Mon April 11th, 11:59pm
You will augment your pathtracer from assignment 3 by adding a realistic camera lens simulator and autofocus! This will allow you to programmatically recreate photographic phenomena such as depth of field and to investigate the different perspective provided by lenses of differing focal lengths.
This project is composed of two required parts and one extra credit part:
- Part 1: Tracing Rays through Lenses
- Part 2: Contrast-based Autofocus
- Part 3: Faster contrast-based autofocus (Extra Credit)
All parts are equally weighted. You'll also need to read this article:
Using the program
git clone https://github.com/CS184-sp16/asst4_lenssim.git
As before, use cmake and make inside a build/ directory to create the executable (help article).
Heads up: you'll have to copy over selected files and functions from your version of Assignment 3 in order to start this assignment! Instructions are below.
Command line options
Below there is a repost of all the info from the previous project. However you should note some new features!!
New features: lenstester
There is a whole new executable! It is named lenstester and takes no arguments. This will help you greatly when debugging your lens ray tracing. In this program you can
- Click to set a fixed point, then move your mouse around to control the endpoint of a ray (or ray bundle). If your fixed point is right of center, you are tracing forwards through the lens. If it is left of center, you are tracing backwards through the lens.
4to switch between the four included lenses.
=to change the number of rays in the ray bundle, and use
]to change the spacing of the rays in the ray bundle.
- Scroll to zoom in and out.
You don't have to change any code for this program -- it's purely to help you debug. Of course, you're welcome to fiddle with the code if it helps you (or interests you).
New features: pathtracer GUI
All these features apply to render mode (the mode you enter by pressing
- You can dump current camera view settings to a file with the
Dkey. You can load them from the command line using the
- If you press
C, you enter cell render mode. Now you can drag a small window with your mouse, which will be highlighted in red. Whenever you use hotkeys or re-press
R, only the small window will render, not the whole screen. This is useful for eyeballing focus quality or just overall render quality at points of interest (like caustics or reflections or edges) without having to waste cycles rendering the whole scene.
- When in cell mode, you can press
Ato print out your focus quality metrics (after implementing them).
- When in cell mode, you can press
Fto run your contrast-based autofocus algorithm on the currently highlighted cell (after implementing it).
- You can press
'to move the sensor plane of the camera back and forth behind the lens. The new position will be printed out on the command line. This is useful when trying to "manually" focus an image before you have autofocus completed. Use this with cell mode to be most efficient.
- You can press
Xrespectively to increase and decrease the aperture of the current lens.
- You can press
4to switch between pinhole camera mode (
0) and the four new lenses (
Same old features
Use these flags between the executable name and the dae file when you invoke the program. For example,
./pathtracer -t 8 -s 64 -l 16 -m 6 -r 480 360 -f spheres.png ../dae/sky/CBspheres.dae
Don't blindly run the program with these arguments! You should use numbers that make sense given what you're trying to see. Also make sure to use the live view and hotkeys. They will save you a ton of time when debugging.
For this assignment, we've provided a windowless run mode, which is triggered by providing a filename with the
-f flag. The program will run in this mode when you are ssh-ed into the instructional machines.
This means that when trying to generate high quality results for your final writeup, you can use the windowless mode to farm out multi-hour render jobs to the s349 machines! You'll probably want to use screen to keep your jobs running after you logout of ssh. After the jobs complete, you can view them using the display command, assuming you've ssh-ed in with graphics forwarding enabled (by using the
Also, please take note of the
-t flag! We recommend running with 4-8 threads almost always -- the exception is that you should use
-t 1 when debugging with print statements, since
cout are not thread safe.
|Flag and parameters||Description|
||Number of camera rays per pixel (default=1, should be a power of 2)|
||Number of samples per area light (default=1)|
||Number of render threads (default=1)|
||Maximum ray depth (default=1)|
||Image (.png) file to save output to in windowless mode|
||Width and height of output image (if windowless)|
||Camera settings file from which to load initial camera viewpoint|
||Print command line help message|
Moving the camera (in edit and BVH mode)
|Rotate||Left-click and drag|
|Translate||Right-click and drag|
|Zoom in and out||Scroll|
|Toggle cell render mode||Press
|Select cell||Click and drag|
|Mesh-edit mode (default)||
|BVH visualizer mode||
|Descend to left/right child (BVH viz)||
|Move up to parent node (BVH viz)||
|Save a screenshot||
|Decrease/increase area light samples||
|Decrease/increase camera rays per pixel||
|Decrease/increase maximum ray depth||
|Enter cell render mode||
|Print focus metrics on selected area of screen||
|Run autofocus algorithm on selected area of screen||
|Change sensor plane position to adjust focus manually||
|Save current camera settings to disk||
|Change aperture of current lens||
Part 0: Set up code
In order to get this new version of
pathtracer up and running, you will need to copy over the version of these files you filled in for your previous assignment (make sure you copy the entire file):
- Part 1: triangle.cpp, triangle.h, sphere.cpp, sphere.h
- Part 2: bvh.cpp, bvh.h, bbox.cpp
- Part 5: bsdf.cpp
Additionally, you will have to copy these specific functions into the new pathtracer.cpp file:
Do not overwrite the rest of pathtracer.cpp! It has many important changes.
We have generously provided implementations of
Camera::generate_ray, so you don't need to copy those :)
Part 1: Tracing Rays through Lenses
First, examine the .dat files in the lenses/ directory, and make sure you understand what the parameters mean. We parse these files for you, but you should have a good grasp on what is going on before attempting to write the lens simulator. Reference the slides from this lecture and consult A realistic camera model for computer graphics by Kolb, et al., for the details.
Look through the new lenscamera.h/.cpp files. This is where all your work will be happening. The
LensCamera class has the ability to trace rays through one of four lenses (or to use the old pinhole model). These lenses are stored as instances of a
Lens class, each of which owns a vector of
LensElement objects (stored from back to front) encoding the set of interfaces making up that lens. (Think of this like having a DSLR and a bag full of detachable lenses! Except virtual of course.)
Lens additionally has a whole host of other parameters, encoding its back element location, infinity focus, near focus, focal length, aperture stop, and current focus setting (i.e., sensor depth). Some of these are calculated upon loading the lens file (back element location, infinity focus, near focus, and focal length) and others are modified while rendering to model what a real camera would do (modify focus depth and aperture size).
The .dat files are loaded up and then processed so that the furthest back lens element (closest to the sensor) is stored first. Additionally, the parameters are regularized so that the spherical center of each element is stored rather than the distance between LensElements. The aperture element is always moved to be at the
z=0 plane. Sensor-to-world ray traversal proceeds along the negative z direction, just like in the previous assignment's camera model. The z axis lies along the optical center of each lens.
All parameters in the
LensCamera are measured in millimeters.
Task 1: Tracing
The core tracing functions are in
LensElement. The publicly exposed function
pass_through passes the given ray through the lens element, returning true if the ray successfully passes through. It is provided with the index of refraction of the previous material (the material the ray was coming from). The job of this function is to
- Intersect the ray with the spherical lens element (returning false if intersection fails). Additionally, return false if the intersection point is farther than
aperture/2away from the z-axis (since
apertureis the diameter of that lens element).
- Refract the ray using Snell's law given its intersection point,
ior. Ignore Fresnel effects, and return false if total internal reflection occurs.
prev_iorto be equal to
rshould now represent the refracted ray before returning true. This hopefully happens automatically since
ris passed around by reference everywhere (so you should return the new ray by modifying
Note that there is a special lens element (the aperture) with radius 0. This element should not be treated as a spherical piece of glass -- it is merely a planar element with a given diameter (
aperture) and you should add in the appropriate special case code to simply intersect the ray with the appropriate plane and check if that point of intersection lies within the given diameter.
LensElement functions are complete, fill in
trace_backwards in the
Lens class. These should be super short! When doing this, remember that the lens elements are stored from back to front. This means that in the normal
trace function, you should be iterating from the 0th element to the size-1-th element. Notes:
traceparameter is an optional vector recording the list of
r.ovalues that occured during refraction. It is critical for using the debug lenstester tool, so you will want to fill it in. After each refraction, you should simply push
r.oon the back of the
trace_backwardsfunction is a bit tricky. The
pass_throughfunction will not give the convenient
prev_iorin this case, so you will have to pull it out from the previous element by hand in each loop iteration.
Edit: it should be noted that you will need to be careful about your intersection and refraction functions in order to get them working seamlessly with
trace_backwards! You can identify a forwards or backwards ray by testing
r.d.z < 0 (true is forwards, false is backwards). Then, intersection can pick its correct quadratic root by testing
r.d.z * radius > 0 ? t1 : t2 (assuming
t1 < t2). In refraction, you will want to flip normals and flip the ratio of
prev_ior/ior for a backwards ray.
Once at least the
LensElement functions and
trace are completed, you should be able to begin to use the lenstester to debug.
Task 2: Lens and LensCamera helper functions
set_focus_paramsis responsible for setting
infinity_focussensor depth by finding the focal point (at $F'$ in the figure above). As shown in the figure, a parallel object-side ray (conceptually starting infinitely far away on the optical axis) that passes through the aperture at a small (paraxial) distance from the axis) will intersect the optical axis on the image-side of the lens at the focal point (conjugate to object infinity). You can trace this ray using the
trace_backwardsfunction (trace rays back from the world into the lens)
- Similarly, you should compute the
near_focussensor depth, which is the image-side position that is conjugate to the closest object focus depth for the lens. (Self check: will
near_focusbe to the left or right of $F'$ in the figure above?) Real lenses have a closest possible object depth that their focus mechanisms can achieve. You will compute the
near_focussensor depth for each lens by using the
trace_backwardsfunction on a ray that starts at the closest object focus position on the optical axis and passes through the lens at a paraxial angle. The intersection of the ray with the optical axis on the image side is the
near_focusvalue. You should use the following closest object focus depths for the 10, 22, 50 and 250 mm lenses: 0.05m, 0.20m, 0.30m, 1.0m. Edit: Inserting these fixed constants does not work so well with the code, so instead we will recommend that you substitute something like 5 times the lens' focal length as the closest object focus depths.
focal_lengthof the lens can be calculated as $|P'-F'|$ referring to the figure above again. The solid line through the lens is the real path of the ray that you computed to calculate
trace_backwards. The dotted line gives the location of the plane $P'$ at which the ray apparently refracted. ($P'$ is one of the principal planes of a thick-lens approximation for the lens. $P$ and $F$ are the principal plane and the focal point corresponding to rays passing through the lens in the opposite direction. This is an ideal thick lens model (as opposed to the ideal thin lens approximation discussed in class)m because $P \neq P'$).
focus_depthcomputes the object-side conjugate of a sensor depth (i.e. the depth in the world corresponding to the by tracing a ray originating from that sensor depth on the z axis through the lens (and then reintersecting with the z axis). Use the
back_lens_samplereturns a point sampled on the 2D circle corresponding to the back of the lens element nearest the sensor (which is
eltsin this case). Remember that you can get the z-axis position of this lens element by taking
elts.center - elts.radius, and that the element's radius is
elts.aperture * 0.5. Sample uniformly on the circle, don't worry about the 3D shape of the lens element.
Finally you need to implement
generate_ray once more. This functions should generate a ray from the given sensor position and then trace it through the lens. Notes:
- The sensor lies at
sensor_depthalong the z axis. The (x,y) position of the sensor point can be calculated by assuming the sensor has the same diagonal as a standard "full-frame" sensor (36x24mm area), adjusted for the current screen aspect ratio (calculate this using
Edit: This has been deemed unclear, so here is what I mean in code:
double film_d = sqrt(24*24+36*36); double screen_d = sqrt(screenW*screenW + screenH*screenH); double film_w = film_d * screenW / screen_d; double film_h = film_d * screenH / screen_d; Vector3D sensor_point(-(x-0.5)*film_w, -(y-0.5)*film_h, sensor_depth);
back_lens_sampleto get a point to trace towards.
- We provide the transform from camera space to world space at the bottom of the starter code
generate_rayfunction. It has some extra parts, since this time the ray can have a nonzero origin, which must be transformed and scaled (to convert from millimeters to meters).
The provided code:
r.o = pos + c2w * r.o * scale; r.d = (c2w * r.d).unit();
Edit: what to do when your ray doesn't make it through the lens? We need to do something principled to ensure that your radiance estimate remains unbiased. Multiple options:
- Return a ray that won't hit anything (backwards ray with direction
c2w*Vector3D(0,0,1)should work for our scenes). This is a bit of a hack.
- The trickier but better option for noise reduction option is to change the function declaration of
generate_rayto include an
int& rays_triedparameter. To do this you need to change camera.h/camera.cpp/lenscamera.h/lenscamera.cpp/pathtracer.cpp. If you try multiple rays in a while loop then you should return number of rays tried here (so 1 if the first is successful, 2 if the first fails but second succeeds, etc.). Then, in
raytrace_pixelin your pathtracer, you capture this value with a local int variable in each iteration of your pixel sampling loop. You add all of these up to get total rays attempted and divide by this instead of the sample-per-pixel count since every failed ray is essentially a black pixel. This will give you the correct vignetting while reducing noise. Caveat: on the fisheye at least, some areas of the sensor physically cannot trace a ray through the lens (they lie outside the lens circle)! What now? To address this, have a condition that breaks out of your loop in
generate_rayonce you have tried something like 10 or 20 rays. Else your loop will never exit. In this case, go back to the (1) method: return a backwards ray (and also the correct
- Note that if you are doing (2), you can pretty easily add the
cos^4(theta)part as well! The correct cosine to take is simply your original sensor-to-back-element ray's
d.z(after normalization of
d), raised to the fourth. You can add this as a
generate_ray, passing it back to
raytrace_pixel, which should multiply the output of
trace_rayby this factor. If you do this, note it in your writeup for extra credit :) Also, you can remove the backwards ray hack in this manner by simply setting the cosine factor to 0 when you are returning an invalid ray. Then your
raytrace_pixelshould test for cos^4 > 0 and simply not trace the returned ray if that test fails. (You still need to add in the
rays_triedto your sample count though!)
generate_ray is complete, you should be able to see results in your pathtracer program by rendering and using the
4 number keys. They'll be super blurry, but you can try to focus them using
' to adjust the sensor position (until you have an autofocus routine implemented!).
Edit New CB*.dae scene files posted to the GitHub repo. Please download. Here is the new CBdragon.dae rendered at the default view, infinity focus, 1spp/1splight/15 max ray depth, through each of the four lenses:
First without cosine factor and correct ray weighting
Now including cosine factor and correct ray weighting
Part 2: Contrast-based autofocus
Now your camera works! You can try to manually focus it using
'. However, wouldn't it be cool if it focused for you automatically?
For part 2, you first need to come up with a way to judge how "in focus" a patch of an image is. Then, you will use a global search heuristic to find the sensor depth where the sharpest focus is achieved for the current image patch. The patch in consideration is selected by you using cell render mode (and then pressing
Important tips on how to use autofocus
- Use high sampling to create a low-noise image patch, e.g. 8 pixel and 8 light samples. The function
PathTracer::bump_settings()does this for you automatically if you call it, by setting by sample parameters to 16.
- Use a small image patch (e.g. 20x20 pixels) to keep things efficient, especially on part 2.
- Note that this is called contrast-based autofocus. It only works on areas of high contrast! Basically, the premise is that we try a bunch of potential sensor depths and see which one is the least blurry. This means that there needs to be some high frequency content in the patch, e.g. an object edge.
Task 1: A simple focus metric - variance
focus_metric, which takes an
ImageBuffer instance and computes a value that is higher when the image is more in focus. For this task we want you to implement the metric as the variance of the image patch. Blurrier (out of focus) image patches will have lower variance, and sharper ones will have higher variance (why?). This is easily confused with noise however, so make sure to pay attention to tip 1 above.
Some helper functions at the top of lenscamera.cpp allow you to extract individual color channels from the ImageBuffer uint32_t data members. Starter code
mean_green shows a demo of how to work with the ImageBuffer.
You can evaluate your metric on the currently selected render cell (with the current sample rate and ray depth settings) by pressing
A in cell render mode. The metric will be written to the command line.
Task 2: Autofocus search
autofocus, whose job is to estimate the depth where the image patch has the highest focus metric. For this task, we want you to simply step through all valid sensor depths and test the focus metric, then set
curr_lens().sensor_depth to the best depth you found. That is:
- Step between sensor depths from
- Think about what step size you should choose. Hint: what is the range of sensor depths where an image point will blur within one output pixel?
- Compute the metric at each sensor position and return the best position.
- The starter code that is provided demonstrates how to request an image patch from the
- This code is triggered by pressing
Fin cell render mode. Whatever patch you have selected will be used for autofocus.
- Pay attention to the tips on image patch size and high-quality sampling above. Autofocus will have trouble if the patch is too low contrast or too noisy. Even with a small patch, this algorithm may take a minute or two to complete.
Part 3: Faster contrast-based autofocus (Extra credit)
For this part your job is to improve the quality and speed of your contrast autofocus algorithm as you see fit.
Some ideas for modifying the search algorithm:
- You could try a non-uniform or recursive scheme for finding the focus position with fewer image patch renders, or with more efficient patch renders.
- Perhaps you could exploit the fact that focus will improve as you approach the correct image depth and then degrade as you move away.
Some ideas for modifying the focus metric:
- You could experiment with a more robust metric. For example, you could try the Sum-Modified Laplacian metric from this paper on Shape from Focus.
- You could modify your metric using a total variation metric to try to reduce the effects of noise and allow you to render more quickly with fewer samples.
As usual, describe in your write-up what algorithms and experiments you tried, what results you got, and what you learned.
Log in to an instructional machine, navigate to the root directory asst4_lenssim of your project, and run
zip -r hw4.zip src zip -r hw4.zip website
Make sure that all images referenced by your website are inside the website/images/ directory! Please also include any new or modified .dae files you create in a dae/ directory. Also, don't forget your competition.png image! You can inspect what's inside your zip file by running
unzip -l hw4.zip
When your zip file is ready, run
This indicates you have succeeded:
Looking for files to turn in.... Submitting hw4.zip. The files you have submitted are: ./hw4.zip Is this correct? [yes/no] yes Copying submission of assignment hw4.... Submission complete.
There are more detailed instructions (including how you can use scp to copy your files onto the s349 machines) here.