This is an archive of a past semester of this course. Go to the current semester.
Lens Simulator

Due Date

Mon April 11th, 11:59pm

Overview

You will augment your pathtracer from assignment 3 by adding a realistic camera lens simulator and autofocus! This will allow you to programmatically recreate photographic phenomena such as depth of field and to investigate the different perspective provided by lenses of differing focal lengths.

Project Parts

This project is composed of two required parts and one extra credit part:

  • Part 1: Tracing Rays through Lenses
  • Part 2: Contrast-based Autofocus
  • Part 3: Faster contrast-based autofocus (Extra Credit)

All parts are equally weighted. You'll also need to read this article:

Using the program

Download the zipped assignment or clone it from GitHub:

git clone https://github.com/CS184-sp16/asst4_lenssim.git

As before, use cmake and make inside a build/ directory to create the executable (help article).

Heads up: you'll have to copy over selected files and functions from your version of Assignment 3 in order to start this assignment! Instructions are below.

Command line options

Below there is a repost of all the info from the previous project. However you should note some new features!!

New features: lenstester

There is a whole new executable! It is named lenstester and takes no arguments. This will help you greatly when debugging your lens ray tracing. In this program you can

  • Click to set a fixed point, then move your mouse around to control the endpoint of a ray (or ray bundle). If your fixed point is right of center, you are tracing forwards through the lens. If it is left of center, you are tracing backwards through the lens.
  • Hit 1-4 to switch between the four included lenses.
  • Use - and = to change the number of rays in the ray bundle, and use [ and ] to change the spacing of the rays in the ray bundle.
  • Scroll to zoom in and out.

You don't have to change any code for this program -- it's purely to help you debug. Of course, you're welcome to fiddle with the code if it helps you (or interests you).

New features: pathtracer GUI

All these features apply to render mode (the mode you enter by pressing R).

  • You can dump current camera view settings to a file with the D key. You can load them from the command line using the -c flag.
  • If you press C, you enter cell render mode. Now you can drag a small window with your mouse, which will be highlighted in red. Whenever you use hotkeys or re-press R, only the small window will render, not the whole screen. This is useful for eyeballing focus quality or just overall render quality at points of interest (like caustics or reflections or edges) without having to waste cycles rendering the whole scene.
  • When in cell mode, you can press A to print out your focus quality metrics (after implementing them).
  • When in cell mode, you can press F to run your contrast-based autofocus algorithm on the currently highlighted cell (after implementing it).
  • You can press ; and ' to move the sensor plane of the camera back and forth behind the lens. The new position will be printed out on the command line. This is useful when trying to "manually" focus an image before you have autofocus completed. Use this with cell mode to be most efficient.
  • You can press Z and X respectively to increase and decrease the aperture of the current lens.
  • You can press 0-4 to switch between pinhole camera mode (0) and the four new lenses (1-4).

Same old features

Use these flags between the executable name and the dae file when you invoke the program. For example,

./pathtracer -t 8 -s 64 -l 16 -m 6 -r 480 360 -f spheres.png ../dae/sky/CBspheres.dae

Don't blindly run the program with these arguments! You should use numbers that make sense given what you're trying to see. Also make sure to use the live view and hotkeys. They will save you a ton of time when debugging.

For this assignment, we've provided a windowless run mode, which is triggered by providing a filename with the -f flag. The program will run in this mode when you are ssh-ed into the instructional machines.

This means that when trying to generate high quality results for your final writeup, you can use the windowless mode to farm out multi-hour render jobs to the s349 machines! You'll probably want to use screen to keep your jobs running after you logout of ssh. After the jobs complete, you can view them using the display command, assuming you've ssh-ed in with graphics forwarding enabled (by using the -X flag).

Also, please take note of the -t flag! We recommend running with 4-8 threads almost always -- the exception is that you should use -t 1 when debugging with print statements, since printf and cout are not thread safe.

Flag and parameters Description
-s <INT> Number of camera rays per pixel (default=1, should be a power of 2)
-l <INT> Number of samples per area light (default=1)
-t <INT> Number of render threads (default=1)
-m <INT> Maximum ray depth (default=1)
-f <FILENAME> Image (.png) file to save output to in windowless mode
-r <INT> <INT> Width and height of output image (if windowless)
-c <FILENAME> Camera settings file from which to load initial camera viewpoint
-h Print command line help message

Moving the camera (in edit and BVH mode)

Command Action
Rotate Left-click and drag
Translate Right-click and drag
Zoom in and out Scroll
Reset view Spacebar

Cell selection

Command Action
Toggle cell render mode Press C
Select cell Click and drag

Keyboard commands

Command Keys
Mesh-edit mode (default) E
BVH visualizer mode V
Descend to left/right child (BVH viz) LEFT/RIGHT
Move up to parent node (BVH viz) UP
Start rendering R
Save a screenshot S
Decrease/increase area light samples - +
Decrease/increase camera rays per pixel [ ]
Decrease/increase maximum ray depth < >
Enter cell render mode C
Print focus metrics on selected area of screen A
Run autofocus algorithm on selected area of screen F
Change sensor plane position to adjust focus manually ; '
Save current camera settings to disk D
Change aperture of current lens Z X

Part 0: Set up code

In order to get this new version of pathtracer up and running, you will need to copy over the version of these files you filled in for your previous assignment (make sure you copy the entire file):

  • Part 1: triangle.cpp, triangle.h, sphere.cpp, sphere.h
  • Part 2: bvh.cpp, bvh.h, bbox.cpp
  • Part 5: bsdf.cpp

Additionally, you will have to copy these specific functions into the new pathtracer.cpp file:

  • estimate_direct_illumination
  • estimate_indirect_illumination

Do not overwrite the rest of pathtracer.cpp! It has many important changes.

We have generously provided implementations of PathTracer::raytrace_pixel and Camera::generate_ray, so you don't need to copy those :)

Part 1: Tracing Rays through Lenses

Intro

First, examine the .dat files in the lenses/ directory, and make sure you understand what the parameters mean. We parse these files for you, but you should have a good grasp on what is going on before attempting to write the lens simulator. Reference the slides from this lecture and consult A realistic camera model for computer graphics by Kolb, et al., for the details.

Look through the new lenscamera.h/.cpp files. This is where all your work will be happening. The LensCamera class has the ability to trace rays through one of four lenses (or to use the old pinhole model). These lenses are stored as instances of a Lens class, each of which owns a vector of LensElement objects (stored from back to front) encoding the set of interfaces making up that lens. (Think of this like having a DSLR and a bag full of detachable lenses! Except virtual of course.)

Each Lens additionally has a whole host of other parameters, encoding its back element location, infinity focus, near focus, focal length, aperture stop, and current focus setting (i.e., sensor depth). Some of these are calculated upon loading the lens file (back element location, infinity focus, near focus, and focal length) and others are modified while rendering to model what a real camera would do (modify focus depth and aperture size).

The .dat files are loaded up and then processed so that the furthest back lens element (closest to the sensor) is stored first. Additionally, the parameters are regularized so that the spherical center of each element is stored rather than the distance between LensElements. The aperture element is always moved to be at the z=0 plane. Sensor-to-world ray traversal proceeds along the negative z direction, just like in the previous assignment's camera model. The z axis lies along the optical center of each lens.

All parameters in the LensCamera are measured in millimeters.

Task 1: Tracing

The core tracing functions are in LensElement. The publicly exposed function pass_through passes the given ray through the lens element, returning true if the ray successfully passes through. It is provided with the index of refraction of the previous material (the material the ray was coming from). The job of this function is to

  1. Intersect the ray with the spherical lens element (returning false if intersection fails). Additionally, return false if the intersection point is farther than aperture/2 away from the z-axis (since aperture is the diameter of that lens element).
  2. Refract the ray using Snell's law given its intersection point, prev_ior, and ior. Ignore Fresnel effects, and return false if total internal reflection occurs.
  3. Update prev_ior to be equal to ior. r should now represent the refracted ray before returning true. This hopefully happens automatically since r is passed around by reference everywhere (so you should return the new ray by modifying r itself).

Note that there is a special lens element (the aperture) with radius 0. This element should not be treated as a spherical piece of glass -- it is merely a planar element with a given diameter (aperture) and you should add in the appropriate special case code to simply intersect the ray with the appropriate plane and check if that point of intersection lies within the given diameter.

Once the LensElement functions are complete, fill in trace and trace_backwards in the Lens class. These should be super short! When doing this, remember that the lens elements are stored from back to front. This means that in the normal trace function, you should be iterating from the 0th element to the size-1-th element. Notes:

  • The trace parameter is an optional vector recording the list of r.o values that occured during refraction. It is critical for using the debug lenstester tool, so you will want to fill it in. After each refraction, you should simply push r.o on the back of the trace list.
  • The trace_backwards function is a bit tricky. The pass_through function will not give the convenient prev_ior in this case, so you will have to pull it out from the previous element by hand in each loop iteration.

Edit: it should be noted that you will need to be careful about your intersection and refraction functions in order to get them working seamlessly with trace_backwards! You can identify a forwards or backwards ray by testing r.d.z < 0 (true is forwards, false is backwards). Then, intersection can pick its correct quadratic root by testing r.d.z * radius > 0 ? t1 : t2 (assuming t1 < t2). In refraction, you will want to flip normals and flip the ratio of prev_ior/ior for a backwards ray.

Once at least the LensElement functions and trace are completed, you should be able to begin to use the lenstester to debug.

Task 2: Lens and LensCamera helper functions

Implement set_focus_params:

  • set_focus_params is responsible for setting infinity_focus, near_focus, and focal_length.

  • Calculate the infinity_focus sensor depth by finding the focal point (at $F'$ in the figure above). As shown in the figure, a parallel object-side ray (conceptually starting infinitely far away on the optical axis) that passes through the aperture at a small (paraxial) distance from the axis) will intersect the optical axis on the image-side of the lens at the focal point (conjugate to object infinity). You can trace this ray using the trace_backwards function (trace rays back from the world into the lens)

  • Similarly, you should compute the near_focus sensor depth, which is the image-side position that is conjugate to the closest object focus depth for the lens. (Self check: will near_focus be to the left or right of $F'$ in the figure above?) Real lenses have a closest possible object depth that their focus mechanisms can achieve. You will compute the near_focus sensor depth for each lens by using the trace_backwards function on a ray that starts at the closest object focus position on the optical axis and passes through the lens at a paraxial angle. The intersection of the ray with the optical axis on the image side is the near_focus value. You should use the following closest object focus depths for the 10, 22, 50 and 250 mm lenses: 0.05m, 0.20m, 0.30m, 1.0m. Edit: Inserting these fixed constants does not work so well with the code, so instead we will recommend that you substitute something like 5 times the lens' focal length as the closest object focus depths.
  • The focal_length of the lens can be calculated as $|P'-F'|$ referring to the figure above again. The solid line through the lens is the real path of the ray that you computed to calculate infinity_focus with trace_backwards. The dotted line gives the location of the plane $P'$ at which the ray apparently refracted. ($P'$ is one of the principal planes of a thick-lens approximation for the lens. $P$ and $F$ are the principal plane and the focal point corresponding to rays passing through the lens in the opposite direction. This is an ideal thick lens model (as opposed to the ideal thin lens approximation discussed in class)m because $P \neq P'$).

Implement focus_depth:

  • focus_depth computes the object-side conjugate of a sensor depth (i.e. the depth in the world corresponding to the by tracing a ray originating from that sensor depth on the z axis through the lens (and then reintersecting with the z axis). Use the trace function.

Implement back_lens_sample:

  • back_lens_sample returns a point sampled on the 2D circle corresponding to the back of the lens element nearest the sensor (which is elts[0] in this case). Remember that you can get the z-axis position of this lens element by taking elts[0].center - elts[0].radius, and that the element's radius is elts[0].aperture * 0.5. Sample uniformly on the circle, don't worry about the 3D shape of the lens element.

Finally you need to implement generate_ray once more. This functions should generate a ray from the given sensor position and then trace it through the lens. Notes:

  • The sensor lies at sensor_depth along the z axis. The (x,y) position of the sensor point can be calculated by assuming the sensor has the same diagonal as a standard "full-frame" sensor (36x24mm area), adjusted for the current screen aspect ratio (calculate this using screenW and screenH).

Edit: This has been deemed unclear, so here is what I mean in code:

double film_d = sqrt(24*24+36*36);
double screen_d = sqrt(screenW*screenW + screenH*screenH);
double film_w = film_d * screenW / screen_d;
double film_h = film_d * screenH / screen_d;
Vector3D sensor_point(-(x-0.5)*film_w, -(y-0.5)*film_h, sensor_depth);
  • Use back_lens_sample to get a point to trace towards.
  • We provide the transform from camera space to world space at the bottom of the starter code generate_ray function. It has some extra parts, since this time the ray can have a nonzero origin, which must be transformed and scaled (to convert from millimeters to meters).

The provided code:

r.o = pos + c2w * r.o * scale;
r.d = (c2w * r.d).unit();

Edit: what to do when your ray doesn't make it through the lens? We need to do something principled to ensure that your radiance estimate remains unbiased. Multiple options:

  1. Return a ray that won't hit anything (backwards ray with direction c2w*Vector3D(0,0,1) should work for our scenes). This is a bit of a hack.
  2. The trickier but better option for noise reduction option is to change the function declaration of generate_ray to include an int& rays_tried parameter. To do this you need to change camera.h/camera.cpp/lenscamera.h/lenscamera.cpp/pathtracer.cpp. If you try multiple rays in a while loop then you should return number of rays tried here (so 1 if the first is successful, 2 if the first fails but second succeeds, etc.). Then, in raytrace_pixel in your pathtracer, you capture this value with a local int variable in each iteration of your pixel sampling loop. You add all of these up to get total rays attempted and divide by this instead of the sample-per-pixel count since every failed ray is essentially a black pixel. This will give you the correct vignetting while reducing noise. Caveat: on the fisheye at least, some areas of the sensor physically cannot trace a ray through the lens (they lie outside the lens circle)! What now? To address this, have a condition that breaks out of your loop in generate_ray once you have tried something like 10 or 20 rays. Else your loop will never exit. In this case, go back to the (1) method: return a backwards ray (and also the correct rays_tried value).
  3. Note that if you are doing (2), you can pretty easily add the cos^4(theta) part as well! The correct cosine to take is simply your original sensor-to-back-element ray's d.z (after normalization of d), raised to the fourth. You can add this as a double& parameter to generate_ray, passing it back to raytrace_pixel, which should multiply the output of trace_ray by this factor. If you do this, note it in your writeup for extra credit :) Also, you can remove the backwards ray hack in this manner by simply setting the cosine factor to 0 when you are returning an invalid ray. Then your raytrace_pixel should test for cos^4 > 0 and simply not trace the returned ray if that test fails. (You still need to add in the rays_tried to your sample count though!)

Once generate_ray is complete, you should be able to see results in your pathtracer program by rendering and using the 1-4 number keys. They'll be super blurry, but you can try to focus them using ; and ' to adjust the sensor position (until you have an autofocus routine implemented!).

Edit New CB*.dae scene files posted to the GitHub repo. Please download. Here is the new CBdragon.dae rendered at the default view, infinity focus, 1spp/1splight/15 max ray depth, through each of the four lenses:

First without cosine factor and correct ray weighting

Now including cosine factor and correct ray weighting

Part 2: Contrast-based autofocus

Now your camera works! You can try to manually focus it using ; and '. However, wouldn't it be cool if it focused for you automatically?

For part 2, you first need to come up with a way to judge how "in focus" a patch of an image is. Then, you will use a global search heuristic to find the sensor depth where the sharpest focus is achieved for the current image patch. The patch in consideration is selected by you using cell render mode (and then pressing F).

Important tips on how to use autofocus

  • Use high sampling to create a low-noise image patch, e.g. 8 pixel and 8 light samples. The function PathTracer::bump_settings() does this for you automatically if you call it, by setting by sample parameters to 16.
  • Use a small image patch (e.g. 20x20 pixels) to keep things efficient, especially on part 2.
  • Note that this is called contrast-based autofocus. It only works on areas of high contrast! Basically, the premise is that we try a bunch of potential sensor depths and see which one is the least blurry. This means that there needs to be some high frequency content in the patch, e.g. an object edge.

Task 1: A simple focus metric - variance

Implement focus_metric, which takes an ImageBuffer instance and computes a value that is higher when the image is more in focus. For this task we want you to implement the metric as the variance of the image patch. Blurrier (out of focus) image patches will have lower variance, and sharper ones will have higher variance (why?). This is easily confused with noise however, so make sure to pay attention to tip 1 above.

Some helper functions at the top of lenscamera.cpp allow you to extract individual color channels from the ImageBuffer uint32_t data members. Starter code mean_green shows a demo of how to work with the ImageBuffer.

You can evaluate your metric on the currently selected render cell (with the current sample rate and ray depth settings) by pressing A in cell render mode. The metric will be written to the command line.

Task 2: Autofocus search

Implement function autofocus, whose job is to estimate the depth where the image patch has the highest focus metric. For this task, we want you to simply step through all valid sensor depths and test the focus metric, then set curr_lens().sensor_depth to the best depth you found. That is:

  • Step between sensor depths from infinity_focus to near_focus
  • Think about what step size you should choose. Hint: what is the range of sensor depths where an image point will blur within one output pixel?
  • Compute the metric at each sensor position and return the best position.

Notes:

  • The starter code that is provided demonstrates how to request an image patch from the PathTracer *pt.
  • This code is triggered by pressing F in cell render mode. Whatever patch you have selected will be used for autofocus.
  • Pay attention to the tips on image patch size and high-quality sampling above. Autofocus will have trouble if the patch is too low contrast or too noisy. Even with a small patch, this algorithm may take a minute or two to complete.

Part 3: Faster contrast-based autofocus (Extra credit)

For this part your job is to improve the quality and speed of your contrast autofocus algorithm as you see fit.

Some ideas for modifying the search algorithm:

  • You could try a non-uniform or recursive scheme for finding the focus position with fewer image patch renders, or with more efficient patch renders.
  • Perhaps you could exploit the fact that focus will improve as you approach the correct image depth and then degrade as you move away.

Some ideas for modifying the focus metric:

  • You could experiment with a more robust metric. For example, you could try the Sum-Modified Laplacian metric from this paper on Shape from Focus.
  • You could modify your metric using a total variation metric to try to reduce the effects of noise and allow you to render more quickly with fewer samples.

As usual, describe in your write-up what algorithms and experiments you tried, what results you got, and what you learned.

Submission Instructions

Log in to an instructional machine, navigate to the root directory asst4_lenssim of your project, and run

zip -r hw4.zip src
zip -r hw4.zip website

Make sure that all images referenced by your website are inside the website/images/ directory! Please also include any new or modified .dae files you create in a dae/ directory. Also, don't forget your competition.png image! You can inspect what's inside your zip file by running

unzip -l hw4.zip

When your zip file is ready, run

submit hw4

This indicates you have succeeded:

Looking for files to turn in....
Submitting hw4.zip.
The files you have submitted are:
    ./hw4.zip 
Is this correct? [yes/no] yes
Copying submission of assignment hw4....
Submission complete.

There are more detailed instructions (including how you can use scp to copy your files onto the s349 machines) here.