Overview

As part of CSE 274 (Image-based Rendering) at UC San Diego, I implemented the Light Field Rendering algorithm described by Levoy and Hanrahan in their seminal paper. In this report, I'll go over the basic theory, the implementation process, and some of my results.

Theory

In their 1996 paper, Levoy and Hanrahan noted that the plenoptic function (initially 7D) could be reduced to 4D in free space and fixed time, making it possible to sample it. They described multiple parameterizations of lines in 4D, with the "best" being that of a two-plane (light slab) parameterization; that is, lines in space are parameterized by their intersection points with two planes (the uv plane and st plane). Capturing a set of images from a grid of positions on one plane, looking at the other, would allow novel views to be reconstructed from arbitrary camera positions and directions (though the camera would have to be near the uv plane for best results). This could be extended to multiple light slabs for 360 degree views of a scene. Levoy and Hanrahan implemented this algorithm by storing a compressed version of vectors corresponding to pixels in the input images. In my implementation, I did not use compression and simply sampled pixels from the original images as needed.

Implementation

The goal of my implementation was produce high-quality novel views in a variety of scenes, both rendered and real. I also wanted to support multiple light slabs. My implementation process was iterative, and ran as follows:

Selecting a renderer for image capture: I intended to develop my algorithm using rendered images, as this would be easier than immediately attempting to generate novel views of real scenes. I decided on using Unity Engine as my rasterizer of choice, and wrote a script that would move the camera along a defined uv plane and take photos of a scene.
Core algorithm, one plane: I next wrote a sampler script in python that synthesizes a novel view, with each pixel being a bilinear interpolation of its closest matches among source images. I initially implemented a one plane parameterization (u, v, theta, phi).
Two plane (light slab): I extended my algorithm to support a (u, v, s, t) parameterization. In conjunction, I tweaked my Unity script to use a sheared perspective projection to take photos of the st plane.
Multiple slabs: I extended my algorithm to support multiple slabs, and to pick a slab for the evaluation of every pixel. This means that one image can draw from multiple slabs, as desired.
Real imagery: I drew fron the Stanford Lightfield Dataset and rendered novel views of real scenes. I had to write some more scripts to standardize image names before they could be processed by my sampling algorithm.
Speedup?: I attempted to use Numba JIT compilation for speedup, but gains were very limited. Perhaps in future I could try adding multiprocessing support.

Github repository: will be posted once I get a chance to write a gitignore and do a refactor of some messy code

Results

Captions (top to bottom, left to right):

First image rendered with my sampling implementation; uses a u,v,theta,phi parameterization and is very blurry.
Same scene, novel view, rendered with a two plane implementation and denser source images.
Screenshot from Unity showing 4-slab setup. Same scene; house and two pagodas.
Novel view from 4-slab capture. Image taken from corner of two slabs, and synthesizes both slabs.
Apple (rendered), novel view from in front of original uv plane.
Apple (rendered), novel view from behind original uv plane.
Buddha (rendered), novel view, also featured in original Levoy and Hanrahan paper.
Different novel view of Buddha.
Novel view of Tarot cards, real imagery. From Stanford light field dataset.
Novel view of lego truck, real imagery. From Stanford light field dataset.

Side-by-side Ground Truth comparisons

The latter scene has significantly wider spacing of source images on the UV plane, and the scene has more depth, making the render blurrier.

Evaluation Metrics

There isn't very much to evaluate the implementation on, but I think it turned out fairly well.

Image Quality: The image quality is good and is indistinguishable from ground truth images when the source image spacing is narrow and when the scene does not have much depth. The sampling is at its best at or near the st plane, and blurriness increases as you move away from it. This effect is especially pronounced in the images of the large scene with the pagodas and house, as well as in the tarot cards image.
Compression: In order to focus on the core algorithm, I did not implement compression.
Speed: I wrote a cache to store recently loaded images in order to reduce I/O calls, which greatly improved speed. Even so, my algorithm takes 10-30 seconds to render a 1280 x 720 image, which is significantly slower than Levoy and Hanrahan. The difference can probably be chalked up to my lack of compression, which means I have to load in a whole image if I need to access one of its pixels. Python being a slow interpreted language may also be another reason. I attempted to refactor my code to use Numba's JIT compilation tool, but this did not result in notable speedup. Multiprocessing would almost certainly help, and it may be something I'll add going forward.
Multiple slabs: My multi-slab implementation appears to work fairly robustly, and synethesizes 2 slabs into 1 image with no problem. I have used it to generate novel views from all around the scene.
Original Images: Levoy and Hanrahan used a buddha and a lion to test their algorithm in the original 1996 paper. I could not find the lion, but I was able to download the buddha and generate novel views of similar quality (above). They did not describe the positioning of the camera that they used for the views they published in the paper, so I could not render an exact match for comparison.