NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

Abstract

This paper achieved state-of-the-art results in synthesizing views from multiple directions by learning a continuous volumetric scene function from a limited number of input images of a 3D scene. The algorithm uses fully connected layers rather than convolutions. Its input is a five-dimensional coordinate consisting of a spatial location $(x,y,z)$ and the viewing-direction parameters $(\theta,\phi)$ from which that location is observed.

From these five-dimensional coordinates, the network predicts values such as RGB color and density along camera rays. Volume rendering then composites those values back into an image. Because the volume-rendering equation is differentiable, the parameters can be optimized by comparing the synthesized scene with the ground truth.

The figure above provides an overview.

Introduction

Many approaches to view synthesis existed before NeRF. After briefly describing the task, we will examine the other methods in more detail under related work. The direction taken in this paper is parameter optimization—in other words, a deep-learning-based approach. The authors first represent a static scene as a continuous five-dimensional function. Here, each dimension is a factor that identifies a point in the scene: as described above, the coordinate system combines the spatial position $(x,y,z)$ with the viewing direction $(\theta,\phi)$. A fully connected MLP takes this five-dimensional input and predicts volume density—literally density, but in practice interpretable as the likelihood that matter exists at that location—and RGB color. As explained later, RGB varies with the observation direction, so it is a view-dependent color.

NeRF can loosely be thought of as a form of few-shot learning, though the actual number of required samples is not especially small. To learn a continuous representation of one scene, cameras must be positioned throughout the surrounding space, and images captured under this setup are needed for optimization. These samples and their corresponding five-dimensional vectors optimize the network described above. For a camera ray at a particular location, the model first produces output queries; a differentiable function then combines them to synthesize the scene. Comparing the result with the ground truth allows the network parameters to be optimized.

This figure best illustrates the model's optimization process.

Perhaps the paper's most important contribution is that it achieves this with a simple network. It is also remarkable that optimization uses no ground-truth 3D representation at all. The model learns to describe objects continuously in space using only 2D projected images of a 3D scene. Positional encoding is applied to the five-dimensional input to improve the fidelity of the generated images.

Related Work

NeRF was not the only method to use MLP optimization for representing 3D scenes. Earlier studies learned or optimized simple 3D shapes from the distance between a point $(x,y,z)$ and a surface. Compared with methods that optimized explicit 3D representations such as voxel grids and meshes, however, these approaches produced scenes of considerably lower quality. The related work discussed in the paper falls broadly into two branches.

Neural 3D Shape Representations

Research contemporary with NeRF often predicted either a signed distance—the distance from an $(x,y,z)$ coordinate to the object's surface—or an occupancy field indicating whether a region was occupied. These studies could capture the broad outline but struggled to predict fine detail, and they also required a 3D representation. Optimizing a 3D occupancy field with deep learning means representing that field, finding each ray's intersections with it, and predicting a color at every resulting point. Methods that avoided an explicit occupancy field instead emitted a feature vector and continuous RGB color at each coordinate, then used an RNN to determine whether a surface existed at each point along the ray.

Such techniques could still predict complex or high-resolution shapes, but they tended either to optimize only simple forms or to render surfaces that appeared oversmoothed. This likely stemmed from the regularization needed to prevent overfitting while optimizing distance- or occupancy-field losses.

NeRF therefore augments the three-dimensional spatial coordinate with direction information, using the five-dimensional vector $(x,y,z,\theta,\phi)$ to obtain a more photorealistic, high-resolution 2D representation from a specified viewpoint.

View Synthesis and Image-Based Rendering

With densely sampled viewpoints—samples from a nearly continuous range of views—a photorealistic image from a particular perspective can be obtained by interpolating a light field. The details involve camera geometry and are beyond the scope of this post. When the sampling is sparse, computer-vision and graphics research instead reconstructs geometry from the observed images, an area in which substantial progress had already been made.

The best-known representation is mesh-based. Some methods diffuse color over a surface—the sense of diffusion here matches the ordinary term used in generative modeling—while others determine appearance as a function of viewing direction.

In computer graphics, a rasterizer performs operations such as clipping, which transforms geometry into clip space and removes objects outside it; perspective division; back-face culling, which handles surfaces according to their orientation relative to the viewer; surface-normal calculation, which distinguishes the front and back of a mesh from vertex order; viewport transformation, which maps clip space to screen space; and scan conversion, which generates fragments inside mesh triangles and can usually be understood as interpolation. Researchers have made this mesh-visualization pipeline differentiable. Path tracing, widely used in engines such as Unreal Engine, has likewise been formulated in a differentiable way (reference). Image reprojection can then be differentiated by backpropagating through the reverse of the tracer. Its disadvantages are a high risk of falling into local minima and difficulty optimizing complex loss landscapes. The requirement that a scene be represented as a mesh also makes real-world scenes with varied backgrounds and object shapes difficult to synthesize.

Another family of methods optimizes volumetric representations. These produce far fewer artifacts than mesh-based approaches and can preserve more realistic detail. Early work directly aligned observed images with color voxel grids. More recent studies used deep learning to optimize scenes from multiple directions, then constructed a volumetric representation of the collected images. A new view could be generated through alpha compositing or a learned aggregation of the multi-view representation. Other approaches jointly optimized a CNN and a scene-specific voxel grid.

These volume-based optimization techniques produced good results, but scaling them to higher-resolution images introduced a severe trade-off between sampling cost and time.

Scene Representation

This is an excerpt from an earlier figure.

To construct a continuous image representation from a five-dimensional vector, the function outputs an RGB value and density at every location. Because the method uses a deep MLP, it optimizes a parameterized function $F_\Theta$ rather than a fixed deterministic formula. The paper denotes the three-dimensional position vector $(x,y,z)$ by $\mathbf{x}$ and the two-dimensional viewing-direction vector $(\theta,\phi)$ by $\mathbf{d}$. Once the representation has been learned throughout the space, all of that information is encoded in the network. To impose a useful constraint during training, the RGB color $\mathbf{c}$ is predicted from both position and direction, while the density $\sigma$ depends only on position. The reason is that RGB appearance can change with viewing direction, but density at the same location should remain constant.

The figure above demonstrates this directly: the color of the diamond from View 1 differs from that of the corresponding square from View 2. Panel (c) visualizes interpolation over all directions from the trained network as a radiance distribution, showing that color changes continuously with viewing direction.

Volume Rendering

Once trained as described above, the network stores continuous spatial information as a function of location. We must now determine how to construct a scene from those learned values.

A five-dimensional neural radiance field is represented by volume density and the color at each point. For a ray, the volume density $\sigma(\mathbf{x})$ at every point can be understood in terms of the differential probability that the ray terminates at location $\mathbf{x}$. Put more intuitively, it describes the accumulated probability from the observation origin to a particular point $\mathbf{x}$. For a ray parameterized by $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$, we can compute the expected color from the near bound $t_n$ to the far bound $t_f$:

$$C(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\,\sigma(\mathbf{r}(t))\,\mathbf{c}(\mathbf{r}(t),\mathbf{d})\,dt$$ $$T(t) = \exp\left(-\int_{t_n}^t \sigma(\mathbf{r}(s))\,ds\right)$$

As explained above, a scene's color is a function of both viewing direction and position, whereas density takes only position as input. $T(t)$ is the accumulated transmittance up to point $t$: it can be interpreted as the probability that a ray reaches $t$ without colliding with an object—that nothing lies between the origin and that point. Equivalently, $C(\mathbf{r})$ is the expected RGB value along a ray under the density distribution.

Rendering a discretized grid normally uses numerical quadrature because continuous samples are unavailable. If only fixed locations are queried from the MLP, however, the representation's resolution is limited much like that of earlier methods. NeRF instead applies stratified sampling. It partitions a ray's interval evenly into $N$ bins and draws one sample uniformly at random from each bin:

$$t_i \sim \mathcal{U}\left(t_n+\frac{i-1}{N}(t_f-t_n),\;t_n+\frac{i}{N}(t_f-t_n)\right)$$

In other words, a random location is drawn uniformly from each interval of width $1/N$. This lets the model optimize over more varied position samples. Although a discrete accumulation still replaces the integral in computation, random sampling reduces the resulting limitation. The color actually rendered by the model is the approximation $\hat{C}(\mathbf{r})$:

$$\hat{C}(\mathbf{r}) = \sum_{i=1}^N T_i\left(1-\exp(-\sigma_i\delta_i)\right)\mathbf{c}_i$$

where

$$T_i = \exp\left(-\sum_{j=1}^{i-1}\sigma_j\delta_j\right).$$

The quantity $\delta_i$ is the distance between adjacent samples and replaces the differential $dt$.

Optimizing a Neural Radiance Field

We have now covered the overall idea, prior work, and the conceptual method for optimizing the neural network. The authors found, however, that this theory alone did not produce state-of-the-art quality, so they introduced two additional techniques. The first is positional encoding, which lifts the coordinates into additional dimensions; the second is hierarchical sampling.

Positional Encoding

A neural network is a useful way to optimize a function, but fitting $F_\Theta$ directly to $xyz\theta\phi$ unexpectedly discards high-frequency details. Deep networks are biased toward lower-frequency functions, a tendency that can arise from their optimization dynamics. Earlier studies showed that transforming the inputs into a higher-dimensional space can reduce this bias. NeRF therefore splits the computation into two parts: a fixed, non-trainable encoder maps values from $\mathbb{R}$ into a high-dimensional embedding space $\mathbb{R}^{2L}$, and $F'_\Theta$ is optimized on the encoded values.

It uses the familiar sinusoidal positional encoding:

$$\gamma(p) = \left(\sin(2^0\pi p),\cos(2^0\pi p),\ldots,\sin(2^{L-1}\pi p),\cos(2^{L-1}\pi p)\right).$$

The encoding function $\gamma$ is applied independently to all three coordinates $x$, $y$, and $z$, normalized to $[-1,1]$. It is also applied to the direction vector $\mathbf{d}$. The experiments use $L=10$ for the position vector $\mathbf{x}$ and $L=4$ for the direction vector $\mathbf{d}$.

The Transformer is a well-known example of positional encoding. There it supplies inductive bias by representing each token's position; NeRF instead uses the encoding to embed coordinates into a higher-dimensional representation.

Hierarchical Volume Sampling

Hierarchical sampling is a classic sampling strategy. Its purpose here is straightforward: only a tiny fraction of the points along a ray contain matter, much as an image often contains far more background than foreground. Object-detection networks also use hierarchical or balanced sampling to avoid the bias caused by a large disparity between foreground and background examples.

Rather than optimize a single network, NeRF therefore optimizes two networks jointly, called the coarse and fine networks. First, it draws $N_c$ random locations by stratified sampling as described above and evaluates the coarse network to predict $\hat{C}_c(\mathbf{r})$. Those outputs are then used to concentrate additional samples in regions more likely to contain volume. The rendered color can first be expressed as a weighted sum of all sample colors $\mathbf{c}_i$ along the ray:

$$\hat{C}_c(\mathbf{r})=\sum_{i=1}^N w_i\mathbf{c}_i,\qquad w_i=T_i\left(1-\exp(-\sigma_i\delta_i)\right).$$

The weights are normalized into probabilities:

$$\hat{w}_i=\frac{w_i}{\sum_{j=1}^{N_c}w_j}.$$

These per-location probabilities define a second sampling distribution. A high weight means that the coarse network assigns a high probability of matter—high density—to that region, so the method samples it more heavily. Evaluating the fine network with the additional $N_f$ samples produces the final rendered color $\hat{C}_f(\mathbf{r})$. This resembles sampling from a nonuniform distribution, with the sampled weights serving as probabilities for a nonuniform discretization of the full integration interval.

Optimization

Optimization requires RGB images of a scene together with corresponding camera poses, intrinsics, bounds, and related metadata. At each training step, the method randomly samples a batch of camera rays and performs hierarchical sampling through the coarse and fine networks. The loss is simply the squared error between the estimated colors—the rendered pixel values—and the ground truth:

$$L = \sum_{\mathbf{r}\in R}\left(\lVert\hat{C}_c(\mathbf{r})-C(\mathbf{r})\rVert_2^2 + \lVert\hat{C}_f(\mathbf{r})-C(\mathbf{r})\rVert_2^2\right).$$

The notation is as follows:

• $C(\mathbf{r})$: ground-truth color
• $\hat{C}_c(\mathbf{r})$: coarse volume color prediction
• $\hat{C}_f(\mathbf{r})$: fine volume color prediction
• $R$: the ray pool, meaning all rays in the batch

Experimental Details

The experiments use batches of 4,096 rays. Each ray begins with $N_c=64$ coarse samples, from which $N_f=128$ fine samples are drawn. Optimization uses Adam with a learning rate that decays exponentially from $5\times10^{-4}$ to $5\times10^{-5}$. Optimizing one scene takes roughly 100,000 to 300,000 iterations, or about one to two days in the authors' setup. The long optimization time appears to be one of the model's drawbacks.

Results

The table above shows that NeRF performs well across metrics such as PSNR, SSIM, and LPIPS. The NeRF project page contains more detailed results.

Dataset

Synthetic Renderings of Objects

The evaluation uses two broad kinds of data. The first is the DeepVoxels dataset, which contains four Lambertian objects—objects whose illumination appears uniform from every direction—with simple geometry. Each object is rendered at $512\times512$ pixels from viewpoints on the upper hemisphere. Of the samples, 479 are used as inputs and 1,000 for testing.

The authors also construct a more realistic dataset with eight non-Lambertian objects—objects whose surface color changes with viewing direction—with complex geometry and path-traced images. Six are rendered from viewpoints on the upper hemisphere, and two from viewpoints over the full sphere, including below the object. Every image is $800\times800$ pixels; 100 views are used as inputs and 200 for testing. In the table above, Diffuse Synthetic corresponds to the Lambertian data and Realistic Synthetic to the non-Lambertian data.

Real Images of Complex Scenes

The paper also evaluates forward-facing captures of eight complex real-world scenes. Five come from the LLFF paper, while the authors captured the remaining three. Each scene contains 20 to 62 images, one eighth of which are held out for testing. All rendered images have a resolution of $1008\times756$ pixels.

Model Comparisons

To assess NeRF, the authors compare it with other high-performing view-synthesis techniques and visualize the results in the paper. All methods except Local Light Field Fusion are trained on the same input views.

Neural Volumes (NV)

Neural Volumes synthesizes views of a bounded volume in front of a separately captured background, meaning the background must be recorded without the object. A deep 3D convolutional network predicts discrete RGB$\alpha$ values on a voxel grid with $128^3$ samples while simultaneously predicting a 3D warp grid with $32^3$ samples. As the figure below shows, the algorithm renders marching camera rays through the warped voxel grid.

Scene Representation Networks (SRN)

SRN represents a continuous scene as an opaque surface. An MLP extracts a feature vector at each $(x,y,z)$ coordinate, and an RNN marches along the ray using those features. The output at each state determines the next step size along the ray. The final decoded feature vector yields the surface color at the terminal location.

Local Light Field Fusion

LLFF is designed to generate photorealistic novel views from forward-facing scene captures. A trained 3D convolutional network produces RGB$\alpha$ grids, and novel scenes are rendered using alpha compositing and nearby multiplane images (MPIs).

An example MPI. One intuitive way to understand it is that moving along the marched scene produces a multiplane image.

Comparison Results

The qualitative results are also strong.

Discussion

The first thing that stands out is simply how good the results look. Unlike earlier approaches that failed to preserve fine details, NeRF produces detailed scenes without using any ground-truth 3D representation. Its practical limitation was the severe trade-off between time and spatial representation. LLFF, for example, takes less time but consumes a large amount of memory on Realistic Synthetic scenes. NeRF's network itself is comparatively simple, however, which gives it the advantage of being optimizable across many datasets.

Fully Connected Layer Structure

The figure shows the fully connected architecture. Positional information encoded as $\gamma(\mathbf{x})$ is supplied as input and passes through eight layers with 256 channels and ReLU activations. Following DeepSDF, the architecture adds a skip connection to the activation of the fifth layer.

An additional layer outputs the volume density $\sigma$. Because density must be nonnegative, its output is rectified with ReLU. A 256-channel feature vector extracted alongside the density is combined with the encoded direction $\gamma(\mathbf{d})$ and passed through the final layers to produce RGB values.

Deriving NDC Ray Space

NeRF uses normalized device coordinate (NDC) space for forward-facing scenes. This section is matrix-heavy, so we will cover only the essentials.

Intuitively, NDC transforms the camera's frustum into a uniformly scaled cube. In homogeneous coordinates, the familiar 3D perspective-projection matrix is:

$$M = \left( \begin{array}{cccc} \frac{n}{r} & 0 & 0 & 0 \\ 0 & \frac{n}{t} & 0 & 0 \\ 0 & 0 & \frac{-(f+n)}{f-n} & \frac{-2fn}{f-n} \\ 0 & 0 & -1 & 0 \end{array} \right).$$

Here, $n$ and $f$ are the distances to the near and far clipping planes, while $r$ and $t$ are the right and top bounds on the near clipping plane. The preceding diagram is useful for interpreting them.

To project a homogeneous point $(x,y,z,1)^T$, multiply it by $M$ and normalize by the final coordinate. A lecture on homogeneous coordinates will provide the full background.

$$\left( \begin{array}{cccc} \frac{n}{r} & 0 & 0 & 0 \\ 0 & \frac{n}{t} & 0 & 0 \\ 0 & 0 & \frac{-(f+n)}{f-n} & \frac{-2fn}{f-n} \\ 0 & 0 & -1 & 0 \end{array} \right) \left(\begin{array}{c}x\\y\\z\\1\end{array}\right) = \left(\begin{array}{c} \frac{n}{r}x \\ \frac{n}{t}y \\ \frac{-(f+n)}{f-n}z-\frac{2fn}{f-n} \\ -z \end{array}\right).$$

Dividing the projected coordinates by the final homogeneous component gives:

$$\operatorname{Project}(x,y,z)= \left(\begin{array}{c} \frac{n}{r}\frac{x}{-z} \\ \frac{n}{t}\frac{y}{-z} \\ \frac{f+n}{f-n}-\frac{2fn}{f-n}\frac{1}{-z} \end{array}\right).$$

Now that we are in NDC space, our goal is to map every point on a ray $\mathbf{o}+t\mathbf{d}$ to a ray $\mathbf{o}' + t'\mathbf{d}'$ in NDC space. Rewrite the projected point as:

$$\operatorname{Project}(x,y,z)= \left(\begin{array}{c} \frac{n}{r}\frac{x}{-z} \\ \frac{n}{t}\frac{y}{-z} \\ \frac{f+n}{f-n}-\frac{2fn}{f-n}\frac{1}{-z} \end{array}\right) = \left(\begin{array}{c} a_x x/z \\ a_y y/z \\ a_z+b_z/z \end{array}\right).$$

The projected origin $\mathbf{o}'$ and direction $\mathbf{d}'$ must then satisfy:

$$\left(\begin{array}{c} a_x\frac{o_x+td_x}{o_z+td_z} \\ a_y\frac{o_y+td_y}{o_z+td_z} \\ a_z+\frac{b_z}{o_z+td_z} \end{array}\right) = \left(\begin{array}{c} o'_x+t'd'_x \\ o'_y+t'd'_y \\ o'_z+t'd'_z \end{array}\right).$$

Without a constraint, this expression contains too many variables. To simplify it, first evaluate it at the starting point:

$$\left(\begin{array}{c} a_x\frac{o_x}{o_z} \\ a_y\frac{o_y}{o_z} \\ a_z+\frac{b_z}{o_z} \end{array}\right) = \left(\begin{array}{c}o'_x\\o'_y\\o'_z\end{array}\right) =\mathbf{o}'=\pi(\mathbf{o}).$$

This confirms that $\mathbf{o}'$ is exactly the projection of the original origin $\mathbf{o}$. Subtracting that origin again while retaining the variable $t$ yields:

$$\begin{aligned} \left(\begin{array}{c}t'd'_x\\t'd'_y\\t'd'_z\end{array}\right) &= \left(\begin{array}{c} a_x\frac{o_x+td_x}{o_z+td_z}-a_x\frac{o_x}{o_z} \\ a_y\frac{o_y+td_y}{o_z+td_z}-a_y\frac{o_y}{o_z} \\ a_z+\frac{b_z}{o_z+td_z}-a_z-\frac{b_z}{o_z} \end{array}\right) \\ &= \left(\begin{array}{c} a_x\frac{td_z}{o_z+td_z}\left(\frac{d_x}{d_z}-\frac{o_x}{o_z}\right) \\ a_y\frac{td_z}{o_z+td_z}\left(\frac{d_y}{d_z}-\frac{o_y}{o_z}\right) \\ -b_z\frac{td_z}{o_z+td_z}\frac{1}{o_z} \end{array}\right). \end{aligned}$$

Removing the common term involving $t$ leaves the direction $\mathbf{d}'$:

$$\mathbf{d}'= \left(\begin{array}{c} a_x\left(\frac{d_x}{d_z}-\frac{o_x}{o_z}\right) \\ a_y\left(\frac{d_y}{d_z}-\frac{o_y}{o_z}\right) \\ -b_z\frac{1}{o_z} \end{array}\right).$$

Substituting the constants from the original projection matrix gives:

$$\mathbf{o}'= \left(\begin{array}{c} a_x\frac{o_x}{o_z} \\ a_y\frac{o_y}{o_z} \\ a_z+\frac{b_z}{o_z} \end{array}\right) = \left(\begin{array}{c} -\frac{f_{\mathrm{cam}}}{W/2}\frac{o_x}{o_z} \\ -\frac{f_{\mathrm{cam}}}{H/2}\frac{o_y}{o_z} \\ 1+\frac{2n}{o_z} \end{array}\right),$$

and

$$\mathbf{d}'= \left(\begin{array}{c} -\frac{f_{\mathrm{cam}}}{W/2}\left(\frac{d_x}{d_z}-\frac{o_x}{o_z}\right) \\ -\frac{f_{\mathrm{cam}}}{H/2}\left(\frac{d_y}{d_z}-\frac{o_y}{o_z}\right) \\ -2n\frac{1}{o_z} \end{array}\right).$$

This is the desired NDC ray representation.