Understanding NSVF: Neural Sparse Voxel Fields

Abstract

Producing realistic real-world scenes is remarkably difficult. NeRF, which we examined in the previous session, uses a simple neural network to improve scene rendering from unseen directions and coordinates. Its major bottlenecks remain long training time and the need to collect a large amount of data—especially the experimental constraint that camera marching must be configured appropriately for the intended setup. Various methods using 3D supervision also limit network capacity to reduce inference time and sometimes settle for blurry renderings. Improving 3D scene rendering or generation is difficult, and collecting suitable data or finding an appropriate metric is often just as hard.

This paper presents NSVF, or Neural Sparse Voxel Fields, which improves rendering quality while also increasing speed. As its name suggests, it uses voxel-based learning. A sparse voxel octree renders and learns only where an object is likely to exist, shortening training while allowing denser, higher-quality sampling. Its explicit 3D representation also opens possibilities for scene editing and composition.

Introduction

In computer graphics, realistic scene rendering can support AR and VR, visual effects in film and broadcasting, and synthetic-data creation in research. The challenge is producing samples of high enough quality for real use. A representation of the actual 3D scene is difficult to obtain—sensors such as LiDAR are expensive—while image-based rendering approaches are difficult to manipulate, offer limited scene diversity, and cannot guarantee quality merely because their inputs are easier to acquire.

Recent work therefore uses deep learning to learn virtual geometry and appearance in an embedding space. Training may incorporate explicit geometric representations such as voxel grids and textured meshes, multi-plane images—which can be understood as multi-channel images containing depth—or point clouds. Neural implicit representations are smoother and more continuous than explicit geometry, which is discrete because it is synthesized in a computing environment. From a signal-processing perspective, they also offer high spatial resolution and reduced aliasing. In practice, however, insufficient network capacity can limit performance, and locating intersections between camera rays and scene geometry is difficult.

NSVF likewise proposes an implicit function, Neural Sparse Voxel Fields. Although sparse voxels may appear to contain explicit 3D geometry, here they are virtual regions defined in the embedding space for sampling during training. Whereas NeRF feeds each point and direction to an MLP as key features, the voxel formulation uses a feature vector summarized from the eight vertices of the cube representing a point. The query point is represented differently, but the formulation can be shown mathematically to contain NeRF as a special case, as we will see later.

During training, voxel-grid regions with little scene information are pruned. The remaining tree nodes are sampled again to capture finer structure and learn its representation, making detailed regions more efficient to learn. In practice, the method is roughly ten times faster than NeRF.

Neural Rendering with Implicit Fields

Let the learnable implicit function be $F_\theta$, with $F_\theta:(p,v)\rightarrow(c,\omega)$. Given the pinhole camera position $p_0\in R^3$, the rendered color $C$ is

$$C(p_0,~v) = \int^{+\infty}_0 \omega(p(z)) \cdot c(p(z),~v)dz$$

where $\int^{+\infty}_0\omega(p(z))dz=1$. Here, $v$ denotes the viewing direction and $p(z)$ is a vector extending from the camera origin along the ray:

$$p(z) = p_0+z\cdot v$$

Not every point along this vector intersects an object, so the integral includes a weight. That weight can be interpreted using object density or the probability that the ray reaches a location without encountering anything. It is therefore independent of the direction vector $v$. Color, by contrast, depends on $v$ because non-Lambertian surfaces and reflection must be considered. This distinction is reflected in the network architecture.

Surface Rendering

Unlike the preceding rendering formulation, surface rendering finds the exact place where the ray meets an object's exterior. Rather than using weights that sum to one throughout the volume, it can be represented with a Dirac delta function:

$$\omega(p(z)) = \delta(p(z)-p(z^*))$$

where $p(z^*)$ is the intersection of the camera ray with the scene geometry.

Volume Rendering

Volume rendering follows nearly the same idea. NeRF renders by sampling according to

$$C(p_0,~v) \simeq \sum_{i = 1}^{N} (\prod_{j = 1}^{i-1} \alpha(z_j,~\Delta_j)) \cdot (1-\alpha(z_i,~\Delta_i))\cdot c(p(z_i),~v)$$

where $\alpha(z_i,\Delta_i)=\exp(-\sigma(p(z_i)\cdot\Delta_i))$ and $\Delta_i=z_{i+1}-z_i$, the sampling interval.

Limitations

Surface rendering learns a color that remains consistent across several views, which can blur the result. Volume rendering needs more information and therefore requires dense sampling for high performance. It is correspondingly slow: NeRF takes roughly 30 seconds to render an $800\times800$ image.

Definition

What, then, does the NSVF proposed in this paper actually mean? We begin with the voxel-bounded implicit field.

Voxel-bounded Implicit Fields

Suppose each non-empty part of a scene viewed from a given direction is represented by a voxel of fixed size, and the $K$ voxels form a set

$V=\{V_1,\cdots,V_K\}$.

With this voxel representation, the scene can be predicted by multiple voxel-bounded implicit functions—effectively a multivariate function for each voxel:

$$F_\theta(p, v) = F_\theta^i(g_i(p), v), \forall p \in V_i$$

Each point is thus represented by a parametric estimate for the voxel that contains it. We can then separate functions for the color, density, and representation of each 3D point $p$:

$$g_i(p) = \zeta(\chi(\tilde{g_i}(p_1^*), \cdots, \tilde{g_i}(p_8^\ast)))$$

The eight vertices of the voxel containing the point are used as inputs, and $\tilde{g_i}$ stores a feature vector at each vertex. The function $\chi(\cdot)$ is trilinear interpolation, while $\zeta(\cdot)$ is positional encoding. NeRF also uses positional encoding to lift inputs into a higher-dimensional space, since learning a manifold in a limited number of dimensions is difficult.

If the point itself is used instead of its vertices, NeRF appears as a special case of NSVF:

$$g_i(p) = \zeta(\chi(\tilde{g_i}(p_1^\ast), \cdots, \tilde{g_i}(p_8^\ast)))$$ $$= \zeta(p)$$

If $\tilde{g_i}(p):p\rightarrow(c,\sigma)$, $\zeta(\cdot)$ is omitted, and $F_\theta^i$ is simply the identity function, the model instead resembles Neural Volumes, storing color and density directly in explicit voxels.

Volume Rendering

NSVF predicts color and density for any scene point contained in its voxels. Unlike NeRF, which renders an implicit representation over the entire space, NSVF ignores empty regions and is therefore much more efficient. Rendering proceeds in two steps:

1. Find the intersections between the ray and the voxels.
2. Perform ray marching inside the intersected voxels.

Ray–Voxel Intersection

NSVF first applies an AABB, or Axis-Aligned Bounding Box, intersection test. For a bounding box aligned to the coordinate axes, it measures the distance from the ray origin to each of the voxel's six faces. This check works well for tasks such as NSVF that use a hierarchical octree structure. Experiments show that roughly 10k to 100k sparse voxels are sufficient to optimize fairly complex scene rendering.

Ray Marching inside Voxels

$$C(p_0,~v) \simeq \sum_{i = 1}^{N} (\prod_{j = 1}^{i-1} \alpha(z_j,~\Delta_j)) \cdot (1-\alpha(z_i,~\Delta_i))\cdot c(p(z_i),~v)$$

$C(p_0,v)$ is sampled using the volume-rendering equation above. A background term $c_{bg}$ is added to account for rays that miss every object.

The transparency $A(p_0,v)=\prod^N_{i=1}\alpha(z_i,\Delta_i)$ and $C_{bg}$ become learnable RGB values for the background. As discussed earlier, rendering the entire space is inefficient in time and produces lower resolution because it undersamples the regions that actually contain objects. NeRF addresses this by training separate coarse and fine networks:

$$\hat{C}c(\rm{r}) = \sum{i=1}^N \it{w}_i\rm{c}_i,~\it{w}_i = T_i(1-\exp(-\sigma_i \delta_i))\rm{c}_i$$

This cannot be trained as a single network; the two serial networks reduce both training and inference speed. NSVF needs no second sampling pass and can sample the region of interest at the same evaluation cost, making it substantially more efficient.

Early Termination

NSVF can represent both transparent and opaque solid objects. For a solid surface, however, only the externally visible voxels need to be processed; accumulation inside the object wastes time once the accumulated transparency $A(p_0,v)$ reaches zero. The experiments therefore use a threshold of $\epsilon=0.01$. If transparency falls below it, rendering terminates early, increasing speed without reducing quality.

Training

Except for the $C_{bg}$ term, the formulation has the same form as NeRF. Because the entire rendering process is differentiable, a quantitative cost against a reference image can be defined:

$$L = \sum_{(p_0,~v) \in R} \parallel C(p_0,~v)-C^*(p_0,~v) \parallel_2^2 + \lambda \cdot \Omega(A(p_0,~v))$$

$R$ denotes a batch of sampled rays, $C^*$ the ground-truth color of a camera ray, and $\Omega(\cdot)$ a beta-distribution regularizer. The beta regularizer can be understood simply as regularization over a variable between zero and one.

Voxel Initialization

The initial bounding box of volume $V$ is divided into ten parts along each of its width, height, and depth axes. This can be regarded as coarse sampling. Self-pruning and rendering then refine it. Self-pruning removes nonessential voxels: based on the density probability—interpreted in NeRF as the chance that a ray encounters an object at that location—a voxel meeting the threshold condition is excluded from computation.

$$V_i \text{ is pruned if} \min_{j = 1, \cdots, G} exp(-\sigma(g_i(p_j))) > \gamma,~p_j \in V_i,~V_i \in \nu$$

$\{p_j\}_{j=1}^G$ are points sampled uniformly within voxel $V_i$. The experiment uses $G=16^3$, dividing the voxel into sixteen sections along each axis, and applies the same threshold $\gamma=0.5$ in every setting.

Progressive Training

This part initially struck me as a little odd. The paper begins by criticizing NeRF's hierarchical networks, yet it also learns finer detail over repeated iterations. My reading is that the speed gain comes largely from pruning, while the quality improvement comes from denser sampling at the same cost. In that respect, this portion still looks quite similar to NeRF.

As shown in the figure, after one iteration each voxel is bisected along every axis into $2^3$ subvoxels, which are then pruned and processed again. Subdivision creates new interior vertices with no stored features, but trilinear interpolation resolves this simply. Trilinear interpolation follows the same principle as bilinear interpolation with one additional axis.