Introduction

Reduced to its core idea, this paper renders 3D objects using multimodal image relationships learned through text representations. Unlike images, which are relatively easy to obtain, 3D generation and rendering require captioned 3D datasets. This limits a 3D generation network to a finite pool of categories, as with ShapeNet.

To move beyond conventional methods with limited structures and textures, the authors guide generation with a CLIP network trained on the web-scale WebImageText dataset, containing roughly 400 million text-prompt/image pairs. The full training process is described later, but the basic idea is to train a Neural Radiance Field (NeRF), optimized across camera views from many directions, to maximize similarity with a target caption. The CLIP network supplies this similarity guidance. A CLIP loss alone can collapse the 3D structure or reduce fidelity, so the authors add simple geometric priors: regularization of transmittance to induce sparsity, scene bounds, and a new MLP architecture.

Why zero-shot is important?

As I began reading, a fundamental question occurred to me. Neural fields have already been studied extensively; given a dataset, rendering a high-fidelity 3D representation or producing images from new angles is not especially difficult. State-of-the-art zero-shot methods can also perform far worse than fully supervised ones. A zero-shot paper therefore needs to explain why the research is necessary. 3D object models are widely used in games and virtual reality. Anyone who has used Unity or Unreal Engine knows that every 3D object—and even many 2D objects—in these environments is created with digital software such as Blender or Maya, while adding textures also requires design work.

Essentially every object used in NeRF research is a product of that process. Unlike WebImageText, free 3D assets online are difficult to collect at consistent quality and sufficient diversity. Deep learning requires datasets, but one cannot hire tens of thousands of graphic designers merely to create training objects, and their differing styles would prevent a uniform distribution anyway.

Among the methods applicable when data collection is difficult—meta-learning, low-shot learning, domain generalization, unsupervised domain adaptation, and others—this paper chooses zero-shot learning. The issues above define the problems and limitations of prior work that motivate its task.

Challenges in Multimedia Applications

For research such as NeRF, a 3D dataset may contain only a few categories and one object form appropriate to each. But to satisfy users in a real multimedia environment, even a simple object such as a chair varies by purpose—sofa, bench, wheelchair—and by material, making its textures difficult to construct.

Unlike prior approaches that represent only shape through a point cloud, voxel grid, or triangle mesh, a practical system must account for view-dependent 3D geometry and texture.

Automatically generate open-set 3D models

To escape these limitations, Dream Fields uses CLIP, which learns image representations from open-set natural-language prompts and performs well across many zero-shot tasks. During NeRF training, Dream Fields optimizes both scene geometry and color to maximize a perceptual metric. As the title suggests, it uses no 3D training dataset at all. Conventional NeRF trains on RGB photographs from many directions as ground truth and reconstructs images from new views.

As shown above, learned function $F_\theta$ captures an implicit 3D space. The model samples its representation of an object according to viewing point and direction, generating an image from a new angle. NeRF offers smooth interpolation and handles color changes naturally. Later work improved NeRF quality, but all such methods share an inability to create novel images from descriptions. Given input images, they can render new views, but cannot use a text description as a 3D semantic feature. Since both CLIP's image-text representation and NeRF's volumetric rendering are differentiable, a “dream field” can be trained zero-shot.

Training only with CLIP representations—supervision from text prompts—and no regularization produces many artifacts, so the authors add geometric constraints.

Contribution

The paper's contributions are clear and stated directly in the introduction:

• A CLIP model trained on image-text pairs can optimize NeRF without 3D shapes or multiview datasets.
• The method generates varied 3D objects from zero-shot descriptions.
• Geometric priors improve fidelity.

The final point about geometric priors is better viewed as a secondary contribution that emerged during experimentation. The first two are the paper's central ideas.

Related works

The authors say DeepDream was their greatest inspiration. That feels slightly overstated to me: creating a completely novel object with NeRF seems so difficult that the idea would have been hard to develop without the much more recent CLIP. Many studies have generated desired images with neural networks—for example, GAN inversion and latent optimization generate images from pretrained networks without additional training. Aside from CLIP, the work most closely related to this paper is differentiable image parameterization from the perspective of style transfer.

Honestly, this part left me puzzled. Using CLIP itself distinguishes the method from prior style-transfer work, and many other papers seem more likely to have inspired it than the authors' own cited work. The opening of the related-work section feels written to increase citations; if that was not intentional, I apologize, but I do not understand why it is there. In any case, replacing conventional style- and content-based losses with an image-text loss increases control over generated style and content through text prompts. Traditional style transfer requires features from a target style image and often produces unstable quality, whereas CLIP-based style transfer needs no style image, only a description. The earlier approach transferred only limited geometry, such as a rabbit object, or optimized texture; Dream Fields enables open-ended, text-guided generation.

Other CLIP-based 3D generation work includes CLIP-Forge, but its decoder generates only geometry and receives guidance only for ShapeNet categories, preventing out-of-domain generation. Text2Shape generates voxelized objects with a text-conditional WGAN, as shown below, but remains confined to ShapeNet categories and, unlike NeRF, is limited in resolution by its voxel representation.

The voxel-based shape-generation results are indeed not especially strong.

The related work also includes an interesting four-page arXiv paper completed by a single researcher at MIT: Evolving Evocative 2D Views.

That work uses a loop algorithm to optimize the 2D projection of a 3D supershape from one viewpoint against a CLIP score, followed by manual coloring. Other CLIP research includes ClipMatrix, which stylizes human SMPL models by modifying vertices and textures, and interfaces for editing signed-distance fields. Image-generation work is even more varied: CLIP can train the weights of a SIREN network, and experiments combine it with VQGAN, StyleGAN, and other generative networks. There are also 3D-aware GANs that combine 2D generative models with NeRF, but most lack the ability to generate from open-set text.

Backgrounds

NeRF

I reviewed NeRF separately before, and it is famous enough that many readers will know the background. NeRF learns a scene's density and color with an MLP. The network queries are the coordinates $(x,~y,~z)$ of a 3D point and viewing direction $(\theta,~\phi)$. As the notation suggests, training uses coordinate systems in canonical and spherical space simultaneously. The MLP implicitly learning canonical space focuses on density or transmittance at a location, while spherical space determines its color (RGB value). The learned parameters can render the object as a 2D image from any angle.

In this paper's simplified form, the MLP takes 3D position $x$ as input and outputs density $\sigma_\theta(x)$ and color $c_\theta(x)$ at that position. An image from a given viewpoint is rendered from this information with the following equation. Each pixel's color is determined by the volume-rendering result along ray $r(t)$.

$$C(r, \theta) = \int_{t_n}^{t_f} T(r, t) \sigma_\theta (r(t)) c_\theta (r(t)) dt$$

Here $T(r,t)$ is transmittance: the probability that a ray starting at $t_n$ reaches position $t$ without being absorbed or attenuated by an object. Equivalently, it is the probability that no object lies between scene boundary $t_n$ and $t$.

$$T(r, t) = \exp\left( -\int_{t_n}^t \sigma_\theta (r(s))ds \right)$$

In practice, we cannot integrate over every point along the ray. Instead, divide it into smaller segments $(t_{i-1}\leq r_i<t_i)$ and assume $\sigma$ and $c$ remain approximately constant within each one. With sufficiently fine segmentation, this assumption approaches the continuous case.

$$C(r, \theta) \approx \sum_i T_i (1 - \exp (-\sigma_\theta (r(t_i))\delta_i))c_\theta (r(t_i))$$ $$T_i = \exp \left( -\sum_{j < i} \sigma_\theta (r(t_j))\delta_j \right),~\delta_i = t_i - t_{i-1}$$

Given MLP parameters $\theta$ and pose $p$, we can sample a ray for each pixel, calculate color and density, and render them to obtain pixel color $C(r,\theta)$ and image $I(\theta,p)$.

Because an MLP struggles to learn an implicit 3D representation, positional encoding lifts features into a higher-dimensional representation.

$$\gamma (x) = (\cos (2^l x),~\sin (2^l x))^{L-1}_{l=0}$$

$L$ is the positional-encoding level. The implementation uses integrated positional encoding (IPE) from mip-NeRF.

IPE(Integrated Positional Encoding)

The figure illustrates the IPE concept. NeRF encodes every frequency equally, which causes aliasing in high-dimensional encoded values when the sampling rate is lower than the encoding frequency, as seen on the left. IPE instead applies a Gaussian over the sampling area, as on the right, so the entire sampled region can be treated consistently.

Imagine a cone extending from the camera's center of projection $o$, with its axis along the normal vector to the pixel plane. Point $o$ is the cone's apex, and vector $d$ represents its height, as shown below.

Consider an image plane parallel to the pixel plane and located a fixed distance $o+d$ from center of projection $o$. Define the pixel's radius or width as parameter $\dot{r}$. Positions $x$ between $t_0$ and $t_1$ in the figure are then defined as

$$F(x, o, d, \dot{r}, t_0, t_1) = \left(t_0 < \frac{d^\top(x-o)}{\vert \vert d \vert \vert_2^2} < t_1 \right) \& \left( \frac{d^\top (x-o)}{\vert\vert d \vert\vert_2 \vert\vert x-o \vert\vert_2} > \frac{1}{\sqrt{1+(\dot{r}/\vert\vert d \vert\vert _2)^2}} \right)$$

This determines whether each sample $x$ belongs to the $n$th conical element and must be incorporated into positional encoding. The simplest method computes positional encoding $\gamma(x)$ for every point and averages points within a given region.

$$\gamma^* (o, d, \dot{r}, t_0, t_1) = \frac{\int \gamma (x) F(x, o, d, \dot{r}, t_0, t_1) dx}{\int F(x, o, d, \dot{r}, t_0, t_1) dx}$$

As the formula shows, averaging over every sample is also intractable, so the mean must be approximated efficiently. Integrated positional encoding therefore assumes a multivariate Gaussian matching the expectation of $\gamma(x)$, as illustrated on the far right.

Image-text models

An image-text model such as CLIP can be summarized by a simple formulation. It consists of image encoder $g$ and text encoder $h$, each mapping data from its modality into an embedding space. If text prompt $T$ corresponds to image $I$, image embedding $g(I)$ and text embedding $h(T)$ should point in similar directions. In CLIP space, text and image are each converted into a vector and normalized to have L2 norm $1$. The following code performs classification with CLIP.

# Prepare the inputs
image, class_id = cifar100[3637]
image_input = preprocess(image).unsqueeze(0).to(device)
text_inputs = torch.cat([clip.tokenize(f"a photo of a {c}") for c in cifar100.classes]).to(device)

# Calculate features
with torch.no_grad():
    image_features = model.encode_image(image_input)
    text_features = model.encode_text(text_inputs)

# Pick the top 5 most similar labels for the image
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)
values, indices = similarity[0].topk(5)

# Print the result
print("\nTop predictions:\n")
for value, index in zip(values, indices):
    print(f"{cifar100.classes[index]:>16s}: {100 * value.item():.2f}%")

As the “top 5 most similar labels for the image” section shows, embeddings produced by .encode_image() and .encode_text() are normalized before comparison. Without magnitude normalization, they may not reflect the true cosine similarity or distance.

Given $N$ text-image pairs, contrastive learning treats the $N$ matching pairs as positive and the remaining $N^2-N$ as negative. Computing InfoNCE as a symmetric matrix turns training into a problem of maximizing the relationship—mutual information—between text-embedding information and image-embedding information.

Methods

Object representation

Built on NeRF's scene-representation learning, Dream Fields uses MLP parameter set $\theta$ to extract outputs $\sigma_\theta(x)$ and $c_\theta(x)$ for queried position $x$. NeRF space is thus embedded implicitly within the MLP, creating a virtual 3D space represented by density and color at every location. Unlike the original NeRF, the paper uses a simplified network independent of viewing direction because including direction offered no advantage. Given camera pose $p$, a ray $r(t)$ can be defined for every pixel on the camera plane, and the sampling equation

$$T_i = \exp \left( -\sum_{j < i} \sigma_\theta (r(t_j))\delta_j \right),~\delta_i = t_i - t_{i-1}$$

renders image $I(\theta,p)$. The number of samples directly affects fidelity; the paper fixes it at 192.

Objective function

The objective for optimizing Dream Fields is surprisingly simple. NeRF space is differentiable with respect to MLP parameters $\theta$. Although CLIP encoders $g,h$ remain frozen during training, gradients can still propagate through them. End-to-end training under a CLIP loss therefore applies the following loss to $\theta$:

$$\mathcal{L}_\text{CLIP}(\theta, \text{pose } p, \text{caption } y) = -g(I(\theta, p))^\top h(y)$$

The embedding produced by passing rendered image $I$ at camera pose $p$ through the image encoder is trained to resemble the desired text prompt, maximizing cosine similarity. The image and text encoders include both the ViT-based models from the original CLIP paper and the LiT baseline. DietNeRF, a few-shot NeRF paper, compares rendered and real images. Dream Fields instead compares the rendering only with a caption, enabling a zero-shot task with no object photograph at all. As the DietNeRF figure shows, both methods use CLIP embedding space for low-shot learning through a consistency loss based on the CLIP image encoder.

Challenges with CLIP guidance

CLIP tells us only how similar an image is to a text description; the text itself does not encode semantic information about image details. This problem is especially pronounced when optimizing NeRF. With images from multiple directions, the network receives abundant supervision and can learn a meaningful 3D representation, minimizing spurious density—the scattering caused by failing to learn a proper representation for novel views—as shown below.

In the zero-shot setting, the only supervision available is the text prompt, leaving NeRF training extremely unconstrained. In optimization, when the target function or set is too large and complex, we reduce the search space—for example, by finding only principal-component directions with PCA or imposing a penalty (regularization) to define a feasible region. Training NeRF on many images and training deep networks on large datasets serve a similar purpose.

More images captured from different angles impose more constraints on how the network renders each view. If the network contains $l$ layers and interactions among $N$ images compound across them, adding even one image can provide a training effect exceeding $N^l$. Applying only a text prompt in zero-shot learning therefore produces poor results, as shown below.

The most obvious problems are empty regions with floating density and density scattered throughout camera space instead of concentrating on the object to produce a sharp, high-fidelity result.

Pose sampling

Common image augmentations such as random crops improved generation in style-transfer work cited by this paper, including DeepDream. Such augmentations, however, apply only to in-plane 2D transformations. For 3D data augmentation, Dream Fields samples different camera-pose extrinsics at each training iteration. It samples uniformly over $360^\circ$ of azimuth around the scene, so every iteration views the object from another direction. Because the MLP shares one scene representation across views, camera sampling alone substantially improves object geometry.

Camera elevation, including focal length and distance to the object, could also be augmented, but the authors found it unnecessary.

Encouraging coherent objects through sparsity

Dream Fields applies opacity regularization during rendering to address near-field artifacts at the field boundary and spurious, sparsely sampled density. Opacity is related to transmittance: the probability that ray $r$ travels between $t$ and near-plane $t_n$ without being absorbed by an object; denser points have higher absorption. The authors define total transmittance as the joint transmittance across $N$ sampling segments. Since the equation is unchanged, the interpretation is straightforward.

$$\begin{aligned} T_i =& \exp \left( -\sum_{j < i} \sigma_\theta (r(t_j))\delta_j \right) \newline =& \prod_{j=1}^{i} \exp \left( -\sigma_\theta(r(t_j))(t_j - t_{j-1}) \right) \end{aligned}$$

They then define transmittance loss as

$$\begin{aligned} \mathcal{L}\_T =& -\min (\tau, \text{mean}(T(\theta, p))) \newline \mathcal{L}\_\text{total} =& \mathcal{L}_\text{CLIP} + \lambda \mathcal{L}_T \end{aligned}$$

Here $\tau$ is the target transparency chosen by the authors. A high joint probability implies little spurious density, while a low value means density is scattered along the ray, so optimizing this loss regularizes the object. For stable early training, the authors begin with $\tau=88\%$, then reduce it to $40\%$ after 500 iterations to prevent complete transparency, where no object appears. Scaling $\tau$ with focal length works well across different object distances. With a plain white or black background, the scene may become biased toward the background even after transmittance converges. Random image backgrounds help the object learn. Dream Fields uses Gaussian noise, checkerboards, and random Fourier textures, smoothed with random Gaussian noise.

The prompt is “An illustration of a pumpkin on the vine.” In the rendered results, regularization reduces sparse image artifacts, while replacing a white background with augmented backgrounds produces a sharper object.

Localizing objects and bounding scene

When Neural Radiance Fields are trained for image reconstruction, scene content is usually centered, as in standard NeRF datasets. Dream Fields can estimate a 3D object's center even when it is not at the scene center and shift rays accordingly. It finds the origin by applying an exponential moving average to the rendered object's center of mass. To prevent excessive drift, it masks density and keeps the object inside a fixed-size cube.

Neural scene representation architecture

NeRF uses an eight-layer MLP whose layers all have the same width, plus two additional layers that output RGB values, as shown below.

This paper replaces that architecture with a residual MLP and repeats a residual connection after every two dense layers. Since the paper includes no architecture diagram, I checked the official code below. As one might expect from Google, it uses flax and jax rather than pytorch.

class MipMLPLate(nn.Module):
  """MLP architecture."""
  activation: str, features_early: Sequence[int], features_residual: Sequence[Sequence[int]]
  features_late: Sequence[int], fourfeat: bool, max_deg: int, use_cov: bool, dropout_rate: float

  @nn.compact
  def __call__(self, mean, cov=None, x_late=None, decayscale=1., *, deterministic):
    """Run MLP."""
    # Integrate the positional encoding over a region centered at mean.
    if not self.fourfeat:
      # Axis-aligned positional encoding.
      feat = 2**np.arange(self.max_deg)[:, None, None] * np.eye(3)
      feat = feat.reshape(-1, 3)
    else:
      # Random Fourier Feature positional encoding. Fix the PRNGKey used for the
      # fourier feature basis so the encoding does not change over iterations.
      fourfeat_key = random.PRNGKey(124124)
      dirs = random.normal(fourfeat_key, (3, 128))
      dirs = dirs / np.linalg.norm(dirs, axis=-1, keepdims=True)
      rads = 2 ** (self.max_deg * random.uniform(fourfeat_key, (128,)))
      feats = (rads * dirs).astype(np.int32)
      feats = np.concatenate([np.eye(3), feats], 1).astype(np.float32)
      feat = feats.T

    mean_proj = (mean[Ellipsis, None] * feat.T).sum(-2)
    if self.use_cov:
      cov_diag_proj = ((cov[Ellipsis, None] * feat.T).sum(-2) * feat.T).sum(-2)
      decay = np.exp(-.5 * cov_diag_proj * decayscale**2)
    else:
      # Disable IPE
      decay = 1.
    x = np.concatenate([decay * np.cos(mean_proj),
                        decay * np.sin(mean_proj)], -1)

    # Network
    activation = nn.__getattribute__(self.activation)
    for feat in self.features_early:
      x = activation(nn.Dense(feat)(x))
      x = nn.Dropout(self.dropout_rate)(
          x, deterministic=deterministic)

    for feat_block in self.features_residual:
      h = nn.LayerNorm()(x)
      for l, feat in enumerate(feat_block):
        h = nn.Dense(feat)(h)
        h = nn.Dropout(self.dropout_rate)(
            h, deterministic=deterministic)
        if l < len(feat_block) - 1:  # don't activate right before the residual
          h = activation(h)
      x = x + h

    if x_late is not None:
      x = np.concatenate([x, x_late], axis=-1)
    for feat in self.features_late[:-1]:
      x = activation(nn.Dense(feat)(x))
      x = nn.Dropout(self.dropout_rate)(
          x, deterministic=deterministic)
    x = nn.Dense(self.features_late[-1])(x)  # don't activate output
    return x

In jax, np plays the same role as numpy, while nn from flax.linen behaves similarly to torch.nn in pytorch. A residual layer appears every two layers, and applying layer normalization before each residual block is reportedly effective. Although the API does not make it explicit, the middle channels expand much like a bottleneck architecture.

Highly transparent scenes, where most rendered values are zero, can cause vanishing gradients. The model therefore uses Swish instead of ReLU and rectifies density $\sigma_\theta$ with softplus.

$$f(x) = x \cdot \sigma(x),~\sigma(x) = \frac{1}{1+e^{-x}}$$

Results

Quantitative results on geometric priors

The authors measure whether the geometric priors used in the paper help retrieve images generated from COCO captions. Retrieval accuracy rises as each proposed component is applied. It also improves whenever the paper's transparency regularization is used.

Compositional generation

The figure shows Dream Fields samples without cherry-picking. Objects are generated with shape and material varied independently. A network such as DALL-E can also create strong images from captions, but unlike this paper it cannot generate a 3D object directly.

Regularization strength

Increasing the transparency regularization strength makes the object more compact. The description used here is “A cake topped with white frosting flowers with chocolate centers.”

Discussion and Limitations

Dream Fields can be viewed as the first paper to apply zero-shot learning effectively to NeRF, but it has several limitations. First, generation requires iterative optimization, so sampling is slow. The authors suggest future acceleration through meta-learning or amortization.

Second, every perspective uses the same prompt. This causes similar patterns to repeat on different sides of an object, resembling texture collapse. The method fails to account for captions that should vary with camera position. Unless an object's texture looks identical from every side, its front and back may need different appearance descriptions.

Finally, CLIP itself has limitations. Even if it learns strong image-text representations, it may assign incorrect scores to some ground-truth training images. CLIP's performance therefore affects generation, and relying on a pretrained model means accepting whatever biases it may contain.