StyleGAN

StyleGAN can be regarded as the first paper to interpret generative models through a style-based approach. I will not explain StyleGAN in detail here, but briefly, it begins with a $4 \times 4 \times 512$ feature-map canvas represented by a constant vector. An affine layer derives style information—the mean $\mu$ and standard deviation $\sigma$—from a latent vector, modulates the preceding feature map, and gradually increases its size. It is also worth reading PGGAN, whose progressive increase in spatial resolution inspired this architecture.

Image manipulation

Rather than discussing StyleGAN's detailed methodology, let us examine the research trends it inspired. If we can obtain style information from a randomly sampled latent vector $z$ or $w$, then conversely, given a particular image, we can extract a latent vector $z$ or $w$ capable of producing it.
This idea of GAN inversion drove a major trend in image manipulation. Because StyleGAN enabled high-resolution image synthesis, its main contribution in this context was high-quality image manipulation. Many image-manipulation papers followed; readers interested in the subject may also consult my post on image manipulation. In short, StyleCLIP studies image manipulation using a pretrained StyleGAN. Unlike earlier methods, it focuses on how text guidance from CLIP, a vision-language contrastive model, can make image manipulation more convenient.

What Is Wrong with Existing Methods?

As noted above, most research on image manipulation finds meaningful semantic information in a pretrained StyleGAN latent space and manipulates it. Searching the latent space directly, however, does not reveal which visual style will change and makes it difficult to obtain the exact edit a user wants.

For example, suppose latent vector $w_1$ produces the image above of a woman with a neutral expression, while $w_2$ produces a woman with a surprised expression. A simple latent-space search does not tell us which manipulation direction creates a “surprised expression,” so we must search randomly. Because the latent vector has $512$ dimensions, a person must perform this laborious search manually in the absence of supervision. GANSpace, Semantic Face Editing, and StyleSpace Analysis use such methods. InterfaceGAN and StyleFlow attempt to solve the problem by using attribute classification as guidance.

If changing a particular latent-space element should alter “facial expression,” the latent is manipulated in the direction that maximizes that attribute change. Other work includes StyleRig, which uses the parametric 3DMM model to sample consistently over a 3D face mesh.
In every case, existing methods retain the process of finding a predetermined semantic direction, limiting how users can manipulate images through latent-space search or classifier guidance. Searching for directions not already mapped in the latent space, as StyleRig does, requires manual effort and enough annotated data.

Image Manipulation with CLIP

The authors therefore focus on a vision-language contrastive model that learned prompt-based image representations from a vast web dataset of text–image pairs. Because the network had already demonstrated prompt-based zero-shot image classification, they reasoned that it could also provide flexibility in image manipulation. Although CLIP and the existing StyleGAN framework can be combined in many ways, the paper describes three methods.

1. Perform text-guided latent optimization for each image. CLIP serves as a loss network: the latent vector is a trainable parameter, and the similarity between the image-encoder and text-encoder outputs becomes the loss that optimizes it.
2. Train a latent residual mapper. Instead of optimizing the latent vector directly, train a mapper that transforms a particular latent in $\mathcal{W}$ into another latent matching the CLIP representation.
3. Learn a text-prompt-conditioned style space independent of the image. The goal is to map styles while preserving the disentanglement of StyleGAN's $\mathcal{W}$ space.

Related works

Vision-Language tasks

The first related area is vision–language representation. Research combining vision and language includes language-based image retrieval, image captioning, and visual question answering. Following BERT's success across language tasks, recent VL methods began using Transformer backbones to learn joint representations—meaningful relationships between images and text. CLIP can likewise be understood as Transformer-based. Its ResNet-50 image encoder is not a pure Transformer, but attention pooling gives its embedding extraction some Transformer-like behavior.

Text-guided image generation and manipulation

Image generation has also been studied in multimodal settings. A conditional GAN, for example, can use a text embedding as its conditioning vector. Because early work preceded the development of Transformer architectures, it extracted text embeddings with a simple RNN and used them to condition generator and discriminator training.

Later research improved image quality with multi-scale GANs or applied attention between text and image features (paper).

The examples above concern image generation. Our focus is image manipulation, in which text supervises edits to a particular image. Because manipulation is harder than generation, it was not studied from the earliest days of deep learning. Even so, some early work used GAN architectures.

Early methods using GAN encoder–decoder structures focused on disentangling the semantics of the image and text description. ManiGAN, one of the more recent image-manipulation papers, proposed a text–image affine combination module (ACM) that improved edited-image quality. Its structure closely resembles StyleGAN's affine-based style modulation. Earlier work generally used basic GANs rather than a StyleGAN foundation; StyleCLIP differs by adopting the StyleGAN approach.
StyleCLIP was not the first to use StyleGAN. TediGAN also did so, but trained an encoder that maps text into StyleGAN's latent space and fused the resulting text-derived style space with the image's style space.

TediGAN appears to assume that because StyleGAN's eight-MLP $\mathcal{W}$ space is based on a normal distribution $\mathcal{Z} \sim \mathcal{N}(0,~I)$, an encoder can provide an implicit functional mapping despite the difference between image and text modalities. This seems like a more direct way to find meaningful text–image representations, but StyleCLIP reportedly performs better, perhaps because of CLIP's strength. DALL-E offers another approach, but its enormous GPU requirements make text-to-image manipulation less convenient. An online post—not a formal paper like StyleCLIP—also combines StyleGAN and CLIP for image manipulation (reference).

The figure reveals a latent-optimization process almost identical to the image-manipulation approach in StyleCLIP. The earlier method optimizes a latent to generate an image from scratch, however, whereas this paper optimizes the style-space latent vector that produces a particular image and thus focuses more directly on image manipulation.

Latent space image manipulation

Many papers already use pretrained StyleGAN generators for image manipulation because its intermediate style space $\mathcal{W}$ has useful properties such as disentanglement. A typical method maps an image into the latent space and uses a representation of the desired edited image as guidance. Approaches can be divided into those using image annotations as supervision and those that discover meaningful directions directly. Most StyleGAN-based manipulation work uses a 512-dimensional $\mathcal{W}$ vector or its extension $\mathcal{W}+$, which supplies a separate style at each feature level. StyleSpace Analysis, however, argues that the style space $\mathcal{S}$ is better and demonstrates more effective disentanglement than earlier spaces.

As noted earlier, StyleCLIP proposes three optimization methods:

1. Text-guided latent optimization for each image, using CLIP as a loss network and the latent vector as a trainable parameter.
2. A latent residual mapper that maps a latent in $\mathcal{W}$ to another latent matching the CLIP representation rather than optimizing it directly.
3. An image-independent style space conditioned on a text prompt, with the goal of preserving the disentanglement of StyleGAN's $\mathcal{W}$ space during style mapping.

Methods 1 and 2, which directly perform latent optimization, retain StyleGAN's existing $\mathcal{W}$ and $\mathcal{W}+$ spaces. Method 3 instead learns a new, input-agnostic $\mathcal{S}$ space.

StyleCLIP text-driven image(latent) manipulation

The paper evaluates these three methods separately. The simplest optimizes each source image according to a text prompt describing the desired style. Because this may be unstable with respect to hyperparameters, another option trains a separate mapping network for each sample. This local mapper learns to infer the latent-optimization step for an image and perform the mapping in one pass. Each mapper is trained for a single text prompt from the latent starting points of many image samples.
The authors found, however, that training a mapper for each manipulation made the mapper's steps similar—reducing manipulation diversity—and failed to exploit disentanglement properly. They therefore trained an input-agnostic mapping network, independent of each sample's starting point, and computed its global manipulation direction in style space $\mathcal{S}$.

Latent optimization

Latent optimization has a fairly simple expression. Rather than explain it at length, let us begin with the formula and interpret it.

$$\begin{aligned} &\arg \min \_{w \in \mathcal{W}+} D\_\text{CLIP} (G(w), t) + \lambda\_\text{L2} \parallel w - w_s \parallel_2 + \lambda\_\text{ID} \mathcal{L}\_\text{ID} (w) \newline &\mathcal{L}\_\text{ID} (w) = 1- \left< R(G(w_s)), R(G(w)) \right> \end{aligned}$$

Here, $R$ is a pretrained feature extractor from the ArcFace face-recognition network, and $\left< \cdot, \cdot \right>$ computes cosine similarity between the feature embeddings of two generated images. The input image is mapped to $w_s$ using e4e. As shown, $D\_{\text{CLIP}}$ reduces the cosine distance between the generated image's CLIP image embedding and the text embedding. The remaining terms measure how much of the original image to preserve, or content retention, controlled by weights $\lambda\_\text{L2}$ and $\lambda\_\text{ID}$.

Latent mapper

The latent-optimization process varies considerably across input images and text prompts, however, so every run requires appropriate hyperparameter tuning. The next approach therefore trains a latent mapper $M_t$ for text prompt $t$, independent of latent starting point $w$ and without fixing an input image, to learn a manipulation step in $\mathcal{W}+$. A mapper implements as a single function the process of optimizing a latent with a pretrained StyleGAN.

Because different StyleGAN layers control different levels of image detail, separate mappers are used for coarse, medium, and fine features. Every mapper is a four-layer fully connected network, half the depth of StyleGAN's mapping network. More precisely, it does not map latent $w_s$ directly to an optimal latent $w$; it predicts the change.

$$\mathcal{L}\_\text{CLIP}(w) = D\_\text{CLIP}(G(w + M_t (w)), t)$$

All other loss terms remain the same. We can regard $M_t(w) = w - w_s$. The paper uses $\lambda\_\text{L2} = 0.8$ and $\mathcal{L}\_\text{ID} = 0.1$.

$$\mathcal{L}(w) = \mathcal{L}\_\text{CLIP}(w) + \lambda\_\text{L2} \parallel M_t(w) \parallel_2 + \lambda\_\text{ID} \mathcal{L}\_\text{ID} (w)$$

These values were not fixed; the authors sometimes used different parameters for different samples. This already reveals a limitation of the paper: the text-based approach makes latent search more convenient, but not hyperparameter search, so the overall burden has merely shifted.

Global direction

The latent mapper guarantees faster inference than per-sample optimization, but the authors found that fine-grained, disentangled manipulation was difficult with existing methods, ultimately attributing this to a limitation of $\mathcal{W}+$. They therefore seek a global direction in style space $\mathcal{S}$. Let $s \in \mathcal{S}$ be a style code and $G(s)$ the image generated from it. Given a text prompt describing the attribute to manipulate, we want to find a direction $\Delta s$ for style code $s$ so that $G(s + \alpha \Delta s)$ maximizes the desired attribute. The style direction $\Delta s$ must leave every other attribute unchanged, while step size $\alpha$ controls the magnitude of the edit.
First, the CLIP text encoder obtains $\Delta t$ from the joint language–image embedding, which is then mapped to manipulation direction $\Delta s$. Text direction $\Delta t$ is derived through natural-language prompt engineering, and the corresponding $\Delta s$ accounts for each style channel's effect on the target attribute. More precisely, suppose $\mathcal{I}$ is the image-embedding manifold and $\mathcal{T}$ the text-embedding manifold. The direction vectors producing semantic change in image and text should be approximately collinear—with high cosine similarity—and nearly identical after normalization.
For original image $G(s)$ and edited image $G(s + \alpha \Delta s)$, denote their embeddings on $\mathcal{I}$ by $i$ and $i+\Delta i$. Since $\Delta i$ expresses the difference between the two images in CLIP space, a text-embedding vector $\Delta t$ describing the style change lets us find a global direction from their similarity.

Natural language to $\Delta t$

This technique appears in most CLIP-based approaches and is worth understanding. Mapping the characteristics of a text prompt to a single image is undesirable because image and text are different modalities. The description “a person wearing glasses” does not specify the person's hair, face, or gender, so a one-to-one mapping between image and text is generally impossible. The learned CLIP space therefore retains differences between these manifolds.
One way to reduce this inconsistency is the template method used for ImageNet zero-shot classification. Although the present task is not based on that dataset, the method remains useful as text-prompt augmentation.

imagenet_templates = [
    'a bad photo of a {}.',
    'a photo of many {}.',
    'a sculpture of a {}.',
    'a photo of the hard to see {}.',
    'a low resolution photo of the {}.',
    'a rendering of a {}.',
    'graffiti of a {}.',
    'a bad photo of the {}.',
    ...
    ...
]

Roughly $80$ prompt-engineering templates are used, leaving a placeholder inside braces as shown above. This acts like an ensemble or regularizer: averaging the embeddings has the effect of mapping a probability distribution rather than one point in embedding space.

Channelwise relevance

The discussion above concerns the relationship between $\Delta i$ and $\Delta t$, but not how to manipulate the style code $s$ itself. To alter a desired attribute by treating the style code channel by channel, we can use the following mechanism. Let $\Delta i_c$ be the change on the CLIP image manifold $\mathcal{I}$ caused by increasing or decreasing style-code coordinate $c$. If the desired image change follows direction $\Delta i$ regardless of coordinate, the relevance of style coordinate $c$, $R_c$, is the projection of $\Delta i_c$ onto $\Delta i$.

$$R_c (\Delta i) = \mathbb{E}\_{s \in \mathcal{S}} \left( \Delta i\_c \cdot \Delta i \right)$$

In practice, the method uses 100 image pairs to estimate the average change. Each pair is $G(s \pm \alpha \Delta s_c)$, where $s_c$ is zero everywhere except coordinate $c$, whose value is the standard deviation of that channel. After computing relevance $R_c$ for each channel, the method ignores channels below threshold $\beta$. The threshold controls how much entanglement is allowed.

$$\Delta s = \begin{cases} \Delta i_c \cdot \Delta i, & \text{if }\vert \Delta i_c \cdot \Delta i \vert \ge \beta \newline 0, & \text{otherwise} \end{cases}$$

Conclusion

The paper's main contribution is its use of StyleSpace to explore several ways of applying CLIP embeddings to image manipulation. Rather than using text embeddings directly, it performs style transfer through similarities in CLIP space. Its limitations include sensitivity to hyperparameters, which can make attribute changes unstable across samples, and difficulty manipulating images when source and target objects are not semantically related, such as a tiger and a lion.
Style mixing works freely for modalities resembling the face images on which StyleGAN was trained, whereas CLIP provides broader and more diverse embedding representations. In my view, the StyleGAN baseline is optimized for image-manipulation research from the perspective of style mapping, but that very specialization may also explain why the area has not produced more varied research and papers.

StyleGAN-NADA

StyleGAN-NADA takes a similar approach to StyleCLIP but uses layer fine-tuning instead of latent manipulation. Its central claim is that if mapping an image generator to a target domain on the image manifold can be guided solely by a text prompt, then supervision images of the style are unnecessary and many forms of styling become possible beyond domain-limited image manipulation. The results below show transformations into a variety of styles with StyleGAN-NADA.

One weakness of StyleCLIP is that manipulation in both $\mathcal{W}+$ and $\mathcal{S}$ ultimately remains within the pretrained StyleGAN domain, so the in-domain constraint is unresolved. Yet applying CLIP text guidance to the new approach is not straightforward either. Incorrect optimization can train the target domain toward an adversarial solution, leaving the realistic image manifold and producing many artifacts. As discussed below, the loss resembles StyleCLIP's: it aligns the text direction with the image direction in CLIP embedding space.

Related works

Many studies have explored text-guided image synthesis. Although the CLIP paper itself focused only on image–text relationships during training, its model proved useful for image synthesis and manipulation from text prompts. As StyleCLIP illustrates, most approaches that use CLIP to optimize a pretrained model such as StyleGAN perform latent optimization, with the primary goal of finding the latent code that produces a specific image. StyleGAN-NADA instead optimizes the image generator itself through text-prompt guidance, enabling manipulation beyond a limited domain.

Another relevant line of work trains generators from limited data. Few-shot methods risk overfitting the generator or causing mode collapse, in which sample diversity declines, so they use augmentation or auxiliary tasks to learn richer representations. StyleGAN-NADA requires no data at all and uses only a CLIP-based text prompt as guidance, eliminating concerns about overfitting or mode collapse caused by data scarcity.

Understanding this paper also requires familiarity with StyleGAN and StyleCLIP, but those topics have already been covered. StyleCLIP is explained above, and I discuss StyleGAN in a separate post.

CLIP based guidance

The figure above summarizes the method. A generator $G$ pretrained on the source domain is copied. One copy is frozen so that it continues producing source-domain images, while the other is fine-tuned to produce the target domain under guidance from a CLIP loss.

Following the same progression of trial and error as StyleCLIP, the authors begin with a global loss and explain why a directional CLIP loss matters. I consider this the most important point in CLIP-based style-transfer and image-manipulation papers, so it deserves a detailed explanation.

Global loss

The simplest approach is to optimize the CLIP loss between an image generated by the model and the target text prompt.

$$\mathcal{L}\_\text{global} = D_\text{CLIP} (G(w), t_\text{target})$$

Given latent code $w$, the generated image $G(w)$ is passed through image encoder $E_I$, and the target prompt through text encoder $E_T$; the loss uses the cosine similarity of their embeddings. Unlike StyleCLIP, the optimized parameters belong to $G$, not $w$. Although this is the simplest approach, it can converge to an adversarial solution. Since the generator parameters are trainable and no Wasserstein or adversarial loss preserves the real-image manifold, the output can leave that manifold. The term can also cause mode collapse, as the next figure illustrates.

Suppose cat images form the target domain and dog images the source domain. The red and blue points represent these two sets in CLIP image-embedding space. The turquoise and purple points represent the text embeddings for the prompts “cat” and “dog,” respectively.

If training merely forces every generated image to match the prompt, it proceeds as in (b). The minimum sends all dog images toward the image embedding nearest the text prompt. Instead of generating the diverse cat images represented by the red distribution, the model generates images in the narrow green region. This is mode collapse.

Directional CLIP loss

To avoid this problem, the method adopts StyleCLIP's global-direction approach. Instead of moving every image toward the target prompt itself, it supplies only the direction from source text to target text. Adding that direction to each source-image vector changes the desired attribute while preserving sample diversity.

$$\begin{aligned} \Delta T =& E_T(t_\text{target}) - E_T (t_\text{source}) \newline \Delta I =& E_I (G_\text{train}(w)) - E_I (G_\text{frozen}(w)) \newline \mathcal{L}_\text{direction} =& 1-\frac{\Delta I \cdot \Delta T}{\vert \Delta I \vert \vert \Delta T \vert} \end{aligned}$$

The figures above illustrate the method. Unlike before, simply aligning directions is enough to generate images resembling the target domain.

Layer freezing

When the domain shift changes texture—for example, turning a photograph into a painting—the same training scheme still produced mode collapse or overfitting. Earlier few-shot domain-adaptation methods improved synthesis quality by restricting part of the network weights, and the authors reasoned that this could also work in a zero-shot task. Optimizing fewer parameters indirectly reduces the size of the trainable network and acts as regularization against overfitting on small datasets.

How, then, should the trainable layers be selected?

The authors explain how they built up the idea step by step. In StyleGAN, style codes at different positions affect different semantic attributes. In the figure, $w_1$ and $w_2$ control relatively coarse features, while codes closer to $w_l$ control fine features. The success of $\mathcal{W}+$ in image manipulation likewise comes from using different style codes at each layer. To generate images in a particular style or domain, it is therefore effective to fine-tune the layers whose $w_i \in \mathcal{W}$ have the greatest influence on the domain shift, thereby preserving high image quality.

To choose $k$ layers, first sample $N$ random latent codes as $M(z) = w$ and replicate each into $\mathcal{W}+$ as a style code for every layer. My post on image manipulation may make this easier to understand. Then run StyleCLIP latent-code optimization for $i$ iterations—optimizing the latent code, not the generator—and measure the change in each code. Select the $k$ layers corresponding to the $w$ codes that changed most and use them for training.

Latent-Mapper mining

The methods above regularize generated images effectively, but they can prevent the generator from fully learning the target domain. In a dog-to-cat task, for example, the fine-tuned network may produce both dogs and cats, or ambiguous hybrids between them. To avoid this, the method also uses StyleCLIP's latent mapping to move latent codes into a cat-related region.

Conclusion

The paper can use both image embeddings and text embeddings as guidance, and the same optimization method works well for both. Unlike StyleCLIP, its contribution is an effective way to fine-tune the parameters of StyleGAN's generator, enabling out-of-domain styles that earlier methods could not apply. Potential drawbacks include the need for latent mapping to prevent domains from mixing when latent codes are excluded from optimization, and the time required to mine trainable layers. Even so, its use of CLIP to depart from conventional style-mixing approaches is impressive.