Image Manipulation with GANs: Image-to-Image Translation and GAN Inversion

GANs support many kinds of image synthesis:

• Texture synthesis: PSGAN, TextureGAN, and Texture Mixer
• Super-resolution: ProGAN, Progressive Face Super-Resolution, BigGAN, and StyleGAN
• Image inpainting, which completes missing or damaged images: DeepFill v1, ExGAN, DeepFill v2, EdgeConnect, and PEN-Net
• Face synthesis: ELEGANT, STGAN, SCGAN, Example-Guided Image Synthesis, SGGAN, and MaskGAN
• Human image synthesis for pose and body shape: Text-Guided, Progressive Pose Attention, Coordinate-Based, and Semantic Parsing

This article focuses instead on changing an existing image:

1. Image-to-image translation: CycleGAN, MUNIT, DRIT, TransGaGa, and RelGAN
2. Image editing: SC-FEGAN, FE-GAN, Mask-Guided, and FaceShapeGene
3. Cartoon generation: CartoonGAN, PI-REC, Internal Representation Collaging, U-GAT-IT, and Landmark-Assisted CycleGAN

Image Translation

The long list shows how broadly GANs can be used. Our real aim is to understand the early ideas behind image manipulation.

Image-to-image translation generates an output image related to an input. It includes semantic labeling and boundary detection in computer vision,

as well as colorization and super-resolution in computer graphics and computational photography.

Supervision maps a source image from domain $S$ into a target image in domain $T$. Every such task reduces to the same outline:

• Define objective $\mathcal{L}$.
• Prepare training pairs $(x,y)$.
• Train network $G$.
• Use translation $G(S)=T$.

$$G^\ast = \arg \min_G \mathbb{E}_{x,y}[\mathcal{L}(G(x), y)].$$

Applying an ordinary GAN creates two problems. First, it cannot control the generation mode: we need a translated version of the input, not merely any plausible target-domain image. Second, early generators produced low-resolution output. Three representative solutions are pix2pix, CycleGAN, and pix2pixHD. We will focus on the first two.

pix2pix: Image-to-Image Translation with Conditional Adversarial Nets

pix2pix is the representative GAN approach to paired image translation. Its best-known demonstration turns a sketch into a realistic photograph.

A generator can make a plausible bag from a sketch while still ignoring the sketch's exact structure. In the upper row, the generated image follows the outline; in the lower row, it resembles a bag but has poor correspondence with the input.

The ordinary GAN objective cannot express this distinction:

$$\begin{aligned} &\min_G \max_D V(D,G) \newline V(D,G) =& \mathbb{E}_{y \sim p_{data}}[\log D(y)] + \mathbb{E}_{z \sim p_z}[\log (1-D(G(z)))]. \end{aligned}$$

It merely asks whether the generated image looks as though it came from the real target distribution. Supplying sketch $x$ to the generator gives

$$V(D,G) = \mathbb{E}_{y \sim p_{data}}[\log D(y)] + \mathbb{E}_{x,z}[\log (1-D(G(x,z)))],$$

but an unconditional discriminator still tells $G$ only whether the output looks real, not whether it corresponds to $x$.

pix2pix uses a conditional GAN objective and adds mean absolute error between output and paired ground truth:

$$\begin{aligned} &\arg \min_G \max_D \mathcal{L}_{cGAN} (G,D) + \lambda \mathcal{L}_{L1}(G) \newline \mathcal{L}_{cGAN}(G, D) =& \mathbb{E}_{x,y}[\log D(x, y)] +\mathbb{E}_{x,z}[\log(1-D(x, G(x, z)))] \newline \mathcal{L}_{L1}(G) =& \mathbb{E}_{x,y,z}[\lVert y-G(x, z) \rVert_1]. \end{aligned}$$

Conditioning discriminator $D$ on $x$ requires the result to be plausible for that source. The L1 term further preserves content and reduces the blurry or structurally unrelated solution; pix2pix uses a large weight around $\lambda=100$.

The generator uses a U-Net, whose skip connections preserve spatial structure. In practice, pix2pix does not supply an explicit latent $z$ in addition to $x$; dropout in the decoder supplies stochasticity. Experiments found that an explicit latent was largely ignored.

The figure isolates the losses. Without L1, output loses the target's broad structure. Without the conditional GAN, it becomes blurry. The discriminator uses Leaky ReLU with slope $0.2$ and, as in DCGAN, omits batch normalization from the first layer.

pix2pix can solve many translation tasks whenever paired datasets are available. Its limitation is precisely the pair requirement. Sketch-to-image learning needs a photograph corresponding to every sketch; depth generation needs a depth map for every scene. Such pairs are expensive and sometimes impossible to collect.

CycleGAN starts from this limitation. Suppose we want to convert photographs into Monet's style. We cannot photograph the exact scenes Monet painted under the same weather, landscaping, and historical conditions. Nor can we resurrect Monet and ask him to paint a new paired dataset.

pix2pix applies only to pairs like those on the left, not unrelated collections like those on the right.

CycleGAN: Unpaired Image-to-Image Translation with Cycle Consistency

Think of generation as language translation. Translate English into Korean and then translate the Korean back into English; the result should reproduce the original sentence. In CycleGAN, generators are the translators.

For horse-to-zebra translation, converting a horse to a zebra and then back to a horse should reconstruct the original horse. This constraint preserves content even without paired examples.

Let $X$ and $Y$ be two domains. Generator $G:X\rightarrow Y$ maps $x$ to $\hat{y}=G(x)$, while $F:Y\rightarrow X$ maps $y$ to $\hat{x}=F(y)$.

The forward adversarial loss is

$$\mathcal{L}_{GAN}(G, D_Y, X, Y),$$

where discriminator $D_Y$ distinguishes real $Y$ examples from outputs of $G$. This trains $G$ and $D_Y$ but says nothing about the reverse mapping, so CycleGAN adds

$$\mathcal{L}_{GAN}(F, D_X, Y, X),$$

where $D_X$ and $F$ compete over domain $X$.

The central intuition—returning should recover the original—is expressed by an L1 cycle-consistency loss:

$$\mathcal{L}_{cyc}(G,F) =\mathbb{E}_{x\sim X}[\lVert F(G(x))-x\rVert_1] +\mathbb{E}_{y\sim Y}[\lVert G(F(y))-y\rVert_1].$$

The implementation resembles pix2pix, but the original CycleGAN uses instance normalization and a modified ResNet generator.

One training step is:

1. Sample $x\in X$ and $y\in Y$ independently.
2. Generate $\hat{y}=G(x)$ and $\hat{x}=F(y)$.
3. Train $D_X$ on real $x$ versus fake $\hat{x}$, and $D_Y$ on real $y$ versus fake $\hat{y}$.
4. Reconstruct $\tilde{x}=F(\hat{y})$ and $\tilde{y}=G(\hat{x})$.
5. Calculate cycle L1 losses between $(x,\tilde{x})$ and $(y,\tilde{y})$.
6. Combine adversarial and cycle terms to update the generators.

The method works across many domains using only two unpaired collections. Its limitation is that a GAN has no explicit object understanding. In horse-to-zebra translation, it can paint stripes onto the background or even onto a rider. If riders are common and balanced in training, the model may learn not to alter them; if they are rare, inference produces these artifacts.

GAN Inversion

GAN inversion approaches image manipulation from the opposite direction.

For a trained generator $G$, latent point $z$ produces fake image $G(z)$. Latents are commonly sampled from $z\sim\mathcal{N}(0,I)$. Inversion instead starts with a real image $x$ and searches for the latent whose generated image best matches it:

$$z^\ast = \arg \min_z \mathcal{L}(G(z),~x).$$

If an ideal $z^\ast$ reconstructs $x$, then moving that latent through meaningful directions edits the real image. The concept is simple once the generator is understood.

Many pretrained StyleGANs expose useful domains for manipulation; see this collection of pretrained StyleGAN models.

Image2StyleGAN

StyleGAN contains an MLP mapping $\mathcal{Z}$ into intermediate space $\mathcal{W}$. Image2StyleGAN performs inversion in an expanded $\mathcal{W}^+$ space. A normal $\mathcal{W}$ latent is copied 18 times and supplied to all 18 style layers. A $\mathcal{W}^+$ code instead has shape $18\times512$, allowing a different row vector at every style layer and therefore a more detailed reconstruction.

Rather than accept an arbitrary StyleGAN sample, optimize a latent to reproduce a reference image, then edit through that latent. Optimization proceeds as follows:

1. Initialize latent code $w^\ast$.
2. Generate $I^\ast=G(w^\ast)$ with a pretrained StyleGAN.
3. Compare $I^\ast$ with reference $I$ using loss $\mathcal{L}$.
4. Update $w^\ast$ by gradient descent while keeping $G$ fixed.
5. Repeat for several iterations.

Because the generator is fixed, two choices dominate performance: the latent space and initialization, and the loss that identifies a faithful code within that space.

Random initialization is possible, but StyleGAN samples poorly in low-density regions. If the initial point lies where feasible gradient directions lead away from supported images, optimization may not reconstruct the target. Starting from the mean latent—the “mean face”—is safer.

Pixel-wise MSE alone also fails to reflect perceptual content and makes high-quality inversion difficult. A perceptual loss adds similarity between deep features:

$$w^\ast = \arg\min_w \mathcal{L}_\text{percept}(G(w), I) +\frac{\lambda_{mse}}{N}\lVert G(w)-I\rVert_2^2.$$

Perceptual loss uses feature maps from an ImageNet-pretrained VGG-16 to compare hidden representations of the two images.

The original perceptual-loss paper optimized style through VGG features, demonstrating that pretrained activations can guide content- and style-aware reconstruction:

$$\mathcal{L}_\text{percept}(I_1, I_2) = \sum_{j=1}^4 \frac{\lambda_j}{N_j} \lVert F_j(I_1)-F_j(I_2)\rVert_2^2.$$

Applications

Morphing

Morphing gradually changes one image into another. If images $I_1$ and $I_2$ invert to codes $w_1$ and $w_2$, interpolate their latents:

$$w = \lambda w_1 + (1-\lambda)w_2,\qquad \lambda \in (0, 1).$$

The morphed image is $G(w)$.

Expression Transfer

Expression transfer uses latent arithmetic. Suppose:

• $w_1$ represents a neutral cat image $I_1$.
• $w_2$ represents a neutral dog image $I_2$.
• $w_3$ represents a smiling dog image $I_3$.

Then

$$w=w_1+\lambda(w_3-w_2)$$

should produce a smiling cat. The difference $w_3-w_2$ represents a direction associated with the expression, which can transfer to another identity or category.

Style Transfer

This resembles StyleGAN's original style mixing but operates in $\mathcal{W}^+$:

• Invert $I_1$ into $w_1\in\mathbb{R}^{18\times512}$.
• Invert $I_2$ into $w_2\in\mathbb{R}^{18\times512}$.
• Use early rows from one and later rows from the other, concatenating coarse styles from one source with fine styles from the second.

Because inversion supplies style codes for real images, this transfer can work even across seemingly unrelated source domains, extending StyleGAN's original face-synthesis demonstration.

Image2StyleGAN++ expands the approach further with mask-based style transfer, image inpainting, local editing, and other applications.