GAN Variants: DCGAN, Conditional GAN, PGGAN, and StyleGAN

DCGAN: Unsupervised Representation Learning with Deep Convolutional GANs

The previous GAN article introduced adversarial training between a generator $G$ and discriminator $D$, using multilayer perceptrons in the original architecture. DCGAN was the first influential demonstration that deep convolutional networks could train GANs effectively. Because adversarial training changes dramatically with architecture and optimization details, the paper is notable for its many practical ablations.

The paper itself illustrates only the generator, so the discriminator diagram on the right comes from another source. The generator is an upsampling convolutional network and the discriminator a downsampling one.

Most notably, DCGAN uses no max-pooling layers. Unlike classification or detection, which can rely on high-level, low-resolution features, generation must reconstruct the image itself; pooling would discard too much information. The generator uses ReLU, while the discriminator uses Leaky ReLU except for its final sigmoid, which produces a probability in $[0,1]$.

$$\begin{aligned} \operatorname{ReLU}(x) =& \begin{cases} x, & \text{if }x \geq 0 \newline 0, & \text{otherwise} \end{cases} \newline \operatorname{LeakyReLU}(x) =& \begin{cases} x, & \text{if }x \geq 0 \newline 0.01x, & \text{otherwise}. \end{cases} \end{aligned}$$

Leaky ReLU retains a small gradient for negative activations. My guess is that this helps the discriminator avoid converging or overfitting too early to the generator's initially poor distribution, which would unbalance the game.

The generator receives a 100-dimensional latent vector and expands it into a form suitable for convolution.

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()

        self.init_size = opt.img_size // 4
        self.l1 = nn.Sequential(nn.Linear(opt.latent_dim, 128 * self.init_size ** 2))

        self.conv_blocks = nn.Sequential(
            nn.BatchNorm2d(128),
            nn.Upsample(scale_factor=2),
            nn.Conv2d(128, 128, 3, stride=1, padding=1),
            nn.BatchNorm2d(128, 0.8),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Upsample(scale_factor=2),
            nn.Conv2d(128, 64, 3, stride=1, padding=1),
            nn.BatchNorm2d(64, 0.8),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(64, opt.channels, 3, stride=1, padding=1),
            nn.Tanh(),
        )

    def forward(self, z):
        out = self.l1(z)
        out = out.view(out.shape[0], 128, self.init_size, self.init_size)
        img = self.conv_blocks(out)
        return img

The code maps latent $z$ through a linear layer, then reshapes the expanded one-dimensional tensor into batch $\times$ channel $\times H\times W$. This is often described as “project and reshape.” Fractionally strided convolutions then expand the spatial dimensions.

An ordinary convolution on the right operates directly over the input grid. A fractionally strided, or transposed, convolution on the left inserts spacing between input positions before convolving, increasing resolution. It proved more effective here than max-unpooling or fixed interpolation. The generator finishes with $\tanh$ to scale its image output.

DCGAN also uses batch normalization selectively: the generator omits it from the final convolution, and the discriminator omits it from the first convolution.

Conditional GAN

DCGAN shows how to build a convolutional GAN. Conditional GAN asks whether its generator can create a requested sample.

An ordinary GAN samples an arbitrary latent from $\mathcal{Z}$ and asks whether its generated image looks as though it belongs to the target domain. No supervision separates meaningful regions of $\mathcal{Z}$, so users cannot directly choose the class—whether an MNIST sample should be a 1 or an 8, for example.

A conditional GAN simply gives class label $y$ to both networks, turning the adversarial objective into a conditional one:

$$\begin{aligned} &\min_G \max_D V(D,G) \newline V(D,G) =& \mathbb{E}_{x \sim p_{data}(x)}[\log D(x \vert y)] + \mathbb{E}_{z \sim p_z(z)}[\log (1-D(G(z \vert y)))]. \end{aligned}$$

After training, generation receives both latent $z$ and the desired class.

class generator(nn.Module):
    # Network Architecture is exactly same as in infoGAN (https://arxiv.org/abs/1606.03657)
    # Architecture : FC1024_BR-FC7x7x128_BR-(64)4dc2s_BR-(1)4dc2s_S
    def __init__(self, input_dim=100, output_dim=1, input_size=32, class_num=10):
        super(generator, self).__init__()
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.input_size = input_size
        self.class_num = class_num

        self.fc = nn.Sequential(
            nn.Linear(self.input_dim + self.class_num, 1024),
            nn.BatchNorm1d(1024),
            nn.ReLU(),
            nn.Linear(1024, 128 * (self.input_size // 4) * (self.input_size // 4)),
            nn.BatchNorm1d(128 * (self.input_size // 4) * (self.input_size // 4)),
            nn.ReLU(),
        )
        self.deconv = nn.Sequential(
            nn.ConvTranspose2d(128, 64, 4, 2, 1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.ConvTranspose2d(64, self.output_dim, 4, 2, 1),
            nn.Tanh(),
        )
        utils.initialize_weights(self)

    def forward(self, input, label):
        x = torch.cat([input, label], 1)
        x = self.fc(x)
        x = x.view(-1, 128, (self.input_size // 4), (self.input_size // 4))
        x = self.deconv(x)
        return x

This implementation comes from a PyTorch generative-model collection, rather than an official repository. It concatenates the latent and label along the feature dimension and otherwise proceeds like a standard GAN. The same conditioning idea applies to deep convolutional GANs.

PGGAN: Progressive Growing of GANs for Improved Quality, Stability, and Variation

I also wanted to cover WGAN, based on Wasserstein-1 distance, but its mathematics deserves a separate post. It is relevant because Progressive Growing GAN trains with WGAN-GP rather than the original minimax objective.

Early GAN papers downloaded quickly from arXiv; as generators began filling papers with high-resolution samples, the files grew much heavier. Not that I blame the authors.

PGGAN stabilizes training while generating $1024\times1024$ images by growing the network progressively.

Training begins with a low-resolution GAN. New layers are added as resolution increases, with the old path supporting the new one during transition.

Coefficient $\alpha$ blends the new high-resolution path with the upsampled old path like a residual connection. It grows linearly from zero to one during each resolution transition. At first the mature low-resolution path stabilizes learning; by the end, the generator should rely entirely on the new layer's output. The $2\times$ generator path uses nearest-neighbor interpolation, while the discriminator's $0.5\times$ path uses average pooling.

The complete training architecture appears above.

The paper also introduced the famous CelebA-HQ dataset, later used alongside datasets such as FFHQ-1024. That leads naturally to StyleGAN.

StyleGAN: A Style-Based Generator Architecture for GANs

StyleGAN retains progressive growing but changes the generator substantially:

• Bilinear upsampling replaces nearest-neighbor interpolation.
• A mapping network $\mathcal{Z}\rightarrow\mathcal{W}$ and AdaIN are added.
• The latent is removed from the synthesis network's direct input.
• Noise is injected into every block.
• Style-mixing regularization is introduced.

StyleGAN's generator is specialized for style-controlled synthesis. The most conspicuous change is that latent $z\in\mathcal{Z}$ no longer enters synthesis network $g$ as its initial spatial input. Instead it is converted into style information and applied through adaptive instance normalization. The generator feels less like a decoder that creates an image in one mapping and more like an artist adding successive properties to a canvas.

Rather than affine-transform $z$ directly, StyleGAN maps it through an eight-layer MLP into a new space $\mathcal{W}$.

Suppose a dataset contains men with glasses but no men without glasses. In figure (a), let the horizontal axis represent glasses and the vertical axis gender. Women with and without glasses and men with glasses are present; men without glasses leave one quadrant empty.

Forcing this irregular data manifold directly into a simple Gaussian latent $\mathcal{Z}$ folds the empty region into adjacent regions. Interpolating one attribute then changes another: attempting to change a woman with glasses into a man might unexpectedly remove the glasses because the dataset never contained the missing combination. This is entanglement.

The mapping network learns an intermediate space whose geometry can follow the data distribution, as in figure (c), and thereby improves disentanglement. The term synthesis network emphasizes that the learned styles are applied progressively to a constant tensor, much as PGGAN grows resolution: broad face structure, gender, hair color, eye size, mouth shape, and other attributes accumulate from coarse to fine.

Each block affine-transforms style $w\in\mathcal{W}$ and applies it through Adaptive Instance Normalization:

$$\operatorname{AdaIN}(x_i,~y) = y_{s,i}\frac{x_i - \mu(x_i)}{\sigma(x_i)}+y_{b,i}.$$

The $i$th feature map is standardized using its own mean and variance, then rescaled and shifted by style factors $y_s$ and $y_b$. The affine transform is simply $y=Aw$.

Styles applied at low resolution, such as $4\times4$ or $8\times8$, control coarse properties including face shape, pose, and broad hairstyle. Middle resolutions, $16\times16$ or $32\times32$, control facial features and finer hairstyle. High resolutions from $64\times64$ through $1024\times1024$ affect texture and minute detail. The figure also suggests a limitation: changing middle or fine styles still entangles parts of the background.

Style mixing uses more than one latent during synthesis. If source 1 comes from $z_1$ and source 2 from $z_2$, the generator switches from one mapped style to the other at a chosen layer.

The crossover determines whether each source contributes coarse structure or fine detail. Training with random crossovers discourages adjacent layers from assuming that correlated styles will always arrive together, further promoting disentanglement.

The final ingredient is noise injection.

A Gaussian noise map matching the spatial dimensions is scaled and added to each feature map. Noise controls stochastic details and prevents a style code from always generating one rigid realization.

Noise in coarse layers creates larger variations; noise in fine layers changes details. In the right figure, (a) adds noise at every layer, (b) uses none, (c) adds it only to fine layers, and (d) only to coarse layers. Compared with (b), coarse noise in (d) changes properties such as hairstyle, while fine noise in (c) affects skin tone, color, and individual strands of hair.

Another important detail is the loss. CelebA-HQ experiments used WGAN-GP, but FFHQ training switched to a non-saturating GAN loss with $R_1$ regularization:

$$\begin{aligned} \mathcal{L}_D &= \max_D V(D,G) \newline V(D,G) &= \mathbb{E}_{x \sim p_{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log (1-D(G(z)))] \newline \newline \mathcal{L}_G &= \max_G V'(G) \newline V'(G) &= \mathbb{E}_{z \sim p_z}[\log D(G(z))] + \frac{\gamma}{2}\mathbb{E}_{p_D(x)}[\lVert \nabla D_\psi(x) \rVert^2]. \end{aligned}$$

Rather than minimizing the saturating $\log(1-D(G(z)))$ form, the generator maximizes the probability that the discriminator calls its samples real. The $\gamma$ term is $R_1$ regularization, with $\gamma=10$.

Finally, latents in very low-density regions of $\mathcal{Z}$ or $\mathcal{W}$ are poorly supported by training and tend to yield lower-quality images. The truncation trick restricts sampling toward the mean:

$$\begin{aligned} w' =& \bar{w} + \psi(w-\bar{w}) \newline \bar{w} =& \mathbb{E}_{z \sim p(z)}[f(z)]. \end{aligned}$$

Intuitively, $\bar{w}$ generates an average face, and $\psi$ controls how far sampling may move from that well-supported center. Smaller $\psi$ improves fidelity at the cost of variation.