An Introduction to Generative Adversarial Networks (GANs)

This article introduces generative adversarial networks (GANs). I had already written about VAEs and diffusion models, so leaving out the most famous and extensively studied family of generative models felt incomplete. GAN research remains active: beyond style-controlled image generation with models such as StyleGAN, it covers 3D scene-aware generation, depth-map generation, and many other fields. One of the greatest practical advantages of GANs is their fast sampling speed.

Background

A generative model attempts to reproduce a desired data distribution. Although a GAN is not an explicit likelihood-based model, its objective likewise constructs a similar distribution. It defines a mapping from a latent space $\mathcal{Z}$ to a data space $\mathcal{X}$. Let us first review the probability concepts needed to understand that mapping.

Probability

In the figure, suppose the red random variable is $y\sim\mathcal{N}(0,1)$ and the blue one is $x\sim\mathcal{N}(0,0.5)$. Both are Gaussian for simplicity.

We can define a distribution over the variables together, their joint probability $p(x,y)$. The two variables may or may not be related. The joint of $p(x)$ and $p(y)$ appears as a three-dimensional Gaussian over the plane between their axes. Because these Gaussian marginals are symmetric around their means, the illustrated joint is isotropic.

Projecting the joint distribution $p(x,y)$ onto either axis recovers the corresponding marginal $p(x)$ or $p(y)$:

$$\begin{aligned} \int_{-\infty}^{\infty} p(x,~y) dy =& p(x) \newline \int_{-\infty}^{\infty} p(x,~y) dx =& p(y). \end{aligned}$$

For discrete probability mass functions, integration becomes summation:

$$\begin{aligned} \sum_y p(x,~y) =& p(x) \newline \sum_x p(x,~y) =& p(y). \end{aligned}$$

Integrating along an axis in multiple dimensions adds all of its values and therefore acts as a projection.

As noted above, the variables may be related. If they are independent—say, the chance that a passerby is my ideal type and the chance of rain tomorrow—the relationship is irrelevant. When they are dependent, conditional probability contains information distinct from the marginal and joint. The distribution of $y$ given a particular $x$ is

$$p(y \vert x) = \frac{p(x,~y)}{p(x)}.$$

Modeling and Sampling a Probability Density

A traditional density-modeling process can be summarized as follows:

1. Collect sample data whose distribution we want to estimate.
2. Estimate an empirical probability density function from the samples.
3. Fit a standard parametric density to that empirical density.
4. Estimate its parameters with maximum-likelihood estimation, Bayesian estimation, or another method.

Sampling reverses the fitted mapping: $p(x)\rightarrow x$. This brief description assumes familiarity with MLE and Bayesian methods, so we will move on rather than derive them here.

Generative and Discriminative Models

In one sentence, a GAN improves performance by making a generator $G$ and discriminator $D$ compete. Modern life is competitive enough already, with everyone worrying about résumés and self-improvement. I am not sure that analogy helps.

The generator and discriminator correspond to two broad classes of statistical model. A generative model represents a joint density $p(x,y;\theta)$ or a marginal density $p(x;\theta)$. Neural networks learn functions implicitly; here we can think of the generator as learning parameters that define a distribution. A discriminative model instead learns a conditional density $p(y\mid x;\theta)$ and distinguishes plausible data $x$ produced by the generator.

Although both were written with $\theta$, the two networks actually optimize separate parameter sets, such as $\phi\in\Phi$ and $\theta\in\Theta$. Examples of generative models include Gaussian mixture models, hidden Markov models, Bayesian networks, deep belief networks, and deep Boltzmann machines. Discriminative models include familiar encoders for low-level tasks: MLPs, CNNs, logistic regression, SVMs, and many others.

General Data Generation

The broad aim of a generative model is to use a decoder-like generator to create plausible data $x$ by learning $p(x;\theta)$. Because the real data density is difficult to obtain directly, we often introduce a joint distribution $p(x,z)$ and optimize quantities we can access without needing every term in Bayes' rule:

$$p(z \vert x) = \frac{p(x \vert z)p(z)}{p(x)}.$$

Here we focus not on explicit optimization of likelihood $p(x\mid z)$, but on the marginal $p(x)$. Integrating the joint over every $z$ averages the likelihood over a prior distribution that we can choose:

$$p(x) = \int_z p(x \vert z) p(z) dz.$$

A common choice is $p(z)=\mathcal{N}(0,I)$. A sampled $z$ is called a latent vector or latent variable. Passing it through the generator produces a sample under the model induced by $p(z)$ and $p(x\mid z)$, which we try to fit to the empirical data distribution.

Characteristics of GANs

The original GAN paper contains the full core idea. A GAN learns an implicit density. Because it receives no direct supervision for a joint probability from which to create $x$, it can be viewed as an unsupervised approach.

The model has two networks: a generative network $G$ and a discriminative network $D$, trained in competition.

The usual analogy is a counterfeiter and the police. The generator, or counterfeiter, tries to create convincing fake data; the discriminator, or police, learns to distinguish fake examples from genuine ones. The minimax objective is

$$\begin{aligned} &\min_G \max_D V(D,G) \newline V(D,G) =& \mathbb{E}_{x \sim p_{data}(x)}[\log D(x)] + \mathbb{E}_{z \sim p_z(z)}[\log (1-D(G(z)))]. \end{aligned}$$

The logarithms are log-likelihood terms applied to probabilities between zero and one.

The Discriminator's Perspective

A real example $x\sim p_{data}(x)$ should be classified as true, so an ideal discriminator makes $D(x)=1$ and $\log D(x)=0$. A fake example $G(z)$ produced from $z\sim p(z)$ should be classified as false, giving $D(G(z))=0$ and $\log(1-D(G(z)))=0$.

The discriminator therefore maximizes $V(D,G)$. If it seems strange that the maximum is zero, remember that the log of a probability in $[0,1]$ lies in $[-\infty,0]$.

The Generator's Perspective

The generator affects only the second term. It wants fake data $G(z)$ to fool the discriminator—that is, it wants the discriminator to call the fake genuine. Under the original minimax form, the generator therefore minimizes $\log(1-D(G(z)))$, whose ideal limit is $-\infty$.

Together, these perspectives explain the minimax game: the two networks optimize in opposite directions, and their competition shapes both parameter sets.

Along one axis the objective is convex and along the other it is concave, so the desired global equilibrium is a saddle point. A figure in the original paper makes this process more intuitive.

Blue shows the discriminator's decision function, black the real data distribution, and green the generator distribution. In (a), neither network is well optimized: the generator's mapping $z\rightarrow x$ does not follow the real data, and the discriminator separates examples only roughly. After discriminator training in (b), it distinguishes real from fake more confidently. Training the generator from this state produces (c), where the green generated distribution moves toward the black data distribution. At convergence in (d), the generator matches the real distribution and the discriminator can no longer tell real and fake apart. This is the saddle-point equilibrium described above.

The training algorithm performs $K$ discriminator updates for each generator update. In terms of the figure,

$$\text{iterations}\times\left(K\times((a)\rightarrow(b))+1\times(c)\right)=(d).$$

The Objective as Jensen–Shannon Divergence

The minimax expression can be rewritten in terms of Jensen–Shannon divergence:

$$\begin{aligned} C(G) =& \max_D V(G,~D) \newline =& \mathbb{E}_{x \sim p_{data}}[\log D_G^\ast (x)] + \mathbb{E}_{z \sim p_z}[\log(1-D_G^\ast(G(z)))] \newline =& \mathbb{E}_{x \sim p_{data}}[\log D_G^\ast (x)] + \mathbb{E}_{x \sim p_g}[\log(1-D_G^\ast(x))] \newline =& \mathbb{E}_{x \sim p_{data}} \left[ \log \frac{p_{data}(x)}{p_{data}(x)+p_g(x)} \right] + \mathbb{E}_{x \sim p_g} \left[ \log \frac{p_g(x)}{p_{data}(x)+p_g(x)} \right]. \end{aligned}$$

Here the distribution induced by the generator is denoted directly by $p_g$, absorbing the explicit $z\rightarrow x$ mapping. At the optimum, $p_g=p_{data}$ and $D_G^\ast=1/2$. Substitution gives

$$\begin{aligned} C(G) =& \mathbb{E}_{x \sim p_{data}} \left[ \log \frac{p_{data}(x)}{p_{data}(x)+p_g(x)} \right] + \mathbb{E}_{x \sim p_g} \left[ \log \frac{p_g(x)}{p_{data}(x)+p_g(x)} \right] \newline =& \mathbb{E}_{x \sim p_{data}} \left[ \log \left(\frac{p_{data}(x)}{\frac{p_{data}(x)+p_g(x)}{2}} \times \frac{1}{2}\right) \right] \newline &+ \mathbb{E}_{x \sim p_g} \left[ \log \left(\frac{p_g(x)}{\frac{p_{data}(x)+p_g(x)}{2}} \times \frac{1}{2}\right) \right] \newline =& -\log(4) + D_{KL}\left( p_{data} \parallel \frac{p_{data}+p_g}{2} \right) + D_{KL} \left( p_g \parallel \frac{p_{data}+p_g}{2} \right) \newline =& -\log(4) + 2 \cdot D_{JS}(p_{data} \parallel p_g). \end{aligned}$$

The figure shows generated results on MNIST, TFD, and CIFAR-10.

Metrics for GANs

A GAN is a generative model, so ordinary supervised metrics for discriminative models do not directly evaluate it. A good generator must produce not only attractive images, but diverse and plausible ones. Three metrics are commonly discussed.

IS: Inception Score

Inception Score measures both quality and diversity:

$$IS(G) = \exp\left(\mathbb{E}_{x \sim G} [D_{KL}(p(y \vert x)\parallel p(y))]\right).$$

If generated images are diverse, the marginal $p(y)$ should spread across classes and have high entropy. This term reflects diversity. At the same time, each generated image $x$ should look unambiguously like its predicted label $y$. Confidence in the individual sample reflects quality, so $p(y\mid x)$ should have low entropy.

The conditional map $p(y\mid x)$ is the softmax output of an Inception network, and the marginal over all samples is

$$p(y)=\frac{1}{N}\sum_{i=1}^N p(y\mid x_i).$$

Low diversity makes $p(y)$ sparse, resembling a one-hot vector; high diversity makes it dense and closer to a high-entropy uniform distribution. Across generated samples,

$$IS = \exp \left( \frac{1}{N} \sum_{i=1}^N D_{KL} (p(y \vert x_i)\parallel p(y)) \right).$$

FID: Fréchet Inception Distance

Inception Score has a serious weakness: a generator can produce only one image per class—a form of mode collapse—and still appear diverse. If it overfits to a single easy sample for every class, IS can remain high.

Fréchet Inception Distance instead extracts intermediate Inception features and models their multivariate Gaussian distributions using means $\mu$ and covariances $\Sigma$:

$$FID(x, g) = \lVert \mu_x - \mu_g \rVert^2_2 + \operatorname{Tr}(\Sigma_x + \Sigma_g - 2(\Sigma_x \Sigma_g)^{1/2}).$$

The metric compares the distributions of real and generated data and is generally more robust to noise than IS. A generator that emits only one image per class differs strongly from the covariance structure of the real data, producing a worse—larger—FID. FID therefore captures within-class diversity more effectively.

LPIPS: Learned Perceptual Image Patch Similarity

LPIPS is comparatively simple:

$$d(x,~x_0) = \sum_l \frac{1}{H_l W_l} \sum_{h, w} \lVert w_l \odot (\hat{y}_{hw}^l - \hat{y}_{0hw}^l) \rVert_2^2.$$

Feature extractors such as AlexNet, VGG, or SqueezeNet produce activation maps for reference image $x_0$ and measured image $x$. LPIPS calculates the Euclidean distance between activations $\hat{y}_{hw}^l$ and $\hat{y}_{0hw}^l$, scales channels by layer weights $w_l$, averages over spatial positions, and sums across layers. Because it compares learned feature maps rather than raw pixels, it serves as a perceptual patch-similarity metric. The layer weights $w_l$ are learned parameters, which explains the “Learned” in the paper's title.