After going through several company interviews and technical interviews, I noticed that certain questions kept coming up. A little embarrassingly, this list is less a collection of model answers gathered from acing interviews than a notebook of mistakes that grew by one line every time I was rejected. Some companies asked me to derive equations, others asked about trade-offs from implementation and systems perspectives, and still others asked me to connect the same concept across vision and language models.

Even topics I was sure I understood became tangled at the definition when I tried to explain them in front of an interviewer. Sometimes I brought up several similar terms at once, only to fail to articulate their differences. The answer would occur to me belatedly on the way home, and only after receiving a rejection would I go back to the papers and code. Apparently, embarrassment over rejection makes us more diligent than the joy of acceptance does.

What I learned in the process is that interviews are not about reciting as many terms as possible. What matters is explaining logically why a method was needed, what problem it solves, and what it gives up in return. For example, saying that FlashAttention is simply “a way to make attention faster” is not enough. You should distinguish memory complexity from compute complexity and explain that it preserves exact attention while reducing HBM I/O and intermediate-matrix storage.

Writing this post does not mean that I can now answer every question perfectly. If anything, each interview showed me how much I still did not know precisely. I simply wanted to reduce the number of times I found myself speechless twice in a row when faced with the same question. This post brings together 30 essential topics I encountered through that cycle of rejection and review. Each section first explains the underlying principle, then closes with an answer suitable for a 30-second to one-minute interview response. Rather than memorizing only the short answer, I recommend understanding the reasoning that precedes it and restating it in your own words.

A basic structure for interview answers: Give a one-sentence definition, explain the mechanism and equations, then discuss advantages, limitations, or a point of comparison. For a systems question, it also helps to finish by stating under what conditions you would choose the method.

1. Why Classification Uses Cross-Entropy Loss, Derived from the MLE Perspective

Given an input $x$, a classification model predicts the conditional probability $p_\theta(y=k\mid x)$ that each class $k$ is correct. If the logit is $z_k$, multi-class classification generally converts logits to probabilities with Softmax.

$$p_\theta(y=k\mid x) = \frac{\exp(z_k)}{\sum_{j=1}^{K}\exp(z_j)}.$$

Assuming that the samples in the dataset $\mathcal{D}=\{(x_i,y_i)\}_{i=1}^{N}$ were observed independently, the full likelihood is as follows.

$$\mathcal{L}(\theta) = \prod_{i=1}^{N}p_\theta(y_i\mid x_i).$$

Maximum Likelihood Estimation finds the parameters that maximize the probability of observing this data.

$$\theta_{\mathrm{MLE}} = \arg\max_\theta \prod_{i=1}^{N}p_\theta(y_i\mid x_i).$$

Because a product is numerically unstable and inconvenient to differentiate, we take the logarithm. Since the logarithm is monotonically increasing, it does not change the location of the maximum.

$$\theta_{\mathrm{MLE}} = \arg\max_\theta \sum_{i=1}^{N}\log p_\theta(y_i\mid x_i).$$

Training code is usually written to minimize an objective, so negating it gives the Negative Log-Likelihood.

$$\mathcal{J}(\theta) = -\sum_{i=1}^{N}\log p_\theta(y_i\mid x_i).$$

If the target is represented as a one-hot vector $q_{ik}=\mathbb{1}[y_i=k]$, the objective can be rewritten as follows.

$$\mathcal{J}(\theta) = -\sum_{i=1}^{N}\sum_{k=1}^{K} q_{ik}\log p_\theta(y=k\mid x_i).$$

This is precisely Categorical Cross-Entropy. In other words, Cross-Entropy is not an arbitrarily chosen loss that happens to work well for classification; it is the Negative Log-Likelihood of MLE under a categorical-distribution assumption.

The same conclusion follows from a distributional perspective.

$$H(q,p_\theta) = H(q)+D_{\mathrm{KL}}(q\Vert p_\theta).$$

Because the target distribution $q$ is fixed, $H(q)$ is independent of the model parameters. Minimizing Cross-Entropy is therefore equivalent to minimizing the KL divergence between the empirical target distribution and the model distribution. Its gradient with respect to the logits also simplifies to $p_k-q_k$, providing a stable signal that lowers the probability of incorrect classes and raises the probability of the correct class.

The same reasoning applies to binary classification. If $y\in\{0,1\}$ follows a Bernoulli distribution, the Negative Log-Likelihood becomes Binary Cross-Entropy.

$$\mathcal{L}_{\mathrm{BCE}} = -y\log p-(1-y)\log(1-p).$$

Interview answer: A classification model's output can be viewed as the parameters of a categorical distribution. The dataset likelihood is the product of the probabilities assigned to the correct classes, and MLE maximizes that product. Taking the logarithm turns the product of probabilities into a sum of log probabilities; negating it to obtain a minimization problem gives $-\sum_i\log p_\theta(y_i\mid x_i)$. With one-hot targets, this is exactly Cross-Entropy. Thus, minimizing Cross-Entropy is equivalent to categorical MLE and can also be interpreted as reducing the KL divergence between the target and predicted distributions.

2. What an LLM Is and Its Full Training Pipeline, Including Pre-training, SFT, and RLHF

An LLM is a language model that uses a large text corpus and many parameters to learn a probability distribution over token sequences. There is no absolute parameter threshold for “large.” More important than scale itself is that the model has general-purpose language representations and generation capabilities, learned with large amounts of data and compute, that transfer to diverse tasks.

The basic objective of a decoder-only LLM is to predict the next token given the preceding tokens.

$$p_\theta(x_{1:T}) = \prod_{t=1}^{T} p_\theta(x_t\mid x_{<t}), \qquad \mathcal{L}_{\mathrm{PT}} = -\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t}).$$

The full pipeline can be divided into the following stages.

1. Data and Tokenizer Preparation

Data is collected from the web, books, code, papers, conversations, and other sources, then deduplicated, filtered for quality, stripped of personal and harmful data, and adjusted for the desired domain mixture. A subword tokenizer such as BPE, WordPiece, or Unigram is then trained. The composition of the dataset directly affects what the model knows and which languages and perspectives it expresses most effectively.

2. Pre-training

The model performs next-token prediction on a large unlabeled corpus. Because teacher forcing is used, the losses at every position in a sequence can be computed in parallel. At this stage, grammar, semantics, factual knowledge, code patterns, and some reasoning ability are compressed into the parameters. Continued pre-training or mid-training may additionally train the base model on domain-specific data to strengthen a particular domain.

3. Supervised Fine-Tuning

A pre-trained model predicts the next token well, but it does not necessarily follow user instructions well. SFT trains it on $(\text{instruction},\text{response})$ pairs to teach it to answer questions and follow requested formats.

$$\mathcal{L}_{\mathrm{SFT}} = -\sum_{t\in\mathrm{response}} \log\pi_\theta(y_t\mid x,y_{<t}).$$

The user-prompt positions are usually masked from the loss, which is applied only to the assistant response. SFT is stable and simple to implement, but because it directly imitates target demonstrations, it struggles to explore better answers absent from the data and may overfit to annotation style.

4. Preference Data and the Reward Model

Several responses are generated for the same prompt, and a human selects the better one. If the chosen response is $y_w$ and the rejected response is $y_l$, the Reward Model can be trained with a Bradley–Terry formulation.

$$\mathcal{L}_{\mathrm{RM}} = -\log\sigma \left( r_\phi(x,y_w)-r_\phi(x,y_l) \right).$$

The Reward Model is an evaluator that assigns a higher scalar score to responses humans are more likely to prefer.

5. RLHF or Preference Optimization

Classical RLHF treats the language model as a policy and uses PPO or a similar algorithm to increase reward. A KL penalty against the SFT or reference model is used to prevent reward hacking and excessive policy drift.

$$\max_\theta \mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)} \left[ r_\phi(x,y) - \beta D_{\mathrm{KL}}( \pi_\theta\Vert\pi_{\mathrm{ref}} ) \right].$$

DPO directly optimizes the policy from preference pairs without separately training a Reward Model or performing on-policy RL rollouts. RLVR has also become important in recent reasoning models; instead of relying on human evaluation, it uses automatically verifiable rewards such as answer checkers, code tests, or equation verifiers.

6. Distillation, Evaluation, and Deployment

A smaller student learns from responses or probability distributions produced by a large teacher, reducing cost. The resulting model is then evaluated for knowledge, reasoning, safety, bias, long-context handling, tool use, and real serving latency. Optimizations such as quantization, KV caching, and speculative decoding make the learned capabilities affordable to serve in practice.

Not every LLM uses every stage above. A base model may stop after pre-training, while an instruct model may use only SFT and DPO. The important distinction is that pre-training provides the language distribution and knowledge, SFT provides the format for following instructions, and the preference/RL stage provides the direction of preference and exploration. I cover this in more detail in How Are LLMs Trained?.

Interview answer: An LLM is a general-purpose language model that learns a probability distribution over token sequences from a large corpus. Pre-training teaches language and knowledge through next-token prediction, while SFT uses high-quality instruction–response pairs to teach the model how to follow instructions. RLHF then trains a Reward Model from human comparison data and raises reward with PPO, while a KL penalty against a reference model prevents excessive change. Alternatives include DPO, which learns directly from preference pairs, and RLVR, which uses automatically verifiable rewards such as mathematical answers and unit tests. Finally, the model is deployed after distillation, safety evaluation, quantization, and serving optimization.

3. The Full Structure of a Transformer Block and the Role of Each Component

Let the input to a Transformer block be $X\in\mathbb{R}^{B\times L\times d}$. A Pre-LayerNorm block common in modern LLMs can be written as follows.

$$U = X+\operatorname{MHA}(\operatorname{LN}(X)),$$ $$Y = U+\operatorname{FFN}(\operatorname{LN}(U)).$$

In other words, a block consists primarily of attention, which mixes information between tokens, and an FFN, which transforms the channels within each token, with a residual connection around each sublayer.

Self-Attention

Queries, keys, and values are produced from the same input.

$$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.$$ $$\operatorname{Attention}(Q,K,V) = \operatorname{softmax} \left( \frac{QK^\top}{\sqrt{d_h}}+M \right)V.$$

$QK^\top$ indicates how much each token should attend to every other token, while $V$ contains the information actually retrieved. Dividing by $\sqrt{d_h}$ prevents Softmax saturation as the variance of dot products grows with dimensionality. $M$ contains a padding mask or a causal mask that blocks future tokens.

Multi-Head Attention divides the $d$-dimensional space into multiple heads that learn different relationships in parallel.

$$\operatorname{MHA}(X) = \operatorname{Concat}(head_1,\ldots,head_H)W_O.$$

Each head does not necessarily separate into a single human-interpretable role such as syntax, position, or pronoun resolution, but multiple heads can represent relationships in more varied subspaces than a single attention map.

Feed-Forward Network

The FFN consists of two linear projections and a nonlinear function, applied independently to each token.

$$\operatorname{FFN}(x) = W_2\phi(W_1x+b_1)+b_2.$$

It generally expands the hidden dimension $d$ to a larger $d_{ff}$ and then projects it back down. In addition to GELU and ReLU, modern LLMs commonly use gated FFNs such as SwiGLU.

$$\operatorname{SwiGLU}(x) = W_o\left[ \operatorname{SiLU}(W_gx)\odot(W_ux) \right].$$

If attention is responsible for transferring information between tokens, the FFN processes that information nonlinearly at each position and creates new features.

Residual Connection

The sublayer output is added to its input.

$$y=x+F(x).$$

The residual path provides a direct route for gradients even in a deep network, allowing each block to learn only the required change to the existing representation instead of reconstructing the entire representation. The identity path preserves information even if attention or the FFN temporarily produces a poor output.

LayerNorm

LayerNorm normalizes the mean and variance across each token's hidden dimension.

$$\operatorname{LN}(x) = \gamma\odot \frac{x-\mu}{\sqrt{\sigma^2+\epsilon}} +\beta.$$

Because it does not depend on other samples in the batch, LayerNorm remains stable with variable-length sequences and small batches. The original Transformer used Post-LN in the form $\operatorname{LN}(x+F(x))$, but Pre-LN, $x+F(\operatorname{LN}(x))$, is widely used in deep models because of its gradient stability. Pre-LN and Post-LN are not identical, however, because their representations and final normalization placement differ.

Positional Information and Architecture-Specific Additional Blocks

Self-attention alone cannot distinguish order, so models use absolute position embeddings, relative biases, RoPE, or similar methods. Encoder blocks generally use bidirectional self-attention. Decoder-only LLMs use causal self-attention. In an encoder–decoder Transformer, the decoder has an additional cross-attention sublayer between self-attention and the FFN, using encoder outputs as its keys and values.

Interview answer: A Transformer block is composed mainly of two sublayers: attention and an FFN. Self-attention computes relationships between tokens with $QK^T$ and mixes information along the sequence by taking a weighted sum of values. The multi-head structure represents relationships in several subspaces in parallel. The FFN is applied independently to each token and nonlinearly transforms its hidden channels. Residual connections provide a direct path for information and gradients, while LayerNorm stabilizes the scale of each token's hidden dimensions. Modern LLMs generally use a Pre-LN structure, and decoders add a causal mask to hide future tokens.

4. Making Attention Efficient in Terms of Memory and Computation

In full attention with sequence length $L$ and head dimension $d_h$, $QK^\top$ has shape $L\times L$. The approximate compute cost is therefore $O(L^2d_h)$, while storing attention scores and probabilities requires $O(L^2)$ memory. Methods that reduce memory are not necessarily the same as methods that reduce total computation.

A Representative Method for Reducing Memory and I/O: FlashAttention

A conventional implementation writes large intermediate matrices such as $S=QK^\top$ and $P=\operatorname{softmax}(S)$ to GPU HBM and then reads them back. FlashAttention divides Q, K, and V into tiles, loads them into SRAM, and accumulates results block by block using online Softmax.

1. Load a block of Q and blocks of K and V into SRAM.
2. Compute a small $QK^\top$ block.
3. Update the row-wise maximum and normalization sum online.
4. Accumulate the normalized value sum without materializing a large $L\times L$ matrix in HBM.
5. During the backward pass, recompute some intermediate values that were not stored.

This method substantially reduces HBM reads and writes and activation memory while preserving exact Softmax attention. It still computes every pair in full attention, however, so the theoretical compute complexity remains $O(L^2d)$. FlashAttention-2 further improves work partitioning and GPU occupancy.

Gradient checkpointing also reduces memory by omitting activations for blocks, including attention blocks, and recomputing them during the backward pass. Sequence parallelism or context parallelism divides a sequence across multiple GPUs to lower per-device memory, but does not automatically reduce the total FLOPs performed by the cluster.

Representative Methods for Reducing Total Computation

Sparse Attention: Instead of considering every token pair, compute only a local window, a dilated or block-sparse pattern, and a small set of global tokens. Longformer combines local sliding-window and global attention to reduce complexity to approximately $O(Lw)$. It exploits the inductive bias that nearby-token relationships matter in long documents, but can miss long-range relationships outside the sparse pattern.

Low-rank and Kernel Attention: Project attention into a lower-dimensional space or reorder operations with a kernel feature map so that the full $L\times L$ matrix is never constructed. Performer approximates the Softmax kernel with random features, targeting approximately linear time and memory.

$$\operatorname{Attn}(Q,K,V) \approx \phi(Q) \left( \phi(K)^\top V \right).$$

This avoids constructing the full $QK^\top$, but introduces trade-offs among approximation error, training stability, practical kernel efficiency, and quality.

Latent bottleneck: Architectures such as Perceiver use cross-attention from a long input into a small number of latent queries, reducing the length of subsequent computation. Compressing all input information into a limited set of latents can create a bottleneck.

Memory Optimization for Autoregressive Decoding

During inference, KV-cache size and memory bandwidth may be greater problems than the attention matrix. MQA has every query head share one K/V head, while GQA has groups of query heads share K/V heads. This reduces the KV cache by a factor of $H_{kv}/H_q$, but it does not simply eliminate the training compute of a conventional, already-trained MHA model. KV quantization, PagedAttention, and sliding-window caches are optimizations from the same perspective.

Interview answer: Full attention has compute complexity $O(L^2d)$ and intermediate-memory complexity $O(L^2)$ because of its $L\times L$ score matrix. FlashAttention, a representative memory-oriented method, uses tiling and online Softmax so that the attention matrix is never stored in HBM, reducing I/O and activation memory. Because it still evaluates every token pair, however, total FLOPs remain quadratic. To reduce computation as well, you must limit the evaluated pairs with sliding-window or block-sparse attention, or use a linear approximation by reordering operations as in kernel-attention methods such as Performer. During autoregressive inference, MQA/GQA, KV quantization, and PagedAttention reduce the KV cache and memory bandwidth.

5. On-Policy and Off-Policy Reinforcement Learning, and Their Relationship to Distillation

Given a policy $\pi$, the key distinction is which policy generated the experience used for training.

On-policy

The policy currently being trained, $\pi_\theta$, is updated with trajectories that it collected itself. Policy Gradient, A2C, and PPO are representative examples.

$$\tau\sim\pi_\theta, \qquad \theta\leftarrow \theta+\alpha\nabla_\theta J(\theta;\tau).$$

Because the current policy's state–action distribution matches the training-data distribution, bias is relatively low. As the policy changes, however, old data becomes less valid, so new rollouts must be generated continually and sample efficiency is low. In LLM PPO as well, the current policy generates responses, rewards and advantages are computed, and after several updates the model generates another set of responses.

Off-policy

The target policy $\pi$ is trained on experience generated by a past policy, another behavior policy $\mu$, humans, or a replay buffer. Representative algorithms include DQN, DDPG, TD3, and SAC. Reusing data gives high sample efficiency, but a large distributional gap between $\mu$ and $\pi$ introduces bias and instability. Importance sampling can correct some of this mismatch.

$$\mathbb{E}_{a\sim\pi}[f(a)] = \mathbb{E}_{a\sim\mu} \left[ \frac{\pi(a\mid s)}{\mu(a\mid s)}f(a) \right].$$

Strictly speaking, PPO belongs to the on-policy family because it uses batches collected from the current policy. It differs from vanilla policy gradient, however, because it reuses a batch for multiple epochs and clips the ratio against the old policy.

Is Distillation On-Policy or Off-Policy?

Distillation is not itself an RL algorithm. It is a training principle in which a student imitates a teacher's probability distribution, hidden representation, or generated sequence.

$$\mathcal{L}_{\mathrm{KD}} = T^2D_{\mathrm{KL}} \left( p_T(\cdot\mid x;T) \Vert p_S(\cdot\mid x;T) \right).$$

It can nevertheless be viewed through an on/off-policy-like lens depending on the inputs and states at which teacher signals are obtained.

• Training a student on a fixed teacher-generated dataset resembles offline or off-policy behavior cloning, independent of the student's current distribution.
• Token-level KD that follows the teacher's token distribution from ground-truth prefixes is easy to train, but does not expose the student to erroneous states it actually visits during generation.
• Online distillation, which obtains the teacher's distribution and feedback on prefixes or trajectories generated by the student itself, learns on the student-induced state distribution and can reduce distribution shift and exposure bias.
• Applying SFT to a small student using the answers and reasoning traces of a large RL-trained reasoning teacher is sequence-level distillation that transfers the results of RL exploration to a cheaper model.

A representative example is DeepSeek-R1, which distilled data generated by a large reasoning model into smaller Qwen- and Llama-family models. The student can imitate reasoning patterns discovered by the teacher without performing RL itself. Policy distillation, conversely, is used to compress several RL expert policies into one student policy or transfer a large RL policy into a smaller serving policy.

Interview answer: On-policy methods train the current policy on trajectories produced by that same policy; PPO is a representative example. The matching distributions make training stable, but continual new rollouts lead to low sample efficiency. Off-policy methods reuse data from past or different behavior policies; examples include DQN and SAC. They are efficient, but must manage distribution mismatch. Distillation itself is neither on-policy nor off-policy RL. Imitating fixed teacher data resembles offline or off-policy learning, while online distillation that receives teacher signals at states actually visited by the student is closer to the on-policy distribution. Applying SFT to a small model using the reasoning outputs of an RL-trained teacher is another important use case.

6. How CLIP Learns, the Limitations of Contrastive Learning, and Subsequent Research

CLIP is a dual-encoder architecture with separate image encoder $f$ and text encoder $g$. It brings embeddings of matching image–text pairs closer while pushing nonmatching pairs apart.

For a batch containing $B$ image–text pairs, define the L2-normalized embeddings as follows.

$$v_i = \frac{f(x_i)}{\Vert f(x_i)\Vert}, \qquad t_j = \frac{g(c_j)}{\Vert g(c_j)\Vert}.$$

Logits are computed from cosine similarity and a learnable temperature $\tau$.

$$s_{ij} = \frac{v_i^\top t_j}{\tau}.$$

The matching $(i,i)$ pair is the target. The image-to-text and text-to-image retrieval losses are computed symmetrically.

$$\mathcal{L}_{I\rightarrow T} = -\frac{1}{B}\sum_{i=1}^{B} \log \frac{\exp(s_{ii})} {\sum_{j=1}^{B}\exp(s_{ij})},$$ $$\mathcal{L}_{T\rightarrow I} = -\frac{1}{B}\sum_{j=1}^{B} \log \frac{\exp(s_{jj})} {\sum_{i=1}^{B}\exp(s_{ij})}.$$ $$\mathcal{L}_{\mathrm{CLIP}} = \frac{1}{2} \left( \mathcal{L}_{I\rightarrow T} + \mathcal{L}_{T\rightarrow I} \right).$$

After training, class names can be converted into prompts such as “a photo of a {class},” and image–text embedding similarities can be compared to perform zero-shot classification without training a separate classifier. CLIP's most important shift was replacing a fixed label space with natural language.

Limitations of Contrastive Learning-Based Training

False negatives: Every other sample in the batch is treated as a negative, even though some may represent the same concept. Two images of dogs or two semantically equivalent captions may be forced apart.

Dependence on batch size: More in-batch negatives enable harder comparisons but require larger batches and more devices. In some cases, the quality of negatives matters more than their number.

Limits of global representations: Compressing each image and text into a single vector can discard fine-grained correspondence between objects and words, as well as position, quantity, relations, and compositional meaning. “A dog chases a cat” and “A cat chases a dog” may end up close because they share similar global concepts.

Noisy web pairs and shortcuts: Alt text may not describe the whole image and may contain biased correlations. Rather than using visual evidence, the model may exploit shortcuts such as text frequency, backgrounds, or watermarks.

The gap between understanding and generation: A dual encoder is efficient for retrieval and zero-shot classification, but does not directly learn token-level fusion or text generation. VQA, captioning, and grounding require additional architectures and objectives.

SigLIP: Replacing Global Softmax with Pairwise Sigmoid

SigLIP retains CLIP's dual encoder and image–text alignment, but replaces InfoNCE's batch-wide Softmax with an independent binary-classification problem for every image–text pair. It assigns $z_{ij}=+1$ to pairs with matching indices and $z_{ij}=-1$ to all other pairs. If $\alpha$ is a learnable logit scale and $b$ a learnable bias, its loss can be written as follows.

$$a_{ij} = \alpha v_i^\top t_j+b, \qquad z_{ij} = \begin{cases} +1, & i=j,\\ -1, & i\ne j, \end{cases}$$ $$\mathcal{L}_{\mathrm{SigLIP}} = -\frac{1}{B} \sum_{i=1}^{B}\sum_{j=1}^{B} \log \sigma\!\left(z_{ij}a_{ij}\right).$$

In CLIP's Softmax loss, the probability of one pair depends on every similarity in the same row or column through normalization. SigLIP separates the loss for each pair, so it needs no global normalization value. In distributed training, rather than constructing the full $B\times B$ logit matrix at once or all-gathering every embedding, text-embedding blocks can circulate among devices while losses over small logit blocks are accumulated. This reduces memory and communication overhead. The paper reports better performance than Softmax loss particularly for batch sizes below $16\mathrm{k}$, while the gains from further increasing batch size saturate quickly.

SigLIP does not solve every limitation of CLIP. In its basic formulation, pairs $(i,j)$ with $i\ne j$ are still treated as negatives, so false negatives remain. Because it remains a global-embedding dual encoder, fine-grained grounding and generation also require separate remedies. More precisely, it is best understood as research that made the training objective tied to large batches and global Softmax normalization simpler and more scalable.

Other General Directions for Improvement

• To mitigate false negatives, allow multiple semantically similar captions for one image to be positives, or use similarity-based soft targets instead of hard labels. Hard-negative mining and debiased contrastive losses can also replace pushing every nonmatching pair in a batch apart equally.
• For fine-grained alignment, learn correspondences between regions or patches and words or phrases, rather than comparing only whole images with whole sentences. Region-level tasks such as phrase grounding, object detection, and semantic segmentation fall into this category.
• To improve data quality, remove duplicates and low-quality pairs, regenerate captions, or filter them to raise the proportion of pairs that accurately describe their images. This improves the quality of the learning signal instead of merely increasing the number of negatives.
• For fusion and generation, directly combine image and text tokens with cross-attention and jointly train objectives such as image–text matching, masked language modeling, and captioning. This extends the model beyond retrieval into VQA, caption generation, and multimodal reasoning.

The post-CLIP trend is therefore not an abandonment of contrastive objectives. It is more accurately described as making the alignment loss itself more efficient, as in SigLIP, or adding token-level fusion, generation, grounding, and data bootstrapping on top of global alignment.

Interview answer: CLIP computes cosine similarity between L2-normalized embeddings from image and text encoders, then uses a symmetric InfoNCE loss in which matching pairs are positives and all other pairs in the batch are negatives. This enables zero-shot transfer by using natural-language prompts like a classifier. Its limitations include false negatives, the cost of large batches and global Softmax normalization, weak fine-grained grounding from global vectors, noisy web captions, and no generative capability. SigLIP applies an independent sigmoid loss to every image–text pair for more efficient training without global normalization, but it does not eliminate false negatives or the limitations of global representations. Other limitations can be addressed with multiple positives or soft targets, patch–token alignment and grounding tasks, data filtering and recaptioning, and cross-attention plus captioning objectives.

7. Representative Vision–Language Models After CLIP and the Evolution of Representation Learning

After CLIP, vision–language research progressed from building a single shared embedding space toward finely integrating the two modalities and using the generative capabilities of language models.

ALBEF: Align Before Fuse

ALBEF first aligns global representations from the image and text encoders with an image–text contrastive loss. It then fuses the two token sequences using cross-attention in a multimodal encoder and performs image–text matching and masked language modeling. Soft targets from a momentum teacher also reduce the problem of treating the single caption attached to web data as an absolute target.

From a representation perspective, it jointly provides separate unimodal spaces suited to retrieval and a fused space needed for VQA and reasoning.

BLIP: Unifying Understanding and Generation

BLIP's MED (Mixture of Encoder–Decoder) uses the same Transformer as a text encoder, an image-grounded text encoder, and an image-grounded text decoder. ITC handles global alignment, ITM handles pair-level fusion, and the LM loss handles caption generation. It also uses CapFilt, in which a captioner generates captions for web images and a filter removes noisy pairs.

Representations thus expand beyond a simple similarity space into a form that supports both bidirectional understanding and autoregressive generation.

CoCa: Combining Contrastive Learning and Captioning in One Graph

The first part of CoCa's decoder omits cross-attention to create a unimodal text representation, while the latter part cross-attends to image features to generate a caption. It applies a contrastive loss to image and unimodal text embeddings, and a captioning loss to the multimodal decoder.

By combining the transferability of a contrastive objective with the fine-grained token supervision of a captioning objective, it creates representations transferable to recognition, retrieval, VQA, and captioning.

Flamingo: Adding Visual In-Context Learning to a Frozen LM

Flamingo leaves most of a pre-trained vision encoder and language model frozen. A Perceiver Resampler compresses variable-length visual tokens into a fixed number of latents, and gated cross-attention layers are inserted between language-model layers. It is trained with next-token prediction over interleaved image and text sequences.

The significance of this architecture is that, instead of relearning visual representations from scratch, it connects a strong visual representation to an LLM's in-context learning space. One model can accept few-shot examples in its prompt and perform a range of vision–language tasks.

BLIP-2: Q-Former as a Representation Bridge

BLIP-2 places a small Q-Former between a frozen image encoder and a frozen LLM. In its first stage, ITC, ITM, and image-grounded text generation train learnable queries to extract text-relevant information from image features. In the second stage, Q-Former outputs are projected into the LLM embedding space, allowing the frozen LLM to generate text conditioned on an image.

Q-Former is both a bottleneck that compresses high-dimensional visual tokens into a small number of query representations and a modality adapter. It showed that well-trained vision and language representations can be preserved while their spaces are connected with relatively few trainable parameters.

LLaVA: Visual Instruction Tuning

LLaVA connects the output of a CLIP vision encoder to an LLM's token space using a linear or MLP projector. After feature alignment on image–caption pairs, it performs instruction tuning with GPT-generated multimodal instruction data and VQA data.

The goal of representation learning shifts beyond retrieval and captioning toward instruction-following representations that select visual information appropriate to a user's query and answer in language. A strong language prior from the LLM can nevertheless override image evidence and cause hallucination, while spatial details or OCR information may be lost in the vision encoder and projector.

The following table summarizes the progression.

Interview answer: After CLIP, research progressed toward adding the fine-grained interaction and generation that global image–text alignment alone lacks. ALBEF first aligns the modalities and then fuses them with cross-attention. BLIP and CoCa combine contrastive, matching, and captioning objectives to support both understanding and generation. Flamingo inserts a Perceiver and gated cross-attention between a frozen vision encoder and LM to enable visual in-context learning, while BLIP-2 efficiently connects two frozen models with Q-Former. After visual feature alignment, LLaVA performs instruction tuning, extending representations from a retrieval space into one for dialogue and problem solving.

8. Types of Positional Encoding and Their Advantages and Disadvantages

Self-attention is permutation-equivariant: if the input tokens are permuted, the outputs are permuted in the same way. Without separate positional information, it has no basis for understanding the order difference between “Dog bites man” and “Man bites dog.”

Absolute Positional Encoding

Assign a position vector $p_i$ to each position $i$ and add it to the token embedding.

$$h_i=x_i+p_i.$$

A learned absolute embedding trains a parameter vector for each position. It is the simplest to implement and expressive within the training length, but positions beyond that length either have no embedding or are out of distribution.

Sinusoidal encoding computes sine and cosine values based on position and dimension.

$$PE(pos,2i) = \sin\left(pos/10000^{2i/d}\right),$$ $$PE(pos,2i+1) = \cos\left(pos/10000^{2i/d}\right).$$

It introduces no additional parameters and can compute values for arbitrary positions, but the ability to calculate longer positions does not guarantee good zero-shot generalization at those lengths. Content and positional information are also mixed by addition at the input stage.

Relative Positional Encoding and Relative Bias

The attention score incorporates the relative distance $i-j$ rather than only the absolute positions of $i$ and $j$.

$$s_{ij} = \frac{q_i^\top k_j}{\sqrt{d_h}} +b_{i-j}.$$

Bucketing distances, as in T5's relative position bias, limits the number of parameters required for large distances. Because token-to-token distances remain the same when an entire sentence shifts, relationships generalize more easily. On the other hand, pairwise biases or relative embeddings must be processed, and implementation and cache management can become more complex.

Instead of a learned table, ALiBi applies a linear distance penalty with a head-specific slope.

$$s_{ij} = q_i^\top k_j-m_h|i-j|.$$

It adds almost no embedding overhead and makes length extrapolation relatively simple, but constraining positional relationships to a linear bias can limit expressiveness.

Rotary Positional Embedding

Instead of adding a position vector, RoPE rotates two-dimensional subspaces of queries and keys by angles proportional to position.

$$\tilde q_i=R_iq_i, \qquad \tilde k_j=R_jk_j.$$

The resulting dot product depends on relative position as follows.

$$\tilde q_i^\top\tilde k_j = q_i^\top R_i^\top R_jk_j = q_i^\top R_{j-i}k_j.$$

Each token is rotated according to its absolute position, yet the relative distance $j-i$ emerges naturally in the attention score. RoPE needs no additional position table and works conveniently with a KV cache, so it is used in many decoder-only LLMs, including LLaMA, Qwen, and Gemma.

It also has limitations. Far beyond the training length, the frequency and distribution of rotation angles change, degrading performance. This motivated context-extension techniques such as position interpolation, NTK-aware scaling, and YaRN. Using RoPE does not automatically solve long-context extrapolation.

More detailed equations are available in Positional Embeddings Derived from the Logic of Attention.

Interview answer: Absolute encoding adds a position-specific vector to the token embedding. It is simple, but weak at extrapolating beyond the training length. Relative encoding adds a distance bias based on $i-j$ to the attention score, generalizing better to positional shifts but requiring pairwise processing. RoPE rotates queries and keys by position-specific angles and uses the property $R_i^TR_j=R_{j-i}$ to encode relative position in their dot product. It needs no additional table and works well with KV caches, so it is widely used in modern LLMs. Far beyond the training length, however, its frequency distribution changes, requiring position interpolation or RoPE scaling.

9. Efficiently Batching Token Sequences of Different Lengths

Let the sequence lengths in a batch be $L_1,\ldots,L_B$, with maximum length $L_{\max}$. A dense batch has shape $B\times L_{\max}\times d$, and the number of padding tokens is

$$N_{\mathrm{pad}} = BL_{\max}-\sum_{i=1}^{B}L_i.$$

If padding is correctly excluded with the attention mask and label ignore index, the main problem is not training noise but wasted computation and activation memory.

Dynamic Padding and Length Bucketing

Pad only to the longest sequence in the current batch rather than the maximum length of the whole dataset. Grouping samples of similar lengths into buckets further reduces the chance that one long outlier lengthens the entire batch. This is the default choice for ordinary fine-tuning because it is easy to implement and broadly compatible.

Slightly padding dimensions to multiples of 8 or 16 can yield better practical throughput because Tensor Cores prefer those shapes. Minimizing the number of padding tokens is not always the same as minimizing wall-clock time.

Token-Budget Batching

Limit the total number of tokens in a batch instead of the number of samples.

$$\sum_{i\in\mathcal{B}}L_i \le T_{\mathrm{budget}}.$$

Include more short samples and fewer long samples to stabilize memory use. Attention compute is proportional to $\sum_iL_i^2$, however, so equal token counts do not necessarily imply equal step times. Loss normalization and gradient accumulation should also be based on the number of valid tokens.

Sequence Packing

Fill one fixed-length row with several short samples to eliminate padding.

[ A A A A A | B B B | C C C C ]

A block-diagonal mask or sequence boundaries must prevent attention from crossing between independent samples.

$$M_{ij} = \begin{cases} 0 & i,j\text{ belong to the same sample} \\ -\infty & \text{otherwise} \end{cases}$$

You must also decide whether position IDs restart for every sample and how to handle EOS and the first token's label. Pre-training commonly concatenates documents with EOS separators and then splits them into fixed-length chunks. Applying this to independent SFT samples without boundary masks can cause cross-contamination.

Bin-packing algorithms such as Best-Fit Decreasing place longer samples first into available space to improve utilization. Greedy or First-Fit is simpler in a streaming environment.

Padding-free Variable-length Attention

Flatten only the actual tokens into one tensor.

$$X_{\mathrm{flat}} \in \mathbb{R}^{(\sum_iL_i)\times d}.$$

Then pass boundary information such as $\mathrm{cu\_seqlens}=[0,L_1,L_1+L_2,\ldots]$ to a variable-length FlashAttention kernel. The kernel computes only actual tokens without crossing boundaries. Merely enabling FlashAttention does not eliminate padding; the varlen interface must actually be used.

Nested tensors and ragged tensors preserve variable length at the tensor-representation level. Operator and model support may be limited, and converting back to a dense tensor midway reintroduces padding costs.

A practical progression looks like this.

$$\text{Global padding} \rightarrow \text{Dynamic padding} \rightarrow \text{Bucketing/Token budget} \rightarrow \text{Packing} \rightarrow \text{Padding-free varlen kernel}.$$

Detailed implementation and trade-offs are covered separately in Efficiently Constructing Variable-Length Batches for Transformers.

Interview answer: The most basic approach is to dynamically pad only to the longest sequence in the batch and bucket samples of similar lengths. When the length distribution is broad, build batches by total token count rather than sample count. For higher efficiency, pack several samples into one row while carefully handling block-diagonal masks, position IDs, EOS tokens, and label boundaries. The most efficient execution is padding-free: flatten the input to retain only actual tokens and pass $\mathrm{cu\_seqlens}$ to variable-length FlashAttention. Dynamic padding and length bucketing are a safe baseline for ordinary fine-tuning, while large-scale LLM training commonly combines packing with varlen kernels.

10. A Recent Paper I Read: EAGLE-3

One recent paper I can discuss is EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. I read it because I was interested in reducing the sequential bottleneck of autoregressive decoding in LLM serving without degrading model quality. A KV cache avoids recomputing keys and values for previous tokens, but the enormous target model must still be run again for every new token.

In speculative decoding, a small draft model first proposes several future tokens, and the large target model verifies several positions in a single forward pass. Rejection sampling and correction against the target distribution provide lossless acceleration without changing the final output distribution.

Earlier EAGLE supplied the target model's top-layer features to a small draft model and autoregressively predicted the next features. EAGLE-3's key insight is to redefine the problem: what we actually want is not the target feature itself, but tokens that the target will accept.

First, it removes the feature-regression constraint and predicts tokens directly. Multiple hidden vectors can produce the same token distribution, so forcing the draft model to match one exact target-feature coordinate can unnecessarily restrict its expressiveness.

Second, removing the feature loss increases distribution shift: during training the model sees only ground-truth features, but during inference it receives its own outputs. EAGLE-3 addresses this with a training-time test that exposes the model during training to multi-step rollouts in which draft outputs are fed back as inputs. Despite its name, this does not perform gradient updates at test time.

Third, instead of using only the target's top layer, it fuses low-, middle-, and high-level features. Because the top layer may be strongly specialized for the immediately next token, combining features from multiple depths broadens the information useful for drafting future tokens.

The paper's contributions can be summarized as follows.

• It changes the objective from feature prediction to direct token prediction.
• It reduces exposure bias in multi-step drafting with a training-time test.
• It strengthens draft inputs through multi-layer feature fusion.
• It allows the draft model to benefit from data scaling.
• It reports up to a 6.5× speedup while preserving the target model's output distribution, and validates gains in batched serving-framework settings.

The approach still incurs draft-training cost and is target-specific, and the same speedup is not guaranteed across acceptance rates, batch sizes, kernels, and hardware. What I found most interesting was the modeling decision to discard a constraint unrelated to the final objective, rather than refine the existing feature-prediction task. I cover the paper in detail in How Does EAGLE-3's Speculative Decoding Speed Up LLMs Without Degrading Quality?.

Interview answer: I recently read EAGLE-3, a paper that addresses the memory-bound, sequential nature of token-by-token LLM decoding with speculative decoding while preserving the target model's output distribution. Earlier EAGLE predicted the next hidden feature, but EAGLE-3 focuses on the actual objective—token acceptance—by removing feature regression and predicting tokens directly. It addresses the resulting multi-step distribution shift with a training-time test that feeds the model's own outputs back into training, and it fuses features from several target-model layers. I was most impressed by how removing an unnecessary intermediate constraint improved draft accuracy and data scaling.

11. The Role of the FFN in a Transformer and Why It Is Necessary

Because Transformers already have self-attention, the FFN can look like a mere auxiliary layer, but the two operations act along different axes. Attention mixes information between tokens along the sequence dimension. The FFN, by contrast, applies the same function independently at every token position to transform hidden channels.

$$\operatorname{FFN}(X)_{b,t,:} = W_2\phi(W_1X_{b,t,:}+b_1)+b_2.$$

The first linear layer expands $d_{model}$ into a larger $d_{ff}$. The original Transformer used an expansion of roughly four times, while gated FFNs in modern LLMs use different ratios to match parameter and FLOP budgets. A nonlinear function such as GELU or SiLU is applied in the expanded space before projecting back to $d_{model}$.

The first reason the FFN is needed is nonlinearity. Softmax makes attention weights nonlinear, but value aggregation itself is a weighted sum. Conditioned on the context gathered by attention, the FFN recombines features and enables complex function approximation.

The second reason is channel mixing. Attention determines “which tokens to retrieve information from,” while the FFN determines “which features of the current token that retrieved information should become.” This resembles the distinction between spatial mixing and channel mixing in CNNs.

The third reason is model capacity. A large fraction of an LLM's parameters resides in its FFNs. Some research interprets an FFN as a key–value memory: an input pattern activates particular intermediate neurons, and the output projection writes related features into the residual stream. We should not conclude that knowledge is stored only in FFNs, however; it is distributed across attention, embeddings, and the full stack of layers.

Modern LLMs commonly use gated activations.

$$\operatorname{SwiGLU}(x) = W_o \left( \operatorname{SiLU}(W_gx) \odot W_ux \right).$$

The gate selects which features to pass based on the input, potentially offering greater expressiveness than a conventional two-layer MLP. MoE replaces this dense FFN with several expert FFNs and selects only a subset for each token.

Interview answer: Attention and FFNs process different axes. Attention mixes information between tokens to gather context, while the FFN is independently applied to each token to nonlinearly transform its hidden channels. It generally expands the dimension, applies GELU or SwiGLU, and projects it back down, providing channel mixing and strong function-approximation capacity. Because a large fraction of an LLM's parameters is also in its FFNs, they matter greatly for model capacity and knowledge representation. Repeating attention alone can aggregate context as weighted sums, but lacks sufficient nonlinear processing of the collected information.

12. Mixture of Experts: Architecture, Operation, and Its Effect Compared with Dense Models

In a dense Transformer, every token passes through every FFN parameter. A sparse Mixture of Experts replaces a Transformer block's dense FFN sublayer with several expert FFNs and a router. Replicating the entire attention module into experts is not the usual baseline architecture.

For a token representation $x$, the router computes expert scores.

$$g(x)=\operatorname{softmax}(W_rx).$$

If only the top-$k$ experts are selected, the output is

$$y = \sum_{e\in\operatorname{TopK}(g(x))} g_e(x)E_e(x).$$

where each $E_e$ is an independent FFN. Switch Transformer uses top-1 routing, while Mixtral sends every token to two of eight experts.

The process is as follows.

1. The router computes expert probabilities for every token in the batch.
2. It selects the top-$k$ expert indices and weights.
3. All-to-all communication dispatches tokens to the GPUs hosting the selected experts.
4. Each expert processes its incoming tokens with its FFN.
5. The results are returned to their original GPUs and token order, then combined using the routing weights.

MoE's key effect is to decouple total parameter count from the number of active parameters per token. Increasing the number of experts can greatly expand model capacity, while each token activates only $k$ experts instead of every parameter as in a dense model. This enables greater capacity under the same FLOP budget, and expert specialization may emerge by language, domain, or token pattern.

But “more parameters at no compute cost” is an oversimplification.

• Every expert's weights must be stored in memory or distributed across GPUs.
• Dispatching and combining tokens requires all-to-all communication.
• If tokens concentrate on one expert, other experts sit idle while the overloaded expert exceeds capacity, causing load imbalance.
• Expert collapse can occur when the router selects only a few experts and rarely uses the rest.
• With small batches, too few tokens per expert can make GEMMs inefficient.

Common mitigations include an auxiliary load-balancing loss that balances the token fraction and mean routing probability across experts, an expert-capacity factor, token dropping or dropless routing, and router z-loss.

$$\mathcal{L}_{\mathrm{balance}} \propto E\sum_{e=1}^{E}f_ep_e,$$

Here, $f_e$ is the fraction of tokens actually routed to expert $e$, and $p_e$ is its mean router probability.

Interview answer: A sparse MoE generally replaces a Transformer's dense FFN with several expert FFNs and a router. For every token, the router selects the top-$k$ experts and combines their outputs as a weighted sum. This allows the total parameter count to grow substantially while limiting active parameters and FLOPs per token, increasing model capacity at the same compute. The trade-offs are storing every expert's weights, all-to-all communication in distributed settings, load imbalance, expert collapse, and poor efficiency from small per-expert batches. Load-balancing losses and capacity management are therefore important.

13. What RLVR Is and How It Differs from Conventional RLHF

RLVR (Reinforcement Learning with Verifiable Rewards) trains a policy with rewards for which correctness can be automatically verified, instead of having humans evaluate every model output or scoring it with a learned reward model. Tulu 3 explicitly named it as a stage in its post-training recipe, and the approach has become widely used in mathematical and code reasoning.

Examples include the following.

• Mathematics: A symbolic or rule-based checker determines whether the final answer matches the ground truth.
• Code: The generated program is executed to see whether it passes unit tests.
• Formal proofs: A proof assistant determines whether to accept the proof.
• Structured output: The output is checked against a JSON schema, compiler, or constraint validator.

Given a prompt $x$, an answer $y$ is sampled and a verifier $V$ returns a reward.

$$y\sim\pi_\theta(\cdot\mid x), \qquad r(x,y)=V(x,y).$$

An algorithm from the PPO, GRPO, or REINFORCE family then maximizes expected reward.

$$\max_\theta \mathbb{E}_{y\sim\pi_\theta}[r(x,y)].$$

The greatest difference from RLHF is the source of reward.

RLVR's advantages are objective rewards, cheap evaluation of repeated samples, and reduced proxy error from a Reward Model. It can also let a model explore multiple solution paths instead of forcing it to imitate a single human reasoning trace. DeepSeek-R1 showed that RL incorporating rule-based accuracy and format rewards can strengthen reasoning behavior.

Its limitation is that only what is verifiable is easy to optimize. Creativity, politeness, and nuanced factual quality are difficult for a single checker to judge. If an outcome reward appears only at the end, credit assignment to individual reasoning steps is hard. Reward hacking can also exploit answer formatting or test loopholes. If the verifier is imperfect, the assumption of an exact reward breaks down.

In a practical pipeline, RLHF and RLVR need not be mutually exclusive. Rule-based rewards can be used for verifiable tasks, while human- or model-based preference rewards cover general conversation and safety.

Interview answer: RLVR reinforces a language model using automatically verifiable rewards such as mathematical answers, unit tests, and proof checkers. RLHF approximates subjective human preferences with a Reward Model trained on comparison data. RLVR directly judges whether an outcome is correct, reducing Reward Model error and annotation cost while enabling large-scale sampling. It is mostly limited, however, to tasks such as mathematics and code that have verifiers, and it faces credit assignment for terminal rewards and verifier hacking. In practice, RLHF-style methods can cover general conversational quality while RLVR handles reasoning with verifiable answers.

14. How Distributed Data Parallel Works, Its Limitations, and Improvements

Distributed Data Parallel places an identical model replica on every GPU and processes a different mini-batch shard on each. Let the world size be $P$.

1. Broadcast the initial model parameters identically to every rank.
2. A distributed sampler divides the global batch among ranks.
3. Each rank independently performs the forward and backward passes to compute its local gradient $g_r$.
4. As soon as a gradient bucket is ready, perform ring All-Reduce.
5. Every rank receives the mean gradient and applies the same optimizer update.

$$g = \frac{1}{P} \sum_{r=1}^{P}g_r.$$

Because every rank has the same initial parameters and mean gradients, its model replica remains identical after the update. PyTorch DDP groups parameter gradients into buckets and asynchronously All-Reduces each bucket as it becomes ready during the backward pass, overlapping computation with communication.

DDP is simple to implement and scales throughput relatively well with GPU count if one copy of the model fits on a GPU. It nevertheless has clear limitations.

Full Replication of Model State

Every GPU holds all parameters, gradients, and optimizer states. Adding GPUs does not reduce per-device memory for model state. Adam training with FP32 states requires roughly 16 bytes per parameter, and activations add further memory, so large models cannot fit on one GPU.

Gradient-Synchronization Cost

Every backward pass All-Reduces gradients proportional to model size. Communication becomes a bottleneck as GPU count grows or when inter-node networking is slow. Poor bucket sizing or ordering also reduces computation–communication overlap.

Global Batch and Stragglers

With the same per-device batch, adding GPUs increases the global batch. The learning rate and optimization behavior must be retuned, and scaling cannot continue indefinitely. Collectives also cannot progress beyond the slowest rank, so long sequences or variation in data-loading time create stragglers.

Remedies and Directions for Scaling

• Gradient accumulation reduces communication frequency and creates a larger effective batch, but activation memory and step latency must be considered.
• Mixed precision and gradient compression reduce communication volume.
• ZeRO Stages 1–3 and FSDP progressively eliminate replication of optimizer states, gradients, and parameters.
• Tensor parallelism divides a layer's matrices across GPUs, addressing models that cannot fit on one GPU.
• Pipeline parallelism divides layers into stages and executes a micro-batch pipeline.
• Sequence or context parallelism divides the activations and attention for long sequences.
• 3D parallelism combines data, tensor, and pipeline parallelism.

DDP is therefore the first choice when the model fits on one GPU but the dataset and training workload are large. When model states or activations exceed one GPU, sharding or model parallelism is required.

Interview answer: DDP replicates the same model on every GPU, performs forward and backward passes on different data shards, All-Reduces and averages the gradients, and applies the same optimizer update. Bucketing gradients lets it overlap backward computation with communication, making it an efficient throughput-scaling method when the model fits on one GPU. Its limitations are that every GPU fully replicates parameters, gradients, and optimizer states, so per-device model memory does not decrease, and that every step incurs gradient All-Reduce and straggler effects. ZeRO/FSDP address this by sharding model states, while tensor, pipeline, and context parallelism divide model and activation computation.

15. Principles and Differences of FSDP and ZeRO Stages 1–3

In ordinary data-parallel training, three main types of model state are duplicated on every GPU.

• Parameters $P$
• Gradients $G$
• Optimizer states $O$: Adam's first and second moments and, depending on the setup, FP32 master weights

ZeRO progressively eliminates this redundancy across data-parallel ranks.

ZeRO Stage 1

Only optimizer states are sharded across ranks. Each rank holds all parameters and gradients, but keeps optimizer states only for its own partition and updates the corresponding parameter shard. The updated parameter shards are All-Gathered so every rank once again has the full parameters.

$$\text{Per-rank state} \approx P+G+\frac{O}{P_{world}}.$$

ZeRO Stage 2

Optimizer states and gradients are sharded. Instead of gradient All-Reduce, Reduce-Scatter leaves each rank with only the gradient shard corresponding to the parameters it will update.

$$\text{Per-rank state} \approx P+\frac{G+O}{P_{world}}.$$

ZeRO Stage 3

Optimizer states, gradients, and parameters are all sharded. Immediately before computing a layer, the required parameter shards are All-Gathered; the full parameters are released once computation completes. Gradients are Reduce-Scattered after the backward pass.

$$\text{Per-rank state} \approx \frac{P+G+O}{P_{world}},$$

Actual memory usage is higher because of the full parameters for the layer currently being computed, communication buckets, activations, and fragmentation. Higher stages save more memory, but additional parameter All-Gathers increase communication and implementation complexity.

FSDP

PyTorch FSDP implements full sharding like ZeRO-3 at the PyTorch module level. It All-Gathers parameters immediately before an FSDP unit's forward pass and reshards them afterward. During the backward pass it also gathers required parameters and Reduce-Scatters gradients. Nested wrapping or an auto-wrap policy can limit the lifetime of full parameters to one Transformer block at a time.

Unlike FSDP1, which hides parameters behind a flat-parameter abstraction, PyTorch FSDP2 provides per-parameter DTensor-based sharding and improved composability. For an interview, the key point is not the API generation, but that FSDP shards parameters, gradients, and optimizer states, gathering parameters at computation time.

Summary of Differences

ZeRO refers to DeepSpeed's concept and implementation for progressively eliminating optimizer/model-state redundancy, while FSDP is the fully sharded implementation in the PyTorch ecosystem. ZeRO-3 and FSDP have very similar algorithmic goals, but differ in wrapping units, parameter representation, prefetching, offload, checkpoint APIs, and runtime integration.

Interview answer: ZeRO progressively shards model states that are duplicated under data parallelism. Stage 1 shards optimizer states; Stage 2 shards optimizer states and gradients; and Stage 3 also shards parameters. In Stage 3, parameters are All-Gathered immediately before a layer's computation, while backward gradients are Reduce-Scattered. FSDP is algorithmically similar to ZeRO-3 in sharding parameters, gradients, and optimizer states, but is integrated with PyTorch modules and DTensor. Higher stages reduce memory further, at the cost of more parameter-gather communication and runtime complexity.

16. Optimizer Types, How Adam Works, and VRAM by Parameter Count

Representative optimizers include the following.

• SGD updates in the direction of the current gradient. It is simple and generalizes well, but is sensitive to learning rate and scale.
• Momentum uses a moving average of gradients to accelerate consistent directions and reduce oscillation.
• AdaGrad adjusts the learning rate using per-parameter accumulated squared gradients, but the rate can keep shrinking.
• RMSProp uses an exponential moving average of squared gradients to mitigate AdaGrad's excessive decay.
• Adam combines a first moment analogous to Momentum with a second moment analogous to RMSProp.
• AdamW applies weight decay directly and separately to the parameter update instead of mixing it into the gradient.
• Adafactor factorizes the second moment to reduce optimizer memory for large matrices.
• LAMB uses layer-wise scaling to support very large-batch training.
• Lion uses sign-based updates and a single momentum state, potentially reducing optimizer-state memory compared with Adam.

For the gradient $g_t$ at step $t$, Adam computes a first moment and a second raw moment.

$$m_t = \beta_1m_{t-1}+(1-\beta_1)g_t,$$ $$v_t = \beta_2v_{t-1}+(1-\beta_2)g_t^2.$$

Because the initial values $m_0=v_0=0$ bias early estimates toward zero, Adam applies bias correction.

$$\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t}.$$ $$\theta_{t+1} = \theta_t - \eta \frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}.$$

The first moment smooths the gradient direction, while the second moment acts as a parameter-wise adaptive learning rate that reduces the step for parameters with recently large gradient scales.

Adam Memory with $N$ Parameters

An accurate answer must begin by stating precision and implementation assumptions. Let us calculate only model state, excluding activations, temporary buffers, the CUDA context, and memory fragmentation.

FP32 training requires the following.

$$M_{\mathrm{Adam,FP32}} \approx 16N\ \text{Bytes}.$$

For example, if $N=7\times10^9$, this is 112 GB in decimal units or approximately 104.3 GiB in binary units. Activations and communication buffers are excluded, so actual required VRAM is greater.

Traditional FP16 mixed precision also uses 16 bytes per parameter when it stores 2-byte FP16 parameters, 2-byte FP16 gradients, a 4-byte FP32 master parameter, and 8 bytes for FP32 $m$ and $v$.

$$2N+2N+4N+4N+4N = 16N\ \text{Bytes}.$$

An implementation using BF16 parameters and gradients without a separate FP32 master weight may require $2+2+4+4=12$ bytes per parameter. Conversely, retaining an FP32 gradient buffer or master weight raises the total to 16 bytes or more. Rather than stating that “Adam always uses a fixed number of bytes,” the precise answer is that the FP32 baseline is $16N$ bytes, while mixed precision may require $12N$ to $16N$ bytes or more depending on master weights and gradient dtype.

ZeRO/FSDP shards these model states across the data-parallel world size, while 8-bit optimizers reduce memory further by storing $m$ and $v$ at low precision.

Interview answer: Adam estimates the gradient's first moment $m_t$ and the squared gradient's second moment $v_t$ with exponential moving averages, then adjusts each parameter's step using $\hat m_t/(\sqrt{\hat v_t}+\epsilon)$ after bias correction. In FP32, parameters, gradients, first moments, and second moments each use 4 bytes, so $N$ parameters require approximately $16N$ bytes for model state alone. Traditional FP16 mixed precision also totals $16N$ bytes: 4 bytes for FP16 weights and gradients, 4 bytes for FP32 master weights, and 8 bytes for Adam states. A BF16 implementation without master weights may use $12N$ bytes, however, so the assumptions must be stated first; activations and buffers are separate.

17. Why KV Caches Are Used and How to Manage and Optimize Them Efficiently

When a decoder-only LLM generates the next token, it feeds in all tokens generated so far. Without a cache, the keys and values for previous tokens would be recomputed at every step.

If the keys and values of past tokens are stored at layer $l$, then only $q_t,k_t,v_t$ need to be projected for the new token $x_t$, after which $k_t,v_t$ are appended to the existing cache.

$$K_{1:t}^{(l)} = [K_{1:t-1}^{(l)};k_t^{(l)}], \qquad V_{1:t}^{(l)} = [V_{1:t-1}^{(l)};v_t^{(l)}].$$ $$o_t = \operatorname{softmax} \left( \frac{q_tK_{1:t}^\top}{\sqrt{d_h}} \right)V_{1:t}.$$

This avoids repeating projections and earlier layer computations for past tokens. Because attention reads the entire past cache, attention cost per token still grows with context length, but the much larger redundancy of forwarding the entire prefix at every step is eliminated.

KV-cache memory is approximately

$$M_{KV} = 2 \times L_{layers} \times L_{seq} \times H_{kv} \times d_{head} \times b_{dtype} \times B.$$

where the factor 2 accounts for K and V, and $B$ is the number of concurrent sequences. With long contexts and large batches, the KV cache can become the largest memory consumer after model weights.

Representative Optimizations

MQA and GQA: MHA has one K/V head for every query head. MQA shares one K/V head across all query heads, while GQA shares one K/V head per group of query heads. Both substantially reduce cache size and decode bandwidth; GQA strikes a balance between MHA's quality and MQA's efficiency.

PagedAttention: Preallocating a contiguous maximum-length buffer for every request causes internal fragmentation and errors in estimating sequence length. PagedAttention divides the KV cache into fixed-size blocks and maps logical blocks to noncontiguous physical blocks. It allocates only the blocks that are needed, while beams or shared prefixes can share blocks with copy-on-write, increasing the serving batch size.

Continuous batching: Instead of retaining a fixed batch until every request completes, fill completed slots with new requests. Combined with a KV-block allocator, this raises GPU utilization.

KV-cache quantization: Store K and V in FP8, INT8, or even fewer bits. This reduces memory and bandwidth, but quality may degrade unless scales and outliers are handled carefully per layer, head, or token.

Sliding windows and eviction: A local-attention model caches only the most recent $W$ tokens. Methods such as H2O retain high-importance “heavy hitter” tokens and evict less important cache entries. This saves memory, but is not exactly equivalent to full-context attention.

Prefix caching: Compute and reuse the KV cache for prefixes shared by many requests, such as system prompts or document prefixes. A higher prompt-cache hit rate reduces time to first token and prefill cost. Cache invalidation and separation by model or adapter are required.

Offloading and distributed KV: When GPU memory is insufficient, move part of the KV cache to the CPU, host memory, or another GPU. This increases capacity but may add latency due to PCIe and network bandwidth.

Interview answer: A KV cache stores past tokens' keys and values at every layer during autoregressive decoding, avoiding repeated projections and forward computation over the previous prefix at every step. The cache grows with layer count, sequence length, number of KV heads, and batch size, and must be read for every token, so it becomes a memory-capacity and bandwidth bottleneck. MQA/GQA reduce cache size by sharing K/V heads, while PagedAttention allocates KV in blocks to improve fragmentation and sharing. Other techniques include KV quantization, prefix caching, continuous batching, sliding-window attention or eviction, and CPU offload.

18. Why Applying Reinforcement Learning to Diffusion Models Is Difficult

From an RL perspective, a diffusion model's reverse process has the current noisy latent $x_t$ as its state, the next latent $x_{t-1}$ as its action, and the denoising transition $p_\theta(x_{t-1}\mid x_t,c)$ as its policy. A reward is received for the final image $x_0$.

$$x_T\rightarrow x_{T-1}\rightarrow\cdots\rightarrow x_0, \qquad r=r(x_0,c).$$

This formulation is possible, but several difficulties are greater than in language models.

Long, Expensive Trajectories and Sparse Rewards

Generating one sample requires dozens of U-Net or DiT forward passes. Because the reward is usually obtained only from the final image, assigning credit to the denoising steps and spatial decisions that contributed to the outcome is difficult. Sampling multiple trajectories for the same prompt is also extremely expensive.

High-Dimensional Continuous Actions

An LM's action is a discrete token from its vocabulary, while a diffusion model's action is a high-dimensional continuous variable close to an entire image latent. Even when transition log probabilities and importance ratios can be calculated, variance is high and small policy changes accumulate across the full image trajectory in complex ways. Deterministic samplers such as DDIM require additional design to be treated as stochastic policies.

Backpropagation Memory and Policy-Gradient Variance

If the reward is differentiable, it can be backpropagated through the full sampling chain, but storing activations for every denoising step requires enormous memory. Recomputation and truncated backpropagation add cost or bias. If the reward is not differentiable, policy gradients must be used, leading to high gradient variance and sample inefficiency from the terminal reward.

Imperfect Rewards and Diversity Collapse

Aesthetic scores, CLIP similarity, and human-preference models are proxies for the quality people actually want. A model may exploit Reward Model shortcuts such as watermarks, oversaturation, or particular compositions. Overoptimizing one reward can reduce image diversity and the model's original generative ability. Defining and efficiently computing KL regularization against a base diffusion policy is also less straightforward than for LLMs.

Representative Approaches

DDPO treats denoising as a multi-step MDP and optimizes the log probabilities of all denoising transitions using policy-gradient and PPO-style methods. DPOK uses online RL with KL regularization. AlignProp/DRaFT-style methods backpropagate differentiable rewards directly through the sampling chain, while Diffusion-DPO methods directly optimize preference pairs.

The issue is therefore not that RL is impossible to apply, but that sample generation itself is long and expensive, while a final reward is sparsely assigned to a high-dimensional denoising trajectory.

Interview answer: A diffusion model's reverse process can be formulated as an MDP in which each denoising step is an action. But one image requires dozens of forward passes, and rewards are usually available only for the final image, making credit assignment difficult. Actions are high-dimensional continuous latents, so policy-gradient variance is high, while differentiating the entire sampling chain requires enormous activation memory. Optimizing loopholes in aesthetic or CLIP rewards can also reduce diversity or cause reward hacking. DDPO applies policy gradients to denoising-transition log probabilities, while AlignProp-style methods directly backpropagate a differentiable reward through the sampling chain.

19. Differences Among BatchNorm, LayerNorm, and GroupNorm, and Their Normalization Dimensions

Let the input image features be $X\in\mathbb{R}^{N\times C\times H\times W}$. The essential difference among the three methods is the axes over which their means and variances are computed.

Normalization generally has the following form.

$$\hat x = \frac{x-\mu}{\sqrt{\sigma^2+\epsilon}}, \qquad y=\gamma\hat x+\beta.$$

Batch Normalization

For each channel $c$, BatchNorm uses the batch and spatial dimensions $N,H,W$.

$$\mu_c = \frac{1}{NHW} \sum_{n,h,w}X_{n,c,h,w},$$ $$\sigma_c^2 = \frac{1}{NHW} \sum_{n,h,w} (X_{n,c,h,w}-\mu_c)^2.$$

During training, it uses the current batch statistics and updates running means and variances. During inference, it uses the running statistics. Noise between batches acts as regularization and is efficient in CNNs, but the method is sensitive to small batches, shifts in batch distribution, sequence length, and distributed settings. On multiple GPUs, SyncBatchNorm synchronizes statistics across ranks at the cost of additional communication.

Layer Normalization

LayerNorm normalizes the full feature dimension at one token or one position in each sample. For a Transformer input $X\in\mathbb{R}^{N\times L\times D}$, it computes statistics over the $D$ axis for every $(n,l)$.

$$\mu_{n,l} = \frac{1}{D} \sum_{d=1}^{D}X_{n,l,d},$$ $$\sigma_{n,l}^2 = \frac{1}{D} \sum_{d=1}^{D} (X_{n,l,d}-\mu_{n,l})^2.$$

It does not depend on other samples in the batch, and computation is identical during training and inference, making it well suited to Transformers and autoregressive models. Computing one mean and variance over all features is not always optimal for every architecture, however. RMSNorm simplifies computation by normalizing only the scale with RMS, without subtracting the mean.

Group Normalization

GroupNorm divides the channels within each sample into $G$ groups and jointly normalizes the channels and spatial dimensions within each group. Let $\mathcal{C}_g$ be the channel set for group $g$.

$$\mu_{n,g} = \frac{1}{|\mathcal{C}_g|HW} \sum_{c\in\mathcal{C}_g,h,w} X_{n,c,h,w}.$$

Variance is computed over the same axes. Because GroupNorm does not depend on batch size, it is suited to vision models whose batches are small because of high resolution, such as detection, segmentation, and diffusion U-Nets. With $G=1$, it resembles LayerNorm over channels and space; with $G=C$, it resembles InstanceNorm over space for each channel. Exact equivalence depends on tensor layout and the definition of normalized shape.

Interview answer: With $NCHW$ layout, BatchNorm uses the mean and variance over $N,H,W$ independently for each channel, and training-time batch statistics differ from inference-time running statistics. LayerNorm normalizes the hidden dimension $D$ of each sample or token, independently of the batch, making it suitable for Transformers. GroupNorm divides one sample's channels into groups and normalizes each group's channel and spatial axes, remaining stable for small-batch vision models. Large-batch CNNs therefore commonly use BatchNorm, Transformers use LayerNorm, and small-batch detection or diffusion U-Nets use GroupNorm.

20. Causal Attention vs. Bidirectional Attention and the Models That Use Them

Suppose that a mask $M$ is added to the self-attention scores.

$$A = \operatorname{softmax} \left( \frac{QK^\top}{\sqrt{d_h}}+M \right).$$

Causal Attention

The upper-triangular region is masked with $-\infty$ so that position $i$ cannot attend to a future position $j>i$.

$$M_{ij} = \begin{cases} 0 & j\le i \\ -\infty & j>i. \end{cases}$$

The prediction of $x_i$ therefore depends only on $x_{<i}$ and satisfies autoregressive factorization.

$$p(x_{1:T}) = \prod_{t=1}^{T}p(x_t\mid x_{<t}).$$

This is used by decoder-only language models such as GPT, LLaMA, and Qwen. During training, next-token losses for all positions can be calculated in parallel. During inference, however, the preceding token must be decided before the next can be generated, so decoding is sequential.

Bidirectional Attention

Except for padding, every token can attend to all tokens on both its left and right. Representative examples include encoder-only models such as BERT and RoBERTa, as well as patch self-attention in ViT. It is effective for classification, token classification, retrieval embeddings, and image understanding when the full input is available.

If BERT could directly see a target token, the reconstruction task would be trivial. It therefore masks some tokens and predicts them from surrounding bidirectional context using Masked Language Modeling.

In an encoder–decoder Transformer, the encoder understands the full source with bidirectional self-attention, while the decoder generates the target with causal self-attention. The decoder's cross-attention allows the current generation position to attend to all encoder outputs.

The masking difference determines whether the future is used during training and which task factorization is available. Causal models are natural for generation, but a token representation does not directly include right-side context. Bidirectional models create strong full-input representations, but cannot naturally generate text from left to right without a separate autoregressive decoder.

Interview answer: Causal attention uses an upper-triangular mask to prevent position $i$ from seeing future tokens $j>i$, enabling autoregressive generation under $p(x_{1:T})=\prod_t p(x_t\mid x_{<t})$. It is used by decoder-only models such as GPT and LLaMA. Bidirectional attention lets every token see the full context on both sides, making it suitable for representation learning and understanding in encoders such as BERT and in ViT. In an encoder–decoder model, the encoder uses bidirectional attention, the decoder uses causal self-attention, and decoder cross-attention attends to the entire source.

21. Teacher Forcing: Definition, Advantages, and Exposure Bias

Teacher forcing trains an autoregressive model by using the previous ground-truth token as the next input, rather than the token generated by the model at the preceding step.

For a target sequence $y_{1:T}$, the training loss is

$$\mathcal{L}_{\mathrm{TF}} = -\sum_{t=1}^{T} \log p_\theta(y_t\mid x,y_{<t}).$$

Because the prefix $y_{<t}$ at every position is already known, predictions and losses for all positions can be computed in parallel in one forward pass with a causal mask. If model outputs were instead fed back as inputs, tokens would have to be sampled one at a time, making training extremely slow, while early random errors would continue contaminating subsequent inputs.

Teacher forcing has the following advantages.

• It provides clear, stable supervision under ground-truth context.
• Every position in the sequence can be trained in parallel.
• It prevents training collapse caused by repeatedly feeding an early model's incorrect tokens back as input.
• It directly optimizes the MLE next-token likelihood.

Training and inference conditions differ, however. During training, the model always sees correct prefixes; during inference, it uses its own predictions as the next inputs. One error can lead into a prefix never encountered during training, allowing errors to accumulate. This is called exposure bias.

$$y_{<t}\sim p_{data} \quad\text{during training},$$ $$\hat y_{<t}\sim p_\theta \quad\text{during inference}.$$

Another problem is the mismatch between token-level likelihood and sequence-level quality. Lower Cross-Entropy does not necessarily improve factuality, coherence, BLEU, or human preference for the complete response.

Scheduled sampling gradually replaces some ground-truth inputs with model predictions, but has issues with objective consistency and sampling bias. Sequence-level training, Professor Forcing, data augmentation, DAgger-style on-policy data collection, RLHF/RLVR, and online distillation instead add feedback on sequences the model actually visits.

Exposure bias does not mean that the usual solution is to abandon teacher forcing. It remains the efficient and stable default for large-language-model pre-training and SFT, while on-policy post-training can supplement sequence-level behavior.

Interview answer: Teacher forcing uses the ground-truth token as the next input during autoregressive training instead of the model's previous prediction. It computes next-token losses at every position in parallel and performs stable MLE without accumulating early errors. But training sees only correct prefixes, while inference feeds the model's own outputs back as inputs. The resulting distribution mismatch is exposure bias: one error can move generation into states never seen during training and cause further errors to accumulate. Scheduled sampling, online distillation, and sequence-level RL can mitigate it, but teacher forcing remains the standard method for pre-training and SFT.

22. How Text Conditioning Works in Stable Diffusion

Stable Diffusion is a latent diffusion model that denoises in a VAE latent space rather than in pixel space. The text prompt enters as a condition that determines which image the denoising U-Net should reconstruct.

1. Text Tokenization and Encoding

The prompt is tokenized with a CLIP tokenizer and passed to a text encoder. The text encoder creates a contextual embedding for every token.

$$C = \tau_{text}(prompt) \in \mathbb{R}^{L_{text}\times d_c}.$$

The text encoder provides a sequence representation that lets U-Net cross-attention refer to token-level conditions such as “red,” “cat,” and “on the table,” rather than producing only one vector for the whole sentence.

2. Latent Generation and Denoising

During training, noise is added to $z_0$, an image compressed by the VAE encoder, to produce $z_t$. The U-Net predicts the noise or another parameterization target. Inference begins with Gaussian noise $z_T$ and repeatedly removes noise according to a scheduler.

$$z_T\rightarrow z_{T-1}\rightarrow\cdots\rightarrow z_0.$$

3. Injecting Text Through Cross-Attention

Intermediate spatial features from the U-Net serve as queries, while text embeddings serve as keys and values.

$$Q=W_Qh_{image}, \qquad K=W_KC, \qquad V=W_VC.$$ $$\operatorname{CrossAttn}(h_{image},C) = \operatorname{softmax} \left( \frac{QK^\top}{\sqrt{d_h}} \right)V.$$

Each spatial latent position learns which text tokens to attend to. This is the core mechanism by which Latent Diffusion injects diverse conditions such as text and bounding boxes using cross-attention.

4. Classifier-Free Guidance

During training, the text condition is dropped with a fixed probability so one model learns both conditional and unconditional predictions. During inference, the difference between the two noise predictions is amplified.

$$\hat\epsilon = \epsilon_\theta(z_t,t,\varnothing) + w \left[ \epsilon_\theta(z_t,t,c) - \epsilon_\theta(z_t,t,\varnothing) \right].$$

A larger $w$ can strengthen prompt alignment, but an excessively large value may oversaturate colors and reduce diversity and naturalness. A negative prompt can be understood as placing embeddings for text to avoid into the unconditional branch instead of empty text.

Text Encoders Across Stable Diffusion Versions

• The Stable Diffusion v1 family uses a frozen OpenAI CLIP ViT-L/14 text encoder.
• The Stable Diffusion v2 family uses an OpenCLIP ViT-H/14 text encoder.
• SDXL uses two text encoders: CLIP ViT-L/14 and OpenCLIP ViT-bigG/14. Their token embeddings are combined to create a broader cross-attention context, while pooled text embeddings are also used alongside micro-conditioning such as image size and crop.

The text encoder transforms a prompt's linguistic meaning into representations the denoiser can use. Tokenizer length limits, conceptual biases in the text encoder, and binding failures in cross-attention mean that the model still does not perfectly follow quantities, spatial relations, or long prompts.

Interview answer: Stable Diffusion converts a prompt into token-level embeddings with a CLIP tokenizer and frozen text encoder. At every denoising step, text conditioning is injected through cross-attention, in which the U-Net's spatial latent features are queries and the text embeddings are keys and values. Classifier-Free Guidance scales the difference between conditional and unconditional noise predictions to strengthen prompt adherence. SD v1 uses CLIP ViT-L/14, v2 uses OpenCLIP ViT-H/14, and SDXL jointly uses the token and pooled embeddings from CLIP ViT-L and OpenCLIP ViT-bigG.

23. How Gradient Checkpointing Works and Its Trade-offs

Backpropagation stores intermediate activations from the forward pass to compute the chain rule. In a deep Transformer or with long sequences, activation memory can exceed parameter memory.

Write a general sequence of layers as

$$h_l=f_l(h_{l-1}).$$

Storing every $h_l$ makes it immediately available during the backward pass, but memory grows with the number of layers. Gradient checkpointing, or activation checkpointing, saves activations only at selected points and discards the rest. When the backward pass reaches a segment, it reruns the forward computation from a saved checkpoint to reconstruct the required activations.

For example, if only outputs 0, 4, 8, and 12 are saved in a 12-block model, the backward pass recomputes the 8–12, 4–8, and 0–4 segments in order. This trades computation for memory.

In theory, dividing $n$ layers into segments of length $\sqrt{n}$ gives a classical configuration that reduces stored activations from $O(n)$ to approximately $O(\sqrt n)$. Actual memory depends on attention tensors, segment boundaries, and framework implementation.

The advantage is a large reduction in activation memory without approximating model parameters or training results, enabling longer sequences, larger batches, and larger models. The disadvantage is increased training time from recomputing part of the forward pass, along with the need to choose recomputation segments and manage RNG state.

When random operations such as dropout are present, recomputation must reproduce the same masks so gradients match the original forward pass. Frameworks preserve RNG state to handle this, at additional cost. Stateful layers or functions with external side effects require care inside checkpointed regions.

FlashAttention uses the same memory–compute trade-off by recomputing some values rather than storing a large attention matrix for the backward pass. Gradient checkpointing is the more general technique at the level of arbitrary module segments.

Interview answer: Gradient checkpointing stores only selected boundaries from the forward pass and recomputes the required forward segments during the backward pass instead of retaining every activation. It substantially reduces activation memory and supports longer sequences or larger batches, but recomputation increases training time. It does not reduce model-state memory, so Adam states still require techniques such as ZeRO/FSDP. With dropout, recomputation must restore the same random state, and the selected checkpoint boundaries determine the memory–compute trade-off.

24. How Speculative Decoding Works and Why It Is Faster

An autoregressive LLM cannot compute the token after next until the next token has been decided. Generating length $T$ therefore calls the large target model sequentially at least $T$ times. Speculative decoding lets a small draft model $q$ propose several tokens in advance, then has target model $p$ verify them at once, increasing the number of committed tokens per target call.

1. Draft

The draft model quickly generates $k$ tokens following the current prefix.

$$\hat x_{t+1:t+k}\sim q.$$

2. Verification

The full prefix and all draft tokens are passed to the target model. A causal Transformer can compute the next-token distributions at all draft positions in parallel in one forward pass.

3. Accept and Correct

Draft proposals are checked from the first token onward. Under sampling, the acceptance probability for a token $x$ proposed by the draft model is

$$A(x) = \min \left( 1, \frac{p(x)}{q(x)} \right).$$

If it is rejected, a corrected sample is drawn from the remaining probability mass.

$$p_{residual}(x) \propto \max(0,p(x)-q(x)).$$

If this process is applied exactly, the final sample distribution is identical to target $p$. With greedy decoding, proposals are accepted while the draft and target argmax tokens match, and the target token is used at the first mismatch.

Why It Is Faster

Token-by-token decoding repeatedly performs small matrix multiplications and reads enormous weights from HBM for every token, so it is often memory-bandwidth-bound at small batch sizes. Verification groups several token positions into larger matrix multiplications.

• It computes multiple positions after reading the target weights once.
• It reduces the number of sequential target-model calls.
• It uses the GPU's parallel compute more effectively.
• If the draft is accurate, several tokens are committed in one cycle.

The speed gain depends roughly on how much the accepted length exceeds the draft and verification costs.

$$\text{Speed benefit} \sim \frac{\text{Accepted tokens per cycle}} {\text{Draft cost}+\text{Verification cost}}.$$

If the draft model is too large, proposing tokens is expensive; if it is too small or its distribution is mismatched, rejection is frequent. If the batch is already large enough for the target GEMM to be compute-bound, verification has less spare capacity and the speedup may be small. EAGLE-3, Medusa, and Lookahead decoding improve draft accuracy or the construction of candidate trees.

Interview answer: Speculative decoding has a small draft model generate several future tokens and a large target model verify every position in parallel in one forward pass. A draft token is accepted with probability $\min(1,p/q)$; on rejection, a correction from the residual distribution $\max(0,p-q)$ preserves the target's exact output distribution. Small-batch decoding is memory-bound because it reads large weights for every token. Verification computes multiple tokens in one larger GEMM after one weight read, reducing sequential target calls. The actual gain depends on draft cost, acceptance rate, and batch size.

25. Definitions and Roles of Positive and Negative Samples in Contrastive Learning

At the heart of contrastive learning is deciding which two views should be treated as semantically equivalent. For anchor $a_i$, positive $p_i$, and negative set $\mathcal{N}_i$, InfoNCE can be written as follows.

$$\mathcal{L}_i = -\log \frac{ \exp(\operatorname{sim}(a_i,p_i)/\tau) }{ \exp(\operatorname{sim}(a_i,p_i)/\tau) + \sum_{n\in\mathcal{N}_i} \exp(\operatorname{sim}(a_i,n)/\tau) }.$$

Positive Samples

A positive pair is defined during training as sharing the same meaning or identity.

• SimCLR: Two views created by applying different augmentations to the same image
• Supervised contrastive learning: Different samples from the same class
• CLIP: An image and caption from the same original pair
• Speaker recognition: Different utterances from the same speaker
• Temporal learning: Nearby moments from the same video or observations from the same track

Positives define which transformations a representation should treat as equivalent—in other words, its invariances. Treating a strong crop and color jitter as positives teaches the model to preserve object semantics despite changes in color and partial position. If augmentation erases information needed for the task, however, the model learns the wrong invariance. For a task in which color determines the class, for example, color removal creates a bad positive.

Negative Samples

A negative is a sample defined as having a different meaning that should remain distinguishable. Negatives prevent representation collapse and create separation or uniformity among different instances and concepts in embedding space. Nearby hard negatives refine decision boundaries, but carry a greater risk of false negatives—mistakenly treating an actual positive as negative.

Negatives may come from in-batch samples, a memory bank, a momentum queue, or hard-negative mining. Increasing batch size raises the number of negatives, but not all are useful; negatives that are too easy produce almost no gradient. A lower temperature $\tau$ focuses more strongly on the most similar negatives.

Their Joint Effect on Representation Learning

Using only positive alignment allows collapse in which every sample maps to the same vector. Emphasizing only negative separation fails to group semantically equivalent variations. Good contrastive learning balances alignment, which brings views with the same semantics together, and uniformity, which spreads embeddings sufficiently across the space.

Non-contrastive or self-distillation methods such as BYOL, SimSiam, and DINO avoid collapse without explicit negatives through mechanisms such as stop-gradient, predictors, centering, and sharpening. Negatives are therefore not the only solution in representation learning, but they are a powerful way to directly define which samples should be distinguished.

Interview answer: A positive sample is a pair with the same meaning that should move closer, while a negative sample is a pair that should remain distinguishable and move apart. In SimCLR, two augmentations of the same image are positives and other images are negatives; in CLIP, a matching image–caption pair is positive. Positives define the transformations to which the model should be invariant, while negatives prevent collapse and create discriminative, uniform embedding spaces. Hard negatives refine boundaries but may push apart false negatives that actually represent the same concept. The definition of positives and negatives therefore determines which information the model preserves and discards.

26. Comparing the Training Methods and Objectives of PPO, DPO, and GRPO

All three methods are used for language-model post-training, but differ in the models and data they require and in whether they operate online or offline.

PPO

PPO is an on-policy RL algorithm that maximizes reward using responses generated by the current policy. It uses the probability ratio between old and new policies.

$$r_t(\theta) = \frac{\pi_\theta(a_t\mid s_t)} {\pi_{old}(a_t\mid s_t)}.$$ $$\mathcal{L}_{\mathrm{PPO}} = -\mathbb{E} \left[ \min \left( r_tA_t, \operatorname{clip}(r_t,1-\epsilon,1+\epsilon)A_t \right) \right].$$

Clipping prevents the policy from changing too much in one update. LLM RLHF requires a policy, a reference model, a Reward Model, and a value/critic model that estimates advantages. The effective reward also includes a KL penalty against the reference.

Its advantage is direct optimization of sequence-level reward while exploring responses actually generated by the current policy. Its disadvantages are rollout cost, memory for several models, critic training, and high implementation complexity.

DPO

DPO raises the relative probability of the chosen response in a fixed preference dataset $(x,y_w,y_l)$.

$$\mathcal{L}_{\mathrm{DPO}} = -\log\sigma \left( \beta \left[ \log\frac{\pi_\theta(y_w\mid x)} {\pi_{ref}(y_w\mid x)} - \log\frac{\pi_\theta(y_l\mid x)} {\pi_{ref}(y_l\mid x)} \right] \right).$$

Using the relationship between the optimal policy and reward in KL-regularized RLHF, DPO replaces the Reward Model and RL loop with a binary-classification objective. It is stable and simple using only offline data, but the current policy does not explore new answers and performance depends heavily on the coverage and quality of the preference dataset.

GRPO

GRPO removes PPO's separate value model and constructs relative advantages from the rewards of a group of responses generated for the same prompt.

It samples $G$ responses $o_1,\ldots,o_G$ for prompt $q$ and normalizes their rewards.

$$A_i = \frac{r_i-\operatorname{mean}(r_1,\ldots,r_G)} {\operatorname{std}(r_1,\ldots,r_G)+\epsilon}.$$

Token probabilities for each response are then updated using a PPO-like clipped ratio and KL regularization. This saves critic memory and is well suited to mathematics and code with verifiable rewards. It must generate multiple responses per prompt, however, and when all group rewards are identical the advantage signal is weak. Group-relative normalization offsets differences in prompt difficulty, but is sensitive to reward scale and batch composition.

More detailed equations are covered in DPO and Reflections on DeepSeek-R1, PPO, and GRPO.

Interview answer: PPO is on-policy RL that applies rewards and critic-estimated advantages to rollouts from the current policy while clipping probability ratios. It can explore and optimize sequence-level rewards, but is expensive because it requires policy, reference, reward, and value models. DPO is an offline method that directly raises the log probability of a preferred answer relative to a reference on fixed chosen–rejected pairs, eliminating the Reward Model and RL loop. GRPO samples several answers to the same prompt, constructs advantages from the group's reward mean and standard deviation, and updates like PPO without a critic. Its costs are multiple rollouts per prompt and dependence on group reward variance.

27. Structural Differences Between Vision Transformers and CNNs, with Their Advantages and Disadvantages

A CNN extracts features by sliding local kernels across an image. Locality and translation equivariance are built into the architecture.

$$y_{i,j} = \sum_{\Delta i,\Delta j} w_{\Delta i,\Delta j} x_{i+\Delta i,j+\Delta j}.$$

Because the same kernel is shared across all positions, parameter count does not directly grow with image resolution, and local pixel relationships are learned efficiently. Stacking layers expands the receptive field to obtain global context.

Vision Transformer divides an image into $P\times P$ patches, flattens each patch, and applies a linear projection.

$$N = \frac{HW}{P^2}, \qquad z_0 = [x_{cls};x_p^1W_E;\ldots;x_p^NW_E]+E_{pos}.$$

The resulting token sequence is passed to a standard Transformer encoder, and classification uses either a CLS token or mean pooling. Self-attention lets the model directly represent relationships among all patches from the first layer.

Structural Differences

• CNNs have strong inductive biases through local receptive fields and weight sharing.
• ViTs use global self-attention over patch tokens and have weaker vision-specific biases.
• In CNNs, convolutions perform spatial mixing, while $1\times1$ convolutions or MLPs primarily perform channel mixing.
• In ViTs, attention performs token mixing and the FFN performs channel mixing.
• CNN compute generally grows linearly with pixel count, while full ViT attention grows quadratically with the number of patch tokens.

Advantages of ViTs

ViTs process global context and long-range dependencies from early layers, and their architecture integrates easily with text and multimodal Transformers. Under large-scale pre-training, weaker inductive biases can become an advantage by allowing more flexible representations to be learned from data, and ViTs scale well. Attention maps and token representations are also easy to reuse for detection, segmentation, and VLMs.

Disadvantages of ViTs

With little data, a CNN's locality bias may be more sample-efficient. Details smaller than the patch size can be lost during patch projection, while attention cost grows with token count at high resolutions. Absolute position embeddings also require interpolation at different resolutions.

Swin Transformer introduced hierarchical features and nearly linear image-size scaling through local and shifted windows, while ConvNeXt showed that CNNs can compete with Transformers under modern training recipes. The comparison is therefore not that “ViTs completely replaced CNNs,” but a choice based on data scale, resolution, latency, and downstream architecture.

Interview answer: CNNs build in locality and translation equivariance through small local kernels and weight sharing, making them sample-efficient with limited data. ViTs convert images into patch tokens and apply global self-attention and FFNs, representing long-range relationships from the first layer and fitting large-scale pre-training and multimodal expansion well. Their weaker vision-specific inductive bias can hurt on small datasets, however, while attention cost grows quadratically with patch count at high resolutions and small details may be missed. Hybrid directions such as Swin use local windows.

28. Comparing Decoder-Only, Encoder-Only, and Encoder–Decoder Transformers

Encoder-only

Encoder-only models use bidirectional self-attention, allowing every token to see every other token. They focus on producing contextual representations of the full input. Pre-training objectives include Masked Language Modeling and replaced-token detection.

Representative models are BERT, RoBERTa, and DeBERTa. They are suited to input understanding and representation extraction, including text classification, NER, extractive QA, search embeddings, and reranking. Because they have no autoregressive decoder, they are not a natural base architecture for free-form long-text generation.

Decoder-only

Decoder-only models use causal self-attention to predict the next token from preceding tokens only.

$$p(x_{1:T}) = \prod_{t=1}^{T} p(x_t\mid x_{<t}).$$

Representative examples include the GPT, LLaMA, and Qwen families. Prompts and responses can be concatenated into one token stream, aligning the pre-training objective with generation and supporting in-context learning and general-purpose text generation. The architecture does not bidirectionally encode the full input, however, and token-by-token inference is sequential.

Encoder-Decoder

The encoder bidirectionally encodes the source sequence, while the decoder applies causal self-attention to the target sequence. Decoder cross-attention uses all encoder outputs as keys and values.

$$p(y\mid x) = \prod_t p(y_t\mid y_{<t},\operatorname{Enc}(x)).$$

Representative models include the original Transformer, T5, and BART. They are suited to sequence-to-sequence tasks with clearly separated input and output roles, such as translation, summarization, and document-grounded generation. The source can be encoded once and reused, while the decoder directly attends to the entire source. On the other hand, maintaining separate encoder and decoder stacks can make architecture and serving more complex than for a general-purpose causal LM.

Decoder-only models are now most visible because they unify many tasks behind one interface. Encoder-only models remain efficient for low-latency embeddings and classification, while encoder–decoder models remain strong choices for tasks with a clear source–target structure.

Interview answer: Encoder-only models create full-input representations using bidirectional attention, making BERT-style models suitable for classification, NER, and retrieval. Decoder-only models use causal attention and next-token prediction, making GPT and LLaMA suitable for generation, chat, and in-context learning. Encoder–decoder models bidirectionally encode the source, then causally generate with a decoder that cross-attends to the full source, making T5 and BART suitable for translation and summarization. No architecture is universally superior; the choice depends on whether the central need is input representation, free generation, or conditional generation.

29. Mixed-Precision Training, Loss Scaling, and Why BF16 Generally Does Not Need It

Mixed-precision training performs large matrix multiplications and stores activations in FP16 or BF16 to increase Tensor Core throughput and reduce memory and communication volume, while retaining numerically sensitive operations and optimizer states in FP32.

A representative configuration mixes the following.

• GEMMs, convolutions, and activations: FP16/BF16
• Reductions, parts of Softmax, and normalization statistics: FP32 accumulation when needed
• Adam first and second moments: usually FP32
• Parameter updates: using FP32 master weights or applying updates stably to BF16 parameters

Why FP16 Needs Loss Scaling

FP16 uses a 5-bit exponent and 10-bit mantissa. Its range between maximum value and minimum normal value is much narrower than FP32's. During the backward pass, a small gradient below FP16's range underflows to zero.

Multiplying the loss by a large scale $S$ also multiplies the gradients by $S$.

$$\nabla_\theta(S\mathcal{L}) = S\nabla_\theta\mathcal{L}.$$

Dividing the gradients by $S$ before the optimizer step after the backward pass is mathematically equivalent to using the original gradients, but keeps intermediate gradients within FP16's representable range.

Static loss scaling uses a fixed $S$. Dynamic loss scaling increases the scale when there is no overflow; when Inf or NaN is detected, it skips the step and lowers the scale, because an excessively large scale causes overflow.

Why BF16 Usually Does Not Need Loss Scaling

BF16 has an 8-bit exponent and 7-bit mantissa. Although its mantissa precision is lower than FP16's, it shares FP32's exponent range and can represent both very small gradients and large values across a wide dynamic range. It therefore generally does not need to magnify the loss to avoid gradient underflow, the main problem under FP16.

BF16 does not eliminate every numerical issue. Its short mantissa may prevent small updates from affecting large parameters, and Softmax and reductions may still require FP32 accumulation. Saying that loss scaling is generally unnecessary does not mean that FP32 states and numerical-stability measures are unnecessary.

Interview answer: Mixed precision computes GEMMs and activations in FP16/BF16 for Tensor Core speed and memory efficiency, while keeping sensitive parts such as reductions and Adam states in FP32. FP16 has only a 5-bit exponent, so small gradients readily underflow to zero. Loss scaling multiplies the loss by $S$ to bring gradients into the representable range, then divides by $S$ before the update. BF16 has a shorter mantissa but the same 8-bit exponent as FP32, giving it a wide dynamic range, so it generally does not need loss scaling. FP32 accumulation and optimizer states remain important.

30. Differences Among Data, Tensor, and Pipeline Parallelism and Their Use Cases

The three methods differ in what they replicate and what they divide.

Data Parallelism

Data parallelism replicates the model on every GPU and divides the data batch. Each GPU computes local gradients and All-Reduces them.

$$\text{Same Model} + \text{Different Data}.$$

It is simple to implement and provides good throughput scaling when the model fits on one GPU. On the other hand, model state is replicated, the global batch grows, and gradient communication is required. DDP, ZeRO, and FSDP fall along this axis. ZeRO/FSDP preserve the computational semantics of data parallelism while reducing state replication.

Tensor Parallelism

Tensor parallelism divides a single layer's tensor operations across multiple GPUs. For example, column-partitioning $W$ in linear layer $Y=XW$ gives

$$W=[W_1,W_2], \qquad Y=[XW_1,XW_2].$$

Depending on the following operation, partial results are combined with All-Gather or All-Reduce. Megatron-LM, which combines column and row parallelism for attention heads, QKV projections, and FFNs, is a representative implementation.

Dividing layer weights and computation solves the problem of very wide layers not fitting on one GPU. Every GPU computes the same layer concurrently, so there is no pipeline bubble. But collectives occur frequently in every Transformer block, making fast intra-node interconnects such as NVLink/NVSwitch important. If the TP degree is too high, matrices per GPU become too small for efficient execution.

Pipeline Parallelism

Pipeline parallelism divides consecutive layers into stages, assigning a different segment to each GPU.

$$\text{GPU 0}:L_1\sim L_k, \quad \text{GPU 1}:L_{k+1}\sim L_m, \quad\cdots$$

The mini-batch is divided into micro-batches. While an earlier stage computes the next micro-batch, a later stage processes the previous one. GPipe uses this kind of pipeline together with activation recomputation.

Dividing layers and activations by stage lets a deep model span multiple GPUs or nodes. Communication between stages primarily consists of activations and gradients, making inter-node deployment possible. The disadvantages are bubbles in which GPUs sit idle during pipeline fill and drain, imbalance in computation across stages, and complexity in micro-batch scheduling and activation management.

Pipeline efficiency generally improves when the number of micro-batches $m$ is sufficiently larger than the number of stages $p$. Under a simple GPipe schedule, the bubble fraction decreases approximately as $(p-1)/(m+p-1)$. A 1F1B schedule interleaves forward and backward passes to reduce activation memory.

Combining Them in Real Large-Scale Training

The three methods are not mutually exclusive.

• Use tensor parallelism over fast links within a node.
• Use pipeline parallelism to divide groups of layers across nodes or GPU groups.
• Replicate the entire model-parallel group and apply data parallelism or FSDP across replicas.
• Add sequence or context parallelism for long contexts.

This is called 3D parallelism or, more broadly, multidimensional parallelism. The degree assigned to each axis depends on model-layer width and depth, sequence length, GPU memory, node topology, network bandwidth, and global-batch constraints.

Interview answer: Data parallelism replicates the model, divides the batch, and All-Reduces gradients, making it effective for throughput scaling when the model fits on one GPU. Tensor parallelism divides a layer's weight matrices or attention heads across GPUs for concurrent computation, allowing wide layers to be split, but requires fast inter-GPU links because every layer performs collectives. Pipeline parallelism divides layers into stages and streams micro-batches through them to place deep models across devices, at the cost of pipeline bubbles and stage imbalance. Large LLMs commonly combine TP within nodes, PP across nodes, and DP/FSDP across complete groups.

Conclusion

The 30 questions may look independent, but they are connected along several broad axes.

The first axis is objectives and training distributions. Cross-Entropy follows from MLE, while teacher forcing makes that likelihood efficient to learn but creates exposure bias. SFT, DPO, PPO, GRPO, RLVR, and distillation ultimately differ in which rewards and targets they provide under which data distributions.

The second axis is representations and architectures. Attention mixes information between tokens, while FFNs transform features within tokens. Positional encodings provide order, and residual connections and normalization make deep networks trainable. The evolution of vision–language models after CLIP likewise expanded the purpose of representations from global alignment to fine-grained fusion, generation, and instruction following.

The third axis is memory, computation, and communication. FlashAttention, gradient checkpointing, KV caching, and mixed precision aim to produce the same result with less memory and I/O. MoE, speculative decoding, and sparse attention reduce active computation or sequential calls. DDP, FSDP, tensor parallelism, and pipeline parallelism move the limits of one device into the combined memory and compute of many devices, but introduce communication as a new cost.

After explaining any method in an interview, it helps to ask yourself the following questions.

What did it reduce, and what increased in return? Does it preserve the exact result or approximate it? Is it a training problem or an inference problem? Under which hardware and data conditions does it provide a real benefit?

If you can distinguish these four points, you can explain a new paper or system from first principles rather than merely reciting its name.

I still often think of a better answer only after the interview ends, and this post may still contain concepts I have misunderstood. So rather than declaring that “my interview preparation is now complete,” this is a record meant to keep me from forgetting where I got stuck. If another rejection teaches me something new, a 31st question will probably appear quietly. It is not the most pleasant update mechanism, but so far it has been quite effective.