Introduction

As its title suggests, this paper investigates the abilities and properties that a Vision Transformer can acquire through self-supervised learning. ViT was inspired by the Transformer architecture developed for language. Follow-up work such as GPT and BERT had already shown in NLP that self-supervised learning over large datasets produces rich semantic information for downstream tasks. Large-scale image pretraining also benefits downstream ViT tasks, but most earlier approaches remained tied to supervised learning.

Emergent Properties

The paper begins with the motivation behind its self-supervised approach, describes the effects that emerge from it, and verifies them through experiments. Its main findings are:

• Features learned by a self-supervised ViT contain semantic-segmentation information, a property not observed in a supervised ViT or convolutional network. As the figure shows, regions highlighted by the model's attention correspond directly to semantic objects in each image.

• The learned features work directly with a similarity-based $k$-NN classifier, and even a small ViT achieves strong ImageNet recognition accuracy.
• Experiments and ablations identify ingredients that matter for self-supervised ViT training: a momentum encoder, multi-crop training, and smaller patches, which create more patch tokens.

Self-Supervised Learning Frameworks

Representative unlabeled or self-supervised approaches include SimCLR, MoCo, and BYOL. They matter here because DINO's emergent properties arise from self-supervised ViT training, and understanding its structure requires the prior work on which it builds. Among the three, DINO draws most directly from BYOL.

SimCLR follows the first diagram. Given input $x$, two random augmentations produce $\tilde{x}_i$ and $\tilde{x}_j$. Neural encoder $f(\cdot)$ maps them to representations $h_i$ and $h_j$, and projection head $g(\cdot)$ produces latents $z_i$ and $z_j$, which train contrastively.

MoCo likewise uses contrastive learning and two branches. SimCLR learns representations with the same encoder inside a mini-batch. MoCo instead uses a momentum encoder that receives no direct gradient and is updated by exponential moving average. A queue is continually refreshed with enqueue and dequeue: positive logits compare matching views, while negative logits draw on samples retained from earlier batches. This makes the key encoder evolve gradually and supplies many consistent negatives without an enormous current batch.

A Short Review of BYOL

BYOL retains the momentum-encoder idea and otherwise resembles SimCLR's two-view training. Its major algorithmic difference is that it does not use contrastive learning and therefore needs no negative pairs.

The central question is how BYOL trains stably without negatives to prevent representational collapse. The answer is tied to the asymmetry between its online and target branches. Predictor $q_\theta(\cdot)$ resembles projection network $g_\theta$ and emits the same output dimensionality; it is not a conventional classifier. The predictor approaching an optimum is a key part of the learning dynamics.

If the converged predictor is $q_\theta^\ast$, then for online representation $z_\theta$ we can write

$$q_\theta^\ast(z_\theta)=\mathbb{E}[z_\xi^\prime\mid z_\theta].$$

The conditional expectation is a function of $z_\theta$, much as an optimally trained neural network on dataset $\{X,Y\}$ can be interpreted as a parameterized posterior $p_\theta(Y\mid X)$. Under this optimal-predictor assumption, a simplified BYOL loss—the full method symmetrizes across the two views and normalizes the latents—is

$$\mathcal{L}_\text{BYOL} = \mathbb{E}\left[\left\lVert \mathbb{E}(z^\prime_\xi\vert z_\theta)-z_\xi^\prime\right\rVert_2^2\right].$$

Its gradient with respect to online parameters $\theta$ becomes the gradient of an expected conditional variance:

$$\nabla_\theta \mathcal{L}_\text{BYOL}= \nabla_\theta\mathbb{E}\left[\sum_i \operatorname{Var}(z^\prime_{\xi,i}\vert z_\theta) \right].$$

This result assumes the predictor has reached its optimum, allowing the online network to update while treating the optimized predictor as fixed. It follows from the envelope theorem and optimality conditions for the corresponding multivariable, Lagrangian optimization problem.

Under this view, the online branch $\theta$ and target branch $\xi$ cannot both reduce the objective through the same collapsing direction. With the target projection $z_\xi^\prime$ and online representation $z_\theta$, the terminal conditional model is treated as fixed once the predictor has approximately converged. Conditioning a random variable on additional information cannot increase its conditional variance. If the online representation collapsed to a constant $c$, then

$$\operatorname{Var}(z^\prime_\xi \vert z_\theta) \le \operatorname{Var}(z^\prime_\xi \vert c).$$

The collapsed online state therefore lies at a less favorable, higher-variance region of this surface and is unstable.

The reverse case explains why $\xi$ is not optimized directly by the loss but updated slowly through EMA. If the target network collapses, $z_\xi^\prime=c$ becomes a deterministic constant, so

$$\operatorname{Var}(c \vert z_\theta)=0\le\operatorname{Var}(z^\prime_\xi\vert z_\theta).$$

Directly optimizing the target would thus favor collapse. BYOL instead uses a similarity loss to bootstrap future online parameters from past online parameters through the momentum target. Avoiding negative pairs reduces dependence on batch size and considerably expands the design space for self-supervised learning.

DINO Approach

DINO stands for “self-DIstillation with NO labels.” It trains a ViT in a student–teacher, or online–target, framework. The teacher produces stable pseudo-labels and is not optimized by the loss. Instead, it tracks the continually trained student through an exponential moving average of the student's parameters.

Knowledge Distillation

During training, the student and teacher act as functions mapping data to outputs that can be interpreted probabilistically. For a student $g$ with parameters $\theta_s$, input $x$ produces logits $g_{\theta_s}(x)$. Applying softmax along the feature dimension gives

$$P_s(x)^{(i)} = \frac{\exp(g_{\theta_s}(x)^{(i)}/\tau_s)}{\sum_{k=1}^K\exp(g_{\theta_s}(x)^{(k)}/\tau_s)}.$$

Temperature $\tau_s$ controls the sharpness of the student's probability distribution. The teacher is formulated similarly:

$$P_t(x)^{(i)} = \frac{\exp(g_{\theta_t}(x)^{(i)}/\tau_t)}{\sum_{k=1}^K\exp(g_{\theta_t}(x)^{(k)}/\tau_t)}.$$

Knowledge distillation treats the teacher output as a soft ground truth and optimizes consistency with the student. In cross-entropy terms, the teacher distribution replaces a one-hot label:

$$\underset{\theta_s}{\min} H(P_t(x),~P_s(x)) = \min_{\theta_s} \{-P_t(x) \log P_s(x)\}.$$

Cross entropy measures how closely one distribution represents the information in another, so the student learns to imitate the teacher's knowledge. DINO adds several mechanisms beyond this basic objective.

Data Augmentation

One weakness of a Transformer is limited built-in local-to-global correspondence. Attention sees global information immediately. That is powerful, but if attention maps converge to global features before learning local structure, self-supervision cannot benefit from the hierarchical correspondences naturally encouraged by a CNN.

DINO addresses this through asymmetric augmentation:

1. Once the teacher has acquired some local–global consistency, it can make a plausible prediction from the global form of an image.
2. The teacher therefore continues to receive only global views.
3. Because the teacher is the EMA of the student, the student must learn the local–global consistency needed for the teacher to converge well.
4. The student consequently also receives local image views.

In concrete terms, only two large global crops $x_1^g$ and $x_2^g$ enter the teacher. A multi-crop strategy supplies the student with both those global views and several smaller local crops:

$$\underset{\theta_s}{\min}\sum_{x \in \{x_1^g,~x_2^g\}} ~~\sum_{x^\prime\in V,~x^\prime \neq x} H(P_t(x), P_s(x^\prime)).$$

The two global crops cover more than 50% of the original image, while each local crop covers less than 50%.

Avoiding Collapse

Self-supervised representation learning has no label-based ground truth and can collapse to a trivial constant. Contrastive negatives, clustering, BYOL's predictor, and even batch normalization all serve in part to prevent collapse. DINO uses normalization and other stabilizers, but its distinctive mechanism is centering and sharpening the momentum teacher's outputs.

Sharpening lowers the softmax temperature to make predictions more decisive. Centering subtracts a running center $c$ from teacher logits, balancing the probabilities. Their effects appear opposed: sharpening reduces entropy, while centering prevents a few dimensions from dominating. Why sharpen a distribution and then rebalance it?

The center is updated exponentially, like the teacher parameters:

$$c \leftarrow mc + (1-m)\frac{1}{B}\sum_{i=1}^B g_{\theta_t}(x_i).$$

Thus $c$ stores a running average of earlier model outputs beyond the current batch.

Sharpening the prediction for a cat image makes a diffuse response and its feature map clearer. If a recurring batch bias is already pushing the model toward collapse, however, sharpening alone intensifies that bias.

The intended goal is to extract discriminative features from different inputs. Without a mechanism like contrastive learning, simply minimizing prediction entropy can make every image produce the same feature—a trivial solution.

Centering reintroduces information accumulated from previous batches and makes training more robust to batch size. Contrastive learning often needs a large batch to obtain many positive and negative pairs that approximate the full data distribution. DINO instead stores a running prototype of past predictions and uses it to smooth later sharpened outputs, gaining some of the same stabilizing effect without enlarging the batch.

Sharpening alone drives entropy toward zero; centering alone keeps outputs overly smooth. Using only one produces representation overfitting or underfitting as training proceeds. Their balance is especially visible in the KL-divergence ablation.

The student–teacher architecture is meant to bootstrap learning: the EMA teacher accumulates representation knowledge, and the student learns to follow its predictions. If the representation collapses, the distinction between student and teacher predictions vanishes and so does that bootstrapping effect.

Experimental Results

Classification

Self-supervised or unsupervised features can be evaluated in several ways, depending on how a downstream head is used:

• Linear classifier: freeze the learned backbone and train only a linear classifier to measure representation quality.
• Fine-tuning: attach a head and fine-tune the pretrained backbone for the task.
• $k$-NN classifier: use embedding retrieval directly, without an additional learned classifier, to measure the representation and metric space themselves.

DINO performs strongly and achieves meaningfully higher classification accuracy than other self-supervised methods using comparable ViT backbones.

ViT Attention Maps

More distinctive than classification accuracy is what appears in the ViT attention maps: DINO can focus attention on local features and thereby localize objects. Ordinary supervised ViT training does not produce the same kind of feature map.

High-level vision tasks such as segmentation require precise pixel-level predictions, so self-supervised methods had generally lagged further behind than on classification. DINO's attention maps nevertheless show that its training method localizes ViT attention effectively, and the quantitative results confirm the effect.

Conclusion

DINO combines a BYOL-inspired self-distillation architecture with sharpening and centering to prevent collapse and stabilize unlabeled ViT training. Once the ViT was trained reliably through self-supervision alone, something remarkable emerged: its attention maps localized semantic objects.

Supervised learning had long been treated as the default. Stable self-supervised methods sometimes exceeded supervised accuracy, but their largest value often appeared through linear probing or fine-tuning rather than a qualitatively new capability of the representation itself. A ViT has little architectural inductive bias for localization, yet after DINO training its attention map behaves like a segmentation mask. This result demonstrates that in computer vision, as in NLP, self-supervision can provide advantages extending well beyond classification.