What is modality?

Before introducing the paper, let us clarify the meaning of modality so we can understand multimodal. The term can be interpreted broadly or narrowly. In a probability distribution, a mode belongs to a probability density function, and multimodal means that the distribution has two or more distinct peaks (local maxima).

That is only a mathematical interpretation based on probability distributions. In deep learning, the term is more often connected to sensory experience.

Human perception has many modalities—broadly, forms or modes—such as hearing, touch, and vision. The interaction and overlap among these senses provides a basis for inference. Multimodality may therefore describe a form of communication or, in evolutionary psychology, a basis for particular behavioral patterns.
Verbal language, for example, includes a lexicon describing the meaning of individual words; syntax, which structures speech and expresses relationships among words; and pragmatics, which interprets meaning by considering context such as the relationship between speakers, time, and place.
Beyond language, visual elements include gestures, body posture, and proxemics—body language defined through relationships with social, cultural, and spatial environments. Eye contact and facial expressions can likewise be viewed as forms of modality.
So far we have considered the bodily organs that gather internal information and the components involved in communicating it. We can now define modality in a narrower sense.

Modality in computational environment

A modality is the way or means by which something is experienced or occurs. In a computing environment, it may mean a particular form of information or a representative format in which information is stored. If an image is saved on a computer, for example, the image itself can be treated as a modality. At a more detailed level, modalities may be distinguished by storage and compression format, such as JPEG, BMP, or PNG.
In keeping with sensory modalities in human perception and modes in mathematical probability distributions, a modality (or medium/media) is a way of storing and communicating information through some method or instrument. It can be understood as a system of transmission and communication.

The world contains social and cultural platforms recognized as many kinds of communities, and modality classifies the information and communication methods defined within them. Categories especially relevant to deep learning include natural language (spoken or written), visual information (images and videos), and auditory information (voice, sounds, and music). Here, multimodal means using two or more modalities together to solve a task. It covers not only tasks combining very different modalities such as text and images, but any task where the modalities can be distinguished, such as human voice plus instrumental performance.

Learning Transferable Visual Models From Natural Language Supervision

I spent so long explaining multimodality before beginning the review because one of this paper's main contributions is its search for a “meaningful relationship between text and images.” This formidable 48-page paper is so famous that many blogs have reviewed it and YouTube offers no shortage of lectures. This post focuses on the strengths, weaknesses, and contributions of the paper, both as a paper and as a method.

Advances in NLP and Limitations of Computer Vision

Deep learning for computer vision and natural language processing (NLP) developed along different paths. Early work began with CNNs (Convolutional Neural Networks) as a baseline and developed recurrent neural networks. The introduction of attention mechanisms and Transformers greatly improved machine-translation tasks, and models such as GPT and BERT, which use the Transformer's encoder and decoder structures, substantially improved many downstream tasks.

Unlike computer vision, large language models could advance in large part because of the ease of collecting datasets. Solving an image-related task usually requires defining a domain, restricting a distribution appropriate to that domain, and then collecting and labeling matching data. To classify dogs, for example, one must collect dog photographs and match each photograph to a breed.

This is where the limitations of image-dataset collection become clear. If thousands or tens of thousands of dog breeds must be classified, each needs an index label. More importantly, the consistency and quality of the dataset distribution cannot be guaranteed: backgrounds may occupy more of an image than the dog because object sizes vary, or a photograph may contain several dogs. Even after laboriously collecting a balanced dataset like the one above, adding a new “Shiba Inu” class requires building data for that class from scratch.

This differs greatly from NLP models, which can simply acquire huge datasets from the web and use the many resulting text prompts as token-level supervision. In computer vision, constructing a meaningful dataset that accounts for both quality and distribution is called “gold labeling,” and good deep-learning performance on computer-vision tasks is difficult without such gold labels.
NLP and CV also differ in zero-shot and few-shot performance on new tasks. Natural-language models can build vast web datasets. Unlike images, whose distribution is nearly unbounded and continuous, language consists largely of forms and variations within human vocabularies and sentences, a discrete signal. A large network trained on massive data can therefore adapt easily to other tasks.

Restricted form of supervision

The paper presents the limitation of state-of-the-art computer-vision systems—that they can learn only data tied to predefined object categories, with no practical alternative during collection—as a problem to solve. Without addressing it, CV can never attain the generality and usability across downstream tasks enjoyed by NLP models.
CLIP therefore uses raw text alongside image training. Rather than learning image supervision itself, it connects text representations and image representations through their relationships with text prompts.
This also addresses limitations in conventional computer-vision data collection. Instead of collecting web images and categorizing them into classes, one can take text prompts that are frequently searched or repeated online and retrieve images that describe them. Training then requires no dataset explicitly constructed around a domain distribution. If “cute cat” is a phrase frequently mentioned on Wikipedia, for example, the model need only associate images crawled from that text with the phrase “cute cat.”

The figure above compares this with the conventional approach. Dataset construction no longer begins by defining a task. Instead, it specifies text prompts for joint image-text learning and collects data from them. Since the data is gathered by prompt, unlike task-based dataset collection, labeling requires no additional step.

Task-robust representation learning with CLIP

Research on learning organic relationships between images and text existed before CLIP. Machine-learning approaches included content-based image retrieval—predicting nouns and adjectives from images supplied with a document—caption prediction with manifold learning, and learning low-level image and text-tag features with a multimodal deep Boltzmann machine. Later deep-learning approaches based on CNNs and Transformers still failed to improve zero-shot learning from the perspective of image representation learning. Most pre-CLIP methods occupied a middle ground between conventional computer vision's gold labels and datasets collected from unlimited text prompts: they trained on huge curated datasets such as JFT-300M or used related prompts as Instagram hashtags. They still relied on limited supervision and traditional softmax classifiers, constraining flexible representation learning and zero-shot performance.
Unlike those weakly supervised methods, VirTex, ICMLM, and especially ConVIRT, which most strongly inspired CLIP, use language information to learn image representations in a direction similar to this paper. Those studies did not use data at massive scale. CLIP adopts the concept of ConVIRT—whose medical-diagnosis setting naturally limits its dataset—but performs contrastive representation learning from scratch on 400 million WebImageText pairs.
The authors explore the most efficient training method and ultimately learn a network applicable to many tasks, much like the GPT family of language models.

Training bag-of-words (BoW) prediction is more efficient than training a Transformer language model as an image-captioning baseline, and combining bag-of-words prediction with CLIP-style contrastive learning is reportedly about 12 times more training-efficient for improving zero-shot performance.

Several approaches

Although the preceding explanation was long, the paper's classification of its approaches is the best place to understand its main contribution. I will summarize each component briefly.

NLP supervision

Earlier work training text-image pairs mostly used natural-language supervision in unsupervised, self-supervised, or weakly supervised settings. It could directly relate text information to image representation learning but did not fully exploit the information in NLP. NLP requires no one-to-many mapping during training and contains more usable information than categorized labels. The paper therefore moves beyond the conventional idea of multimodal image-text learning that predicts text from images under NLP supervision. Instead, it simply connects image representations to NLP representations.

Dataset Collection

Datasets such as MS-COCO and Visual Genome were much too small, while even a large dataset such as YFCC100M made it difficult to extract information from image metadata such as filenames. Building a sufficiently large dataset was therefore hard.
Inspired by web-based collection for NLP supervision, the authors augment frequently used internet queries (text prompts) to create a set of 500,000 and collect up to 20,000 image-text pairs for each query. The resulting WebImageText dataset rivals the scale of WebText, the GPT-2 training dataset, as intended.

Selecting an efficient pre-training method

As the dataset grew, training time also increased, making an efficient pretraining method for rapid convergence essential. The authors describe several approaches they tried along the way.

Jointly trained an image CNN and text transformer from scratch

This follows the approach of VirTex. As the Transformer-based learning graph above shows, however, it converges about three times less efficiently than a simple baseline using ResNet and a BoW encoder.

Bag of Words With a Simpler Baseline

The BoW and Transformer approaches differ only in architecture—parameter count, CNN versus Transformer. Both remain tied to the NLP-supervision concept of finding exact words for every image.
This restricts an image to a single description. Consider the following image, for example.

Its text prompt need not be only “an image of two cats.” It could be “two cats who like each other,” “two cats standing in a field,” or any representation that can occupy a nearby region of the related text embedding space. Contrastive learning—increasing similarity to related text embeddings and decreasing similarity to unrelated ones—therefore trains more efficiently. This is the method CLIP adopts.

Contrastive learning

Suppose a batch contains $N$ image-text pairs. CLIP makes predictions over all $N \times N$ possible pairings. Since each image has one correct text prompt, $N$ of these predictions are positive pairs and the remaining $N^2-N$ are negative pairs.

As the figure shows, the symmetric cosine similarities between sample embeddings can be represented as a matrix, and training optimizes a normalized cross-entropy loss over it. Batchwise similarity-based cross-entropy was introduced in work on multi-class $N$-pair loss and InfoNCE loss; CLIP uses the same mechanism.

Training from scratch

With 400 million examples, overfitting is not a major concern. The network is therefore pretrained from scratch without representation-specific initialization, and the only data augmentation is cropping. It also uses no nonlinear projection when mapping into the multimodal embedding space. Unlike self-supervised learning, this setting apparently does not need that technique for training efficiency, so omitting it reduces computation. The temperature-normalization hyperparameter $\tau$, which controls entropy, is learned because fixing it manually would make ablation difficult.

Network selection

CLIP uses the ResNet-50 from the BoW baseline, along with several encoder improvements proposed in ResNet-D. To match the image-embedding dimensions, it replaces global average pooling for feature extraction with an attention-pooling layer.
The ViT (Vision Transformer) architecture is otherwise unchanged except for an additional layer normalization applied to the combined patch and position embeddings.
The text encoder follows GPT-2. Its base configuration has 12 layers, a channel width of 512, and 8 attention heads.

Results

This paper became exceptionally long for three reasons:

1. There is an enormous amount of related work.
2. The authors claim a broad range of contributions.
3. Each concept receives an extensive buildup.

The experimental section is correspondingly long and somewhat difficult to absorb in a full read-through. I will therefore use the figures to focus on the most important results.

Zero-shot with prompt engineering

CLIP's central claim is that, like natural-language models, image models trained for representation learning on enough data can achieve performance on new tasks in a zero-shot setting. The downstream procedure for a new classification task is shown below.

Rather than using a class name alone as the prompt, the authors improve performance with dataset-specific prompt engineering such as “A photo of a {object}.” Their main analysis is that training image-text pairs contain descriptions of image scenes, not merely words naming objects, creating a difference between the learned domain distribution and the class-name distribution.

The model computes similarity with $N$ text prompts for $N$ classes and predicts the index with the highest similarity. Prompt engineering does indeed improve average classification performance.

Zero-shot vs fully-supervised baseline

Because the baseline study uses ResNet-50, CLIP can be compared with a linear probe. Across the 27 datasets in the table, it outperforms fully supervised training on 16 of them, yielding a positive $\Delta$.

Zero-shot vs Few-shot

Comparing the best results among few-shot methods given a fixed number of samples, zero-shot CLIP performs on par with a 4-shot method. Even with 16 shots, existing approaches such as BiT-M and SimCLRv2 struggle to surpass the accuracy of zero-shot CLIP.

The table reports how many few-shot samples are needed to match CLIP's zero-shot performance. Most datasets require an average of 20.8 samples, demonstrating CLIP's data efficiency and task robustness.

Zero shot performance & Linear probe

The paper also shows that CLIP does more than loosely connect text and images: its image encoder learns representations useful to a classifier-based linear probe. The graph reveals that poor zero-shot performance tends to coincide with poor linear-probe performance; CLIP also struggles on tasks where classification itself is difficult.

Zero shot performance with network scale

Following the same trend reported by existing methods such as EfficientNet, increasing the image encoder's representational capacity—that is, scaling the network—improves CLIP's zero-shot performance.

CLIP is also a good representation learner

The graph compares average score against GFLOPs per image during training. It is not a zero-shot comparison; instead, it measures how much CLIP's pretrained representations help linear-probe performance. The best CLIP network in the graph is compared across several datasets with EfficientNet-NoisyStudent, the strongest representation learner among the other networks.

Since this is CLIP trained with ViT-L/14, stronger performance is not surprising, but it nevertheless achieves a positive $\Delta$ on 21 of 27 datasets.

Robustness to natural domain shift

Although supervised deep-learning networks have become very strong, they remain vulnerable to distribution shift. A network trained only on ImageNet, for example, may fit the neural network to the in-distribution of its training dataset instead of learning and recognizing meaningful features of each class's objects. The authors validate on seven natural distribution shifts of ImageNet: ImageNetV2, ImageNet Sketch, YouTube-BB and ImageNet-Vid, ObjectNet, ImageNet Adversarial, and ImageNet Rendition.

I do not quite understand why the paper gives no explanation of this figure, but my interpretation is that zero-shot CLIP models preserve more of their ImageNet performance when transferred through a linear probe and thus handle domain shift well. This is not the main figure, however. The next one provides experimental evidence for why CLIP became a game changer in zero-shot and domain-shift tasks.

Ideally, performance on ImageNet and a domain-shifted dataset would remain identical despite the shift ($y=x$). A model trained only on ImageNet loses $2\sim30\%$ performance, whereas zero-shot CLIP shows that average performance over the seven naturally shifted datasets can improve by as much as roughly $75\%$ relative to that baseline.

One might ask whether evaluating purely zero-shot performance differs meaningfully from using a dataset-specific, fine-tuned logistic-regression classifier. Fine-tuning on ImageNet improves accuracy on the original dataset by a remarkable $9.2\%$. On ImageNetV2, collected similarly to ImageNet, the gain is about half as large. Performance declines on every other dataset except the video datasets YouTube-BB and ImageNet-Vid.
If CLIP with its text encoder performed only zero-shot classification, the experiments above would support a straightforward conclusion: text guides predictions and makes them robust to domains. The surprising result is that accuracy under domain shift can remain strong even after a classifier fine-tunes the downstream task. The authors move on without clearly explaining why.
Another experiment addresses the fact that transfer-dataset classes do not always match ImageNet's 1,000-way classifier. The previous procedure max-pooled predictions over ImageNet subclasses, which was imprecise. The authors replace it with a CLIP-native method: text embeddings that prompt class names. This improves the video datasets YouTube-BB and ImageNet-Vid, but produces only a small gain on ObjectNet.

The figure shows that as few-shot supervision is added to zero-shot CLIP, its robustness advantage over fully supervised learning declines. In other words, using more dataset samples fits the model more closely to the in-distribution, reducing robustness at the same ImageNet performance relative to the original zero-shot CLIP model.

Comparison to Human Performance

Measuring human zero-shot performance is very difficult. The Oxford-IIIT Pets dataset, which asks subjects to identify 37 dog and cat breeds, nevertheless allows a comparison of zero-shot and few-shot task performance. Even a human is unlikely to know every dog and cat breed. Show a person an image; if they know the breed, they answer, and otherwise they say “I don't know.”

Intuitively, one image (one shot) usually improves a person's accuracy on that breed, but showing $n$ additional images does not continue to raise it. “Humans know what they don't know”: because people clearly recognize the gaps in their knowledge, learning the missing information once is enough.

The authors note that deep-learning networks, like humans, fail to turn prior knowledge into meaningful few-shot gains, and that solving this will be an important advance for CLIP. The graph also shows that humans and CLIP find broadly similar breeds difficult in zero-shot classification.

Data overlap analysis

WebImageText trains on a massive number of web images. Some validation datasets also contain samples available online, so a validation set could leak into the pretraining data. One way to prevent this is to remove all duplicate images—samples identical to validation examples—before training. But checking every dataset is time-consuming.
Instead of this brute-force approach, the authors define overlap and detect it through the following procedure.

1. Run a duplicate detector, described in the appendix, on every example in each validation dataset. Manually inspect nearest neighbors, choose a threshold based on an inter-sample distance metric, and divide examples into Overlap (similarity above the threshold) and Clean (similarity below it). To measure dataset cleanliness—the absence of overlapping data—compute the ratio of Overlap to All (Overlap + Clean).

2. Compute CLIP RN50x64 zero-shot accuracy on All, Overlap, and Clean. Since (All accuracy) minus (Clean accuracy) is the accuracy difference caused by contamination, a positive value indicates how much the model overfits overlapping data.

3. Because the Overlap set may be small, additionally perform a binomial significance test. Treat Clean accuracy as the null hypothesis and obtain a one-tailed (greater) $p$-value for the Overlap subset. Also compute $99.5\%$ Clopper–Pearson confidence intervals for the Dirty subset.

Binomial distribution test

Before examining the results, let us understand the binomial-distribution test through its equations. Let the test hypothesis $H_0$ be the null hypothesis. Here, the null hypothesis is Clean accuracy: zero-shot validation performance after excluding the small overlapping subset.

$$H_0 = \pi = \pi_0$$

Accuracy lies between $0$ and $1$, so treat it as the probability that CLIP correctly classifies a particular sample under the non-overlapping dataset.
If the Overlap dataset contains $n$ examples and CLIP correctly classifies $k$ of them, the probability is

$$\text{Pr}(X = k) = \begin{pmatrix} n \newline k \end{pmatrix} p^k (1-p)^{n-k}$$

We want the $p$-value for the probability $p$ in the Overlap dataset exceeding this value, because only then can we posit overfitting.

$$p = \sum_{i=0}^{k} \text{Pr}(X = i) = \sum\_{i = 0}^k \begin{pmatrix} n \newline i \end{pmatrix} \pi\_0^i (1-\pi_0)^{n-i}$$

The one-tailed test for this $p$ is shown above, because the reference point is an expected $n\pi_0$ successes.

The results appear above. Among 35 datasets, the authors conclude that 9 with higher Clean accuracy than Overlap accuracy show no overlap effect. Overall, overlap is not severe for most datasets. At a $0.1\%$ threshold, overall accuracy shifts substantially on 7 datasets; under Bonferroni correction, the two largest effects are on Birdsnap and Country211.

Conclusion

The remainder of the paper discusses its limitations and detailed implementation points not covered elsewhere, including the appendix. Its main contributions can be summarized as follows.

1. It constructs a massive dataset like those used for text models and trains an image model efficiently.
2. It learns meaningful text-image relationships, enabling multimodal learning.
3. It dramatically improves zero-shot transfer learning.

The experiments reveal several additional, secondary contributions.

1. Linear-probe and logistic-regression performance confirms that the model learns useful representations.
2. It produces a model robust to domain shift.
3. Rather than collecting data indiscriminately, it analyzes dataset overlap and shows that the performance gains do not result from overfitting.