Introduction

We train large-scale pretrained networks on massive datasets so that their learned representations can be transferred quickly to many downstream tasks. In deep learning, this is viewed as representation transfer or knowledge transfer. The CoCa paper likewise proposes a way to train a large network on image–text pairs so that it can be used robustly across a wider range of tasks. CoCa unifies encoder-level contrastive learning for understanding, as in CLIP and ALIGN, with encoder–decoder generative learning, as in SimVLM.

In an ordinary Transformer encoder–decoder, decoder attention cross-attends to the encoder output. CoCa splits the decoder instead: its first half acts as a unimodal text encoder and its second half as a multimodal decoder. This lets the architecture learn contrastive alignment and captioning at the same time. The name CoCa comes directly from this combination of Contrastive learning and Captioning.

The key is not merely that dividing the decoder makes room for two loss terms, but that both can be optimized end to end in a single computational graph. CoCa was not the first attempt to combine contrastive and generative learning. Its significance lies in training from scratch with a comparatively simple architecture that fuses the two central pillars of vision–language pretraining.

Why Is a New Architecture Needed?

Introductions to vision–language pretraining papers often sound alike. Still, it is useful to identify the specific problem this paper sets out to solve.

Early deep-learning approaches to vision used single encoders such as convolutional neural networks and trained them for individual tasks. Familiar pretrained backbones—including ResNet, GoogLeNet, and EfficientNet—support representation transfer by fine-tuning task-specific heads for image understanding, such as classification, or video understanding, such as action recognition and grounding.

More recently, CLIP- and ALIGN-style work introduced vision–language pretraining on large collections of web image–text pairs. Their dual encoders learn with batch-level contrastive loss: a matching image and text form a positive pair, while mismatched examples form negative pairs. The image and text encoders learn their respective modalities independently, map them into a shared embedding space, and use contrastive loss to produce cross-modal alignment.

Where CLIP and ALIGN approach image–text alignment contrastively, methods such as SimVLM frame it as generation. The difference is where and how the modalities align. With dual-encoder contrastive learning, no explicit supervision is applied inside the process by which each encoder implicitly understands its modality. A generative model, by contrast, attends to an image—or an image plus a prompt—and generates descriptive text, so alignment occurs inside the decoder. The encoder output also provides keys and values to the decoder queries, creating alignment indirectly at the encoder–decoder boundary.

The central objective of both methods is to associate images with text, yet neither architecture is completely task-agnostic. Contrastive learning performs very well on understanding-oriented downstream tasks such as zero-shot classification, but may struggle with captioning or reasoning about individual image regions. A generative encoder–decoder cannot produce a text-only representation without an image because it lacks a unimodal text encoder. Its image-encoder output can also yield weaker representations when probed for understanding tasks.

CoCa therefore seeks to combine the single-encoder, dual-encoder, and encoder–decoder paradigms. It also unifies their training so that image–text relationships can be optimized end to end.

The figure shows three broad paradigms. The left is a vision-recognition or understanding task without multimodal input. The middle is contrastive vision–language learning such as CLIP and ALIGN. The right is the encoder–decoder structure used for generative learning. Personally, I think a single multimodal decoder in the rightmost diagram would match the paper's narrative more closely.

Is This the First Attempt?

No. Papers such as ALBEF had already proposed ways to solve multiple image–text tasks with one large network.

As the figure shows, however, ALBEF does not train a single batch in one unified process. Its encoder architecture also needs an additional decoder head for generative tasks. Finally, because it depends on representations from a pretrained model such as BERT, it is not a method that can be trained from scratch. It also requires distillation through a momentum module.

Approach

The paper's goal is to derive the supervision for single-encoder, dual-encoder, and encoder–decoder learning from natural language. In ordinary single-encoder classification, cross-entropy trains a scoring network by minimizing divergence from a human-annotated class vector:

$$\mathcal{L}_\text{Cls} = -p(y) \log q_\theta (x)$$

Here $p(y)$ is a one-hot class vector and $q_\theta(x)$ is the prediction of a scoring network parameterized by $\theta$. Training only against a class vector treats the label as a discrete index rather than natural language, preventing organic interaction with language.

Contrastive learning addresses this by using image and text encoders $\mathcal{I}$ and $\mathcal{T}$ to model image embeddings $x$ and text embeddings $y$ contrastively:

$$\mathcal{L}_\text{Con} = -\frac{1}{N} \left( \sum_i^N \log \frac{\exp (x_i^\top y_i / \sigma)}{\sum_{j=1}^N \exp(x_i^\top y_j/\sigma)} + \sum_i^N \log \frac{\exp (y_i^\top x_i / \sigma)}{\sum_{j=1}^N \exp(y_i^\top x_j/\sigma)} \right)$$

Captioning is relatively straightforward. An autoregressive language decoder observes only the causal prefix before time $t$, and training sums the negative log-likelihood of the conditional next-token probability. Assuming each generation step is conditionally factorized, the captioning loss optimizes the probability of producing the complete caption:

$$\mathcal{L}_\text{Cap} = -\sum_{t=1}^T \log P_\theta (y_t \vert y_{< t}, x)$$

Sequence-to-sequence training generally uses teacher forcing for efficiency: the ground-truth prefix, rather than the model's previous prediction, provides the condition.

Adding the contrastive and captioning losses gives an end-to-end objective for an encoder–decoder with a masked decoder divided into two parts:

$$\mathcal{L}_\text{CoCa} = \lambda_\text{Con} \cdot \mathcal{L}_\text{Con} + \lambda_\text{Cap} \cdot \mathcal{L}_\text{Cap}$$

Decoupled Text Decoder and CoCa Architecture

Training with both losses requires captioning and contrastive learning to occur together. Captioning optimizes a conditional likelihood, while contrastive learning operates on the unconditional representations of the two modalities—or, more precisely, on a joint model that treats each encoder's output as independently produced.

A conventional single decoder cannot expose appropriate representations for both objectives. CoCa therefore decouples the decoder into unimodal and multimodal portions. The unimodal decoder layers encode input text into latent vectors using the usual causal mask; these are the lower $n_\text{uni}$ layers. The upper $n_\text{multi}$ layers then cross-attend between that text representation and the image encoding. The paper divides the decoder evenly, so $n_\text{uni}=n_\text{multi}$.

A Transformer vision encoder usually has a token such as [CLS] that encodes information about the entire image; each output token can be viewed as an $n$-dimensional vector. To perform contrastive learning, the authors similarly append a [CLS] token at the end of the unimodal text decoder and calculate contrastive loss from the whole-sentence representation it produces.

Attention Pooling

For an understanding-oriented objective such as contrastive loss, summarizing each complete modality into a single pooled token is common and effective. Other downstream tasks—generation in the paper's discussion—do not merely summarize the entire image or text. They benefit from more visual tokens.

The architecture therefore defines one attention pooler for contrastive learning and another whose output is cross-attended by the multimodal text decoder. Each attention pooler uses the image encoder's outputs as keys and values and optimizes $n$ parameterized queries. The generative loss uses 256 queries, while the contrastive loss uses one.

CoCa for Downstream Tasks

A trained CoCa model supports many forms of zero-shot transfer over images and text: classification as in CLIP and ALIGN, image–text retrieval, video–text retrieval, and more.

For tasks that are not zero-shot, the image encoder can remain frozen. Because an attention pooler can learn queries appropriate to each task, training a task-specific attention pooler provides a way to reuse the encoder's representations in different forms.

For video action recognition, for example, each frame passes through the encoder. Attention pooling over those embeddings learns a query for the spatial or temporal information needed from the video, and the attended result feeds a softmax cross-entropy objective for action recognition. Because the pooler needs only one spatial or temporal query, computation remains modest.

Video–text retrieval is simpler still. It averages the embeddings of 16 frames and uses the encoded caption of each video as the target embedding.

Experiments

CoCa's advantage is a training method and network architecture that can be applied easily to many tasks. In the table, previous state-of-the-art results are highlighted in yellow; CoCa exceeds them broadly.

Even with its encoder frozen, CoCa remains competitive with earlier networks. Fine-tuning for each task pushes it beyond the previous state of the art across the table.

Because CoCa's image encoder follows the ViT architecture, the model variants are named accordingly. The table reports fine-tuned and zero-shot classification performance from the smallest to the largest parameter count.

The largest model's image–text retrieval results on Flickr and MSCOCO appear above. Image-to-text retrieval finds text matching a supplied image, while text-to-image retrieval finds an image matching a supplied text.

CoCa also performs very well zero-shot on the ImageNet variants used to test domain generalization. It beats the previous state-of-the-art BASIC model on every dataset and follows a similar trend even under relatively severe domain shift. The paper reports many additional results for video–text retrieval, captioning, and other tasks, but their shared conclusion is simply that the method performs strongly.

Ablations

The table summarizes the authors' ablations. LE denotes linear evaluation, ZS zero-shot evaluation, AE attention evaluation, and FT fine-tuning. The ablations mainly defend the design choices and are not essential to understanding the central idea, so I will end the review here.