Introduction

Large pretrained vision-language (VL) models such as CLIP and ALIGN have shown that representations can be transferred effectively to many downstream tasks. Their zero-shot and linear-probing performance in particular demonstrates that combining vision and language provides a training structure better suited to open-world settings. Traditional representation learning largely relied on discretized labels, such as one-hot encodings that map classes to labels. Vision-language pretraining (VLP), by contrast, studies how to place images and text in the same feature space with meaningful relationships. This work uses prompting to solve each downstream task; here, prompting means a “method for composing text.” Since this post focuses on prompt learning and repeatedly uses the concept of a prompt, let us begin with that rough definition.

As discussed in detail later, the paper identifies the difficulty of prompt engineering as a problem. Ordinary deep learning tunes hyperparameters to improve performance; similarly, in a VLP task, performance varies so dramatically with prompt design that prompting becomes an indispensable preprocessing step. Yet testing a virtually unlimited number of possible prompts merely to improve one task is extremely time-consuming. Even after training with the best-performing prompt, a new task requires starting prompt engineering again from scratch. Creating prompts to learn representations for a particular language also demands domain expertise.

The paper therefore adapts prompt-learning methods from NLP and proposes Context Optimization (CoOp), which optimizes the prompt itself. CoOp treats the prompt's context words as learnable vectors and freezes the VL model parameters while learning the optimal prompt for each downstream task. With fewer shots (training samples), CoOp matches hand-crafted prompts; after sufficient training, it improves downstream-task performance by roughly $15\%$.

Limitations of existing methods and appearance of VLP

As noted in my CLIP review, CLIP focused on web-based image datasets large enough to learn many representations, but I did not explain precisely why conventional training for the image modality was a poor method of representation learning. In classification research such as ResNet and Vision Transformer, there is a fixed number of object categories, and each category description becomes a discrete label through indexing. For example, CIFAR-10 uses

a dataset with 10 classes. Category information such as airplane, cat, and frog does not supervise training through the meanings of those words; instead, it is represented by one-hot encodings used to optimize cross-entropy loss. ImageNet contains about 1,000 classes, including the following:

281: 'tabby, tabby cat',
282: 'tiger cat',
283: 'Persian cat',
284: 'Siamese cat, Siamese',
285: 'Egyptian cat',

This supervision ignores linguistic and contextual similarity to entirely unrelated categories such as orange or mushroom and simply assigns different indices to different categories. The problem with these discretized labels can be illustrated as follows.

If we have a Siamese cat and a tiger cat, the two images belong to different categories, but—as their names show—they share the concept of a cat. More importantly, the objects have similar attributes: both are animals with fur and whiskers. The apple image below, by contrast, shares essentially no attributes with those two images.

Category labels in conventional classification cannot connect relationships expressed in text with image properties. To recognize the wide variety of objects encountered in an open-world setting, simply mapping a continuous image signal to a discrete one-hot signal is inappropriate. Moreover, as category descriptions become more detailed, the number of categories the network must classify increases, and nodes for objects other than the correct label inevitably become distractions. Conventional computer-vision deep learning treats visual recognition as a closed set; whenever data is added, a new classifier must be trained on both the old and new data. Readers interested in this problem may want to explore continual learning and domain generalization.

More recently, VLP models such as CLIP and ALIGN began pretraining on text embeddings instead of discretized labels, substantially improving zero-shot transfer over conventional vision representation learning. The training idea is simple: align the outputs of an image encoder and a text encoder. Both CLIP and ALIGN use contrastive loss as their objective. A positive pair—an image paired with text that describes it—is trained to move closer together in embedding space, while a negative pair is pushed farther apart. In this way, CLIP and ALIGN transfer knowledge from networks pretrained on massive datasets through prompting tailored to different tasks.

The task-specific prompting process appears on the left (CLIP framework) in the figures above. The object name of each class is formatted as “A photo of (Class)” or another task-specific prompt, and the text embedding produced by the text encoder is used as the classifier weight.

Limitations of prompt engineering in VLP

In VL tasks using pretrained networks, effective prompt engineering had a major influence on performance. Defining an appropriate prompt is difficult, however, making tuning extremely time-consuming; on some tasks, even a tiny prompt change dramatically affects performance. On Caltech101, for example, replacing “a photo of (class)” with “a photo of a (class)” improves performance by as much as $5\%$, revealing striking sensitivity.

Prompt engineering also requires prior knowledge of the task and domain knowledge about how language models operate.

Specific descriptions such as “texture” or “centered satellite photo” are difficult to use in prompt engineering without a solid understanding of the dataset or field. The largest problem, as noted above, is that we cannot know whether the engineered prompt is actually optimal, so we must settle for whatever performance gain it happens to provide.

Prompt engineering in VLP task

To address these problems, CoOp draws on recent prompt-learning research in NLP to find prompts optimized for downstream tasks. Conventional prompt selection resembles tuning and validating hyperparameters; CoOp instead automates the search for an appropriate prompt by treating its context words as learnable vectors. To cover varied tasks, the paper proposes two approaches. The first learns a single prompt for all classes through a unified context; the second learns a separate prompt for each class. The first performs better in most cases, though the second is more effective for some tasks with fine-grained categories.

Contributions

The paper's prompt-learning method treats text prompts as continuous signals rather than discretized descriptions (categories), as in conventional prompt engineering. To evaluate the effect of prompt optimization, the authors use 11 benchmark datasets spanning categories such as objects, scenes, and actions. They summarize their contributions as follows.

• Rather than addressing VLP networks directly, the paper timely studies their downstream applications and identifies the inefficiency of prompt engineering in existing VLP methods.
• To automate prompt engineering for pretrained VL models, it introduces continuous prompt learning in both unified and class-specific forms, making the method applicable to a wider range of recognition tasks.
• On downstream tasks, it optimizes transferred VLP representations more efficiently than hand-crafted prompts and linear probing, outperforms existing methods, and makes VL models more robust to domain shift.

Related works

Vision Language models

The motivation for prompt learning is to transfer and use pretrained representations from VLP tasks effectively. Recent systems such as CLIP and ALIGN, which contrastively train the outputs of text and image encoders, provide the standard architecture for image-text multimodality. Both use massive web datasets and perform contrastive learning with large minibatches. Research predating CLIP and ALIGN also sought to place text and images in the same embedding space, but its methods for extracting text embeddings (Word2Vec, TF-IDF, and so on) and matching them (metric learning, multi-label classification, n-gram language learning, and so on) differ from today's state-of-the-art contrastive representation learning.

The authors emphasize that their work takes a different direction from conventional vision-language-model research. Existing VLP tasks focus on aligning images and text in the same embedding space; this paper focuses on transferring knowledge that has already been pretrained. It argues that replacing hand-crafted prompt engineering with prompt learning should be effective.

What is prompt learning?

For knowledge probing in large pretrained language models (LLMs), researchers proposed cloze text—“fill in the blank”—which laid the groundwork for prompt learning in NLP.

In this fill-in-the-blank setting, the basic idea of probing is simply to make the model generate the correct answer for a given cloze-style prompt. How Can We Know What Language Models Know? proposes generating candidate prompts through text mining and selecting the prompt with the best training accuracy as optimal. AutoPrompt, as illustrated below, searches for the token that produces the largest gradient change in label likelihood—a gradient-based search—and uses it to generate the prompt.

This paper uses continuous prompts. One drawback is that, unlike searching over discrete tokens in a text embedding space, we cannot visualize exactly what prompt a learned word represents. The authors nevertheless choose continuous prompts because the goal of a VLP task is not to extract an interpretable prompt embedding, but to automate the tuning process for prompts that perform well when a VL model is applied to a downstream task.

Method

The method is very simple: replace conventional prompt engineering with learnable context vectors and optimize them. As the authors note, this is orthogonal to training the VLP model itself; in the CLIP baseline framework, it corresponds to the transition from step (2) to step (3). Context can be optimized because each downstream task provides supervision.

CLIP baseline

The network uses CLIP, which contains a vision encoder and a language encoder. The vision encoder may be a CNN baseline such as ResNet-50 or a Transformer baseline such as ViT; the language encoder is a Transformer. CLIP encodes text with BPE. To restate the training process briefly, batches contain image-text pairs. Training maximizes cosine similarity for matched pairs (positive pairs in contrastive learning) and minimizes it for mismatched pairs (negative pairs). To learn varied image and text representations, CLIP trains on 400 million paired web examples.

One of CLIP's main contributions is strong zero-shot inference. Because CLIP trains on many text prompts from the web, it can perform downstream tasks on classification datasets with varied categories. Let $f$ be the feature obtained by passing image $x$ through the image encoder, and $(w_i)_{i=1}^K$ the weights obtained by passing each class description $t_i$ through the text encoder. Here, $K$ is the number of classes in the downstream task. A prompt such as “a photo of a (class)” places a class name such as “cat” or “dog” in the marked position. The prediction then uses a softmax probability based on cosine similarity. Cosine similarity acts as a score: the greater the similarity, the higher the probability of that class.

$$p(y = i \vert x) = \frac{\exp (\cos (w_i, f) / \tau)}{\sum_{j=1}^K \exp(\cos (w_j, f)/\tau)}$$

$\tau$ is a temperature parameter learned during CLIP pretraining. Conventional classifiers learn discrete labels only for closed-set visual concepts—a task defined around distinguishing fixed classes. CLIP's ability to learn open-set visual concepts organically enables a high-capacity network.

Context Optimization

As the method above shows, a human must tune prompts such as “a photo of a (class)” by searching for one that performs well on each task. CoOp proposes two ways to automate this process.

Unified Context

This approach shares one context across all classes. The prompt supplied to text model $g(\cdot)$ can be defined as

$$t = (V)_1(V)_2 \cdots (V)_M (\text{CLASS})$$

Each $(V)_m$ denotes the embedding of a particular word (512 dimensions in CLIP). $M$, the number of word embeddings to use, is a hyperparameter. Passing prompt $t$ through the text encoder yields a classification weight vector for each class, and the prediction probability is computed with the same expression as above.

$$p(y = i \vert x) = \frac{\exp (\cos (g(t_i), f) / \tau)}{\sum_{j=1}^K \exp(\cos (g(t_j), f)/\tau)}$$

The optimal context structure might be “a photo of (class),” but it might instead be “a photo of (class), a type of object.” We can therefore define the learnable prompt as follows.

$$t = (V)\_1 \cdots (V)\_{\frac{M}{2}}(\text{CLASS})(V)\_{\frac{M}{2}+1} \cdots (V)\_M$$

The prompt can thus learn to fill later cells, or to stop filling them with a termination signal.

Class-specific context

The preceding approach learns the same context for every class. The alternative learns a context for each class. For different class indices $i$ and $j$,

$$(V)_1^i(V)_2^i \cdots (V)_M^i \neq (V)_1^j(V)_2^j \cdots (V)_M^j$$

it learns distinct contexts as shown above. This approach is reportedly effective when fine-grained classification is required.

Experiments

The authors experiment on 11 datasets: ImageNet, Caltech101, Oxford-Pets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, and UCF101.

The statistics of the datasets appear above. The hand-crafted prompt baseline uses the best-performing prompt identified through ablation.

The figure above reports CoOp's mean results over the 11 datasets. “end” optimizes a context with the CLASS description at the end, while “mid” places it in the middle. CSC means class-specific context. Overall, the unified prompt tends to perform better.

The unified prompt is not best in every case; on some datasets, CSC performs better as the sample count increases. I see the paper's main contribution in its few-shot result. When zero-shot CLIP is fine-tuned on different datasets through linear probing, fewer than four shots do not reliably outperform zero-shot inference. With prompt learning, however, few-shot performance tends to rise to or above zero-shot performance.

The graph makes the advantage clearer. Compared with zero-shot CLIP, a 16-shot CoOp network improves performance by as little as $1.24\%$ and as much as $45.97\%$. Food101 is an exception where performance declines, but CoOp improves 10 of the 11 datasets.

Training with CoOp is also more robust under the domain shifts shown above. On datasets such as ImageNet-R and ImageNet-Sketch, performance can improve by an even larger margin than the gains on the source domain.

The experiments use a context length of 16 and include an ablation of this choice, along with the effect of the vision backbone on CoOp's trends. Results follow a pattern similar to zero-shot CLIP and show that CoOp improves performance regardless of backbone architecture.

The appendix above shows the vectors most similar to the context learned for each dataset. Because continuous prompt learning does not make a learned vector correspond to a specific word, this only identifies indirectly which words resemble the learned prompt. My impression is that learning prompts continuously inevitably breaks the conventional structure of text. CoOp may be suitable for improving performance, but it has many limitations in revealing image-text relationships. I suspect this is why the authors describe their work as distinct from VLP in the related-work section.

Limitation in CoOp and appearance of CoCoOp

CoOp has a critical problem. Because its context overfits the downstream task during training, it performs well on in-domain classes but poorly on out-of-domain classes with a similar distribution.

For seen SUN397 categories such as “Arrival gate” and “Cathedral,” a learned prompt achieves higher accuracy than the zero-shot baseline. However,

on categories with a similar distribution from the perspective of scene understanding, such as “Wind farm” and “Trail railway,” performance is actually worse than the zero-shot baseline with hand-crafted prompts. In other words, an optimized text prompt overfits only the seen classes of a particular dataset.

This problem is hardly surprising and already appears in the appendix table at the end of the CoOp experiments. Visualizing nearest neighbors of learned prompts produces no sensible words or sentences, only descriptions largely disconnected from the image context. The paper varies the CLASS prompt's position—placing it in the middle or at the end—to test a flexible context, but these variations seem to contribute no intuition from the text domain.

Conditional Context Optimization

To solve this weak-generalization problem, CoCoOp uses an approach similar to image captioning. A meta-network lets the input image guide the prompt being learned, regularizing its tendency to overfit classes. Image-captioning research has likewise shown that instance-dependent optimization makes training more robust to class shift. The authors seem to conclude that optimizing a prompt solely under the influence of the text encoder weakens transfer of the image representation.

CoCoOp therefore trains a meta-network from the class token of an image encoder (ViT or ResNet), producing a conditioned meta-token for each image instance. If the context tokens ($v_1,v_2,\cdots,v_M$) optimize a prompt solely for one dataset, they can overfit. The context tokens therefore learn only a broadly generalizable prompt, while $\pi$ supplies the remaining image-specific information by conditioning the tokens through a lightweight meta-network. I think of learning the context tokens as preparing a well-designed canvas, and learning $\pi$ as assembling a palette with which to paint a good representation. CoOp inevitably overfits because it tries to create a plausible prompt from nothing. CoCoOp's conditioning regularizes the process by separating the roles of the text and image encoders.

The CoOp vs. CoCoOp results for the two examples appear above. The upper result shows roughly similar performance for class-specific prompts, while the lower result shows that CoCoOp is far better on unseen-class prompts. In cases where CoOp falls as much as $10\%$ below zero-shot performance, CoCoOp instead improves it.

Conditional Context Optimization (CoCoOp)

As the paper makes clear, the related work and method sections differ only slightly from the authors' original CoOp paper. Since they cite their own work directly, there is no real reason to object, though it made me think I would like to write a paper that way someday. The experimental setup also looks straightforward because the method essentially adds only a meta-network to the original work.

The remaining equations are therefore unchanged, and we need only examine the meta-token $\pi$ produced by the meta-network. The authors' simplest initial idea was to design $M$ neural networks, each trained separately for one of the $M$ context tokens. Under CoOp's settings, however, that would require training 16 networks. It would be too expensive and not parameter-efficient, so they construct a meta-network that generates one token added to all $M$ context vectors.

For Meta-Net $h_\theta(\cdot)$ with learnable parameters $\theta$, the context vector for input image embedding $f=E_I(x)$ is constructed as follows. Starting from the CoOp expression,

$$t = (V)_1(V)_2 \cdots (V)_M (\text{CLASS})$$

define a fixed number of learnable prompt vectors. The conditioned vector $V_m(f)=(V)_m+\pi$, which adds meta-token $\pi=h_\theta(f)$ to each vector, can then replace the prompt.

$$t(f) = (V_1(f),~V_2(f), \cdots ,~V_M(f), (\text{CLASS}))$$

The original prediction probability becomes

$$p(y = i \vert x) = \frac{\exp (\cos (g(t_i(f)), f) / \tau)}{\sum_{j=1}^K \exp(\cos (g(t_j(f)), f)/\tau)}$$

During training, the meta-network's parameters $\theta$ and the context vectors $V_m$ are updated together by gradients.

The meta-network is a very simple two-layer structure: a Linear–ReLU–Linear MLP. The authors leave more complex architectures to future work, though merely obtaining meaningful results with a more complicated network would probably not be a major contribution (there goes another possible paper topic).

Experiments

The experiments reuse the same 11 datasets as the original CoOp study. Their mean accuracies are shown below.

CoCoOp outperforms CoOp on both New (unseen classes) and H (Base + New). CLIP still has the best zero-shot performance on unseen classes, but its performance on the base dataset trails by $11\%$. This shows that CoCoOp can be applied to both seen and unseen classes.

The comparison between unseen and base classes for each dataset appears above. CoOp often performs better on base classes, but the authors attribute that advantage to overfitting, so it is not meaningful. The notable result is that CoCoOp outperforms every existing method on unseen classes.

Because the work emphasizes generalization, it also evaluates domain generalization. CoCoOp remains more robust than CoOp even as the gap between source and target domains grows.

Conclusion

Both CoOp and CoCoOp seek to improve CLIP downstream tasks through prompt learning. Of the prompt-learning techniques developed in NLP, CoOp adopts continuous prompt learning that can be applied directly to optimization and focuses on improving downstream performance. CoCoOp addresses the overfitting to seen classes overlooked by CoOp's performance gains and introduces a meta-network for generalization to unseen classes.

Their contribution lies in applying domain-generalization methods from NLP to VL networks and developing a line of work orthogonal to VLP itself. CoOp and CoCoOp are strong papers in performance terms, but they also left me thinking, “So this is how one writes a paper.”