Introduction

There are actually two MaskCLIP papers to review: one from ECCV 2022 and another from CVPR 2023. Their titles differ, but using the same name for the central method can be confusing. The paper I originally wanted to read was Extract Free Dense Labels from CLIP, published at ECCV 2022. The second is MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining, published at CVPR 2023. The former investigates how CLIP's zero-shot representations can be used for segmentation. The latter applies masked modeling, a generative approach to self-supervised learning, to improve CLIP's image representations.

What Makes CLIP Useful?

CLIP uses vision-language contrastive learning to discover relationships between images and text. It opened a path to solving downstream tasks—including classification, object detection, and segmentation—on many datasets in zero-shot or few-shot settings.

Outside of large language models such as GPT, CLIP was among the first computer-vision systems to train on a web-scale dataset built around text prompts. More than solving downstream tasks, its connection between language and vision enabled prompt-based deep learning applications such as image captioning and view synthesis. In that sense, it helped open a new era of multimodal learning.

Can We Use It for Segmentation?

The ECCV 2022 paper Extract Free Dense Labels from CLIP focuses on precisely this advantage: how can CLIP's zero-shot performance be extended to segmentation?

Suppose we need to segment the image above. The Joker and Batman have to be labeled as separate instances. We could manually label each character and train with that supervision, but annotating every frame and every character in every film is expensive both during dataset construction and training.

Conventional state-of-the-art segmentation methods commonly initialize representations with ImageNet-pretrained weights. This paper instead uses CLIP's global image representation, learned at web-image scale. Compared with an ImageNet-based representation, CLIP offers three advantages:

1. It can learn local image semantics, with individual feature dimensions describing parts of an image. This is broadly similar to the benefit of ImageNet-pretrained baseline weights.
2. It learns open-vocabulary concepts, making it possible to segment a desired object without a dedicated label for every class.
3. It learns rich contextual information about object interactions, relationships, and spatial locations.

If these properties can supervise a segmentation model, we can expect much more efficient task performance.

Failure and Success

The authors did not succeed immediately. The most obvious idea is to initialize a segmentation backbone with CLIP's image-encoder weights and fine-tune it. DeepLab models, for example, were strong segmentation baselines, so one might initialize DeepLab from CLIP and adapt the backbone. This avoids using CLIP's text embeddings, but it also completely discards one of CLIP's central advantages: segmenting unseen classes.

MaskCLIP takes another route. It uses patch-level features from the CLIP image encoder and performs dense prediction without the final attention-pooling layer. Text embeddings for each class become the weights of a $1\times1$ convolution. This is not an arbitrary trick: classification by image-text similarity computes a correlation between the text and image embeddings, while a $1\times1$ convolution likewise performs an inner product across feature dimensions.

Because it keeps the CLIP baseline intact, MaskCLIP preserves strong zero-shot behavior and works with both ResNet and ViT. Attention pooling is structurally equivalent to self-attention by ViT's image class token. The authors also introduce two training-free refinements, key smoothing and prompt denoising.

The MaskCLIP Approach

We will discuss MaskCLIP+ later; first consider the dark-gray MaskCLIP section. MaskCLIP requires no training. It creates text prompts for the desired class labels, encodes each prompt, and uses the resulting vectors as $1\times1$ convolution weights to turn the image embedding into dense predictions. Conventional attention pooling treats the average image feature as a class token and attends over all pixels. MaskCLIP instead retains every pixel feature before attention pooling.

The raw result is not especially good. Because prediction depends on similarity to a text prompt, it lacks the locality prior common in segmentation—that nearby pixels are likely to share a class. Patch-level pixel predictions can also be noisy. Key smoothing and prompt denoising address these problems.

Key Smoothing

If attention pooling is removed, its keys and values no longer participate directly in prediction. Yet attention influenced how CLIP learned its image representation. Its keys encode relationships between pixels and thus contain information about which regions a local semantic feature should consult. In the figure, $K_1$ belongs to grass while $K_2$ and $K_3$ belong to the cat. Similarity should be low between $K_1$ and $K_2$, but high between $K_2$ and $K_3$.

$$\text{pred}_i = \sum\_j \cos \left( \frac{k_i}{\parallel k_i \parallel_2},~\frac{k_j}{\parallel k_j \parallel_2} \right) \text{pred}_j$$

Key smoothing uses this property. The prediction for patch $i$ incorporates its key similarity to patches at other locations. The paper prints $i$ as the index of the prediction multiplied by the similarity, but the surrounding explanation appears to imply $j$, so I have written the equation accordingly.

Prompt Denoising

Another problem appears when an image contains many candidate classes. If the task distinguishes “cat,” “grass,” and “tree stump,” the stump occupies only a tiny area. During key smoothing, it can interfere with confidence for the other classes. The paper therefore removes a class from prediction when its key similarity is below $0.5$ for every feature pixel, improving performance.

From MaskCLIP to MaskCLIP+

The paper goes further and introduces MaskCLIP+. MaskCLIP enables zero-shot segmentation with CLIP, but that also means it cannot directly use segmentation-specific architectures such as DeepLab or PSPNet. In particular, designs such as ASPP are unavailable, limiting how much the CLIP-based architecture can specialize for segmentation.

MaskCLIP+ uses the frozen CLIP network to create pseudo-labels for another network. Its backbone starts from an ImageNet-pretrained model, just as in conventional fine-tuning. The difference is that its classifier is not trained; it uses the $1\times1$ convolution generated from CLIP text prompts. This demonstrates knowledge distillation from CLIP space into a non-CLIP network.

Continually supervising the segmentation backbone with pseudo-labels would cap its performance at the CLIP architecture's upper bound. The authors therefore use CLIP pseudo-labels for roughly the first tenth of the schedule, then remove them once performance begins to converge and continue with self-training.

The result is remarkably strong segmentation performance. Instead of ordinary fine-tuning, CLIP-guided text representations make zero-shot segmentation possible even with architectures designed for supervised segmentation.

Fully supervised mIoU can be regarded as the practical upper limit for zero-shot performance. MaskCLIP+ surpasses the previous state of the art and approaches a level where further gains become difficult.

CLIP's Limitation

CLIP, impressive as it is, still has limitations. The ECCV MaskCLIP and the CVPR MaskCLIP approach them from different directions. The ECCV paper emphasizes that CLIP's text-prompt representation contains enough context to support segmentation. The CVPR paper argues that CLIP's training objective alone is insufficient to learn a complete image representation.

Suppose the image contains cats with the caption “Two cats are on the grass.” The text describes the objects and their relationship, but provides little information about background details or textures unrelated to those objects. Language guidance can learn image-text relationships at scale, but visualization shows that regions mentioned by the description enter the representation more strongly than regions the text ignores.

Learning Image Representations with Self-Supervised Learning

The proposed solution adds self-supervised learning to CLIP: contrastive methods such as SimCLR and MoCo, or generative approaches such as MAE, BEiT, and DINO, can supply supervision from the image itself. Earlier work such as SLIP had already combined SSL with CLIP, but contrastive variants or methods focused on salient objects did not fill the missing contextual information around those objects.

BEiT and MAE learn image representations through self-supervision at token or patch level. CLIP, by contrast, aligns high-level image and text embeddings. Their target representations therefore live at different levels, making a generative objective awkward to attach directly. The paper instead uses feature-level rather than pixel-level supervision and adopts CLIP itself, which learns global representations, as a self-teacher for patch-level knowledge distillation.

The figure summarizes the ablations, with the final MaskCLIP design on the far right. Its contributions are threefold:

1. It proposes a new vision-language contrastive framework whose masked self-distillation improves transfer performance.
2. Rather than merely attaching one SSL method to the visual model, it performs ablations over several MaskCLIP variants.
3. The self-distillation architecture performs well in zero-shot, linear-probing, and fine-tuning settings.

Results

MaskCLIP substantially outperforms the original CLIP. The paper particularly emphasizes faster convergence from the additional loss on image representations. My interpretation is that image space carries a more complex signal and learns more slowly than the comparatively simple text-embedding space; extra visual supervision helps narrow that convergence gap.

As the paper claims, the regions attended to for a given text prompt become visibly more reasonable.