Why did we need SigLIP after CLIP?

CLIP aligns images and text in a shared embedding space. It learns to bring semantically matching image-text pairs closer and push unrelated pairs apart, making it possible to use a natural-language prompt like a classifier. The idea has become very familiar, but from a training perspective it carries one important condition:

To find the text corresponding to one image, the model needs relative scores against every other text in the same batch.

CLIP’s Softmax contrastive loss therefore cannot be computed from one pair in isolation. Every image-text similarity in the batch must be collected and normalized. When training across multiple devices, the embeddings held by each device must also be exchanged.

SigLIP asks a remarkably simple question at this point:

Must this be a Softmax problem that selects one correct answer among many candidates? Could we independently decide whether each image-text pair matches?

Released in 2025, SigLIP 2 broadens the question again:

Is strong global alignment between images and text enough to make a good vision encoder? What else is needed to understand patch-level positions, dense features, many languages, and varying aspect ratios?

The names of the two papers form a sequence, but the nature of their contributions differs substantially. SigLIP redesigns the loss and distributed training, while SigLIP 2 integrates several representation-learning techniques into one encoder-training recipe.

Revisiting CLIP’s Softmax contrastive loss

Let the batch size be $B$, and let the L2-normalized image and text embeddings be $x_i$ and $y_i$. CLIP computes every similarity to form a $B \times B$ matrix.

$$s_{ij}=t x_i^\top y_j,$$

where $t=\exp(t')$ is a learnable temperature scale. Diagonal entries $(x_i,y_i)$ are positive pairs; all others are negative pairs.

The CLIP loss averages cross-entropy in the image-to-text and text-to-image directions.

$$\mathcal{L}_{\text{CLIP}} = -\frac{1}{2B} \sum_{i=1}^{B} \left[ \log \frac{\exp(s_{ii})}{\sum_{j=1}^{B}\exp(s_{ij})} + \log \frac{\exp(s_{ii})}{\sum_{j=1}^{B}\exp(s_{ji})} \right].$$

The first term sees image $i$ and finds the correct answer among $B$ texts. The second sees text $i$ and finds the correct answer among $B$ images.

The intuition is clear, but the Softmax denominator depends on the full batch. If the batch is distributed across several devices, local embeddings alone cannot compute the global denominator. All-gather is needed to collect the full set of embeddings, as is materialization of the $B \times B$ similarity matrix.

Larger batches provide richer negative samples, but also increase the communication and memory required to compute the loss. The statistical advantages of contrastive learning become coupled to the cost of the distributed system.

SigLIP: Classify every pair instead of selecting one answer

SigLIP turns the problem of selecting one of $B$ classes into $B^2$ binary-classification problems.

$$z_{ij} = \begin{cases} +1, & i=j \\ -1, & i\neq j \end{cases}$$

The logit and Sigmoid loss are

$$\ell_{ij}=t x_i^\top y_j+b,$$ $$\mathcal{L}_{\text{SigLIP}} = -\frac{1}{B} \sum_{i=1}^{B} \sum_{j=1}^{B} \log \sigma\left(z_{ij}\ell_{ij}\right).$$

Logistic loss encourages $\ell_{ij}$ to grow for a positive pair and shrink for a negative pair. Softmax asks, “Which text matches this image best?” Sigmoid repeatedly asks, “Do this image and this text match?” for every combination.

At first glance, it may look as though one activation was simply changed from Softmax to Sigmoid. The important difference is that each $\ell_{ij}$ contributes to the loss independently of every other pair’s score. No normalization factor over the full row or column is needed.

The differences can be summarized as follows.

Are so many more negatives than positives a problem?

For batch size $B$, there are $B$ positive pairs but $B^2-B$ negative pairs. At $B=16{,}384$, for example, there are about 16 thousand positives and 268 million negatives.

If initial logits are near zero, every pair begins with match probability $0.5$. Compared with the true prior, this prediction is far too positive, and the vast number of negatives can dominate the loss and cause a large correction early in training.

SigLIP mitigates this by adding a learnable bias $b$, initialized as

$$t'=\log 10,\qquad t=10,\qquad b=-10.$$

Since $\sigma(-10)\approx 4.54\times 10^{-5}$, the initial match probability is very low. This is a natural initialization when the true positive ratio in a batch of 32k is approximately $1/B$. In the paper’s ablation, $b=-10$ consistently outperformed both no bias and $b=0$.

The bias affects the objective itself. Softmax probabilities across the candidates sum to one, so class imbalance is partly built into normalization. After reformulating the problem as independent binary classifications, we must directly choose the logit corresponding to the initial match prior. The learnable bias sets that value.

Why the Sigmoid loss simplifies distributed training

Across the full batch, SigLIP’s pairwise loss still computes $B^2$ pairs. It would therefore be wrong to conclude that “Sigmoid eliminates quadratic computation.” What changes is that the complete $B^2$ similarity matrix need not be held in memory and globally normalized at once.

Suppose there are $D$ devices and each has local batch size $b=B/D$. Each device performs the following steps:

1. Compute the $b \times b$ local loss from its own image and text embeddings.
2. Circulate its text-embedding chunk to the neighboring device.
3. Compute and accumulate the loss between the newly received text chunk and the local image embeddings.
4. Repeat until every image and text chunk has met once.
5. Sum only the final scalar losses across devices.

This method does not need to all-gather every embedding, and the similarity matrix materialized at any moment has size $b^2$ rather than $B^2$. It is possible precisely because the loss is independent for each pair.

SigLIP’s contribution therefore lies at the intersection of loss design and system design. The mathematics is simpler, and that simplicity changes both the communication pattern and peak memory.

Does performance keep improving as batch size reaches one million?

The Sigmoid loss lets the paper scale batch size as far as 1M. Its most interesting conclusion is not that “1M is good,” but that roughly 32k was already sufficient.

At batch sizes below 16k, Sigmoid loss clearly outperformed Softmax. Softmax caught up as the batch grew, but gains for both losses saturated rapidly around 32k. In the multilingual setting, XM3600 retrieval actually declined beyond 32k.

A larger batch sees more negatives per step, but if the total number of examples seen remains fixed, the optimizer takes fewer update steps. Continually adding easy negatives also has limited value. Rather than supporting the conventional wisdom that “contrastive learning always benefits from enormous batches,” the result suggests that the balance between sufficiently diverse hard negatives and sufficiently many optimizer updates matters more.

The paper distinguishes two terms:

• SigLIP jointly trains the image encoder and text encoder.
• SigLiT, like LiT, freezes a pretrained image encoder and trains the text tower.

The abstract highlights an ImageNet zero-shot result of 84.5% achieved in two days on four TPUv4s. This is not from-scratch SigLIP, but SigLiT with a frozen ViT-g/14 image encoder. By contrast, SigLIP jointly training randomly initialized B/16 image and text encoders reached 72.1% after two days on 32 TPUv4s and 73.4% after five days.

Both settings demonstrate the efficiency of Sigmoid loss, but cost comparisons must not conflate training with and without a frozen image backbone.

SigLIP’s performance and actual contributions

In the scale-up results toward the end of the paper, SigLIP B/16 at resolution 224 reaches 76.2% ImageNet zero-shot accuracy and COCO retrieval Recall@1 of 64.4% image-to-text and 47.2% text-to-image. OpenCLIP B in the same table achieves 70.2%, 59.4%, and 42.3%, respectively. The shape-optimized 400M model reaches 83.2% ImageNet zero-shot accuracy.

These numbers should not be attributed solely to the loss. Training data, examples seen, optimizer, architecture, and resolution also differ. The more persuasive evidence in the paper is the batch-size ablation comparing Softmax and Sigmoid under the same setting.

In my view, SigLIP contributes three main ideas:

1. It reformulates image-text contrastive learning from global categorical classification into independent pairwise classification.
2. This removes global normalization and full-similarity-matrix materialization from loss computation.
3. It separates batch size and the negative-to-positive ratio from the definition of the loss, enabling experiments such as 1M batches and negative masking.

More important than gaining a few percentage points is the finding that global Softmax, once considered essential to CLIP-style training, is not actually a requirement.

Is SigLIP alone enough to produce a good vision encoder?

SigLIP is strong at aligning pooled image and text representations. It is well suited to tasks asking about the meaning of an entire image, such as image retrieval and zero-shot classification.

Similar pooled embeddings do not guarantee, however, that each patch represents what appears at each location. Global similarity alone struggles with tasks such as

• finding the bounding box of “the person holding a red umbrella” in referring-expression comprehension,
• predicting a class for every pixel in semantic segmentation,
• estimating per-patch depth or surface normals in dense prediction,
• preserving OCR information in tall documents or wide screens,
• and retrieving images with languages other than English.

SigLIP 2 was proposed to address these limitations. It is more accurately understood as a recipe paper integrating ideas from LocCa, SILC, TIPS, the DINO family, active data curation, and NaViT/FlexiViT on top of SigLIP than as a study introducing one new loss.

SigLIP 2’s basic architecture and data

The fixed-resolution SigLIP 2 model retains the same dual-encoder architecture as SigLIP. The image and text towers use ViT structures, and a MAP (Multihead Attention Pooling) head extracts the pooled representation. Apart from tokenizer and vocabulary differences, a standard SigLIP 2 checkpoint can therefore replace SigLIP comparatively easily.

Notable aspects of the training setup include

• 10B images and 12B alt texts from WebLI,
• data in 109 languages,
• a mixture containing 90% pairs from English web pages and 10% from non-English pages,
• text length 64 and a multilingual Gemma tokenizer with a 256k vocabulary,
• batch size 32k and a total of 40B training examples,
• and model releases at ViT-B (86M), L (303M), So400m (400M), and g (1B) scales.

The original SigLIP concluded that “32k is enough,” and SigLIP 2 uses exactly that batch size. The follow-up adopts a sensible point from the earlier extreme-batch experiment as its practical recipe.

SigLIP 2’s training recipe

The center of the figure is SigLIP’s existing image-text alignment; the left and right show training signals added by SigLIP 2. It preserves the existing structure while introducing three kinds of signal.

1. Jointly train the Sigmoid loss and LocCa decoder

The first stage applies the existing Sigmoid image-text loss and LocCa decoder loss with equal weight. Sigmoid loss aligns the image encoder’s pooled output with the text embedding, while the pre-pooling patch sequence passes into an autoregressive decoder with cross-attention.

The decoder learns three targets:

• Captioning generates text describing the full image.
• Referring-expression prediction estimates bounding-box coordinates from text describing a particular region.
• Grounded captioning generates a caption for a region given its bounding box.

The global Sigmoid loss learns “which sentence is close to the full image?” The decoder loss injects “which region corresponds to which text?” into the patch representation. This explains the large improvement in localization performance.

Importantly, this decoder is a training-only module for representation learning. It is omitted from released checkpoints. Training supplies richer supervision while inference retains the efficiency of the original dual encoder.

2. Add self-distillation and masked prediction for the final 20%

Caption and bounding-box supervision alone do not improve every local feature. Once 80% of training is complete, SigLIP 2 introduces an EMA teacher and performs image-only self-supervised learning.

The first objective is local-to-global self-distillation:

• The teacher sees a global view of the original image.
• The student sees eight local views cropped from the same image.
• Training makes the student representations match the teacher’s global representation.
• Teacher weights are updated as an EMA of student weights.

The second objective is masked prediction:

• Replace 50% of the student image patches with mask tokens.
• The teacher sees the same global view without masking.
• Train student patch features at masked positions to follow the teacher patch features.

This recipe is familiar from DINO/iBOT-family methods. Sigmoid loss handles pooled semantic alignment; self-distillation and masked prediction fill unpooled patches with local semantics.

To prevent augmentation from interfering with image-text alignment, the paper uses the original image for SigLIP/LocCa losses and separate augmented views only for the self-supervised losses. It avoids forcing every objective onto one input where their training signals would conflict.

3. Distill into small models through active data curation

Small fixed-resolution models such as ViT-B/16 and B/32 receive additional fine-tuning with ACID active data curation. This is not conventional knowledge distillation in which a student directly imitates teacher logits.

The teacher and current learner assess the “learnability” of each example, selecting the most valuable 32k examples from a 64k super-batch. A strong teacher transfers knowledge through which data it chooses to show the student, rather than through labels or logits. The paper describes this as “distillation through data.”

This stage is not shared by every model; it supplements the performance of smaller models. Direct comparisons between the gains of B-sized SigLIP 2 and larger models must therefore account for active curation.

4. NaFlex does not distort images into squares

A conventional ViT resizes every image to a fixed square such as $224\times224$ or $384\times384$. This may not be a serious issue for natural images, but it distorts documents, screens, and charts where aspect ratio itself carries information.

NaFlex combines NaViT’s native aspect ratio with FlexiViT’s variable sequence length.

1. Preserve the original aspect ratio as much as possible while resizing height and width to multiples of the patch size.
2. Pass patch coordinates and a padding mask together.
3. Interpolate the learned positional embedding to the target non-square patch grid.
4. Train one checkpoint to process sequence lengths $\{128,256,576,784,1024\}$.

The model can spend more tokens on long documents and fewer on simple images, making it easy to trade inference cost against performance.

NaFlex is not always better than the standard model. A fixed-square variant can perform better on natural-image benchmarks, and extrapolation beyond sequence lengths seen during training was weak. To reduce implementation complexity, NaFlex training also omits the self-distillation and masked prediction described above.

NaFlex is therefore not an “unconditionally better SigLIP 2,” but a separate trade-off for applications where documents, OCR, and variable resolution matter.

How much does SigLIP 2 improve on core tasks?

Under the same B/16, resolution-256 conditions, the paper reports the following SigLIP and SigLIP 2 results.

English-centric classification and retrieval improve, but the largest differences appear in multilingual retrieval. The original SigLIP was strongest with English-centric training and needed a separate mSigLIP for multilinguality. SigLIP 2 substantially closes the gap between the two domains with a single model.

It is interesting that 90% of the data comes from English pages and only 10% from non-English pages. Rather than balancing every language equally, the recipe finds a mixture that adds multilingual coverage while preserving English-task performance.

The gap is even larger for dense features and localization

SigLIP 2 aims at more than a one- or two-point gain in zero-shot accuracy. Its more important objective is improving the patch representations of an encoder that previously excelled mainly at global pooled features.

With So400m/14 at resolution 224, SigLIP 2 raises PASCAL segmentation mIoU from SigLIP’s 72.0 to 77.1, and ADE20k from 37.6 to 41.8. NYUv2 depth RMSE falls from 0.576 to 0.493. Since these results freeze the image encoder and attach probes, they indicate that the pretrained representation itself contains denser information, rather than merely benefiting from better downstream fine-tuning.

The difference is larger in localization. For an L-sized model with sequence length 256, RefCOCO validation Acc@0.5 rises from 67.33 with SigLIP to 86.04 with SigLIP 2. The LocCa decoder’s joint training on captions, bounding boxes, and region captions likely contributes more than the self-supervised loss alone.

Interestingly, LocCa—which uses the same decoder-based loss but focuses on English captions—still outperformed SigLIP 2 on some referring-expression benchmarks. Multilingual generality and the best performance on a particular English localization task do not always move in the same direction.

Is it also a better vision encoder for VLMs?

Today many people encounter SigLIP not as a standalone zero-shot classifier, but as the vision encoder in a VLM such as LLaVA or PaliGemma. The vision encoder converts an image into a token sequence, and a projector connects it to the LLM’s embedding space.

The SigLIP 2 paper freezes SigLIP, SigLIP 2, and AIMv2 encoders, connects each to a Gemma 2 LLM, and compares them under PaliGemma-like conditions. SigLIP 2 outperforms SigLIP on average across model sizes and resolutions.

This finding also relates to better dense features. A VLM receives image patch tokens—not one pooled vector—to perform OCR, counting, and spatial reasoning. Improving patch representations can improve segmentation and the quality of the visual tokens delivered to the LLM.

SigLIP 2 is therefore both an alternative to CLIP for zero-shot tasks and something closer to a general-purpose visual tokenizer for VLMs.

Multilinguality and fairness are not the same problem

Along with a multilingual data mixture, SigLIP 2 applies a debiasing filter to its training data. The paper evaluates cultural diversity and fairness with separate benchmarks.

For example, with L/16 at resolution 256, a representation-bias metric that associates random objects with a particular gender falls from about 35.5% for SigLIP to 7.3% for SigLIP 2. Several metrics also improve in geographic-diversity evaluations using Dollar Street and GeoDE.

This should not be interpreted as a general guarantee that “SigLIP 2 is fair.” The measurements use particular sensitive attributes, prompts, and datasets; other cultures or downstream applications may reveal new biases. Even the paper finds income-level and geographic-region slices with small or negligible improvement.

Having a multilingual vocabulary, including data from many regions, and reducing social bias are related but distinct problems. It is valuable that SigLIP 2 addresses all three, but each still needs independent evaluation.

Summary of the differences between SigLIP and SigLIP 2

SigLIP 2 does not discard the SigLIP loss. It keeps Sigmoid loss as a global semantic anchor and supplements the local semantics that this loss teaches poorly with other objectives.

Limitations and cautions in interpretation

Sigmoid does not solve false negatives

The assumption that every off-diagonal batch pair is negative remains. Two different captions may express the same meaning, or different images may contain the same object, producing false negatives. Sigmoid loss shows greater robustness to label noise in experiments, but the underlying problem does not disappear.

Distinguish memory efficiency from computational complexity

A chunked implementation reduces the peak memory of the full $B\times B$ matrix and eliminates all-gather. If it computes every pair, however, the total number of similarity operations remains quadratic in batch size. The fact that a 1M batch is possible does not mean that a 1M batch is economical.

SigLIP 2’s gains do not come from one loss

SigLIP 2 changes the data mixture, tokenizer, caption/box decoder, self-supervised learning, active curation, number of training examples, and compute at the same time. We therefore cannot say, “SigLIP 2 is better than SigLIP, so one particular objective is the answer.” Rather than isolate and prove one new principle, the paper aims to combine several validated components into a strong open-weight encoder.

Training is much heavier than in the original work

SigLIP emphasized an efficient loss that could train on relatively few devices. SigLIP 2 trains on 40B examples using as many as 2,048 TPUv5e chips. Removing the decoder after training keeps inference simple, but the pretraining cost required to obtain the representation is not lightweight.

Choose a NaFlex checkpoint according to the goal

NaFlex is attractive when native aspect ratio and a variable token budget matter. If natural-image classification is the main objective, a fixed-resolution checkpoint benefiting from both self-distillation and active curation may be preferable.

Conclusion

I consider SigLIP an especially good kind of research. It does not overturn the existing framework; it simply shifts the loss perspective from categorical classification to pairwise binary classification. Yet that change simultaneously affects the equations, memory, device communication, and freedom to experiment with batch size.

SigLIP 2’s appeal, by contrast, lies less in one new idea than in arranging well-known ideas so they do not interfere with one another.

• Sigmoid loss aligns global image-text semantics.
• The decoder injects relationships between captions and regions into patch features.
• Self-distillation connects local and global views.
• Masked prediction reconstructs the semantics of hidden patches.
• Active curation lets the teacher choose which data a small model sees.
• The multilingual mixture and debiasing broaden representation coverage.
• NaFlex preserves real image aspect ratios and the inference budget.

The two papers can therefore each be summarized in one sentence:

SigLIP makes language-image pretraining simpler and more scalable by removing the global normalization required by CLIP’s loss.

SigLIP 2 extends that efficient global alignment into a general-purpose vision encoder by adding localization, dense features, multilinguality, and flexible resolution.

The papers and public checkpoints are available in the Google DeepMind big_vision repository.