Why BLIP Was Proposed

Training that combines vision and language, as in CLIP, is called vision–language pretraining (VLP). Earlier VLP models tended to excel at either understanding tasks, where encodings transfer to downstream prediction, or generation tasks, where a decoder produces a modality, but not both.

Methods such as CLIP improved VLP by training on noisy image–text pairs collected from the web. “Noisy” means that mass crawling collects irrelevant or mismatched examples, such as images poorly related to their text prompts. This is a practical substitute for better supervision, not an ideal solution. The ideal would be billions of perfectly matched pairs, but constructing them is unrealistic.

The BLIP paper proposes a framework that learns understanding and generation together. Unifying them supports a broader range of vision–language tasks through multi-task learning and improves both. More importantly, the authors use the decoder from the generation task as a caption synthesizer and the encoder from the understanding task as a caption filter. Together they reduce noise in web image–text data and improve performance across tasks.

Limitations of Earlier VLP

The limitations can be viewed from model and data perspectives.

Model Perspective

Most VLP systems use either encoder-based architectures such as CLIP and VILLA, or Transformer-like encoder–decoders such as Unifying Vision and Language and Deep Learning Library. Encoder-only representations are hard to apply directly to text generation. Encoder–decoders focus on generating text and often underperform on cross-modal understanding tasks such as image–text retrieval.

Data Perspective

State-of-the-art systems including CLIP, ALBEF, and SimVLM train on massive web image–text datasets. More data generally improves performance, but simply scaling noisy captions unrelated to their images is not necessarily the best route. BLIP addresses both the architectural and data-quality limitations.

Contributions

Multimodal Mixture of Encoder–Decoder (MED)

MED supports more organic transfer learning than a simple encoder or encoder–decoder. It combines a unimodal encoder, an image-grounded text encoder, and an image-grounded text decoder. Three objectives train the structure:

• The unimodal encoders use contrastive learning between image and text embeddings.
• The image-grounded text encoder optimizes image–text matching.
• The decoder optimizes image-conditioned language modeling.

Captioning and Filtering (CapFilt)

CapFilt bootstraps noisy image–text pairs with a synthetic-caption captioner and an understanding-oriented filter. It removes bad examples from both original web captions and generated synthetic captions. Captioning-only and filtering-only variants underperform their combination, demonstrating that the two successfully bootstrap one another. Generating diverse captions also acts as augmentation.

With MED and CapFilt, BLIP improves image–text retrieval, captioning, visual question answering, and other tasks. Transferring the representation directly to video–language tasks even produces state-of-the-art zero-shot performance.

Related Work

Vision–Language Pretraining

VLP relies on web pairs because high-quality human-annotated text is difficult to obtain. Most web datasets receive only simple filtering, leaving substantial caption noise. Researchers tolerated that drawback because the benefits of scaling the dataset were larger. BLIP treats noisy web text as a compromise rather than an inevitability and proposes CapFilt to improve it.

Earlier systems also struggled to optimize one architecture for both understanding and generation. BLIP mixes structures by modality and claims to support diverse downstream tasks with a relatively simple unified pretraining process.

Knowledge Distillation

Knowledge distillation traditionally uses a high-performing teacher and a smaller student. For classification, the student imitates the teacher's class probabilities, often by minimizing KL divergence. Self-distillation uses teacher and student instances of the same architecture.

BLIP contains a related idea, but the student does not simply copy a teacher prediction. The captioner distills semantically rich synthetic captions; the filter edits that knowledge by discarding noisy captions. It is less like attending a live lecture and more like studying a carefully edited online course.

Data Augmentation

Computer-vision augmentation is a widely studied form of dataset regularization. Language augmentation is harder because grammar and context are domain-specific. Recent generative language models made synthesis a practical augmentation technique, usually for low-resource NLP. BLIP differs by applying a synthetic-caption dataset to VLP.

BLIP Method

BLIP provides an efficient way to learn VLP from noisy image pairs and uses CapFilt for its central contribution: bootstrapping the dataset.

Model Architecture

The architecture is a multimodal mixture of encoder–decoder (MED) with three roles.

For CLIP-style contrastive learning, unimodal encoders process images and text separately: a Vision Transformer for images and BERT for text. An image-grounded text encoder then acts as the understanding module used to filter meaningful captions. Its cross-attention layers incorporate image embeddings into the text encoder and represent the match between the two.

Finally, an image-grounded text decoder synthesizes captions from images for dataset bootstrapping.

Pretraining Objectives

Image–text contrastive loss (ITC) trains the unimodal vision and text encoders. Matching pairs should embed nearby; mismatched pairs should separate. This improves unimodal vision and language understanding in the familiar VLP contrastive-learning framework.

Image–text matching loss (ITM) trains the image-grounded text encoder as a binary classifier. Cross-attention combines semantic image features with text and predicts match or mismatch. Because ordinary binary cross-entropy does not distinguish easy from difficult negatives in the way a contrastive denominator can, BLIP uses hard-negative mining so confusing mismatches contribute more.

Language-modeling loss (LM) trains the image-grounded text decoder autoregressively with cross entropy. Unlike a masked-language-modeling objective, it learns to generate meaningful captions conditioned on visual information.

Four Transformer components could make the parameter count explode. BLIP therefore shares all parameters between the text encoder and decoder except self-attention layers. Encoder self-attention is bidirectional; decoder self-attention must be causal and cannot view future tokens. Their cross-attention and feed-forward layers are otherwise similar enough to share.

CapFilt

Because annotation is expensive, web-crawled pairs $(I_w,T_w)$ rarely have the quality of human pairs $(I_h,T_h)$. An alt-text string that does not describe its image moves training away from the ideal distribution. Earlier work accepted this noise in exchange for scale; CapFilt tries to correct it.

BLIP uses both human-annotated $(I_h,T_h)$ and web-crawled $(I_w,T_w)$ data. The clean human pairs fine-tune the filter and captioner. The fine-tuned decoder generates synthetic caption $T_s$ for each web image $I_w$. The fine-tuned encoder then grounds on $I_w$ and filters both original $T_w$ and synthetic $T_s$, removing either if it fails to match the image.

The resulting dataset combines clean human pairs $(I_h,T_h)$, retained web pairs $(I_w,T_w)$, and retained synthetic pairs $(I_w,T_s)$. A new network pretrains on this bootstrapped dataset. Repeating the process can improve data quality and therefore network performance.

Limitations and the Emergence of BLIP-2

BLIP still has important limitations. Parameter sharing simplifies an otherwise fully fine-tuned network, but the method remains end-to-end. Its ambition is to combine strong image and text encoders with a text decoder into a high-performing architecture for many tasks, so reducing the modality gap still requires a complicated training process.

BLIP-2 begins from this limitation.

Its main figure nearly explains the entire framework. Where BLIP optimizes four Transformer components, BLIP-2 aligns vision and language through two pretraining stages and a small bridge module.

Where Is VLP Research Heading?

Many attempts followed CLIP in jointly learning vision and language, but most introduced expensive training. VLP naturally tries to combine the best unimodal vision and NLP models. BLIP-2 seeks a generic, compute-efficient method that preserves pretrained vision and language models by keeping them frozen. They still guide the bridge's training, but their parameters do not change.

Large language models such as GPT-3 and BERT have powerful generation and zero-shot-transfer capabilities. Applying them to another modality risks catastrophic forgetting, so preserving their pretrained representation is essential.

Cross-modal alignment is the key. If text embeddings form a topological space $T$ and image embeddings a space $I$, the ideal is a near one-to-one mapping that also preserves neighborhoods: similar images should correspond to similar text. Multimodal alignment research tries to discover that relationship.

Freezing an LLM such as BERT or GPT-3 preserves its representation, but because it was not supervised on images, connecting the two spaces is difficult. Earlier methods such as Frozen and Flamingo depended largely on image-to-text generation loss, which the authors consider insufficient for optimizing alignment.

BLIP-2 inserts a Q-Former, or Querying Transformer, between a frozen image encoder and frozen LLM. It extracts from image embeddings the features most useful to the LLM. At first glance it resembles a complicated spatial transformer or ordinary Transformer encoder mapping one topology into another. A better interpretation is that it packages the cross-attention used throughout VLP into one reusable module, improving scalability.

Because neither the image encoder nor text model must be retrained, Q-Former inherits the efficiency and light weight of modular research while learning a meaningful vision–language relationship.

Contributions of BLIP-2

The full title, “Bootstrapping Language–Image Pre-training with Frozen Image Encoders and Large Language Models,” summarizes its contributions:

• BLIP-2 uses frozen pretrained image and text networks efficiently and reduces their modality gap with Q-Former. Training separates a representation-learning stage from a language-generation stage. Despite its simpler structure, it achieves state-of-the-art results across vision–language tasks.
• With capable LLMs such as OPT and Flan-T5, BLIP-2 performs zero-shot image-to-text generation. Stronger LLMs can directly improve visual reasoning and conversation through the same module, making the approach scalable.
• Training only a lightweight Q-Former between frozen unimodal models is compute-efficient. BLIP-2 exceeds Flamingo's zero-shot VQA performance by $8.7\%$ while using 54 times fewer trainable parameters.

Related Work

End-to-End Vision–Language Pretraining

The figure, from the survey Multimodal Learning with Transformers, shows recent multimodal architectures. Different tasks favor different structures. That freedom enables research but demands extensive ablation. Although many pretraining objectives have been proposed, image–text contrastive learning, image–text matching, and masked-language modeling dominate recent systems.

Most VLP methods train end to end on large web image–text datasets. Modern Transformers and LLMs improve on older CNN-based systems partly by spending far more parameters and computation, so the same bottleneck carries into VLP. Efficiently leveraging the strong representations already present in LLMs is difficult as well.

Modular Vision–Language Pretraining

Earlier systems primarily treated pretrained models as components to fine-tune—what the authors call “on-the-shelf.” BLIP-2 instead develops an off-the-shelf method centered on freezing pretrained networks.

Early work extracted visual features with a frozen object detector. More recently, LiT froze a pretrained image encoder for CLIP.

LiT found that freezing the image encoder and fine-tuning the text encoder performed best, suggesting that preserving a good visual representation helps. The inverse family freezes an LLM for vision-to-language tasks. Aligning image semantics with a frozen text model is difficult, so Frozen uses the image encoding as a soft prompt prepended to the LLM's text input.

The LLM stays fixed while the vision encoder learns with its guidance. Flamingo instead aligns at the feature-map level. It inserts cross-attention layers into a frozen LLM so text can attend to visual features, and trains only the added layers.

Both freeze the LLM. Frozen fine-tunes the image encoder for input-level alignment; Flamingo freezes the image encoder too and trains newly inserted cross-attention layers. Neither changes the core pretrained models, but fine-tuning or modifying their surrounding architecture remains computationally expensive. BLIP-2 aims to leverage both representations more efficiently.

BLIP-2 Method

BLIP-2 is simpler than BLIP. Instead of separately optimizing unimodal representation learning, grounded matching, and grounded text generation across a large network, it learns a bridge module.

The Q-Former bridges vision and language. Its training has two phases: vision–language representation learning with the image encoder, followed by generative learning with the LLM. The main remaining inconvenience is that these stages are sequential.

Q-Former contains learnable query tokens. Given text, they extract image features useful for alignment. BLIP used separate networks and objectives for image–text matching, contrastive learning, and grounded generation. BLIP-2 changes only attention masks and causality within Q-Former's encoder-like and decoder-like paths to support each objective.

The solid connections in the figure show bidirectional image–text matching and unidirectional image-grounded generation. Dashed or crossed connections separate the unimodal branches for contrastive learning. Matching can consult all semantic information. Generation is causal because it is autoregressive. Contrastive learning distinguishes positive pairs in white from negatives in gray for each query token.

Bootstrapping Vision–Language Representation from a Frozen Image Encoder

In stage one, Q-Former connects to a frozen image encoder and learns from image–text pairs. The goal is to train its query parameters so that they extract visual features meaningfully related to text. Inspired by BLIP, it optimizes ITC, ITM, and ITG objectives.

The paper does not deeply ablate the loss terms, which is unfortunate. The table shows a small gain from image-grounded text generation (ITG). I expected ITG to dominate because stage one must extract text-relevant visual features, yet ITC plus ITM already accounts for most of the result.

One omitted detail from the BLIP discussion is negative sampling. BLIP used a momentum queue; BLIP-2 draws positives and negatives within the current batch. The MoCo and SimCLR papers provide useful background.

Bootstrapping Vision-to-Language Generation from a Frozen LLM

After stage one aligns Q-Former with the frozen image encoder, stage two aligns Q-Former with the frozen LLM.

Q-Former is itself a lightweight Transformer and emits visual embeddings. Rather than feed them directly into the LLM, BLIP-2 projects them through a fully connected layer. The resulting soft visual prompt is prepended to the text input, resembling Frozen's input-level alignment, with Q-Former playing the role of the image encoder—plus the additional projection.

The authors test both decoder-only and encoder–decoder LLMs. With decoder-only OPT, the projected soft visual prompt conditions output-text generation directly. With encoder–decoder Flan-T5, the visual prompt is prepended to the original text prefix supplied to the encoder.

Results

BLIP-2 substantially outperforms the open-source BLIP baseline and shows gains of up to eight points over recent systems such as Flamingo and BEiT-3. Q-Former depends critically on staged representation alignment, as the comparison between joint and separated training shows.

Because stage two supports both decoder-only and encoder–decoder LLMs, Flan-T5 can perform prompted image question answering as shown below. The paper calls this “instructed zero-shot image-to-text generation” rather than VQA, although the practical distinction appears small.

The right-hand diagram shows the architecture used for VQA fine-tuning.

Conclusion and Discussion

BLIP and BLIP-2 explore increasingly efficient ways to combine vision and language models. Both were fascinating, and even where I did not understand every detail, they offered useful architectural insight.

Questions remain. BLIP-2 does not fully explain why the additional fully connected projection is necessary, and ITG appears to contribute less than expected. The authors also acknowledge that the model has not acquired robust few-shot in-context learning and that its frozen LLM lacks up-to-date information, creating failure cases in image-to-text generation.

Reusing a powerful web-trained model is a major advantage, but if an image requires knowledge that did not exist when the LLM was trained, the frozen language model may still fail to reason about it.