5.1. Cross-Modal Alignment

CLIP (Contrastive Language–Image Pretraining), by OpenAI, was the first major success in aligning text and vision representations.

It’s trained on hundreds of millions of (image, caption) pairs from the web — for example:

🖼️ “A golden retriever running through grass.”

Two separate encoders are used:

• Image Encoder: A Vision Transformer (ViT) turns pixels into embeddings.
• Text Encoder: A Transformer converts captions into embeddings.

The goal: make embeddings of matching pairs close together and non-matching pairs far apart in the shared latent space.

Training Objective: Contrastive Loss

Here:

• $I$: image embedding
• $T$: text embedding
• $\text{sim}(I, T)$: cosine similarity
• $\tau$: temperature scaling

So the model doesn’t classify objects directly — it learns the concept of correspondence.

After training, you can:

• Zero-shot classify: “Find which label’s text best matches this image.”
• Retrieve: “Find the image that best matches this caption.”

Flamingo (DeepMind) extends CLIP-style learning to multimodal reasoning.

It combines:

• A pretrained vision encoder (like a frozen CLIP).
• A large language model (like Chinchilla or PaLM).
• Cross-attention layers that let the LLM “look” at visual features.

Instead of fusing everything early, Flamingo injects visual information midway through the LLM — so the model reads text, then attends to visual embeddings when needed.

Example:

“What color is the car in the picture?” The model uses cross-attention to attend to the region in the image embedding that encodes “red car.”

This flexible fusion allows few-shot multimodal learning — you can show it a few examples and it generalizes quickly.

PaLI (Pathways Language and Image) unified the text and vision pipeline even more tightly.

It introduced:

• A shared multimodal Transformer backbone.
• A single vocabulary for text and vision tokens.
• End-to-end training on captioning, VQA, and OCR tasks simultaneously.

Unlike Flamingo, PaLI doesn’t keep encoders frozen — both vision and text parts co-train, aligning deeply within a unified Transformer.

The result: models that could caption images, translate text, and perform reasoning across modalities in one architecture.

GPT-4o (“omni”) represents the culmination of multimodal alignment — a single Transformer that processes text, vision, and audio in the same attention stream.

Unlike earlier models that had separate encoders for each modality, GPT-4o uses a shared attention backbone. All modalities are projected into a common embedding space — effectively turning everything into “tokens.”

• Text → tokenized normally
• Images → projected patches → embedding tokens
• Audio → waveform chunks → embedding tokens

Then, the same Transformer layers process all these tokens jointly. This allows GPT-4o to “see and say” in real time — answering visual questions, describing sounds, and reasoning about scenes seamlessly.

Key Benefit:

• No more late fusion or task-specific adapters.
• Fully unified understanding and generation across modalities.

Different modalities (like vision or audio) can’t be fed directly into an LLM — they must first be projected into the model’s token embedding space.

For example:

• An image patch from a Vision Transformer → linear projection → same dimensionality as text tokens.
• Audio spectrogram chunks → temporal convolution → token embeddings.

Once projected, these embeddings behave like “words” — so the model can attend to them, combine them with text, and reason over both seamlessly.

This projection step is the “language bridge” between modalities.

Text and images don’t share structure — but they share semantics. The trick is to map both into a joint embedding space where “dog,” “🐕,” and an image of a dog all occupy nearby coordinates.

Contrastive pretraining (like CLIP) builds this alignment; cross-attention (like Flamingo and PaLI) lets the model use it; unified backbones (like GPT-4o) make it native.

Cross-modal alignment is the foundation for multimodal reasoning — the ability to ground language in perception.

It moves AI from symbolic understanding (“the word ‘apple’”) to embodied understanding (“the image, sound, and description of an apple mean the same thing”).

• $I_i, T_i$: image and text embeddings for the $i$-th pair
• $\text{sim}$: cosine similarity
• $\tau$: temperature parameter

• “CLIP classifies images.” No — it aligns them with text; classification is an emergent behavior.
• “GPT-4o just combines vision and text models.” It uses a single Transformer backbone with unified embeddings, not modular fusion.
• “Flamingo uses early fusion.” It uses cross-attention mid-fusion, where vision tokens enrich the LLM stream dynamically.