5.2. Multimodal Reasoning & ReAct Paradigm

Traditional LLMs generate a single, long text output — but ReAct models interleave reasoning traces with actions.

Example flow:

User: What’s happening in this picture?
Model (internal thought): I see a person wearing a lab coat and holding a flask.
Model (action): [use_vision_tool("describe_objects")]
Tool: Returns → {"objects": ["person", "flask", "lab coat"]}
Model (reasoning): The person is likely conducting an experiment.
Model (final answer): This image shows a scientist doing an experiment.

Each step alternates between:

1. 🧠 Reasoning — internal chain-of-thought (thinking, planning).
2. ⚙️ Acting — calling a tool or sensory module to gather more info.

This “reason–act–reason–act” cycle continues until the model reaches a confident answer.

Why it matters: It lets LLMs use tools, see images, hear audio, and query knowledge bases dynamically — rather than trying to do everything inside text generation.

MM-ReAct (Multimodal ReAct) extends this reasoning–action paradigm to include visual and auditory information.

Here’s how it works conceptually:

• Text, vision, and audio inputs are all converted into embeddings.
• The LLM performs multimodal reasoning, combining these signals.
• When it identifies a missing piece of information (e.g., “what color is that object?”), it invokes a modality-specific tool — like an image captioner, object detector, or OCR model.
• The retrieved result is fed back into the reasoning chain as new context.

Example:

Question: “What is the person doing in the photo?” 1️⃣ Reason: “I can’t infer action just from the face.” 2️⃣ Action: [use_pose_estimator(image)] 3️⃣ Result: “Detected hand raised, holding racket.” 4️⃣ Reason: “Likely playing tennis.” ✅ Final Answer: “The person is playing tennis.”

This dynamic reasoning allows the model to ask itself questions, gather new evidence, and update beliefs.

Chain-of-Thought (CoT) refers to step-by-step reasoning traces that LLMs use internally.

In Multimodal CoT, this reasoning is grounded in visual or sensory context.

Example:

“Is the stoplight green?” 1️⃣ Visual grounding: Focus on the region of the image containing the stoplight. 2️⃣ Reasoning: “The top bulb is off, the middle bulb is on, and it’s green.” 3️⃣ Answer: “Yes, the light is green.”

So the chain-of-thought doesn’t just rely on text tokens — it refers to visual regions, colors, or relations. This process builds explainable perception — we can see why the model concluded something, not just what it concluded.

Even large multimodal LLMs can’t do everything internally — they rely on external tools.

Examples of tools they might use:

• 🖼️ Vision API — for object detection, segmentation, or OCR.
• 🔊 Audio API — for transcription or sound classification.
• 🔍 Search API — for retrieving background knowledge.
• 🧮 Code Executor — for math or data reasoning.

When the model detects a cue that it lacks information (like “analyze this sound” or “plot these coordinates”), it generates a special token (e.g., <|call_tool|>) that triggers the right API.

Once the result is returned, it’s fed back into the reasoning trace — forming a closed cognitive loop.

Conceptually:

This decision process — when to keep thinking vs. when to act — is handled by policy mechanisms inside the model.

There are two common approaches:

1. Explicit Policy Head:

   • A small classifier predicts whether to act or reason next.
   • Example: policy = sigmoid(W_p * h_t) → decides between generating text or triggering a tool call.

2. Action Tokens:

   • The model learns special tokens like <reason>, <act>, <stop>.
   • During training, it observes patterns and learns when to produce each token based on context.

Over time, this teaches the model to self-regulate its reasoning depth — if visual cues suffice, it answers immediately; if ambiguity remains, it “acts” to seek more input.

This mechanism is crucial for multimodal autonomy — models that interact intelligently with the world.

Reasoning consumes compute, while acting consumes external resources (API calls). An efficient system balances both:

• Reason when internal knowledge suffices.
• Act when external evidence is needed.

This is analogous to human behavior — you don’t Google every question, but you also don’t rely only on memory when facts are uncertain.

ReAct turns LLMs from passive pattern matchers into active cognitive agents. They don’t just generate — they decide, query, perceive, and synthesize.

This paradigm underpins the design of autonomous multimodal systems like GPT-4o, Gemini 1.5, and Claude 3.5, which can see, reason, act, and explain — moving closer to embodied intelligence.

• $s_t$: current model state (context embeddings).
• $a_t$: next action — “reason,” “act,” or “stop.”
• $W_p$: learned policy weights.

• “ReAct is just chain-of-thought.” Not quite — CoT is reasoning-only; ReAct interleaves reasoning with real actions.
• “Multimodal means seeing and describing.” Modern multimodal LLMs reason and plan across vision, audio, and language.
• “Tool use is hardcoded.” It’s emergent — the model learns when and how to use tools during fine-tuning.