6.1. LLaMA, Mistral, and Falcon Families

LLaMA (Large Language Model Meta AI) is a family of open, dense Transformer models. Unlike MoE models (like Mixtral), every parameter is used in each forward pass.

🧩 Key Engineering Features

1. Grouped-Query Attention (GQA): Normally, each attention head has its own query, key, and value projections. GQA reduces memory and compute by sharing key and value projections among groups of query heads.

   This allows faster inference with almost no accuracy loss — crucial for deployment on smaller hardware.

   Conceptually:

   • 32 query heads share 8 key/value groups → fewer KV caches, less memory.

   Mathematically:

   $$ \text{GQA: } Q = W_Q x, \quad K,V = W_{K,V}^{(g)} x $$

   where $g$ indexes shared groups.

2. RoPE (Rotary Positional Embeddings): Enables smooth generalization to longer context windows (>4K tokens).

3. Efficient Pretraining: Trained on 2 trillion tokens with high-quality, deduplicated datasets. LLaMA 3 introduced multi-turn instruction fine-tuning and better safety alignment, rivaling GPT-4-level quality for many tasks.

Result: LLaMA models became the “default foundation” for open research — used in Alpaca, Vicuna, Mistral-based Mixtral, and countless custom fine-tunes.

Mistral (2023) redefined what a small, open LLM can do.

It outperformed LLaMA-2 at similar or smaller sizes by combining architectural refinements and smarter training practices.

⚙️ Innovations in Mistral

1. Sliding Window Attention: Each token only attends to a fixed window of recent tokens instead of the entire sequence. This keeps context management efficient — great for streaming and long-text reasoning.

   Unlike naive truncation, Mistral uses overlapping windows, allowing information to “flow” across segments without abrupt breaks.

2. Better Tokenization (Byte-Pair Encoding Enhancements): Improved subword segmentation increases efficiency on multilingual and code data.

3. Data Curation: Uses highly filtered, instruction-tuned datasets (high ratio of clean English text and reasoning data).

4. Parallel Residuals: Residual connections and normalization are restructured to reduce vanishing gradients and improve throughput.

Result: Mistral-7B matches or beats LLaMA-13B performance — a 50% smaller model with equal reasoning ability and higher speed.

Mixtral (Mistral’s MoE variant):

• Combines 8 experts (like MoE) but activates only 2 per token (Top-2 routing).
• Gives the power of a 48B model at the cost of ~13B compute.

Falcon, developed by the Technology Innovation Institute (TII), focused on speed, efficiency, and high-throughput causal decoding.

It was trained entirely on web text + curated corpora, emphasizing open availability and reproducibility.

🦅 Key Features

1. Causal Decoder-Only Architecture: Like GPT — optimized purely for generation (autoregressive tasks).

2. FlashAttention: An attention kernel that computes softmax(QKᵀV) in chunks, directly in GPU memory — massively reducing memory overhead and improving latency.

   It enables faster training and inference on long sequences without precision loss.

3. Parallelized Tensor Operations: Designed for multi-GPU efficiency, making Falcon a favorite for open-source deployment.

4. Training Efficiency: Trained on 1 trillion tokens using 80% less compute compared to comparable closed-source LLMs.

Result: Falcon-40B and Falcon-180B achieved top rankings on Hugging Face’s Open LLM Leaderboard for months — proving that careful optimization beats massive scaling.

Closed models (like GPT-4) benefit from massive private datasets and proprietary scaling laws. Open models can’t match raw scale — but they compete through efficiency and accessibility.

Trade-offs include:

• Smaller model sizes → easier to fine-tune.
• Open weights → reproducibility and transparency.
• Faster inference → wider hardware compatibility.

The brilliance of LLaMA, Mistral, and Falcon lies in scaling intelligence sustainably. They optimize architecture, attention, and memory, not just parameter count.

Each group $g$ of query heads shares a single set of key and value projections. If there are $h$ total heads and $g$ groups, then

For token $i$, attention is computed only over a local window of $w$ tokens:

This reduces computational cost from $O(n^2)$ to $O(nw)$.

• “LLaMA and Mistral are just copies of GPT.” Not true — they use fundamentally different attention designs and optimizations.
• “Open models can’t match closed models.” They can, when engineered efficiently (Mixtral matches GPT-3.5 on many tasks).
• “FlashAttention changes the model’s behavior.” It doesn’t — it’s a computational optimization, not a learning modification.