6.2. Fine-Tuning & Evaluation of Open Models

Full fine-tuning (updating all model weights) is computationally expensive — often requiring hundreds of GBs of GPU memory. Parameter-Efficient Fine-Tuning (PEFT) solves this by freezing most of the model and only training a small number of new parameters.

The most common PEFT method: LoRA (Low-Rank Adaptation).

🧩 How LoRA Works

LoRA inserts small trainable matrices ($A$ and $B$) into each linear layer of the Transformer:

• $W$: original pretrained weight matrix (frozen)
• $A$ and $B$: small low-rank adapters (trainable)

During fine-tuning:

• Only $A$ and $B$ are updated.
• The main model remains untouched.
• The resulting fine-tune is saved as delta weights (tiny patch files).

Benefits:

• 100x smaller training footprint.
• Easy to merge or remove.
• Compatible with multiple base models.

Axolotl is the go-to open-source framework for fine-tuning LLaMA, Mistral, or Falcon.

It simplifies the entire process:

1. Dataset Loading: Supports text, JSON, Alpaca, and ShareGPT-style data.
2. Training Setup: PEFT, QLoRA (quantized LoRA), or full fine-tuning.
3. Evaluation: Integrated validation and logging with Weights & Biases.
4. Export: Generates Hugging Face-compatible adapters or merged models.

Why It’s Popular:

• Works across architectures (decoder-only, encoder–decoder).
• Hugely configurable via YAML — no deep code needed.
• Supports instruction tuning, chat formatting, and multi-turn conversations.

In essence, Axolotl is like a fine-tuning autopilot for open LLMs — plug in your dataset, define your adapter type, and fly. ✈️

After fine-tuning, inference (serving the model) becomes critical.

vLLM is an optimized inference framework that:

• Uses PagedAttention — a GPU memory manager for large attention caches.
• Allows continuous batching, where new requests join mid-inference.
• Supports quantized and LoRA-adapted models.

It dramatically reduces latency, especially for chatbots or APIs handling many concurrent users.

In short: Axolotl fine-tunes → vLLM serves.

Llama.cpp is a lightweight C++ runtime for running quantized models on CPUs, mobile devices, or edge hardware.

It supports:

• 4-bit, 5-bit, and 8-bit quantized versions of LLaMA, Mistral, and Falcon.
• GGUF (compressed model format).
• Multi-threaded inference and GPU acceleration via Metal, Vulkan, or CUDA.

Its magic lies in quantization, which shrinks models without retraining.

Example: A 13B model → quantized → fits in 8 GB of RAM and runs on a MacBook.

Result: Anyone can experiment, deploy, or test models without a data center. 🌍

Quantization reduces model precision — say, from 16-bit to 4-bit — to make it smaller and faster. However, this compression can distort the model’s internal “reasoning pathways.”

To preserve accuracy, we apply quantization-aware fine-tuning, which:

• Emulates low-precision during training.
• Lets the model “adapt” to quantized arithmetic.

This process trains the model to survive aggressive compression while maintaining logic fidelity.

Trade-off: QAFT can retain factual recall and task performance, but deep reasoning (multi-step CoT) often degrades first — because these pathways depend on fine-grained activation patterns.

Model merging fuses multiple fine-tuned models into one — using delta weight arithmetic.

If model A is good at reasoning and model B is good at coding:

This creates a new model that blends both skill sets — without retraining from scratch.

Popular approaches:

• Delta weight addition (as above).
• Layer-wise merge (weighted by task importance).
• Feature-space merge (via embeddings).

Benefit: Fast cross-domain specialization — “combine the brains, skip the training.” 🧩

Once fine-tuned, evaluation is vital. Three major open benchmarks dominate:

1. MMLU (Massive Multitask Language Understanding):

   • 57 academic tasks (math, history, coding, logic).
   • Measures reasoning and general knowledge transfer.

2. ARC (AI2 Reasoning Challenge):

   • Science exam-style questions.
   • Tests commonsense and deductive reasoning.

3. TruthfulQA:

   • Designed to test model honesty and factual correctness.
   • Penalizes plausible-sounding but false answers.

Together, these benchmarks reveal whether a model:

• Knows facts (MMLU),
• Can reason (ARC), and
• Avoids hallucination (TruthfulQA).

Fine-tuning balances plasticity and stability:

• Too much updating → catastrophic forgetting.
• Too little → under-adaptation.

PEFT and LoRA maintain stability by freezing the base brain, while Axolotl and QAFT enable efficient, safe adaptation.

Benchmarks then measure whether this adaptation preserved intelligence or introduced bias.

This is the practical end of the LLM lifecycle:

• Pretraining builds general intelligence.
• Fine-tuning personalizes it.
• Quantization deploys it.
• Evaluation validates it.

It’s where research meets engineering — theory becomes product.

• $W’$: adapted weight
• $W$: frozen base weight
• $A, B$: small trainable matrices
• $r$: rank of the adaptation (usually 8–64)

• “LoRA changes the base model permanently.” No — it only adds small adapters; you can merge or remove them anytime.
• “Quantization always reduces accuracy.” It depends on task type — factual recall survives better than reasoning.
• “Benchmarks show intelligence.” They show competence under structure, not creativity or adaptability.