3.2. Instruction Fine-Tuning & Mixture Objectives

Flan (Fine-tuned Language Net) was Google’s first major framework to fine-tune large models like T5 on explicit task instructions.

Instead of raw text, the model was shown inputs formatted like this:

Input: “Classify the sentiment of this review: The movie was amazing.” Output: “positive”

or

Input: “Translate to French: Where is the train station?” Output: “Où est la gare ?”

These natural language task prompts taught the model to associate task intent with behavior.

Key Concept: Multi-task Fine-Tuning Flan fine-tuned models across hundreds of tasks simultaneously — translation, summarization, reasoning, classification, etc. This taught it how to recognize instructional patterns and adapt behavior dynamically.

Result: Flan models could perform zero-shot learning — solving unseen tasks simply because they understood the style of human instructions.

OpenAI’s InstructGPT took instruction tuning a step further by adding human judgment to the loop.

1. Base Model: Start with a pretrained GPT-3 model.
2. Supervised Fine-Tuning (SFT): Train it on human-written instruction–response pairs (like “Explain quantum physics simply”).
3. Reward Modeling: Train a smaller model to predict which outputs humans prefer.
4. Reinforcement Learning (RLHF): Use Proximal Policy Optimization (PPO) to fine-tune the model toward responses that humans rate higher.

This process aligned the model not just with instructions, but with human intent, helpfulness, and tone.

UL2 (Unifying Language Learning) built on T5 but introduced a more flexible pretraining strategy: the Mixture of Denoisers Objective.

Instead of a single masking style (like T5’s span corruption), UL2 uses a blend of denoising tasks:

• R-denoising: like span corruption (T5-style).
• S-denoising: short masked spans — tests local reasoning.
• X-denoising: extreme masking — forces global reasoning and generation.

Each denoising task type is randomly selected during training, giving the model exposure to multiple reasoning styles.

Effect: UL2 became capable of smoothly switching between understanding (like BERT) and generation (like GPT), enabling better transfer to instruction-tuned settings.

Prompt prefixing is the idea of prepending short cues to guide the model’s behavior:

“Translate English to German: …” “Summarize: …” “Answer the question: …”

These prefixes serve as task labels embedded in text form — no separate model heads or configurations needed.

Modern instruction-tuned models (Flan, UL2, and beyond) learn that “Translate to French” → translation mode, “Summarize” → condensation mode, etc.

This makes task-switching linguistically natural — the model thinks in instructions instead of APIs.

When fine-tuning on new tasks, models risk forgetting earlier knowledge — this is called catastrophic forgetting.

Instruction tuning mitigates this by:

• Mixing diverse task types during training (multi-task).
• Using continual fine-tuning — small batches of new tasks rather than total retraining.
• Adding regularization that keeps weights close to their pretrained values.

The trade-off is subtle:

Too narrow fine-tuning → better task accuracy but worse generalization. Too broad fine-tuning → better generalization but less specialization.

Instruction tuning aligns models with meta-learning principles: Instead of memorizing task rules, the model learns how to learn tasks from natural instructions.

Every instruction becomes a mini-lesson — by observing hundreds of phrasing styles (“Summarize…,” “Write briefly…,” “In short…”), the model internalizes task intent inference.

Instruction tuning bridges the gap between pretraining and alignment:

• Pretraining builds knowledge.
• Instruction tuning teaches communication.
• RLHF adds values and preferences.

Together, they transform raw predictive engines into cooperative assistants capable of following natural language directions gracefully.

• $T$: set of all tasks (translation, summarization, QA, etc.)
• $w_t$: weight for each task (balances learning importance)
• $P(y|x, \theta)$: probability of generating the correct output given input $x$

• “Instruction tuning just rephrases prompts.” No — it reconfigures the model’s internal attention patterns to associate intent with task type.
• “Flan and InstructGPT are the same.” Flan focuses on task diversity; InstructGPT adds human feedback and RL.
• “UL2 is just a T5 variant.” It’s a general-purpose learning framework that unifies multiple denoising styles for flexibility and better reasoning.