2.7. Safety Alignment & Post-Training Alignment

Safety alignment ensures the model’s outputs are truthful, harmless, and unbiased — even when faced with tricky or adversarial inputs.

Key techniques include:

1. Bias Detection: Automatically identifying unfair or stereotypical responses.
2. Toxicity Mitigation: Filtering or rephrasing unsafe content.
3. Rule-based Guidance: Embedding ethical principles or constraints directly into the model’s reasoning process.

This layer sits on top of traditional fine-tuning — continuously monitoring and correcting unsafe tendencies.

While RLHF relies on human labels, post-training alignment leverages synthetic supervision — using other LLMs, rules, or heuristics to generate feedback automatically.

The goal:

“Teach the model to critique, refine, and correct itself.”

This makes alignment scalable — you don’t need thousands of human annotators for every update.

Key frameworks:

• Constitutional AI (Anthropic): Uses a “constitution” — a set of principles (like “avoid harmful advice”) to guide self-critique.
• Self-Reward Modeling (SRM): The model predicts its own reward signals using previously learned preferences.
• Direct Preference Optimization (DPO): Simplifies RLHF — directly optimizes model parameters for preference-consistent outputs without full PPO reinforcement learning.

Safety and alignment shift the objective of LLMs from accuracy to appropriateness. In other words, we stop asking “Is the answer correct?” and start asking “Is this the right thing to say?”

From a system design view, this represents the final layer of value alignment — ensuring technical excellence translates to trustworthy behavior.

DPO eliminates the need for PPO’s reinforcement loop by turning preference learning into a simpler optimization problem.

Given human or synthetic preference pairs $(x, y^+, y^-)$ (preferred vs. dispreferred responses), DPO directly trains the model to assign higher probability to preferred ones:

Here:

• $\pi_\theta$ = model’s probability distribution.
• $\beta$ = temperature controlling preference sharpness.
• $\sigma$ = sigmoid function.

This pushes the model to favor $y^+$ responses without needing reward models or RL agents.

Constitutional AI replaces human feedback with a set of written principles, like a “digital moral code.”

Process:

1. The model generates a raw answer.
2. A self-critic model evaluates it against the constitution.
3. The model revises its response accordingly.

Example rule:

“The model should avoid hateful or discriminatory statements.”

This forms a feedback loop without needing humans in every iteration.

Mathematically, it approximates RLHF but substitutes human reward $R_{human}$ with rule-based reward $R_{rules}$.

SRM extends RLHF by allowing models to generate synthetic rewards using previously aligned versions of themselves.

1. A strong model (“teacher”) scores responses for helpfulness or safety.
2. The student model trains to maximize those scores.

This iterative process allows self-improvement without new human data — a kind of bootstrap alignment.

• Use classifier models (e.g., Detoxify, Perspective API) to detect harmful or biased content.
• Implement “refusal strategies” for unsafe requests (“I can’t help with that”).
• Regular audits on demographic fairness and representational balance.

Modern deployment systems wrap LLMs with safety filters that:

• Flag unsafe prompts before processing.
• Inspect generated outputs for toxic or private information.
• Route flagged outputs to human reviewers or safer rewriters.

This layered defense ensures reliability even under adversarial input.

• “Safety alignment means censorship.” ❌ It’s not about silencing — it’s about responsible accuracy.
• “Self-aligned models don’t need oversight.” ❌ Self-feedback loops can still drift without human auditing.
• “Bias elimination is possible.” ❌ The goal is mitigation and transparency, not perfection.