LLM Alignment Explained: RLHF, DPO, Constitutional AI and Reinforcement Learning from Verifiable Rewards
How large language models are trained to be helpful, honest and safe: RLHF with reward models, Direct Preference Optimization and its variants, Constitutional AI and AI feedback, reinforcement learning on verifiable rewards with GRPO, how the methods compare, known failure modes and practical advice for fine-tuning.
A model trained only to predict the next token learns to continue text, not to be helpful, honest or safe. It may ramble, follow harmful instructions or confidently make things up. Alignment training is the set of techniques used to close that gap after pre-training. This guide explains the main methods, from reinforcement learning from human feedback to newer approaches that reward verifiably correct answers, how they compare and where they still fall short.
RLHF: the method that made chat assistants work
Reinforcement learning from human feedback (RLHF) has three stages. First, the model is fine-tuned on examples of good responses written by people (supervised fine-tuning). Second, people compare pairs of model outputs and say which is better, and those comparisons are used to train a reward model that predicts human preferences. Third, the language model is optimised with reinforcement learning, usually an algorithm called PPO, to produce responses the reward model scores highly, with a penalty that keeps it from drifting too far from the original model.
OpenAI’s InstructGPT paper showed how powerful this is: human evaluators preferred outputs from a 1.3-billion-parameter InstructGPT model over the 175-billion-parameter GPT-3 it was derived from. The same recipe underpinned ChatGPT.
RLHF has well-known drawbacks. Human feedback is expensive and slow to collect, people disagree, and the reward model is only an approximation of what people want. A model optimised hard against an imperfect reward model can learn to exploit its blind spots, a problem known as reward hacking. Preference training can also reward telling people what they want to hear.
DPO: preference training without a reward model
Direct Preference Optimization (DPO), introduced in 2023, showed that the same goal can be reached more simply. Instead of training a separate reward model and running reinforcement learning, DPO adjusts the language model directly on pairs of preferred and rejected responses with a classification-style loss. It is more stable, cheaper and easier to implement, which made it popular for fine-tuning open models.
Variants address different needs. KTO works with simple thumbs-up or thumbs-down labels rather than paired comparisons, which are easier to collect. IPO and ORPO modify the objective to reduce overfitting or combine supervised and preference training in one step. DPO’s main limitation is that it learns from a fixed dataset of preferences collected in advance; it cannot explore and receive fresh feedback the way online reinforcement learning can, so the quality and coverage of the preference data matter a great deal.
Constitutional AI and AI feedback
Anthropic’s Constitutional AI replaces much of the human labelling for harmlessness with feedback from an AI model guided by a written set of principles, the constitution. In the supervised stage, the model drafts responses, critiques them against the principles and revises them. In the reinforcement learning stage, an AI model judges which of two responses better follows the principles, and those judgements train the preference model. This is often called reinforcement learning from AI feedback (RLAIF).
The advantages are scale and transparency: feedback can be generated cheaply, and the principles are written down rather than implicit in thousands of individual human judgements. Anthropic has published the constitution it uses for Claude. The risk is that AI feedback inherits the blind spots and biases of the model producing it, so human review and red-teaming remain necessary.
Reinforcement learning from verifiable rewards
The biggest shift since 2024 has been training on tasks where correctness can be checked automatically, such as maths problems with known answers or code that must pass tests. Instead of a learned reward model, the reward comes from a verifier. DeepSeek’s GRPO (Group Relative Policy Optimization), introduced in the DeepSeekMath paper, simplifies the reinforcement learning step by comparing a group of sampled answers to each other rather than training a separate value model.
DeepSeek-R1 showed what this approach could do: its precursor, R1-Zero, developed long step-by-step reasoning through reinforcement learning on verifiable tasks without human-written reasoning examples (arXiv:2501.12948). Verifiable rewards are much harder to game than a learned reward model, though not impossible; models can still find shortcuts such as exploiting weak tests. The approach also only works where answers can be checked, so preference-based methods remain necessary for open-ended qualities such as tone, helpfulness and safety.
The core papers
- Christiano et al., 2017 — “Deep Reinforcement Learning from Human Preferences” introduced learning a reward model from human comparisons (arXiv:1706.03741).
- Ouyang et al., 2022 — InstructGPT applied supervised fine-tuning, a reward model trained on human rankings, and PPO to GPT-3; human evaluators preferred outputs from a 1.3B InstructGPT model over the 175B GPT-3 (arXiv:2203.02155).
- Bai et al., 2022 — Constitutional AI replaced much human harmlessness labeling with AI feedback guided by a written set of principles (RLAIF) (arXiv:2212.08073). Anthropic has published the constitution it uses for Claude.
- Rafailov et al., 2023 — Direct Preference Optimization (DPO) showed that the RLHF objective can be optimized directly on preference pairs with a simple classification-style loss, without training a separate reward model or running reinforcement learning (arXiv:2305.18290).
How the methods compare
| Method | Feedback source | Needs RL loop? | Strengths | Drawbacks |
|---|---|---|---|---|
| RLHF (PPO) | Human preferences via a reward model | Yes | Flexible, proven at scale | Complex, expensive, prone to reward hacking |
| DPO and variants | Offline preference data | No | Simple, stable, cheap | Limited by data quality and coverage |
| Constitutional AI / RLAIF | AI judgements guided by written principles | Often | Scales feedback; principles are explicit | Inherits biases of the feedback model |
| RL from verifiable rewards (e.g. GRPO) | Automatic checks: answers, tests | Yes | Hard to game; drives reasoning gains | Only for checkable tasks |
In practice, frontier labs combine several: supervised fine-tuning, preference optimisation, AI feedback guided by principles, reinforcement learning on verifiable tasks and extensive red-teaming, followed by separate safety classifiers at deployment.
Unresolved problems
Sycophancy. Preference training can reward agreement with the user. Anthropic researchers documented this tendency across several assistants (Sharma et al., 2023), and it remains an active problem.
Honesty and calibration. Models trained to be helpful may produce an answer when they should say they do not know. Training them to express uncertainty helps but can make them seem less useful, and balancing the two is difficult.
Jailbreaks. Alignment training reduces but does not eliminate harmful outputs under adversarial prompting. That is why deployed systems add separate input and output classifiers and monitoring.
Whose values? “Helpful and harmless” can mean different things across cultures, communities and contexts, and decisions about defaults are made largely by the companies training the models. Published principles and model specifications make those choices more visible, but they do not settle the question of who should decide.
Oversight of more capable systems. As models become better than people at some tasks, it gets harder for humans to judge their outputs. Research on scalable oversight, interpretability and evaluation aims to address this, but there is no complete solution.
Practical takeaways for builders
If you fine-tune open models, start with supervised fine-tuning on good examples, then use DPO or a variant such as KTO or ORPO to steer behaviour with preference data. If your task has checkable answers, such as structured extraction validated against a schema or code validated by tests, reinforcement learning with verifiable rewards is worth exploring. Libraries such as Hugging Face TRL implement DPO, KTO, ORPO and GRPO.
Collect preference data from real use cases rather than generic prompts, keep a held-out evaluation set, and test for regressions in helpfulness, accuracy and safety after every training run. Watch especially for sycophancy and for the model learning to game whatever signal you reward.
Our analysis
Alignment has evolved from a single recipe, RLHF, into a toolkit: preference optimisation for tone and helpfulness, written principles and AI feedback for scale, and verifiable rewards for reasoning. Each method trades off cost, stability and susceptibility to gaming, and none removes the need for evaluation and red-teaming. For builders, the practical lesson is to match the method to the kind of feedback you can reliably collect, and to measure what the model actually learned rather than what you intended to teach.