DPO Alignment — align models with preferences without reward models
Use Direct Preference Optimization to fine-tune a model on chosen/rejected response pairs, aligning outputs with human preferences directly without a separate reward model.
Fine-tuning4.5
Prompt template
N/A — this is a preference optimization technique. Prepare a dataset of prompt, chosen response, rejected response triples.
Format:
{
"prompt": "user question",
"chosen": "high-quality preferred response",
"rejected": "lower-quality response"
}How to use
- 1Collect or generate preference pairs: for each prompt, a 'chosen' (good) and 'rejected' (worse) response
- 2Ensure pairs have meaningful quality differences — trivial pairs (typo only) waste training signal
- 3Load the base model and its reference (frozen) copy; DPO compares log-prob ratios between them
- 4Configure DPOTrainer with beta 0.1-0.5 (controls how far from reference the model can move)
- 5Train for 1-3 epochs with learning rate 5e-7 to 5e-5, monitoring reward margin
- 6Evaluate alignment with a held-out preference test set and LLM-as-judge scoring
Pro tips
- DPO is simpler and more stable than PPO-based RLHF — no reward model or value function needed
- Beta controls the KL penalty: lower beta (0.1) allows more deviation, higher beta (0.5) stays closer to reference
- Preference data quality matters more than quantity — 1,000 high-quality pairs can beat 10,000 noisy ones
- You can combine SFT (supervised fine-tuning) first, then DPO for the final alignment step
Works best with
trlpefthuggingfacedatasets
DPORLHFAlignmentPreference LearningTRL
Ratings & Reviews
Loading reviews…
