Direct Preference Optimization (DPO)

DPO trains a model to prefer chosen responses over rejected ones using a classification loss — no separate reward model needed.

Preference Data

chosen vs rejected

→

Reference Policy π_ref (frozen)

→

DPO Loss β-weighted

→

Aligned Model π_θ

The DPO Loss

Lθ=−Ex,y_chosen,y_rejected∼D[log⁡σ(βlog⁡πθ(ychosen∣x)πref(ychosen∣x)−βlog⁡πθ(yrejected∣x)πref(yrejected∣x))]

\mathcal{L}{\theta} = -\mathbb{E}{x, y_{\text{chosen}}, y_{\text{rejected}} \sim \mathcal{D}}\left[\log\sigma\left(\beta\log \frac{\pi_{\theta}(y_{\text{chosen}}|x)}{\pi_{\text{ref}}(y_{\text{chosen}}|x)} - \beta\log \frac{\pi_{\theta}(y_{\text{rejected}}|x)}{\pi_{\text{ref}}(y_{\text{rejected}}|x)}\right)\right]

Lθ​=−Ex,y_chosen​,y_rejected​∼D​[logσ(βlogπref​(ychosen​∣x)πθ​(ychosen​∣x)−βlogπref​(yrejected​∣x)πθ​(yrejected​∣x))]

• πθ\pi_{\theta}πθ​ — current policy being trained
• πref\pi_{\text{ref}}πref​ — reference model (frozen, typically the pre-DPO checkpoint)
• β\betaβ — controls preference strength (higher = more conservative)

DPO vs RLHF

DPO eliminates the need for a separate reward model by directly optimizing the policy. Simpler and cheaper than classical RLHF, but requires the base model to already be in-distribution with the preference data.

Available Datasets

Key Hyperparameters

• dpo_beta — Start with 0.1. Higher values are more conservative.
• learning_rate — Lower than SFT, typically 1e-5 to 1e-6.
• Base model — Should be in-distribution with the preference data. Start with a light SFT phase or collect on-policy preferences.

Training Metrics

• dpo_loss — classification loss (should decrease)
• accuracy — how often the model prefers the chosen response
• margin — average reward difference between chosen and rejected
• chosen_reward / rejected_reward — average implicit rewards

Learn More

• DPO & Preferences Tutorial — hands-on interactive walkthrough
• RLHF Pipeline Tutorial — full 3-stage RLHF with reward model
• DPO Recipe — production training script
• tinker_cookbook.preference API — Comparison, PreferenceModel, Config