Cover illustration

Date: 2026-08-27
Suitable for readers: graduate students, new scientific research, engineering readers who want to check the preferences and alignment experiments
Search date: 2026-08-27

Abstract

DPO (Direct Preference Optimization, Direct Preference Optimization) is often said to be "RLHF that does not require a Reward Model". This sentence is only half right: DPO does omit the individual fitting of the explicit reward model and the use of PPO The phase of the update strategy is sampled online, but it still relies on the Preference pairs given by humans or other annotators, and defines "implicit rewards" through the logarithmic probability ratio of the reference model through the policy relative to the reference model.

This article focuses on an acceptable target: use a triple prompt/chosen/rejected Construct the minimum preference data, push from the formula to each sample loss, and then use a small script to verify that the loss is reduced, and the implicit reward for margin rises. This is not a reproduction of the scale of the paper, but a minimum checkpoint that exposes symbols, masks, templates and reference model errors.

Contents

  1. What does DPO save?
  2. The smallest unit of preference data
  3. From KL Constraints to DPO Loss
  4. A guide to the official implementation
  5. A minimal experiment and evaluation protocol
  6. Failure Analysis and Evidence Boundary
  7. Questions for further research
  8. Summary and References

What does DPO save?

The classic RLHF pipeline usually has three steps: first supervise fine-tuning (SFT), then train the Reward Model with answer sorting, and finally use PPO and other reinforcement learning methods to choose between rewards and deviation reference strategies. The InstructGPT paper is one of the representatives of this pipeline. The key practice of a DPO paper is to write the optimal strategy in the problem of KL constraint reward maximization as a closed expression, and then re-parameter Reward into the log-ratio of the current strategy and reference strategy. In this way, you can directly do binary classification training for preference pairs.

Therefore, the safest explanation of "Reward-Free" is: no longer train an independent, queries and explicit Reward Model, and do not need to be repeatedly sampled from the strategy during training and then scored by the model. It does not mean "no reward hypothesis", nor does it mean "no human preference cost". Preference tags are the supervised signals of DPO, and the reference model gives a relative scale.

The smallest unit of preference data

A DPO sample is (x, y_w, y_l): x is prompt, y_w is the Chosen completion preferred, y_l is a rejected completion. The two answers must correspond to the same prompt; otherwise the model may learn the prompt difference, not the answer quality difference. TRL The current official document supports ordinary string format and dialogue message format, and recommends explicit reserved prompt.

{
  "prompt": "用一句话解释梯度下降。",
  "chosen": "沿损失增长最快方向的反方向更新参数。",
  "rejected": "随机改变参数。"
}

Checking that all three fields exist is only the first step. Verify that the two responses differ and that applying the chat template gives them identical prompt-token prefixes. Check that truncation does not remove the chosen response’s key conclusion while preserving the rejected response. Accumulate loss over completion tokens only, so the prompt does not count twice in the preference comparison. When splitting the data, keep near-duplicate prompts together to prevent leakage across splits.

There is also a "Preferential Pairing Physical Examination Form". Record the number of tokens, length difference, source model, sampling parameters, labeling rules and labeler consistency for two answers for each sample. If Chosen is almost always longer, or rejected always comes from a weaker model, the Training Set has a shortcut that can be easily exploited. For the conversation data, training and testing should be divided on a complete session or prompt family, instead of randomly breaking up the rewrite of the same problem.

Draws and weak preferences should not be forced to be hard labels. If the labeler cannot distinguish two answers, the pair can be deleted, the probability of soft preferences is retained, or at least layered separately in the analysis. Otherwise, a randomly decided Chosen will be lost as deterministic evidence, which is label noise, not more supervision.

From KL Constraints to DPO Loss

The paper starts with a KL constraint reward maximization goal. Given a prompt x, the reference strategy is pi_ref, the training strategy is pi_theta, you can write the target as:

max⁡π  Ey∼π(y∣x)[r(x,y)]−βDKL[π(y∣x)∥πref(y∣x)]\max_{\pi}\;\mathbb{E}_{y\sim\pi(y|x)}[r(x,y)]-\beta D_{KL}[\pi(y|x)\|\pi_{ref}(y|x)]

r(x,y) is to answer the reward, beta>0 It is the scale parameter between the reward and the deviation reference strategy. its optimal solution pi_r(y|x) ∝ pi_ref(y|x) exp(r(x,y)/beta). After sorting, the reward can be expressed as beta * log(pi_r/pi_ref) add one only with x related normalization. When comparing the two answers of the same prompt in the Bradley–Terry preference model, this normalization term is eliminated and gets:

LDPO=−Elog⁡σ(β[log⁡πθ(yw∣x)πref(yw∣x)−log⁡πθ(yl∣x)πref(yl∣x)])L_{DPO}=-\mathbb{E}\log\sigma\left(\beta\left[\log\frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)}-\log\frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}\right]\right)

DPO Minimum Reproduction Process

for batch size as B, the sequence length is T, the size of the vocabulary is V The autoregressive model, the original logits shape is [B,T,V]. Use the label to remove the log-prob of the target token, then multiply the completion mask and T Sum, get one each of Chosen/Rejected [B] Sequence log-prob. Do the current strategy and reference strategy once, and finally generate [B] Reward margin and scalar batch loss. This shape chain is the fastest way to check masks and symbols.

The important thing is that the DPO optimization is the "chosen/rejected advantage of the relative reference model", which is not the absolute probability of maximizing the chosen alone. Therefore, the loss of loss does not guarantee that the Chosen log-prob must rise; it may also expand the margin mainly by lowering the rejected .

You can check and understand with a numerical example. When policy and reference are exactly the same, both log-ratio are zero, and each pair of loss is -log sigmoid(0)=log 2, about 0.6931. If the relative log-prob of subsequent chosen increases 0.8, the relative log-prob of rejected is reduced 0.2, when beta=0.1 when, DPO Logit is 0.1*(0.8-(-0.2))=0.1, loss to about 0.6444. This also shows that the beta and the optimization step are coupled: you can't guess the training intensity just by looking at the name of the beta, you should jointly observe the amount of margin and kl proxy and the quality of generation.

A guide to the official implementation

The author's reference repository gives a clear route: first use preference_datasets.py See how preferences are organized, and then look at trainers.py In policy/reference log-prob, DPO loss and training loop, and finally back to train.py Check the configuration delivery. Repository README divides the pipeline into two stages of SFT and preference learning, and recommends that the preference data is as consistent as the SFT distribution.

If you need modern engineering implementation, read TRL's DPOTrainer. The order is: data format and chat template; tokenization and EOS of prompt/chosen/rejected; padding and truncation; policy and reference forward; default sigmoid loss; last is rewards/chosen、rewards/rejected、rewards/margins and rewards/accuracies log. The current TRL also supports a variety of DPO variants, but the minimum reproduction should first fix the default sigmoid loss to avoid using the framework switch as the original paper conclusion. Documents and default values will change. Real experiments must record TRL, Transformers, tokenizer and model revision. The current API details of this article are marked for manual review.

One easily overlooked cost is that every batch contains both chosen and rejected responses, and both the policy and the reference model must score them. A naïve implementation may take four forward passes. Implementations can batch the two responses together to reduce scheduling overhead. If the reference log probabilities stay fixed throughout training, precompute them and store them with the data to trade storage for training time. Pin the tokenizer, template, truncation settings, and reference-model revision together, or the cached values can silently become stale.

When the policy uses LoRA, you can get the base reference by disabling the adapter, avoiding another complete model. However, this optimization cannot only see whether the memory is down, you must use a fixed batch to compare the "Policy" with a separate reference. log-prob; when the two are inconsistent, prioritize the merged weights, dropout status, precision and adapter switching logic.

A minimal experiment and evaluation protocol

The companion script is in code/minimal_dpo.py. You can run standard library checks without installing any dependencies:

python3 code/minimal_dpo.py --check-only

The script first verifies the triple, then uses three pairs of manual Sequence log-prob to calculate the DPO loss, and finally performs the parsing gradient update for each pair of policy margins. In this actual Smoke test, lost from 0.693147 Down to 0.415586, the average implicit reward margin from 0 rise to 0.663086, three pairs of Reward Accuracy by 0 Turn into 1. These numbers only prove that the loss direction of the script is the same as the inspection volume, and is not a language model alignment result.

After installing PyTorch, you can run a [3,2] TOY optimization for Logits:

python3 -m pip install -r code/requirements.txt
python3 code/minimal_dpo.py --steps 80 --seed 7

Real small model experiments should have at least four sets of records. Data layer records labeling rules, draw processing, length distribution, near-repeat deduplication and segmentation; model layer records SFT checkpoint, reference structure, chat template and whether to use LoRA; optimization layer record Beta, learning rate, valid batch, sequence length and SEED; the evaluation layer simultaneously reports Held-out DPO loss, Reward accuraacy/margin, generation length, and reference KL Agent quantity and independent person evaluation or task indicator.

Don't just report the Reward Accuracy. It only asks whether implicit rewards put Chosen on top of rejected, and cannot prove that the model is more realistic, useful or safe to generate new prompts. It is best to pre-freeze the evaluation Prompt, decoding parameters and Rubric, make a blind test comparison between SFT/Reference and DPO Checkpoint, and keep the sample-by-sample output.

An actionable three-layer acceptance sequence is: first do the math unit test, and use four known log-prob confirmation symbols and logsigmoid; Do the overfitting test again, confirm that dozens of preference pairs can be remembered by the small model, and sample the probability changes of Chosen/Rejected; finally do generalization evaluation, and compare the generation and task indicators on the no prompt. The failure of the first layer is an implementation error, and the second layer failure is mostly optimization or data pipeline error. Only after the first two layers are passed, the third layer failure is worth discussing methods and data hypotheses.

To evaluate the variance, at least run multiple SEEDs for small experiments, reporting mean, standard deviation, and SEED logs. If the resource is only enough to run once, the result should be clearly marked as an exploratory run, and priority should be reserved for sample output, length, margin and failure type, rather than only one total score.

The final product should be replayed by third parties: keep the immutable version or checksum of the original preference pair, preprocessing scripts, templates, full startup commands, environment lock files, sample-by-sample JSONL and failure sample annotation for each evaluation. Only checkpoint and no data and protocol, it is impossible to prove that the two DPO experiments are really comparing the same problem.

Failure Analysis and Evidence Boundary

The first category is data fake alignment. Chosen and rejected use different system prompts, different conversation history, or different truncation rules, and the model will read these shortcuts. The repair method is to directly compare the two prompt token prefixes after tokenization, and count the truncated ratio.

The second category is lost symbols or mask errors. You can make loss by reversing the Chosen/rejected, or inverting the reference log-ratio, or the padding/prompt token into the sequence log-prob. Seems to decline normally. A must-do unit test is: after manually improving the Chosen relative to the policy log-prob, the loss must fall, and the reward margin must rise.

The third category is the Reference misalignment. The reference model should be frozen, and its tokenizer, template and complex mask should be comparable to policy. If the reference model is also updated by mistake, the log-ratio ruler will drift; if the adapter is not switched correctly in the LoRA training, the policy and the reference may even point to the same state.

The fourth category is preference over optimization. Preference data may mistake "longer, more formatted, and more confident" as quality, and training can successfully expand margin, but enlarge these label deviations. Accuracy should be reported in length intervals, check the length advantage of selected answers, and add authenticity or execution result indicators to verifiable tasks.

The evidence needs to be strictly stratified. The DPO paper reports a comparison with the PPO pipeline on emotional control, abstract and single-wheel dialogue tasks; this paper does not rerun these settings. This actual verification only includes first-hand data open verification, Python syntax check and standard library Toy Loss Smoke Test. PyTorch Toy and the real language model fine-tuning is not running and should not be written as a reproducing result of the paper.

Questions for further research

  1. Is the beta and learning rate discernible on the same SFT checkpoint, or is it just coupled through effective gradient scales?
  2. After stratifying the preferences on the difference in length, difficulty and labeler consistency, which one can best improve the independent reviewer?
  3. How does a periodic update reference change stability and overfitting compared to a fully frozen reference model?
  4. When both Chosen and rejected are poor, do paired relative preferences require a lower quality limit or SFT loss as a guardrail?
  5. In terms of code, mathematics and other executable tasks, can the test results be recorded separately from human preferences, and can the training direction of "more favorable but more incorrect" be identified?

Summary

The smallest essence of DPO is: answer to the choice/rejected of the same prompt, compare the two log-ratios of the current policy relative to the reference strategy, and then use -log sigmoid Expand Preferences Margin. It bypasses the explicit Reward Model and PPO training rings, but does not bypass the preference data, reference strategy, length deviation and evaluation design.

For the reproduction of scientific research, the first step is not to immediately Fine-Tuning billions of parameter models, but to use manual log-prob to confirm symbols and masks, and then use small models to fix data, reference, beta and decoding protocol. Only when "loss changes as expected" and "independent evaluation really improves" can be interpreted as a preference alignment success.

If there is only the former but not the latter, you should stop expanding the training and go back to the data deviation, the reference model and the evaluation protocol to re-examine, instead of continuing to pursue a smaller training loss.

References

Search date: 2026-08-27. The following preferences are used for paper pages, conference pages, author repository and official framework documents. The TRL code and the default configuration will change, and the locked version needs to be checked before the actual operation.

  1. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn, Direct Preference Optimization: Your Language Model is Secretly a Reward Model, arXiv 2023,v3 2024.
  2. NeurIPS 2023, DPO conference page.
  3. Eric Mitchell et al., DPO official reference implementation, including data processing, trainer and SFT/DPO configuration.
  4. Hugging Face TRL, DPOTrainer documentation, data format, loss, log metrics and PEFT integration.
  5. Hugging Face TRL, dpo_trainer.py, the current trainer source code.
  6. Long Ouyang et al., Training language models to follow instructions with human feedback, arXiv 2022 / NeurIPS 2022.
  7. Yuntao Bai et al., Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, arXiv 2022.