
Date:2026-08-04
Who this is for: already understand the concept of SFT, Transformer and basic reinforcement learning, and want to systematically read the graduate students of RLHF, DPO, GRPO and preference data papers, newcomers and engineering background readers.
Abstract
RLHF / Preference Optimization is the most easily overwhelmed by "formal nouns" in modern LLM training: PPO, Reward Model, DPO, KTO, ORPO, SimPO, GRPO, DAPO It looks like a list of algorithms, but there are three things that really need to be read: how the Preference signal is defined, how the Optimization goal is to convert the Preference into a language model update, and how the evaluation proves that the model is more in line with the goal rather than longer, more intriguing or will cater to Judge. This article gives an intensive reading route: first read the classic RLHF pipeline, then read the offline Preference Optimization of the DPO family, then read GRPO / RLVR inference training, and finally use RewardBench, AlpacaEval and the reproduction log to examine the paper evidence.
Contents
- Why RLHF / Preference Optimization is important
- Core reading method: from data, objective function to evaluation closed loop
- A reading guide to the key papers
- A closer look at key papers
- Methods and experiments compared
- How to reproduce the results
- Common misconceptions
- Research questions for graduate students
- Summary
- References
Why is this topic important
The pre-training model learned the "statistic law of the next token", SFT makes it more like an assistant who can answer questions according to instructions, but "what answer is better" often cannot be defined by just a reference answer. Whether the summary is concise, whether the abstension is moderate, whether the mathematical reasoning is reliable, whether the code answer is executable, and whether the long answer is just verbose requires some kind of preference signal. The contribution of RLHF is to convert human comparisons, rule validation, or model feedback into optimized goals, so that the model is updated in a "more preferred" direction.
Preference training carries several risks. Preferences depend on data sources, annotation guidelines, and the people supplying the judgments. A policy can exploit its reward model, and training can produce reward hacking, a bias toward verbosity, excessive refusals, or regressions in safety. Offline methods such as DPO simplify the pipeline but make preference-pair quality, the reference model, length normalization, and hyperparameter sensitivity especially important. Since DeepSeek-R1, GRPO and RLVR have drawn attention to combining verifiable rewards, multiple samples, and long reasoning traces. Many reproductions fail even with the right equations because data filtering, templates, sampling, length penalties, logging, and evaluation protocols differ.
So reading this direction can't just memorize the formula. You need to ask: who is the preference from, what is the optimization of the reward, what component is omitted by the method, what deviation is introduced after the component is omitted, and whether the paper evidence can rule out "model length", "benchmark leak", "judge bias", "base" Model would have been explained.
Core reading method

Read the RLHF / Preference Optimization paper, which can be disassembled by four layers.
The first layer is data. See what the sample unit is: one prompt with two answers, or one prompt with multiple candidates; feedback is human preference, AI feedback, rule validation, single thumbs-up/down, or Reward model Scoring; whether Chosen is really high-quality, whether rejected is just a weak model output; whether the Training Set and review set share Prompt templates, data sources, or generative models.
The second layer is the objective function. The PPO class method explicitly trains the policy to de-maximize the REWARD, and at the same time uses the KL constraint to prevent the model from deviating from the reference strategy; the DPO rewrites the optimal policy relationship of the Reward model into a classification loss; KTO uses binary Desirable / undesirable; ORPO and SimPO attempts to reduce the number of reference models or stages; GRPO uses a set of sampling results for the same prompt to make a relative advantage, and reduce the criticalic / The cost of the value model. When reading the formula, don't stop at "Whether RL", but write what each loss encourages, what is punished, how to deal with the default length, whether the reference model is involved, and whether online sampling is required.
The third layer is the training system. Many conclusions of RLHF and RLVR depend on engineering details: Rollout speed, vLLM/SGLang generation backend, ZeRO/FSDP, Reference Model placement, KL Calculation, Reward Server, Breakpoint Renewal, SEED, Sampling Temperature, Number of Prompt Sampling, Max Tokens, whether to filter all pairs or all wrong samples. If you only write "we use GRPO", the reproduction value is not enough.
The fourth level is the evaluation. Preference optimization papers are commonly used for MT-Bench, AlpacaEval, Arena-Hard, RewardBench, MATH, AIME, HumanEval, IFEval or safety evaluation. You have to distinguish: whether to evaluate the policy, or to evaluate the Reward Model; whether it is automatic judge, or human preference; whether there is length control; whether there is multiple sampling average;
A reading guide to the key papers
1. Classic RLHF Pipeline
From the Human Preferences language model of Ziegler et al. Fine-Tuning, to the Summarization from Human Feedback of OpenAI, and then to InstructGPT, the main line is "Collecting Compare Data -> Training Reward Model -> Optimizing Policy with PPO -> Adding KL to Control Drift". The focus of reading this group of papers is not whether the exact same pipeline is used today, but to understand why the Reward Model is necessary, why the PPO is complicated, and why KL has become the basic constraint in post-training.
2. DPO and Offline Preference Optimization
The key contribution of DPO is to rewrite the RLHF target into a supervised pair of unexpedily trained Reward Model, not online Rollout. Subsequent KTO, ORPO, SimPO, f-DPO, RSO and other methods continue to transform from feedback form, reference model, divergence, length normalization and sampling sources. When reading this branch, you should ask: After the method becomes simple, does it sacrifice the ability to explore? Are you more dependent on preference data quality? Will Chosen be made shorter, longer or more patterned?
3. GRPO / RLVR and Inference Model
DeepSeekMath proposes GRPO, and DeepSeek-R1 pushes "reinforced learning on verifiable tasks" to the research center. The preferences here do not necessarily come from human comparisons, but may come from verifiable rewards such as math, code, formatting, or unit tests. The DAPO, Dr. GRPO and asynchronous RL systems further point out that the key to long inference training is not just REWARD, but the variance, length deviation, long penalty, dynamic sampling and system throughput of the sampling group.
4. Reward Model and Review
RewardBench and RewardBench 2 indicate that evaluating the Reward Model itself is already an independent issue. A Reward Model has a high score on Pairwise Benchmark, which does not necessarily mean that it can stably improve the policy; a policy that wins on AlpacaEval does not necessarily mean that the reasoning ability or security is improved. When reading evaluation papers, "the preferences of the evaluator" should be used as the research object, rather than as an objective referee.
A closer look at key papers
Learning to summarize from human feedback / InstructGPT
These two articles are suitable for reading in the first week. Draw three stages of the process when reading: SFT initialization, Reward Model learning comparison, PPO optimization policy. Focus on how to collect preference data, how to control the consistency of the annotator, how KL Penalty prevents the model from deviating, and why artificial preferences can beat automatic metrics such as ROUGE. Reproducing training does not need to run the large model at the beginning, you can use the open source Summarization data, a small Reward Model and a PPPoTrainer of TRL to make a minimum closed loop.
DPO
DPO needs to intensively read formula derivation. You need to push yourself from the KL-regularized RLHF Objective to the Reward-Policy relationship, and then to Bradley-Terry Preference loss. After reading, write a page of "What does DPO save": save the explicit training of Reward Model, save online sampling, and save the Critic of PPO, but keep the Reference policy and the preference pair. The experimental part focuses on whether the comparison between DPO and PPO, SFT and Rejection Sampling controls the model initialization, data and evaluation protocols.
KTO / ORPO / SimPO
These three are suitable for reading side by side. KTO is asking "whether desirable / undesirable can be trained if there is no pairing preference"; ORPO tries to combine SFT and preference punishment into a stage; As an implicit reward, and remove the Reference Model. Their common training point is: don't just look at the paper claiming "easier", but dismantle the assumptions behind each simplification. For example, whether reference-free really saves memory, whether it is more difficult to control the drift; whether the binary feedback is cheaper, and whether it is more ambiguous; learning mixed together.
DeepSeekMath / DeepSeek-R1 / DAPO / Dr. GRPO
This group of papers is a new branch that must be supplemented by RLHF after 2025. The motivation for GRPO in DeepSeekMath is to reduce the cost of the Value Model in the PPO, and use the Reward mean and standard deviation of multiple outputs under the same prompt to estimate the Advantage. DeepSeek-R1 allows researchers to see that rule-based reward can drive long inference modes in verifiable tasks such as mathematics and code. The value of DAPO is to disclose more complete training recipes: dynamic sampling, clip adjustment, token-level loss, and too long punishment. Dr. GRPO reminds everyone that the standardization and length processing of GRPO may introduce optimization deviations.
RewardBench / RewardBench 2
These two are used to train your review sensitivity. Reward Model can be used for RLHF training and best-of-N or reranking; but Reward Model Benchmark and Downstream Policy There is no automatic equivalent between improvements. RewardBench 2 has made it harder to make multi-skilled Reward Model reviews harder and reports relevance to training uses such as downstream reasoning extensions and PPOs. When reading such papers, it is best to divide the Benchmark item into Instruction Following, Reasoning, Safety, Style, abstention, nuance, etc., and then see which buckets the model is on.
Methods and experiments compared
| Method | main signal | Whether explicit Reward Model | Whether to sample online | The most checked when reading a paper |
|---|---|---|---|---|
| PPO-RLHF | human preference -> RM score | Be | Be | KL, Critic, Reward Overoptimization, Rollout logs |
| DPO | chosen/rejected pair | No | No | Reference Model, Beta, Pair quality, length deviation |
| KTO | desirable / undesirable binary feedback | No | No | Binary label source, positive and negative sample ratio, task adaptability |
| ORPO | SFT data + preference pair | No | No | Whether single-stage training confuses SFT and alignment benefits |
| SimPO | pairwise preference + average log probability | No | No | reference-free drift, beta/gamma, answer length |
| GRPO | Multi-sampling REWARD with Prompt | Optional | Be | GROUP SIZE, REWARD variance, zero variance sample, length processing |
| DAPO | RLVR Reward + Dynamic Sampling | Optional | Be | Data filtering, dynamic sampling rules, too long penalties, reproduced scripts |
How to reproduce the results
Don't start with a full RLHF large system. It is recommended to do three-layer reproduction.
Tier 1: DPO Minimum Reproduction. Select a 1B-3B Instruction Model, use a small subset of HH-RLHF or UltraFeedback, fix the Prompt template, and train a round with TRL DPOTrainer. Records the result of Train Loss, Chosen/Rejected Reward Margin, Response Length, MT-Bench, or AlpacaEval. The goal is not to swipe the list, but to observe how the beta and learning rate affect the output length and abstent style.
The second layer: Reward Model is reproduced. Train a Pairwise Reward Model, and then measure Accuracy on RewardBench or self-built small collections. Then use it for best-of-N, and observe whether the REWARD score improvement really improves the quality of manual inspection. It's easiest to find here Reward Hacking: the model may be lengthy, variable, over-safe, or cater to judge.
At the third level, run a small RLVR/GRPO reproduction. Choose GSM8K, a subset of MATH, or a coding task with unit tests. Assign rewards with a rule-based verifier, sample multiple responses per prompt, and run GRPO or RLOO. At every step, log group rewards, answer length, pass rate, the proportions of all-correct and all-incorrect groups, KL divergence, entropy, and truncation rate. Without those logs, it is hard to tell whether training improves reasoning or simply changes the length of the samples.
The reproduction report writes at least five fields: data version, model checkpoint, training command, key hyperparameters, failure examples. Preference optimization experiments are very sensitive to SEED, template and generating parameters, and a single result cannot be a conclusion.
Common misconceptions
Myth 1: DPO "must be better" than PPO. DPO is simpler and more stable, but it is mainly off-line preference optimization. Online RL is still valuable when you need to explore new behaviors, rely on environmental feedback, or verifiable rewards.
Myth 2: The higher the Reward Model score, the better the policy. Reward Model will have blind spots. An important reminder of such work as RewardBench 2 is: Reward Model Evaluation and Downstream Training Performance are to be verified separately.
Misunderstanding 3: Long CoT is the emergence of reasoning ability. In GRPO/RLVR training, length growth may come from task requirements, or from Loss, standardization, truncation, and too long penalty designs. Accuracy, token number, truncation rate and error samples must be reported at the same time.
Myth 4: Preference data only depends on the scale. DPO data research has repeatedly pointed out that Chosen quality, pair contrast, on-policy and labeling consistency may all be more critical than the number of simple samples.
Myth 5: Automatic evaluation can replace manual reading. AlpacaEval, Arena-Hard, LLM-as-judge are useful, but in the Preference Optimization paper, they are objects that may be optimized in themselves. High score samples are subject to manual inspection.
Research questions for graduate students
- Preferences for Quality Diagnostics: Design indicators predict which DPO samples will bring effective updates, compare Chosen quality, rejected difficulty, semantic distance and length difference.
- Drift analysis of the reference-free method: Do multi-task evaluation on the SimPO/ORPO class method, and check the common changes of safety, fact, reasoning and answer length.
- GRPO Length Deviation Reproducing Experiment: The system changes the GROUP SIZE, MAX tokens, the long penalty and standardized method on the system on the small mathematical model, and verify the relationship between the length growth and the Accuracy.
- Reward Model Correlation of Downstream Policy: Train multiple Reward Models to do correlation analysis between RewardBench, best-of-N, PPO or DPO.
- Preference optimization failure sample library: Collect samples such as excessive abstent, cater to Judge, long-wrong, correct format but semantic error, code test overfitting, etc., to form a reusable evaluation card.
Summary
The reading order of RLHF/Preference Optimization should start with "how human preferences get into training" rather than with algorithm names. Classic RLHF teaches you the basic closed loop of Reward Model, PPO and KL; the DPO family teaches you how to optimize your preferences into a lighter monitoring loss; GRPO / RLVR Teach you how rewards can be driven by reasoning training; RewardBench and replicate papers remind you that there may always be a crack between Optimization and evaluation goals. The real scientific research training is not to remember the differences in the formulas of DPO, KTO, ORPO, SimPO, and GRPO, but to ask four questions about a new paper: Is the data credible, loss What is encouraged, whether the system details are enough to reproduce, and whether the evaluation ruled out a simpler explanation.
References
Search date: 2026-08-04.
- Schulman et al., “Proximal Policy Optimization Algorithms,” arXiv, 2017. https://arxiv.org/abs/1707.06347
- OpenAI, “Learning to summarize with human feedback,” 2020. https://openai.com/index/learning-to-summarize-with-human-feedback/
- Stiennon et al., “Learning to summarize from human feedback,” arXiv, 2020. https://arxiv.org/abs/2009.01325
- OpenAI, “Aligning language models to follow instructions,” 2022. https://openai.com/index/instruction-following/
- Rafailov et al., “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” arXiv, submitted 2023, revised 2024. https://arxiv.org/abs/2305.18290
- Ethayarajh et al., “Model Alignment as Prospect Theoretic Optimization,” ICML / PMLR, 2024. https://proceedings.mlr.press/v235/ethayarajh24a.html
- Hong, Lee and Thorne, “ORPO: Monolithic Preference Optimization without Reference Model,” arXiv, 2024. https://arxiv.org/abs/2403.07691
- Meng, Xia and Chen, “SimPO: Simple Preference Optimization with a Reference-Free Reward,” NeurIPS 2024. https://papers.neurips.cc/paper_files/paper/2024/hash/e099c1c9699814af0be873a175361713-Abstract-Conference.html
- Shao et al., “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,” arXiv, 2024. https://arxiv.org/abs/2402.03300
- DeepSeek-AI et al., “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” arXiv / Nature, submitted 2025, revised 2026. https://arxiv.org/abs/2501.12948
- Yu et al., “DAPO: An Open-Source LLM Reinforcement Learning System at Scale,” arXiv, 2025. https://arxiv.org/abs/2503.14476
- DAPO project page, ByteDance Seed / Tsinghua AIR / HKU / SIA-Lab. https://dapo-sia.github.io/
- Liu et al., “Understanding R1-Zero-Like Training: A Critical Perspective,” arXiv, 2025. https://arxiv.org/abs/2503.20783
- Malik et al., “RewardBench 2: Advancing Reward Model Evaluation,” arXiv / ICLR 2026. https://arxiv.org/abs/2506.01937
- Hugging Face TRL documentation, accessed 2026-08-04. https://huggingface.co/docs/trl/en/index
- OpenRLHF documentation, “RL Training Guide,” accessed 2026-08-04. https://openrlhf.readthedocs.io/en/latest/agent_training.html
- Anthropic HH-RLHF dataset, Hugging Face, accessed 2026-08-04. https://huggingface.co/datasets/Anthropic/hh-rlhf/tree/main
- OpenBMB UltraFeedback repository, accessed 2026-08-04. https://github.com/OpenBMB/UltraFeedback
- AllenAI RewardBench repository, accessed 2026-08-04. https://github.com/allenai/reward-bench
- Tatsu Lab AlpacaEval repository, accessed 2026-08-04. https://github.com/tatsu-lab/alpaca_eval