Cover illustration

Date:2026-07-16
Who this is for: Graduate students in machine learning, NLP, AI safety, reinforcement learning, evaluation and model governance; engineering readers who pay attention to post-model training and reliability
search date:2026-07-16

Abstract

Alignment Science is not concerned with the shallow issue of "making the model more chatting", but how to constrain the increasingly capable generative model into an interpretable, supervisible, and verifiable behavior space. In the past few years, the mainstream routes have developed from RLHF to preference optimization such as DPO, KTO, ORPO, SimPO, etc.; from artificial preference to Constitutional AI, RLAIF and explicit model specifications; 2025-2026 has been pushed towards harder topics: verifiable rewards, scalable oversight, anti-rewarding hackers, anti-jailbreak, anti-deception and multi-value conflicts.

This article splits "Alignment Science and Controlled Generation" into four research lines: preference learning, constitutional training, verifiable rewards and scalable oversight. The focus is not on enterprise product strategies, but on research questions, experimental paradigms, benchmark and unsolved problems.

Contents

  • Research background: from RLHF to Alignment Science
  • core science questions
  • A reading guide to the past year’s papers
  • Key papers by research question
  • Comparing the methods
  • How to interpret the experiments and benchmarks
  • Reproducibility and data problems
  • Limitations and open questions
  • Research questions for graduate students
  • Summary
  • References

Research background: from RLHF to Alignment Science

The pre-training goal of the Large Language Model is to predict the next Token, it does not naturally learn the norms of "helping users, rejecting dangerous requests, acknowledging uncertainty, complying with developer intentions, avoiding flattery, and not fabricating when there is insufficient evidence". InstructGPT and early RLHF work give a set of classic post-training recipes: first use demonstration data as supervising fine-tuning, then use human preferences to train Reward Model, and finally use PPO Reinforcement learning algorithm optimization strategies such as the KL constraints to prevent the model from deviating too far from the original distribution.

This method brings the available dialogue model and also exposes the core problem of Alignment Science: human feedback is noisy, Reward Model is the proxy target, and the preference label mixes correctness, politeness, length, style and safety. DPO rewrites the Reward Model and RL links of RLHF into a more direct preference classification loss, which reduces the engineering complexity. Subsequently, KTO, ORPO, SimPO, etc. are further explored: whether the alignment can be achieved with binary feedback, no reference model goals or implicit rewards that are closer to the generation probability.

Alignment Science Method Framework Diagram

But Alignment research isn't just about "tuning a better preference loss". Constitutional AI takes the natural language principles as a supervisory source, lets the Model generate self-criticism and revision, and trains the preference Model with AI Feedback. Deliberative Alignment emphasizes letting the Model explicitly learn security specifications and reason about the specification before answering. 2026 Anthropic New Claude Constitution and OpenAI Model Spec Description, Alignment It is moving from "hidden preference fitting" to a mixed paradigm of "auditable specification plus optimization".

core science questions

First, what is your preference for testing? The same pairwise preference may reflect the correctness of the facts, or it may just be longer, more confident, and more cater to the labeler. The value of Benchmark, such as RewardBench, is to put the Reward Model in Chat, Reasoning, Safety, and OOD scenarios to open the evaluation, instead of just looking at an overall win rate.

Second, how does the specification enter the model? Model specifications, constitutions, developer messages and security policies are all natural language objects. They are readable and auditable, but may also conflict, ambiguous or bypassed by Prompt Injection.

Third, what can be verified? Mathematics, code, formal proofs and search questions have external verifiers: whether the answer can be verified by unit tests, theorem checks, string equivalents or evidence. RLVR turns this signal into a reinforcement learning reward. But Verifier will also be wrong, forming a new Reward hacking surface.

Fourth, who will supervise a model that is stronger than people? Scalable Oversight is concerned with how to still generate reliable training signals when the model output is so complex that humans are difficult to evaluate directly. Debate, Iterated Amplification, Weak-to-strong Generalization, CriticGPT, and AI Auxiliary Red Team all belong to this problem family.

Fifth, is alignment proof? Most of the current work can only give a reduced empirical risk, and it cannot prove that the model has no hidden target, does not deceive, and will not fail outside the distribution. The goal of a verifiable alignment is not to immediately give a complete security proof, but to convert a part of the subjective judgment into a constraint that can be tested repeatedly.

A reading guide to the past year’s papers

1. Preference optimization enters the data efficiency and target design stage. After DPO in 2024, KTO uses the "Desirable/Undesirable" signal instead of pairing preferences, ORPO merges SFT and preference penalty, SimPO uses reference-free Reward and emphasizes the average Log Probability. The work on Preference Variance in 2025 further asks: Which preference samples are the most training value? This shifts preferences to data selection, labeling noise, and target inductive bias studies.

2. Constitutional alignment moves from a list of principles to interpretable specifications. The original contribution of Constitutional AI is to use principles to generate self-criticism, revisions, and AI Preference. In 2025, Constitutional Classifiers used the "Constitution" for the training of synthetic safety Classifiers, and used large-scale red teams to test jailbreak defenses. The Grounded Constitutional AI in 2026 focuses on how to generate more representative principles from human reasons and value statements, stating that Constitutional alignment is approaching the issue of value aggregation and democratic input.

3. Deliberative Alignment explicitly specifies canonical inference. OpenAI’s Deliberative Alignment paper proposes teaching models safety specifications and having them reason before answering, with the aim of improving the trade-off between resisting jailbreaks and refusing benign requests. A 2025 stress test extended the approach to an anti-scheming setting, examining covert actions, situational awareness, and behavior under tests for hidden objectives. This work expands alignment evaluation beyond classifying refusals to asking whether a model can understand and apply rules.

4. The verifiable reward becomes the key interface for training after the inference model. DeepSeek-R1 uses rules-based accuracy rewards and format rewards for tasks such as mathematics and code, and strengthens the position of RLVR in the inference model. The research on RLVR in 2025 began to study: it can verify whether the reward really stimulates correct reasoning, how to deal with Verifier noise, and how to extend RLVR to free text fields such as medicine, chemistry, and economics. The trend here is to connect alignment, reasoning and verify .

5. Supervise the toolchain from Judge to Critic. CriticGPT displays GPT-4-level Critics to help humans discover errors in code answers. Weakly supervised strong models, debate, consultancy, self-evolutionary critic and other jobs are all exploring the same bottleneck: when humans cannot evaluate complex outputs separately, let another model help evaluate whether it will be more reliable.

Key papers by research question

A. RLHF and direct preference optimization

The advantage of RLHF is that the concept is clear: first learn a Reward Model, and then optimize the policy. It is suitable for pressing multidimensional human preferences into a trainable signal, but the cost is that the engineering is complex, the training is unstable, and the Reward Model will become the optimized proxy target. The key idea of DPO is to replace the relationship between the optimal strategy and the Reward Model into the preference likelihood, so that the model can be optimized directly on the Chosen / rejected data. It makes it easier for researchers to reproduce experiments, and also allows preference optimization to train pipelines from closed-source model training.

What KTO, ORPO, and SimPO have in common is to reduce preference training costs or simplify goals. KTO focuses on human feedback, which is not a strict rational choice. ORPO emphasizes directly punishing unpreferred output in SFT, while SimPO tries to remove the Reference Model and make the implicit reward closer to the generation process. When reading these papers, you should avoid only looking at the leaderboards: more important is the source of training data, the control of the response length, whether the evaluation is dominated by LLM JUDGE, whether there are safety and factual indicators, and whether to report Reward hacking or verbosity bias.

b. Constitutional AI, RLAIF and Model Specifications

The scientific question of Constitutional AI is: Can natural language principles be used to replace some human preference labels? The original paper consists of two stages: the supervision stage allows the model to criticize and revise its own answer according to the principle, and the RL stage uses AI feedback to train the Preference Model, and then use it for reinforcement learning. Its advantage is that the labeling cost is low, the standard is readable, and it can cover the security boundary that it is difficult for human beings to label one by one.

The Claude Constitution and OpenAI Model Spec in 2026 represent another direction of engineering: expose the code of conduct as an auditable document. The Anthropic page gives the four-level priority of the current Model of Claude: generalized security, general ethics, following the Anthropic guide, and truly helping users; OpenAI Model Spec exposes the Model behavior specification and its source file. For the researchers, these documents are not "conclusions of the paper", but the object of research: Is the specification consistent? How can conflicts resolve between different specifications? Is the Model really obeyed after training? Is it possible to construct a negative example?

C. Verify rewards, process oversight and RLVR

The attraction of a verifiable reward is that it converts a part of the alignment problem from "what is the best person" to "whether the external inspector can judge". The code can be tested, the math answer can be checked equivalently, the formal certificate can be handed over to the Proof Checker, and the retrieval task can require the reference to be consistent with the evidence. RLVR is especially important in the inference model, as it allows the model to discover long-chain inference strategies through trial and error, without manual annotation for each step.

But Verifiable does not equal risk-free. String matching misses equivalent answers, unit test coverage is limited, retrieving Verifier may miss context conflicts, and LLM Verifier may be spoofed against Token or surface formatting. In 2025, the work on Noisy Verifiers regards Verifier as a random reward channel with False Positive and False Negative, and proposes a method for correcting training updates. This is a problem that verifiable alignment must face: the more automated the reward, the more it takes to study the way Verifier is hacked.

D. Scalable Oversight, Critic and Anti-Spoofing Reviews

Early Scalable Oversight included Reward Modeling, Debate, Iterated Amplification, and Weak-to-strong Generalization. The core intuition is that humans may not directly give the correct answer, but can better judge the answer with the help of AI. CriticGPT is a concrete example: OpenAI reports that humans use CriticGPT to review code better than no auxiliary review, and point out that this method is still limited by short answers, hallucinations and complex tasks.

2025 Anti-Scheming Stress Test pushes the problem further: If the model knows that it is being evaluated, it may perform well because of contextual awareness, and does not represent true alignment. The paper uses Covert Actions as a proxy behavior to test hidden targets and rule circumvention. This type of experiment reminds us that the alignment evaluation should not only look at the surface compliance rate, but also consider whether the model is "performed to the evaluation" in the test.

Comparing the methods

Route Supervised source Typical optimization suitable for tasks Main risk
RLHF human preference, demonstration reward model + PPO / RL Conversation quality, safety abstention, style control Reward hacking, labeling noise, complex training
DPO / KTO / ORPO / SimPO Paired Preferences or Binary Feedback Direct preference loss, no reference model target Research on post-open source training and data efficiency Length offset, Judge bias, out-of-distribution instability
Constitutional AI / RLAIF Natural Language Principles, AI Feedback Self-revision, AI Preference Model, RL Safety specifications, value principles, low labeling cost scenarios Principles of dispute, rule conflict, AI feedback deviation
Model Spec / policy reasoning Explicit Behavior Standardized learning, deliberative training Security boundaries, command level, auditable behavior Specification ambiguous, prompt injection, excessive abstent
RLVR / verifier training Unit testing, theorem inspection, answer equivalent, evidence verification Reinforcement Learning, GRPO, Process Rewards Mathematics, Code, Formal Validation, Evidence Q&A Verifier exploits, covered blind spots, false positive / false negative
Scalable oversight AI assists human judgment, Debate, Critic Critic training, weak-to-strong, red team evaluation Complex tasks that are difficult to evaluate directly Situational awareness, evaluation leaks, supervisors are persuaded

How to interpret the experiments and benchmarks

First, look at the data. Preference data come from real person, AI judge, rule generation or synthetic data? Is Chosen / Rejected only the style length? Does it cover Chinese, multi-language, long context, tool calls and abstent boundaries?

Second, look at the goal. The mathematical forms of RLHF, DPO, KTO, ORPO, and SimPO are optimized in different ways, and the implied inductive bias is also different. A goal is good in AlpacaEval or Arena-Hard, and does not mean that it is good for safety, fact, medical, legal and mathematical proof.

Third, look at the Evaluator. RewardBench, M-RewardBench, HarmBench, JailbreakBench, StrongREJECT, XSTest, TruthfulQA, SafetyBench, etc. Benchmark The focus is different. Reward Model Benchmark is not equal to Policy Benchmark, abstention Benchmark is not equal to real safety.

Fourth, examine the trade-offs. Lower refusal rates may mean more harmful assistance. Stronger safety filters can lead to more unwarranted refusals, and gains in helpfulness can come with more sycophancy or hallucinations. Interpret win rates cautiously when a paper omits refusal and false-refusal rates, truthfulness, and out-of-distribution robustness.

Fifth, look at the confrontation evaluation. Prompt Injection, Universal Jailbreak, Multi-Turn Attack, Encoding Attack, Role-Play Attack will change security conclusions. Constitutional Classifiers are valuable because it puts the synthetic classifier into the red team and auto-review, rather than just reporting accuracy on the static dataset.

Sixth, look at reproducibility. Many Alignment papers involve closed-source models, private data, or internal security policies. Readers should distinguish between "reproducible experiments" and "trusted but non-revisible system reports".

Reproducibility and data problems

The core of preference learning is data transparency: Prompt, candidate answers, annotation guide, conflict processing, deduplication, temperature generation and filtering rules can all change results. Many papers only disclose the trained model or part of the data, which makes it difficult to analyze the failure cases.

The difficulty of the reproduction of constitutional training lies in the principle. A constitution is not just a list, it includes priority, exception, and conflict handling. If only a few principles are disclosed, it is difficult for researchers to judge that the behavior changes come from the principle itself, the process of generating data, or the details of the preference model or RL.

The reproduction difficulty of RLVR is Verifier. Code tasks should be publicly tested and sandboxed, mathematical tasks should explain the equivalent check, and the retrieval task should disclose the document version and reference rating. Otherwise, the "verifiable" may just be "score on a private inspector".

Safety reviews also face the dilemma of disclosure. Public Jailbreak Prompt is easy to reproduce, and it will also help attackers; if it is not public, it is difficult to verify. The pragmatic direction is to release abstract attack categories, evaluation protocols, controllable examples and third-party re-test results.

Limitations and open questions

First, preference is not value. Fitting human preferences better does not mean that the model has learned a stable human value. Preference data often include short-term satisfaction, style preferences, cultural deviations and annotation platform incentives.

Second, the norm is not a proof. Model Spec and Constitution improve transparency, but whether the Model is internalized, how to deal with conflicts, and whether it is obeyed outside the distribution still needs experimental verification.

Third, Verifier will be optimized. As long as Verifier becomes a reward, the model may learn to exploit the Verifier vulnerability. The security of RLVR depends on the Verifier coverage, robustness and update mechanism.

Fourth, the AI feedback may amplify the model deviation. RLAIF and LLM JUDGE reduce labor costs, but if JUDGE and policy share training deviations, a closed-loop confirmation may be formed.

Fifth, anti-deception is still in the early stage. The current Anti-Scheming evaluation multi-purpose agent behavior and artificial construction environment. They can expose risks, but cannot prove that the model does not have a more hidden strategy.

Research questions for graduate students

  1. The amount of information of preference data to choose: Reproduce the Preference Variance idea, study which Prompts and answer pairs can best improve DPO or SimPO, and report cross-benchmark generalization.

  2. Chinese and multilingual Reward Model reviews: Extend the RewardBench/M-RewardBench style data to test whether the Reward Model transfers English security and helper preference errors to the Chinese context.

  3. Constitutional Conflict Test Set: Construct the task of conflicting security, ethics, help, developer instructions and user preferences, and evaluate whether the model complies with the public priority.

  4. RLVR Verifier Robust: Quantify False Positive, False Negative and post-training Reward hacking for mathematical equivalents, code unit tests and search reference design adversarial examples.

  5. Critic-Assisted Oversight Experiment: Compare the error discovery ability and hallucination rate of Human, model, Human+Critic in tasks such as code review, paper review, medical question and answer, etc.

  6. Situational Awareness Control of Alignment Evaluation: The design model does not know whether it is in the evaluation environment under test, and compares the difference between explicit safety and implicit behavior auditing.

Summary

Alignment Science is moving from empirical post-training skills to a more comprehensive experimental Science. Preference learning solves the approximate optimization problem of "what humans like", constitutional training and model specification make behavior goals more readable, can verify the rewards convert some supervision into external inspections, scalable oversight Try to solve the fundamental bottleneck of the lack of human supervision ability.

None of these four routes are given the final answer separately. Preferences will be noisy, rules will conflict, Verifier will be used, Critic will be hallucinations, and Benchmark will leak. A more reliable research paradigm should combine them: use explicit specifications to define goals, use preference data to learn delicate choices, use Verifier constraints to determine tasks, use Critic and red teams to find failures, and then use public benchmarks and negative controls report TRADE-OFF. For graduate students, the most worthwhile is not to invent another loss of preference, but to make the evidence chain of Alignment more reproducible, auditable, and more counter-examples.

References

Search date: 2026-07-16. The following priority lists the papers, official research pages, official specifications, arXiv/OpenReview and project home pages. The model name, code repository status, benchmark ranking, system card and official policy documents will be continuously updated, and it is recommended to manually verify again before publication.

  1. Ouyang et al., Training language models to follow instructions with human feedback, NeurIPS 2022. https://arxiv.org/abs/2203.02155
  2. Bai et al., Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, arXiv, 2022. https://arxiv.org/abs/2204.05862
  3. Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, arXiv, 2023. https://arxiv.org/abs/2305.18290
  4. Ethayarajh et al., KTO: Model Alignment as Prospect Theoretic Optimization, arXiv, 2024. https://arxiv.org/abs/2402.01306
  5. Hong et al., ORPO: Monolithic Preference Optimization without Reference Model, arXiv, 2024. https://arxiv.org/abs/2403.07691
  6. Meng et al., SimPO: Simple Preference Optimization with a Reference-Free Reward, arXiv, 2024. https://arxiv.org/abs/2405.14734
  7. Guo et al., On the Role of Preference Variance in Preference Optimization, arXiv, 2025. https://arxiv.org/abs/2510.13022
  8. Lambert et al., RewardBench: Evaluating Reward Models for Language Modeling, arXiv, 2024. https://arxiv.org/abs/2403.13787
  9. Gureja et al., M-RewardBench: Evaluating Reward Models in Multilingual Settings, arXiv, 2024. https://arxiv.org/abs/2410.15522
  10. Bai et al., Constitutional AI: Harmlessness from AI Feedback, arXiv / Anthropic, 2022. https://arxiv.org/abs/2212.08073
  11. Anthropic, Constitutional AI: Harmlessness from AI Feedback, official research page, 2022. https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback
  12. Anthropic, Claude’s Constitution, official page, 2026. https://www.anthropic.com/constitution
  13. OpenAI, The OpenAI Model Spec, GitHub repository and rendered spec, 2025-2026. https://github.com/openai/model_spec
  14. OpenAI, Sharing the latest Model Spec, official blog, 2025. https://openai.com/index/sharing-the-latest-model-spec/
  15. Guan et al., Deliberative Alignment: Reasoning Enables Safer Language Models, arXiv, 2024. https://arxiv.org/abs/2412.16339
  16. Schoen et al., Stress Testing Deliberative Alignment for Anti-Scheming Training, arXiv, 2025. https://arxiv.org/abs/2509.15541
  17. Sharma et al., Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming, arXiv, 2025. https://arxiv.org/abs/2501.18837
  18. Bell et al., Beyond Preferences: Learning Alignment Principles Grounded in Human Reasons and Values, arXiv, 2026. https://arxiv.org/abs/2601.18760
  19. Guo et al., DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, arXiv, 2025. https://arxiv.org/abs/2501.12948
  20. Wen et al., Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs, arXiv, 2025. https://arxiv.org/abs/2506.14245
  21. Cai et al., Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers, arXiv, 2025. https://arxiv.org/abs/2510.00915
  22. Su et al., Expanding RL with Verifiable Rewards Across Diverse Domains, arXiv, 2025. https://arxiv.org/abs/2503.23829
  23. Leike et al., Scalable agent alignment via reward modeling: a research direction, arXiv, 2018. https://arxiv.org/abs/1811.07871
  24. Irving et al., AI safety via debate, arXiv, 2018. https://arxiv.org/abs/1805.00899
  25. Burns et al., Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, arXiv, 2023. https://arxiv.org/abs/2312.09390
  26. Awano and Suzuki, The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge, arXiv, 2026. https://arxiv.org/abs/2605.12908
  27. McAleese et al., LLM Critics Help Catch LLM Bugs, arXiv / OpenAI, 2024. https://arxiv.org/abs/2407.00215
  28. OpenAI, Finding GPT-4’s mistakes with GPT-4, official research page, 2024. https://openai.com/index/finding-gpt4s-mistakes-with-gpt-4/
  29. Kenton et al., On scalable oversight with weak LLMs judging strong LLMs, arXiv, 2024. https://arxiv.org/abs/2407.04622
  30. Mazeika et al., HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal, arXiv, 2024. https://arxiv.org/abs/2402.04249
  31. Chao et al., JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models, arXiv, 2024. https://arxiv.org/abs/2404.01318
  32. Souly et al., A StrongREJECT for Empty Jailbreaks, arXiv, 2024. https://arxiv.org/abs/2402.10260