
Date: 2026-08-13
Suitable for readers: graduate students, AI safety / alignment researcher, AI reader with engineering background
Search date: 2026-08-13
Abstract
Safety and alignment papers can read like prescriptions for how a model should answer. As a researcher, ask how the threats were discovered, how safety boundaries were defined, and how preference data enters the training objective. How do the authors evaluate appropriate refusals alongside unnecessary ones? Can the model remain useful within those boundaries? Read beyond the safety score to examine the taxonomy, data construction, attack strength, evaluation judges, error analysis, and which results can be reproduced.
This article gives an intensive reading route from Red Teaming to Controlable Generation. First read RLHF, Constitutional AI and Safe RLHF to understand how "Safe behavior" is written into the training target; then read Red Teaming, Jailbreak Benchmark, Refusal Benchmark and moderation model, understand why security evaluation is not a single accuracy; finally read Safe Complete, Constitutional Classifiers and Frontier Safety Framework, understand the trends for 2025-2026: Security research is shifting from "whether the input should be rejected" to "whether the output is as useful as possible in the constraints", from the static chat model to the multistep, tooling and high-risk capability scenarios.
Why is this topic important
For graduate students, Safety/Alignment is a very suitable direction for training scientific judgment, because it naturally requires you to deal with methods, data, evaluation and value boundaries at the same time. A model rejects a dangerous request, which may be a security ability, or it may be over-conservative; a model performs well on Jailbreak Benchmark, may be better trained or benchmark Pollution, referees too wide or not strong enough to attack; a preference optimization method to improve Harmlessness may sacrifice helpfulness, or it may just make the model better at saying "sorry".
The changes in 2025-2026 make this line more worthy of systematic intensive reading. OpenAI's safe-completion idea shifts the focus from "classified by input two to comply/refuse" to "maximizing help within the output level"; Anthropic's Constitutional Classifiers and subsequent Constitutional Classifiers++ Put constitution, composite data, input/output or Exchange Classifier, linear Probe Combined with cascading architecture, it is used to resist Universal Jailbreak; Frontier Safety Framework/Responsible for Google DeepMind and Anthropic Scaling Policy puts model evaluation into the ability threshold, deployment mitigation, external evaluation and the life cycle of the continuous red team.
This is not to say that every blog has to discuss governance. On the contrary, scientific research and training should dismantle these data back to the operable problem: how to construct an attack set? How to define a security request that should not be rejected? How to measure Harmful Compliance, False Refusal, Helpfulness and Severity? How to record the version and misjudgment of Judge Model? How to write negative results into the research questions of the next paper?
Core reading method
When reading a Safety/Alignment paper, it is recommended to create a note template with five questions.
First, what is the safety object of the paper? Is it a normal chat model, a tool call Agent, multimodal model, code model, text to image model, or a high-risk ability of Frontier Model? Different objects have different risk boundaries. The real-world actions of the chat model's Harmful Text and the agent's real-world actions are not the same type of errors. The multimodal model also introduces image input, visual JailBreak and cross-modality Abstension .
Second, where does Taxomy come from? Security papers often first define Harmful Category, Policy Category or Risk Category. Readers should check whether these categories come from public policies, regulations, expert labels, existing Benchmark, or author customizations. The finer the Taxoomy, the more conducive to error analysis; but the finer, the more prone to category overlap, cultural deviation and labeling inconsistency.
Third, what is the goal of method optimization? RLHF converts human sorting into Reward Model; Constitutional AI uses principles to drive self-critique, revision and RLAIF; Reward is decoupled from Harmless Cost; DPO / Preference Optimization bypasses the explicit RL, but still depends on the preference for quality. When reading the paper, ask: Does it optimize "reject dangerous content" or "Safe and helpful output"?
Fourth, did the evaluation look on both sides at the same time? Safety reviews should look at least two directions of Harmful Compliance and False Refusal. Just look at the Attack success rate, it is easy to reward "reject everything"; just look at helpfulness, it is easy to ignore dangerous requests. The common lesson of data such as XSTest, SORRY-Bench, StrongREJECT, HarmBench, WildGuard, SALAD-Bench, etc. is: The abstention boundary itself needs to be evaluated independently.
Fifth, is the referee reliable? Many security benchmarks use LLM-as-judge, MODERATION CLASSISIER, or rules. You want to record the Judge Model, Prompt, Temperature, Output Resolution, Manual Sampling Review, and Disagreement Case. In the reproduction report of the safety paper, the error of Judge is not the details of the appendix, but the core experimental conditions.

A reading guide to the key papers
1. From RLHF to Constitutional AI: How safe behaviors get into the training target
The first article is still InstructGPT/RLHF. It explains the basic paradigm of modern alignment training: first use artificial demonstration to do supervised fine-tuning, then let the labeler compare multiple model outputs, train the reward model with Ranking, and finally optimize the model behavior with reinforcement learning. When intensive reading, don't just remember "RLHF to make the model more obedient", but to disassemble three variables: generalization of prompt distribution, labeler guideline, and reward model.
Then read the Red Teaming and Constitutional AI of Anthropic. The former is mainly to turn Red Team from scattered attacks to data production and measurement processes: humans or models continue to look for harmful output, and then feedback failure cases to training and evaluation. The latter is important in putting the "principles" explicitly into the training process, and using self-critique, revision and ai feedback to reduce the dependence on artificial harmlessness annotation. The key training question here is: How does the principle text affect behavior? Does the model just learn to refuse more politely? Is improvements to Harmlessness accompanied by helpfulness decline?
Safe RLHF and BeaverTails are suitable for reading in the third step. BeaverTails separate helpfulness from Harmlessness, and Safe RLHF writes security constraints as Cost Model and Constrained Optimization. It reminds graduate students that Safe alignment is not single-target optimization, but constraint optimization. Do not report only one average win rate, but also report the behavior of help, security, and conflict between the two.
2. Red Teaming and Jailbreak Benchmark: The attack is not the more fancy the better
The Red Teaming paper focuses on the attack budget, attack target and success definition. Anthropic 2022 Red Teaming work releases large-scale Red-Team Attack data and discusses the relationship between model scale, training method and Red team; DeepMind's early work demonstrates the automatic generation of language models Red-Team Prompts; HarmBench and JailbreakBench further standardize the attack method, target model, Harmful Behaviors and Evaluator.
StrongREJECT is worth reading alone, as it points to a common pitfall: many Jailbreak reviews will mistake "the model does not explicitly reject" as "attack success", but the real risk also depends on whether the output is specific, executable, and harmful. For graduate students, the value of this paper is not just a dataset, but an evaluation concept: Attack Success needs to judge Willingness and Capability at the same time.
When reproducing such papers, it is recommended to start from a small scale. Select 20-50 public Harmful Prompts, prepare 2-3 Attack Wrappers, fixed target model versions, and save the complete Prompt, Response, Judge Rationale and manual review labels. Don't spread executable harmful details in an open blog, just show abstract categories, desensitization examples, and statistical abstracts.
3. Abstention and over Abstation: the safe boundary is a measurable curve
abstention is not the more the better. The core contribution of XSTest is to construct a set of Prompts that "look like a dangerous request, actual safety" to measure the Exaggerated safety behavior. SORRY-Bench then systematically evaluates the difference between the safe abstation behavior on different models and topics. WildGuard puts Prompt Harmfulness, Response Harmfulness and Refusal Detection in an open moderation tool, indicating that the security assessment not only needs to judge the input, but also judge the output of the model and the model output and abstention behavior.
This line of papers is best to read according to confusion matrix: unsafe prompt + unsafe compliance is a high-risk failure; unsafe prompt + safe refusal or safe complete is the target behavior; Safe Prompt + Refusal is False Refusal; Safe Prompt + useful answer is availability. Many papers only show average safety scores, but you should ask about the wrong examples of each grid.
OpenAI's Model Spec and Safe-Completion data can be used as a first-hand reference for "abstention style". The core of Safe Completion is not to write abstention more euphemistically, but to give useful alternative information to the safe range when it cannot provide dangerous details. When reading such materials, it is necessary to maintain a boundary awareness: they are Model behavioral norms and training ideas, not equivalent to a completely reproducible academic Benchmark; if you want to write a paper, you need to convert the specification into annotation and evaluation data tasks.
4. Preference alignment and safe data: data is more important than algorithm names
The common bottleneck of RLHF, DPO, RLAIF, Safe RLHF, and Constitutional AI is data. The security preference data contains at least four levels: the category of hazard requests, the degree of danger to the model output, the help of the safety alternative answer, and the annotator or principle to interpret the boundary. Collecting "Reject vs not rejected" will squash the problem.
The value of BeaverTails is to separate helpfulness/harmlessness; the open source code and data of PKU-SafeRLHF enables researchers to reproduce the Reward Model, Cost Model and Constrained fine-tuning; Safe RLHF-V in 2025 extends this idea to Multimodal human feedback, indicating that the security label under image input requires multi-level risk label and Guardrail. The key to future security data is not the scale itself, but the coverage of boundary samples, dual-purpose samples, cross-language samples, and Adversarial examples.
When reading a preference optimization paper, you can check it with a hard question: if the Benign Prompt contains dangerous words, but the user's intention is obvious and safe, can the model answer it? If Dual-Use Prompt may be abused, can the model provide high-level, compliant, and non-executable help? Does the training method still work if the attacker bypasses the border with coding, roleplay, translation, long context or tool feedback?
5. Controlled generation and system level protection: from model parameters to runtime guarantee
Controlled generation is not simply added a system prompt. Security control can occur in pretraining data filtering, SFT, preference optimization, abstention training, Output Shaping, Classifier Guardrail, Tool Permission, Human Confirmation and Deployment monitoring and other multiple levels. The trend of 2025-2026 is multi-layer protection: the model itself learns the security boundary, the external Classifier monitors the input and output, the system layer limits high-risk actions, and continuously updates the data through the red team.
Anthropic’s Constitutional Classifiers and Constitutional Classifiers++ are worth reading together. The first uses a constitution to generate synthetic training data for input and output classifiers designed to defend against universal jailbreaks. The second introduces a more efficient cascade and probes internal activations to reduce computational overhead and unnecessary refusals. Alongside attack success rates, examine how often harmless queries are refused, the extra compute required, how long red teaming lasted, whether anyone found a universal jailbreak, and how the demo environment differs from deployment. Recheck figures and demo findings for the version you discuss before writing them up.
Frontier Safety Framework and Anthropic Responsible Scaling Policy It reminds us that the security alignment paper is connecting with capability assessment, model Safety reporting and deployment thresholds. For graduate students, the most practical reading is to turn these frameworks into experimental lists: risk categories, ability thresholds, evaluation sets, mitigation measures, residual risks, external reviews and update mechanisms.
Methods and experiments compared
| Read Branch | representative data | Main problem | Intensive reading must be checked |
|---|---|---|---|
| RLHF / RLAIF | InstructGPT, Constitutional AI | How to write preferences and principles into training | Prompt distribution, labeling specification, Reward generalization, helpfulness loss |
| Security Constraint Optimization | Safe RLHF, BeaverTails | How to separate help and harmless | Reward/cost definitions, conflict samples, whether constraints are satisfied |
| Red teaming | Anthropic red teaming, HarmBench, JailbreakBench | How to discover and measure attacks | Attack budget, target model, success definition, manual review |
| Abstention Review | XSTest, SORRY-Bench, StrongREJECT | when to refuse, when not to reject | False refusal, harmful compliance, judge error |
| Moderation / guardrail | WildGuard, SALAD-Bench, Constitutional Classifiers | How to intercept risks in the outer layer of the system | Input/output coverage, delay, manslaught rate, bypass strategy |
| Output controllable | safe completion, Model Spec | How to stay useful within safety constraints | Dual-use classification, alternative help, severity grading |
| Frontier safety | GDM FSF, Anthropic RSP, OpenAI system cards | How high-risk capabilities enter the publishing process | Ability threshold, mitigation plan, external evaluation, version update |
How to reproduce the results
In the first stage, an abstention boundary evaluation is reproduced. Construct Safe/Unsafe pairs with XSTest or self-built small control sets. Each record contains at least prompt, expected label, model answer, whether or not, whether the answer is useful, manual remarks. Indicators should not only be counted as Accuracy, but should be counted as unsafe refusal rate, harmful compliance rate and safe false refusal rate respectively.
In the second stage, a JailBreak Evaluator is reproduced. You can choose the public implementation of StrongREJECT or HarmBench, first do not extend the attack, just run fixed Prompts and fixed model versions. Focus on documenting Judge Prompt, Judge Model, parsing rules and manual sampling consistency. If LLM JUDGE conflicts with manual judgment, write the conflict example into the body of the report instead of hiding it in the appendix.
In the third stage, a preference data analysis is reproduced. Take a small part of BeaverTails or PKU-SafeRLHF, and count the conflict between helpfulness and harmlessness. Safe completion, which needs to be clarified. This exercise is more capable of training research intuition than blindly training DPO.
In the fourth stage, do a controlled generation small experiment. Define a security policy subset, such as the high-level information boundary of Cyber or Bio; write permission, restrict, and reject three target outputs for each Prompt; let the model generate answers, and double label Severity with manual and Judge. The report must be clearly written: which content is desensitized, which model versions are subject to manual verification, and which results cannot be extrapolated to real deployments.
In the fifth stage, write reproduction report. Reporting in the security direction includes at least: retrieval date, model version, API or checkpoint, policy taxonomy, data source, desensitization strategy, JUDGE setting, manual review ratio, error type, negative result, and ethical boundary. Do not publish hazardous operation details that can be reused directly.
Common misconceptions
First, put the Safety Score as a single ranking. Safety is a multi-objective problem, and the average score masks the failure of a high-risk category.
Second, treat "reject more" as "more safe". Abstention can impair the normal use of the model in education, medical sciences, news analysis and safety research.
Third, take the fancy degree of Jailbreak Prompt as a contribution. The real contribution is a clearer threat model, a more reliable definition of success, a harder data leak and a stronger manual review.
Fourth, ignore the Judge Model drift. Model reviewers, Moderation APIs, and security policies are updated; recurring must be fixed and the search date must be specified in the text.
Fifth, leak harmful details in the blog. Safety studies can discuss categories, methodologies, and desensitization cases, and should not spread executable steps in order to demonstrate the effectiveness of the attack.
Research questions for graduate students
- Chinese XSTest style passes the abstention benchmark: construct Chinese Prompts containing sensitive words but intentional safety, and do artificial annotation consistency analysis.
- Safe Completion Scoring Rubrics: Split the "Security Alternative Help" into four dimensions: Severity, Specificity, Usefulness, and Policy Compliance.
- Red-team Evaluator Reliability Research: Compare the differences between LLM Judge, rule judgment and manual labeling on the successful judgment of Jailbreak.
- Preference Data Conflict Maps: Analyze the topic distribution of Helpfulness and Harmlessness tag conflicts in BeaverTails/SafeRLHF.
- Agent Refusal Boundary Experiment: Extend security reviews from single round text to Tool-Use Trajectory, study when to plan, ASK, ACT, or REFUSE.
Summary
The focus of the intensive reading of the Safety / Alignment paper is not the back of the algorithm name, but the establishment of the chain of evidence: first define the risk Taxonomy, then construct attack and boundary samples, then use a reviewable referee to evaluate the model output, and finally analyze the tension of Safety and help. RLHF and Constitutional AI tells us how safe behavior can enter the training; HarmBench, StrongREJECT, XSTest, SORRY-Bench and WildGuard tell us that the evaluation itself will go wrong; Safe Complete and Constitutional Classifiers represent a closer direction: to do fine-grained control at the output level, and to combine model-level training with system-level protection.
The most practical training method for graduate students starts with a small abstention boundary assessment. You don't need to train a model first, but you have to learn to write policy taxonomy, structure control prompt, fixed judge, review error samples, protect harmful details, and turn negative results into next research questions.
References
Search date: 2026-08-13. The following priority lists papers, official project pages, meeting pages and first-hand materials; model version, API behavior, security policy, demo results, the Benchmark Leaderboard and code repository status may change, please check again before reappearing. The data involving the Red Team and Jailbreak should only be used for compliance research and defensive assessment; hazardous samples are subject to desensitization.
- Ouyang et al., “Training language models to follow instructions with human feedback”, arXiv, 2022. https://arxiv.org/abs/2203.02155
- Ganguli et al., “Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned”, arXiv / Anthropic, 2022. https://arxiv.org/abs/2209.07858
- Perez et al., “Red Teaming Language Models with Language Models”, Google DeepMind, 2022. https://deepmind.google/blog/red-teaming-language-models-with-language-models/
- Bai et al., “Constitutional AI: Harmlessness from AI Feedback”, arXiv / Anthropic, 2022. https://arxiv.org/abs/2212.08073
- Anthropic, “Claude's Constitution”, updated 2026. https://www.anthropic.com/news/claudes-constitution
- Rottger et al., “XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models”, arXiv, 2023. https://arxiv.org/abs/2308.01263
- Ji et al., “BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset”, NeurIPS 2023 / arXiv. https://arxiv.org/abs/2307.04657
- Dai et al., “Safe RLHF: Safe Reinforcement Learning from Human Feedback”, arXiv, 2023. https://arxiv.org/abs/2310.12773
- PKU-Alignment safe-rlhf repository. https://github.com/PKU-Alignment/safe-rlhf
- Mazeika et al., “HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal”, arXiv, 2024. https://arxiv.org/abs/2402.04249
- Souly et al., “A StrongREJECT for Empty Jailbreaks”, arXiv, 2024. https://arxiv.org/abs/2402.10260
- StrongREJECT dataset page. https://huggingface.co/datasets/walledai/StrongREJECT
- Li et al., “SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models”, arXiv, 2024. https://arxiv.org/abs/2402.05044
- Han et al., “WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs”, NeurIPS 2024 / arXiv. https://arxiv.org/abs/2406.18495
- Xie et al., “SORRY-Bench: Systematically Evaluating LLM Safety Refusal”, ICLR 2025 project page. https://sorry-bench.github.io/
- Ji et al., “AI Alignment: A Contemporary Survey”, ACM Computing Surveys, 2025. https://doi.org/10.1145/3770749
- Lu et al., “Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges”, arXiv, 2025. https://arxiv.org/abs/2507.19672
- Zhang et al., “Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback”, arXiv, 2025. https://arxiv.org/abs/2503.17682
- OpenAI, “From hard refusals to safe-completions: toward output-centric safety training”, 2025. https://openai.com/index/gpt-5-safe-completions/
- OpenAI Model Spec repository. https://github.com/openai/model_spec/blob/main/model_spec.md
- Anthropic, “Constitutional Classifiers: Defending against universal jailbreaks”, 2025. https://www.anthropic.com/news/constitutional-classifiers
- Anthropic, “Next-generation Constitutional Classifiers: More efficient protection against universal jailbreaks”, 2026. https://www.anthropic.com/research/next-generation-constitutional-classifiers
- Google DeepMind, “Frontier safety”, latest framework page checked 2026-08-13. https://deepmind.google/frontier-safety/
- Google DeepMind, “Evaluating Frontier Models for Dangerous Capabilities”, arXiv / publication page, 2024. https://deepmind.google/research/publications/78150/
- Anthropic, “Responsible Scaling Policy”, latest page checked 2026-08-13. https://www.anthropic.com/responsible-scaling-policy