
Date:2026-08-03
Who this is for: Graduate students, new scientific researcher who is ready to train after making a large model, readers with engineering background but want to systematically understand SFT and instructions to follow
Prerequisites: Understand language model pretraining, cross entropy loss, prompt, few-shot/zero-shot, Transformer basic structure and common LLM benchmark
Abstract
Instruction Tuning is most likely to be wrong in one sentence: it is not just "training the model with quiz data", it is not equivalent to a complete RLHF, DPO or GRPO. More precisely, it is a supervised post-supervised training method that pulls the pre-trained language model from "predicting the next token" to "understand the description of natural language tasks and output as required". Earlier Flan and T0 prove that multitasking natural language instructions can significantly improve zero-shot generalization; InstructGPT puts SFT into "demonstration data + preference sorting + RLHF" human feedback process; Self-Instruct and Flan Collection drive synthetic Instruction data; LIMA proposes a strong hypothesis that "small amount of high-quality alignment data may also be enough for church interaction formats". By 2025-2026, Tulu 3, Qwen3, GRAPE, IFEval/IFBench/LexInstructEval Work to advance the problem to open formula, data adaptation, verifiable constraints and stricter evaluation.
This article gives a intensive reading route of Instruction Tuning. Readers need to learn: first distinguish SFT, Instruction Following, Preference Optimization and RLVR; then dismantle the data source, task mixing, training objectives and evaluation evidence; and finally transform a paper into a small experiment that can be reproduced by itself.
Contents
- Why Instruction Tuning is worth reading
- Core reading method: start with "task-data-train-evaluation" four-grid table
- A reading guide to the key papers
- A closer look at key papers
- Methods and experiments compared
- How to reproduce the results
- Common misconceptions
- Research questions for graduate students
- Summary
- References
Why is this topic important
Pre-training language models learn language distribution and world knowledge, but the input to the model by users is usually not "please continue this web page", but "please summarize, rewrite, translate, reason, write code, press JSON to output, no more than 100 words". The scientific research value of Instruction Tuning is in this dislocation: it studies how to use natural language task description and target output, so that the model is more like an interactive system on new tasks that have not been seen before.
For graduate students, this theme is especially suitable as a post-training entry, because it connects three levels. First, the method is simple: the most basic version is to do Teacher Forcing on the Instruction-Output Pairs, training with cross entropy. Second, the variables are complex: also called SFT, the data may come from manual annotation, templated NLP dataset, strong model distillation, self-generated, user log or preference screening; the model may be encoder-decoder, decoder-only, dense, MoE, multimodal; reviews may be MMLU, BBH, MT-Bench, AlpacaEval, IFEval or artificial preference. Third, it is easy to generate reproducible topics: you can also study data quality, task coverage, template design, constraint compliance and failure mode without training the 70B model.
The core judgment of this route is: Is an Instruction Tuning paper proving "model learning tasks", or only learning "answer format"? Does the data volume work, or does data quality, task diversity, model scale, teacher model, chat template, evaluation prompt, or benchmark contamination work?

Core reading method: start with four tables
When reading such papers intensively, don't rush to read the rankings. It is recommended to fill in a four-sheet form for each paper:
- Task: Task definition. Is the Instruction in the paper a traditional NLP task template, an open user request, a multi-round dialogue, a format constraint, a tool call, or a domain task? Are training tasks and testing tasks the same distribution?
- Data: Data source. Is the data written manually, transcribed from public datasets, generated by strong models, model bootstrap, and screened by preference data, or mixed recipes? Do you report deduplication, filtering, licenses, pollution checks and versions?
- SFT: training target. Is it full-parameters fine-tuning, LoRA/QLoRA and other PEFTs, or is it the first step in multi-stage training? Does loss only count assistant tokens? Is Chat template, special token, packing, learning rate, batch size reproducible?
- Eval: Evidence intensity. Is the evaluation Held-out Task, Held-out Instruction, Held-out Constraint, or the same origin Benchmark? Is it program validation, human evaluation, LLM-as-judge, or traditional Accuracy? Is there an ablation, negative result and error analysis?
After filling out the four tables, read the result table. Instruction Tuning The table of papers often rubs together multiple abilities: knowledge quizzes, mathematics, code, dialogue preferences, safe abstent, format compliance, long context. You have to take them apart, because a data mix may improve the Instruction format, but it damages the facts; improves the reasoning benchmark, but reduces the concise answer; improves the open question and answer in English, but it does not help Chinese, code or multiple constraints.
A reading guide to the key papers
First stop: FLAN.
2021 Google Finetuned Language Models Are Zero-Shot Learners It is proposed to rewrite multiple NLP tasks into natural language instructions for the fine-tuning language model. Its focus is not the "FLAN model name", but the experimental paradigm of "generalization of cross-tasks with natural language task descriptions": training tasks, no tasks, instruction templates, model scale and ablation.
The second stop: T0 and Super-NaturalInstructions.
T0 uses large-scale multitask prompted training to study zero-shot task generalization. Super-NaturalInstructions extends this question to more than 1600 NLP tasks with expert-written instructions. Look closely at what is held out: the test may include unseen tasks, types of instruction or whole task families.
The third stop: InstructGPT.
In 2022 OpenAI's InstructGPT paper put SFT in a more complete human feedback pipeline: first use the indexer demonstration data to supervise fine-tuning, and then collect model output sequencing training Reward Model, finally optimized with RLHF. When intensive reading, it should be clear: SFT is the baseline and initialization, and RLHF is the optimization of subsequent preferences. Do not simplify the entire paper into "Instruction Tuning paper".
Fourth station: Scaling Instruction-Finnetuned Language Models and Flan Collection.
The Flan-PaLM/Flan-T5 series shifts the problem to Scale: how the task number, model scale, chain-of-thought data, how the prompting setup and multi-benchmark performance work together. The value of Flan Collection is to expose tasks, templates and methods, so that subsequent research can analyze data recipes more systematically.
Fifth station: Self-Instruct.
The key question of Self-Instruct is: when the manual instructions are limited, can the model generate instruction, input, and output, and then filter and use it to use fine-tuning? This line opens up the large-scale use of "synthetic instruction data", and also leaves problems such as quality, deviation, pattern collapse and teacher-student distribution mismatches.
Sixth stop: LIMA.
LIMA uses a small number of high-quality samples to make SFTs, and proposes the strong assumption that "most of the knowledge and abilities come from pre-training, and the main teaching interaction style is in the alignment stage". When reading it, don't just remember "1000 pieces of data", but look at sample screening, base model, manual evaluation, comparison objects and the security, complex inference and verifiable constraints that it cannot cover.
The seventh stop: After opening the training formula.
Tulu 3 presents modern open post-training into more complete engineering research objects: tips and data planning, SFT, DPO, RLVR, evaluation and pollution removal. The official technical data such as Qwen3 and Llama 3/4 also shows that the modern Instruct model is usually not a single-stage SFT, but the result of a combination of multi-stage data screening, SFT, preference optimization, RL, long context and tool capabilities.
Eighth Stop: Data adaptation and verifiable instructions for 2025-2026 are followed.
GRAPE reminds us that the strongest teacher's answer is not necessarily the most suitable for a target model; "fit" can use signals such as the probability of the target model itself. IFEval uses program to verify format constraints, IFBench further points out that the model may be overfitting a few Constraint templates, AAAI-26's LexInstructEval Then, the fine-grained vocabulary is constrained into a checkable rule. These work push Instruction following from "subjective good" to "whether it can be verified and generalized".
A closer look at key papers
FLAN: Reading cross-task generalization, not reading single-point scores
The core design of FLAN is to convert many tasks into natural language instructions, and then evaluate the performance of zero-shot on the task. It is recommended to draw three columns during intensive reading: training task type, test task type, instruction template. The real question is: the model improvement comes from "seeing more tasks" or from "seeing the natural language mission description"? Ablation in the paper points out that the number of tasks, model scale and natural language instructions are all critical, which is a small problem that readers should reproduce.
Minimum reproduction can choose 8-12 public categories/question answers/summary tasks, and use several of them as held-out tasks. Train a small T5 or small Decoder-only model and compare three groups: no instruction template, single template, multi-template. Your goal is not to reproduce the absolute score of the original paper, but to observe whether the "unseen task generalization" changes with the template diversity and the number of tasks.
2. T0 / Super-NaturalInstructions: read Benchmark construct
The value of t0 and Super-NaturalInstructions is to extend the instruction followinging from a single dataset to a collection of tasks. When reading such papers, the dataset section is more important than the result table: how are tasks grouped? How to split training/test? What counts as unseen task? Will multiple prompts for a task leak to the test? Is the evaluation EXACT MATCH, ROUGE, BLEU, F1 or manual judgment?
If you want to do follow-up, you can start with "task split". Many papers claim zero-shot, but there may be semantic similar tasks in the Training Set; it may just be a Prompt expression, not a real ability. A reliable topic is to redo the task family, such as separating all emotional classifications, factual quiz, and generative rewriting, and observe which boundary the model generalization has crossed.
3. InstructGPT: Read the role of SFT in the full alignment chain
InstructGPT is often regarded as an introductory RLHF paper, but in the Instruction Tuning route, its intensive reading focuses on how SFT has become a starting point for preference optimization. The paper first collects the labeler-written definitions to do supervised learning, then uses the model output sort to train the Reward model, and finally do RLHF. The research question here is: What has SFT already taught the model? In which Prompt distributions continue to improve RLHF? How to deal with the regression of public NLP Benchmark?
Do not attempt to fully replicate OpenAI's private data and human feedback processes when reappearing. A more reasonable training is to use public Instruction data as SFT, and then use small-scale preference data or regular verification data to do DPO/RLVR, and compare the difference between SFT-only and Preference stage. Reports must be separated: helpfulness, format follow, trustfulness, toxicity, or safety is not the same indicator.
4. Self-Instruct: read the filter chain of synthetic data
The Self-Instruct method is very suitable for dismantling: starting from a small number of seed tasks, generate a new instruction, judge the task type, generate instances, and then filter similar, invalid or low-quality samples, and finally train the model. When intensive reading, you should focus on two links: whether the generation increases the real task diversity, and whether the filter is strong enough.
Graduate students can do a small reproduction experiment: let an open source Instruct model generate 2K-5K Instruction-Output for a specific field Pairs, respectively train four versions of unfiltered, deduplicated filtering, regular filtering, and manual sampling filtering. Use the Held-out Human-Written Prompts and IFEval style constraints to evaluate whether the model will be more "responsible" or just more imitating the teacher's tone.
5. LIMA: Read the boundary of "Mass is greater than quantity"
LIMA owes much of its influence to a counterintuitive question. If a pretrained model is already strong, can a few high-quality alignment examples teach it how to interact? The paper helps distinguish learning a capability from learning how to express it. If the model already knows the material, SFT may simply elicit an assistant-style response. Whether a few examples still suffice for new domain knowledge, strictly verifiable constraints, reliable refusals, or complex multi-step reasoning calls for fresh evidence.
Avoid two extremes when reading LIMA: don't understand it as "the less data is better", and don't deny large-scale instruction data just because it has few samples. The better question is: what type of data must be of high quality and small amounts? What abilities need to cover a lot of long tail tasks? What ability cannot be solved by SFT alone?
6. Tulu 3 / Qwen3: Read Modern Open Recipes
The official data of Tulu 3 emphasizes open data, code, training formula and evaluation, which is a good sample of post-research training. It splits Post-Training into prompt planning and synthesis, SFT, preference data and DPO, RLVR and other stages. Qwen3 official repository shows the 2025 Qwen3-2507 Instruct/Thinking variants continue to strengthen Instruction Following, Reasoning, Tool Usage, alignment and long context, and provide training entry instructions such as SFT/DPO/GRPO.
This type of material should be read as a "technical report", not as a general paper. You need to check the model card, the transparency of the training data, whether the code can run, whether the benchmark can be reproduced, and whether there is a contamination description. If the official only gives the summary without data details, the article should indicate "waiting for manual verification", and the published copy cannot be written as an experimental conclusion.
7. IFEval/IFBench/LexInstructEval: read verifiable constraints
Traditional Instruction Following evaluations often rely on human preferences or LLM-as-judge, which are expensive and may be biased. IFEval narrows the problem to programmable constraints, such as length, keywords, format, case, etc.; IFBench further points out that the model may overfit ifval fixed Constraint Templates, which cannot be generalized to new constraints; LexInstructEval constructs a more fine-grained vocabulary evaluation using regular syntax.
This line is especially suitable for reproduction training: you can define 20 Chinese verifiable constraints, such as "exactly three paragraphs", "no use of a certain word", "every one starts with a verb", "exports a legal JSON and the field order is fixed". Then evaluate the Strict/Loose Accuracy of several open source models, and analyze the failure types. This is more controllable than scoring with a large model.
Methods and experiments compared
| Papers/Data | What to read | data form | Training/Evaluation Focus | Minimum reproduction |
|---|---|---|---|---|
| FLAN | Can multitasking natural language instructions improve zero-shot | Traditional NLP Task Template | held-out task generalization | multitasking model + template ablation |
| T0 | Mass Prompted Multitask Training | PromptSource/Multi-task Prompt | No task generalization | Test by task family |
| Super-NaturalInstructions | Instruction Benchmark | 1600+ NLP Task Description | task type coverage | Sampling task redo train/test split |
| InstructGPT | Division of SFT and RLHF | Demonstration + Preference Sort | helpfulness/truthfulness/safety | SFT-only vs DPO/RLVR small experiment |
| Self-Instruct | Synthetic instruction data bootstrap | seed task + model-generated data | Filtration and Diversity | Filtering Policy Ablation |
| LIMA | Boundary of a small number of high-quality samples | curated prompt-response | alignment style vs capability | 1K/5K/20K high-quality data comparison |
| Tulu 3 | Post-open training formula | Hybrid Public/Synthetic Data | SFT+DPO+RLVR+ Decontamination | Reuse open-instruct to do 1B/3B experiments |
| GRAPE | Whether the data is adapted to the target model | Multi-teacher response candidates | target-model fit | Press Target likelihood to choose an answer |
| IFEval/IFBench | verifiable instructions follow | Rule constraints Prompt | strict/loose programmatic eval | Self-built Chinese Constraint Suite |
| LexInstructEval | lexical-level fine-grained constraints | Rule syntax + human-in-loop | lexical control | Generate Rules + Verifier |
How to reproduce the results
Experiment 1: Comparison of SFT data quantity and quality.
Select a 0.5B-3B base model, and prepare three sets of data: small and manual screened data, larger synthetic data, deduplication and regular filtered composite data. Keep training steps, learning rate, chat template, tokenizer and eval set unchanged. Report training Loss, Held-Out Prompts, IFEval style constraints and manual sampling error analysis.
Experiment 2: Teacher-student fit.
For the same batch of Instruction, let multiple teachers generate answers. Calculate the average Log Probability or length-normalized loss of the answer with the Target Base Model, and then select the answer training for "Easier Learning of the Target Model". This is a miniature version of GRAPE's ideas. The point is not to prove that GRAPE must be better, but to observe whether the complex answer of strong teacher will make the small model learn badly.
Experiment 3: Generalization of instructions.
Instead of randomly dividing the samples, they are divided by task family, constraint type or domain. For example, training only contains length and keyword constraints, test format and vocabulary constraints; train English commands, test Chinese instructions; train a single round, and test multiple rounds. This distinguishes Memorization and Generalization.
Experiment 4: SFT VS preference optimization.
Do SFT first, then use small-scale Chosen/Rejected Pairs to do DPO, or use the rule Verifier to do RLVR. The open question and answer quality, verifiable format compliance and safe abstention are measured separately. The improvement of many models is not unified improvement, but one type of behavior is optimized, and the other type of behavior may be degraded.
Save each reproduction:
- Data files Hash, data sources, licenses, filtering scripts, and sampling seeds.
- Chat template, special token, maximum length, whether to pack, loss mask rules.
- Model CheckPoint, training framework version, graphics card, precision, learning rate, batch size, warmup, epoch.
- eval prompt, decoding parameters, verifier code, manual evaluation rubric.
- Failed samples: not following the format, answering questions, overlying length, hallucination, abstention error, multiple rounds of forgetting.
Common misconceptions
Myth 1: All post training is called Instruction Tuning.
SFT, DPO, PPO/RLHF, RLVR, Reject Sampling, Distillation and Safety Fine-Tuning are different stages. If the paper is not clear about the training target, it is difficult to judge the contribution.
Misunderstanding 2: Just look at the number of data.
100,000 low-quality synthetic data may not be as good as 1,000 high-quality samples; but a small number of samples may not cover long-tail constraints, domain tasks and security boundaries. Look at quality, diversity, coverage and target model adaptation at the same time.
Misunderstanding 3: Improve the tone of the model as an improvement in ability.
SFT can easily make the model more like an assistant, more typography, and more able to say "of course". This does not mean that fact, reasoning, code correctness, or tool usage really improves.
Misunderstanding 4: Ignore Chat Template.
Many reproduction failures are not because the method is invalid, but the Template, EOS, Role Token, Loss mask or tokenizer is inconsistent. The input format of the Instruction Tuning itself is part of the method.
Misunderstanding 5: Use the same origin BENCHMARK to prove generalization.
If the training data comes from an IFEval template, Alpaca style issue, or a near-neighbor data from a Leaderboard, then the Benchmark boost may just be a template familiar. Requires Held-out Constraint, Held-Out Task and pollution checks.
Research questions for graduate students
- Chinese verifiable instructions follow Benchmark. Construct 30-50 Chinese formats, vocabulary, paragraphs, references, and JSON constraints, providing Verifier and Strict/Loose evaluation.
- Data selection for the target model. Compare the impact of Strongest-teacher, random-teacher, target-likelihood, and diversity-aware selection on the small model SFT.
- Instruction data Deduplication and pollution audit. Contamination detection at the N-GRAM, Embedding, and Prompt-Template level is performed on common open source SFT data.
- SFT side effects on calibration and abstention. Evaluate whether the model is more confident, less uncertain, or easier to fabricate without knowing after SFT.
- Multiple rounds of constraints are maintained. After the 1st round is subject to the format constraints of the research model, whether the 3-5 rounds still follow the constraints and which training data can improve memory.
- Small model SFT training dynamics. Reproduce 2025 ICLR Small Model SFT paper learning rate, batch size, early loss/gradient signal, verify whether it is true in Chinese or code tasks.
Summary
Instruction Tuning is the entrance to post-training research, but not the whole of post-training. When reading this route, the most important thing is to establish a decomposition ability: what is the task, where does the data come from, what behavior SFT trains, whether the evaluation can prove generalization, and what mechanism does the failure sample reveal. The classic paper provides a paradigm: FLAN/T0 talks about the generalization of cross-task instructions, the location of the InstructGPT talks about SFT in the human feedback chain, Self-Instruct talks about synthetic data bootstrap, LIMA The relationship between quality and pre-training ability. In recent years, the open formula and verifiable evaluation have reminded us that today's Instruction Following research has entered the stage of "whether the data is adapted, whether the constraints are verifiable, and whether the results can be reproduced" from "Looks more likely to answer".
A controlled small experiment is a good starting point for research. Hold the model fixed while systematically varying the data, or hold the data fixed while varying the template. With a fixed benchmark, examine how well the model generalizes to different constraints. Save each failure case. Those examples can reveal research questions that leaderboard scores alone will miss.
References
Search date: 2026-08-03. The following links give priority to the paper page, conference page, official project page or official repository; model performance, code status, data version and benchmark results may change, and it is recommended to verify again before official release.
- Jason Wei et al., Finetuned Language Models Are Zero-Shot Learners, arXiv:2109.01652, 2021/2022. https://arxiv.org/abs/2109.01652
- Victor Sanh et al., Multitask Prompted Training Enables Zero-Shot Task Generalization, ICLR 2022 / OpenReview. https://openreview.net/forum?id=9Vrb9D0WI4
- Yizhong Wang et al., Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks, EMNLP 2022, ACL Anthology. https://aclanthology.org/2022.emnlp-main.340/
- Long Ouyang et al., Training language models to follow instructions with human feedback, arXiv:2203.02155, 2022. https://arxiv.org/abs/2203.02155
- Hyung Won Chung et al., Scaling Instruction-Finetuned Language Models, arXiv:2210.11416, 2022. https://arxiv.org/abs/2210.11416
- Google Research Blog, The Flan Collection: Advancing open source methods for instruction tuning, 2023. https://research.google/blog/the-flan-collection-advancing-open-source-methods-for-instruction-tuning/
- Yizhong Wang et al., Self-Instruct: Aligning Language Models with Self-Generated Instructions, arXiv:2212.10560 / ACL 2023 camera ready. https://arxiv.org/abs/2212.10560
- Self-Instruct official repository. https://github.com/yizhongw/self-instruct
- Chunting Zhou et al., LIMA: Less Is More for Alignment, arXiv:2305.11206 / NeurIPS 2023. https://arxiv.org/abs/2305.11206
- Shengyu Zhang et al., Instruction Tuning for Large Language Models: A Survey, ACM Computing Surveys, 2026. https://doi.org/10.1145/3777411
- Yifan Lou et al., Large Language Model Instruction Following: A Survey of Progresses and Challenges, Computational Linguistics, 2024. https://direct.mit.edu/coli/article/50/3/1053/121669/Large-Language-Model-Instruction-Following-A
- Nathan Lambert et al., Tulu 3: Pushing Frontiers in Open Language Model Post-Training, arXiv:2411.15124, revised 2025. https://arxiv.org/abs/2411.15124
- Ai2 Tulu 3 official project page. https://allenai.org/tulu
- AllenAI open-instruct official repository. https://github.com/allenai/open-instruct
- QwenLM, Qwen3 official repository and model documentation. https://github.com/QwenLM/Qwen3
- Aldo Pareja et al., Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs, ICLR 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/b6e2c96bc4702f761d7d108d6e31930f-Abstract-Conference.html
- Dylan Zhang, Qirun Dai, Hao Peng, The Best Instruction-Tuning Data are Those That Fit, arXiv:2502.04194, 2025/2026 revision. https://arxiv.org/abs/2502.04194
- Jeffrey Zhou et al., Instruction-Following Evaluation for Large Language Models, arXiv:2311.07911, 2023. https://arxiv.org/abs/2311.07911
- Valentina Pyatkin et al., Generalizing Verifiable Instruction Following, arXiv:2507.02833 / NeurIPS 2025 Datasets & Benchmarks. https://arxiv.org/abs/2507.02833
- Huimin Ren et al., LexInstructEval: Lexical Instruction Following Evaluation for Large Language Models, AAAI-26. https://ojs.aaai.org/index.php/AAAI/article/view/39701
- Hugging Face TRL, SFTTrainer documentation. https://github.com/huggingface/trl/blob/main/docs/source/sft_trainer.md