
Date:2026-07-28
Who this is for: graduate students, early-career researchers and AI practitioners with an engineering background
Goals of this article: Upgrade the "Running paper code" to "Audit paper proposition", and learn to use the environment, data, weight, randomness and log to construct a trusted LLM recurrence report.
Abstract
Reproducing an LLM paper, the most common mistake is from pip install and bash run.sh start. The real reproduction should be asked first: what exactly does the paper claim? Which results are the main conclusions and which are just presentations? What needs to be reproduced is training, fine-tuning, reasoning, evaluation, or complete pipeline? If the resources are insufficient, which claims can be verified in small-scale experiments, and which can only be done for Implementation Check or Partial reproduction?
From 2025 to 2026, the reproduction has changed from "engineering details in the appendix of the paper" to independent research objects. MLRC 2026 becomes the official Track of NeurIPS, ReproEvalCard proposes a standard for LLM Pipeline to evaluate the workpiece report, and PaperBench has made "Reproducing AI papers from scratch" into Agent Benchmark, PyTorch, vLLM, Hugging Face and other tool documents also repeatedly remind: the same SEED does not guarantee that cross-hardware, cross-version, and cross-Serving scheduling are exactly the same. This article gives a set of reproducing processes suitable for postgraduate training: first disassemble the Claim, then audit Artifacts, then run the Minimal Run, and finally do the controlled reproduction and reproduction report.
Contents
- Why LLM papers are more difficult than traditional deep learning
- Define goals before reappearance: claim, range and success criteria
- Artifact audit: environment, data, weights, SEED, logs
- Five-stage reproduction workflow
- Representative papers and workpiece roadmap
- Methods and experiments compared
- Actionable checklist
- Common misconceptions
- Research questions for graduate students
- Summary
- References
Why LLM papers are more difficult than traditional deep learning
The reproduction of traditional model papers can often be abstracted into three things: get code, get data, retrain or evaluate according to hyperparameters. Of course, these are also needed for LLM papers, but there are more difficulties.
First, there are more hierarchies of model workpieces. A "model" may include tokenizer, base checkpoint, SFT checkpoint, DPO / RLHF adapter, system prompt, chat template, generation Config, Retriever, reranker, Tool Schema, Judge Prompt, and post-processing scripts. A score in the paper table may not be the attribute of a single checkpoint, but the output of a complete set of pipelines.
Second, there is a huge difference in budget calculations. Many pre-trained papers cannot be fully reproduced by ordinary labs. Reproducible training is not equal to retraining the 100 billion parameter model, but to clearly select the reproducing hierarchy: the reproduction model loading and evaluation, the reproducing small-scale training curve, the reproducing key ablation, the reproducing data processing, the reproducing reasoning behavior, or the complete training recipe. The focus of research training is to make CLAIM auditable, rather than blindly pursuing the same GPU scale.
Third, randomness and numerical details are enlarged. The official PyTorch document clearly states that the fully reproducible result does not guarantee cross-pytorch release, commit, platform, CPU/GPU; the vLLM document also shows that the default is performance and does not guarantee the reproducibility of the result, online serving Scheduling is especially difficult to be fully determined. The 2025 study on Reproducibility further pointed out that the batch size, number of GPUs, GPUs Both models and precision settings may disagree with the inference output, especially in long-chain reasoning, the tiny differences in early tokens will be cascaded to the final answer.
Fourth, the evaluation itself is becoming more and more like system engineering. ReproEvalCard emphasizes in ACL 2026 that the reproduction of LLM Pipeline requires not only model weights and data, but also prompt configuration, snapshot and intermediate execution traces. For Agent, RAG, Tool-use, and Reasoning Model, the evaluation results without trace can only be partially reviewed, and it is difficult to reproduce completely.
Therefore, the reproduction of the LLM paper is not "can I run out the same number", but "can I explain under what conditions the number is true and which claims my reproduction evidence supports or weakens the paper".
Define goals before reappearance: claim, range and success criteria
Before you start writing the code, first rewrite the paper proposition into a testable object. It is recommended to use the following sentence pattern:
Under the data set D, split S, metric M, model/weight w, reasoning or training budget B, environment E, random control r, the paper claims that method A produces difference x relative to baseline C, and uses this difference to support the assumption mechanism H.
If you are incomplete, the reproduction target is not clear. Don't be in a hurry to take the environment, go back to the paper's Method, Experiment, Appendix, Repo Issue and Model Card first.
The reproduction range is usually divided into five levels:
| Level | Target | suitable for resources | Success standard |
|---|---|---|---|
| L0 sanity check | Confirm that the code, model and data can be loaded | Single card or CPU | Can execute minimal example, no shape/dtype/tokenizer error |
| L1 evaluation reproduction | Reproduce a review form for a paper | Single card to multi-card | The same checkpoint, the same data and metrics are close to the result |
| L2 component reproduction | Reproduce key modules or ablation | Single card to small cluster | Module switches produce directional differences consistent with the paper |
| L3 scaled-down training | Small-scale recurrence of training trends | snan | Loss curves, validation trends and failure patterns are explained |
| L4 full recipe reproduction | Try to reproduce complete training or system as much as possible | Big cluster | Results, costs, logs, and model artifacts are auditable |
Graduate training generally starts with L1-L3. L4 is valuable, but should not be the sole goal. The contribution of many good reproducing reports is precisely: under the affordable resources, dismantling the most critical assumptions of the paper, and explaining what settings it is established, weakened or failed.
Artifact Audit: environment, data, weights, Seed, logs
1. The environment is not requirements.txt, but executable boundary
Minimum to record:
- Operating system, CUDA/ROCm, driver version, GPU model and video memory;
- Critical dependency versions such as Python, PyTorch, Transformers, Datasets, Accelerate, vLLM, PEFT, Bitsandbytes, Flash-attn, etc.;
- Compile related variables, such as
TORCH_CUDA_ARCH_LIST, NCCL, cuDNN, compiler version; - Whether to use Docker, Conda, uv, pip-tools, Poetry or Nix;
- Run entry commands, environment variables, configuration files, and commit hashes.
Don't just save requirements.txt. It may not lock Transitive Dependencies, nor does it indicate system libraries, drivers, and hardware. The reproduction report should include at least env.lock、run_command.sh、hardware.md and git_commit.txt. If the paper repository does not have a lockfile, you can generate one yourself, but indicate "reproduced, not provided by the author".
2. The data should be changed from "name" to MANIFEST
The LLM paper says "We use GSM8K" or "We evaluate on MMLU" is not enough. You need to record:
- Dataset source URL, version, revision, download time;
- train / validation / test split;
- Preprocessing, filtering, dedup, packing, shuffle rules;
- prompt template and fEW-Shot examples;
- Whether to use streaming, cache, private data or manual annotation;
- Data license and access restrictions;
- File checksum or sample count.
The official documentation of Hugging Face Datasets is supported revision Specify the version of the Hub dataset, which is a detail that is easily overlooked in reproduction. Only writing dataset name is equivalent to leaving future version drift to the reader.
3. Weights must be positionable to immutable identification
There is not enough record model name. At least need:
- Hugging Face repo / ModelScope repo / GitHub release / DOI;
- Checkpoint revision, commit hash or release tag;
- tokenizer revision;
config.json、generation config、chat template;- whether to use safeensors;
- The corresponding relationship between adapter and base model;
- Quantization formats and parameters, such as GPTQ, AWQ, Bitsandbytes 4-bit/8-bit, GGUF;
- Weights license and gated access states.
Transformers document reminder, from_pretrained() Will give priority to loading SafeTensors, if using the Custom Model trust_remote_code=True, it is better to bind the specific revision. The PEFT documentation also states that Adapter CheckPoint usually only contains Adapter weights and configurations, not Base Model. The reproduction report must clearly write the pairing relationship between base and adapter, otherwise it will be difficult to rebuild the same model after a few months.
4. Seed is not a magic button
Common SEED records include random、numpy、torch、torch.cuda, DataLoader Worker, DataSet Shuffle, Generation Sampling SEED, vLLM SEED, Distributed Sampler SEED. Accelerate documentation is recommended for set_seed() Unified set of random sources in distributed training; PyTorch documents also remind at the same time that even if you set up random seeds, there is no guarantee of complete consistency across Releases, platforms and devices.
Therefore, there are three conclusions to be distinguished by the reproduction report:
- Deterministic within setup: the same machine, the same dependency, the same command can be repeated;
- Statistically Stable: The mean and variance under multiple SEEDs support the same direction;
- cross-system reproducible: The hardware or framework is basically the same.
Many LLM papers can only reach the first two. Don't mistake them as the third one.
5. The log is the chain of evidence for reproducing reports
It is recommended to record four types of logs:
- config log: complete configuration, command line parameters, environment variables;
- data log: number of samples, number of filters, shuffle seed, data hash;
- runtime log:loss、learning rate、grad norm、throughput、GPU memory、checkpoint step;
- Evaluation log: raw output, parsing results, metric, failed samples, judge prompt, judge model, statistical scripts.
Saving RAW Generations is more important than saving the final score if you do a LLM review. Without the original output, you can't distinguish between models, prompt errors, parser errors, metric errors, or Judge bias.

Five-stage reproduction workflow
Stage 1: Paper Claim
Choose only one master claim. Don't come up and reproduce the entire paper. First locate the table, graph, or ablation that can support the most contributions in the paper, and write:
- Which claim am I reproduced?
- What workpieces are needed?
- What is the success threshold?
- Which differences are acceptable and which will overthrow claim?
- What is the resource budget?
For a large model paper, a reasonable goal may be to "reproduce a review of an open-weight checkpoint on a designated benchmark", rather than "retrain the complete model".
Stage 2: Artifact Audit
Establish artifacts/ table:
| Category | Need to record | Deletion when missing |
|---|---|---|
| Code | repo、commit、branch、patch | fork and record diff |
| Environment | lockfile、Dockerfile、GPU、CUDA | Replace and label with the repeater environment |
| Data | URL、revision、split、hash、preprocess | If you can't get it, you will be degraded to a partial reproduction |
| Weights | checkpoint、tokenizer、adapter、quantization | Bind a commit or write for manual verification |
| Evaluation | prompt、metric、parser、judge、raw output | Scorer first, then run the model |
| Journal | train/eval logs、config、seed | Self-built log template |
The product of this step is not a paper abstract, but a "reproduction feasibility map". If critical data or weights are not available, adjust your goals as early as possible.
Stage 3: Minimal Run
Run the smallest closed loop first, don't run the complete experiment directly:
- Load tokenizer and model;
- Take 3-10 samples;
- run the same prompt;
- Save raw outputs;
- run Scorer;
- Generate a minimum metrics file.
A minimal run is designed to expose problems in the pipeline before you try to improve the score. Common failures include a mismatched tokenizer, a changed chat template, incorrect end-of-sequence handling, truncation at the token limit, inconsistent bf16/fp16 data types, a metric parser that is too sensitive to formatting, loading the wrong dataset split, and presenting few-shot examples in the wrong order.
Stage 4: Controlled Reproduction
After the minimum closed loop passes, it enters the controlled reappearance. The recommended sequence is:
- Fixed model, data, prompt, metric, only change the SEED, and estimate the variance;
- Fixed seed, compare batch size, number of GPUs, dtype, serving backend;
- Fixed inference budget, compare baseline with methods;
- Reproduce a small subset in the main table of the paper;
- Extend to the full table or key ablation.
For tweeting theories, especially to record the decoding parameters: temperature, top_p, top_k, max_new_tokens, stop tokens, repetition penalty, num_return_sequences, beam search, best-of, self-consistency samples. For training or fine-tuning papers, record Effective Batch Size, Gradient Accusation, Learning Rate Schedule, Warmup, Weight Decay, Optimizer, LoRA Rank, Target Modules, Packing, Sequence length, Checkpoint Selection rules.
Stage 5: Reproduction Report
The reproduction report is recommended to be divided into four sections:
- Target: which claim to reproduce and why choose it;
- Workpiece: Which are from the author, which are supplemented by the repeater, and which are not available;
- Result: Main results, variance, failure samples, and the difference between the paper;
- Explanation: Confirm, partially confirm, fail to reproduce or cannot judge, and explain the reason.
A good reproduction report does not have to make all the numbers into "the same form of the paper". It should let the reader know: if you want to continue to study the paper, which parts are trusted, which parts are fragile, and which places may have followed-up ideas.
Representative papers and workpiece roadmap
Classic Reproduction Framework: From Checklist to Reproducibility Science
The NeurIPS 2019 Reproducibility Program report of JMLR 2021 put Code Policy, Reproducibility challenge and Checklist In connection, it means that the reproduction is not a single-point engineering task, but the result of the meeting system, author norms and community challenges. In 2026, MLRC became the official Track of NeurIPS, and further confirmed that reproduction, reproducing failure, generalization test, and meta-recurring research can all be formal scientific research contributions.
LLM pipeline:ReproEvalCard
ReproEvalCard audits the 2022-2025 Pipeline-Based LLM paper, emphasizing Prompt, Judge configuration, Retrieval Snapshot and The necessity of the indexate traces. Its most important inspiration for graduate students is: When reproducing LLM systems, don't just stare at model weights. The evaluation result of a RAG or Agent paper may be mainly determined by the retrieval library version, tool return, Judge Prompt, and trajectory resolution.
Paper Reproduction Benchmark: PaperBench
OpenAI released PaperBench in 2025 to grade AI paper reproductions against a rubric. The official page describes an agent’s task as understanding the paper’s contribution, building a codebase, and running experiments. The GitHub README divides a run into three stages: agent rollout, reproduction, and grading. The underlying idea is to divide a reproduction into subtasks that can be scored separately. Graduate students can apply it by organizing a reproduction plan around four checklists covering artifacts, implementation, experiments, and analysis.
Open Source Model Report: OLMo 2, Qwen3, DeepSeek-V3
OLMo 2 emphasizes Fully Open Artifacts in the OpenReview page: Weights, Training Data, Training Code, Recipes, Logs, and Intermediate checkpoints. Ai2/OLMo repo also gives training configuration, W&B links and different stages of checkpoints. This type of paper is suitable for high-quality reproducing training, because it exposes enough workpieces.
Qwen3 GitHub / Technical Report provides model family, documentation, Hugging Face / ModelScope links, reasoning, deployment, quantization and training guidance. DeepSeek-V3 Repo public model download, operation mode, weight description and technical report reference, but complete pre-training data is not fully re-engraved. When reading such technical reports, it is necessary to distinguish "Open Weights Evaluation Reproduction" and "Full Training Reproduction".
Numerical and Inference Reproduction: 2025 Reproducibility
"Give me FP32 or Give me Death?" and "a Sober Look at Progress in Language Model Reasoning" remind us: Reasoning Benchmark The scores may be affected by precision, hardware, prompt, SEED, decoding parameters and software framework. For graduate students, this type of paper is suitable for training "negative results interpretation": when the reproducing numbers are inconsistent, do not immediately attribute the paper errors, first systematically scan the values and evaluate the pipeline.
Methods and experiments compared
| Reproduce object | The most easily missing workpiece | Minimum feasible reproduction | critical risk | Suggested log |
|---|---|---|---|---|
| Open Source Model Evaluation | checkpoint revision、chat template、scorer | Fixed weight run 100-500 samples | Data version, parser, decoding | raw output、metric json、config |
| LoRA / QLoRA Fine-Tuning | base-adapter correspondence, data processing | Small data run through training and compare direction | target modules、packing、seed | train loss、eval loss、adapter hash |
| RAG / Agent papers | Retrieve snapshots, tool return, trace | Reproduce 20-50 samples full trajectory | Dynamic web page, API version, Judge bias | retrieval docs、tool traces、judge outputs |
| Reasoning paper | Prompt, number of samples, precision, hardware | Fixed Prompt run more seed | BATCH/DTYPE/GPU causes disagreement | per-sample answer、token length、seed |
| Pre-training technical report | Data mixing, full compute, training logs | Small model reproducing trends or ablation | Resources are incomparable, data is inaccessible | loss curve、throughput、checkpoint |
| Benchmark Papers | Hidden Split, Scoring Code, Pollution Detection | Reproduce Scorer and a set of baselines | Leakage, selective report | scorer version、failure cases |
Actionable checklist
Before the beginning
- Is only one main claim selected?
- Is the reproduction level L0-L4 written clearly?
- Is it to confirm that the code, data, and weight licenses allow research to be used?
- Estimate GPU, time, storage and API costs?
- Is the success threshold and failure interpretation criteria defined?
Environment
- Is OS, GPU, Driver, CUDA/ROCm recorded?
- Is it locked with Python and critical dependency versions?
- Do you save repo commit, patch, and run commands?
- Whether to record
trust_remote_code、custom kernels、flash attention、serving backend? - Is it possible to re-run the Minimal Run in a clean environment?
Data
- Is the DataSet Revision or file hash fixed?
- Is it recorded Split, sample number, filtering rules?
- Do you save Prompt template and fEW-Shot examples?
- Check test set leak or near duplication risk?
- Is it preserved the processed intermediate file manifest?
Weights
- Is it fixed base model, tokenizer, adapter, quantization revision?
- Does the Model Config and Generation Config are saved?
- Is the checkpoint selection rule logged?
- Do you distinguish between Open Weights, Open Data, Open Training Code and Fully Open?
- Is it confirmed that the local actual load is the expected weight?
Randomness and Reviews
- Is it set to Python, NumPy, PyTorch, CUDA, DataLoader, generation seed?
- Do you report multiple SEED variances instead of just reporting the best one time?
- Are the decoding parameters and serving scheduling recorded?
- Are Raw Outputs, Parser Outputs and Final Metrics saved?
- Can it explain the source of the difference with the paper number?
Common misconceptions
Misunderstanding 1: Running through the author's repository is equivalent to reappearing. Running is only L0. Reproduction should at least connect the paper claim, input artifact, result and difference interpretation.
Misunderstanding 2: Inconsistencies in the numbers indicate that the paper is wrong. Check the data version, weight revision, prompt, dtype, batch size, GPU, dependency and scorer first. The vulnerability of the LLM review may come from Pipeline, not necessarily from methods.
Misunderstanding 3: Only save the final score. The final score is the least enough. RAW outputs, trace, config and logs are the evidence for analyzing failure samples.
Misunderstanding 4: Take SEED as full control. SEED can only control a portion of random sources. Differences across hardware, framework, and across Serving Backend require separate experiments.
Misunderstanding 5: Reproducing the goal is too large. If the paper is pre-training of the 100 billion model, do not start with a complete retraining. First reproduce the evaluation, data processing, small model trends, or key ablation.
Research questions for graduate students
- LLM evaluation artifact audit: Select 30 RAG / Agent / Reasoning papers, and count the disclosure of Prompt, Judge Config, Retrieval Snapshot, Trace, Seed and Raw Outputs.
- Hardware-sensitive reasoning reproduction: Fixed model and data, the system compares the impact of GPU model, batch size, dtype, serving backend on the reasons of reasonsing benchmark.
- Model card to reproduction checklist: Convert the Model Card, Repo, and DOCS in the open source model technology report to the automatic reproduction list, and evaluate the information gap.
- Minimal reproduction templates: Build the minimum reproducing template for LoRA, RAG, Agent, and Reasoning Eval respectively, and require the automatic output of env/data/model/log manifests.
- Reproduction report rubric: Refer to PaperBench, split a paper reproducing report into scoring Rubric for coursework or laboratory Reading Group.
- Negative reproduction taxonomy: Organize and reproduce failure cases, distinguish between environment failures, data failures, implementation failures, numerical failures, statistical failures and claim failure.
Summary
The core of reproducing LLM papers is not to "run the code up", but to turn the paper into auditable evidence. You need to fix the environment, freeze the data version, locate the weights and tokenizers, control the randomness, save the raw outputs and logs, and then write the results as reproduction report.
For newcomers to scientific research, the most valuable movements for training are: small-scale, strong control, and complete records. Don't pursue the entire model technical report from the beginning. First choose a claim, run a minimal run, create an artifact manifest, and then gradually expand to a key ablation or a complete review. If you persevere for a long time, you will form a more solid ability than "reading the paper": judging where the evidence chain of a paper is strong, where it is fragile, and where it can develop into its own research question.
References
Search date: 2026-07-28. The following give priority to official pages, conference pages, paper pages, project repository and tool documents; models, lists, dependencies and repository status may change, and it is recommended to re-verify before the official release.
- NeurIPS Blog, “MLRC 2026: Reproducibility as an Official Track at NeurIPS”, 2026-05-04. https://blog.neurips.cc/2026/05/04/mlrc-2026-reproducibility-as-an-official-track-at-neurips/
- ML Reproducibility Challenge official site, MLRC 2026. https://reproml.org/
- NeurIPS 2026 Main Track Handbook, Paper Checklist and Data Guidelines. https://neurips.cc/Conferences/2026/MainTrackHandbook
- ICLR 2026 Reviewer Guide, reproducibility and evidence assessment guidance. https://iclr.cc/Conferences/2026/ReviewerGuide
- Priyaranjan Pattnayak and Apoorv Bhatia, “ReproEvalCard: A Reporting Standard for Reproducible Evaluation of LLM Pipelines”, ACL 2026. https://aclanthology.org/2026.acl-short.22/
- Joelle Pineau et al., “Improving Reproducibility in Machine Learning Research”, JMLR 2021. https://www.jmlr.org/papers/v22/20-303.html
- OpenAI, “PaperBench: Evaluating AI’s Ability to Replicate AI Research”, 2025-04-02. https://openai.com/index/paperbench/
- OpenAI PaperBench GitHub README. https://github.com/openai/preparedness/blob/main/project/paperbench/README.md
- PyTorch Documentation, “Reproducibility”, last updated 2025-10-03. https://docs.pytorch.org/docs/stable/notes/randomness
- vLLM Documentation, “Reproducibility”. https://docs.vllm.ai/en/v0.9.1/usage/reproducibility.html
- Hugging Face Datasets Documentation, “Load”. https://huggingface.co/docs/datasets/loading
- Hugging Face Transformers Documentation, “Loading models”. https://huggingface.co/docs/transformers/models
- Hugging Face Transformers Documentation, “Parameter-efficient fine-tuning”. https://huggingface.co/docs/transformers/peft
- Evan Pete Walsh et al., “2 OLMo 2 Furious (COLM’s Version)”, OpenReview / COLM 2025. https://openreview.net/forum?id=2ezugTT9kU
- Ai2, “OLMo 2 32B: First fully open model to outperform GPT 3.5 and GPT 4o mini”, 2025. https://allenai.org/blog/olmo2-32b
- AllenAI OLMo GitHub repository. https://github.com/allenai/olmo
- QwenLM Qwen3 GitHub repository and technical report links. https://github.com/QwenLM/Qwen3
- DeepSeek-AI DeepSeek-V3 GitHub repository. https://github.com/deepseek-ai/DeepSeek-V3
- DeepSeek-AI, “DeepSeek-V3 Technical Report”, arXiv:2412.19437. https://arxiv.org/abs/2412.19437
- Jiayi Yuan et al., “Give Me FP32 or Give Me Death? Challenges and Solutions for Reproducible Reasoning”, arXiv:2506.09501. https://arxiv.org/abs/2506.09501
- Andreas Hochlehnert et al., “A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility”, arXiv:2504.07086. https://arxiv.org/abs/2504.07086