
- Date: 2026-09-23
- For graduate students, early-career researchers and research engineers
- Scope of reproduction: organize the evidence index of 052–080, encapsulate and replay the 080 report experiment; do not retrain the model.
Abstract
After a month, it is often not the code that is most likely to be lost, but "Which experiment did this number get this number?" This round, from attention, retrieval and training methods, to interpretive, calibration and results reporting, and finally need to leave research assets that can be re-run, examined and questioned. The goal of this article is very specific: organize an existing experiment into a self-contained small package, recalculate the report in two empty work directories, and actively modify the code, input and environmental records to check whether the process can find changes. The criterion for success is that the chain of evidence is clear, not the folder neat.
Contents
- Recover value and this round of harvest
- Core Ideas and Validation Formulas
- A guide to the official implementation
- Minimum Experiment and Codebase Structure
- Evaluation protocol and results
- Troubleshooting and limits of the evidence
- Questions for further research
- Summary and References
1. Recover value and this round of harvest
What the paper reports. Pineau et al. summed up the reproducibility plan of NeurIPS 2019 in JMLR 2021, including code submission, community recurrence challenges, and a paper checklist. The barriers to discussion in the paper are not only the lack of code, but also include data differences, insufficient training instructions, unclear indicators, selective reports and conclusions beyond evidence. [1] Therefore, "repository energy start" is only a condition for the study of checkability.
What we tested. The three-digit number in the scan directory, 052-080, has a total of twenty-nine articles, all of which are kept through records. This time, the syntax of its metadata, Chinese characters, local links, picture files and 47 Python source files is re-checked; the historical operation and visual acceptance records are preserved, and the entire model is not run again. See the Research Codebase Index for full navigation.
Three things from these articles are worth reusing. One is the mapping from concepts to code, such as the tensor operations that implement the attention formula. Another is the experimental protocol, including how retrieval candidates, data splits, and abstention thresholds are chosen. The third is a set of ways to challenge a result, such as ablations, input perturbations, and independent audits of the counts. A research repository should preserve the links among all three as well as the training scripts.
These teaching experiments share a limit. Small synthetic datasets can expose interface and metric errors, but a replication on real data with the official models and the original paper’s budget remains a separate undertaking. The monthly review should turn those unverified points into experiments for the next round. A “passed” article status gives no basis for generalizing a scientific conclusion.
2. Core ideas and verification formula
This article defines "experimental packages" as a collection of code, configuration, input, environmental records, and expected output. The manifest, that is, manifest, is the directory and content fingerprint of these objects; SHA-256 is an algorithm that maps the file bytes into a fixed-length summary to find out whether the content has changed.
Here (p_i) is the relative path in the packet, (h_i) is the summary of the saved, and (k=5) is the number of load files in this example. The verification is both a summary and a collection of files, so missing and unexpected new files will be found. Hash only states whether the bytes are consistent with the baseline, can't prove the label is correct, and can't prove the author's honesty.
(c) is the report code, (d) is the stored experimental summary, (\theta) is the statistical protocol, (e) is the operating environment, (s) is the random seed; (O_A, O_B) is the result generated by the two working directories. Compare the byte digests of the five deterministic outputs here, and require them to match the baselines that were saved from 080 in advance, instead of just checking the two new runs that are identical to each other.
The logical shape of the input score is [3,5,4]: Three old model runs, five disturbance budgets, four attacks. It is not a new trained tensor, nor is it sixty independent seeds. The replay of the two reports does not increase the sample size of the model experiment, and cannot automatically narrow the original error bar.
3. Guide to the official implementation
Read PyTorch's Reproducibility documentation first: the same seed does not guarantee that the results of different versions, platforms, or CPU/GPU are exactly the same. [2] When reading, the random number source, operator determinism and operating environment are marked respectively, and all changes are not attributed to the seeds.
Then open the official torch/random.py. [3] This time manual_seed Delegate to the internal implementation, which sets the seeds for the available devices and calls the default generator; get_rng_state Returns the CPU default generator state. From this, you can ask the code reading question: Which generator states are saved in breakpoint recovery, and whether the data loading sequence is also restored? Recording an integer seed does not automatically answer these questions.
Finally, read the stage and lock file description of the DVC. [4] cmd describe the execution command, deps Describe input dependencies, outs description output; dvc.lock Record stage status and content summary. Leaking the external files read by the script out of the dependency list will make the judgment of "no change in input" lost basis. Borrowing this explicit dependency idea, this paper uses the standard library to achieve a smaller fixed process.
The actual opening this time is the 2.14 page and search date displayed by the PyTorch document. main The source code should not be assumed to be the same commit; the source code has not been fixed yet. DVC also only reads the documentation. Relevant framework operation, version compatibility and breakpoint recovery experiments are all subject to manual verification.
The minimum experiment and code base structure

In the figure, the input is verified first, and then divided into two working directories, and the final comparison result; fork means two independent directories, and the code is executed in sequence. It does not represent two machines, containers isolated or accelerated in parallel.
Execute from this directory, the output directory should be a new path that does not exist yet:
python3 code/replay.py --out code/my_replay
PYTHONPYCACHEPREFIX=/tmp/081-pycache python3 -m py_compile code/replay.py code/capsule/report.py code/capsule/audit.py
Only the Python standard library is required, and the actual baseline is CPython 3.9.6. The entry will check the interpreter version and implementation; when replacing the version, another reviewed baseline should be established, rather than still claiming the same environment after deletion check. Dependencies and complete commands see Code description.
capsule/ Save the 080 reporter, independent auditor, and three input files as it is. capsule_manifest.json Save the summary, source, random seed, and expected output summary of these files. The runner uses the current interpreter to create a child process, keeping the standard output, error output and results of each command; deny the overriding when the old output directory already exists.
The Monte Carlo inside the reportr is sample fixed to 80, and the runner also sets the Python hash seed to 81. No new training, data sampling, or model downloads are added here. The independent auditor does not import the reporter function, but rechecks the statistical results from the integer failure count and score operations; it can find some implementation errors, but still share the same batch of upstream data.
The public layer of the personal library is only responsible for the running identity, evidence path and verification results, and the specific indicators are left in each experiment. For example, even if the recall, abstent coverage and attack success rate are both decimals, you cannot use a vague meaning. score Field mix comparison. The accompanying research card template requires a clear denominator, direction, division and scope of conclusions.
5. Evaluation protocol and actual results
This test is to preserve the reproducibility of the abstract. Before starting, the five load summaries and five deterministic output abstracts are taken from the 080 file. Each working directory starts with load copying, and there is no pre-release result; then the reporter and auditor are executed, and then compared to the baseline.
What we tested. The two working directories have successfully processed 60 conditional logs and twenty statistical groups; the five deterministic outputs are the report JSON, the list of exhaustive weights sample, the CSV table, the Markdown table and the audit JSON. They are all bytes between replays and between 080 baselines. see for detailed records Replay acceptance log.
| Check object | judgment rule | actual result |
|---|---|---|
| Enter and code | Consistent file collections and summaries | five loads through |
| Report recalculation | The two new directories match the old baseline respectively | Each time five outputs pass |
| Counting and Statistics | Independent audit recalculation | passed twice |
| drift example | explicitly refuse to change | Seven categories were discovered |
Seven types of samples change the code, configuration, data, file missing, additional files, interpreter version records and output tables respectively. The fault is only injected into a temporary copy, and the original experiment is not modified. The first three categories simulate file drift by adding bytes, which proves that the integrity check will take effect, not all semantic errors will be detected.
The time-consuming log is reserved, but does not require the same bytes, because the real run time will change; the number of records, the shape of the score and the Python version are still checked. This exclusion rule is stated in advance in the list, and you cannot arbitrarily delete disliked fields after seeing the difference.
6. Failure investigation and evidence boundaries
The first type of failure is that the input identity is unknown. The same file name does not mean that the content is the same, and the default value of the configuration miss is also different from the two "experiment with the same name". The load summary and configuration should be checked first, and then the results should be checked, instead of directly adjusting the tolerance for comparison. If the list and the file are modified at the same time, this case will not authenticate its authenticity; the official archive requires an independent version record.
The second type of failure is to hide dependencies. Scripts read caches from personal directories, rely on the current working path, or reuse old results, can produce false replay successes. This example uses a relative package path and executes in a new directory, but does not implement file access tracking or network sandbox, so it cannot claim that all external dependencies exist.
The third type of failure is the same as the error. Running the same error formula twice runs the same table. Therefore, raw counting, independent implementation and manual semantic checking are required. If there is a label error or test pollution upstream, the summary audit cannot be repaired; it also does not re-verify the sample-by-sample prediction and model weight.
The fourth type of failure is that the statistical object is replaced. Replaying the same result is not a new random repetition; cross-device differences are not equal to method effect differences. The current conclusion is strictly limited to this machine, this interpreter and this load. Cross-operating system, dependency upgrade, real GPU training, official Benchmark and interval coverage are not running, wait for manual verification.
Syntax checks and CPU Smoke test passed. The existing result directory refuses to be overwritten, which is convenient for retaining the evidence of failure; the pre-check failure will directly print the error, and if the execution phase fails, it will write an unfinished state in the new directory. Don't leave the failure record empty and leave only the final run successfully.
7. Follow-up scientific research issues
Author’s interpretation. The most valuable question in the next round is whether this recording method can help third-party positioning reproduce differences. The data and code can be fixed, the interpreter, numerical library and hardware can be changed item by item, the prediction error tolerance is defined in advance, and then the stage of comparison of changes occurs. The current byte equality standard is suitable for small reporting processes and cannot be used for all floating-point training results without argument.
You can also study the minimum evidence preservation amount: only aggregate indicators, save sample predictions, and then increase intermediate activation, what failures can be found? The errors to be detected should be listed first, and then the storage cost and detection ability should be compared. Saving more files itself is not guaranteed to be more reliable, the key is whether the specific error explanation can be refuted.
For graduate students, you can choose an experiment that still lacks real data verification from the index, fix the official submission, and make up a set of strong baseline and failure analysis to form a new research card. For beginners, try to deteriorate an input and then observe the error information, which is often easier to understand the chain of evidence than to install many experimental management tools at the same time.
8. Summary
The personal research code base should allow each conclusion to return to the source, agreement, input and operational records. This time, we have left 29 articles with both experimental navigation and a small package that has been replayed and faulty in dual directory. It has proven the processability of the process within a limited scope; the task of subsequent scientific research is to continue to test the hypotheses that these teaching experiments have not yet covered.
References
Search date: 2026-09-23. The following first-hand page and source code have been actually opened. Online documentation and branch content may change, and the unrun version is compatible with manual verification.
- Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d'Alche-Buc, Emily Fox, Hugo Larochelle. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). JMLR 22(164):1–20, 2021: Paper page、Formal PDF.
- PyTorch Contributors. Reproducibility, the page displays 2.14; the framework version is not installed or executed this time.
- PyTorch Contributors. Official torch/random.py, retrieve the day main, not pinned commit; used to read the seed and generator state path.
- DVC Contributors. dvc.yaml Files:Stages and dvc.lock, the online document searches the Japanese version; this time the DVC is not executed.