
- Date: 2026-10-19
- For graduate students, early-career researchers and research engineers
- Scope: Real GPT-2 Small, 16 controlled synthetic development samples, three candidate heads, only interfere with the end of the sentence.
Abstract
The previous article found that changing how names were corrupted changed the ranking of heads under activation patching. Restoring behavior with clean activations leaves open how much those heads explain when removed. Patching and ablation start from different states and use different replacement values. Here we fixed the candidate components and ran 336 ablations across three baselines and seven component subsets. Removing two heads together caused a smaller loss than the sum of their individual losses. The conditional effect of a third head changed sign with the baseline. These intervention effects can be checked, but they establish neither a complete circuit nor the use of a backup mechanism.
Reproducing value: first ask what "removing" means
Indirect object recognition, i.e. IOI, requires complementary sentences similar to "Christopher and Michael to the store, Christopher gave the ring...", the goal is Michael. The model has been able to do the right thing, and the research problem turns to internal computing. The "head" here refers to a sub-module in the long attention; "circuit" also needs to explain how the component is connected and how the information related to the task is transmitted.
Paper report: Wang et al. used intervention analysis to use intervention analysis, and put forward test standards such as loyalty, integrity and minimumness; the paper also reports the phenomenon that other components undertake the function of name handling after ablation is ablation. This is the motivation for further inspection of compensation, and it cannot be directly transplanted as the conclusion of this experiment. Original paper
The actual verification of this time: The model, template and 16 development materials of the previous article are used, and according to the local damage ranking that has been disclosed, the first three candidates are frozen: a is L8H10, b is L8H6, and c is L9H9. Both the layer and the head are numbered from zero. In this article, only the replacement baseline and the set of replaced components are changed; compared with the scan of 32 batches in the previous article, this time is unified to recalculate the baseline in batches to avoid mixing the batch value difference into the effect. Ranking and ablation use the same batch of development materials, so this is exploratory follow-up, not independently confirmed.
Core Idea: Zero, Mean and Resampling to answer different questions
Zero ablation turns the selected activation into zero, which is easy to execute, but changes the mean, scale and content at the same time. Mean Ablation uses the mean activation of the reference distribution, and retains some common components. Resampling ablation uses the real activation of a reference sample to retain the internal structure of the sample, but the result may depend on who is drawn. All three need to have a clear reference object, and "real activation" cannot be regarded as a guarantee of "unpartial replacement".
In this experiment, the three names in each template were changed to three different names, and six arrangements of Michael, Christopher, and Jessica were enumerated, forming six items per template, with a total of twelve ABC reference sentences. Here ABC represents three different name slots, independent of the tokens of candidate heads A, B, and C. The reference sentence does not repeat the subject identity, and does not give the only correct indirect object label. Take the mean value according to the template, or map a donor according to the pre-saved seed 103; all heads of the same target share the donor with the subset.
The original paper also discusses the reasons for using the ABC reference distribution and template grouping; this time, only a very small reference library is enumerated, and does not claim to reproduce its complete sampling and circuit extraction process. The reference mean may not be the actual state of any natural input, and a single donor distribution cannot represent the overall average of resampling. Ablation definition in the paper

The three replacement branches in the figure are parallel conditions, and the dotted line sends the original model output to the pairing audit; the graph only explains the protocol, and does not carry experimental measurements. Individual and unselected components are only used in the last digit, and the calculations continue to be calculated from other locations and unselected components.
Formulas and modules: write component effects as recalculable amounts
The logit difference with the original correct name and the interference name:
is a clean prompt, is the fraction of the output layer without softmax, and are the correct name and subject name respectively. is the header set by the ablation, is the replacement baseline; It is the difference after the intervention. The positive indication is that the performance decreases, and the negative means is improved. It is not Accuracy, nor is it a fixed "contribution share" owned by a component.
The non-additive quantities of the two heads are defined as:
If the value is zero, if the value is zero, the two independent losses can be added together; the negative value means that the joint loss is less than the sum prediction. Calculate line by line first, and then take the mean value of the fixed sixteen rows. Also look at the marginal effect of removing C after AB has ablation, i.e. . This amount can be different from the effect of individual Ablation C.
Implement cache attention output before projection , the tensor view is : Batch is one, Input tokens for fifteen or nineteen, each of the twelve heads has sixty-four dimensions. We only replace the selected slice of the last bit, and the output projection, residual path and subsequent networks are still running normally. So it's not the weight that removes the entire header, nor does it close it in all locations.
A guide to the official implementation
Look at the fixed version first GPT2Attention: Find the attention weighted value and merge it into a hidden dimension, enter c_proj position, confirm which head the slice corresponds to. Look again GPT2LMHeadModel The final output layer confirms that the comparison is the same location and the same answer token. Finally read the IOI repository get_circuit_replacement_hook and do_circuit_extraction, which checks the interface for location replacement, reference data and packet mean. Official GPT-2 source code、IOI Ablation Implementation
The local implementation uses the official Transformers forward process, and the self-written hook to complete the teaching experiment, does not run EasyTransformer or TransformerLens, and does not perform weight conversion. The source map is marked with the original multiplexed self-writing module, the rewriting part and the independent arithmetic implementation. The four official documents have been re-downloaded for verification, where the model implementation is consistent with the installed version bytes.
A minimal experiment and evaluation protocol
The experiment freezes the GPT-2 model and the tokenizer revision 607a30d783dfa663caf39e06633721c8d4cfcd7e, use PyTorch 2.6.0, Transformers 4.49.0, CPU single precision, four threads, batch one. The material consists of two templates, two name sequences, two overlapping name pairs and two role directions. There are only three different names, and sixteen lines cannot be regarded as independent natural samples. Leave the name and template unreferenced, not trained, generated or truncated.
The seven subsets are three single-headed, three double-heads and three heads combined. The main comparison is written as zero value and ab loss and non-additive under the reference mean in advance; the complete report of resampling and the rest of the combination is complete, and the "most explanatory" setting is not selected after reading the results. Configuration, source code, data, resource and protocol hashes are frozen before the test run, and the output cache is not reused.
The single sample test run includes 21 interventions and a complete reference library, which takes 13.31 seconds, and the full amount is estimated to be about 21.24 seconds, which is lower than the 300-second budget. The actual wall clock is officially running for 11.99 seconds, and the peak memory of the process is about 1.47 GB; the test run includes the cost of cold start, and the two do not constitute a speed comparison. Each stage state, progressive score, donor vector and exit code are reserved.
The actual result of this time: non-additive translation cannot be converted into compensation
The mean difference between the clean model is 4.295. The numbers in the table below are the average point difference loss, and the larger the interference, the stronger the intervention damage; the last two rows are the pairing and comparison.
| pre-order amount | zero ablation | ABC Mean | Fixed ABC Resampling |
|---|---|---|---|
| Losses of Individual A | 1.218 | 1.256 | 1.195 |
| Losses for the individual B | 1.416 | 1.513 | 1.519 |
| Losses of the individual C | 0.320 | 0.702 | 0.667 |
| Loss of Union AB | 2.340 | 2.493 | 2.453 |
| Losses of Union ABC | 2.281 | 2.537 | 2.477 |
| ab innocence | −0.294 | −0.276 | −0.262 |
| Continue with Ablation C after AB | −0.060 | 0.044 | 0.025 |
The non-additives of the sixteen lines AB of the three baselines are all negative. If you simply add a single head loss, the average loss of joint intervention will be overestimated. What's more worth tracking is C: A single ablation is obviously harmful, but after AB has been ablation, the marginal loss is close to zero, and the average improvement is slightly improved at zero ablation. There are still fourteen lines after AB under the mean baseline, and only twelve lines are preferred for the full vocabulary table; only two names will be missed, and other words will be missed than the correct answer.
**Author’s interpretation:** These results rule out the simple idea that each component has a fixed contribution regardless of context. They do not identify a compensatory pathway. In particular, there is no evidence here that removing A and B together makes C more indispensable. Recomputing the residual stream, layer normalization, nonlinearities, and score saturation can all produce nonadditive effects. Establishing a backup mechanism would require localizing it and intervening on its pathways. A biological name for the interaction term supplies no such evidence.
Troubleshooting and limits of the evidence
Implementation errors are excluded first. The full vocabulary error of the clean activation substitution and empty set intervention is zero; the maximum error between the complete official output and the last bit read is about 0.000191. Another independent implementation is to write the linear difference amount produced by adding the replacement after the output projection, and the maximum error is about 0.000061 compared with the pre-projection coverage, which is lower than the preset threshold. The three-head joint of all modes performs this control, avoiding multiple hooks covering each other.
An independent script using the standard library recomputed every condition, mean, correct count, interaction and reference-vector statistic from the raw candidate scores. It also passed seven deliberate corruption tests. This was an automated arithmetic check. A second implementation used a different post-projection intervention and the full official forward pass to replay 16 baselines and all 336 conditions. The top choice over the entire vocabulary agreed throughout. The largest candidate-score difference was approximately 0.000153, and the largest difference in the double-precision probability check was approximately 0.000043. The two implementations did not produce bitwise-identical values throughout.
It is also necessary to distinguish between necessity and adequacy. In a certain ablation, the score is reduced, which only means that these activations are related to the behavior under the intervention conditions; it cannot be explained that only these components are involved, and it cannot indicate that the behavior can be reconstructed by retaining these components. This article does not turn off the remaining network, and does not check the edges between the components, so a complete cause and effect diagram cannot be drawn from the three candidate heads. The complete record retains two two-headed combinations that are not shown in the table, and the reader can check whether the conclusion depends only on a selected subset.
Finally check the replacement scale. The clean vector average of C is about 9.376, zero is replaced by zero, and the reference mean is about 4.438. The three types of interventions are different, and they cannot be regarded as equal-intensity disturbances; nor the difference in the norm cannot prove that a state is outside the distribution. The real boundary is: the reference library is very small, there is only one donor allocation, development selection and analysis of shared data. The old temporary environment lacks dependence and network recovery process has been recorded, and the environmental repair is not regarded as a scientific result.
From phenomena to verifiable research questions
In the next step, these three headers and replacement rules should be fixed, and the behavior and conditional effects should be compared on the reserved names and templates; if only the name is changed after the name is changed, first check whether the reference library has coded the name and identity. The two interpretations require different controls, and they cannot be re-started after failure, and then the new combination is packaged as a null hypothesis.
Another distinguishing experiment is the fixed intervention set, changing the size of the reference library and repeating the donor seeds: if the symbol is stable and the size fluctuates, the main problem may be the estimation accuracy; if the symbol system changes, the definition of "component importance" itself depends on the reference distribution. What is proposed here is the hypothesis to be verified. These new settings are not run this time, and the migration is not pre-promised to succeed in advance.
Code and instructions for reproducing the results
Fixed versions published in this article: Full source code directory、README and run instructions、SOURCE_MAP vs upstream mapping. Forty-four distribution files have been anonymously downloaded, consistent with the local source code and archive-by-file hashing; the public copy is tested by independent audit and real model.
After installing the lock dependency in README, the minimum entry is:
python run.py --cache ./cache --out smoke-new --smoke
python audit.py smoke-new
The first command to verify or download fixed resources, run a real goal and twelve reference sentences, and generate a condition-by-condition record, reference activation, a summary, environment and stage states; the second article performs independent recalculation. The formal scope is sixteen goals, 336 interventions and all realizations. The archive contains all the self-written modules and necessary evidence, and the model weights are obtained by a fixed link. Do not mix the minimum test run output with the official full result.
Summary
Ablation is an experiment with reference values, action locations, and index selection. The progress of this verifiable is to write the non-additive and conditional dependence of the joint effect into a complete record; what is still missing is the compensation path and the allowance for migration evidence. Only by separating the two can the positioning results have the opportunity to develop into a mechanism that can be tested.
References
Retrieval and access date: 2026-10-19. The above results are from this log, and the following first-hand data provides concepts, models and implementation basis.
- Kevin Wang、Alexandre Variengien、Arthur Conmy、Buck Shlegeris、Jacob Steinhardt,2022: Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small,v1.
- Fred Zhang、Neel Nanda,ICLR 2024: Towards Best Practices of Activation Patching in Language Models: Metrics and Methods,v2. The sensitivity background of the method does not mean that the complete experiment is reproduced this time.
- Redwood Research: Fixed IOI templates. The license and formula correspondence are shown in the source map.
- Hugging Face: GPT-2 Fixed Model Card. The model and tokenizer file identity are locked by eight SHA-256 items.