
- Date: 2026-10-18
- For graduate students, early-career researchers and research engineers
- Scope: The real GPT-2 small, publishes sixteen development samples constructed by the IOI template, two names are counterfactual; not a natural corpus or a complete paper benchmark.
Summary: The most important head depends on what was asked
In the previous article, the attention head of the clean input was activated and put back into the rewritten input, and it was found that the average recovery effect of the L8H10 at the end was the largest. The layers and heads here are numbered from zero. There is still a lack of a check: if the sentence is changed in another way, does the ranking remain? This article freezes the model, sample, intervention and score, only change the name and replace the rules, and run the complete grid again. What we tested: the top ten are the same, and the rank correlation of all heads is 0.857, but the first place has changed from L8H10 to L9H9; the average recovery ratio of L8H10 has dropped from 28.6% to 12.5%. This is a local stability experiment, and the same ranking cannot be regarded as a complete circuit, nor can it be called the interpretation failure directly.
1. Recover value: first clarify the semantics of counterfactuals
Indirect object recognition (IOI) requires the model to determine who is handed over to the model based on the previous article. For example, "Michael and Christopher go to the store, Christopher hand over the ring to...", and expected to continue to write Michael. Remember that the recipient is IO, and the recurring subject is S. The so-called corruption, this article refers to a regular rewriting input to form a different task from the original sentence; it does not mean adding noise to the model at will.
We keep two counterfacts. S2 Flip only changes the second occurrence of S to IO; Global Swap switches all the IO and S in the same sentence at the same time. The former changes the last actor, and the latter exchanges the identity of the character consistently. The two rewriters are the original S, so the model is more inclined to the original S, which can correctly understand the new sentence. The full text always calculates the score of the original IO relative to the original S, to avoid the replacement of the answer and then falsely claiming that the recovery was successful.
| Set up | An example of the original name sequence | rewrite post-sequence | Change Tokens |
|---|---|---|---|
| clean input | IO、S、S | Constant | 0 |
| S2 Flip | IO、S、S | IO、S、IO | 1 |
| Global Swap | IO、S、S | S、IO、IO | 3 |
The names here can be encoded as a single token with leading spaces. The two input lengths, the frequency of names are the same as the non-name tokens, but different locations have been changed. The difference cannot be attributed only to the "number of changes": the order of the first mention and the replacement position also change accordingly.
2. The paper gives the motivation, the problem is limited in this experiment
What the paper reports: In their IOI study, Wang and colleagues combined several interventions to investigate components in GPT-2 small. They evaluated their explanation for faithfulness, completeness, and minimality [1]. Zhang and Nanda compared corruption methods and scoring choices for activation patching and found that these choices can change which components are localized [2]. They distinguish Gaussian noise from semantically related token substitutions and discuss why the choice of tokens to replace matters.
This article only compares two discrete names counterfactuals, without redoing its Gaussian noise experiments. We also did not run the complete model transformation of TransformerLens: the actual forward Transformers are actually forwarded, and the convention for substituting slices is referenced to the TransformerLens fixed source code [3][4]. This distinction makes the realization, the conclusion of the paper and the author's interpretation have a clear boundary.
An additional design constraint is balance. The sixteenth article covers two templates, two reference sequences, two adjacent name pairs and two role directions, and only involves three overlapping names. In the complete material, the multiple sets of each rewriting sentence are equal to the multiple collections of clean sentences, but the matching relationship with the original sentence changes. Therefore, the overall baseline difficulty is controlled, but the information path entered by a single entry is not the same. This property is checked item by item by tokens sequence, and does not rely on post-mortem selection of samples with similar scores.
3. Core ideas, formulas and modules

The tokens in the figure use the example sequence of the original IO, S, and S; the warm color highlights the changed position, and does not encode the identity of the character. The two paths share clean sources, node definitions and original candidate answers, and the ranking results will not be fed back to this selection rule.
Define the candidate logit first. Logit is the model output before normalization to probability:
Among is a sample, is a condition of destruction, A layer-head combination at the end of the main ranking; is the output score of the original candidate, It is an increment of the intervention relative to the same damage baseline. The positive value indicates that the preference moves to the original answer, and there is no guarantee that the correct prediction has been restored.
The main indicators are averaged according to the sixteenth article, and the proportion of auxiliary reports is restored:
. The denominator is average clean-destruction spacing, which must be greater than the predetermined threshold; do not average the ratio of each question, and do not cut the negative value to zero. The denominator is the same under the same conditions, so the two indicators give the same top ranking, and they cannot be regarded as independent support evidence.
Intervention object is an attention weight vector , before the output projection. The cache shape is , view as : batch, sequence position, twelve heads, sixty-four dimensions per head. Only one layer, position, and header are replaced at a time, and then the subsequent network is calculated normally. It is different from directly modifying the attention weight, and does not replace multiple heads at once.
The guide to the official implementation
Read the model card and resource lock first, and confirm that GPT-2 shares a fixed revision[5] with the tokenizer. Look at the official again GPT2Attention The merge of the weighted value output and c_proj, make sure the hook is inserted on the correct boundary; finally compare the TransformerLens layer_pos_head_vector_patch_setter, to verify the meaning of the location and head dimension. Specific submissions, line numbers, licenses and local maps are all in the source map.
Local run.py The forward, cache, single-point replacement and controls follow the previous article; the new part is only the counter-fact structure, condition loop and log marking. analyze.py aggregation ranking, audit.py Independent recalculation, the latter does not import experiments and aggregate scripts. public template by rebuild_samples.py Obtain the fixed source code and resolve the constant reconstruction [6], and do not execute the entire upstream repository. All required self-writing modules are distributed with this article.
V. Minimum Experiment and Evaluation protocol
The model and tokenizer are fixed in 607a30d783dfa663caf39e06633721c8d4cfcd7e; the template is fixed in ea15315dd24481e9e2ac5c3ef335d82907a1dc34. The configuration continues with seed 101, CPU float32, EAGER attention, four threads, and 32 intervention batches. No training, generation, truncation, additional initiation tokens or set-out reasoning.
Save the protocol with the script hash before the official execution. First run the two conditions of a material to complete the grid to estimate the resources, and then run all sixteen. Each condition scans the twelve, twelve, and all 15 or 19 positions, totaling 78,336 single-point replacements. The last position END is the main ranking position that has been stipulated in the previous article. The actual resource is the official process wall clock 396.294 seconds, peak resident memory 1,607,516,160 bytes; after import, the timer is 395.671 seconds, the system is macOS, Python 3.12.14, PyTorch 2.6.0, Transformers 4.49.0; undetected energy consumption or independent video memory peak.
The main comparison was pre-fixed to the rank correlation of all one hundred and forty-four END heads, the top ten overlapping degree, and the pairing changes of the previous experiment winning head L8H10. The coincidence is the intersection size divided by ten, and Jaccard is divided by the union size. Sorting in descending order of average increments, the order is determined according to the layer and head number when the precise equality is equal. At the same time, we check the difference between the neighbors and avoid exaggerating the ranking changes that are close to the numerical error. Samples share templates and names, and you cannot draw Confidence interval in sixteen independent natural samples.
6. Result: Look at the stable part and the change part at the same time
The average clean score difference between the two conditions is 4.295, the damage difference is −4.295, and the average spacing is 8.591. The single tokens condition reproduces the 144 END rankings and increments of the previous article. The following are the true values of the same node under two conditions:
| END node | Single tokens increment/rank | Full Name Exchange Increment/Ranking |
|---|---|---|
| L8H10 | 2.455/1 | 1.078/3 |
| L8H6 | 2.215/2 | 0.113/11 |
| L9H9 | 1.859/3 | 1.992/1 |
| L10H0 | 1.332/5 | 1.364/2 |
| L10H7 | −3.841/144 | −3.782/144 |
The top ten coincidences are 90%, and Jaccard is 0.818; L8H6 exits the top ten, and L10H2 enters. There is a considerable part of the overall ordering, but the expression "the only most important head" does not hold. The L8H10, which was prescribed in the previous experiment, decreased in increments and increased by one on fifteen materials, and the average pairing change was −1.377. The exception is Sample 2, from 2.363 to 2.735, indicating that the average drop cannot be used as a case-by-case illustration. Even if the single head with the strongest conditions was used separately, the average score difference after the intervention was still negative, and the complete behavior was not restored.
Neither condition had an exact tie. Among the 143 adjacent rank pairs, however, 102 in the first condition and 107 in the second differed in gain by less than 0.001. The gaps between tenth and eleventh place were 0.00355 and 0.10615, respectively. Neither gap establishes statistical significance. L8H3 moved the farthest, dropping from fourteenth to 138th, even though its raw gain changed only from 0.079 to −0.035. A fall of more than a hundred places need not imply a behavioral loss of comparable magnitude.
Negative effects also constitute evidence. In the full-position grid, the number of nodes with increments less than −0.001 is 3,142 and 6,964, respectively; this contains different layers and positions, not the number of independent failures. The L10H7 is arranged at the end of both conditions, and the clean activation writing to the current destruction context will further bias the answer to the original interference. At this stage, this conditional effect can only be reported, and it cannot be directly labeled as a general "harmful component".
A common misreading is to regard the global rank correlation as a "percentage of interpretation credibility". The one hundred and forty-four heads here contain many near-zero effects, and the order may be sensitive to minor changes. Therefore, the report not only retains the overall order, but also displays the original increment of the main head, the interval between the first ten boundaries and the pairing results of one by one. Changes near the threshold and significant amplitude changes should be understood separately.
7. Failure investigation and independent acceptance
The replacement experiment first verifies the implementation, and then explains the heat map. After the entire paragraph is embedded, the state is changed to a clean state, and the forward direction should be restored; the destruction cache should be written back to itself, and it should be completely unchanged. We also use the linear increment after the output projection to redo the pre-specified node, replace the control with the pre-projection, and check the realization of the complete official model and the read-only last position. The node before the real modification of the position should only have a numerical difference within the allowable range.
This complete status control and self-rewriting error are both zero. The maximum error of the official complete forward check is 0.000191, the independent projection increment check is 0.0000534, and the batch shape difference is 0.000221; 0.0000610, all less than the predetermined tolerance of 0.001. This tolerance is the realization threshold, not a Confidence interval for the real effect.
Independent scripts recalculate the original candidate difference, complete grid uniqueness, mean, ranking, recovery ratio and multiple sets; the Pearson formula of the sequence number vector is calculated separately. Eleven damage tests intentionally change candidates, change positions, miss nodes, duplicate nodes, or forged rankings and coincidences, all should be rejected. Automatic arithmetic re-calculation is not called the real-life semantic gold standard. The source code networking is restricted by the sandbox for the first time, and the subsequent anonymous fixed version is successfully read, and the experimental protocol or model assets have not been changed.
8. Research questions and boundaries
Author’s interpretation: If the same clean source has different effects under different counterfactual paths, the component contribution may depend on the downstream context. This is an explanation to be further differentiated, and the current ranking cannot determine which information path has changed. At the same time, the name position and the order of mention have been changed, and there is no independent manipulation of one of the factors; there is no evidence that the two paths are equivalent to any natural text.
The following questions that can be verified are: fixed the heads selected for this development collection, is the loss of joint ablation equal to the sum of the individual losses? If not, there may be interaction or compensation, which requires different ablation baselines. After that, the migration was tested on the reserved name and template. You can't start again after the migration fails, and then the same batch of data is called a confirmation set. This is still the development stage evidence, and repeated samples, limited templates and potential pre-training contacts all restrict extrapolation.
Code and instructions for reproducing the results
Fixed versions that have been publicized and verified: Full source code directory、README and running steps、SOURCE_MAP official implementation mapping. The 49 files are consistent with the local archives item by item; the open copy has been tested by independent audit and a real test run of 4,320 substitutions. The official results of this paper are sixteen double-conditional complete scans.
After installing a lock dependency by README, the minimum real running command is:
python run.py --cache ./cache --out replay-smoke --smoke
The first execution is performed to download and verify about 550 megabytes of model resources by fixed revision; the verification cache can be reused. Output conditional samples, all intervention scores, summary, stage state and operating environment. python audit.py --results replay-smoke Can be recalculated independently. Get rid of --smoke That is, it is the complete scope of the sixteen articles of this article; the output directory must be a new catalog to avoid overwriting the original evidence. The public package contains the original results and audit records, and does not contain the private paths of weights, virtual environments, or author machines.
Summary
We can now give “Which head matters most?” a testable meaning. With the model, candidate answers, and intervention sites fixed, how does the ranking of contributions change when we change the counterfactual pairing? A robust account should identify which heads retain their standing, how large the changes are, and where the experiment’s conclusions apply. The ranking can guide the next intervention. Explaining the mechanism takes further work.
References
Search and access date: 2026-10-18. The following are all first-hand data; the hash of the fixed source code and the license to see the source map of the source code.
- Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, Jacob Steinhardt, 2022 first edition: Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small,v1.
- Fred Zhang、Neel Nanda,ICLR 2024: Towards Best Practices of Activation Patching in Language Models: Metrics and Methods,v2.
- TransformerLens: Fixed version activation replacement implementation.
- Hugging Face Transformers: Fixed version GPT-2 forward.
- OpenAI / Hugging Face: GPT-2 Fixed Model Card.
- Redwood Research: IOI public template fixed source code.