
- Date: 2026-10-17
- For graduate students, early-career researchers and research engineers
- Experiment Scope: Real GPT-2 Small, 16 English samples for public template construction, 39,168 single-point interventions; development positioning, does not represent a complete benchmark.
Abstract: Recovering part is not the same as explaining all
The previous article confirmed that the model has indirect object recognition ability on fixed names and sentence patterns. This question: How much can the answer preference be restored if one internal vector in the correct operation is transplanted into another operation? We scan the entire layer, head and position, and the Attention Head, which has the strongest performance at the end, restores an average logit gap of about 28.6%, but the average preference after recovery still points to the interference name of the original question. This result provides a component worth continuing to check, and also leaves a clear explanation gap.
The original paper offers a mechanistic account based on its full study. Our experiment tests a limited set of activation interventions, and we use our interpretations only to frame hypotheses that subsequent work could refute. We did not train a model or remove the other components to establish that a particular head could perform the task on its own.
Reproduce value: make "relevant" a manipulative quantity
IOI is indirect object recognition. Taking this sample as an example, the sentence introduces Christopher and Michael, and then said that Christopher gave the ring..., the name to be continued should be Michael. The attention map can show where the model is looking, but cannot directly tell us whether the message has changed the answer. Activating the replacement changes one of the forward passes on the way to compare the final output change.
The original paper of Wang et al. combines path replacement, ablation and other analysis to propose circuits, and discusses loyalty, integrity and minimumness; this article only reproduces the relevant positioning ideas, and does not equate single-point recovery rankings with this circuit. Original paper
Compared with the previous article, model weights, tokens, templates and scoring candidates remain fixed. The new variables are only the intervention node and the clean/corrupt conditions; the new product is the original candidate score by sample, layer by layer, heading, and position by one. The "true" here refers to the real pre-training model and official calculation implementation, and the language material is still a controlled synthetic sample.
The core idea: first make the destruction clear
clean means the original input, and corrupt represents the controlled modification input. We only replaced Christopher for the second time with Michael. At this time, the correct recipient of the sentence also changes; it is not random garbled, let alone the correct response of the model to the new sentence is called a decrease in ability.
In order to measure whether the original question preferences can be restored, both conditions use the original question Michael to reduce the score of Christopher. The program confirms that the two names are a token with leading spaces, and the change happens to happen in a token, the length remains unchanged, and the decoding and re-encoding are consistent. Scoring status is not secretly exchanged with conditions, otherwise both conditions may get high scores, but the direction of intervention cannot be measured.
The formal sample is determined before the new reasoning: from the previous article, the first two adjacent name pairs are selected, and the two templates, two initial name orders and two-way role exchanges are exchanged, a total of sixteen items; in fact, only three overlapping names are actually involved. They share structures and cannot be regarded as sixteen independent language phenomena. The eight names and the other two templates left are not used, and the development samples that have been seen are not renamed as test set .
Formulas and modules: which one to replace
Attention head first weights the value vector to sum to get the header output , and then write the residual flow through the output projection. The residual flow can be understood as an intermediate representation of common reading and writing by each module. We do the replacement before projecting:
Among is the layer, is the token position, Yes, the right side is run from the same pair of cleans; other nodes continue to calculate by corrupt input. GPT-2 Small has twelve layers, twelve heads each, and each head is sixty-four dimensional. The single sample cache shape is , It is fifteen or nineteen; in the middle of realization, the rear two-dimensional is flattened into seven hundred and sixty-eight dimensions before the projection is realized, so it must be divided according to the head, and the entire layer output cannot be regarded as a head.
The main metric follows the logit difference of the two candidates. Logit is an unnormalized fraction before softmax:
here represent samples, It is the final position vocabulary score, and IO and S always refer to the original receiver and the subject; It is the original question preference increased by this intervention. The auxiliary recovery ratio is:
The horizontal line represents the arithmetic average of sixteen samples, and END refers to the last input position of each. We first demand an average and then divide, and the denominator is pre-required that the denominator is greater than 0.1; there is no dividing by one by the gap that may be close to zero, and the negative value or more than one value is not cut off. The recovery ratio is not a probability, and it cannot be interpreted as "the amount of intelligence is borne by this head".
A guide to the official implementation
Read the name, template, and sequence conversion of the public IOI data script first, and then compare the sample rebuilder in this article. Then look at the position of the TransformerLens - head replacement function: it only replaces the specified slice with the clean cache, and the output scan axis is the layer, position, and head. fixed implementation
Finally, enter the GPT-2 of Transformers for attention forward: the weighted results of each head are spliced and then sent into c_proj. This article is linked at the input of this module, and the official Forward Pass is used to independently transplant and replace the operation; the entire set of TransformerLens is not installed and run, and there is no claim that the two sets of frameworks are equivalent after weight transformation. The source map records are pinned commit, specific lines and MIT/Apache licenses; the official GPT-2 file has been downloaded and checked to be consistent with the installation file bytes.
A minimal experiment and evaluation protocol

The blue cache in the figure only passes values to one node running orange. The three boxes below are independent controls, not continuous steps; FULL STATE actually replaces the entire embedded residual STATE, and replaces them using the corrupt cache of each layer. The bar in the figure is only a score symbol, not a measurement value.
Run a sample's complete scanning estimation resources first, and then implement the official 16 articles. Only one node is changed at a time, and all grids are reserved; pre-specified main display end position head rankings, sorted by average gain, draw by layer and header number. Each row of the batch size of 32 represents an independent intervention, and the final batch is discarded after the batch is completed and the excess output is discarded; the same batch shape is also used for the comparison baseline.
Three controls are indispensable. The unreplaced clean/corrupt gives the real denominator; the entire embedded state is changed to clean, and the clean output should be reconstructed. The latter two terms are precisely equal to the complete vocabulary scores on the sixteenth item. The complete status control proves that the running link can be restored, and does not prove that "all attention heads contain all mechanisms".
Officially runs using CPU, single precision, four threads and fixed random seeds. The wall clock takes 190.646 seconds, and the process peak memory is 1,600,782,336 bytes. A single clean activation cache is about 0.76 MB, but it cannot be used instead of the process memory that contains weights and runtime. Download the verification, preprocessing, reasoning and evaluation states to save them respectively; there are no training or generation steps.
What we tested: the highest head still has no answer preferences
The following are all derived from the save results, the layer and header are numbered from zero. The mean difference between clean and modified inputs is 4.295 and −4.295, respectively, and the average gap is 8.591. The mean value of the two sides is just reversed from this symmetrical character design, not the nature of the intervention method.
| final position intervention | average gain | Recovery ratio | Mean difference after intervention |
|---|---|---|---|
| Eighth Floor Tenth Head | 2.455 | 28.6% | −1.841 |
| Eighth floor sixth head | 2.215 | 25.8% | −2.080 |
| ninth floor ninth head | 1.859 | 21.6% | −2.437 |
| 10th floor seventh head | −3.841 | −44.7% | −8.136 |
The first three heads have positive gains on each sample, but this does not mean that each bar is correct, let alone a combined recovery of the proportions. The average absolute probability of the answer to the original question ranked highest from Corrupt 0.0108 to 0.0657, still lower than Clean's 0.5177. We didn't pick up a better-looking head again with probability.
Negative results are equally important: some of the clean header outputs have been transplanted in the past, but instead enhance the preferences in the opposite direction. This may involve downstream interactions or changes in conditions, and the global "hazardous" label cannot be labeled accordingly. The original mesh saves all the effects, not only the top few are exported.
Failure investigation: rank numerical problems first, and then talk about the mechanism
This model test run and the official operation are successful, but passing does not mean that there is no error. In order to save the scanning cost, the program only sends the last position into the official output layer; the maximum difference between the score of the vocabulary and the complete model call is about 0.000019. The maximum difference in batch size changes is about 0.000022. They are lower than the pre-set 0.00-1 tolerance, and the records are still intact.
Another check uses the algebraic writing method of "replacement of vector difference multiplied output projection" independently. The maximum error is about 0.0000046 for the predetermined node compared to the input hook version. The causal mask provides structural negative control: the position before the change of tokens cannot see future changes, the transplant effect should be close to zero, and the actual maximum gain absolute value is about 0.0000061.
This also explains why there are not all the effects that are strictly less than zero to declare the inhibitory mechanism: many tiny negative values of more than 39,000 Gree are numerical scales. There are 3,142 grids below negative 0.001, but these grids share samples and calculation paths, and are still not the number of independent discoveries. The independent audit also checks the unique complete grid, candidate identity, difference and ranking, and confirms that the inspection can indeed report errors with eight kinds of intentional damage to evidence.
From positioning results to verifiable research questions
The author's inference is: the current ranking may reflect the wrong path of the output copy ability and the production of this role exchange. The next step is to freeze this realization, only change the damage method, and compare the recovery effect and sorting of the same set of heads. If the ranking is stable, the evidence of cross-destruction conditions is added; if the change is obvious, check which information path is changed.
Another competitive explanation is that a certain head only carries a signal that adapts to the current name. Leaving the name and template can distinguish the more general role mechanism and term dependence, but the candidates must be frozen first, and the results cannot be seen before changing the head. Whether the joint replacement is compensated or offset also requires a real run of the combined experiments; this paper does not have these results. There is currently no independent and same distribution of related sample sets, and no development ranking is written as a significant conclusion.
Code and instructions for reproducing the results
This article has been published: Source tree at the pinned commit、README and run instructions、SOURCE_MAP corresponds to the official implementation. The four entrances of this version can be anonymous, and forty-seven files are consistent with the local archives one-by-one hashing; the complete audit of the downloaded copy, the eight damage tests and the real model test run have all passed.
After installing the lock dependency by README, the minimum real running command is:
TOKENIZERS_PARALLELISM=false python run.py --cache ./cache --smoke --out smoke-new
python audit.py --results smoke-new
The fixed model resources will be downloaded and verified for the first time; there is already a correct cache that can be reused. It is expected to obtain sample-by-sample records, all single-point scores, summary, environment and stage states. The test run covers 2,160 nodes of a sample; remove the test run parameters is the scope of this article. Offline auditing can recalculate the preserved evidence, but does not replace the model forward. The distribution package includes all self-write modules, configurations, source correspondences and permissions, and does not contain implied paths for weights, credentials, or author machines.
Summary
Activation patching lets us investigate a component’s importance with a repeatable intervention. We found heads that partially restored the model’s preference on the original prompt. We also recorded cases where patching a head alone failed to restore that preference or shifted it in the opposite direction. The findings apply to this model, corruption procedure, and sample. A mechanistic explanation still has to survive alternative corruptions, joint ablations, and transfer tests on held-out examples.
References
Search and source code verification date: 2026-10-17.
- Kevin Wang、Alexandre Variengien、Arthur Conmy、Buck Shlegeris、Jacob Steinhardt,2022: Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small,v1.
- Fred Zhang, Neel Nanda, 2023 first edition / 2024 revision, ICLR 2024: Towards Best Practices of Activation Patching in Language Models: Metrics and Methods,v2. The influence of this paper and the influence of selection on positioning conclusion is supported, and the motivation of this paper to continue to test the sensitivity of the protocol is supported.
- TransformerLens: Exploratory Analysis Demo. The specific porting basis is subject to the pinned commit in the source map of this paper.
- Redwood Research: Fixed IOI data construction source code.
- Hugging Face: Fixed GPT-2 model card and Transformers 4.49 corresponds to GPT-2 source code.