cover

  • Date: 2026-10-21
  • For graduate students, early-career researchers and research engineers
  • Scope: Real GPT-2 small, 256 synthetic materials with public template structure; only check the last attention node, not the complete IOI circuit recurrence.

Abstract: The three important heads are not enough to be an explanation

The previous experiment fixed a set of three attention heads and found that the direction of the intervention effect carried over to new names and sentence patterns. Showing that changing these heads affects an answer establishes their importance, but leaves open how much of the answer they explain. Here we reverse the test. We retain the three candidate heads, replace the other attention outputs at the final token position with reference values, and compare the resulting system with the full model.

A clear negative result was obtained in this actual verification: at zero replacement, the preferred number of correct answers to the four groups of correct answers is all zeros; More importantly, the gap can be widened by continuing the same candidate head in the two systems at the same time. We don't deliver a circuit diagram that declares success, but a set of control records that can overturn strong interpretations.

1. Reproducing value: from positioning components to inspection and interpretation

Indirect object recognition (IOI) requires the model to make up for the person who accepts the item, such as "A and B go to the store, B hand over the ring to..." should take A. The attention head here is a set of computing units that read the context and write the representation inside the Transformer; the last bit END is the location of the last token of Prompt.

The paper report distinguishes between loyalty, integrity and minimum: whether the reserved explanation can be close to the original model; whether it can still be consistent with the original model after continuing to knock out the component; whether it contains unnecessary components. The paper also reports its interpretation of an unremoved gap. This article borrows these test problems, does not reuse the paper scores, and does not claim to reproduce its complete circuit. The fourth section of the original paper

The research value of this step is to turn the narrative of "finding three high-segment heads" into a failed prediction. If the three heads are already enough to bear the claimed last bit calculation, then the output after retaining them should not deviate from the original model seriously. After the result fails, the scope of interpretation should be narrowed, rather than modifying the success criteria.

2. Core ideas and formulas: the average closeness may still be missed to calculate

set the input as xx, the complete model is MM, the correct name and the interference name are IOIO and SS. Use unnormalized output fractions, i.e. logit, to define:

DM(x)=ℓIO(M(x))−ℓS(M(x)).D_M(x)=\ell_{IO}(M(x))-\ell_S(M(x)).

here ℓ\ell It is the score of the corresponding token. The positive difference only indicates that the correct name is better than this interference name, and it cannot guarantee that it exceeds the entire vocabulary. Therefore, the preferred vocabulary and the correct name probability are recorded at the same time, and the advantage of two choices is not called complete Accuracy .

Make UU It is a twelve-layer, twelve heads of each layer at one hundred and forty-four nodes of END; candidate set HH It is A=L8H10, B=L8H6, C=L9H9, the layer and head are numbered from zero. Hold UU does not belong HH After the output is replaced, the local reserved system is obtained C∗C^*. All non-last positions, MLPs, embeds, residuals, and normalization still exist; they are shared backgrounds. Therefore, this notation does not mean that there are only three heads left in the model.

against the same group nn input, define F(Q)=n−1∑iDQ(xi)F(Q)=n^{-1}\sum_i D_Q(x_i), in QQ can refer to any system after intervention. First report:

E=∣F(M)−F(C∗)∣,R=1n∑i∣DM(xi)−DC∗(xi)∣.E=|F(M)-F(C^*)|,\qquad R=\frac{1}{n}\sum_i|D_M(x_i)-D_{C^*}(x_i)|.

EE is the difference of the group mean, RR It is a sample-by-sample absolute residual mean. Looking at the two together, the positive and negative errors will not be offset. Integrity pressure test re-enumeration HH eight subsets KK:

G(K)=∣F(M∖K)−F(C∗∖K)∣.G(K)=|F(M\setminus K)-F(C^*\setminus K)|.

The set difference here means that the specified header output is replaced with a reference value, not a weight of deletion. Use the same on both sides KK And the same replacement rules, the remaining nodes re-formate pass . We are exhaustively a subset of three candidate nodes, and there is no exhaustive model of all nodes, let alone global integrity.

3. Guide to the official implementation: read the reverse of "reserve" first

Read the official first process_heads_and_mlps: The reserved list will be transferred to the list of removed locations. Read again get_circuit_replacement_hook, check which head and which token is overwritten. Final reading do_circuit_extraction the reference data and the group mean. Omitting this step makes it easy to mistake "other heads to continue working normally" for "other heads have been excluded". Official extraction code

A complete GPT-2 forward process that uses Transformers independently, and the projection is output in the attention output c_proj Replace slices before. The input shape is [1,T,768][1,T,768], can be understood as twelve heads as [1,T,12,64][1,T,12,64]; TT is the current prompt length, each head contains sixty-four numbers. Only the last bits are covered, and the uncovered parts are checked by precise equal assertions. The output is still projected by the official subsequent layer and vocabulary, and the clean answers of the cache are not forced back to the reserved system. Official GPT-2 implementation

4. Minimum Experiment and Evaluation Protocol: Change the reserved scope and freeze the selection

Follow the four sets of samples in the previous article: old names and old templates, new names and old templates, old names and new templates, new names and new templates, each group has 64 pieces. The first eight and the last eight of the public list are taken, and the template is index zero to three; the location and item are fixed. Eight loop name pairs, two role directions, and two appearance sequences form a balanced design. Official Template and Name

All these data have been deduced in the previous article, and this article is an exploratory test on the observed data. The previous head selection remains unchanged, and you cannot see the failure to pick the head again, and then call the same batch of samples as the confirmation set. The new variable is the pairing relationship between "Remove Candidate Subset" and "Remove All Last Heads of the Candidate Set".

There are two types of replacement baselines. Zero replacement clears the specified vector; the ABC mean replacement is filled with three different names, and the average output of the six arrangements is taken. The four templates each have six references, a total of 24, all from the unintervented model. The reference names are fixed as Michael, Christopher, and Jessica, and the sub-template is averaged in END, and the reference is not used to use the target answer screen.

This small reference library is not equal to points for all names: it retains constant information of the reference distribution, and may also be biased towards the old name. The gap between old and new groups cannot be attributed to circuit migration alone. We chose to report the difference between two substitutions and keep this boundary, rather than pick the better baseline as the only answer.

Partial Circuit Evidence Verification Process

The green area in the figure represents the background module that always participates in the calculation; the orange control enters two branches respectively. The method diagram only explains the flow and does not carry the measured value.

Running a revision with GPT-2 607a30d783dfa663caf39e06633721c8d4cfcd7e, CPU four threads, float32, batch 1. Random seeds 105; no training, generation, completion or truncation. First do a single sample resource test run, and then execute 256 articles: a total of 7,936 independent scoring forwards, and another 1,024 times to control the forward direction and 24 times to refer to the forward direction. The official process measured about 231 seconds, and the peak resident memory is about 1.465 GB; the GPU video memory and energy consumption are not measured.

5. This actual verification: reserved signal is not equal to reserved behavior

The following tables are all recalculated for this saved log, FF Indicates the above group average, and each group has a full score of 64 in each group. There is no mix with the original benchmark of the paper.

Name/Template complete model FF Zero replacement reserve FF Mean Replacement Reserved FF Full/mean reserved preferred number
old/old 4.446 0.125 2.608 64/24
new/old 2.947 0.061 1.688 64/0
old/new 3.877 0.115 1.976 56/15
new/new 2.479 0.046 1.287 53/0

The number of preferred zero replacements is zero, but there is still a correct name that does not fEW-Shot truces the interference name. The mean replacement also occurs in the case of "the preference between the two names is still there, and the answer to the vocabulary is already wrong", so a positive average logit difference is not enough to explain the remaining errors.

Replacement of old/old group mean E=1.838E=1.838, R=2.053R=2.053, which means that the offset does exist. The maximum absolute residual is taken according to the rules written in advance. The correct answer to sample 46 is Amanda: the difference of the complete model is 5.656, and it becomes −0.455 after local reservation. Not only does it become weak, it also changes the direction between the two names; this sample is kept in a case-by-case record, along with the Prompt and Token codes.

When continuing with the Ablation C header, the average system of the same group is 3.975, and the reserved system is 0.415, and the gap widens to 3.559; the preferred number of corresponding vocabulary tables is 63 and 5. All eight subsets are reserved, and the maximum difference between the four sets of mean substitutions occurs at Ablation C. This maximum value comes from the exhaustion of the predetermined set, which is a descriptive pressure result, not a significant discovery of the verification on the new data.

6. Failure investigation: first exclude measurement errors, and then explain negative results

For every target example, the no-op intervention and self-patching of all 144 heads at the final token position matched the full model exactly. An alternative implementation adds a vector difference after the output projection. On the specified controls, its largest discrepancy across vocabulary scores was about 0.000221, below the preset threshold of one thousandth. An independent script recomputed all per-example differences, group means, absolute residuals, counts, and counterexample selections. Eight deliberately corrupted records confirmed that the audit rejects invalid evidence.

All 7,936 scoring conditions have been completed for the replay of the independent model: the first choice of the vocabulary is all the same, and the maximum difference between the candidate scores is about 0.000183, the maximum probability difference is less than 1/100,000; the double-precision recalculation difference of the reference mean is less than one in a million. The arithmetic re-calculation can only prove that the aggregation is consistent, and the argmax of the entire vocabulary cannot be proved by the two saved scores, so the two layers of verification are recorded separately. After the first read of the source network fails, the recovery has been restored through authorization anonymous reads, and the failure record is reserved; the experimental configuration has not been modified to cater to the results.

There are still three limitations at the interpretation level. Wide range of zero substitutions may cause a distribution shift; the ABC mean is not a neutral "no information"; the non-terminal and MLP paths continue to work and may provide support beyond the candidates. It is found that the error does not automatically locate which missing edge, and after discovering the gap, the gap expansion does not automatically prove that there is a backup circuit.

Samples share names, templates, and role pairs, so they don't report the common independent confidence interval. What is measured is the gap under the definition of this set of construction materials, these replacement values and local node definitions, not the overall error rate of natural texts.

Seven, the verifiable research questions and the stop boundary

The author infers that the candidate head does have a local effect, but it is not enough to independently bear the omitted last bit calculation. The next step should distinguish two explanations of "there is a lack of key downstream headers" and "reference replacement carrying name bias". You can freeze a more balanced reference name library first to check whether the gap between groups has changed; then expand the candidates on the development data, match the number of nodes and keep a simple control. It is only suitable to confirm the improvement only if the sample is not observed.

These are follow-up questions that have not yet run, and cannot be released from the current table which interpretation has been established. This article stops searching here, and does not regard a beautiful picture as a discovery, nor does it continue to add components that can improve the score of the same batch in order to collect the circuit.

Code and instructions for reproducing the results

The supporting source code has been disclosed and anonymous download verification has been completed: Source tree at the pinned revision、Running Instructions README、The official implementation corresponding table SOURCE_MAP. Fifty-four files are consistent with the local archive by byte, and the audit of the public copy is passed.

After installing the lock dependency, the minimum real running command is python run.py --cache ./cache --out smoke-new --smoke. The first time you need to download about 550 MB model resources; you can also reuse the cache that has been verified by file. Expected products include one-by-case scores, reference mean, stage state, environment and time-consuming records. Get rid of --smoke And the output directory can run 256 complete articles; audit.py without loading the model, replay.py The independent model playback is performed. This time, the test run and the full experiment were actually carried out, and the complete paper benchmark was not run.

Summary

This article promotes the positioning of components as an explanation audit: reserve candidates, compare residuals, and then apply the same subsets ablation on both sides. The real result does not support the complete circuit, but it is clear that how many behaviors will be missed by the three last-bit heads. The verifiable failure record is better suited as a starting point for the next study than a successful graph that lacks boundaries.

References

Search date: 2026-10-21. The following are first-hand materials that are actually opened and verified; the official documents are fixed and the source code map is fixed.

  1. Kevin Wang、Alexandre Variengien、Arthur Conmy、Buck Shlegeris、Jacob Steinhardt,2022, Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, arXiv v1 and full text.
  2. Redwood Research, Implementation of a pinned version of the circuit extraction and data template,MIT.
  3. Hugging Face,Transformers 4.49.0, GPT-2 Official Forward Implementation,Apache-2.0.
  4. OpenAI/Hugging Face, GPT-2 Fixed Model Card, MIT; the model is consistent with the exact revision of the tokenizer.