cover

  • Date: 2026-10-20
  • For graduate students, early-career researchers and research engineers
  • Scope: The real GPT-2 small, the synthetic material of the IOI template structure is disclosed; the four groups are 64 each, which does not represent the complete benchmark of natural language.

Abstract: The direction has migrated, but the intensity has not been copied

The previous experiment compared clearing, mean and resampling ablation, indicating that "decreasing the score after deleting several heads" cannot directly prove that the complete circuit has been found. Add a constraint today: no longer look for better heads, and walk into reserved names and templates with the same set of components.

Intervening on the three heads together had the same direction of effect on every example. Zeroing them lowered the score of the correct name relative to the distractor, while patching their activations raised it. Yet when both names and sentence patterns changed, the mean drop under ablation shrank from 2.387 in the original group to 1.360, and the number of correct top choices by the full model fell from 64 to 53. Recovery was about 60% in each case, concealing the difference in the strength of the original signal. The replication provides local evidence that the component set transfers. A complete circuit remains unverified.

1. Reproducing value: first write "replacement and valid" as a problem that can fail

Indirect object recognition, referred to as IOI, requires the model to predict the person receiving the item at the end of the sentence. For example, two people enter the store, and one of them gives the ring to the other; candidates include the recipient IO and the giver S. What is observed here is the next tokens, the tokens is the discrete unit that the tokenizer is sent into the model, and the free continuation is not performed.

Original paper by Wang et al. The IOI behavior of GPT-2 was investigated with causality intervention, and explained by test of loyalty, integrity, and minimumness. What the paper reports Support the value of this task, but do not endorse the three-head experiment. This article also does not transplant the entire list of components in the paper.

The ranking of this collection continues the development stage: L8H10, L8H6, and L9H9 with zero starting numbers, that is, the layer number and the number one. They have been selected on sixteen development materials, and have been used in the previous experiment; the number one, joint operation and end-end positions are all frozen during migration. The presumptions are that the average ablation decline and recovery increments of the three migration groups are positive. The same intensity is a stronger proposition, and this time there is no equivalent limit for it.

2. Experimental design: four groups of materials answer two dimensions of change

Four sets of migration experiments for freezing components

The original name is taken in the first eight of the official list, and the name is given to the eight next; the original template is index zero, one, and the new template is two and three. These reserved materials have been registered at the baseline stage of the behavior, and the first reasoning is done after this freezing plan. "New name" means that the project component selection is not used for this project, and it cannot claim that the pretrained model has not seen these names.

Each group consists of eight loop adjacent name pairs, two character directions, two first mention sequences and two templates, a total of 64 items. The location and items are fixed for shops and rings. All names are verified as single tokens with leading spaces; four groups of combined indexes are one-to-one, which is convenient for item-by-item comparison. The pairing here is that the design position is the same, which does not mean that different people's names have completely consistent language attributes. Two groups of words frequency or gender associations that do not match names, nor do they measure their appearance in pretrained corpus. Therefore, the "name change effect" can only refer to the difference between the two frozen lists, and cannot be further attributed to a certain name attribute.

The original group has reused the previous 64 behavioral materials, of which 16 have participated in the selection of components, so it is a development reference. The remaining 192 articles are the reserved migration materials. The loop names overlap each other, and the same template is reused, and these rows cannot be used as natural sentences extracted independently. This time, only the descriptive difference is given, and the seemingly accurate line-by-service self-help method Confidence interval is not drawn.

The new template contains "Working" and "Want to go somewhere", while changing the expression and length of subsequent verbs. They maintain an open IOI structure, but do not replace a strict minimum control of just one grammatical component. Only change the name, only the template, and the three groups at the same time, you can locate the source of change separately; you still can't get the causal effect of the pure syntax.

3. Core formula and intervention: the recovery ratio must be seen with the denominator

Define the candidate score difference first:

D(x)=ℓIO(x)−ℓS(x).D(x)=\ell_{IO}(x)-\ell_S(x).

xx is an input prefix without answers, ℓ\ell It is the un-normalized output score at the last position of the model, that is, logit. D>0D>0 It only means that IO is more preferred between two names, and there is no guarantee that IO will be ranked first in the entire vocabulary. Therefore, the first-choice answer and IO probability of the full vocabulary are also recorded.

The main indicators are the joint clearance drop volume:

Ag=1ng∑i∈g[D(xi)−D0(xi)].A_g=\frac{1}{n_g}\sum_{i\in g}\left[D(x_i)-D_0(x_i)\right].

gg represent an experimental group, ng=64n_g=64, D0D_0 It is the difference of the score difference after the output of the three selected heads at the end of the sentence is activated and cleared at the same time. The model weights are not deleted; all subsequent calculations are done as usual. The larger the value, the greater the impact of this clearing operation on the score of the group, which is not equivalent to explaining the coverage rate.

The recovery experiment replaces the last subject that appears with the original IO, and gets the counterfactual prefix x~\tilde x; Still scored by the original IO and the original S. Then activate the end of the sentence corresponding to the three heads in the clean input and put it into the counterfactual run to get DpD_p. Define:

Gg=Dp−D(x~)‾,Rg=GgD(x)−D(x~)‾.G_g=\overline{D_p-D(\tilde x)},\qquad R_g=\frac{G_g}{\overline{D(x)-D(\tilde x)}}.

The horizontal line is the mean within the group; GgG_g is the original recovery increment, RgR_g It is the ratio of the mean, not the average of the row-by-row ratio. If the absolute value of the average denominator is not more than one in a million, the script will return a null value without hard division. Negative effects are also not cropped with a value of more than one. Zhang and Nanda The influence of indicators and destruction methods on the positioning results is studied, so this paper retains the original increment and denominator at the same time.

The migration is the component location and operating rules, and each new sentence still needs its own clean activation. This does not prove that a fixed vector can be universal across sentences, nor does it call the simultaneous replacement of the three heads as a side-by-side path repair.

4. Guide to the official implementation and minimum operation

Read the official first ioi_dataset.py The name, template and first mention change, see again ioi_utils.py Candidate difference; finally catches the attention output projection of Transformers c_proj. The source code map gives a pinned commit, specific line number and license to prevent readers from facing the changing main branch.

Activation before projection can be interpreted as a shape [1,T,12,64][1,T,12,64]: Batch 1, length TT, twelve heads, sixty-four dimensions each. The official actual incoming projection is after splicing [1,T,768][1,T,768], the local hook only changes the three slices at the last position. When the length of the template changes, the end of the sentence is positioned by the actual prefix length, and the old integer position cannot be used.

The use of the official complete language model is forward, and the output shape is [1,T,50257][1,T,50257], no longer read out with a separate last position matrix multiplication. Each row is additionally used in increments after projection to achieve cross-check, and the two intervention expressions are consistent within the floating point tolerance. TransformerLens is not installed or running, and there is no mixing weight of the different frameworks mixed in.

The model and tokenizer are fixed in 607a30d783dfa663caf39e06633721c8d4cfcd7e, the template is fixed in ea15315dd24481e9e2ac5c3ef335d82907a1dc34. The eight resource files are checked one by one; the multiplexing has a cache, and re-downloading is not regarded as an experimental progress. CPU, single precision, four threads, batch one, turn off training, generation and truncation.

5. This result: the same direction does not mean the same explanatory power

The following tables are all from the saved real model logs; the original group is the development reference, and the other groups are the frozen migration measurements. The unit of descent and increment in the table is a logit difference.

Conditions, sixty-four in each group clean mean Clean preferred right Clearance drop recovery increment Recovery ratio
Original name, original template 4.446 64/64 2.387 5.387 60.59%
new name, original template 2.947 64/64 1.720 3.834 65.04%
Original name, new template 3.877 56/64 1.947 4.207 54.24%
new name, new template 2.479 53/64 1.360 2.997 60.43%

On all 256 examples, zeroing the heads lowered the score and patching raised it, supporting the prespecified directional hypothesis on these materials. But when both names and templates changed, the drop under zeroing was 1.027 smaller than in the original group, and the gain under patching was 2.391 smaller. The recovery fractions were close because the mean clean-to-counterfactual gap also shrank, from 8.892 to 4.959. Quoting only the roughly 60% recovery figure could give a misleading impression that the strength of the mechanism stayed constant.

After recovery, the first choice and correct number of the full vocabulary list are 46, 47, 23, and 26. Especially when the original name is equipped with a new template, there are 40 candidates who have a positive difference between the two candidates after the restoration, but only 23 are the real preferred IO. The local improvement between the two names is still far from the recovery of the answer to the complete vocabulary.

This official operation is about 62 seconds, the peak memory of the process is about 1.466 GB, and the detailed byte value and environment are stored in the operation record; the video memory or energy consumption is not separately measured. A total of 1,024 scoring conditions were recorded, and another 1,280 times were achieved to control the forward direction. The length of the template prefix is 15, 19, 18, and 20. The difference in calculation amount is truthfully reserved, and there is no short new template to forge and other costs.

6. Failure investigation: from specific counter-examples to measurement definitions

In the sample 136, Jennifer intends to give the ring to Ashley; the difference between the model is positive for Ashley compared to Jennifer, but the first vocabulary is the pronoun "her". This is the fact that the name of this protocol fails to be precisely preferred, and it should not be interpreted as the model does not understand the recipient at all. Other failures also appear "the", indicating that the open vocabulary competition is not the same task as the two candidate judgments.

Two hundred and forty-two samples of double migration are more direct: in the prefix of David and James, the candidate difference is negative 0.440. This article does not exclude it and then reports the recovery rate, nor does it return to the reserved set to start again. A single failure can only reveal the boundary, and cannot prove that a certain attribute is the cause.

We first checked tokenization where the answer joins the prompt, then checked the subject count, the unique replacement position, the final token index, and the scope of each hook. The no-op hook and both self-patching controls gave zero error across the vocabulary. Two implementations of the projection differed by at most about one ten-thousandth. An independent script recomputed all the arithmetic and rejected seven kinds of deliberately corrupted data. A separate implementation replayed every scoring condition and agreed on every top token. Small floating-point differences in candidate scores and probabilities precluded a claim of bitwise agreement.

7. Research questions: what kind of counter-evidence is needed next time

Author’s interpretation Yes: The local effect of this set of header bearings can maintain the direction across this material, but the recovery of strength and complete answer still depends on the input distribution. It has not excluded the external disturbance caused by clearing; the previous article has shown the importance of replacing baselines, and did not re-run all the mean and resampling conditions today.

One verifiable follow-up question is: on the materials that are separately registered, use a template with a more matching length and a pre-fixed replacement baseline. Is the original recovery increment still reduced with the clean score? Competitive interpretations include the weakening of the signal of the complete model itself, and the relative effect of the selected component on new expression. It is necessary to add new controlled evidence to distinguish, and the conclusion of the current four mean substitution mechanisms cannot be obtained.

These reserved names and templates are now used and cannot be re-named as unconfirmed sets in subsequent articles. The next step to assess the remaining errors between the reserved component and the full model must also acknowledge the failures seen today. Integrity requires stronger exclusion evidence, and forward migration is not the end point.

Code and instructions for reproducing the results

This article has been published: Source tree at the pinned revision、Running Instructions README、The official implementation corresponding table SOURCE_MAP. Forty-eight distribution files have been anonymously downloaded and checked one by one locally; the public copy has also passed the real test run. After installing the lock dependency, the minimum command is:

python run.py --cache ./cache --out smoke-new --smoke

It runs the four conditions and control checks of the original group of a real sample, and produces a row-by-line record, a summary, a stage state and a resource log. Get rid of --smoke And use the new output directory to run all 256 articles; python audit.py results The results of the archives can be audited offline. Dependency, downloader, independent recalculation, fixed upstream mapping and license are fully provided, and the large file of the model is downloaded by a fixed version and is not distributed with the source code.

Summary

The most valuable thing about the migration experiment is to make an explanation face the material it does not participate in the selection. This evidence supports the migration of the three fixed heads in the direction of the local intervention, and at the same time, the intensity is reduced, the vocabulary competition and recovery are insufficient. Keeping these phenomena side by side is closer to the testable scientific research problem that only shows the stable percentage.

References

Retrieve and fixed source code access date: 2026-10-20.

  1. Kevin Wang、Alexandre Variengien、Arthur Conmy、Buck Shlegeris、Jacob Steinhardt: Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, 2022, read version v1.
  2. Fred Zhang、Neel Nanda: Towards Best Practices of Activation Patching in Language Models: Metrics and Methods,ICLR 2024,v2.
  3. Author's official repository: Fixed template and name implementation、Candidate Score Implementation.
  4. Hugging Face Transformers: Fixed GPT-2 implementation,4.49.0.
  5. OpenAI / Hugging Face: GPT-2 Fixed Model Card.