Cover illustration

  • Date: 2026-10-16
  • For graduate students, early-career researchers and research engineers
  • Actual Scope: Real GPT-2 Small; Expose 64 English samples constructed by IOI templates; only behavioral evaluation.

Summary: Determine the behavior to be explained first

In the previous unit, the score was prompted, the score and the selection process were changed, and this article began to ask how a specific behavior was generated within the model. The turning point is not to change a heat map of attention, but to establish a research object that is clear enough and can repeat the measurement: when the sentence says that two people go to the store together, and one of them gives the thing to the other, whose name is more inclined to make up for the model?

The correct name ranked first across the entire vocabulary on all 64 examples. The mean logit difference was 4.446, and the mean probability of the correct name was 0.600. This establishes a behavior worth investigating on these examples. It falls short of replicating the full paper or demonstrating that the model uses the grammatical rules we might ascribe to it. On the weakest example, the correct name had a probability of only 0.172, a distinction that accuracy alone conceals.

Reproduce value: turn an answer into a verifiable object

Indirect object recognition, referred to as IOI, is the name of the person who receives the item at the end of the model to complete the sentence. For example, this time entry said that Christopher and Michael went to the store, and then Christopher handed the ring to... The goal was Michael, and the interference answer was Christopher. The "correct" here comes from the role convention of the controlled task, rather than the exclusive judgment of all possible continuation of the natural language.

In a 2022 preprint, Wang, Variengien, Conmy, Shlegeris, and Steinhardt studied this behavior in GPT-2 small. Using interventions, they proposed an explanation involving 26 attention heads. They went beyond visualizing the heads in heatmaps and tested the explanation for faithfulness, completeness, and minimality. Those interventions belong to the original study. We did not run them here. Original paper

Our reproduction goals are narrower: fixed weights, names and templates, confirm the signal direction, and leave each input, position, original score and aggregation relationship. It provides a reference point for subsequent activation replacements. Some clean inputs have not yet produced a stable signal, and any "recovered percentage" denominator may be meaningless.

Core Ideas and Formulas: Compare two names, and also look at the entire vocabulary

Logit is the real number score before softmax is converted into probability, not the probability itself. antiward ii The main indicators are:

Δi=zi,IO−zi,S,Δˉ=1n∑i=1nΔi\Delta_i=z_{i,\mathrm{IO}}-z_{i,\mathrm{S}},\qquad \bar\Delta=\frac{1}{n}\sum_{i=1}^{n}\Delta_i

Among n=64n=64, IO refers to the person who should receive the item, and S refers to the subject of the second half of the sentence; zz It is the vocabulary score of the last input position before the answer is generated. The difference is positive, which only means that the correct name is more preferred than the specified interference name. The average difference is the main indicator pre-specified in this article, to avoid seeing the most favorable statistic after the temporary use of the whole pair.

qi=ezi,IOezi,IO+ezi,S=σ(Δi),pi=ezi,IO∑v∈Vezi,vq_i=\frac{e^{z_{i,\mathrm{IO}}}}{e^{z_{i,\mathrm{IO}}}+e^{z_{i,\mathrm{S}}}}=\sigma(\Delta_i),\qquad p_i=\frac{e^{z_{i,\mathrm{IO}}}}{\sum_{v\in V}e^{z_{i,v}}}

qiq_i is a conditional probability restricted within two candidates, σ\sigma is a sigmoid function; pip_i is the correct name in the full vocabulary VV probabilities in. The two have different denominators and cannot be called "confidence" to each other. This time average qiq_i It is 0.969, but it cannot be claimed that the model gives the correct answer an absolute probability of 96.9%.

The actual output tensor of GPT-2 is [1,L,50257][1,L,50257]: The batch size is one, LL For Input Length, this time is fifteen or nineteen, and the last dimension is a vocabulary. The final position hidden vector is [768][768]. The hidden state after the final normalization is mapped into a vocabulary through the output layer of the language model. Location checking and model calls are equally important, and a misreading grid becomes an experiment that predicts another word.

A guide to the official implementation

First read the author's repository's name table and BABA_TEMPLATES, and then look at the ABBA template obtained after the initial two-person name exchange. A represents the target person, and B represents the recurring subject; this string includes the last answer that has not been entered. Read after gen_prompt_uniform Symmetrical exchange, understand why the answer must also be updated when changing roles. fixed generator

The second stop is ioi_utils.logit_diff: It reads IO and S scores from the end position of each sample and subtracts. The third station is the GPT-2 output layer of Transformers. The official implementation of version 4.49.0 has been checked byte for byte by byte with fixed submissions; there is no complete set of old Easy-Transformer installation, and no claim to verify TransformerLens Equivalence after weight transformation. The source map clearly distinguishes template adaptation, official model call and independent arithmetic audit. Fixed scoring function

Minimum Experiment: Which Variables Freeze

IOI Behavior Baseline Process

Both the model and tokenizer are fixed in 607a30d783dfa663caf39e06633721c8d4cfcd7e, the official template is fixed in ea15315dd24481e9e2ac5c3ef335d82907a1dc34. The downloader checks the SHA-256 of eight files, and the samples are also identified with the full file hash. The data comes from the deterministic expansion of the public template, and there is no fictional dataset revision. Model card at the pinned revision

The experiment takes the first eight names of the official list, and forms eight pairs according to the list and end to end, and each pair is exchanged for the characters; the first two sentence patterns are taken, and each is in two orders, ABBA and BABA. From this, four conditions are obtained, each of sixteen. The location is fixed as a store, and the item is fixed as a ring. Compared with the original author randomly selecting more names, locations and items generators, we have actively contracted the scope of changes, and we cannot compare the high scores here with the large-scale results of the papers.

The configuration is frozen before reasoning: CPU, float32, eager attention, four threads, random seeds 100, batch size 1; use evaluation mode, no training, no free generation, no special symbols at the beginning of sentences, and no truncation. Each candidate must have a leading space and is exactly a token, and the complete "prefix plus answer" code must be equal to the prefix token plus the answer token. The target name appears once in the prefix, the subject appears twice, and all three positions are saved.

The resources are tested with two samples first, and the budget is 1,200 seconds and four GiB peak memory; only 64 pieces are run after passing. The official process wall clock takes 5.418 seconds, of which the recorded forward loop takes 1.878 seconds, and the peak resident memory is 1,604,501,504 bytes. They are measured locally, not throughput publicity; download time and environment installations are not included. There is no activation of all attention heads that are cached, so the cost of subsequent circuit scans cannot be estimated accordingly.

For details, look at the zero, the token number of the target name is 3899, and the interference name is 12803; the zero starting position is four, and two, ten, and the last input position is fourteen. Readers can directly match these positions with the saved token strings one by one. The leading spaces are part of the name code, which cannot be removed first and then asserted with the naked eye the same. There is also no punctuation or terminator after the candidate, so the scoring object in this article is always the name of the next single token.

This design also has a practical constraint for beginners: first write the condition table clearly, and then download the model. Eight pairs of names are not eight non-overlapping groups, and the same person's name will enter two adjacent pairs; symmetrical exchange ensures that each pair of names has been the target and subject. It helps to avoid always letting a common name act as an answer, but it can't make the sample suddenly satisfy statistical independence. Explain the experimental unit clearly, which is more valuable than adding error bars to the table mechanically in the later stage.

Evaluation protocol and this results

Condition number of samples average logit difference Average probability of correct name The first place in the full vocabulary table is correct
Sentence zero, BABA 16 3.808 0.534 16
Sentence Zero, ABBA 16 4.997 0.663 16
Sentence 1, BABA 16 4.294 0.567 16
Sentence 1, ABBA 16 4.685 0.635 16
Entire 64 4.446 0.600 64

All four groups are all pairs, but the score intervals are not the same. Since the same name recurs across multiple conditions, these lines are not sixty-four independent natural samples; this article claims that the general Bootstrap is not generally significant after the line is a normal Bootstrap. The difference between groups in the table is descriptive evidence of this fixed design, and it does not mean that the internal reasons for the position deviation are discovered alone.

Article 8 is the weakest sample of the main indicator: Jennifer gives the Ashley ring, the probability of the target Ashley is 0.172, and the Jennifer is 0.105, the difference is only 0.489. It still ranks first in the full vocabulary, as the probability of other words is also low. This reminds us that continuous metrics should continue to be retained when Accuracy is saturated, not to create temporary additions and deletions of failures.

The independent standard library script re-checks the score subtraction, probability ratio, position, role map and all mean of the scores of the 64 articles; another four types of intentional damage records are used to confirm that the audit will report an error. Scripts, input hashes, commands, and exit codes are archived together. This is automatic recalculation, and the machine inspection is not called a real person review.

If you choose the follow-up intervention material only according to the positive and negative signs or the first place, you will confuse the easy-to-measure example with the strong signal example. We retain all 64 articles as development materials, and also retain the minimum and maximum differences; in the next article, if we need to narrow the scanning range, we should choose according to the pre-established budget rules, and explain whether to choose the scores of this article. The subsets drawn again cannot be called the unseen confirmation set after the full result is visible.

Another easily overlooked detail is the aggregation sequence: first apply sigmoid to each difference and then average, which is usually different from the average difference first and then sigmoid. The main text reports the first one; the independent script also uses the logarithm of the probability ratio to check the original difference to avoid adding the conditional probability of the two names into the absolute probability column. Such deterministic checks can be handed over to the script, but it does not replace the judgment of the legality of the template semantics and future disruption operations.

Failure investigation: check the measurement chain first, then explain the model

The first real test run failed at the output layer: multiply the output matrix of the last position vector alone. Compared with the batch matrix multiplication result, the maximum error is 0.000160, which is slightly higher than the preset absolute threshold. The double-precision control supports floating-point rounding interpretation, and the same shape recalculation is exactly the same. Before the official operation, change the hard check to exactly the same shape and continue to save the single-vector error; the original failure script, log and revision records are kept. There is no name change or a template for this problem.

All inputs for the supplementary audit are independently re-run, and the standard library is used to take the maximum value of more than 50,000 vocabulary scores. However, when summing the absolute probability of double-precision index and recalculation, the maximum difference from the official float32 output is 0.000045; the mean of each group of the text is reserved for three decimal places. This difference has been documented, and the probability is not consistent bit by bit as a passed conclusion.

When resuming the experiment, first check the model and sample hash; secondly, check whether the leading spaces and answers are split into multiple tokens; then check whether the last position is the end of the input, rather than feeding the answer into it. If these passes, look at the actual score. When replacing the model library, adding a sentence head symbol, doing half-precision or changing padding, it should be regarded as a new protocol and re-verified, and the beautiful results in the cache cannot be used.

From behavioral evidence to testable problems

The authors infer that this group of strong, incompletely saturated continuous signals is suitable for the next step of causal intervention. There are at least two competition explanations that need to be distinguished: the model uses the information of duplicate names to exclude the subject, or depends on the combination of name frequency, initial position and fixed sentence pattern. Symmetrical exchange reduces the frequency of some roles, but it cannot rule out these alternative interpretations alone.

In the next article, you should first define a clear semantic clean and destructive pairing, keep the no replacement and complete replacement control, and then ask whether the activation of a certain position can restore the score. Component sorting must only be selected on the development material. The next eight names and templates 2 and 3 of the official list have been reserved this time, and there is no reasoning for them; in the future, if the migration fails in the future, you can't go back and pick a set that can win to be called verification.

Code and instructions for reproducing the results

The full source code has been released and anonymous download verification: Directory at the pinned commit、Running Instructions README、The official implementation corresponds to SOURCE_MAP. All forty-five public documents are consistent with the local archives one by one, and the two independent audits, destructive inspections and real test runs of the open copy are all passed. The original output is kept together with the first test failure record.

After installing the lock dependency and downloading fixed resources, the minimum real entry is:

python run.py --cache ./cache --smoke --out smoke-new
python audit.py --results smoke-new

The first article generates two real model records, aggregation results, stage status and environmental identities; the second one is independently recalculated. The complete command can be replayed by removing the Smoke parameter with the complete command. This time, the official scope has indeed been run; the re-executive is a re-executive material that has been seen and is not a new blind test. The source code does not carry weights or hidden native modules, and the template reconstruction script can restore the same script from the fixed official file.

Summary

The core assets left in this article are a set of behavioral baselines that can answer "what is the input, which position you read, and how you get the score". All correct answers do not end the study; low interval samples, sequence differences and retained materials give the next question a chance to be falsified. Any explanation of the circuit should be made after these reviewable facts.

References

Search and actual opening date: 2026-10-16; see the source code map for the fixed version and license.

  1. Wang, K., Variengien, A., Conmy, A., Shlegeris, B., Steinhardt, J. (2022). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small,v1.
  2. Author team: Official IOI Code and Experimental Entrance.
  3. OpenAI community / Hugging Face: GPT-2 Fixed Model Card.
  4. Hugging Face: Transformers 4.49.0 corresponds to the official implementation of GPT-2.
  5. TransformerLens: Exploratory Analysis Demo, follow-up reading portal; this example is not run in this article, and the dynamic page is not used as this execution version.