Readers: postgraduate, early-career researchers and research engineers; search and experiment date: 2026-09-17.

Copy intermediate state scientific research device concept cover

Abstract

If patching a layer corrects a previously wrong answer, have we located where the model stores that knowledge? We start with the three forward passes used in activation patching, examine the official ROME and TransformerLens code, and train a 402-parameter network to test patch direction, whole-layer controls, joint effects, and coordinate rotations. The aim is to reproduce an auditable intervention protocol and establish what its results can support. We ran no language model and make no claim to have replicated the original paper’s results with the toy network.

Contents

  1. Reproducing Value: From Correlation Graph to Intervention Issues
  2. Core ideas and formulas
  3. A guide to the official implementation
  4. A minimal experiment and its results
  5. Evaluation protocol and independent audit
  6. Failure Troubleshooting and Interpretation Borders
  7. Questions for further research
  8. Summary and References

Reproducing Value: From Correlation Graph to Intervention Issues

Mechanism interpretability attempts to illustrate how the calculation is done within the neural network. The attention is very large and the neurons are very active, which only shows the observed values; active replacement will actively change the intermediate state, and then observe the output. Activation is a vector generated by a forward run, and the parameter is the weight obtained by training. Replacing the former is not equal to editing the latter.

Paper report: Meng et al. used the work of CAUSAL TRACING to study the facts in 2022 in NeurIPS, and then proposed the weight editing method ROME. This article checks the fifth edition of the revised version in January 2023, focusing on reading the tracking agreement, and will not repeat the unrun edited results. title, author and version

Another starting point is Zhang and Nanda's ICLR 2024 paper: disturbance methods, scoring indicators and replacement ranges may change positioning conclusions. Therefore, the question in this article is not "Which grid is the most popular", but "what the input pair and intervention, what has this grid changed". Paper version page

Core ideas and formulas

Independent baseline, cache replacement and output comparison process

First run the original input clean, save the activation; then run the control input corrupted, save the baseline; finally rerun the corrupted, only put the clean activation at the specified position, and continue the calculation. The orange branches in the figure are separated before replacement, representing the independently completed unreplaced baseline; grid is a conceptual diagram, not an experimental heat map.

h~S(xb)=hS(xc),h~Sˉ(xb)=hSˉ(xb).\widetilde h_S(x_b)=h_S(x_c),\qquad \widetilde h_{\bar S}(x_b)=h_{\bar S}(x_b).

xc,xbx_c,x_b are the original and control inputs, respectively. SS It is the coordinate set of this replacement, and the rest of the coordinates keep the control value. In language models, residual activation is often [B,T,d][B,T,d]: batch size, tokens and hidden width; replace a position usually replaces the entire dd dimension vector. There is no tokens axis in this small network, and the hidden layer is [B,16][B,16], can replace one or more neurons.

m(x)=zy(x)−zy′(x),Δm=mp−mb,R=mp−mbmc−mb.m(x)=z_y(x)-z_{y'}(x),\qquad \Delta m=m_p-m_b,\qquad R=\frac{m_p-m_b}{m_c-m_b}.

zz is the logit before softmax, that is, the unnormal class score; yy is the clean answer, y′y' It is a competitive answer, three subscripts indicate patched, corrupted, clean. R=0R=0 Indicates back to the control score, R=1R=1 Indicates back to the original fraction; it is not a probability, it can be crossed, and cannot be cut into a beautiful percentage. When the denominator is close to zero, the original difference should be retained and the restoration rate should be stopped. This article also saves the target probability change, to avoid confusing the movement of the score with the flip of the answer. Definitions of Indicators and Disturbances

The direction also determines the problem: clean→corrupted check whether it can be restored, and reverse replacement check whether the original behavior is damaged. The conditions required for recovery and damage may vary, and redundant pathways will make them asymmetrical. Methods of Heimersheim and Nanda

A guide to the official implementation

The first stop is ROME causal_trace.py. Along calculate_hidden_flow Enter trace_with_patch: Batch zero samples are kept clean, and the rest of the samples accept embedded noise; specify the layer and position to copy the zero number activation. The output takes the target word probability of the last position, and averages the disturbed sample. The noise range, target answer and batch index should be checked, and this set of probability protocols cannot be directly regarded as the logit difference protocol of this paper.

The second stop is nethook.py. Hook is a callback for forward execution, and can read or replace the module output. Check whether the return value is a tuple, whether the cache is copied, and whether the hook is removed after leaving the context; otherwise, the intervention left by a scan may contaminate the next baseline.

The third stop is TransformerLens patching.py. generic_activation_patch Re-run Corrupted index by index; layer_pos_patch_setter Assign a value to the specified location. resid_pre、mlp_out The meaning and axis order of attention pattern are different and cannot be interchanged. The residual position scan returns the layer × position result, but this does not automatically prove a complete calculation loop.

The above links are all the official main branches that are actually opened, and the submissions are not locked, and the dynamic version compatibility is subject to manual verification. The static reading is completed this time, and the official environment is not executed. The attached real model script only supports local GPT-2, and replaces the entire block output, and cannot be called a separate feedforward module.

A minimal experiment and its results

The task is deliberately simple: enter two values a,b∈{−1,1}a,b\in\{-1,1\}, selector qq Decide which one to output. q=−1q=-1 Selected aa, q=1q=1 Selected bb; the input has two independent Gaussian noise characteristics, the standard deviation is 0.30.3, tags ignore them. Flip the selected value will change the answer, and flip the unselected value should not change the answer.

The model is a multi-layer perceptron, that is, MLP: the five-dimensional input passes through two sixteen-dimensional tanh hidden layers, and then gets the scores of the two categories. The data seeds are 760, and the eight core modes are matched with different noises, generating 512 pieces of training, 128 verifications, and 64 diagnosis. The three sets do not have duplicate floating-point samples, but share all core patterns, so only the behavior within the task is checked and cannot claim combinatorial generalization.

Tanh is a hyperbolical tangent function that pushes the real number to the negative one and one, and provides nonlinearity here; Adam is an optimizer that adjusts the update amplitude according to the first and second-order historical statistics of the gradient. These options are just for small networks to learn to choose tasks, and do not simulate the attention, tokenization or factual knowledge of the language model.

Model seeds are 76, 77, 78; full batch Adam, learning rate 0.010.01, fixed a thousand steps. The final weight is saved, and the verification curve is not used for early stops. Four neurons to be replaced in each layer, averaged by the Validation Set Δm\Delta m Select, the diagnostic set does not participate in the selection. Execute in the article directory:

python3 code/patching_numpy.py
python3 code/audit_results.py

Dependency and environment description see code/README.md. The actual environment of this time is Python 3.12.14, NumPy 2.3.5, CPU double precision; the input is [64,5], the two layers of activation are [64,16], the output is [64,2].

What we tested: the Clean Accuracy of the three seeds is all one; after the selected value is flipped, the Accuracy of the original answer is all zero. The following table is the three seed averages on the same diagnostic set, and the recovery first calculates and then averages one by one; the standard deviation indicates the initialization difference, not Confidence interval.

Intervene Relative to the original answer Accuracy Recovery Rate Mean ± Seed Standard Deviation
Do not replace 0.0000 0.0000 ± 0.0000
Restore all coordinates of the first layer 1.0000 1.0000 ± 0.0000
Restore the first layer Validation Set selected four-dimensional 0.2865 0.3836 ± 0.0467
Restore the selected four-dimensional value of the second layer Validation Set 0.1615 0.4113 ± 0.0368

The recovery rate of the second layer is slightly higher, but the answer Accuracy is lower, indicating that continuous scores and discrete decisions are not equivalent. The success of the entire layer recovery is interface control: the subsequent calculation of this unhop-connected network only depends on the restored entire state. Success is originally expected, and cannot be regarded as a discovery of precise positioning knowledge.

Do coordinate control again: insert orthogonal transform only after the second hidden layer h′=hQh'=hQ, restore the pre-calculation Q⊤Q^\top. QQ For [16,16][16,16], satisfying QQ⊤=IQQ^\top=I. The output remains unchanged without intervention, but replaces a new coordinate equal to changing one direction of the original space. Maximum single coordinate average of seed 76 Δm\Delta m From 2.62352.6235 Turn into 5.97065.9706, the sum of all coordinate effects is still about 23.791223.7912.

Author’s interpretation: The "concentration" here depends on the coordinates that are allowed to be replaced. The experiment did not show that the original neurons were meaningless, and did not find new language circuits; it showed that the single coordinate ranking is not the same level as the entire function. Rotation occurs after Tanh, and cannot be generalized as any rotation of the non-linear layer is equivalent.

Evaluation protocol and independent audit

First confirm that the model will do the original task, the disturbance will indeed change the target behavior, and then talk about recovery. When flipping the unselected value, the answer to the three seeds, Accuracy is still one, but the logit difference will move; this article uniformly records the recovery rate of this condition as empty, and only observes the original score. This is more clear than forcing each disturbance to force normalization.

Each seed keeps 8,320 records, including no replacement, self-replication, whole layer, single point, four-point unity, reverse, zero and mismatch donors. The mismatching adopts batch cyclic shift. Since the data is grouped by core mode, many donors still share tags; it is only a weak diagnosis, and it cannot be called a strict random negative control. Zeroing occurs on the corrupted base, and the clean base is used in reverse replacement, and the two answer different questions.

Sixty-four diagnostic inputs do not represent sixty-four independent inference rules: each core model has eight noise variants. In the future, if the uncertainty of cross-problem is to be estimated, it should be sampled in groups according to the core mode or semantic template, and at the same time, the types of tasks should be added, and a large number of similarities cannot be rewritten as the same as much independent evidence.

24,960 records were recalculated with scalar cycles for independent audit, and the maximum absolute error was 8.88×10−158.88\times10^{-15}, and recheck the Validation Set point and summary metrics. The finite difference check covers the selected elements of the six sets of parameters, and the maximum error is 7.85×10−127.85\times10^{-12}, not every parameter has a numerical gradient check. All data, weights, one case logits, training curves and logs are saved in code/results/.

The sum of the four-point combined effect of the first layer is not the same as the sum of the four single-point effects: the maximum difference between seed 76 is about 3.36173.3617. Only linearly read out after the second layer, its logit difference effect can be added, and the error is in floating-point accuracy. This structure is compared with the description, the combination replacement must actually run, and the heatmap grid cannot be added directly.

Failure Troubleshooting and Interpretation Borders

When the result is completely unchanged, first check whether the hook hit, whether the position is aligned, and whether the donor is from the correct sample; when the result is abnormally perfect, check whether the entire status, answer position or subsequent logits are replaced. The cache should be replicated independently, the weights are frozen, and each scan restarts from the same baseline. No substitution and self-copying must be restored to the baseline, and the entire layer recovery must meet the expectations of the network structure.

The real text should also check tokenization: the same number of characters does not guarantee that the tokens are the same, and the tokens have the same number do not guarantee that the semantic position corresponds. Multiple tokens answers need to set a separate sequence score, not just the first tokens. local_gpt2_patch.py It will print the location, check the tokens answer and remove the hook; the current lack of dependencies and local weights, only through syntax checks, the model operation and example prompts are manually verified.

Recovery is evidence under specific input pairs, replacement sets and indicators. It does not automatically prove that the location is the only storage point, nor does it tell us how to write upstream and how to read downstream. The donor activation produced by the natural input is spliced with the remaining activations of another input, and still may form an unnatural internal state. Subsequently, the input coverage must be expanded and the verification of the combination, reverse and path level must be expanded.

Questions for further research

First, fixed model and input pair, only change the disturbance mode, is the positioning and sorting stable? At the same time, the baseline damage strength should be reported, and the recovery space caused by stronger damage should be avoided as a more important mechanism.

Second, under the same replacement dimension budget, who is more stable between the original coordinates and the learned subspace? The subspace can only be learned on training or validation data, and then compared on independent prompts, and can't pick the direction after reading the test heatmap.

Third, can the joint recovery of the candidate sets be maintained across rewriting? Scans first put forward hypotheses, and then freeze the sets for confirmatory experiments; if only the sentences found on it are valid, the failure cases should be retained, rather than continuously expanding the collection until the recovery is full.

Summary

Reproducing an activation-patching result takes more than a heatmap. It requires the input pairs, patch coordinates, baselines, metrics, and controls. Here we ran and audited those interventions on a small network. Their behavior in a real language model remains untested. The next step is to transfer the checks to that setting before interpreting particular layers and circuits.

References

The following are all first-hand sources, and the search date is 2026-09-17, which has been actually opened.

  1. Kevin Meng、David Bau、Alex Andonian、Yonatan Belinkov. Locating and Editing Factual Associations in GPT. NeurIPS 2022; check arXiv v5, 2023-01-13.
  2. Fred Zhang、Neel Nanda. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. ICLR 2024;v2,2024-01-17; Full text.
  3. Stefan Heimersheim、Neel Nanda. How to use and interpret activation patching. 2024,v1,2024-04-23; Full text.
  4. ROME authors and contributors: causal_trace.py、nethook.py, retrieve the main branch of the day.
  5. TransformerLens maintainers: patching.py, retrieve the main branch of the day.