
Date: 2026-08-12
Suitable for readers: graduate students, introductory mechanisms for mechanics, AI readers with engineering backgrounds
Search date: 2026-08-12
Abstract
Interpretability, especially mechanistic interpretability, has grown beyond plotting heatmaps of model outputs. Researchers now study internal representations through causal interventions and reproducible experiments. When interpreting a neuron, ask what units represent information in the model, how stable those units are, and whether they actually contribute to the output. If you patch, ablate, or steer one of them, does the model’s behavior change as the proposed mechanism predicts?
This article gives a intensive reading route suitable for scientific research training: first read representation and probing, understand "whether the information is readable", then read circuits, understand "how the components are combined into algorithms"; then read the sparse features / SAE, understand "why single neurons are not enough, why features fail"; finally read Causal Intervention, Circuit Tracing, Jacobian Lens and Natural Language Autoencoders, train themselves to advance interpretation from relevance evidence to mechanism evidence.
Why is this topic important
The stronger the model, the easier it is to be masked by Prompt, evaluation set, and sampling settings. If a model answers the math problem correctly, it does not mean that it is really inferable according to the traceable intermediate step; a model rejects a harmful request, which does not mean that there is no first identification, planning or suppression of something internally; an SAE feature It seems to have clear semantics, which does not mean that it is the only causal variable actually used by the model.
Progress in 2024 to 2026 has made this direction more specific. OpenAI released a hyperscale Sparse AutoEncoder study characterization of GPT-4; Anthropic advances from the Monosemantic features of Claude 3 Sonnet 2025 Attribution Graph / Circuit Tracing, and then Jacobian Lens, J-space and Natural Language in 2026 Autoencoders; Gemma Scope and Gemma Scope 2 of Google DeepMind push SAE, Transcoder and interactive analytics resources on the open model to the community. At the same time, SAEBench, feature absorption, feature explanation falsesification, etc. in NeurIPS 2025 also remind us: the interpretation tool itself also needs to be Benchmark, counterexamples and falsified evaluations.
Therefore, the reading of Interpretability papers is more demanding than that of ordinary method papers. Instead of "proposing an indicator and running several models", it is trying to answer "what is going on inside the model". Such conclusions can easily stay in beautiful visualization without Intervention, Negative Control, cross-Prompt reproduction and tool failure analysis.
Core reading method
When reading Interpretability papers, it is recommended to split each article into five questions.
First, what is the explanation object? Is Token Embedding, Residual Stream, Attention Head, MLP Neuron, SAE Latent, Transcoder feature, or Jacobian Lens read it verbaliizable representation? Different objects have different meanings. A PROBE reads out the syntax information, which only means that the information can be extracted linearly; an ablation changes the output, and then starts to approach "this part of the calculation is used by the model".
Second, what is the purpose of explaining the goal? Is the paper explaining a specific behavior, a specific data distribution, a certain type of ability, a certain safety-related tendency, or a general representation space of the model? Explanation Induction Head is not the same level of task and explanation Claude's Evaluation Awareness, and cannot use the same standard of evidence.
Third, is evidence relativity, predictability, or causality? The highest sample of visual activation is correlation; linear probe is readable; activation patching, path patching, ablation, feature steering, model editing It is the stronger evidence of causality. The best papers usually string together these evidences, rather than just showing a beautiful case.
Fourth, is there a self-checking of the interpretation tool? SAE To check Reconstruction Loss, Sparsity, Feature Consistency, Dead Features, Feature Splitting, Absorption and interpretation falsity; Circuit tracing to check the replacement model approximate error; Probe controls Probe capacity; Jacobian Lens Remember that it is an average linearization tool, not a complete read model thinking.
Fifth, where is the reproduction boundary? Mechanism interpretability is very suitable for small model training, but it is easy to get out of control in engineering. Graduate students are better suited for small models from GPT-2 Small, Pythia, Gemma, public SAE, TransformerLens, NNsight, Neuronpedia and Circuit-Tracer Tools like this start, first reproduce a small behavior, and then expand to a stable problem.

A reading guide to the key papers
1. Representation and Probing: first learn to distinguish between "readable" and "used"
Getting started can start with Probing Literature. Belinkov's review is very suitable for building basic alerts: training a classifier to predict language attributes from hidden states, it is not directly proof that the model uses this attribute in downstream tasks. Probe is too strong to learn tasks by yourself, and Probe is too weak and may miss real information.
There are three actions to master at this stage: layer selection, position selection, and Probe capacity. When reading a paper, don't just look at Probe Accuracy, but look at Control Task, Selectivity, Minimum Description Length, cross-model comparison and intervention validation. A good Probing experiment should be able to tell you "Which layer, which position, and what information can be extracted with complex reader", but it usually can't tell you "whether this information is driving output".
After reading the Probing firmly, it will be clearer to read Logit Lens, Tuned Lens and Jacobian Lens. They are essentially asking whether the middle layer indicates whether it can be mapped to a human-readable vocabulary or semantic space. The important thing about J-space work in 2026 is not just to propose a Lens, but to put "Verbalize representation" with Report, Modulation, Internal Reasoning, Flexible Functional properties such as generalization, selectivity, etc. are connected, and check whether these representations really affect behavior through Intervention.
2. Circuits: From Components to Algorithms
The Transformer Circuits line of work teaches you to analyze a network as a program. The 2021 paper A Mathematical Framework for Transformer Circuits decomposes an attention head into QK and OV circuits and uses path expansion to analyze attention-only Transformers. The 2022 work on induction heads traces some instances of in-context learning to identifiable combinations of heads. Certain heads find a pattern seen earlier in the context and copy the token that followed it.
When reading a Circuits paper, the focus is not to memorize the terminology, but to learn its chain of evidence: first define a behavior, such as repeated token / indirect object identifier / factual Recall; then locate the relevant head, MLP, or feature; then use activation patching, path patching or abration to prove that these components contribute to the output cause and effect; finally check whether the mechanism is cross Prompt, cross-model, or cross-training stages are stable.
Circuit Tracing in 2025 has taken this route further at scale. Instead of just staring at a few attention heads, it uses cross-layer transcoder to construct a more interpretable replacement model, and then expand the calculation on the prompt into an attribute graph. A key limitation should be paid attention to when reading: Replacement Model is an approximation model, which makes the mechanism diagram readable, but also introduces approximate errors. The output difference between the original model and the alternative model must be recorded when reappearing.
3. Sparse features and SAE: from "neuronal semantics" to "feature space"
A single neuron is often polysemantic: the same Neuron may be involved in multiple concepts, and a single concept may also be distributed over multiple Neurons. Anthropic's Dictionary Learning / Monosemanticity, OpenAI's GPT-4 Concept Extraction, DeepMind's Gated SAE and Gemma Scope is developed around a core hypothesis: the model may use sparse features to represent many concepts, and SAE can decompose the Dense activation into more interpretable latent features.
The reading order of this line can be: first read Toy Model of SuperPosition, understand why "features more than dimensions" will lead to overlapping representation; How to find interpretable features in the small model; read Scaling Monosemanticity and Extracting Concepts from GPT-4 to understand the training and limitations of scaled SAE; finally read Gemma Scope/Gemma Scope 2, learn tooltized resources on open models.
But SAE is not an interpretive silver bullet. 2025 SAEBench reminds us that proxy metrics such as reconstruction error and sparsity are not necessarily equivalent to the actual interpretability utility; feature absorption The hierarchy concept of the work description will make the seemingly single feature not activated at this activation, and the quilt feature is absorbed; examples to falsify. When reading an SAE paper, you must read both "Successful Papers" and "Fail/Evaluation Papers".
4. Causal Intervention: Interpretation must be interrupted, replaced and verified
The most important training action of the interpretability of the mechanism is Intervention. ROME's causal tracing is a classic example: first use Corruption + Restoration to find the most critical intermediate state of the Factual Association for the output, and then use the Rank-One Model Editing Modify specific facts. It states that "location" and "editing" can be verified: if you say that a certain MLP middle layer stores the facts, you should be able to change the fact through constraint editing, and keep the specificity and generalization.
In LLM reproduction, the most common Intervention includes four categories: activation patching, put a certain activation run of the clean run into corrupted run; ablation, put a certain Head, feature or direction is zero; Steering, enhances or suppresses the concept in a certain direction; path patching only replaces the path from a component to a downstream component. Negative control is designed for each: random layer, random feature, irrelevant prompt, synonymous rewrite and reverse intervention.
The new work of 2025-2026 uses Intervention on more abstract objects. Circuit tracing can intervention on the feature of the Attribution Graph; the J-space work can be patched on the Verbaliizable Representation Ablation; Natural Language Autoencoders try to reconstruct the Activation through the Natural Language bottleneck to make the internal state readable in the form of text interpretation. They are all worth reading, but they all need to be cautious: the closer the tool to "read out the model thinking", the more serious the tool output, model output and real causal mechanisms are required.
Methods and experiments compared
| method line | Representative work | Main answer | Evidence of the most checked |
|---|---|---|---|
| Probing | probing classifiers, probing as inductive bias | Whether there is a certain type of information in the representation | Probe capacity, control task, cross-layer trend, whether there is an intervention |
| Transformer circuits | Mathematical Framework, Induction Heads | How components are combined into algorithms | Head/MLP positioning, path patching, cross-prompt stability |
| SAE / sparse features | Monosemanticity, GPT-4 SAE, Gemma Scope | How Dense Activation is decomposed into Features | Reconstruction, Sparsity, Feature Consistency, counter-example |
| SAE evaluation | SAEBench, feature absorption, explanation falsification | Explain when the tool expires | Is Proxy Metric effective, close negative, hierarchical concept |
| Causal tracing/editing | ROME, activation patching | Which state causality affects the output | corruption-restoration、specificity、generalization |
| Circuit tracing / CLT | Attribution graphs, circuit-tracer | How the Prompt level calculation diagram expands | Replacement Model error, edge validation, intervention |
| Lens / workspace | Jacobian Lens, J-space | Intermediate representation can Verbalize and participate in reasoning | Lens hypothesis, layer location, patch/ablation, task boundary |
| Natural language bottleneck | Natural Language Autoencoders | Can Activation be translated into a text explanation | Reconstruct quality, explain loyalty, whether excessive humanization |
How to reproduce the results
In the first stage, a small behavior is reproduced. It is recommended to start with the GPT-2 Small or Pythia small model, and use TransformerLens to cache Residual Stream, Attention Pattern and Logits. Tasks can be Repeated Token, Induction Head, IOI, or Factual Recall. The goal is not to do new SOTAs, but to write clearly clean prompt, corrupted prompt, target token, patch location and logit difference.
In the second stage, a SAE feature is reproduced. You can use Neuronpedia, Gemma Scope, or expose the SAE weight, choose a relatively clear feature, and collect Top Activating Examples, Near Misses and Close Negatives. Don't just show positive examples; to record samples that feature should not be activated but should be activated but not activated.
In the third stage, the intervention is reproduced once. Do ablation/steering on a head, MLP Activation, SAE feature, or Residual Direction, compare logits, generate text, and task Accuracy . At least three controls are matched at each Intervention: random same-level position, unrelated feature, and synonymous prompt.
In the fourth stage, write reproduction report. The report includes at least: model name and version, code base commit, data sample, prompt template, random seed, hook point, indicator definition, all failure samples, manual interpretation rules and artificial verification items. In particular, the reproduction report of Interpretability should write "I can't prove anything", otherwise it is easy to misread the visualization as a mechanism.
Common misconceptions
First, take the attention map as an explanation. Attention weights can provide clues, but are not equal to causal contributions. At least match the value path, logit attribute or patching.
Second, use Probe Accuracy as a mechanism. Probe can read the information, which does not mean that the model uses this information to complete the task.
Third, take the SAE feature name as a fact. Many feature interpretations come from Top Activating Examples or are automatically named, and must be checked with Negative Examples, cross-seed consistency, and Intervention.
Fourth, only the successful Prompt is reproduced. Mechanism interpretation is most afraid of cherry-picking. Simultaneous, antisense, irrelevant, boundary, and out-of-distribution should be actively constructed.
Fifth, overly personified. Words such as J-space, Workspace, Thoughts, Natural Language Autoencoders, etc., can easily make readers think that the model has a complete human thinking. When writing paper notes, these terms should be restored to tool definitions and experimental operations, and do not take philosophical explanations as experimental evidence.
Research questions for graduate students
- The Close Negative dataset explained by SAE Features: Constructs a set of counter-examples that "look similar but should not be activated" for a set of public Feature systems, and evaluates the quality of automatic interpretation.
- Small model Circuit reproduction benchmark: make the Induction Head, IOI, Factual Recall, and Refusal Toy Task into a unified Patching Notebook.
- Evidence upgrade from Probe to Intervention: Select a Linguistic or Reasoning property to compare whether the three types of evidence of Probe, Ablation, and Steering are consistent.
- Gemma Scope / Neuronpedia feature stability study: Compare the consistency and interpreted quality of different layers, different SAE architectures, and different SEEDs.
- Jacobian Lens vs Logit Lens reproduction: Compare the readability and causal positioning ability of the two Lens to the intermediate layer content on the open model.
Summary
The reading core of the Interpretability paper can be summarized as one sentence: first find a readable Representation, then prove the effect of causality, and finally recognize the tool boundary. Representation and probing training We should not mistake "readable" for "used"; Circuits trains us to combine components into algorithms; SAE Train us to upgrade from neurons to feature spaces, and at the same time be alert to feature splitting, absorbing, and interpreting hallucinations; causal intervention trains us to use patch, ablation, steering and editing to check mechanism hypotheses.
For graduate students, the best way to get started in this direction is not to directly explain the maximum model, but to reproduce a controllable mechanism on the small model, and write the experimental log of each step clearly. The real valuable explanation is not a beautiful picture, but a chain of evidence that can be interrupted by counterfactuals, reproduced across Prompt, and can explain when it fails.
References
Search date: 2026-08-12. The following will give priority to the papers, official project pages, conference pages and first-hand materials; model version, tool API, open source weight, interactive platform content and benchmark results may change, please check again before reappearing.
- Elhage et al., “A Mathematical Framework for Transformer Circuits”, Transformer Circuits Thread, 2021. https://transformer-circuits.pub/2021/framework/index.html
- Olsson et al., “In-context Learning and Induction Heads”, Transformer Circuits Thread, 2022. https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/
- Belinkov, “Probing Classifiers: Promises, Shortcomings, and Advances”, Computational Linguistics, 2022. https://aclanthology.org/2022.cl-1.7/
- Meng et al., “Locating and Editing Factual Associations in GPT”, NeurIPS 2022 / ROME project page. https://rome.baulab.info/
- Bricken et al., “Towards Monosemanticity: Decomposing Language Models With Dictionary Learning”, Transformer Circuits / Anthropic, 2023. https://www.anthropic.com/research/towards-monosemanticity-decomposing-language-models-with-dictionary-learning
- Templeton et al., “Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet”, Anthropic, 2024. https://www.anthropic.com/research/mapping-mind-language-model
- Wu et al., “Extracting Concepts from GPT-4”, OpenAI, 2024. https://openai.com/index/extracting-concepts-from-gpt-4/
- Rajamanoharan et al., “Improving Dictionary Learning with Gated Sparse Autoencoders”, Google DeepMind / arXiv, 2024. https://deepmind.google/research/publications/88147/
- Lieberum et al., “Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2”, 2024. https://deepmind.google/models/gemma/gemma-scope/
- Google DeepMind, “Gemma Scope 2: helping the AI safety community deepen understanding of complex language model behavior”, 2025. https://deepmind.google/blog/gemma-scope-2-helping-the-ai-safety-community-deepen-understanding-of-complex-language-model-behavior/
- Ameisen et al., “Circuit Tracing: Revealing Computational Graphs in Language Models”, Transformer Circuits Thread, 2025. https://transformer-circuits.pub/2025/attribution-graphs/methods.html
- Marks et al., “Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models”, ICLR 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/3ba4d47a83e498c2b1a0868cba20f6de-Abstract-Conference.html
- Karvonen et al., “SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability”, arXiv, 2025. https://arxiv.org/abs/2503.09532
- Chanin et al., “A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders”, NeurIPS 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/764ff7477b8e24dbe01531f6791e8bdf-Abstract-Conference.html
- Ma et al., “Revising and Falsifying Sparse Autoencoder Feature Explanations”, NeurIPS 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/5b6ce2529a79862a2d8ac57ac0e01d90-Abstract-Conference.html
- Gurnee et al., “Verbalizable Representations Form a Global Workspace in Language Models”, Transformer Circuits Thread, 2026. https://transformer-circuits.pub/2026/workspace/
- Anthropic, “A global workspace in language models”, 2026. https://www.anthropic.com/research/global-workspace
- Lindsey et al., “Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations”, Transformer Circuits Thread, 2026. https://transformer-circuits.pub/2026/nla/index.html
- TransformerLens official repository. https://github.com/TransformerLensOrg/TransformerLens
- NNsight official documentation. https://nnsight.net/index.html
- Neuronpedia official platform and documentation. https://www.neuronpedia.org/
- Anthropic circuit-tracer package. https://pypi.org/project/circuit-tracer/