
Date:2026-07-15
Who this is for: Graduate students in machine learning, NLP, AI safety, interpretability and deep learning systems; engineering readers who focus on the internal mechanisms of the model and reliable evaluation
search date:2026-07-15
Abstract
Mechanistic Interpretability (mechanism explanation, hereinafter referred to as MI) is not concerned with "whether the natural language explanation given by the model is like a human language", but a harder question: within a trained neural network, which activations, features, attention heads, MLP direction and cross-layer path actually lead to a certain behavior? Does the model behavior change as expected if we remove, replace, or enhance these internal variables?
Five lines of work frame this account of mechanistic interpretability in 2025–2026. Early circuits research decomposed Transformers into identifiable circuits. Activation patching, causal tracing, and automated circuit discovery test proposed causal pathways. Sparse autoencoders (SAEs), transcoders, and dictionary learning aim to disentangle superposed representations into sparse features. Linear representation analysis and steering connect interpretable directions with control over generation. Finally, infrastructure such as SAEBench, AxBench, Neuronpedia, and Gemma Scope exposes unresolved evaluation problems. Across all five, the question is what evidence establishes that a plausible explanation faithfully captures the causal mechanism.
Contents
- Research background: Why MI is more difficult and more important in the era of large models
- core science questions
- A reading guide to the past year’s papers
- Key papers by research question
- Comparing the methods
- How to interpret the experiments and benchmarks
- Reproducibility and data problems
- Limitations and open questions
- Research questions for graduate students
- Summary
- References
Research background: Why MI is more difficult and more important in the era of large models
Explainable AI has long used saliency maps, attention visualization, probes, and feature attribution. Mechanistic interpretability has a goal closer to reverse engineering: identify the structures that actually perform a computation. The Transformer Circuits series is a good starting point. In A Mathematical Framework for Transformer Circuits, researchers from Anthropic and OpenAI describe attention heads, the residual stream, and MLPs as computational components that can be analyzed. Later work on induction heads shows how particular heads support copying and pattern continuation within the context. Toy models of superposition help explain how a network can pack multiple semantic features into the same neurons or directions.
The large model makes this problem more pressing. The stronger the Model, the more likely it is to be delegated to higher risk tasks in medical, code, scientific reasoning and agent scenarios; but the internal representation of the Model is also more distributed, more context-dependent, and more difficult to rely on a single Neuron interpretation. 2023 Towards Monosemanticity introduces Dictionary Learning to a small Transformer, trying to learn the sparse characteristics of "more single meaning" from activation; 2024 Anthropic extends this route on Claude 3 Sonnet, showing features that can be found in Large Models that are related to behaviors such as concepts, abstention, code, bias, and spoofing. By 2025, "Circuit Tracing: Revealing Computational Graphs in Language Models" and Anthropic's On The Biology of A Large "Language Model" further pushes the focus to "from feature to calculation graph": not only to find out a single feature, but also to describe how the feature is formed by the weight and activation of the causal chain through the Model weight and activation.

core science questions
First, what is the "mechanism"? An attention head focuses on certain tokens, a SAE feature activates a certain type of text, and a probe can linearly read entity attributes, these are only evidence of correlation. The stronger goal of MI is that if this internal variable is patch, ablate, clamp or Steer, the output behavior will change according to the mechanism assumption. Mechanisms should support counterfactual interventions.
Second, is it decomposing? The SuperPosition point of view believes that neural networks often press features that exceed the number of dimensions into a limited representation space. A single Neuron does not necessarily correspond to a single concept, and a single concept may also be scattered in multiple directions. SAE's commitment is to decompose the activation into sparse feature codes, but whether the feature learned by SAE is stable and whether the causal variables used by the model still require Benchmark and intervention verification.
Third, how is the circuit combined across the layers? Early Induction Head is very suitable for teaching examples, because the path is relatively short and the function is relatively clear. Real LLM behavior often involves multiple layers of Residual Stream, MLP read and write, attention routing, token position, context conditions and decoding strategy. If an explanation is only positioned to "important at a certain layer and a certain head", it is far from the complete calculation.
Fourth, explain how to evaluate? Natural language interpretation can make people illusion of understanding. The MI review needs to be answered: Can the explanation be able to predict behavior on Unseen Prompts? Can you change your behavior through intervention? Can the model behavior be compressed while retaining the key output? Can you locate a failure case? New Benchmarks like SAEBench and AxBench are trying to turn "features well" into reproducible measurements.
A reading guide to the past year’s papers
1. From feature visualization to Circuit tracing. The Circuit Tracing work in 2025 combines the SAE/Transcoder features, the local substitution model and the Attribution Graph, and the goal is to give the language model the input token from the input token Calculated graph to the output prediction. Its meaning is not just "drawing a better-looking picture", but pushing the feature-level interpretation to the graph-level mechanism assumption. It should be noted that such graphs still rely on the selected alternative models, thresholds, attribution rules and manual interpretation, and cannot be automatically equivalent to the real causal mechanism.
2. SAE enters the Benchmark stage. After 2024, SAE has entered the infrastructure stage from a small number of concepts: Gemma Scope has released an open SAE resource for Gemma 2, and Neuronpedia provides the interface of feature browsing, interpretation, search and intervention experiments; 2025 SAEBench proposed a system evaluation kit, comparing dimensions such as Reconstruction, Sparsity, Feature Interpretability, and Downstream Behavior. Then some papers began to question: Is SAE Benchmark reliable? Can seemingly interpretable features also be learned on models that are initialized or under-trained at random? These doubts pull the direction from "showing beautiful features" back to the experimental design.
3. Steering reconcile with the representation analysis. Linear direction, Activation Steering, Representation Engineering is not the whole of traditional MI, but they provide a strong intervention interface: if a direction really encodes a certain type of attribute, then add and subtract that direction should change the output controlly. Comparing Steering, Control and Interpretability together makes researchers realize that control is not equal to interpretable, and explanations do not necessarily lead to better control.
4. Automation and toolchain become the research object. Tools such as TransformerLens, NNSight, Neuronpedia, SAELens, Circuit Tracer, etc. have lowered the experimental threshold. At the same time, automatic Circuit discovery, LLM generation Neuron/Feature interpretation, and automatic scoring Explanation Quality have also become common. The stronger the tool, the more you need to be alert to the "automatic interpretation pipeline" that packages relevance, visualization and natural language description into over-determined conclusions.
5. Controversy in 2026 is focused on mobility and reliability. As of 2026-07-15, the new papers that can be seen are asking more questions: Can SAE explanations be stable across models, datasets, and cross-layers? Is the Benchmark score consistent with human understanding? Does the automatically discovered feature/circuit really have causal Faithfulness? These questions are more suitable for graduate students than a single Feature Demo.
Key papers by research question
A. Transformer Circuits: decomposition from components to a named circuit
The basic work of Transformer Circuits disassembles the Transformer into information read and write in Residual Stream. Attention head can be seen as writing the value vectors of some tokens to the current position; MLP can be regarded as key-value memory or non-linear feature transformation; different layers overlay information through Residual Stream. The value of this framework is that it gives the researchers a set of languages for "look inside the model's calculations".
Induction Head is one of the most famous success stories of this line. Models are seen in the sequence A ... B ... A when some HEADs will go from the previous A The latter token copy information to help predict the second A Continued writing later. This type of work description MI can discover reusable mechanisms, rather than just post-event visualization. But the Induction Head also reminds us that being able to explain a concise mechanism does not mean that it can explain all complex behaviors.
b. Causal tracing and activation patching: turning interpretation into an intervention assumption
The basic idea of Activation patching is: run clean prompt and corrupted prompt for the same model, replace the Activation of a certain layer and a position from clean run to corrupted run, and observe whether the output is restored. If restored, this Activation carries the task-related information. Methods such as Causal tracing, path patching, attribute patching, and ACDC extend this idea at different granularities: the positioning layer, header, MLP, path or subgraph.
The advantage of this type of method is direct: it asks whether the internal variables lead to changes in behavior. Disadvantages are also very realistic. The patching results depend on the Prompt pair design, metric selection, intervention granularity, and whether to introduce out-of-distribution activation. When a component is patched, it may indicate that it participates in the mechanism, or it may just carry alternative information. Good papers usually do multiple sets of Prompt, negative control, random control, cross-model reproduction, and report failure examples.
C. SAE, Transcoder and Dictionary Learning: from Neuron to Feature
SAE activates a layer x encoded into sparse vectors h, and then from h restructure x. If the training is successful, h A single feature in a single feature may be more singular than the original Neuron: for example, a feature is strongly activated for parentheses, a certain language, code patterns, abstent, or place name context. Transcoder goes one step further and attempts to learn the sparse map of the output from one layer to the latter or a module.
The strong point of this route is scale: SAE can generate a large number of candidate features at one time, which supports search, browsing and manual annotation. Neuronpedia makes this exploration more engineered, and Gemma Scope provides SAE resources on the open model, which is convenient for different teams to reproduce and compare. The risk is that "feature naming" is easy to over-explain. A high-activation example of a feature looks like a concept, which does not mean that it only represents this concept, nor does it mean that the model actually uses it in the task.
D. Circuit tracing: connect the features into a calculation diagram
2025 Anthropic's Circuit Tracing route attempts to answer a more difficult question: if SAE gives a local feature, then these features How to influence each other through model weights and activation, which eventually leads to the output? The paper and technical report use the Attribution Graph to describe the influence path between the features, and analyze the internal calculation of the model on facts, reasoning, abstention, etc. in the case.
Its importance is to advance the MI from the "Characteristics Catalog" to the "Mechanism Graphology". However, it is very cautious to read such results: the edges in the figure come from attribution or local approximation, which is not equal to the complete causal proof; the readability of the graph is affected by the threshold and screening; the human interpretation is still deeply involved. The safest reading is to use Circuit Graph as a verifiable hypothesis generator, and then use intervention and external tasks to verify.
e. Indicates direction, Steering and control
Represents the commonly used tools such as linear Probe, PCA/CCA, Logit Lens, and Activation Steering. They can quickly answer whether a layer linearly encodes a attribute, and can also change the model output style or factual tendency through addition and subtraction directions. A series of work instructions of Representation Engineering, there is an actionable direction within LLM, which can affect honesty, emotion, absention, toxicity, preferences and other behaviors.
The relationship between this line and MI has tension. It provides a clear intervention interface, but does not necessarily give a complete mechanism. A Steering Direction is valid and may just take advantage of the model's local control panel, rather than explaining how the model produces that behavior. AxBench and other reviews are valuable because it compares Steering Success, Control Specificity and Interpretability Evidence in the same experimental framework.
Comparing the methods
| Route | Main object | typical evidence | Representative Resources/Paper | Main risk |
|---|---|---|---|---|
| Transformer circuits | attention head、MLP、residual stream | Named circuit, path analysis, manual verification | Mathematical Framework, Induction Heads | Success cases may be small and difficult to extend to complex behaviors |
| Activation patching / causal tracing | Layer, location, header, path activation | clean/corrupted intervention in recovery behavior | Causal Tracing, ACDC, path patching | Prompt pair and metric selection have a great impact |
| SAE / dictionary learning | sparse feature, feature dictionary | Reconstruction, sparsity, high activation sample, intervention | Towards Monosemanticity, Gemma Scope, SAEBench | Feature Naming over-explanation, Benchmark is still unstable |
| Circuit tracing / attribution graph | Calculation diagram between feature | feature graph, side attribution, local intervention | Circuit Tracing, On the Biology of a LLM | The graph structure depends on approximation and thresholds, not necessarily complete cause and effect |
| Indicates direction / Steering | Linear direction, Probe, Activation Edit | Control effects, attribute reading, counterfactual editing | Representation Engineering, AxBench | Controlling does not mean interpretable, there may be a lack of mechanism closed loop |
How to interpret the experiments and benchmarks
When reading a MI paper, it is not recommended to only look at beautiful visualizations or several Anecdotes. At least seven questions to ask.
First, what is the object of interpretation? Is it a single Neuron, SAE Feature, Attention Head, MLP Channel, Residual Direction, or cross-layer Circuit? Different granularity and different requirements for evidence.
Second, is there a causal intervention? Showing only high activation samples or correlation curves is not enough. More evidence includes Ablation, Patching, Feature Clamping, Direction Steering, Path Intervention, and the impact of the task metric after intervention.
Third, is there a negative control? For example, random feature, random direction, random layer, irrelevant prompt, token at the same frequency but with different semantics. Without negative control, it is difficult to judge whether the interpretation is caused by data distribution or general sensitivity of the model.
Fourth, did the Benchmark detect the real goal? SAEBench focuses on the multidimensional quality of SAE, AxBench focuses on Steering/Control/Alignment style tasks, but any Benchmark will induce optimization. Score improvements do not necessarily mean that the interpretation is closer to the real mechanism, and needs to be combined with manual review and intervention experiments.
Fifth, is it reproduced across models, layers, and across data? If a mechanism is only established on a single model, a single Prompt Family, and a single layer, the conclusion should be narrowed. The reproducibility of models such as Pythia, Gemma, Llama, and Claude is more convincing, but closed-source models limit verification.
Sixth, is the failure to report a sample report failure? A good MI paper should show the case of explanation failure: feature polysemantic, patching is unstable, side effects of Steering, and circuit graph miss path. Failure examples can help follow-up research more than just showing success graphs.
Seventh, is it different from "human readable" and "model actual use"? A feature name can help people understand, but the name is not the mechanism itself. The real key is whether the feature has a measurable effect in the model calculation.
Reproducibility and data problems
The difficulty of reproducing MI mainly lies in the details. The Patching experiment requires a clear model version, tokenizer, prompt pair, layer and position index, logit metric, sampling settings; SAE needs to report training corpus, activation cache, layer, hidden Expansion, sparse punishment, dead feature processing, refactoring error and training steps; Circuit tracing needs to describe the Attribution Graph construct, threshold, local approximation and manual screening process.
Data is also very sensitive. When interpreting a fact-recall mechanism, the wording of the prompt, the frequency of the entity, and the pollution of the training corpus will affect the results; when explaining safety-related behaviors, the public prompt may trigger model policy differences; when interpreting code or mathematical abilities, tokenization and format details will change the activation path. Therefore, this article does not write the conclusion that "a method reaches SOTA" that is easy to expire. For BenchMark results, repository status, model coverage, and online platform functions, manual verification should be performed on the paper homepage, GitHub Release, OpenReview, or arXiv latest version before publication.
Limitations and open questions
First, the MI does not have a reliable "microscope". SAE, Probe, Patching, Attribution Graph are all tools, not instruments that guarantee the truth. Tools will bring their own deviations.
Second, features may not be natural concepts. Humans tend to name Features, but the variables inside the model may mix semantics, syntax, location, frequency, and task policies. Polysemanticity is not an exception, but may be part of an efficient representation.
Third, interpretation and control may be separated. A direction can Steer model, it does not mean that the model is output; a Circuit graph can explain certain examples, it does not mean that it can control all related behaviors stably.
Fourth, scale will dilute manual understanding. There may be tens of millions of candidates for the large model. Even if tools can automatically generate interpretations, humans cannot review them one by one. In the future, better sampling, auditing, hierarchical organization and error discovery mechanisms are needed.
Fifth, the AI Safety context is easy to overestimate the short-term ability of MI. MI has a potential for finding risk mechanisms, but there is currently no guarantee that large models do not have hidden targets, spoofing strategies, or unknown paths to failure. It is reasonable to treat MI as part of a security demonstration, and it is too early to treat it as a complete proof of safety.
Research questions for graduate students
SAE Feature's Causality Evaluation Protocol: not only evaluate refactoring and sparseness, but also require feature ablation, feature swap, cross-prompt family and negative controls.
Reproduce across model Circuit: Compare open source models such as Pythia, Gemma, Llama and other open source models on the same task, and study whether similar behaviors are implemented by similar paths.
Interpret Failed Sample Dataset: Collect cases of feature naming errors, patching false positive, Steering side effects, benchmark scores and manual judgment conflicts.
Credibility Calibration for Automatic Circuit Discovery: Let the algorithm output the mechanism diagram and output uncertainty at the same time, and use the pre-registration intervention experiment to verify.
Combining MI with formal verification: Convert the found circuit or representation into a testable specification, such as whether the abstention path must be triggered when some safe features are activated.
Mechanism of multimodal model: Study the causal path between visual token, language token, and cross-attention, and avoid only putting the text LLM method directly to VLM.
Summary
Mechanistic Interpretability is moving from small Circuits Demo to the internal mechanics project of the large model. The most noteworthy change from 2025 to 2026 is the expansion of the research subjects from a single Neuron/head to SAE Feature, Transcoder, Attribution Graph, and BENCHMARK; research method shifts from "visualized interpretation" to "intervention, reproducible, auditable mechanism assumptions".
The prospects of this route are great, but it is not advisable to be carried away by the beautiful map. For scientific readers, the most valuable judgment standard is still simple: Can the explanation predict new examples? Can you withstand intervention? Is there a negative control? Do you admit failure? If the answer is no, it is at most an interesting visualization; if the answer gradually becomes affirmative, MI may become the core scientific tool for understanding and governing a large model.
References
Search date: 2026-07-15. The following priority lists papers, official research pages, project homepage and tool documents; model coverage, code repository status, benchmark ranking and online platform functions may change. It is recommended to manually verify before publication.
- Elhage et al., A Mathematical Framework for Transformer Circuits, Transformer Circuits, 2021. https://transformer-circuits.pub/2021/framework/index.html
- Olsson et al., In-context Learning and Induction Heads, Transformer Circuits, 2022. https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html
- Elhage et al., Toy Models of Superposition, Transformer Circuits, 2022. https://transformer-circuits.pub/2022/toy_model/index.html
- Bricken et al., Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, Transformer Circuits, 2023. https://transformer-circuits.pub/2023/monosemantic-features/index.html
- Templeton et al., Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet, Transformer Circuits / Anthropic, 2024. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html
- Marks et al., Circuit Tracing: Revealing Computational Graphs in Language Models, Transformer Circuits, 2025. https://transformer-circuits.pub/2025/attribution-graphs/methods.html
- Anthropic, On the Biology of a Large Language Model, 2025. https://transformer-circuits.pub/2025/attribution-graphs/biology.html
- Anthropic, Tracing the Thoughts of a Large Language Model, 2025. https://www.anthropic.com/research/tracing-thoughts-language-model
- Meng et al., Locating and Editing Factual Associations in GPT, NeurIPS 2022 / project page. https://rome.baulab.info/
- Conmy et al., Towards Automated Circuit Discovery for Mechanistic Interpretability, NeurIPS 2023. https://arxiv.org/abs/2304.14997
- Wang et al., Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, ICLR 2023. https://arxiv.org/abs/2211.00593
- Bills et al., Language Models Can Explain Neurons in Language Models, OpenAI, 2023. https://openai.com/index/language-models-can-explain-neurons-in-language-models/
- Google DeepMind, Gemma Scope: helping the safety community shed light on the inner workings of language models, 2024. https://deepmind.google/discover/blog/gemma-scope-helping-the-safety-community-shed-light-on-the-inner-workings-of-language-models/
- Bloom, Neuronpedia: Interactive Reference and Tooling for Mechanistic Interpretability, project page, 2024-2026. https://www.neuronpedia.org/
- Karvonen et al., SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability, arXiv, 2025. https://arxiv.org/abs/2503.09532
- Wu et al., AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders, arXiv, 2025. https://arxiv.org/abs/2501.17148
- Chanin, Are Sparse Autoencoder Benchmarks Reliable?, arXiv, 2026. https://arxiv.org/abs/2605.18229
- Heap et al., Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers, arXiv, 2025, revised 2026. https://arxiv.org/abs/2501.17727
- Huben et al., Sparse Autoencoders Find Highly Interpretable Directions in Language Models, ICLR 2024. https://openreview.net/forum?id=F76bwRSLeK
- Nanda and Bloom, TransformerLens, documentation and repository. https://github.com/TransformerLensOrg/TransformerLens
- nnsight team, nnsight: interpreting and manipulating the internals of deep learning models, documentation. https://nnsight.net/
- Sharkey et al., Mechanistic Interpretability for AI Safety: A Review, arXiv / TMLR, 2025. https://arxiv.org/abs/2404.14082