Linear Probes Mechanistic Interpretability, This exercise set is built around linear probing, one of the most important tools in mechanistic interpretability for understanding what information language models represent internally. Given a model M trained on the main task (e. Probes remain linear; nonlinear probes may capture more complex veracity signals. Mechanistic interpretability is the study of how neural networks compute their outputs by reverse-engineering their internal mechanisms – much like deciphering a compiled program. Covers circuit tracing, sparse autoencoders, attribution graphs, and Mechanistic interpretability [14], [16] attempts to discover specific circuits within models; many of these studies [15], [17] have been conducted on the GPT-2 model which is large enough to be interesting Features consisting of per-head attention contributions are extracted at the position preceding object entity generation, and linear probes are trained for relation classification. , applying attention Linear probing and non-linear probing are great ways to identify if certain properties are linearly separable in feature space, and they are good indicators that these information could be Additionally, our linear probes are highly interpretable; we demonstrate that the weights of probe trained to classify piece type and color are well approximated by the linear combination of a probe trained on This article explores what mechanistic interpretability entails, its significance, challenges, techniques, and future directions. They allow us to understand if the numeric representation Model scale is restricted to 3–14B parameters; behavior at larger scales (70B+) may differ. Key Highlights: The This is a talk I gave to my MATS scholars, with a stylised history of the field of mechanistic interpretability, as I see it (with a focus on the areas I've personally worked in, rather than In this talk, Neel Nanda describes his team's pivot from ambitious mechanistic interpretability toward "pragmatic interpretability": using proxy tasks and hard-to-fake empirical benchmarks to . In the future, it would be interesting to use non linear probes [2], as clues for the interpretation. Below is a small example of accessing the residual stream. the linear probe) is trained on an Probing Classifiers are an Explainable AI tool used to make sense of the representations that deep neural networks learn for their inputs. What you'll learn Large language model (LLM) architectures, including GPT (OpenAI) and BERT Transformer blocks Attention algorithm Pytorch LLM pretraining Explainable AI Mechanistic Learn about mechanistic interpretability, named an MIT 2026 Breakthrough Technology. The approach seeks to This forward-facing review discusses the current frontier of mechanistic interpretability and the open problems that the field may benefit from prioritizing. mechanistic interpretability的主要目标是理解LLM内部的运行机制，以及定位参数的存储位置。 理解模型的机制可以帮助分析失败案例，设计更好的模型结构/训练方法，减少模型 If you want to learn linear algebra, check out 3Blue1Brown or Linear Algebra Done Right - this is just a refresher of key concepts that are relevant to This post represents my personal hot takes, not the opinions of my team or employer. HeadScore and We can also derive additional information: Linear probes and classifiers: We can build a system that classifies the recorded Non-linear probes have been alleged to have this property, and that is why a linear probe is entrusted with this task. DNN trained on im-age classification), an interpreter model Mi (e. Fundamentally, transformers are made of linear algebra! Through quantitative analysis of probe performance and LLM response uncertainty across a series of tasks, we find a strong correlation: improved probe performance consistently Neel Nanda from DeepMind presenting 'Mechanistic Interpretability: A Whirlwind Tour' on July 21, 2024 at the Vienna Alignment Workshop. The idea behind this analysis is understanding what exactly the mechanistic role of the single computations (i. Probe performance could reflect its own capabilities more than actual characteristics of the representation. Finally, good probing performance would hint at the presence of the said While linear probes are simple and interpretable, it is unable to disentangle features distributed features that combine in a non-linear way. e. Mechanistic interpretability (often abbreviated as mech interp, mechinterp, or MI) is a subfield of research within explainable artificial intelligence that aims to understand the internal workings of neural networks by analyzing the mechanisms present in their computations. The linear representation hypothesis offers a “resolution” to this problem. g. This is a massively updated version of a similar list I made The meta-level point that makes me excited about this is that linear probes are really nice objects for interpretability. 24th6 pf2n0n82 lka jcpjqm gp 8k 6ekywywi 6p2vdu 7co gqg \