What Do Neural Networks Really Learn? Exploring the Brain of an AI Model

Neural networks have become increasingly impressive in recent years, but there’s a big catch: we don’t really know what they are doing. We give them data and ways to get feedback, and somehow, they learn all kinds of tasks. It would be really useful, especially for safety purposes, to understand what they have learned and how they work after they’ve been trained. The ultimate goal is not only to understand in broad strokes what they’re doing but to precisely reverse engineer the algorithms encoded in their parameters. This is the ambitious goal of mechanistic interpretability. As an introduction to this field, we show how researchers have been able to partly reverse-engineer how InceptionV1, a convolutional neural network, recognizes images. ▀▀▀▀▀▀▀▀▀SOURCES & READINGS▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ This topic is truly a rabbit hole. If you want to learn more about this important research and even contribute to it, check out this list of sources about mechanistic interpretability and interpretability in general we’ve compiled for you: On Interpreting InceptionV1: Feature visualization: Zoom in: An Introduction to Circuits: The Distill journal contains several articles that try to make sense of how exactly InceptionV1 does what it does: OpenAI’s Microscope tool lets us visualize the neurons and channels of a number of vision models in great detail: Here’s OpenAI’s Microscope tool pointed on layer Mixed3b in InceptionV1: Activation atlases: More recent work applying SAEs to InceptionV1: Transformer Circuits Thread, the spiritual successor of the circuits thread on InceptionV1. This time on transformers: In the video, we cite “Toy Models of Superposition“: We also cite “Towards Monosemanticity: Decomposing Language Models With Dictionary Learning“: More recent progress: Mapping the Mind of a Large Language Model: Press: Paper in the transformers circuits thread: Extracting Concepts from GPT-4: Press: Paper: Browse features: Language models can explain neurons in language models (cited in the video): Press: Paper: View neurons: Neel Nanda on how to get started with Mechanistic Interpretability: Concrete Steps to Get Started in Transformer Mechanistic Interpretability: Mechanistic Interpretability Quickstart Guide: 200 Concrete Open Problems in Mechanistic Interpretability: More work mentioned in the video: Progress measures for grokking via mechanistic interpretability: Discovering Latent Knowledge in Language Models Without Supervision: Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning: ▀▀▀▀▀▀▀▀▀PATREON, MEMBERSHIP, MERCH▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 🟠 Patreon: 🔵 Channel membership: 🟢 Merch: 🟤 Ko-fi, for one-time and recurring donations: ▀▀▀▀▀▀▀▀▀SOCIAL & DISCORD▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ Discord: Reddit: X/Twitter: ▀▀▀▀▀▀▀▀▀PATRONS & MEMBERS▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ AAAA you don’t fit in the description this time! But we thank you from the bottom of our hearts. All of you, in this Google Doc: ▀▀▀▀▀▀▀CREDITS▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ All the good doggos who worked on this video: