Visualizing Attention, a Transformer’s Heart | Chapter 6, Deep Learning
Demystifying attention, the key mechanism inside transformers and LLMs.
Instead of sponsored ad reads, these lessons are funded directly by viewers:
Special thanks to these supporters: #thanks
An equally valuable form of support is to simply share the videos.
Demystifying self-attention, multiple heads, and cross-attention.
Instead of sponsored ad reads, these lessons are funded directly by viewers:
The first pass for the translated subtitles here is machine-generated, and therefore notably imperfect. To contribute edits or fixes, visit
------------------
Here are a few other relevant resources
Build a GPT from scratch, by Andrej Karpathy
If you want a conceptual understanding of language models from the ground up, @vcubingx just started a short series of videos on the topic:
If you’re interested in the herculean task of interpreting what these large networks might actually be doing, the Transformer Circuits posts by Anthropic are great. In particular, it was only after reading one of these that I started thinking of the combination of the value and output matrices as being a combined low-rank map from the embedding space to itself, which, at least in my mind, made things much clearer than other sources.
Site with exercises related to ML programming and GPTs
History of language models by Brit Cruise, @ArtOfTheProblem
An early paper on how directions in embedding spaces have meaning:
------------------
Timestamps:
0:00 - Recap on embeddings
1:39 - Motivating examples
4:29 - The attention pattern
11:08 - Masking
12:42 - Context size
13:10 - Values
15:44 - Counting parameters
18:21 - Cross-attention
19:19 - Multiple heads
22:16 - The output matrix
23:19 - Going deeper
24:54 - Ending
------------------
These animations are largely made using a custom Python library, manim. See the FAQ comments here:
#manim
All code for specific videos is visible here:
The music is by Vincent Rubinetti.
------------------
3blue1brown is a channel about animating math, in all senses of the word animate. If you’re reading the bottom of a video description, I’m guessing you’re more interested than the average viewer in lessons here. It would mean a lot to me if you chose to stay up to date on new ones, either by subscribing here on YouTube or otherwise following on whichever platform below you check most regularly.
Mailing list:
Twitter:
Instagram:
Reddit:
Facebook:
Patreon:
Website:
37 views
6
0
2 months ago 03:33:33 1
Manifest Love & Miracles | 528Hz Positive Change & Self-Love Healing Frequency | Meditation & Sleep
2 months ago 00:12:31 1
АСМР Осмотр врача после Нового Года | Ролевая игра | ASMR Roleplay A Doctor
2 months ago 00:03:45 1
빅스(VIXX) - Error Official Music Video (Lip&Dance ver.)
2 months ago 00:03:10 1
HOTSHOT - Take A Shot MV
2 months ago 00:05:28 1
빅스(VIXX) - Error Official Music Video
2 months ago 00:01:21 1
JESUS: Son of God Movie – Teaser Trailer | Keanu Reeves
2 months ago 04:00:02 2
[4 Hours] Harry Potter ASMR Ambience + Dialogue ◈ Study & Relax w/ the Characters ◈ Multiple Scenes
2 months ago 01:39:00 2
Fläskkvartetten. МДМ. 1991
2 months ago 00:21:21 1
АСМР Аяно Аиши похитила тебя и угрожает расправой | Ролевая игра “Yandere Simulator“
2 months ago 00:17:43 1
ASMR Role Play | Ear Exam with Breathy Whispers “GOOD“ “OKAY“ and “HMM“ Deep in your Ears
2 months ago 00:02:17 9
Dracula | Official Trailer (2025) | Keanu Reeves, Jenna Ortega
2 months ago 00:20:56 1
BBC Presenter WRECKS Israel’s Spokesman, Video Goes Viral in Britain!
2 months ago 00:22:45 1
АСМР Ульяна расскажет тебе страшилки перед сном | Ролевая игра “Бесконечное лето“
2 months ago 00:19:09 1
ASMR | Up-Close Attention & Mouth Sounds ❤️
2 months ago 00:03:34 1
Paris Hilton - ADHD (Official Music Video) | Paris Hilton
2 months ago 00:15:13 1
АСМР итинг роллов 🍣 ASMR eating🍱
2 months ago 01:22:58 1
Deep Chill Music — Calm and Stress Relief Mix
2 months ago 00:00:00 7
😍WHAAT??😲 Live on THE Miami Swim Week 🔥 BLACK TAPE PROJECT LIVE 2024🔥 2024 LA Fashion Week🔥
2 months ago 00:03:21 1
빅스(VIXX) - [hyde] Official Music Video
2 months ago 10:10:08 1
SHAMANIC DRUMMING and DEEP VIKING CHANTS • DEEP TRANCE Humming Journey for Spiritual Awakening
2 months ago 00:11:50 1
“Otu“ (Demon) by CreatorCreepy, KineticFrost, SnowEye & more | Geometry Dash 2.2
2 months ago 01:53:17 1
David Byrne. Горбушка. 1994 год
2 months ago 00:26:26 1
Banshee Comforts You in the Haunted Forest | ASMR Roleplay
2 months ago 00:04:25 1
Deliver Us (Prince of Egypt) - Cover by Caleb Hyles and Jonathan Young