1. The spelled-out intro to neural networks and backpropagation: building micrograd
The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement two fundamental functions: encode() from strings to tokens, and decode() back from tokens to strings. In this lecture we build from scratch the Tokenizer used in the GPT series from OpenAI. In the process, we will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We’ll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.
Chapters:
00:00:00 intro: Tokenization, GPT-2 paper, tokenization-related issues
00:05:50 tokenization by example in a Web UI (tiktokenizer)
00:14:56 strings in Python, Unicode code points
00:18:15 Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32
00:22:47 daydreaming: deleting tokenization
00:23:50 Byte Pair Encoding (BPE) algorithm walkthrough
00:27:02 starting the implementation
00:28:35 counting consecutive pairs, finding most common pair
00:30:36 merging the most common pair
00:34:58 training the tokenizer: adding the while loop, compression ratio
00:39:20 tokenizer/LLM diagram: it is a completely separate stage
00:42:47 decoding tokens to strings
00:48:21 encoding strings to tokens
00:57:36 regex patterns to force splits across categories
01:11:38 tiktoken library intro, differences between GPT-2/GPT-4 regex
01:14:59 GPT-2 released by OpenAI walkthrough
01:18:26 special tokens, tiktoken handling of, GPT-2/GPT-4 differences
01:25:28 minbpe exercise time! write your own GPT-4 tokenizer
01:28:42 sentencepiece library intro, used to train Llama 2 vocabulary
01:43:27 how to set vocabulary set? revisiting transformer
01:48:11 training new tokens, example of prompt compression
01:49:58 multimodal [image, video, audio] tokenization with vector quantization
01:51:41 revisiting and explaining the quirks of LLM tokenization
02:10:20 final recommendations
02:12:50 ??? :)
Exercises:
- Advised flow: reference this document and try to implement the steps before I give away the partial solutions in the video. The full solutions if you’re getting stuck are in the minbpe code
Links:
- Google colab for the video:
- GitHub repo for the video: minBPE
Supplementary links:
- tiktokenizer
- tiktoken from OpenAI:
- sentencepiece from Google
34 views
8979
3340
7 hours ago 00:00:50 1
Servicemen of the assault units of the Vostok group of forces hone their combat training skills at one of the training grounds i
7 hours ago 00:04:35 1
[Мицуко Утида – тема] Mozart: Piano Sonata No. 16 in C Major, K. 545 “Sonata facile“: I. Allegro
7 hours ago 00:00:10 37
4ed8b0b42735a3c5cfdb2fae6879bbe3 (1).mp4
7 hours ago 00:00:09 1
«Четыре героя бамбукового леса» Вечная ночь звëздной реки / Love Game in Eastern Fantasy (永夜星河)
7 hours ago 00:00:25 6
Video by Актриса и телеведущая Екатерина Гетрик
7 hours ago 00:21:20 1
[Mark Rober] Этот Робот Решает Пазлы в 200 Раз Быстрее Человека
7 hours ago 00:13:07 1
[Анимэр] Яндере-Ангел Уничтожила Мир, Чтобы Навсегда Остаться с Любимым!
7 hours ago 00:43:42 3
Сериал Вавилон 5 сезон 1 серия 11-выжившие
7 hours ago 00:00:26 1
[Sounds Recorded] Old Church Bell Sound Effect
7 hours ago 00:55:53 1
Джоэл-дезинсектор! The Last Of Us Part I Стрим#9 ФИНАЛ
7 hours ago 00:17:40 1
Первый Финал: Изгнание! 201 – The Bad Number #14
8 hours ago 00:01:05 1
[The Times and The Sunday Times] Caver rescued after 12 days trapped underground in Turkey
8 hours ago 00:06:20 1
[ABC News] What Biden and Trump need to answer in their 1st presidential debate
8 hours ago 00:02:39 1
[Movieclips] Doubt (2/10) Movie CLIP - I Am Concerned (2008) HD
8 hours ago 00:02:39 1
[Movieclips] On the Waterfront (2/8) Movie CLIP - Am I Gonna See You Again? (1954) HD
8 hours ago 00:00:23 7
дежурными средствами ПВО уничтожены 2 БЛА над Курской и 1 БЛА над Белгородской областью
8 hours ago 00:21:15 1
[NISM] TNO(СУПЕРИВЕНТЫ) - ЩИТПОСТЫ И ПРИКОЛДЕСЫ
8 hours ago 00:00:10 1
Video by Защитники Отечества I Республика Дагестан
8 hours ago 00:45:30 1
[PRO-onLineIGRY] ToCA Race Driver 3 (МТ, сложно): этап 17, 1960’S Grand Prix