Let’s Build the GPT Tokenizer (text version)

Announcements
1

We just published Let’s Build the GPT Tokenizer: A Complete Guide to Tokenization in LLMs, a translation of Andrej Karpathy’s excellent video into a book chapter. It’s likely to be of interest to anyone wanting to learn about how modern large language models work.

Next week we’ll post an article showing how we made it using Solveit.

Reply here if you have any questions or thoughts about the article or tokenization in general.

18 Likes
3

Thats great i am creating DeepSeek from scratch using fastai now a days

4 Likes
4

Hi Jeremy, the post looks great. I look forward to going through it in detail over the next few days.

Any time I try to click on the Solveit link on any browser I get a message saying “This site can’t be reached” or “The connection was reset”. I also get a message from Norton warning of a dangerous website or connection. This is happening in Firefox, Edge and Chrome. Just wondering if anyone else is having this issue?

1 Like
5

Sounds like your anti-virus has a false positive. Does it have a way to add exceptions? If so, add solve.it.com

2 Likes
6

it’s nice post, i have some questions:

  1. how to eval tokenizers from pretrained or trained one on a specific langauge!
  2. tokenizers with GPU instead of cpu
  3. tokenizer with task-heel
1 Like
7

Hi @jeremy. Thanks for the awesome blog post. I found it after listening to the MLST(Machine Learning Street Talk) podcast. I am very keen to see the meta-article on how you created the tokenizers post. Thanks.