Recommended reading on Transformers?

[msp]

I watched a very interesting interview of Ilya Sutskever by Lex Fridman,
and he talks about “transformers” (see this part around the 1:00:00 mark).

I see that fastai has support for such models (e.g. tutorial here), but I was wondering, do you have some recommended videos or papers to understand what transformers are?

[stefan-ai (Stefan Josef)]

I recommend starting with the fastai NLP course (videos 17 and 18).

Then, I would go through these resources:

• excellent blog post by Jay Alammar: The Illustrated Transformer
• blog post with code from Harvard NLP: The Annotated Transformer
• read the original paper: Attention is all you need