Machine learning
Attention and transformers
Attention, self-attention, multi-head attention and the transformer, the architecture behind every modern LLM.
About 20 hours of study.
Where to learn it
Read
Watch
Neural Networks: Zero to Hero
Andrej Karpathy
Let's build GPT, from scratch
You write a working transformer yourself, line by line.
Section not checked yet
Practise
Understanding Deep Learning notebooks
Simon J. D. Prince
Notebooks for chapter 12
Sixty-eight Python notebooks with the key code left for you to fill in.
Other options
- The Illustrated Transformer (read)
The whole post
Wrong chapter, dead link, or know something better? Tell us on GitHub.
Which roles ask for it
- Machine learning engineer in the first year
- Deep learning engineer (vision / language) to get hired
Before and after
Attention and transformers