Beyond self-attention: How a small language model predicts the next token(shyam.blog)
411 points by tplrbv | 72 comments
A long blog post analyzing how a small ~10M parameter transformer trained from Karpathy's tutorial turns attention outputs into next-token predictions.