You could have invented transformers from memory(alexwang.ai)
602 points by speckx | 133 comments
Blog post by Alex Wang explaining transformer layers (MLPs, gated MLPs, self-attention) as associative memory stores, with code and an interactive app.
#self-attention on Hacker News
5 stories tagged self-attention, newest first.
Writing an LLM from scratch, part 13 – attention heads are dumb(gilesthomas.com)
322 points by gpjt | 61 comments
Blog post, part 13 of a series working through Raschka's 'Build a Large Language Model (from Scratch)', explaining why self-attention's matrix multiplications work.
Writing an LLM from scratch, part 8 – trainable self-attention(gilesthomas.com)
350 points by gpjt | 26 comments
Blog post, part 8 of a series working through Raschka's 'Build a Large Language Model (from Scratch)', explaining self-attention with trainable weights.
The FFT Strikes Back: An Efficient Alternative to Self-Attention(arxiv.org)
438 points by iNic | 161 comments
arXiv paper introducing FFTNet, an FFT-based adaptive spectral filtering framework offering O(n log n) token mixing as an alternative to self-attention.
Beyond self-attention: How a small language model predicts the next token(shyam.blog)
411 points by tplrbv | 72 comments
A long blog post analyzing how a small ~10M parameter transformer trained from Karpathy's tutorial turns attention outputs into next-token predictions.