4 stories tagged attention-mechanism, newest first.
27 points by baraa_ | 2 comments
An online book guiding readers through building a large language model from scratch in NodeJS and TypeScript, covering dataloaders, embeddings, and attention.
525 points by weirdcat | 166 comments
arXiv paper introducing Diff Transformer, which computes attention as the difference of two softmax maps to cancel noise and improve long-context modeling and reduce hallucination.
959 points by elbasti | 307 comments
Evan Miller's blog post arguing that the softmax in the transformer attention formula has an off-by-one flaw and proposing a modified softmax.
380 points by tomcam | 91 comments
Cohere blog post explaining what transformer models are and how they work in natural language processing.