The Softmax function and its derivative(eli.thegreenplace.net)
33 points by costco | 2 comments
A blog post by Eli Bendersky explaining the softmax function mathematically and deriving its derivative for machine learning use.
#softmax on Hacker News
3 stories tagged softmax, newest first.
New exponent functions that make SiLU and SoftMax 2x faster, at full accuracy(github.com)
368 points by weinzierl | 72 comments
GitHub pull request to llama.cpp by jart that upstreams llamafile's vectorized expf() functions to compute SiLU and softmax faster and more accurately on CPU.
Attention Is Off By One(evanmiller.org)
959 points by elbasti | 307 comments
Evan Miller's blog post arguing that the softmax in the transformer attention formula has an off-by-one flaw and proposing a modified softmax.