Why Is Speculative Decoding Fast?(jbarrow.ai)
41 points by gmays | 5 comments
Blog post by Joe Barrow explaining that speculative decoding speeds up LLM inference by changing the kind of GPU work rather than reducing FLOPs.
First screen, captured .
First screen, captured .
41 points by gmays | 5 comments
Blog post by Joe Barrow explaining that speculative decoding speeds up LLM inference by changing the kind of GPU work rather than reducing FLOPs.
120 points by adityaathalye | 29 comments
