Understanding GPT tokenizers(simonwillison.net)
380 points by simonw | 110 comments
Simon Willison blog post explaining how GPT tokenizers convert text to integer tokens, with an Observable notebook tool for encoding, decoding and searching tokens.
First screen, captured .
Looks similar
Show HN: AutoThink – Boosts local LLM performance with adaptive reasoning
386 points by codelion | 62 comments
Show HN post introducing AutoThink, a technique that adaptively allocates thinking tokens by query complexity and uses steering vectors to improve local LLM reasoning.
Finding Optimal Tokenizers(blog.aqnichol.com)
25 points by mcyc | 1 comments
Nanochat(simonwillison.net)
26 points by bilsbie | 4 comments
OpenAI Tokenizer(platform.openai.com)
289 points by tosh | 152 comments
