Generate fonts where every LLM token is the same width(ampdot.mesh.host)
80 points by z-mach9 | 16 comments
A web tool and demo that compiles a font plus an LLM tokenizer into a token-space font where every token renders at equal width.
#tokenizers on Hacker News
2 stories tagged tokenizers, newest first.
Understanding GPT tokenizers(simonwillison.net)
380 points by simonw | 110 comments
Simon Willison blog post explaining how GPT tokenizers convert text to integer tokens, with an Observable notebook tool for encoding, decoding and searching tokens.