Understanding GPT tokenizers(simonwillison.net)
380 points by simonw | 110 comments
Simon Willison blog post explaining how GPT tokenizers convert text to integer tokens, with an Observable notebook tool for encoding, decoding and searching tokens.
380 points by simonw | 110 comments
Simon Willison blog post explaining how GPT tokenizers convert text to integer tokens, with an Observable notebook tool for encoding, decoding and searching tokens.
94 points by throw0101b | 64 comments
97 points by gyosifov | 29 comments