Looks similar
Understanding GPT tokenizers(simonwillison.net)
380 points by simonw | 110 comments
Simon Willison blog post explaining how GPT tokenizers convert text to integer tokens, with an Observable notebook tool for encoding, decoding and searching tokens.
Optimi-Zi(n)g Sudoku-Solving(log.pfad.fr)
14 points by todsacerdoti | 2 comments






