Retrofitting language models to operate over bytes(nature.com)
99 points by theanonymousone | 20 comments
Open-access Nature research article describing a method to retrofit existing language models to operate directly over bytes instead of tokens.
#tokenization on Hacker News
9 stories tagged tokenization, newest first.
Font where each token is equal-width(twitter.com)
24 points by ampdot | 10 comments
An X post showing a prototype font in which every LLM token is rendered at equal width, illustrating how models see text and their quirks.
Show HN: Jevgpt – reinventing the wheel with another wheel
8 points by kunggaochicken | 0 comments
Show HN post for Jevgpt, a hobby autoregressive chatbot built from small-vocabulary token classifiers with a speculative decoding pass, linking to its GitHub repo.
GigaToken: ~1000x faster Language model tokenization(github.com)
585 points by syrusakbary | 116 comments
GitHub repository for gigatoken, an open-source library for fast language model tokenization at GB/s speeds, MIT licensed.
Karpathy on DeepSeek-OCR paper: Are pixels better inputs to LLMs than text?(twitter.com)
381 points by JnBrymn | 150 comments
An X post by Andrej Karpathy discussing the DeepSeek-OCR paper and whether pixels might be better inputs to LLMs than text tokens.
Why do LLMs freak out over the seahorse emoji?(vgel.me)
618 points by nyxt | 312 comments
Blog post by vgel examining why large language models produce confused, looping output when asked whether a seahorse emoji exists.
Byte Latent Transformer: Patches Scale Better Than Tokens(ai.meta.com)
366 points by zxexz | 83 comments
Meta AI research publication page for the Byte Latent Transformer paper, which proposes byte-level patches instead of tokens for scaling language models.
![Screenshot of youtube.com: Let's Build the GPT Tokenizer [video]](https://hn.urlbox.com/shots/renders/2024/2/20/3a4e4c3f-c4ce-4d7d-967c-d91df03c18b1.jpg)
Let's Build the GPT Tokenizer [video](youtube.com)
585 points by davidbarker | 42 comments
YouTube video lecture by Andrej Karpathy building the GPT tokenizer (byte-pair encoding) from scratch in a Jupyter notebook.
GPT-3.5 crashes when it thinks about useRalativeImagePath too much(iter.ca)
421 points by goranmoomin | 129 comments
Blog post showing that asking ChatGPT 3.5 to repeat a string containing "useRal" tokens consistently triggers an error in the message stream.