Retrofitting language models to operate over bytes

There has been extensive research into token-free LLMs, but for some reason we are still operating in a token domain, so there is something to that.

Byte Latent Transformers (BLT): https://arxiv.org/abs/2412.09871

Charformer: https://arxiv.org/abs/2106.12672

subword tokenizers were never causal in the first place, so BPE was peeking at future bytes all along! TIL

I don't understand how this is different from BPE tokenization.

Wow nature got rolled. Should have stayed closer to their expertise. They were already being hustled by a lot of the applied AI work they were accepting.

[flagged]

[flagged]

Weird paper. Models have had tokens for each byte for ages now. They can read and write individual bytes just fine, in addition to also having multibyte tokens.