alt Hacker NewsIsn't iterative chunking the whole point of the transformer though? Or does it add that much overhead to do it at the byte level?
Replies
Ohentis • yesterday at 8:49 PM
No. Most transformer variants work on either a flat stream, or a fixed sized internal state.
What do you mean by iterative chunking? There are Transformer variants that hierarchically build token- or word-level representations from bytes, then reverse it to decode, but they never really took off. Didn't help much compared to a good tokenizer.