alt Hacker NewsI strongly suspect it's tokenization that drives this. If we trained with ASCII or even UTF8, I think we'd have much better results.
I strongly suspect it's tokenization that drives this. If we trained with ASCII or even UTF8, I think we'd have much better results.
Self-attention is an O(n^2) operation. You don't want to train on individual characters, or bytes.