Aren't tokenizers a hack to turn Latin alphabet texts into a pseudo Chinese with better per-unit information density?
It is desirable for all alphabets/abjads/abugidas/syllabaries, not just Latin. Chinese is rather the special case of a human language with very high character inventory and hence low character/word ratio.
alt Hacker News
It is desirable for all alphabets/abjads/abugidas/syllabaries, not just Latin. Chinese is rather the special case of a human language with very high character inventory and hence low character/word ratio.