logoalt Hacker News

dosinga • yesterday at 11:29 PM • 0 replies • view on HN

roughly the tokenizer is optimized such that the training data (for the tokenizer) can be represented with the smallest number of tokens. and yes, English tends to be overrepresented in training data.