logoalt Hacker News

robrenaud • yesterday at 3:05 PM • 0 replies • view on HN

Use different tokenizers for different languages. Or at least, use different tokenizers for different scripts.