logoalt Hacker News

stephantultoday at 7:28 AM0 repliesview on HN

I think that some of these choices (as others have commented) show that the author has not investigated how tokenization works.

Tokenization is not some black box, you can run tokenizers and check them.