alt Hacker NewsWhy is it replacing true/false with T/F? true/false is already 1 token in all tokenizer I've seen. Even worse is replacing null with ∅. ∅ is a special unicode symbol that takes up 2 tokens compared to the 1 token for null...
Replies
wehnsdaefflae • today at 9:12 AM
Maybe they want to be tokenizer agnostic? Then they would need to go by character count, right? Despite these inconsistencies, has anyone actually verified their promise? 350 vs. 10.000 tokens would still be very valuable even if they mess up some edge cases
➕ show 1 reply
cedws • today at 7:58 AM
Brand new GitHub account, brand new HN account. I stay far away from projects like this these days, they can easily be malicious. GitHub needs some kind of indicator for projects authored by tenured developers with a real identity.
➕ show 1 reply
What's even weirder is that the substitution is explained in "How TOON works" section. Yet TOON never describes such behavior anywhere in its spec.