alt Hacker NewsI'm guessing this is an allusion to reinventing something that already exists, but do you mind explaining what that is to me, since I don't know?
Replies
BobaFloutist • yesterday at 5:19 PM
I suspect they're talking about LLM tokens.
I'm guessing this is an allusion to reinventing something that already exists, but do you mind explaining what that is to me, since I don't know?
I suspect they're talking about LLM tokens.
It is describing essentially how tokenisation is done for LLMs (Large Language Model → Expansive Dictionary Model).