alt Hacker News> Given you also don't want it to memorise [for all tokens, count([for all letters])
Why not? You've memorized how words are spelled, and how sounds correspond with letters, and how concepts correspond with words. To the extent that there are shortcuts that enable compression you use these, and the model will do something similar.
Replies
mdp2021 • yesterday at 8:51 PM
> Why not?
Because to "123x456" we want a reply that goes "this times that plus that...", not "Was that not nnnnnn?". If it does not perform its duty (returning solid checked answers) it is a liability.
Combinatorial explosion, and facts merely memorised is a huge waste of parameters that are better dedicated to effective reasoning. Not that we really know how to split facts from skills, though we are trying various approaches.
Being able to spell all the words then count letters is simpler, and more generalisable to other tasks, than memorising answers to all possible word questions.
That said, we're so bad at splitting facts from skills that trying to get them to memorise a bunch of facts might force them to learn a skill and generalise anyway.