logoalt Hacker News

mordaetoday at 10:34 AM0 repliesview on HN

It needs to argue with itself to extract most of the knowledge embedded in the weights into the context. Asking it to synthesize ideas directly in a single go is simply unreasonable. And MoE models need to walk multiple experts to extract all the knowledge on top of that. So you need to give them the reasoning trace to first spill all the associations into.