logoalt Hacker News

mordaetoday at 10:49 AM0 repliesview on HN

I think that in this case there is also the problem of trying to transfer MoE-style reasoning into a dense model. I mean, MoE needs reasoning to walk multiple experts, but dense model already has all the weights. So when you push it hard to reproduce the MoE traces, you are effectively asking a small mouse to role-play as an anthill. Not great.