alt Hacker NewsI am curious if there has been work to remove experts from an open-weights model. The goal would be to reduce the size to be able to run on desktop GPUs without compromising quality. For a focused usecase - say coding, you dont need a model that knows world history. And, I am not talking about quantization. If it is possible to determine which experts are active for some usecases, and surgically remove the others.
Replies
Tepix • today at 7:37 PM
Yes. It's called REAP and from what I've seen, results aren't stellar.
monster_truck • today at 6:11 PM
Glazing over a lot, that's how they work already, just not in the way you think. A relatively small fraction of the model is active at any given time
Experts aren't trained on separate tasks. More recent routers are designed to spread out requests even more evenly, and they were already pretty even.