alt Hacker NewsGiven the frontier-level capabilities of Kimi K3, I'm wondering if it's possible to extract the core capabilities (fundamental reasoning and tool calling) of the model into a smaller one that consumer devices could run? Not sure exactly how, but either by heavy distillation or some other surgical method since Kimi has a Mixture of Experts architecture.
I think it's very valuable to have a smaller model that doesn't have any domain knowledge or facts built into its weights, but given the right context, could accurately reason about what to do and use the right tools.
I'm aware of colibri [1], but so far I've only seen extremely slow performance.
Replies
Why not generate an artificial dataset using commercial APIs and then finetune a small model on this data?
I’ve had success adapting even a 7B model for single-domain tasks that way, including reasoning and tool calling.
You can use an open model. The point is just to outsource the inference, so you don’t have to deal with running the larger model yourself.
There’s another way besides distillation that’s way cheaper: You can have the big model build prescriptive skills that the small model follows.
Take the “train” portion of tasks on some benchmark, have K3 complete it, and then output detailed descriptions of tools used and why, then run the validation tasks with some small model that has access to the skills.
"I'd like a car that goes 300mph and gets 100mpg while doing it. I'm aware of a car that gets 100mpg but it is extremely slow."
You are describing fundamental tradeoffs. Getting more performance relative to model size and training token amount is what all of the labs are solving.