I'm far behind the curve with local/open weight models, but I thought it was common practice that you should do some last-mile fine-tuning on the particular harness that you're using and would get a reasonable return on investment for doing so (particularly on quantized models)?
Can anyone confirm/deny this is the case?
If so, it seems a shame not to include this practice into the reporting methodology for full transparency.
It's typically easier to tune the harness around the models' existing behavior. In my experience the primary failures in models > 5B params on coding tasks are just that they aren't able to translate instructions into a correct description or implementation of the problem (will write wrong code, wrong tests, and wrong documentation). The basic tool calling and syntax of generated code is fine.
I don't think this is common, though it would be interesting if ThinkingMachines/HuggingFace/OpenRouter/BaseTen offered fine-tuned variants of the most popular open models that worked with specific harnesses
In MiMo's v2.6 public RL training run, they explicitly trained on 23 different harnesses in order to minimize overfitting on any single harness. I imagine that this technique has to be common in RL training runs given that model providers have little control over where their models are used.
https://mimo.xiaomi.com/rl/#overview