alt Hacker NewsKeep in mind all this kind of stuff can make the model less capable. If it has to think in "plain" English, it may well be squashing quality of code etc output.
I'm not sure how true this is, but when using "forced" json output it def had a big drop off in quality - https://arxiv.org/html/2408.02442v3.
I think you're better not fighting it with hacks like this and find a different model.
Replies
firemelt • today at 5:45 PM
how about stuff likes https://github.com/JuliusBrussee/caveman is it also make the model less capable?
silverwind • today at 2:56 PM
Changing output style shouldn't affect thinking at all.
I would not overgeneralize from paper. Firstly: forcing JSON output is, in my opinion, a bigger change than asking it to match the above style guidelines, and secondly, as is always the case with these kinds of papers, what was true for the model tested in the paper may either be completely false, or greatly reduced, in later models. That paper is almost 2 years old and models today have been trained in very different ways (or more accurately post trained in very different ways) and are in general far more capable.
Based on that paper, I would maybe try to check if it was true for a modern use case, I would very much not assume it was still true.