> We disabled code execution for all models in the demo, to prevent the LLMs from writing a brute-force backtracking solver in Python and running it, which would tell us something about their coding ability but nothing about their reasoning ability.
This seems sketchy. Reasoning how to solve the problem with a program is still reasoning, and if it yields programs which are more efficient (bc even a slow interpreter is faster/cheaper than an LLM) and potentially reusable, then so much the better!
I think the goal in this case is to isolate the LLM from that avenue in order to determine whether or not the approach is improving the LLM intrinsic ability, not because it is a better solution than having the LLM create programs.