logoalt Hacker News

abeppu • today at 3:58 PM • 1 reply • view on HN

> We disabled code execution for all models in the demo, to prevent the LLMs from writing a brute-force backtracking solver in Python and running it, which would tell us something about their coding ability but nothing about their reasoning ability.

This seems sketchy. Reasoning how to solve the problem with a program is still reasoning, and if it yields programs which are more efficient (bc even a slow interpreter is faster/cheaper than an LLM) and potentially reusable, then so much the better!


Replies

throwaway0q5347 • today at 4:04 PM

I think the goal in this case is to isolate the LLM from that avenue in order to determine whether or not the approach is improving the LLM intrinsic ability, not because it is a better solution than having the LLM create programs.

➕ show 1 reply