I think the goal in this case is to isolate the LLM from that avenue in order to determine whether or not the approach is improving the LLM intrinsic ability, not because it is a better solution than having the LLM create programs.
What is reasoning if examining a problem and constructing a program to solve it doesn't count? Changing the original represention into one that is more familiar (particular sequences of tokens) seems like it is all about "intrinsic ability".
I'm not saying they don't have a point, but I think this is actually not great evidence.
What is reasoning if examining a problem and constructing a program to solve it doesn't count? Changing the original represention into one that is more familiar (particular sequences of tokens) seems like it is all about "intrinsic ability".
I'm not saying they don't have a point, but I think this is actually not great evidence.