>I feel a lot of these tools are barking up the wrong tree.

Its the right tree, but not the right branch. RAG systems are the next big thing. You can take a small model running locally, feed it a bunch of context and it will be able to generate the correct answer on par with a larger model with no context.

Context doesn't have to be documents/code either, it can be LLM generated context that guides next iterations of inference on how to do things.