Built exactly this for work a few months ago. Claude has a read-only account to the dev DB and a bunch of python postgres tools, I point it at queries/views and ask it optimize, and it has to:

1) verify identical query results

2) run repeatedly to get average, worst, best, etc duration of runs

Sped up so many legacy things that none of us were ever going to bother with.

Just like humans, AI needs a "reality" to check things against. If we don't provide that in an automated way, somehow, it will always make mistakes and we'd have to point it out. Since, at that point, we're just mediating with what reality gives us and what an AI should do with it.

AI doesn't have enough senses yet. It's trapped in a box.

Exactly. In the above case, AI should be able to verify it's own hypothesis by having access to the database's table statistics and planner.

I’ve had a lot of success with this. Give it a good enough harness, leave it for a few days and see what it finds that actually holds up.

What tooling makes this go?

But them it's more like AI assisted performance fuzzing? Just throw stuff against some measuring harness and see what sticks?

Very few problems really have "all the tooling to verify its hypotheses" though. even if you want to construct such an harness.

Also let me ask you why we need better and better and models if what we have already can produce good output with 'all the tooling to verify its hypotheses'

That’s how I do it in my Laravel apps. I simply ask it to hypothesize, benchmark, apply and verify. Laravel has great tools for that.

Do you people just not know about the massive, intractable security issues of doing this? Our do you just not care? Do you think it isn't going to hit you? Why do you continue to think this way in the face of the OpenAI/Hugging Face attack?