I've had pretty good luck with the following process for performance optimization loops:

- Have an agent generate unit tests until it gets to 100% path (not just statement) coverage, with every numerical test asserting checks against golden values to prevent regressions

- Let it rip on a performance improvement loop, for the widest E2E representative test case you have. Have it generate flamegraphs along the way so you can check in and steer it as necessary.

- Optionally allow for 1 ULP changes in output values so that it doesn't kill itself getting bit-exact results.

- Have it flag correctness errors as it goes, since your code probably isn't bug free.

This is also how I've done language ports from python to rust, and having the ironclad test coverage protects you from drifting.