Not OP, but I went through this last week. I (ok, codex) optimized a hot path in some Java code from ~350ms to ~60ms, which made a substantial difference in "is this whole business going to work".

My Java profiling knowledge is... let's call it "antique". I was really not looking forward to ramping back up for this work. Turns out, I didn't have to do any of it. The LLM chose the tools (flight recorder) and even built a JMH (also new to me) harness to experiment with different algorithms.

About half of the optimizations were things that I would have figured out on my own; the other half were definitely "wow" moments.

The whole thing was done in a couple hours, with just a few back-and-forths. Sans AI, it would have taken a week, with nowhere near the same gain. I'm impressed.

"Figure out how to make this process fast" is really a perfect activity for LLMs. And the prompt doesn't really have to be much more sophisticated than that.

I've been using:

JMH as the framework to write microbenchmarks. It takes care of dealing with JIT warmup, etc. It's the standard way to write rigorous Java microbenchmarks.

async-profiler (https://github.com/async-profiler/async-profiler) for profiling. Java has a problem where many profilers are based on safepoints, which are biased toward particular program points. async-profiler is not biased in this way.

Java Flight Recorder for memory allocation data.

One thing I've observed in all of this is that it's really useful to have expertise in the programming language and ecosystem you're writing in, otherwise it's all Greek to you and you can't really guide the agent to do the right thing. I have opinions about e.g. profilers and I can point the agent to one that I think is more accurate than other options.