logoalt Hacker News

Retr0id • today at 1:24 AM • 1 reply • view on HN

> This is variation of scientific method and it significantly helps agents when they are getting stuck on some problem. Stops it going in circles. By forcing it to do definitive experiments. No endless guessing and trying random things.

These "guessing loops" are my main annoyance when REing with Claude (mostly Opus 4.8, due to the restrictions on newer models). On harder/larger targets, I have to constantly babysit it and tell it to "stop guessing".

For truly black-box reverse engineering, yes, hypothesis-testing is the way to go (it's all you can do). But in most cases all the information is available to arrive at an answer deterministically, it just needs to be worked through meticulously. As an experienced human I have strong intuitions and I can work much faster under a "guess and check" model, but I prefer to use Claude as an "unstoppable RE juggernaut" with superhuman patience and determination, tracing method invocations all the way from A to distant subsystem B (for example).

Another big win for hardware RE is from tasking the LLM with writing unicorn emulation harnesses, e.g. for bootroms and firmwares. It's pretty good at the classic "figure out which MMIO needs to be stubbed/simulated next to progress the boot" loop, although sometimes it tries to take egregious purpose-defeating shortcuts.


Replies

cyanydeez • today at 1:33 AM

On llamacpp and other engines, you can inject a think budget which truncates at specified token count. You can also specify a message at the limit. You can set this in the header, so a dynamic engineering harness could relax/tighten. Combine with a decision model and some robust messages like, "run a test to rule out items" and all that would reduce.

I've noticed the same habit with qwen3.8-flash-next. no matter how much I tell it a problem I see, it often fills up it's thinking with "the user says this, but maybe they're confused, etc". I eventually got vision mode turned on and that's significantly cut down on UI based pathologic skepticism; I got it access to my browser and that's further reduced it.

But nonetheless, if you can get them to test their thinking, they seem more grounded.