alt Hacker NewsCan anyone think of any lessons to draw from this for normal development, where we don't have oracles to serve as guardrails? I write specs but the agents still find ways to insert bugs between the lines. Oh well, job security.
Replies
It's not really comparable. When decompiling you only care about a one time result. Code quality doesn't matter much. Stability doesn't matter much. Only whether the result is accurate. Once you have the matching decompiled code, all tools used until that point will be usually thrown away.
Normal development isn't so lucky, you do care about code quality. So you have to review the code the agents write and handle the testing yourself. The agents will not do what you want, they'll do what they think you want, and you're the only person who knows what you think.
[flagged]
The obvious lesson is that you need some kind of test/harness, even for normal development. Your spec will always leave room for interpretation in a sense that is undesired.
Obviously matching functions byte-byte is not it, but I think with enough creativity, you can have guardrails and oracles almost everywhere.
E.g. before diving into feature development, you can have agents generate a test suite based on your implementation first. Then, when developing features, this test suite shows what broke and agents can double check if that was intended.
How that test suite looks like obviously highly depends on the environment.