alt Hacker NewsThis is very similar in root cause and exploitation to Copy Fail.
Which illustrates pretty well something that's lost when relying heavily on LLMs to do work for you: exploration.
I find that doing vulnerability research using AI really hinders my creativity. When your workflow consists of asking questions and getting answers immediately, you don't get to see what's nearby. It's like a genie - you get exactly what you asked for and nothing more.
The researcher who discovered Copy Fail relied heavily on AI after noticing something fishy. If he had to manually wade through lots of code by himself, he would have many more chances to spot these twin bugs.
At the same time, I'm pretty sure that by using slightly less directed prompting, a frontier LLM would found these bugs for him too.
It's a very unusual case of negative synergy, where working together hurt performance.
Replies
No, unless I'm misreading it it's the *same* root cause: high 32 bits of Extended ESN in IPsec == authencesn module/cipher mode.
The wrong thing got fixed for copy.fail, because people jumped to blame AF_ALG.
[ed.: yes it's the same authencesn issue. https://github.com/V4bel/dirtyfrag/blob/892d9a31d391b7f0fccb... it doesn't say authencesn in the code, only in a comment, but nonetheless, same issue.]
[ed.2: the RxRPC issue is separate, this is about the ESP one]
Or a follow up prompt: "find similar classes of bugs". Once the actual case has been layed out finding like bugs isn't too hard. I hear you on the creativity bit. Like any tool, AI can put blinders on. Using it to augment without it fully taking over your workflow is tough.
I don't know... after they found a high profile bug like copyfail, I wouldn't attribute not looking for similar bugs to them being overly dependent on AI. It's easy to stop exploring, for a while at least, after you've struck on a major find. Maybe they would've returned to it in a few months. It certainly inspired others to explore similar areas and find these new bugs. Isn't that enough?
I don't follow. LLMs spotted these bugs in the first place. You seem to be saying that these discoveries are indications that they're bad for vulnerability discovery.
Just on a side note. Negative synergy does not seem so uncommon with machine learning. We did some research maybe 10 yrs ago an human/ML based duplicate detection (for a municipal support ticket system) . Research showed that pure AI and pur human outperformed co-working. Human oversight often e.g. overcorrected machine work. I think it is a nice HCI problem to solve actually to amplify creativity and unique skills in such processes. Particularly if they can be to some degree repetitive and tiresome.
It’s very hard to see a root vuln similar to, but not the same as, another discovered by AI, as a lesson about AI not exploring.
Is there a counterfactual where you would say it explored well enough, besides both vulnerabilities published as one?
Evidence or are you just riffing?
These are all page cache poisoning attacks (dirtyfrag, copyfail, dirtypipe). Maybe the page cache should have defense-in-depth measures for SUID binaries?
> When your workflow consists of asking questions and getting answers immediately, you don't get to see what's nearby.
That's why is very very important to just step out and use saved time to go for a walk, to a park, sit on a bench, listen do birds, close eyes and zoom out.
The state we are in is actually brilliant.
> When your workflow consists of asking questions and getting answers immediately, you don't get to see what's nearby.
Very much aligns with my experience. For me this is the most unsatisfying thing about AI-based workflows in general, they miss stuff humans would never miss.
All the time I wonder what am I missing that's right nearby? It's remarkable how many times I have to ask Claude code to fully ingest something before it actually puts it into context. It always tries to laser through to target it's looking for, which is often not what you want it to look for, at least not all you want it to look for. Getting these models to open up their field of vision is tough.