alt Hacker NewsData-only attacks are somewhat low-hanging fruit. Classical static analysis could already find them before AI got this strong, and LLMs make identification even easier. But the real threat is risk buried in business logic, especially abuse of normal business logic. Take e-commerce refund abuse. Bug hunters would not even call it a risk, yet fraud rings have arbitraged millions off this kind of logic. And because the logic is legitimate business logic, it is very hard to detect.
Replies
I still remember me and my friends on club live finding that the games you could just submit the scores for and get free xbox stuff, and then doing some research online years later we found the entire thing was setup by employees to abuse themselves with plausible deniability.
Club live lost msft millions of dollars by itself.
> And because the logic is legitimate business logic, it is very hard to detect.
Hard to detect at n=1, yes. But larger scale - are you assuming that no Accounting or Sales managers are watching the returns ratios, nobody in Shipping is minding carrier delivery failure metrics, and nobody in Returns is raising alarms about the bricks they're receiving?
Going on a bit of a tangent:
'Classic' non-AI fuzzers like AFL are still insanely useful and powerful, as are static analysis tools.
LLMs make all of these much, much easier to use. The other night, before I went to bed I told Kimi to go and fuzz filesystem code in the latest Linux kernel. I woke up to 26 crashes with reproducers and fixes. I'm still busy reviewing and upstreaming them. (Some have already landed.)