alt Hacker NewsHow many of those automated fixes were reverted? How many introduced a new bug? What's the false positive rate on the finding agents? The post has counts for everything that went right and nothing for what could go wrong.
Replies
They also don’t really provide an explanation for the big uptick in bugs found M146+ in the first place.
Is that better testing? Or more bugs were being introduced in the first place?
> The post has counts for everything that went right and nothing for what could go wrong.
That's AI for you.
At Amazon we have many forums to share our AI wins, but none to share AI failures or disappoinments.
No wonder execs make bad decisions regarding AI, they only hear completely one-sided stories.
I assumed it was really just AI that found them, which allowed Google to know to fix them.
> How many introduced a new bug?
I'd say that one is not really an issue. In 2012 the Pinkie Pie exploit chain already required chaining 6 bugs to lead to an exploit [1]. Since then we've seen chains requiring more than 10 bugs (!).
If you fix any one of those bugs, the exploit is non-functional anymore. Sorry out of luck.
So if, say, for every ten bugs you fix, you introduce two new ones then it's still a very net win. Unless of course it introduces a bug so bad it becomes a simple exploit not requiring a long chain of exploits.
But in the case of browsers we've only ever been moving to longer and longer chains of exploits required to pwn a browser.
A great many window of opportunities are closing for dark-side hackers / north korean intelligence etc.: there were probably exploit chains still open for exploitation in April that just got closed by Google.
If anything, besides the supply chains attacks in amateur-land, the world didn't stop working: projects (not just browsers but OSes too) are being hardened left and right.
Using AI to find potential bugs is an amazing use case and there really aren't many downsides.
> The post has counts for everything that went right and nothing for what could go wrong.
I'm not saying there aren't a few downsides but the benefits are just too good to ignore.
[1] https://blog.chromium.org/2012/05/tale-of-two-pwnies-part-1....
Exactly, also it lacks a lot of context as to why they did that.
Possible (probable?) scenario:
- Marketing: "we found and fixed lots of bugs thanks to AI"
- Reality: the KPI is now to fix as many bugs as possible with the help of AI, so they used AI to search old and easy bugs in the backlog, and then fixed it manually