alt Hacker NewsGoogle fixed more Chrome bugs in June than over the past two years, thanks to AI
Comments
Not that I don't believe its possible to fix a lot of bugs, I also wonder what the actual dynamic was. Were the people in team working much more than usual as well? Given its Google, I wouldn't be surprised if there was an "internal push" to fix more bugs over next X sprints so that they can publish this blog and some manager can show impact and AI adaption to his superior.
The real datapoint was Firefox not paying any money in Berlin's Pwn2Own competition round this May just gone. Unheard of to have nothing confirmed... they've paid out every event since 2007 (I checked). Does this mean we must move past the low-hanging fruit now? Probably... Certainly indicates some usefulness of these models.
How many of those automated fixes were reverted? How many introduced a new bug? What's the false positive rate on the finding agents? The post has counts for everything that went right and nothing for what could go wrong.
AI critique often funnels itself into a narrow bucket: creating code blindly with AI is bad. That's easy to grant.
Adversarial testing, checking developer assumptions, refactor suggestions, small dev tools and even some guided coding all sit on the other side of the spectrum of what you can do with coding and AI. For larger and larger codebases even simple things like tracing dependencies or behavior might be greatly aided.
And the critiques reserved for that narrow bucket on the other end, blindly generating code, are too easily conflated with the rest.
Worrying. Extrapolating a (speculative) future, this means (Google will feel) that soon the chromium base will no longer need the crowd-sourced bug hunting that is open-source. I expect Google to eventually stop working on chromium (in the open) and all current chromium-flavours will become de-facto forks of the last published version of chromium. These forks won't be equally easy to maintain given that the groups running them do not have access to the same level of subsidy as Chrome does with Gemini.
Elephant in the room : how many of these bugs were written by LLMs in the first place ?
Because creating 100x more bugs and fixing 100x more isn't something to be proud of.
Have LLMs shown any decent capabilities in resolving non-security bugs yet? Security bugs are important, but my day to day frustration with tech tends to be caused by UX/functionality bugs. And Google seem to struggle more than most other companies with those.
Google Home became so unbelievably buggy, and I felt so betrayed by that, that I stopped using every single Google product - from Gmail to Android to Google Cloud Platform. The few physical Google devices that I'm yet to replace account for 10% of my tech, but still cause 90% of my problems.
This is a tacit admission it was Mythos, not Gemini, despite alluding to experiments with Gemini earlier on:
> Added support for model interoperability to leverage the unique strengths of both open-weights and proprietary models
A lot of people here seem to be living in a different universe than me or simply don't know how to work with AI. I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you. They get mad at Excel for the poor investment returns. At this point, this is such a strawman, it isn't worth counter arguing.
I think I'll abandon this discussion and keep using AI quietly while exchanging tips with like-minded people who are interested in using it properly and efficiently.
I've recently been using AI a lot for performance optimisation during a particularly busy period at work. I would say it was almost completely useless at the high-level direction - it would point out suspicious parts of SQL queries for example but on back to back testing these almost never resulted in any performance change.
In fact, if it wasn't for the fact that it made making the actual changes I identified much easier (move these joins into a CTE etc) it would have been a detriment. Not only did I get sidetracked by a bunch of useless suggestions but I also had to put up with others dumping their raw AI output at me as if it was somehow a meaningful contribution.
This situation seemed familiar... https://i.ibb.co/RGvQgfmX/i-fixed-more-bugs.png
Did they fix the bug where Chrome tries to track you and your behavior wherever you go and whatever you do in the world?
In my experience LLMs do find a lot of embarrassing bugs as Linus says but it can constantly turn into a game of whack-a-mole where most of the bugs it finds were written in previous LLM sessions. It's a huge struggle to get it to actually fix the root cause of the bug instead of patching the symptom.
Convenient that the LLM vendor is the one saying their own product works, isn’t it?
This is only a flex if AI also didn’t cause an increase in bugs that needed to be fixed.
It's so hard to believe these kind of affirmations coming from a firm that has so many hundreds (thousands ?) of billions invested directly and indirectly in generative AI...
I wrote a pretty insignificant book. So many complaints over the years because there are a lot of mistakes. Not something to be proud of, but I’m the only person working on a pretty large project. Thanks to AI I fixed a large number of mistakes last month.
Title is inaccurate. Fixed more “security bugs” not “bugs”. The article is specifically about security vulnerabilities.
They keep breaking inertial scroll on Mac though, which is something that irritates me thousands of times a day.
First they made it so when it came to rest it would then start scrolling again and overshoot by 2 lines before coming to a halt.
They fixed that after a few weeks then it worked until last week and now it’s left in a state where as it’s coming to a halt it jerks every time like dropping frames of the scroll almost.
I know it’s not my machine, config or trackpad because I’m seeing it on multiple computers.
On a related note: Visual Studio still has dialogs that cannot be resized, even after years of people begging Microsoft to make them resizable. Shouldn't this be a trivial fix for AI?
I'll believe in AI when I get to resize my Configuration Manager!
https://developercommunity.microsoft.com/t/Resize-configurat...
Code related Vulnerability discovery is one of those bright spots where AI can shine because its shaped as a learning-test: every example it runs gives it feedback that is deterministic. If you have ever pointed claude/codex to your a binary executable and asked it to figure out something it can run against full throttle, I fond it always gets back to you with some good insights.
Just admit that Chromium is too complex at this point, we've all seen show HNs doing what could be considered magic in Chromium. We probably need to issue a mortarium in new platform features and focus on a deep clean in bug fixes. I've said that Mozilla is too busy doing redesigns than bug fixes as well.
I assume this was thanks to Project Glasswing + Mythos Preview?
There's no way in hell those are all exploitable bugs. I'll bet my hat that those are "insecure practices".
Nice marketing Google, now tell us how many of the bugs that were fixed were actually introduced by AI usage to begin with?
I have a software degree and some work experience but have been doing other things for a decade or so. I'm curious to try out some "vibe engineering" but it seems a bit daunting to get started, there are so many new tools available on top of the AI itself. Are there good resources for somebody like me to get started, that'll guide me though how to think about prompting, and using CI and how go have the agent successfully write specs and tests and what-not. I don't mind paying some for a good course, but I find it hard to figure out which one would be worthwhile, or whether there are some youtubers that would be better to follow.
Any tips?
What I like is that this is a measurable productivity increase that can be directly linked to AI and they go over their methodology. AI-positive posts skew heavily anecdotal and hyperbolic.
No worries, writing with AIs can improve the speed bugs enter the code base as well. Would be interesting to know the breakdown.
> the “latent security issue.” Code that is safe and robust in isolation can be transformed into a critical vulnerability by an entirely unrelated, minor logic change elsewhere in the tree.
Similarly, you can accidentally fix a bug without discovering/triaging it, so the pointless repeated bug lifecycle is incomplete
The article is great reflection how LLMs are actually doing Quality assurance on both security and codebase sides, but i still doubt the creative part that means UI side, So far in my experience LLMs at best are unable to determine what the good UI is , may be its not inherently a bug detection rather a creative process
The next step is to build software without bugs
When I see this, and I recall that same guys mobile OS been acting crappy software wise (10fold). I find this as a joke.
The privatization and obfuscation of the security industry is worrying.
Google claims to help open source developers, but donates millions to the Alpha Omega Foundation that did not give access to Mythos to Curl and only found one issue.
The Alpha Omega people appear to be selected on physical appearance and take the money away from open source without doing much.
The entire Chrome article is obviously directed at suits and unsurprisingly hypes AI for better promotion chances.
I can believe it. So many small CSS quirks that have bothered me for years have suddenly been fixed, meaning I can remove my work-arounds.
I recently switched from Chrome to Edge after using Chrome for decades. It is so bloated and slow. I can have same number of tabs open in both and Edge is always 30% to 50% more efficient.
spent a several months working on their bug bounty program earlier this year while working full time and lost a lot of motivation after a duplicate combined with lower bounty awards and claude safety guards all at the same time
Good. Now please fix youtube. Amount of bugs that I'm getting in youtube app on iphone should be embarassing for company like Google. I cannot believe how bad it become in last couple years.
Okay, now how many bugs would have been fixed if Google had spent $130B (Google's AI capex) in human resources instead of AI?
Just to have a sense of proportion, considering an average software engineer salary of $200k, that same money would buy you 650,000 engineer-year salaries. Of course there are other expenses, but that gives you an idea of the order of magnitude.
The current title says "Google fixed more Chrome bugs in June than over the past two years, thanks to AI."
This is wrong on almost every count.
1. The numbers are by MILESTONE not month.
2. The latest milestone did NOT exceed the previous two years of milestones. (However, the latest few combined did.)
3. The only thing discussed are SECURITY bugs, not bugs in general.
What is true: the article is about Chrome.
Cool. How many did they add?
> Google fixed more Chrome bugs in June than over the past two years, thanks to AI
And introduced how many new bugs? thanks to AI ...
Lies, obviously. AI is worthless.
Did they fix spell check? Because it still doesn't work and all the solutions are basically "wipe and start over".
I can't comment on Chrome bugs, but I can comment on Safari bugs. According to a search of the WebKit Bugzilla, I'm the most prolific filer of web inspector bugs outside of the WebKit team itself.
Recently I noticed an unusually high number of web inspector bugs fixed by one longtime WebKit engineer, and I suspected AI assistance, though the commit messages include no disclosure of this. Nonetheless, the increase in volume was quite dramatic.
The other day, an update to Safari Technology Preview was released, and I attempted to verify whether a couple of my bugs were indeed fixed. However, it was impossible for me to verify, because the latest version of Safari Technology Preview introduced a new web inspector bug that totally broke the features I was testing.
Thus, I remain unimpressed.
Without a doubt, LLMs have demonstrable skills and can produce code much faster than humans. I never thought that producing code fast was wise, though, even before LLMs arrived. For many years I've criticized software development based on management-driven release schedules, where developers are forced to pump out code regardless of quality, regardless of whether it's ready. Your boss may not like it, but code is done when it's done, not when your boss says it has to be done. If software complexity were predictable and reproducible, then indeed management could replace engineers with automation, but that's not how it works in reality.
Back to Chrome, the browser is notorious for continually adding invasive new "features" to the web that nobody wants except advertisers. I wonder how many Chrome bugs were the result of Google push push pushing all of this new crap on users over the years?
So it uses less RAM now, right? ... right??
I’d rephrase this to something like: “People fixed more Chrome bugs (…), thanks to people who leveraged AI tools”
Interesting to see how many new bugs were created from all these changes.
with excellent test cases, AI can do wonders, human's task in AI world is to just write as many tests cases as we can.
To me this merely signals how broken C++ development really is. Most if not all of the bugs being uncovered are memory related and therefore intimately tied to the mental memory model of C and C++, namely manual memory management.
It's fine for a C or C++ program encompassing a couple hundred lines but beyond that it's a liability.
C and C++ are simply not fit for purpose when large scale software projects are concerned. All of these need to be ported to Rust or another memory-safe language ASAP to prevent mayhem.
The hundreds if not thousands of developers working on Chrome weren't idiots who didn't know what they're doing. The complexity of programming in C/C++ is simply beyond most intelligent individuals' ability to get perfect all the time.