alt Hacker NewsMaybe I'm missing something obvious, but if I was going to have my team use this, I'd want someone to answer the following question
If AI is good enough to explain what the change is and call out what to focus on in the review, then why isn't AI good enough to just do the review itself?
I understand that the goal of this is to ensure there's still a human in the review cycle, but the problem I see is that suggestions will quickly turn into todo lists. Devs will read the summary, look at the what to review section, and stop reviewing code outside of things called out in the what to focus on section. If that's true, it means customers need to be able to trust that the AI has enough context to generate accurate summaries and suggestions. If the AI is able to generate accurate summaries and suggestions, then why can't we trust it to just do the review itself?
I'm not saying that to shit on the product, because I do get the logic behind it, but I think that's a question you should have a prepared answer for since I feel like I can't be the only one thinking that.
Replies
No worries at all, that's a very fair point and a question we've gotten a lot!
I think our perspective is that: software design has always had a subjective element to it. There's never been a "right" way to design a system, there are always trade offs that have to be made that depend on things like business context etc.
To that extent, most engineers probably still want to be part of that decision making process and not just let agents make all the high level decisions, especially if they're responsible for the code that ultimately gets merged
One thing that comes to mind is that an AI might see the code and say "Yeah, this should compile / no obvious runtime errors", but the AI doesn't have the context to know your teams coding standards (every team has different standards). That said, there are ways to feed that context to the AI, but still risk hallucinations, etc.
I mean, that's likely where it's going.
Most of human review I see of AI code is rubber stamping at this point, the volume is too big for human to keep up. What used to take Developers a few days to do is taking a few hours so PR volume is higher and human reviewing can't keep up. At this point, human review seems like CYA then anything else, "Why yes SOC2 auditor, we review all PRs."
I'm also seeing a lot more outages as well but management is bouncing around all happy about feature velocity they are shipping so :shrug:
Imo human review is important for context/knowledge sharing even if a machine or tool can mechanically determine the change is reasonable