In our new world of non-deterministic output (that's why we love LLMs! they say such helpful/agreeable/sometimes wrong stuff!), I think CI won't be sufficient. CI is in the realm of Quality Control; when I build the thing, is it to spec and does it do what I need it to do?

But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.

In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time. For example, if a hotel chatbot starts telling guests checkout time is at 11pm instead of 11am, that's a real operational problem and ideally should be caught before it impacts customers.

LLMs introduce new classes of problems/risks that we are just starting to understand and develop the tools to manage.

Our agents automatically review our PR's - the authors agents automatically see the feedback and make fixes, and automatically merge when everything is green.

A well authored CI review process is significantly better than any human could do. We have the AI review not only the changes but clone and investigate all related repositories that integrate with the code to evaluate interactions, check all open and historical issues that may be related, review production/development logs for the feature, etc.

It will then also go update issues/tasks to reflect the changes after the merge.

Similarly you can have automated agents watching post-deployment to mitigate risk in the "a new feature just went out" time-frame. Just having someone check the related logs in the hours after deployment can catch a lot of issues that otherwise would turn into a fire-drill.

> You wouldn't ask an agent to review a PR then just copy/paste the output int PR would you?

Hasn't everyone already got agents directly adding themselves to PRs and leaving comments (occasionally useful)?

In my experience with the Claude Github integration, I found it to be pretty helpful. It’s had a pretty good success rate of catching bugs before they get to master, and for simple ones I can ask it to fix itself.

> you should have 2+ developers looking at most PRs

It’d be nice, but usually not the case in my experience. More eyes is better. AI review should not replace human review, it should supplement it. I find myself spending more time doing end-user testing instead of looking at code vs before.

We had a two human PR requirement until recently we dropped it. It was slowing us down too much now the human developer creating the future is obviously writing it all with AI so they need to check it then depending on the feature and it’s use it requires a PR but it’s not universal and we’ve stepped up our automated test Tan X what it used to be it’s been so far fewer bugs better delivery

> You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?

I would if the PR was clearly written by AI. I'm fine with the PR author using AI, but only to draft the PR. They should be editing the shit out of it for the final version before submitting it.

We’ve added AI to our auto review process. It does expose when the author is not confident in their solution to push back. Which is interesting. But we do try to target specifically at our patterns and safety.

Human reviews are going away for most PRs these days. If the LLM decides the risk level of the change is low, the PR can be merged without another human in the loop. That's at least the direction many companies are taking. Only require a human judgement when necessary.

Nah, we have AI code review at Google and it is shockingly good at catching bugs no one would have noticed. I absolutely depend on it now.

> You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?

of course not, it's disrespectful to the author even if they used an LLM to generate the code. what you should do is actually try to understand what the LLM is saying about the proposed changes, check whether it's talking shit or legit, and if legit, rephrase in your own words why you think a certain thing should be changed

I would and do.

works perfect

https://github.com/dzmitry-lahoda/dz/tree/main/agents/skills...

burns half of day sub of astra for 200 USD. runs 1 hour on our repo.

finds bugs missed by coderabitai, devin-integration-bot, codex and copilot (I ask them first until nothing found, yet my orchestration finds more).

I do not hardcode our application in orchestration, but run subagent for applicationdomainproduct detection.

also I use agy 200usd sub for second-opinion as one of steps for false positive elimination.

We totally use it in CI. We were thoughtful in how we built our review bot and it catches a lot of issues before humans lay eyes on the code. Typically they are lower level code issues that humans would not have noticed. It's not the comprehensive review solution, which is why we still require some level of human attendance to the code, but it's very much worth using. We do set limits on PR size and description quality as well to mitigate the harsher AI slop issues.

Echoing phrasing from the bef-ai-re times: this is the way

I'm glad I work in places where there's no such silly pointless rules like how many people need to review a PR.

The PR author asks for feedback if it needs feedback, otherwise it merges it, period.

I don't know why and when the world got convinced that all this bureaucracy is a "best practice", when it's just a practice, that can be good, or a waste of time depending on many factors.

Like do doctors when analyzing your medical records are like "I have a diagnosis, but first let me have two colleagues review it"? No, unless they are at the beginning and it's part of the process or they need one.

And that's way more serious than the crap most HN is writing.