> Improving the system over patching the output
This really connect with the latest conversations I had with coworkers regarding how can we review gigantic PR generated by AI. One way could be to ask the submitter to give their prompt as part of their contribution. This way it's easier for the reviewer to understand what was the thought process of the contributor and if this very process is sound. Not saying that this way to review is a silver bullet, but at least it removes some weight.
> One way could be to ask the submitter to give their prompt as part of their contribution
Zed's delta (https://delta.dev/) seems to be solving this problem by extending the version control to the conversation layer. I don't know if I'd want all my conversation to be tracked, some stuff belongs to the draft sheet.
Look into [openspec](https://openspec.dev). My team is routinely passing 1-2-5k line PRs through review, with the following caveats:
1. PRs come with "the prompt". In openspec parlance this is a set of docs: a proposal, design, specs, task list.
2. I have a tiny skill to slice and dice PRs, so I can review each part of the system separately by rewriting commits in order of reviewal: specs/docs, production code, tests, evidence. I sometimes pull DB queries to a separate commit for focused review.
3. The evidence bit is built by another skill: go over all added spec scenarios and produce a markdown document with links to the code implementing the scenario and the tests proving the implementation works. Emacs helps me jump back and forth via those links and read the parts I'm interested in.
My next goal is to extend openspec schemas with custom workflow steps to make PRs land in my preferred review structure and contain the evidence report, so I don't have to build it.
Do most people generate PRs from a one-shot prompt? I have a whole pipeline of prompts, tests and evals, which loop back and potentially restart if the result is inadequate. How would I include that in the PR?
How does reviewing the long discussion the author had with the LLM prove the code is correct?
If a PR is gigantic it means the team lead screwed up slicing the work.
Just ban large prs. They can always be split up, and usually as well as helping review it makes them less risky and easier to find bugs and roll back if there was a problem.
The discipline of splitting them up helps in almost every dimension.