alt Hacker NewsAt Plotly we are dealing with similar issues. We're doing our best to keep up, but it's genuinely hard not to just cut our losses on passer-by community contributions. 3 page issue writeups nobody has proofread, nobody but a bot to follow up with when we try, huge volume of plausible but weak PRs eating up reviewer time. All while we have core work we really want to be focused on rather than merging another bug fix we don't expect anyone but this bot has really run into.
Replies
We disabled PRs for PgDog, but left the issues open, for now.
Most bugs that are reported via AI-generated GitHub issues are edge cases, which are technically bugs, but not something anyone actually ran into.
Contributions are exclusively AI-generated, and it's much easier to just go off of a GitHub issue myself than asking the bot behind the GH account to remove the 5 redundant tests it added, or to refactor what has become a 3k line file.
AI doesn't care about code quality, maintenability, or long term goals. It's just a local maximum optimized with RL. Great for debugging, OK for reviews, but can't produce anything that's above average.
Yeah I sympathize with this, but I’ve experienced the other side of this: I ran into a bug (an integration for home assistant has to be manually kicked every time HA restarts due to a bug in the integration) that my AI fixed with a one-liner. It works great for me. I submitted a PR which is languishing for weeks. No problem. My AI will just maintain a personal fork with my patch until my PR gets merged (or not). Previously it was way too much work maintaining your own fork, merging in upstream changes, and keeping your patch working.
I think this is going to be a common outcome from the fact that many people have AI agents that can one-shot fix and deploy. Maybe it’s not a bad thing for those people and the maintainers who each get what they want.
Low quality PRs, along sometimes with concerns of providence that might cause future licensing issues, were a problem before the Eternal LLM September made it easy for far less capable (or caring) people to submit them than before. This is at least part of why a number of projects switched to being “open source but not open contribution”.
Possible idea: tag them with AI, and leave them aside for 6 months and review them with 2027 AI
I’m a little perplexed though. The problem as stated is there’s too many drive-by contributions with AI slop. Isn’t the solution to define a contributor network (eg a GitHub org that you can apply to) based on trust?
The other option they proposed was a proof-of-humanity by payment, which seems much more in line with the above.
The problem with just banning AI is you didn’t solve trust, and your untrusted users are just going to drop drive-by PRs anyway. How do you know if an untrusted user did or did not use AI? In practice this likely means just aggressively closing “slop-looking” PRs from users you don’t recognize?
But then, you’re preventing your trusted community members from benefiting from AI assistance.
I’ve been advocating for defining a higher standard for AI contributors, eg an AGENTS.md that specifies some meaningful correctness bar. In this case, perhaps the PR that adds you to CONTRIBUTORS.md could to add a large novel and non-trivial end-to-end test; if you can construct problems that are easy to verify but hard to get right, then this is your “proof of work” to establish competence.
From reading the GH issue it does seem this is a placeholder solution, so I hope they find a more stable long-term resolution.
The linux kernel team set something up that they are pretty happy with. I don't know the details, but they have dealt with the influx and feel comfortable their solution will last.
https://www.youtube.com/watch?v=NnV_cWeoo5Q
One thing I've been thinking is something like the DCO checks. Set up some account bridge between (eg.) Discord and GitHub, then auto close anything submitted by accounts not sufficiently involved with a message linking to the new contributor policy and rationale.
This has basically been the story for us at Hatchet too, and it’s been similarly challenging. A lot of the OSS PR reviews I’ve done lately from non-core contributors are basically just me going back and forth with Claude via someone else, which is frustrating when I feel like I could 1) just not or 2) do the back and forth with Claude myself and cut out the middle man.
I have a sneaking suspicion that people are having some agent check out 500 repos and just prompting with “find a bug in this repo and fix it” and submitting the results blindly.
Edit: typos, on mobile