alt Hacker NewsOne of the most wild parts of the "Is AI useful" arguments is how different people are getting so astoundingly different results from it. One guy will claim that he can one-shot "Port this unmaintained app from 2017 to modern Android," go do the dishes, and come back to a working application. And the next guy will say, "I can't get it to do anything groundbreaking and have to micro-babysit it every step of the way to the point where it is more work." Assuming both people are truthful and operating in good faith, how can this be explained? How can simply writing a prompt in English be a "skill issue?"
I'm personally much more towards the "I struggle to get it to work well" side, but there have been so many anecdotes from people who are earnestly saying it does all their work for them and that they hit enter, go to bed, and wake up to working software. I'm trying to imagine a hammer or band saw that produces such wildly different results in the hands of different users.
Replies
I think the answer to your question is the simple and unfortunate one: groupthink.
So many businesses are pushing AI hard. People with varying levels of job (in)security want to be seen as "leading the charge" into a post-AI era.
LLMs can do some really impressive things, don't get me wrong. But I regularly find massive blunders in the code it produces.
And that actually makes sense if you understand how LLMs operate and the quality of software on GitHub.
My guess is they do something like "Hey dingus, make my app send emails" and then Claude or GPT just goes off and does a thing that makes it look superficially like it sends emails but it really only sends email exactly on Tuesdays and if the subject line is "send email to user". And there's no tests, so who knows if the next LLM can fix the bugs without introducing new ones.
For the last several years (since GPT 4o) I have written several apps in use by people, making money, entirely with AI/LLMs (me neither writing nor reviewing the code in any meaningful fashion - other than high level architecture, schemas, etc.) and - yes, in a few hours it can do things that would take a normal human weeks (if ever!). But left unconstrained, it will just pile more and more garbage on the pile.
Fable MAX is the thing that gets me much closer to "just send emails" but even then it doesn't look in my specs directory for the spec, it just goes off and does a Jurassic Park-style "It's a Ruby-on-Rails system! I know this!" and disregards all the other ways we send email and writes its own thing. And often I'll go "Where's the button to do X" and it will say "You're right! Nobody asked for it, so this page is an orphan!"
I happen to use Superpower's (https://github.com/obra/superpowers) "brainstorm -> spec -> plan" workflow in e.g. Fable MAX (for anything non-trivial) then I have Fable send to GPT 5.6 Sol XHIGH for execution, with Fable (either the original, or a separate one, depending on the criticality of the task and the blast radius) being the critical reviewer.
Even then, though, I need to continually guide it against the norms and conventions of the codebase/app, because it makes a ton of assumptions.
It doesn't surprise me that if people don't take a fairly rigorous approach to AI-software development then they'll end up with a mess. Even if you buy into "just re-write it" (I happen to think that's where we're ending up) - we aren't there yet and without e.g. a strong test suite, re-writing it is just as likely to create more bugs than it is to fix the existing ones.
> Assuming both people are truthful and operating in good faith, how can this be explained?
Quite simply, difference in skill.
The sysadmin that has configured a few applications in their career would likely get annoyed at how many mistakes an LLM makes.
The manager who barely knows their way around a shell, uses an LLM to configure an application in less than the three business days it will take them, and will claim they are 10x more productive now.
Once again, LLMs are effectively great at triggering the Dunning-Kruger effect. The least you know, the more productive you feel.
>"I'm trying to imagine a hammer or band saw that produces such wildly different results in the hands of different users.
r/DiWHY
I'm the GP that got Claude to port the app, set up the SMTP server etc. etc. And I do see stuff on that level of complexity, multiple times a week. I'm not a full-time software dev any more, but did learn the craft at a really good company before AI came along - and I've been doing basic sysadmin stuff by hand, almost exclusively on the terminal, for almost a decade now.
A lot of pithy responses to your comments that contain some pretty bad-faith assumptions about AI "power users", but to answer your question I think part of it genuinely does come down to a "skill issue" - or at least familiarity.
AIs really do have huge blind spots, and there are things it can't do well. But on the flip side, there are modes of operation that do get good results and if you keep it in this "zone" you can get amazing results.
For example, here is my CLAUDE.md that's global to all projects: https://github.com/EspoTek/.claude/blob/master/CLAUDE.md
Note the "Working with unfamiliar data or systems" section - it doesn't stop the models from making wrong assumptions but it does get them to test these assumptions with simple experiments and course-correct before human eyes ever see the result.
Setting up things like CLAUDE.md files and skills so that it doesn't make the same mistakes over and over is a big help too. The model doesn't literally get smarter, but it knows how to avoid pitfalls its fallen into in the past.
If you want to have a chat about it earnestly, my email's in my profile.