alt Hacker NewsI did a test a few months ago. A friend of mine wanted to develop what i understood to be a simple single page web app. But since she didn’t have any software engineering experience she asked me to help. Around that time everyone was talking about how literally anyone can develop software with LLMs i asked her if she could give it a try first, and if I could watch the attempt.
I was fully expecting that writing the code will pose no problem for the AI. But i was curious if the AI will realise that my friend is a novice and needs extra help with things like: copy pasting the code into a text file and saving it with an html extension, helping her host the file online so she can share it with others, buying a domain for it, etc. I assumed they will get there eventually, but i also assumed that it will take a lot of stumbling around and misunderstandings.
But i was completely wrong. They didn’t even get to that point. Because my friend didn’t have the vocabulary to ask the AI to write code. They were just going around in circles where the AI was brainstorming with her about possible features and getting thints more and more complicated. We terminated the experiment after one and a half hours and many many messages exchanged between her and the LLM.
Whereas to me who knows the terminology would have probably just taken a single exchange of messages to get the result she described to me. I would have prompted with something like “Please write to me an html page which does X, Y, Z.” But since she didn’t know the right terminology she got into a vortex of feature discusion, and she didn’t find a way to tip the AI into “just do it, write it now” mode. In other words in that case the LLM would have rewarded even just a little bit of expertise, but without it there was a confusion about goals between the human and the machine.
Replies
Interesting, but... wrong tool, wrong job. And by tool, I mean web based chat interface, not LLMs in general. (Maybe wrong delivery mechanism, if you like.)
Your friend needed an agent, not a chatbot. I use Claude within VS Code (as per many others) but I certainly wouldn't recommend that for a beginner. They needed a tool that's specifically aimed at people who want to build software but don't know the first thing about how to do it. I think there are a bunch of these now but the one I'm most aware of is Lovable, and I'm pretty surprised you didn't recommend one of these.
An HR person I know was searching for a way get something build, found Lovable, and managed to build a somewhat functional application with it on their first attempt within an hour or two. It was full of holes and far from perfect but they got something working - at least the outline of a potential solution.
As I say, you should have recommended your friend to try building with a tool like that: a tool that they're a member of the target market for. They would have got a lot further. I'm not vouching for the quality of the result, but they would have got something.
Even for experiened engineers, chatbots have always been a pretty grim experience for software development: from the mind-numbing drudgery of endlessly copying and pasting code, commands, and prompts around, to the fact that they just can't see enough of what you're doing to generate the best quality output or advice. You can do software development with a ChatBot but it seriously sucks, and better tools are (a) probably being shoved at you day in, day out via ads, and (b) only a Google search or a ChatGPT recommendation away.
(Obviously, nobody's going to search for "Lovable" without knowing about Lovable, but they might search, or ask ChatGPT or whatever, something like, "How would I build a website without knowing anything about building websites?", which might get them an advert or recommendation.)
>she didn’t find a way to tip the AI into “just do it, write it now” mode
worse, the longer an LLM conversation goes on, but especially with constricted/free models (yes the simple chat interface they are likely using) the harder it is to get an LLM into this mode even *IF* you know the right words to say
at that point the best way forward is to terminate the exchange entirely, and to start off with the right initial message, instantly getting into coding mode. a non technical person will not know this and be stuck in feature theory crafting mode in perpetuity, or worse in an endless "excuses' mode as the LLM diverts ant attempt at coding into reasons why its not going to: "i wont output incomplete/broken code! that would require too many lines of code sorry i wont do it! i wont be able to get it perfect so i wont attempt it! but heres more features and theory crafting"
will a non technical person know to end the conversation and start fresh? not likely unless they have a lot of experience already with LLMs
You can see the same thing with restaurants generating their own menus/pictures. Some of them look absolutely terrible, visually ugly, way too information dense, the classic piss filter, etc. idk how they do it, even the most basic prompt I can come up with makes something 10x better, and when I put in my amateur photography knowledge/keywords in it gets pretty close to what I'd consider a good pre LLM quality menu. Literally just adding "make it look nicer/cleaner" seems to get rid of most problems, but people just don't care apparently.
I think this is about teaching problem solving at a young age and it is an abomination that our education system does so poorly at. The key is to know what kind of questions to ask and knowing when to go deeper and what to pay attention to.
But that is not how our education system aligns us. One typical example of problem solving kids, and I too, learn in school is how to apply a concept in physics to a free-body-diagram(Indian and Chinese education cram schools are famously good at teaching kids how to do this, the usefulness of which I debate). But it all stops at the exam room. No architect or jr structural engineer position for you kiddo.
Another kind of problem solving skill might be how to invest money and understand your own risk appetite to construct portfolios to manage your money. All that is taught in school is a dry compound interest formulae, time discounted cash flows and a black scholes model. Only to find out later I don't need most of it to manage my money.
> They didn’t even get to that point. Because my friend didn’t have the vocabulary to ask the AI to write code.
What harness did you use?
In e.g. claude, there are two modes:
1. Spit out code 2. Draft a plan, ask questions, GOTO 1
You literally have to go out of your way to get it NOT to write code. I keep mine on a tight-ish leash because it modify code way too happily even when there's no intention or instruction to do so
I’d love to see this experiment executed with Claude design.
Particularly with something static, I don’t think they’d fail to get a result.
But without domain knowledge I think they’d misunderstand prototype with finished product.
Without knowing what it’s doing, it’s hard to know what it’s not doing.
My experience is similar, people succeed with Loveable or similar platforms because they don't need to know _deployment_ and _runtime_ experience. It's like using Adobe vs Canva, most people are now used to the latter and don't even have a mental model of "runs on a server" or "database and server are two different things". This is the key difference for me compared to earlier Low Code solutions such as OutSystems that still required a SDLC mental model. However, at some point this breaks, like in your example of feature discussions, because people struggle to test out new ideas _without_ immediately showing them. So it becomes Canva vs Figma, a system-based model. And we don't even have good terminology for that ourselves yet.
Are you describing a "chat window" experience here? This is apples to oranges.
I find there's a lot of variation between LLMs / models...
But there's also some psychology in play too; that we (engineers) see a lot: Some people just let their imagination run away and forget to "do"; without someone in the conversation pushing for results and action, the conversation will just stay within imagination and everyone will be happy in the moment but nothing will get done.
We've seen plenty of examples of apps successfully vibe coded by non technical people, including apps making real revenue.
Your friend could start with telling the LLM that they are a non technical person who wants to make an app and it will explain all the successive steps.
I'm wondering: What did she tell you that made you build the website? Did she ask the same thing from the AI? Did you bridge any gaps the AI didn't do for you? I don't know the answer, but I suspect that she treated the AI differently and would've gotten better results if she had asked what she told you. At least if it's an agentic coding assistant like Claude Code or GitHub Copilot. Of course a simple chat will leave manual tasks for you.
Every now and then I'll write a single prompt to Claude to see how much of a web app I can build in one shot. I usually pick Django (though I assume any other web framework could suffice) because of the battery included nature, I still sometimes describe that I want users to login / register, have access to x, y, z
I also noticed a friend of mine had way more success by having Claude write code by doing TDD and giving Claude scenarios for things the code should be able to handle, if you do this correctly, and cover edge cases, Claude will work with these in mind.
OTOH my wife's friends got drunk and made "tinder for horse purchases". They prompted to read typical horse advertisements (we're all horse people) and create an app with mock tinder like entries to swipe right and left to buy horses.
A web app was produced with lots of mock "Hi i'm Dominique and i love running through fields and having a bucking good time" type entries complete with silly horse photos. A huge amount of drunken fun even if it boiled a towns water supply and blew through half a subscription to create.
I was looking at the results as a dev with 30 years experience and thinking fuck me. The little apps i made here and there before AI are being outdone by a bunch of drunk people on a whim!
This sort of thing has been studied in academic experiments. Although the models studied are now “old”, and I expect the floor is higher, the lack of vocabulary and basic concept familiarity sets the ceiling.
I have counter evidence of this. My wife with absolutely zero skills in computers was at the terminal doing things that Claude was telling her to do and generated some impressive tools. One was a tool to help her organize her day. It involved scripts, PDF generation, printing. She did it all without even asking me. Maybe it fails in some cases but I'm not sure the anecdote above is the average experience.
I have the same experience giving my brother an OpenClaw as his personal assistant.
His words were: It feels as if I need to know how to program it.
I was expecting he could say something like "Oh, it seems like you don't remember the people I'm referring to, perhaps you need some kind of CRM system. Can you investigate if there are any easily available CRM systems you can interface with, so we don't need to make one for you?"
Whereas my OpenClaw moment was trying to make it manage its own NixOS installation, so that if I ask it to do something, it doesn't yolo `apt install` commands, but rather improves on the same overview of its own installation.
A lot of people had success making their OpenClaw do things without being Linux experts. But you need a tinkerer's mindset, is what I came to conclude.
Having a similar experience. Watching a muggle try to build a website with basic functionality, e.g. auth, database, etc. is an enlightening experience. They simply do not have the vocabulary to guide the LLM. My good friend calls me every night frustrated with his results, and when I watch what he is doing it's amazing what we take for granted being in the software industry. Don't get me started on the UX, that's even more mind boggling.
Do your friend at least know what Claude Code (or any harness) is?
Of course she wouldn't be able to make a website if she doesn't even know the right tools to use. But I don't think it prove anything. Knowing and installing Claude Code might not be a common sense, but nor is it "expertise" or "skill."
I've seen in first hand that people struggle installing Steam. Yes, "people" in the plural. But just because some people struggle with it, it doesn't mean that installing Steam isn't an objectively easy task. Your friend's experience doesn't change the fact that building a website is something that an average person can do in hours if not minutes.
did she try saying "I want to make a webpage"? Even ChatGPT will just build, deploy and host a webpage for you with that request. I don't really understand what system she must have been using.
I've seen some designers and product managers get pretty far with LLMs but mostly because they already know how to build apps just from a non-technical perspective.
I have been pondering this and I think it's likely a gap that will get filled sooner or later.
Right now there's just so much value in building LLM tools for experts that everyone is focusing on that. But surely at some point we'll have bespoke harnesses that exist exactly to solve this kind of thing.
I think this can start with constrained problem spaces like "you are a WordPress developer, you solve problems for people with enough expertise to know they are looking for a WordPress developer" and incrementally expand from there. Maybe I'm naive but I think you can probably get pretty far with this today just by writing loads of skills and picking the right technical preferences to encode in them.
Reminds me of watching someone who has no idea how to use a search engine try to use a search engine
I am reminded of this one:
https://thedailywtf.com/articles/Could-You-Explain-Programmi...
Weird because I had a friend that wanted a web app, also no experience, and he just told Claude (on the iOS app) to make it for him and Claude just made an artifact and put everything there.
Buddy sent me a share link and he was just like “dude this is crazy I just told it what I wanted and it just spit it out in a few minutes”. No “experiment” needed he just did it because he knew Claude could do it and it worked exactly like you’d expect. Hell, he did it on the Free tier.
I just gave this a quick try with Sonnet 5 Medium [0] on a free Anthropic account. It's a bit of a contrived example I guess, but it probably isn't too far off what a completely non-technical person seeing this for the first time would do.
The output is EXTREMELY misleading, as all the data here lives purely locally, yet the AI says that you can "just share the link" and other people will see the schedule you set on the generated artifact. Also, what link? To the Claude chat? It doesn't explain what to do with that `Booking` artifact other than "link to it".
I can so easily see someone tapping out a few steps down the line of this once they realize it doesn't work and they have no clue what to do or say to make it work. What do you even ask as a non-technical person at this point? I guess they could explain "The other person doesn't see it", but would the AI actually clarify that it's because it's not fucking hosted anywhere and has no mechanism of persisting the data outside of the current machine, or would it - as I'm almost 100% sure would be the case - not actually point out this error?
[0] https://claude.ai/share/0cbfe698-3886-4d4d-86e4-7c697b61dc00
I've seen the same thing happen to my brother when he tried making an app with zero experience. Only difference was that he got a front end that didn't work.
same thing as people just entering a question prompt and copying and pasting the response as gospel. e.g. politicians using it to write speeches, or lawyers for testimonials.
the output is programmed to look correct so unless you have some sort of background you won't actually know what errors to look for.
...didn't have the vocabulary...
That is exactly the key or the sign there. Even with a couple of decades of engineering expertise, when I try to do / research something that I don't know enough about, I find myself in the exact position of not having the vocabulary.
To the point that I sometimes have to ask the AI "nicely" to cut the pleasantries and be ruthless against nonsense, whether from its/their side or from mine.
Interesting, thanks for sharing.
> Whereas to me who knows the terminology would have probably just taken a single exchange of messages to get the result she described to me. I would have prompted with something like “Please write to me an html page which does X, Y, Z.” But since she didn’t know the right terminology she got into a vortex of feature discusion, and she didn’t find a way to tip the AI into “just do it, write it now” mode.
Even as a developer, when I've been using the web chat interface for things which I know the AI can do easily, I've had this happen to me a few times. I was very surprised the first time I saw ChatGPT respond ~"this would be a few thousand tokens, I can't do that".
Even more surprising: ChatGPT was accurate when responding that way this time, despite this being trivial for Claude and well within what ChatGPT could do using the web chat interface 6 months earlier. The ChatGPT output was extremely meh.
Question is if the outcome was LLM related or just a result of a personality trait. After all, talking in circles without really approaching an end-goal is something certain humans do all day long.
What terminology is required to make Claude Code write code?
My experience with friends has been the opposite. A PM friend made a custom tool. A friend who has never written a line of computer code has an app.
If you tell Claude Code "I want a website that does X, Y, and Z" it will write code.
I’ve had the absolute opposite experience with a friend of mine. I started by setting her up with a terminal emulator on the web linked to Claude Code, and written a CLAUDE.md that told it how to deploy. These days (with no further intervention by me) she’s running Claude Code natively on her laptop, and she tagged me yesterday on Facebook in some update about how much more she enjoyed using Claude Code than plain Claude.
copying and pasting code? a few months ago?
2025 called and wants its test back
[dead]
So maybe your friend need a role reversal: A prompt guiding the llm to act as a consultant, guiding people in implementing a software project, asking questions, creating a shared understanding, limiting scope or creating milestones.
That seems like something that could be done using an llm, not that complicated probably.
And maybe, in other fields as well.
I ran a similar test and got completely different results. My girlfriend (hair stylist/artist) with zero coding background mentioned a Telegram bot idea. I asked "Why not build it yourself?" I gave her a Windows laptop, but she said she wanted what I have instead. So I handed her a USB stick and told her she was on her own now. Fast forward: she now runs Arch Linux with Hyprland (I use Xorg/i3 though), fully riced with cats. I only interfered on partitioning to preserve my data. She even installed Steam and got Portal running (that was her "watch this" flex after I told her not to even try). She pulled all of this off using a free-tier Gemini chat.
For the bot, a friend gave her a Kimi 2.7 key. She set up their harness and built a working bot in a matter of days. She even got a free Oracle VPS for deployment, though I stopped her there to check security first (still haven't had time, unfortunately). She uses that laptop daily now and says she enjoys it over Windows by a mile.