alt Hacker News> With all due respect to folks working on web and phone apps, I keep getting the feeling that AI is great for high level, routine sorts of problems and still mostly useless for systems programming.
As someone working on routine problems in mainstream languages where training data is abundant, LLMs are not even great for that. Sure, they can output a bunch of code really quickly that on the surface appears correct, but on closer inspection it often uses nonexistent APIs, the logic is subtly wrong or convoluted for no reason, it does things you didn't tell it to do or ignores things you did, it has security issues and other difficult to spot bugs, and so on.
The experience is pretty much what you summed up. I've also used Claude 3.5 the most, though all other SOTA model have the same issues.
From there, you can go into the loop of copy/pasting errors to the LLM or describing the issues you did see in the hopes that subsequent iterations will fix them, but this often results in more and different issues, and it's usually a complete waste of time.
You can also go in and fix the issues yourself, but if you're working with an unfamiliar API in an unfamiliar domain, then you still have to do the traditional task of reading the documentation and web searching, which defeats the purpose of using an LLM to begin with.
To be clear: I don't think LLMs are a useless technology. I've found them helpful at debugging specific issues, and implementing small and specific functionality (i.e. as a glorified autocomplete). But any attempts of implementing large chunks of functionality, having them follow specifications, etc., have resulted in much more time and effort spent on my part than if I had done the work the traditional way.
The idea of "vibe coding" seems completely unrealistic to me. I suspect that all developers doing this are not even checking whether the code does what they want to, let alone reviewing the code for any issues. As long as it compiles they consider it a success. Which is an insane way of working that will lead to a flood of buggy and incomplete applications, increasing the dissatisfaction of end users in our industry, and possibly causing larger effects not unlike the video game crash of 1983 or the dot-com bubble.
Replies
> You can also go in and fix the issues yourself, but if you're working with an unfamiliar API in an unfamiliar domain, then you still have to do the traditional task of reading the documentation and web searching, which defeats the purpose of using an LLM to begin with.
Oh, see, this is where I disagree. I think it's incredibly helpful to get past the "blank page". Yes, I do usually end up going and reading docs, but I also have a much better sense of what I'm looking for in the docs and can use them more effectively.
I feel like this is the same pattern with every new tool. Google didn't replace reference books, but it helped me discover the right ones to read much more easily. Similarly, LLM based tools are not replacing reference texts, but they're making it easier for me to spin up on new things; by the time I start reading the docs now, I'm usually past the point of needing to read the intro.
> The idea of "vibe coding" seems completely unrealistic to me.
That's what happens to "AI art" too. Anyone as a non-artist can create images in seconds, and they will look kind of valid or even good to them, much like those "vibe coded" things look to CEOs.
AI is great at generating crap really fast and efficiently. Not so good at generating stuff that anyone actually needs and which must actually work. But we're also discovering that a lot of what we consume can be crap and be acceptable. An endless stream of generated synthwave in the background while I work is pretty decent. People wanting to decorate their podcasts or tiktoks with something that nobody is going to pay attention to, AI art can do that.
For vibe coding, right now it seems that prototyping and functional mockups seems to be quite a viable use.