For me it feels a lot like generated images or video. I've made lots of things now, but those that are 100% LLM written "work" but are uncanny, weird and the details are wrong everywhere you care to look in detail.
I think how I'd put it is like, it's a deterministic common denominator thing iterated on, but it's not really intuitive, it's not really "what makes sense" but rather "what would x look like", similar to how you point out it is when it comes to image generation
it's useful for scaffolding but after that I'm not sure how you could rely on it without being in the loop and directing how the code should be like
This has been my experience has well. For code generation, you have to really constrain these LLMs on your coding style, design goals, test cases, and overall expectations. I mainly use these tools to help my understanding of the code and to generate code for very specific problems. Even after all that setup and careful review I'd say it's still a net big speed up for certain software engineering tasks.
Jason Turner gave an excellent talk at last year's CppCon explain how he thinks tools can be used to make generative AI coding assistance safer and more productive. https://www.youtube.com/watch?v=xCuRUjxT5L8