This is almost the same as saying "The problem is not the fentanyl, but that everyone keeps getting addicted or dying." In a very literal sense, yes, absolutely. There is a way to safely use fentanyl. But from a holistic point of view, the way to do so is so specific (ie multiple medical professionals prescribing and possibly even administering and monitoring vitals afterwards) and limited that for the general public is much more useful to just say "It's unsafe to use fentanyl unless following instructions from your medical provider." There's absolutely a way to use AI for coding that doesn't result in not actually knowing about the produced code, but it's so damn hard to get that right and you won't even realize you've fallen away from that knowledge until it's too late.

The point about PMs is I think illustrative of this issue. Yes, great PMs who actually understand the product they want to build and can then turn that into something using engineering teams as a black box system exist. But by and large, no they don't. A massive part of why waterfall fell out of fashion is because it turns out most orgs don't have and can't find PMs/PM adjacent people who can build that knowledge, and you can see how agile attempts to correct for this by turning a pipeline into an OODA loop. If you can prevent pipeline flushes, it will beat that OODA loop, but it turns out as an industry we can't actually prevent enough pipeline flushes for that to be the case and we can (for the most part) use that OODA loop to eventually iterate into something a paying customer might give us money for.

Part of the issue is that we're treating the AI like it's just another compiler or static code generator. It's not. The reason they're not is because they require a complete specification of what to output, ie the code. That level of specificity is what allows engineers to mostly stop caring about the assembler and just stay in their higher level code (the main exception being extremely hot loops in places too complex for the compiler to optimize well) and legitimately be able to say they understand the code without ever looking at the compiler/linker output. The entire point of AI for code generation is taking an incomplete specification and turning it into a completed one. If you review and understand every line of output the way almost no one did even before AI was a thing, then you're not going to fall into the trap of not knowing anything about your system anymore. But at the same time, you're not really going to get much benefit from AI either, because you're just replacing time spent coding with time spent reviewing code. It might even take you longer to review than it would to just code it yourself (i think every senior engineer who's mentored a junior dev knows this exact feeling).

To actually get the benefit from AI you have to let go of looking at the code output at all, because looking at that output is the slow part and because just looking and reviewing still won't give you the same knowledge as actually implementing. The passenger aviation industry handles the second issue by preferring/requiring manual control in critical areas (takeoffs and landing) and using simulators to extensively train for manual recovery outside of the abilities of autopilot. For developers the second bit is actually why reviewing AI output is a fool's errand - it will give you the false confidence in your understanding of the system. That doesn't mean we have to embrace vibe-coding or accept slop. It means changing the engineering process from one that deals with a deterministic system to a stochastic one. As an aside, I think this is why upper level managers and executive are leaning into AI code generation so hard - engineering was already a stochastic black box system to them.