Hell, on top of that (i.e. the confounders you mention are very very apparent everywhere,) the challenge of folks understanding how to use AI to be more productive.

Even last week I had a case where, I was reviewing code that I know the dev was vibe coding, and honestly was good up to that review.

Buuuuuut, for the stuff I reviewed that day, there was some base-class inheritance use that was so ugly that my initial teams message to the dev was along the lines of 'What crack was Copilot smoking on that one'.

Turns out, they had asked some other random-ish questions in the chat they were doing their actual work in, and those questions more or less polluted the context.

Sure, I was able to guide them for how to not do that in future (i.e. keep unrelated questions around the work in question to a separate chat, etc.) but this was also a dev that is normally both very good, but also overworked semi-related to the general 'AI will save us on dev costs thing'.

I can't help but ask whether, for the even less-technical people that are told 'yeah you lost two colleagues but here is Copilot/Claude/etc' and they don't even get enough advice to have it actually increase productivity versus chasing down gremlins in the actual work.