Im quite sceptical of articles making claims, as opposed to data that explains the productivity over a large enough scope.
OUTSIDE OF TECH: From what I am personally seeing, there are more incremental improvements in productivity than step changes in firms.
For the degree of hype, we would expect more step changes.
For the actual capability of these tools, it should be more incremental changes.
The greatest determinant of a step change in productivity is the presence of 3 things, typically in one person.
1) Domain expertise
2) project expertise/knowledge
3) familiarty with AI tooling
All of which are useful because this allows the operator to identify when a session is going of the rails quickly, discard the output, and switch to a better prompt.
Otherwise, the productivity gains are incremental, and arise out of capability expansion in doing early career work. i.e., AI removes the communication friction of reaching out to another firm or team.
There are going to be MANY counter examples and anecdotes on how this is not true. Good counter examples are people using LLMs to assist with legal issues or handle translation. This is usually within the boundaries of early stage work, which the operator has enough experience and common sense to understand.
Another boundary is review - people are getting spammed from inside now, because the cost/ease of generation has gone down, while verification capacity has not kept pace. The ratio of availble V/G vs required V/G for a process depends on how mission critical that process is. Most firms are stuck in a hype driven cycle to just produce more, and review is not a valued part of the process.
I expect that this is going to change as soon as people start asking who is responsible and accountable for things going wrong. Velocity is great as long as you don't break someone else's things.
I am going to regret writing this.