alt Hacker NewsSo, a couple years ago Microsoft was the first large, public-facing software organization to make LLM-assisted coding a big part of their production. If LLM's really delivered 10x productivity improvements, as claimed by some, then we should by now be seeing an explosion of productivity out of Microsoft. It's been a couple years, so if it really helps then we should see it by now.
So, either LLM-assisted coding is not delivering the benefits some thought it would, or Microsoft, despite being an early investor in OpenAI, is not using it much internally on things that really matter to them (like Windows). Either way, I'm not impressed.
Replies
The argument I usually hear is that you only truly get the 10x improvements with <new-model> (right now, Opus 4.5), so they've only had a few months, not years. In a few months, it'll turn out that <new-model> wasn't actually capable of that, but <new-new-model> is, and as that's not been out long its unfair to judge so early. And so the cycle begins anew
I fully believe highly skilled people can get a great benefit from LLM tools; probably not 10x; but enough that its noticeable.
The key thing for me is that it only works when the LLM is used for tasks below the devs skill level; It can speed up somebody good, but it also makes the output of low-skill devs much harder to deal with. The issues are more subtle, the volume is greater, and there is no human reasoning chain to follow when debugging.
So you combine that with a company that has staff in low skill regions, and uses outsourcing, and while there might be some high skill teams that got a speed up, the org is structured in a way that its irrelevant.
They weren't great before LLMs either.
Also, it seems from the outside like a dysfunctional organisation, or at least with incentives heavily misaligned with their users. Replace LLMs with a bunch of 10x engineers and it will still be bad in an environment like this.
So not sure how much to blame the LLMs - or in fact how much MS is really using them. Poor souls have to use MS AI tools, I almost feel sorry for them.
Oh it did help.
Microsoft went all in on do more with less and fired/reorged significant part of the company.
Wouldn’t be surprised if the outage is caused by new team taking something over with near zero documentation while all the tribal knowledge was torched away
Or LLMs weren’t good enough yet years ago, but the growth curve looked so promising that an investment seemed a good idea.
Also: do you have a reference for “a couple years ago Microsoft [made] LLM-assisted coding a big part of their production”?
I know they started investing, mentioning future benefits, but don’t remember them saying their Windows development team (heavily) relying on it.
I think it's naive to believe AI is used primarily for productivity boost. It's used mainly for cost reduction and to increase profits, even if quality and productivity take a hit in the process.
exactly my thoughts as well - if LLM really were massive productive booms - then we would see the number of bugs in major software platforms going down - we would see more features - but neither is happening
so yeah we're being sold a bag of air
Everyone should read "The Bear Case for AI" thread:
"The bear case for AI is that bringing 10x or 100x or 1000x more intelligence to America will not change anything because U.S. institutions are already designed to ignore or waste intelligence and have no idea what to do with any more of it."
Microsoft is not even using dotnet core and what not, internally. SLT is very hard on adopting AI, but not much on getting results
If they used copilot and it was years ago, I'm actually impressed there are no reports of Windows PC's exploding
this reasoning is flawed.
wouldn't a for-profit company just balance the workforce for the productivity gained to increase overall profit?
some person is 10x 'more productive' (whatever that means) , let's cut 9 jobs.
Although to your grander point, employment during the LLM-embrace period seems fairly stable.[0]
(1) a couple of years ago, LLMs for coding sucked pretty bad.
(2) LLMs are a force multiplier. If you start with a negative number, then your coefficient makes things worse.
(3) Microsoft has never been a place of quality. It's not organized for that, it doesn't have that as its philosophy, and so you should never be surprised that it doesn't deliver that.
If anything, we see a decrease, not an increase.
Imagine a world where Microsoft was pushing “Copilot” integration everywhere, just as they are in this one—but the proof was, actually, in the pudding. Windows was categorically improving, without regression, with each subsequent update. Long-standing frustrations with the operating system experience were gradually being ironed out. Parts of the system that were slow, frustrating, convoluted, or all three, were being thoughtfully redesigned without breaking backwards compatibility, and we were watching this all unfold in real time, in awe of the power of “AI”, eyes wide with hope for the future of software, and computing in general.
Think of how dramatically this hypothetical alternate reality differs from the one we live in, and then consider just how galling it is that these people have the nerve to piss on our leg and then tell us it's raining. Things are not getting better. This supposedly-magical new technology isn't observably improving things where it matters most—rather, it's demonstrably hastening the decline of the baseline day-to-day software that we depend upon.
[dead]
> If LLM's really delivered 10x productivity improvements, as claimed by some, then we should by now be seeing an explosion of productivity out of Microsoft. It's been a couple years, so if it really helps then we should see it by now.
That productivity may not be visible. I think MS's move-everything-to-rust initiate would be one hell of an endorsement if they manage to make visible progress on that in the next couple of years.
I know blaming everything on LLMs is in vogue right now; but this is much more to do with Microsoft very publically firing the QA department[0][1] as a cost savings measure and claiming developers will do their own QA (long before LLMs were on the scene). It started in 2014 and the trickle never stopped.
Microsoft has a cultural problem; it went from an "engineers" company to an MBA directed one, trying to maximize short-term shareholder value at the cost of long-term company reputation/growth. It is very common and typical of US Corporate culture today, and catastrophic in the long-run.
[0] https://arstechnica.com/information-technology/2014/08/how-m...
[1] https://www.reuters.com/article/business/microsoft-expected-...