Amazon Web Services – Four Years and Out

Over the last month I contacted Support for the first time in many years.

This was for a question about how billing works.

It went like this;

1. Case created.

2. Unassigned for seven days.

3. Open real-time chat, talk for 25 or so minutes where I guide a first-line Indian chap who plainly doesn't know about the subject in hand and who is as we talk reading the AWS docs I've already read. At the end, just as I couldn't find an answer, he couldn't - which is good, he didn't try to give me the wrong answer - he escalates. That's fine - a lot of questions are simple and even silly, and first line support is there to handle them - but they could have done all this without me, if they'd opened the ticket themselves rather than me having to chase.

4. Eleven days later, comes back with exactly the wrong answer. In the meantime, I had figured out the correct answer, and reply, explaining it to him.

5. Next day, I get a wall of plainly AI generated text telling me my answer is correct.

It seems to me a key issue here relating to AI generated text, is a misunderstanding on the part of AWS that I as a consumer will value that answer exactly (or indeed, even remotely) as I would value the answer from a human.

I do not. I almost ignore AI generated text, as I think it as unvalidated response.

I used to see AI generated images with lots of unintelligible writing or misspelled words in slides, but the speaker left them in anyway. “Good enough” is not customer obsession.

This enforced adoption of immature GenAI reminds me of Milo Minderbinder trying to make people eat cotton in Catch 22, because he had inadvertently obtained a huge amount of it.

I think a key goal of senior management at any big company in the last 6 months is to make rank and file fungible or obsolete. It’s one big experiment. There are precedents like the Industrial Revolution. Things get worse for the workers for a generation or so.

I'm also an AWS alumni from many years back now, and truthfully, the organizational problems really took off when Jassy moved to being CEO of amazon as a whole and major leaders left the company (Charlie Bell, et al.).

There were always other problems too, pressure on the company in both directions across many different product lines on both cost (any number of cheaper baremetal providers who are much faster at providing customers instances than they were a decade ago), and product quality (any number of startups to now bigger companies, databricks probably being the biggest success) along with a number of expensive bets that were made that didn't work out especially as interest rates began to rise (there were numbers of of different services ranging from IoT, AI, business support, robotics, groundstation, that essentially all failed).

AI infra being their latest bet, along with doubling down on custom hardware is smart, but these roles don't require the same number of SWEs and instead require a different type of high skilled professional.

#actual-aws-memes mentioned!

I also joined in 2022, and it aligns so much with my experience. Good manager that moves on, then a gradual erosion of "insist in the highest standards" towards a dreaded "good enough", GenAI only accelerated it IMO.

All the AI hype aside, I wonder if there is a way to avoid becoming one of these faceless corporations where customers are just numbers. For years Amazon has been fantastically customer centered, but at some point they just lost it. I could compile a list where Amazon is actually way more customer unfriendly than in the past now, but I guess everybody already got their own anecdotes about that. So what exactly went wrong and how could that be avoided at other companies?

I’ve been hearing Amazon is going to run out of bodies for years now and yet they keep chugging along.

The story about recovering the account rings very close to me. At least they had coworkers cheering for him, I feel teams are shrinking so much that we'll end up with just the LLM of choice to pat our back with "good work" and "you're absolutely right"

Our company also requires everyone to use more AI-related tools, and I don't think there's anything wrong with that. But the quality of work produced using these tools really depends on the individual's ability. Some people don't put in much effort, and the results they produce are really sloppy, which bothers me a lot.

> When AWS first introduced a viable cloud to the world, it was amazing. Back in the 1990s when you wanted to implement an enterprise software solution, you first had to take a guess at what computing power you would need. Next, you would have to order hardware from companies like Sun Microsystems or Dell and that could take weeks if not months to be delivered. It would then need to be racked, powered and provisioned, and then you were screwed if you happened to undersize it or criticized if you spent too much and oversized it.

This is how many large enterprises still operate today. Ironically, the main argument is that it's faster to provision VMs on-prem than it is to get approval to run in the cloud.

Bureaucracy always beats tech.

I have been advocating within my org to replace "fungible" with "flexible" or "generalist."

"Fungible" implies they are a commodity, easily swapped for someone else. In other words, they are so low-value that they are interchangeable.

"Flexible" or "generalist" instead connotes that they are so high-value that they can operate well in multiple domains, easily shifting to where they are needed most.

Not that I disagree with the points in the article, but 2022 is hardly the high point of Amazon. That ship sailed decades ago.

> the goal seems to be to create as many things as fast as possible, throw them into the world and see which ones gain traction, whether or not they serve a real need

The goal was never to solve a real problem, like we evangelized for decades. That was how it was explained when resources (mainly time, but also money) were scarce and we could not just throw things at walls. Now we can, and you won't see anyone talk about "make something people need".

Things will be low quality until something sticks, and then money will be poured into it. It's not a bad strategy, but my takeaway from this is: there are multiple plausible explanations for the same thing. People have an incentive to not give you the correct one if it helps you compete with them. But they will give you a sensible one. AI won't protect you from this, experience and real knowledge will.

AWS lost its way. S3, SQS, EC2 and VPC were great innovations and those services were done by a bunch of engineers who wanted to have a reliable elastically scalable system. This was coincidentally cost effective at the same time. What came after especially the data stack and now the AI services were done by a MBA heavy management team who does not understand innovation and treats engineering like a bank does: putting it in the cost category. Recent financial results show the impact: Google grew almost twice as much as AWS did. Maybe it is just coincidence.

The account recovery story says a lot. At some size, companies start handling people as tickets. Sometimes it only gets fixed because one person inside still cares.

Being fired for calling out Corruption. That's how I read this.

> In this whole pivot to GenAI, AWS has lost its focus on the customer. Instead of working backwards from a genuine customer need, the goal seems to be to create as many things as fast as possible, throw them into the world and see which ones gain traction, whether or not they serve a real need.

Anecdotally, this seems to be the new "mission statement" of many companies.

Agents are already getting budgets, what ‘customer obsession’ looks like when you’re marketing and selling to clankers?

I thought Amazon only did memos, not slide decks.

Has that changed, or is it the non-AWS part of Amazon?

This genai is going to bring about huge quality drop in software across the stack and across the domains. I already see orgs that had reasonable software processes transform into orgs where the only metric is how much generated code you can slap and slop together and how fast. There's no success here for anyone.

And this is not a dink on the ai tooling itself but on the organizationan processes that provide the context in which the AI code generation is being used.

Bad processes will always produce bad low quality outcomes regardless of tbe technology.

Was inspiring to meet you at NixCon. Thanks for all your energy and advocacy! You'll always be welcome in our open source community.

Thank you for writing this

Sad to hear. I was at AWS between 2016 and 2021 and had a much different experience.

Side note, and unpopular opinion ahead: while it takes a lot of courage to write things like this and I respect it, but being fired and writing negatively (no matter how justly) about your former employer is considered by many employers as a red flag and can hurt you going forward (even if you are 100% right).

The "fungible" point sounds as though the "cattle, not pets" ethos of the infrastructure management has leaked into the management of the staff.

> Amazon has a really odd viewpoint when it comes to the people who work there. They view almost all employees as “fungible”.

Hardly an Amazon-only thing. In fact, enterprises need this mindset, because people moves on, retires, or just suddenly die. With that said, due to its late-stage capitalistic ethos, Amazon is just too overly gleeful about this tasteless reality of life. It's the equivalent of a nephew coming to an aunt's funeral and shouting "A week ago, I told her everybody dies! And now she did! Wasn't I right??? Everybody dies!"

> Also, last year the focus at AWS turned fully and almost desperately toward GenAI.

I wonder if I'm being too cynical, but late-stage capitalism companies also love profiteering, and the mere prospect of firing all those pesky workers and not having to pay their salaries is like cocaine to those organizations. Which is why I think Amazon fulfillment centers will at some point rent robots at a price point between 2x and 3x their current human labor costs, in the hope that it will eventually make economic sense.

> In this whole pivot to GenAI, AWS has lost its focus on the customer. Instead of working backwards from a genuine customer need, the goal seems to be to create as many things as fast as possible, throw them into the world and see which ones gain traction, whether or not they serve a real need.

AWS has been this way for a lot longer than GenAI, since the basic infrastructure products were built out early on. But when I read this line about throwing things out there quickly, I also think of Google and even Anthropic. Google has a long list of products that got created and killed, as part of their internal politics and promotion culture. Anthropic is currently rushing vibe coded slop all the time to try and win over OpenAI and set up their IPO.

Maybe all the rich high funding companies can afford to this and maybe it is the right thing for them to do. They can afford to make big mistakes without hurting their stability. A true startup or smaller company can’t - they would shutdown because one big investment that fails is enough to destroy the whole company.

“I have to say being fired from AWS is actually a relief. There have been a lot of changes to the company since I joined in 2022, and the company I wanted to work for is no longer the same company.”

Many storied companies can be described this way. It’s a shame. Have any companies hit such scale and kept the ethos and magic of before? Is it inevitable for companies to enshitify themselves in the pursuit of their shareholder’s goals?

S3 is quite good. The rest ranges from meh to no thanks.

LOL That meme. Funny because it's true.

I was enjoying the article and then he makes some of the most bizarre claims about what cloud did and how we had to provision servers

If any of you young'uns read this, that is not how we had to do provisioning before cloud.

VMs already existed before AWS came out. You could already provision a new server usually in minutes and rent it month to month.

In fact, all the existing VM server companies had to start calling themselves cloud companies because pointy haired bosses couldn't understand what cloud really was.

> Long story short, I was able to get his resources restored. All I did was manage to poke the right bear and the support team did the rest of the work (and they were amazing).

No they utterly failed and needed a special non fungible employee to get them to do their job.

[flagged]

[flagged]

[flagged]

> They view almost all employees as “fungible”

I'm glad to see that one core amazon principle has endured the 10 years since I worked there, even if none of the actual leadership principles have survived /s

[dead]

[flagged]

AWS has lost its way.

It’s well into the IBM phase now. Primarily providing important but boring commodity infrastructure, but the top talent that can drive real innovation has long since left the building.

It’s race to stay relevant in AI but always seeming 2-3 steps behind everyone else is one such example of the current sad state of affairs.

I read every comment on this thread at the time of my posting and believe it’s fair to categorize the general sentiment that Amazon has lost its way and may even be in its “IBM stage.”

AWS services still are generally reliable and available. I’d think we’d be seeing cracks here if the organization were in shambles. AWS seems to keep humming along.