They don't use their own product, and they don't want their engineers to use their product either. They want velocity, and you can't have velocity if you're bogged down by doing end-to-end testing and finding friction and whatnot.

Its being pounded into their head almost daily (and from all directions) that AI is the future and the more AI the better. Their boss, the industry, other managers, heck probably even their children.

So any big AI initiative they are apart of had succeeded before it even starts!

Goodhart's Law will always be a factor, and "the bottom line" is largely the ultimate perverse metric for Goodhart's Law. If doing something everyone hates still has a chance to increase revenue, it's still "winning" to management.

Low quality "spam" tactics still reel in enough fish to be monetarily worth the "backlash" from customers that find it distasteful and or start to lose trust in the company. The "We promise we don't spam people" metrics and "Consumer distrust" metrics don't talk enough to the "revenue" metrics, but especially have very different cycles: big customer satisfaction metrics like J. D. Power are big annual things, not quarterly reports like earnings. In my experience things like "how often are we calling the same disinterested people to the point where it starts to feel like very personal spam" metrics don't ever really get prioritized in internal reports unless there's enforcement from Legal departments, and even then Legal departments can't "upset the bottom line" and only care about such compliance when it becomes News and/or Lawsuits, both of which don't even merit even an annual review cycle. (In fact, the modern class action lawsuit pretty effectively prevents that feedback mechanism from cycling, because generally the terms of agreement in a settled class action lawsuit is that the class is no longer allowed to sue again for the same problem, even if the same problem keeps happening and is never actually fixed.)

Unless quarterly earnings reports need to include things like client satisfaction and spam tracking, the only metric management will continue to care about, because it also is the only metric shareholders claim to ever care about, is the "bottom line", no matter how ugly the sausage is made to bump it from quarter to quarter.

The incentive to replace employees with ai systems?

VS. the inadequacy of ai systems (nondeterministic output, no reference with reality, unknown signal to noise ratio, low effort etc).

They care more about short term cost reduction than long term effects of upsetting customers. Even moreso if there is a lack of significant competition, as is often the case for companies doing this, since customers can't just easily switch to something else.

If you can save millions in costs every quarter for the degraded experience, I can see why leaders want to take the risk. Really depends on who is making the calls I guess, for the services that depend on customer experience I would think an AI service agent would probably be out of the question.

Money. Often employees are the largest cost center.

The same people who liked outsourcing customer service to cheaper countries are of course going to love doing the same to AI

I recommend everyone in this thread call OpenAI and talk that support agent. I had some issue and tried to trip it up, spoke naturally and ambiguously, and it actually did a pretty good job.

In my experience, AI customer service agents can be really helpful for some things, but when they aren’t helpful they are the most annoying thing in the world.

I think this leads to a problem because leadership might see metrics showing that the AI service agent successfully helped with 80% of the questions it is asked, but they don’t realize how damaging that 20% is. Over time, more and more of your customer base is going to hit that 20%, so eventually everyone is pissed off at you.

It would be like if 1 in 100 Cheerios in a box were made of poop. It doesn’t matter that most of them are fine, people are going to remember that one cheerio more than the 99 others.

They know, because in their personal capacity, they hate interacting with AI customer service agents from other companies.

But their bonus depends on driving down costs in their company.

It's like how these automated call handlers that say "Just tell us what you need help with" and then no matter how you phrase the sentence it just doesn't understand what you mean. Great idea in theory, horrendous execution. Bonus points if it automatically disconnects after 3 attempts without finding the right magic word from its dictionary of options.

> the customer can tell INSTANTLY they're AI

I had to call Geico's claims department yesterday and their AI customer service agent had a surfer bro accent, said "no worries" and "hang tight" and "I gotchu" while trying to follow-up on my claim which made it even more infuriating to interact with. Like... stop with the affectations and fake trying to be hip just be professional and succinct please!

Really? The modern ones I’ve run into lately are like 100x better than the old flows. “Press 1 for sales. Press 2 to know our hours. Press 3 if you have a ticket. Press 4 to hear our website address…”.

Of course nothing beats a human with real agency at the company but like, these modern agents could be 100x better than what airlines and internet service providers currently have.

They don't care. They have metrics to meet and don't care how. Full stop.

Have you seen "Severance"? It's a wonderful show that shows us a sick truth:

Many people are a different person at work. That different person is devoid of morals & ethics. They are machines intent on meeting metric targets. Nothing else, not managers, not employees, not people, just machines hitting targets they didn't decide on.

How could we possibly expect more?

It’s when they see the money they saved that quarter with layoffs. Then the next quarter customer satisfaction is down and they just don’t know why…