Comparable being the key word there. AI marketing is threatening to eventually eliminate most white collar work. The exact high paying jobs (at least in the US) that enable upward class mobility and fuel the consumption based economy.

Take those away and tell everyone "sorry, go do physical labor now for half or worse of the salary" and that's a big problem.

Automation is a boon when it automates physical labor, not when it automates away knowledge work.

The other big worry is, what if it just doesn't do what is promised and these trillions of dollars that were spent assuming magic would happen were all for nothing? I mean, other than to make a handful of extremely wealthy individuals even more wealthy at the expense of everyone's retirement funds.

There are precedents for a lower workforce. It was not so long ago that women did not participate much in formal labor, but rather spent their creative energies improving their families and homes. That might not be an empowering choice today, and I'm not advocating for it, but it shows that the economy has in the past and probably still can get along perfectly well with a lot of sidelined labor capacity. The important thing is that the sidelined labor find some useful purpose outside the workplace rather than simply consuming welfare: in the past, domestic work like childrearing and social/emotional work like building community soaked up excess labor capacity and still had pro-social effects.

There's also the worry whether productivity is real.

There is a basic problem with framing though. Why does the labour need to find somewhere to go, but capital doesn't? Why can't the increase in productivity be captured by labour and denied to capital?