I think the concern is that humans who are given back their time won't have any means to make use of that time, or even possibly means to survive. The resources will be concentrated in the hands of the few more than ever.

When cars made horses obsolete, it didn't go so well for horses.

In the short term, AI is disempowering the vast majority of people in favor of a very small subset. In the also way too short term, AI is disempowering all people.