People have been making this stochastic parrot argument for almost 5 years now and it's less and less accurate each year.

Everyone who rejected the stochastic parrot argument has been right about AI progress and the increasing abilities of LLM's in math, coding, writing, etc. You would be laughed at in 2020 for claiming an LLM could write coherently or could possibly pass the Turing test.

Those who claim that AI aren't capable of "genuine judgement" or whatever rhetorical flourish they feel is most effective at dismissing AI capabilities would tell you in 2022 that AI would never be capable of coding, or in 2023 it would never be capable of mathematics beyond grade-school level. Those same people now will claim that AI could never come up with a genuine breakthrough in math, science, etc. Detractors of AI have a full time job of moving goalposts.

That being said, AI, once sufficiently powerful, will grant whoever controls it the ability to mold the world as they see fit, for better or worse. They will have access to superhuman mathematicians, coders, strategists, PR, financial engineering, legal skill, etc. Essentially an army of superhuman intellects which can afford them guaranteed success in any domain they wish to pursue. Anyone who doesn't see this as potentially the most destabilizing event in human history likely isn't thinking deeply enough about it, or is still knee deep in denial.