Certainly, but at that point it becomes a philosophical question, rather than an empirical one.

Can we even measure an intelligence greater than our own? And at that point -- what is the point exactly? Why make an intelligence that we don't even know is intelligent? If its output is indiscernible to the human mind, what good does it do us?

The only "barrier" I can see that is valuable and that might be measureable to humans is this underlying mechanic of being able to create new ideas/concepts from little to no existing human generated training data. Otherwise, and as I see it right now, these machines will not become sentient, or go rouge, or be able to cure all future disease or solve all of physics, becasue they are entirely reliant on human made trainingsets.

Historically, for humans, this can not be the case, otherwise we would see no progress. Someone has to have a thought for the first time. Perhaps the solution to this for AI agents is some inherent randomness, and massive amounts of brute force compute. Who knows?