But then if you observe human behaviour at a macro level don't we do the same?

I think we like to pretend that we don't. But even if you think about it from a bio/physics point of view, all we are is just a bunch of biological wires and neurotransmitter packages.

I know there's the whole unproven quantum tubule nonsense but even that would be a physics effect that we could learn about, manipulate and extort just as we have with everything else we've learnt about our realities.

That suggests they can't make inferences or produce any original insights, which is wrong. They can absolutely have takes.

Even if it's a simulacrum, it's a simulacrum based on nearly everything that was ever written. No one who has ever lived can claim to know what these models have trained on.

Aren’t human takes also a simulacrum of other human takes based on training?

How do you know they are not conscious?