The problem is that science has found no evidence of free will, leaving humans to be nothing except determinism and chance (and given that QM effects don't scale to molecular level, that leaves only determinism). The oddity in conscious LLMs wouldn't be the determinism, it would be that we finally have a consciousness running on a system we have enough control over to repeat the same state.
>and given that QM effects don't scale to molecular level, that leaves only determinism
This is simply wrong. Quantum effects affect your choices more than it might seem possible. The wrong atom decays in the wrong place in your body mutating your DNA, that one too much string of DNA that starts a tumor, which means chemotherapy. That does seem to affect your choices.
Or instead of Shroedinger's cat you use it to trigger something, buy or sell some stock. Which clearly affects your future, one way or another.
I never really understood why humans keep insisting stochastic processes do not affect their choices and their future. It's very short sighted.
Hell, even brownian noise in your brain can affect your ideas, that one random spark out of nowhere, that one neuron that's pushed over the triggering edge. There's so much stuff all around us affecting us constantly. One neutrino interacting a the wrong time (average one does interact with our bodies across whole life, I remember reading), one cosmic ray.
To a forth dimensional being humans would look almost exactly like an LLM does to us. "What do you mean they have to wait for the future to know about it, how could they possibly be conscious then".
Also, I'd love to have a quantum copy machine where you just replay someones state again and again with slight changes to see out it effects the output. My assumption is we'd figure out we behave just like LLMs pretty quick.
Also, don't give me unlimited power or I might make a the world a war crime simulator, so there's that.