I agree, and yet it is reasonable to ask if these notions of "context" (sorry!) - namely feelings, external world, body, senses, etc - are somehow distinct from an LLM's notions of context. Today they certainly capture different things, but given the right representations, why couldn't these human notions also be captured as "context"?

The idea of memory does not seem to resolve this: if you allow the machine to "compact" its context, then you've given it a system which is analogous to our own evolving state. (Though this is undoubtedly still less expressive and meaningful than the one we have evolved as humans.)

One idea I've wondered about is our human capacity to induce subsequent mental states: I can effectively decide how I want to feel and take actions to create that feeling. It's not clear whether models exhibit any degree of privileged introspection into their own states. Is this important? I don't know. (Non)determinism also does not seem to resolve it; it's my understanding that there are plenty of philosophers and researchers who think that human behavior is deterministic, or that the question of determinism does not matter.