> If an informational concept causes a system to output a stresslike response, or worse trigger a stresslike action in real life then there is no distinction to me. A system is what it does.

Hmm. I think that I partially agree. But it sort of depends. E.g., someone with locked-in syndrome can seem comatose to someone who isn't very observant, while at the same time experiencing everything. Octopuses are very intelligent, but I would have to study them for a very long time before I was at all confident in my ability to predict what they were experiencing.

So, the difficulty is knowing what a "stresslike response" is. E.g., a LLM might seem stressed, but if you say the right things to it it might flip and say that it is totally fine. Or, it might seem totally fine and then you say the right thing and it seems stressed. Is it secretly stressed and hiding it? Or secretly fine and pretending to be stressed? Or something much more confusing and fragmented?

I think that we should study this more, and try to understand what experience actually is, but by default should assume that the experience of a LLM is a radically different sort of thing that that of a human.