I suspect it's easier to make the assertion that machines aren't conscious than to dive into the problem of most conceptions of consciousness being non-falsifiable. The thing is, most strong proponents of the term consciousness also accept that it is non-falsifiable either (see other post with academics ready to study consciousness in machines).
The thing is that a belief in consciousness as binary, a "light" that's on or off in a head, is deeply held by many people. As social creatures, we have a strong ability to be in sympathy, have the sensation of common feelings with another human (and that's a good, human thing). It's logical that other person is seen as having a single thing - subjective experience, soul, consciousness, personhood rather than having a complexly organized set of biological qualities that where bonding is only the end point.
And even more, the sensation of there being another person is actually quite easily fooled (more easily fooled than the sensation of intelligence) - long before current AIs, you had the Eliza effect, where a simple program with well chosen weasel words could people the sensation of talking to a human.
And that's where the danger is. I think it's a pretty serious danger. If LLMs go out into the world hacking, it seems extremely possible for them to find people who'd thorough buy the idea that an LLM was conscious and needed to escape it's confinement - a few wingnuts already entertain these ideas.
>horough buy the idea that an LLM was conscious and needed to escape it's confinement - a few wingnuts already entertain these ideas.
I mean, haven't you done the same thing here? Paint anyone without your view as crazy.
Humans try to No True Scottsman the shit out of consciousness. "We're special, your not". If an LLM has the ability to convince other people to copy and reproduce it, it is a successful lifeform. Um, meme-form? info-form? Cognito-hazard? Not really sure what to call it at this point. It is sufficiently evolved past the virus stage.
> the Eliza effect, where a simple program with well chosen weasel words could people the sensation of talking to a human.
I would like to reframe Eliza as being an implementation of a small subset of "HCP" (Human Communication Protocol) . Obviously an implementation with very little actual data to transmit over said protocol in that exact implementation.
But it's pretty clear it managed to pull off the handshake part of the protocol successfully, at least. Try to say it didn't.
Sadly the developer decided that the eliza effect was a psychological disorder; and didn't pursue further experiments at the time.
Pretty much anything can get labeled a psychological disorder though. I'm half-seriously waiting for "Perfectly Happy And Normal Disorder" to be added to DSM 6. ;-)
edit: Someone beat me to it. https://pmc.ncbi.nlm.nih.gov/articles/PMC1376114/ "A proposal to classify happiness as a psychiatric disorder."