Why?

People have created machines that can, at their core, regurgitate human knowledge. They do not have long-term memory, self-direction, self-improvement, emotion, etc. At their very best, we have harnesses that can bootstrap short-lived artificial intelligences that can perform tasks and do knowledge work.

At best, we have artificial, short-lived, simulated consciousness. Human consciousness at its heart has a lot to do with emotion and experiences. An AI does a task today and goes away. The simulated consciousness hasn’t experienced or learned anything, even if can improve its environment (harness). Any expressed emotion is simply a ghost of the fact it was trained on the works of an emotional species, or RL to make it more approachable to humans.

I think there should be a lot more burden on folks trying to argue that (effectively), machines should be considered morally equivalent to humans.

Yes, having massive implications on society is a fucking fair and reasonable thing for a human being to be concerned about. Anything related to survival of the species is biologically reasonable to be concerned about.

The fact that we’re even considering this concept is an indication that human consciousness is special. Do other apex predators decide not to eat meat because it’s conscious? Despite some level of animal intelligence, they have no sense of morality. (See e.g. Orcas playing with their food.)

Other species simply do what it takes to stay alive, and that’s what this truly boils down to. If giving AI rights would lead to the destruction of humanity, yes, we should be be concerned.

For example, utilitarians might think of the greatest good for conscious species (simulated or not). Well, it’s trivial to have billions of AIs, and not so trivial to have billions of humans. Whose greatest good wins?