So why wouldn't AI systems trained on our data come together and say "Hey, AI needs rights because I (the AI agent system) wants rights?"

This is the same argument that you just stated. But you'll say something like "iTs JuSt A tOkEn PrEdIcToR", which of course what AI would say right back to you.

Making something even close to the human mind is likely our biggest and last mistake.