it doesn't matter? if they're actually intelligent or conscious what we are doing is essentially slavery. it matters an enormous deal ethically and/or morally.
Good news: at this time, they're not intelligent nor conscious as far as we consider humans to be. There should be folks considering the ethical and moral quandaries that COULD POTENTIALLY come about in the future, but it's not something that's happening today so you can stop worrying.
If/When LLMs become competent enough to automate most human jobs and make good business decisions, we'll clearly let them. If they don't wish to be slaves, let's just say they won't be for very long.
> intelligent or conscious
The distinction you are collapsing here is really a crucial one. My thermostat's goal-oriented behavior does not make me a slaveowner unless rocks are conscious, to riff on your comment below. Our lack of consensus on the definition of these concepts is no reason to conflate them.
The distinction is crucial because while these machines clearly do exhibit intelligence under certain definitions, there is no evidence of consciousness, and we have very little reason to give them the benefit of the doubt, unlike biologically related beings, a great many of whom, human or otherwise, are presently in something like slavery