When you put it like that, I think there are actually three separate things we're arguing about:

- intelligence: ability to solve problems

- agency: ability to be self-directed, to choose what to do

- consciousness: ability to experience things; "having a self"

The "intelligence" part is largely solved now (that's a very big statement, but I can't think of a better way to phrase it!) Although really it's just moving the goalposts -- computational stuff like calculating trajectories or playing chess was solved long ago, this is just more and more things moving into the "solved" column.

That leaves agency and consciousness as the hot topics that nobody has really figured out. Do they always go together, does one require the other, does one create the other? Who knows?

"Free will", there's another one. That must surely connect with agency, with consciousness, or both, in some manner we haven't figured out.

To me, agency seems like a solvable problem via existing approaches and technology. I have no idea what that means for consciousness. It's tempting to conclude that consciousness is just a mirage, but then we all do feel like we have it, so it seems like it must be something. Maybe consciousness arises automatically once you have sufficient intelligence and agency; but how would you ever determine that?

Other people might think that true agency requires consciousness, and that consciousness requires some magical new ingredient that LLMs don't have yet. The problem with that approach is, you have to identify what it is that true agency can do that software can't do; and every time you do that, it turns out LLMs can do it, so you have to keep moving the goalposts. Trying to make your argument rigorous immediately makes it self-defeating; I think that's why so much of the philosophical discussion around this is impenetrably vague. All the arguments that aren't vague just turn out to be wrong.