alt Hacker NewsI wasn't going solely for "we don't know and we can't know." There are other relevant observations.
A matmul (or a fixed finite sequence of matmuls) is not Turing-complete. An LLM that produces answer in the number of steps that is proportional to the length of the answer is not Turing-complete. An LLM with CoT that produces answer after a number of steps, which is determined by the LLM itself, is Turing-complete.
So, we do have a qualitative step between a matmul and modern LLMs. Of course, it doesn't guaranty having feelings, but it doesn't allow to simply dismiss the notion due to sheer computational inadequacy.
Thanks for this comment. It's an angle worth remembering.
A machine is not turing complete, but by running it recursively on its own output, turing completeness emerges.
Who knows at what stage of intricacy consciousness emerges?
Arguably, either it does at some point, or it never does and we requiere the supernatural soul.
So, if it does, we can't disregard the possibility that we've reached it already.