On the contrary, I would argue that it's true in a totality of AI cases.

To your point, I agree that nominally there should be a way to give conceptual names to paths of weights, and when answering a question, notice which weights were and were not applied and retrospect on that.

That's not what reasoning traces as they currently exist are, though.