> If it solves the problem, you can ask the AI to explain it you.

Will I get an understandable and verifiable explanation? And, if the system cannot solve the problem, will I get an understandable and verifiable explanation of that? As a researcher I'd think that shared understanding is the point.