Can a layman be sure of an AI model's output if their proof is sound and follows through?

Wouldn't that still require mathematians?

Genuine question.