In my experience, calibration is only ever in expectation, even for a classic classifier. Any given prediction is often still over- or under-confident.