alt Hacker NewsWouldn't learning a calibrated probability distribution just be a consequence of the CE loss being a proper scoring rule? So, by construction, they are approximating the distribution of the labels. Of course in practice there might be issues with overfitting the log-likelihood and the with the labels being a finite sample of the population distribution
Yes, in a sense, with some important caveats. Cross-entropy will tend to yield reasonable marginal (i.e., not class-wise, not approximately conditional) calibration for in-distribution (i.i.d. data). The challenge with LMs, and neural networks more generally, is that the resulting estimators that are based on the output logits are very brittle to covariate shifts and out-of-distribution data, and that for real applications, we often need robust estimates in high-probability regions (e.g., of class- and prediction-conditional accuracy at least some value near 1) for conditional-branching decisions (by a human or a model). (ECE is the wrong metric for that quantity.)
The various decision APIs don't give you that, and would themselves need to be recalibrated with constraints in feature-representation space over the applicable reference class, as with https://pypi.org/project/reexpress-sdm/.