Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty

arXiv:2603.15232v1 Announce Type: cross
Abstract: Calibration is a conditional property that depends on the information retained by a predictor. We develop decomposition identities for arbitrary proper losses that make this dependence explicit. At any information level $mathcal A$, the expected loss of an $mathcal A$-measurable predictor splits into a proper-regret (reliability) term and a conditional entropy (residual uncertainty) term. For nested levels $mathcal Asubseteqmathcal B$, a chain decomposition quantifies the information gain from $mathcal A$ to $mathcal B$. Applied to classification with features $boldsymbol{X}$ and score $S=s(boldsymbol{X})$, this yields a three-term identity: miscalibration, a {em grouping} term measuring information loss from $boldsymbol{X}$ to $S$, and irreducible uncertainty at the feature level. We leverage the framework to analyze post-hoc recalibration, aggregation of calibrated models, and stagewise/boosting constructions, with explicit forms for Brier and log-loss.