Rho-Perfect: Correlation Ceiling For Subjective Evaluation Datasets

arXiv:2602.08552v1 Announce Type: cross
Abstract: Subjective ratings contain inherent noise that limits the model-human correlation, but this reliability issue is rarely quantified. In this paper, we present $rho$-Perfect, a practical estimation of the highest achievable correlation of a model on subjectively rated datasets. We define $rho$-Perfect to be the correlation between a perfect predictor and human ratings, and derive an estimate of the value based on heteroscedastic noise scenarios, a common occurrence in subjectively rated datasets. We show that $rho$-Perfect squared estimates test-retest correlation and use this to validate the estimate. We demonstrate the use of $rho$-Perfect on a speech quality dataset and show how the measure can distinguish between model limitations and data quality issues.