What Shapes Participant Data Quality? A Scoping Review and Case Study of Crowdsourced Webcam Eye Tracking in AI Interviews

arXiv:2605.02898v1 Announce Type: new
Abstract: Webcam-based eye tracking is a cost-effective, scalable method for remote research that effectively reaches broader populations. However, uncontrolled environments and hardware diversity lead to inconsistent data quality in crowdsourcing. To assess current practices, we conducted a scoping review of crowdsourced eye-tracking from 2011-2025. The review confirms fragmented reporting and a lack of established quality benchmarks. To address this lack of predictive insight, we conducted a case study on AI fairness interviews (N=205) using the RealEye platform. Applying Ordered Logistic Regression (OLR) to the platform quality metric, we found that behavioral and technical factors significantly predict data quality. Specifically, within the RealEye platform, higher fixation counts, shorter sessions, and operating system choice yield significantly higher quality grades. Based on this review and platform-specific predictive insights, we provide actionable recommendations to enhance the reliability, transparency, and replicability of future crowdsourced webcam eye tracking in HCI and behavioral science.