Rip van Winkle’s Razor, a Simple New Estimate for Adaptive Data Analysis
Can you trust a model whose designer had access to the test/holdout set? This implicit question in Dwork et al 2015 launched a new field, adaptive data analysis. The question referred to the fact that in many scientific settings as well as modern machine learning (with its standardized datasets like CIFAR, ImageNet etc.) the model designer has full access to the holdout set and is free to ignore the (Basic Dictum of Data Science) “Thou shalt not train […]