Accounting for Variance in Machine Learning Benchmarks
Abstrak
Strong empirical evidence that one machine-learning algorithm A outperforms another one B ideally calls for multiple trials optimizing the learning pipeline over sources of variation such as data sampling, data augmentation, parameter initialization, and hyperparameters choices. This is prohibitively expensive, and corners are cut to reach conclusions. We model the whole benchmarking process, revealing that variance due to data sampling, parameter initialization and hyperparameter choice impact markedly the results. We analyze the predominant comparison methods used today in the light of this variance. We show a counter-intuitive result that adding more sources of variation to an imperfect estimator approaches better the ideal estimator at a 51 times reduction in compute cost. Building on these results, we study the error rate of detecting improvements, on five different deep-learning tasks/architectures. This study leads us to propose recommendations for performance comparisons.
Topik & Kata Kunci
Penulis (17)
Xavier Bouthillier
Pierre Delaunay
Mirko Bronzi
Assya Trofimov
B. Nichyporuk
Justin Szeto
Naz Sepah
Edward Raff
Kanika Madan
Vikram S. Voleti
Samira Ebrahimi Kahou
Vincent Michalski
Dmitriy Serdyuk
T. Arbel
C. Pal
G. Varoquaux
Pascal Vincent
Akses Cepat
PDF tidak tersedia langsung
Cek di sumber asli →- Tahun Terbit
- 2021
- Bahasa
- en
- Total Sitasi
- 184×
- Sumber Database
- Semantic Scholar
- Akses
- Open Access ✓