Reconciling modern machine-learning practice and the classical bias–variance trade-off
Mikhail Belkin, Daniel J. Hsu, Siyuan Ma
et al.
Significance While breakthroughs in machine learning and artificial intelligence are changing society, our fundamental understanding has lagged behind. It is traditionally believed that fitting models to the training data exactly is to be avoided as it leads to poor performance on unseen data. However, powerful modern classifiers frequently have near-perfect fit in training, a disconnect that spurred recent intensive research and controversy on whether theory provides practical insights. In this work, we show how classical theory and modern practice can be reconciled within a single unified performance curve and propose a mechanism underlying its emergence. We believe this previously unknown pattern connecting the structure and performance of learning architectures will help shape design and understanding of learning algorithms. Breakthroughs in machine learning are rapidly changing science and society, yet our fundamental understanding of this technology has lagged far behind. Indeed, one of the central tenets of the field, the bias–variance trade-off, appears to be at odds with the observed behavior of methods used in modern machine-learning practice. The bias–variance trade-off implies that a model should balance underfitting and overfitting: Rich enough to express underlying structure in data and simple enough to avoid fitting spurious patterns. However, in modern practice, very rich models such as neural networks are trained to exactly fit (i.e., interpolate) the data. Classically, such models would be considered overfitted, and yet they often obtain high accuracy on test data. This apparent contradiction has raised questions about the mathematical foundations of machine learning and their relevance to practitioners. In this paper, we reconcile the classical understanding and the modern practice within a unified performance curve. This “double-descent” curve subsumes the textbook U-shaped bias–variance trade-off curve by showing how increasing model capacity beyond the point of interpolation results in improved performance. We provide evidence for the existence and ubiquity of double descent for a wide spectrum of models and datasets, and we posit a mechanism for its emergence. This connection between the performance and the structure of machine-learning models delineates the limits of classical analyses and has implications for both the theory and the practice of machine learning.
1965 sitasi
en
Mathematics, Computer Science
Open3D: A Modern Library for 3D Data Processing
Qian-Yi Zhou, Jaesik Park, V. Koltun
Open3D is an open-source library that supports rapid development of software that deals with 3D data. The Open3D frontend exposes a set of carefully selected data structures and algorithms in both C++ and Python. The backend is highly optimized and is set up for parallelization. Open3D was developed from a clean slate with a small and carefully considered set of dependencies. It can be set up on different platforms and compiled from source with minimal effort. The code is clean, consistently styled, and maintained via a clear code review mechanism. Open3D has been used in a number of published research projects and is actively deployed in the cloud. We welcome contributions from the open-source community.
2034 sitasi
en
Computer Science
Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors
Jonathan Huang, V. Rathod, Chen Sun
et al.
The goal of this paper is to serve as a guide for selecting a detection architecture that achieves the right speed/memory/accuracy balance for a given application and platform. To this end, we investigate various ways to trade accuracy for speed and memory usage in modern convolutional object detection systems. A number of successful systems have been proposed in recent years, but apples-toapples comparisons are difficult due to different base feature extractors (e.g., VGG, Residual Networks), different default image resolutions, as well as different hardware and software platforms. We present a unified implementation of the Faster R-CNN [30], R-FCN [6] and SSD [25] systems, which we view as meta-architectures and trace out the speed/accuracy trade-off curve created by using alternative feature extractors and varying other critical parameters such as image size within each of these meta-architectures. On one extreme end of this spectrum where speed and memory are critical, we present a detector that achieves real time speeds and can be deployed on a mobile device. On the opposite end in which accuracy is critical, we present a detector that achieves state-of-the-art performance measured on the COCO detection task.
2667 sitasi
en
Computer Science
Method of Modern Mathematical Physics
Mark S. C. Reed, H. Simon
Introduction to Classical and Modern Test Theory
L. Crocker, J. Algina
5131 sitasi
en
Psychology
Classical and modern regression with applications
R. H. Myers
Modern Multidimensional Scaling: Theory and Applications
I. Borg, P. Groenen
4376 sitasi
en
Mathematics, Computer Science
Introduction to the Modern Theory of Dynamical Systems
J. Zukas
3773 sitasi
en
Mathematics
Sources of the Self: The Making of the Modern Identity
Charles Taylor
6511 sitasi
en
Philosophy, Psychology
Modern Quantum Mechanics
J. Sakurai, San Fu Tuan, R. Newton
1. Fundamental Concepts. 2. Quantum Dynamics. 3. Theory of Angular Momentum. 4. Symmetry in Quantum Mechanics. 5. Approximation Methods. 6. Identical Particles. 7. Scattering Theory. Appendices. Supplements. Bibliography. Index.
[Modern epidemiology].
M. Graffar
Modernity and Self-Identity: Self and Society in the Late Modern Age
A. Giddens
Acknowledgements Introduction 1. The contours of high modernity 2. The self: ontological security and existential anxiety 3. The trajectory of the self 4. Fate, risk, and security 5. The sequestration of experience 6. Tribulations of the self 7. The emergence of life politics Notes Glossary of concepts Index.
4318 sitasi
en
Sociology, Psychology
Governmentality: Power and Rule in Modern Society
Michael Lait, Mitchell Dean
3601 sitasi
en
Political Science
Modern Applied Statistics with S Fourth edition
W. Venables, B. Ripley
3521 sitasi
en
Computer Science
Modern Control Engineering
P. Paraskevopoulos
4018 sitasi
en
Computer Science
Reversal of Fortune: Geography and Institutions in the Making of the Modern World Income Distribution
D. Acemoglu, D. Acemoglu, D. Acemoglu
et al.
4572 sitasi
en
Geography, Economics
A Guide to Modern Econometrics
Marno Verbeek
3277 sitasi
en
Economics, Computer Science
Computer Vision: A Modern Approach
D. Forsyth, J. Ponce
4431 sitasi
en
Computer Science
The Modern Practice of Adult Education: From Pedagogy to Andragogy
M. Knowles
4436 sitasi
en
Psychology
Modern Applied Statistics with S-Plus.
W. Venables, B. Ripley
4545 sitasi
en
Mathematics, Computer Science