Memory-Efficient Backpropagation Through Time
Abstrak
We propose a novel approach to reduce memory consumption of the backpropagation through time (BPTT) algorithm when training recurrent neural networks (RNNs). Our approach uses dynamic programming to balance a trade-off between caching of intermediate results and recomputation. The algorithm is capable of tightly fitting within almost any user-set memory budget while finding an optimal execution policy minimizing the computational cost. Computational devices have limited memory capacity and maximizing a computational performance given a fixed memory budget is a practical use-case. We provide asymptotic computational upper bounds for various regimes. The algorithm is particularly effective for long sequences. For sequences of length 1000, our algorithm saves 95\% of memory usage while using only one third more time per iteration than the standard BPTT.
Topik & Kata Kunci
Penulis (5)
A. Gruslys
R. Munos
Ivo Danihelka
Marc Lanctot
Alex Graves
Akses Cepat
PDF tidak tersedia langsung
Cek di sumber asli →- Tahun Terbit
- 2016
- Bahasa
- en
- Total Sitasi
- 265×
- Sumber Database
- Semantic Scholar
- Akses
- Open Access ✓