ICML 2021: Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers

-------------------------------------------------------------------------------- Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers Robin M. Schmidt, Frank Schneider, and Philipp Hennig International Conference on Machine Learning (ICML) 2021 -------------------------------------------------------------------------------- ► Paper: ► Results: ► Framework: Choosing the optimizer is considered to be among the most crucial design decisions in deep learning, and it is not an easy one. The growing literature now lists hundreds of optimization methods. In the absence of clear theoretical guidance and conclusive empirical evidence, the decision is often made based on anecdotes. In this work, we aim to replace these anecdotes, if not with a conclusive ranking, then at least with evidence-backed heuristics. To do so, we perform an extensive, standardized benchmark of f