Learning the Optimizer by Luke Metz

Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
was a paper from September, 2020, by Luke Metz and co. It is another
step towards replacing hand-designed features with learned functions, this time the
optimizer. This has been a three year journey for Luke; listen to him describe what
he’s learned along the way and where are the pain points, from research to
engineering.