Recent Readings for Optimization of Deep Neural Networks (since 2017)


4Optimization (Index of Posts):

No. Read Date Title and Information We Read @
1 2017, Nov, 9 Optimization IV - change DNN architecture for Optimization 2017-W12
2 2017, Nov, 7 Optimization III - Optimization for DNN 2017-W12
3 2017, Nov, 2 Optimization II - DNN for Optimization 2017-W11
4 2017, Oct, 31 Optimization I - Understanding DNN Optimization 2017-W11
5 2017, Apr, 22 Optimization17- Optimization in DNN 2017-team

[1]: Optimization17- Optimization in DNN

4Optimization optimization scalable EM propagation mimic
Presenter Papers Paper URL Our Slides
Muthu Optimization Methods for Large-Scale Machine Learning, Léon Bottou, Frank E. Curtis, Jorge Nocedal 1 PDF PDF
Muthu Fast Training of Recurrent Networks Based on EM Algorithm (1998) 2 PDF PDF
Muthu FitNets: Hints for Thin Deep Nets, ICLR15 3 PDF PDF
Muthu Two NIPS 2015 Deep Learning Optimization Papers PDF PDF
Muthu Difference Target Propagation (2015) 4 PDF PDF

[2]: Optimization I - Understanding DNN Optimization

4Optimization optimization Curriculum Differentiation
Presenter Papers Paper URL Our Slides
Ceyer An overview of gradient optimization algorithms, 1 PDF PDF
Shijia Osborne - Probabilistic numerics for deep learning 2 DLSS 2017 + Video PDF / PDF2
Jack Automated Curriculum Learning for Neural Networks, ICML17 3 PDF PDF
DLSS17 Johnson - Automatic Differentiation 4 slide + video  

[3]: Optimization II - DNN for Optimization

4Optimization Architecture Search RL Few-Shot Optimization
Presenter Papers Paper URL Our Slides
GaoJi Neural Architecture Search with Reinforcement Learning, ICLR17 1 PDF PDF
Ceyer Learning to learn 2 DLSS17video PDF
Beilun Optimization as a Model for Few-Shot Learning, ICLR17 3 PDF + More PDF
Anant Neural Optimizer Search with Reinforcement Learning, ICML17 4 PDF PDF

[4]: Optimization III - Optimization for DNN

4Optimization Architecture-Search Hyperparameter dynamic Optimization
Presenter Papers Paper URL Our Slides
GaoJi Forward and Reverse Gradient-Based Hyperparameter Optimization, ICML17 1 PDF PDF
Chaojiang Adaptive Neural Networks for Efficient Inference, ICML17 2 PDF PDF
Bargav Practical Gauss-Newton Optimisation for Deep Learning, ICML17 3 PDF PDF
Rita How to Escape Saddle Points Efficiently, ICML17 4 PDF PDF
  Batched High-dimensional Bayesian Optimization via Structural Kernel Learning PDF  

[5]: Optimization IV - change DNN architecture for Optimization

4Optimization Forcing Optimization
Presenter Papers Paper URL Our Slides
Shijia Professor Forcing: A New Algorithm for Training Recurrent Networks, 1 NIPS16 PDF + Video PDF
Beilun+Arshdeep Mollifying Networks, Bengio, ICLR17 2 PDF PDF / PDF2
BackTop