4Optimization


Recent Readings for Optimization of Deep Neural Networks (since 2017) (Index of Posts):

No. Read Date Title and Information We Read @
1 2020, May, 5 Optimization and New Loss in Deep Learning 2020-W7
2 2017, Nov, 9 Optimization IV - change DNN architecture for Optimization 2017-W12
3 2017, Nov, 7 Optimization III - Optimization for DNN 2017-W12
4 2017, Nov, 2 Optimization II - DNN for Optimization 2017-W11
5 2017, Oct, 31 Optimization I - Understanding DNN Optimization 2017-W11
6 2017, Apr, 22 Optimization17- Optimization in DNN 2017-team


Here is a detailed list of posts!



[1]: Optimization and New Loss in Deep Learning


optimization mutual-information
Index Papers Our Slides
1 Review on Semi-Supervised Learning Zhe Survey
2 Review on Generative Adversarial Networks Zhe Survey
3 Information theory in deep learning Zhe Survey
4 Lagrange Optimization Zhe Survey
5 Deep Learning and Information Theory, and Graph Neural Network Derrick Survey
6 Loss Functions for Deep Structured Models Jack Survey
7 Group Sparsity and Optimization Zhe Survey

[2]: Optimization IV - change DNN architecture for Optimization


Forcing Optimization
Presenter Papers Paper URL Our Slides
Shijia Professor Forcing: A New Algorithm for Training Recurrent Networks, 1 NIPS16 PDF + Video PDF
Beilun+Arshdeep Mollifying Networks, Bengio, ICLR17 2 PDF PDF / PDF2

[3]: Optimization III - Optimization for DNN


Architecture-Search Hyperparameter dynamic Optimization
Presenter Papers Paper URL Our Slides
GaoJi Forward and Reverse Gradient-Based Hyperparameter Optimization, ICML17 1 PDF PDF
Chaojiang Adaptive Neural Networks for Efficient Inference, ICML17 2 PDF PDF
Bargav Practical Gauss-Newton Optimisation for Deep Learning, ICML17 3 PDF PDF
Rita How to Escape Saddle Points Efficiently, ICML17 4 PDF PDF
  Batched High-dimensional Bayesian Optimization via Structural Kernel Learning PDF  

[4]: Optimization II - DNN for Optimization


Architecture Search RL Few-Shot Optimization
Presenter Papers Paper URL Our Slides
GaoJi Neural Architecture Search with Reinforcement Learning, ICLR17 1 PDF PDF
Ceyer Learning to learn 2 DLSS17video PDF
Beilun Optimization as a Model for Few-Shot Learning, ICLR17 3 PDF + More PDF
Anant Neural Optimizer Search with Reinforcement Learning, ICML17 4 PDF PDF

[5]: Optimization I - Understanding DNN Optimization


optimization Curriculum Differentiation
Presenter Papers Paper URL Our Slides
Ceyer An overview of gradient optimization algorithms, 1 PDF PDF
Shijia Osborne - Probabilistic numerics for deep learning 2 DLSS 2017 + Video PDF / PDF2
Jack Automated Curriculum Learning for Neural Networks, ICML17 3 PDF PDF
DLSS17 Johnson - Automatic Differentiation 4 slide + video  

[6]: Optimization17- Optimization in DNN


optimization scalable EM propagation mimic
Presenter Papers Paper URL Our Slides
Muthu Optimization Methods for Large-Scale Machine Learning, Léon Bottou, Frank E. Curtis, Jorge Nocedal 1 PDF PDF
Muthu Fast Training of Recurrent Networks Based on EM Algorithm (1998) 2 PDF PDF
Muthu FitNets: Hints for Thin Deep Nets, ICLR15 3 PDF PDF
Muthu Two NIPS 2015 Deep Learning Optimization Papers PDF PDF
Muthu Difference Target Propagation (2015) 4 PDF PDF



Here is a name list of posts!