Optimization III  Optimization for DNN
07 Nov 2017 4Optimization ArchitectureSearch Hyperparameter dynamic OptimizationPresenter  Papers  Paper URL  Our Slides 

GaoJi  Forward and Reverse GradientBased Hyperparameter Optimization, ICML17 ^{1}  
Chaojiang  Adaptive Neural Networks for Efficient Inference, ICML17 ^{2}  
Bargav  Practical GaussNewton Optimisation for Deep Learning, ICML17 ^{3}  
Rita  How to Escape Saddle Points Efficiently, ICML17 ^{4}  
Batched Highdimensional Bayesian Optimization via Structural Kernel Learning 

_{ Forward and Reverse GradientBased Hyperparameter Optimization, ICML17/ We study two procedures (reversemode and forwardmode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm such as stochastic gradient descent. These procedures mirror two methods of computing gradients for recurrent neural networks and have different tradeoffs in terms of running time and space requirements. Our formulation of the reversemode procedure is linked to previous work by Maclaurin et al. [2015] but does not require reversible dynamics. The forwardmode procedure is suitable for realtime hyperparameter updates, which may significantly speed up hyperparameter optimization on large datasets. We present experiments on data cleaning and on learning task interactions. We also present one largescale experiment where the use of previous gradientbased methods would be prohibitive. } ↩

_{ Adaptive Neural Networks for Efficient Inference, ICML17 / We present an approach to adaptively utilize deep neural networks in order to reduce the evaluation time on new examples without loss of accuracy. Rather than attempting to redesign or approximate existing networks, we propose two schemes that adaptively utilize networks. We first pose an adaptive network evaluation scheme, where we learn a system to adaptively choose the components of a deep network to be evaluated for each example. By allowing examples correctly classified using early layers of the system to exit, we avoid the computational time associated with full evaluation of the network. We extend this to learn a network selection system that adaptively selects the network to be evaluated for each example. We show that computational time can be dramatically reduced by exploiting the fact that many examples can be correctly classified using relatively efficient networks and that complex, computationally costly networks are only necessary for a small fraction of examples. We pose a global objective for learning an adaptive early exit or network selection policy and solve it by reducing the policy learning problem to a layerbylayer weighted binary classification problem. Empirically, these approaches yield dramatic reductions in computational cost, with up to a 2.8x speedup on stateoftheart networks from the ImageNet image recognition challenge with minimal (<1%) loss of top5 accuracy. } ↩

_{ Practical GaussNewton Optimisation for Deep Learning, ICML17 / We present an efficient blockdiagonal approximation to the GaussNewton matrix for feedforward neural networks. Our resulting algorithm is competitive against stateoftheart first order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike firstorder methods, for which hyperparameter tuning of the optimisation parameters is often a laborious process, our approach can provide good performance even when used with default settings. A side result of our work is that for piecewise linear transfer functions, the network objective function can have no differentiable local maxima, which may partially explain why such transfer functions facilitate effective optimisation. } ↩

_{ How to Escape Saddle Points Efficiently, ICML17 / Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M. Kakade, Michael I. Jordan/ This paper shows that a perturbed form of gradient descent converges to a secondorder stationary point in a number iterations which depends only polylogarithmically on dimension (i.e., it is almost “dimensionfree”). The convergence rate of this procedure matches the wellknown convergence rate of gradient descent to firstorder stationary points, up to log factors. When all saddle points are nondegenerate, all secondorder stationary points are local minima, and our result thus shows that perturbed gradient descent can escape saddle points almost for free. Our results can be directly applied to many machine learning applications, including deep learning. As a particular concrete example of such an application, we show that our results can be used directly to establish sharp global convergence rates for matrix factorization. Our results rely on a novel characterization of the geometry around saddle points, which may be of independent interest to the nonconvex optimization community. } ↩