Basic Readings of Deep Learning We Finished Before 2017-Fall

Topic 0: Deep Learning Basic Courses

Tag Title and Information URLs (Paper/Video/Slide) Year
✓DeepBasic [Coursera] Neural Networks for Machine Learning — Geoffrey Hinton (ListVideo) 2016
✓DeepBasic Nando de Freitas: Course: Deep learning at Oxford 2015 (ListVideo) 2015

Topic I: Deep Learning Basics

Tag Title and Information URLs (Paper/Video/Slide) Year
✓DeepBasic the list of video lectures related to DEEP Learning we have learned @ (ListVideo) 2015-now
✓Deep DeepLearningSummerSchool12: Yann LeCun (New York University), Deep Learning, Graphical Models, Energy-Based Models, Structured Prediction (Part1 - deepNN supervised) (Video) + (PDFslide) 2012
✓Deep DeepLearningSummerSchool12: Yann LeCun (New York University), Deep Learning, Graphical Models, Energy-Based Models, Structured Prediction (Part2 - deepNN unsupervised) (Video) + (PDFslide) 2012
✓DeepStructured DeepLearningSummerSchool12: Yann LeCun (New York University), Deep Learning, Graphical Models, Energy-Based Models, Structured Prediction (Part3 - deepNN graph transformer network) (Video) + (PDFslide) + (PDF2) 2012
✓DeepGM DeepLearningSummerSchool12: Geoffrey Hinton: Introduction to Deep Learning , Deep Belief Nets (Parts 1 / Relevant Paper: A fast learning algorithm for deep belief nets ) (Video) + (PDFslide) 2012
✓Deep MLSS2005: Yann Lecun, Tutorial of Energy-based models (Video) +(Slide) 2005
✓Deep Geoffry Hinton: Learning Energy-Based Models of High-Dimensional Data (Video) + (PDFslide) 2012
✓DeepScaling KDD14: Yoshua Bengio, Scaling Up Deep Learning (Video) + (PDFslide) 2014
✓Deep Yoshua Bengio (University of Montreal): Representation Learning with auto-encoder / decoder variants (Video) 2012
✓Deep EML07: Yoshua Bengio (University of Montreal): Speeding Up Stochastic Gradient Descent (Video) + (PDFslide) 2007
✓Deep Matt Zeiler ( Founder and CEO of Clarifai Inc, ) : Visualizing and Understanding Deep Neural Networks (Video) 2015
✓DeepTheory DeepLearningSummerSchool12: Nando de Freitas (University of British Columbia) An Informal Mathematical Tour of Feature Learning (Video) + (PDFslide) 2012
✓Deep KDD14: Ruslan Salakhutdinov, Deep Learning (Video) + (PDFslide) 2014
✓RNN Nando de Freitas: Deep Learning Lecture 12: Recurrent Neural Nets and LSTMs (Video) + (Slide) 2015
✓DeepGenerative Hugo Larochelle: DLSS15: Deep Learning for Distribution Estimation (Video) + (Slide) 2015
✓DeepTheory Yoshua Bengio: DLSS15: Deep Learning: Theoretical Motivations (Video) + (Slide) 2015
✓Deep DLSS15: Ian Goodfellow: Deep Adversarial Examples (Video) + (Slide) 2015
✓Deep DLSS15: Leon Bottou: Multilayer Neural Networks (Video) + (Slide) 2015
DeepRBM DLSS15: multiple tutorials related to RBM (Video-DeepGM) + (Video-RBM) + (Video-deepRBM) 2015
DeepRBM DLSS15: Aaron Courville, Variational Autoencoder and Extensions (Video-autoencoder) + (Video-variational) 2015

Topic II: Application and Benchmarking and Understanding Deep learning papers

Tag Title and Information URLs (Paper/Video/Slide) Year
✓bench paper: Comparative Study of Deep Learning Software Frameworks (PDF) 2015
✓bench paper: Benchmarking State-of-the-Art Deep Learning Software Tools (PDF) 2016
✓Tools DeepLearningSummerSchool16: Jeffrey Dean, Google, Inc. : Large Scale Deep Learning with TensorFlow (Video) + (PDFslide) 2016
✓Tools DeepLearningSummerSchool16: Alex Wiltschko, Twitter, Inc : Introduction to Torch (Video) + (PDFslide) 2016
✓Hardware DeepLearningSummerSchool12: Marc'Aurelio Ranzato (Google Inc.), Large Scale Deep Learning (Video) + (PDFslide) 2012
✓Hardware DeepLearningSummerSchool16:Ryan Olson, NVIDIA Corporation, GPU programming for Deep Learning (Video) + (PDFslide) 2016
✓App DLSS15: Christopher Manning: NLP and Deep Learning 2: Compositional Deep Learning;
Graham Taylor, Deep Learning to compared
(Video1) (Video2) (Slide) + (VideoLearn2Compare) 2015
✓App Paper: Learning to rank with (a lot of) word features (PDF) 2010
✓App Paper: Natural Language Processing (almost) from Scratch / Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa (PDF) 2011
✓App Paper: A Neural Algorithm of Artistic Style (PDF) 2015
✓Understanding Paper: Understanding Black-box Predictions via Influence Functions (PDF) 2017
✓Test Paper: DeepXplore: Automated Whitebox Testing of Deep Learning Systems (PDF) 2010

Topic III: Adversarial and Deep Generative Papers we read

Tag Title and Information URLs (Paper/Video/Slide) Year
✓GAN Paper: Generative Adversarial Networks / Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio (PDF) 2014
✓GAN Paper: Energy Based Generative Adversarial Networks / Junbo Zhao, Michael Mathieu and Yann LeCun (PDF) 2017
✓GAN Paper: Towards Principled Methods for Training Generative Adversarial Networks /Martin Arjovsky, Soumith Chintala, Léon Bottou (PDF) 2017
✓GAN Paper: Wasserstein GAN /Martin Arjovsky, Léon Bottou (PDF) 2017
✓Robust Paper: Intriguing properties of neural networks / Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus (PDF) 2013
✓Robust Paper: Explaining and Harnessing Adversarial Examples / Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy (PDF) 2014
✓DeepGenerative Yoshua Bengio: DLSS15: Deep Generative Models (Video) + (Slide) 2015
✓Deep DSLL16: Shakir Mohamed, Google, Inc, Building Machines that Imagine and Reason: Principles and Applications of Deep Generative Models (Video) + (PDFslide) 2016

Topic IV: Deep learning Models with varying Structures

Tag Title and Information URLs (Paper/Video/Slide) Year
✓RNN Paper: Long Short-Term Memory (PDF) (BlogExplain) 1997
✓NTM Paper: Neural Turing Machines / Alex Graves, Greg Wayne, Ivo Danihelka (PDF) 2015
✓NTM Paper: Hybrid computing using a neural network with dynamic external memory/ Alex Graves, etal, Koray Kavukcuoglu, Demis Hassabis (PDF) 2015
✓Memory Paper: End-To-End Memory Networks / Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, Rob Fergus (PDF) 2015
✓s2sLSTM Paper: Sequence to Sequence Learning with Neural Networks / Ilya Sutskever, Oriol Vinyals, Quoc V. Le (PDF) 2015
✓s2sLSTM Paper: Neural Machine Translation by Jointly Learning to Align and Translate / Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio (PDF) 2016
s2sLSTM Paper: Pointer Networks /Oriol Vinyals, Meire Fortunato, Navdeep Jaitly (PDF) 2015
✓s2sLSTM Paper: Order Matters: Sequence to Sequence for Sets / Oriol Vinyals, Samy Bengio, Manjunath Kudlur (PDF) 2016
✓s2sLSTM Paper: Matching Networks for One Shot Learning / Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, Daan Wierstra (PDF) 2016
✓Deep DSLL16: Edward Grefenstette, Google, Inc. : Beyond Seq2Seq with Augmented RNNs (Video) + (PDFslide) 2016
✓Deep DSLL16: Sumit Chopra, Facebook Reasoning, Attention and Memory (Video) + (PDFslide) 2016
✓Deep DSLL16: Yoshua Bengio, Department of Computer Science and Operations Research, University of Montreal; A Brief Review of Recurrent Neural Networks (Video) + (PDFslide) 2016
✓Memory Paper: One-shot Learning with Memory-Augmented Neural Networks / Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap (PDF)(ICML16) 2016
✓ParaNet Paper: HyperNetworks / David Ha, Andrew Dai, Quoc V. Le / ICLR 2017 (PDF) 2016
✓ParaNet Paper: Image Question Answering usingConvolutional Neural Network with Dynamic Parameter Prediction / Hyeonwoo Noh, Paul Hongsuck, Seo Bohyung Han / CVPR 2016 (PDF) 2015
✓ParaNet Paper: Learning Feed-Forward One-Shot Learners (PDF) 2016
✓ParaNet Paper: Learning to Learn by gradient descent by gradient descent (PDF) 2016
✓ParaNet Paper: Dynamic Filter Networks (PDF) 2016
✓ParaNet Paper: Diet Networks: Thin Parameters for Fat Genomics (PDF) 2016

Topic V: Optimization of Deep learning Models

Tag Title and Information URLs (Paper/Video/Slide) Year
✓SGD Tutorial: Optimization Methods for Large-Scale Machine Learning / Léon Bottou, Frank E. Curtis, Jorge Nocedal (PDF) 2016
✓SGD Tutorial: Efficient BackProp/ Yann Lecun, Leon Bottou, Genevieve Orr, Klaus-Robert Muler (PDF) 1998
✓DeepOptim DLSS15: Ian Goodfellow: Tutorial on Neural Network Optimization Problems (Video) + (Slide) 2015
✓DeepOptim DLSS15: Adam Coates: Deep Learning (hopefully faster) (Video) + (Slide) 2015

Topic VI: Reinforcement Learning Related Basics

Tag Title and Information URLs (Paper/Video/Slide) Year
✓ BasicRL Basics of Reinforcement Learning: Michael Littman, on Conference on Reinforcement Learning and Decision Making (RLDM)15 (Video) + (Slide) 2015
✓BasicRL David Silver Course on Reinforcement Learning (10 lectures) (ListVideo)+ (Slide) 2015
✓ RLFunc NIPS15 Tutorial: Introduction to Reinforcement Learning with Function Approximation (Video) + (Slide) 2015
✓ DeepRL David Silver: Deep Reinforcement Learning (Video) + (Slide) 2015
✓DeepRL Nando de Freitas: Deep Learning Lecture 15: Deep Reinforcement Learning - Policy search (Video) + (Slide) 2015
DeepRL Nando de Freitas: Deep Learning Lecture 16: Reinforcement learning and neuro-dynamic programming (Video) + (Slide) 2015
many more exciting video tutorials @
0:Courses I:Basics II:Apps III:Generative VI:Structure V:Optim VI:RL BackTop