Image Super-Resolution Using Deep Convolutional Networks

Super resolution convolutional neural networks was introduced as a new deep learning method for single image super-resolution compared to traditional methods such as sparse coding. They also show that conventional sparse-coding-based SR methods can be...

Neural Turing Machines

Neural Turing Machines extend recurrent network architectures with an external memory bank that can be read from and written to at each time step. This process is fully differentiable and can be trained like any...

Distilling the Knowledge in a Neural Network

NeurIPS 2014

We chose this paper as it describes the foundations of network compression, a deperature from some of the previous papers we’ve read that are more domain-specific. The approach that the authors used addresses what knowledge...

You Only Look Once: Unified, Real-Time Object Detection

CVPR 2016

We chose this paper since YOLO is the most widely used technique in real-time object detection, which is an important application of computer vision. The paper was relatively easy to understand as one of its...

Statistical Modeling: The Two Cultures

Statistical Science, Vol. 16, No. 3 (Aug., 2001)

We took a break from machine learning papers to review our statistical foundations. This paper discusses the tradeoffs between data models and algorithmic models as seen by an eminent statistician. It parallels the contrast between...

Attention is All You Need

NeurIPS 2017

We chose this paper since it kicked off a revolution in NLP architectures, introducing attention-based transformers. While the paper provided a good technical approach, it didn’t have the best visualizations to help us understand how...

Visualizing and Understanding Convolutional Networks

ECCV 2014

We chose this paper to give us context: this is a buoy in the sea of papers and improvements in image recognition that took place after the 2012 ImageNet challenge showed such remarkable improvements. This...

Natural Language Processing (Almost) from Scratch

JMLR 2011

Before we delve into learning about the state-of-the-art in NLP, we wanted to get a strong foundation. This NLP from scratch paper explains how you can go about training neural networks that perform well on...

A Neural Algorithm of Artistic Style

IEEE 2016

The style transfer paper is very unique in the ranks of seminal deep learning papers. The authors are more active in neuroscience than in machine learning, but showed the amazing results you can get when...

Generative Adversarial Networks (GANS)

NIPS 2014

This week, we had two presentations: GANs (presented by Alan Zheng) and Style Transfer (presented by Jack Morris). We thought that the GANs paper was as good of a place as any for us to...

The Lottery Ticket Hypothesis

ICLR 2018

For our first academic paper reading as a group, we chose The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks by Frankle et al. We chose this paper to try and make a smooth transition...

Overview of Deep Learning

Neural Networks and Deep Learning, Chapter 6

This was our final reading from Neural Networks and Deep Learning. This chapter explained the basics of convolutional networks. It goes on to talk about the ImageNet competition and the amazing progress in image recognition....

Training Deep Neural Networks

Neural Networks and Deep Learning, Chapter 5

This week, we read Training Deep Neural Networks, Chapter 5 of the textbook. This chapter outlines some of the fundamental problems in training deep neural networks, like the vanishing and exploding gradient problems. This chapter...

The Universal Approximation Theorem

Neural Networks and Deep Learning, Chapter 4

This chapter gives a visual proof for the Universal Approximation Theorem, based on the sigmoid activation function. It’s an explorable explanation full of animations that start from a basic step function and show you how...

Improving the way neural networks learn

Neural Networks and Deep Learning, Chapter 3

This chapter explains the basics of the deep neural network learning process. It explains the cross-entropy loss function in depth and gives an intuition for the learning slowdown across layers. It also covers overfitting vs....

How the backpropagation algorithm works

Neural Networks and Deep Learning, Chapter 2

This chapter is the most theoretical of the six chapters in Neural Networks and Deep Learning. It derives the equations for gradient descent and provides strong mathematical intuitions for why they actually make sense. (These...