We took a break from machine learning papers to review our statistical
foundations. This paper discusses the tradeoffs between *data models* and
*algorithmic models* as seen by an eminent statistician. It parallels the
contrast between old-school statistical modeling algorithms and new-school
machine learning algorithms. Simplicity is sometimes better– when you value
interpretability over predictive accuracy. But if you want to minimize
predictive error, you’re going to sacrifice some level of understanding.
The author discusses three principles of statistical models:

- Rashomon: there will be many models that give about the same error on the test set
- Occam: Simpler is better (or Einstein: as simple as possible, but no simpler)
- Bellman: Dimensionality is a curse (or perhaps a blessing, to machine learnists)

These three principles encapsulate a lot of the important message from the paper. He also gave good advice for anyone looking to fit a model to data: dive into the data first. Spend some time perusing it and analyzing it – gain a real understanding of your data before attempting to fit a model to it. Although it was written in 2001, Breiman’s manifesto provides messages that are just as important in 2020.