Out-of-Distribution Taxonomy (preview)

Below is a high-level preview of our high-level taxonomy of Out-of-Distribution (OOD, sometimes referred to as ODD for “Out-of-distribution-detection” as well) errors in machine learning, as it was presented at the recent Harvard Machine Learning Theory Virtual Meetup on October 30th, 2020.

Unlike other papers on the subject, our guide will provide actionable recommnendations for how to fix OOD error $x$, detected with diagnostic method $y$, by doing $z$. For example, consider the following popular OOD-detection methods (described with little context by Ovadia, Yaniv, et al.):

• (Vanilla) Maximum softmax probability [Hendrycks and Gimbel, 2016]
• (Temp Scaling) Post-hoc calibration by temperature scaling using i.i.d. validation set [Guo et al., 2017, Platt, 1999]
• (Dropout) Monte-Carlo Dropout [Gal and Ghahramani, 2016, Srivastava et al., 2014] with rate p
• (SVI) Stochastic Variational Bayesian Inference [Blundell et al., 2015, Graves, 2011]

• (LL) Approximate Bayesian inference for the parameters of the last layer only [Riquelme et al., 2018]

  • (LL SVI) Mean field SVI on the last layer only
  • (LL Dropout) Dropout only on activations before last layer
• (Deep Ensembles) Ensembles of M networks trained independently on the entire dataset using random initialization [Lakshminarayanan et al., 2017]

There is a time and place for all of these, but details on those times and places are hard to find (if they exist at all). Our aim is to provide a concrete guide on when it’s necessary to make the leap to bayesian machine learning, and when you can fix your problems with softmax tuning.

Cited as:

@article{Out-of-Distribution Taxonomy,
    title = "Out-of-Distribution Taxonomy (preview)",
    author = "Matthew McAteer and Vinay Prabhu",
    journal = "matthewmcateer.me",
    year = "2020",
    url = "https://matthewmcateer.me/posts/ood-taxonomy/"
}

If you notice mistakes and errors in this post, don’t hesitate to contact me at [contact at matthewmcateer dot me] and I would be very happy to correct them right away!

See you in the next post 😄