1. In a binary classification task (with fairly balanced classes), in what scenario the AUC is not a reliable performance measure for model selection? We are looking at a general case. As a hint, imagine we plotted the ROC curves of two models -- think about in what case the AUC is too simplistic for comparing these models.

Exercises in this section cannot be answered by just reading the chapter and require further reading and research.

1. In the following paragraph, replace ‘__’ by the letters a-c, which are defined as
   a): Out-of-Bag (OOB) Bootstrap
   b): .632 Bootstrap
   c): .632+ Bootstrap

"For estimating the generalization performance, the _(1) is less optimistically biased than the _, while (1) and _ are less pessimistically biased than the _."

Suggested reading to answer this question:

• Sebastian Raschka (2018). Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning.
  https://arxiv.org/abs/1811.12808