OOB score vs Validation score

I asked a similar question in class @yzhao :slight_smile:

Can i reframe your question about how the oob_score 's sample and the validation set sample is created and illustrate how the sample set formation can affect the scores?

The oob_score uses a sample of “left-over” data that wasn’t necessarily used during the model’s analysis, and the validation set is sample of data you yourself decided to subset. in this way, the oob sample is a little more random than the validation set. Therefore, the oob sample (on which the oob_score is measured) may be “harder” that the validation set. The oob_score may on average have a “less good” accuracy score as a consequence.

For example, Jeremy and Terence use only the last 2 weeks of grocery store data as a validation set. The oob sample may have unused data from across all four years of sales data. The oob_score 's sample is much harder because it’s more randomized and has more variance.

If the oob_score never improves, but the validation set score is always very good. You need to re-think how to subset the validation set. In the case of Jeremy and Terence, they might decide to take a more random sample of data across all years rather than strictly the last 2 weeks of data.

Hope that helps some!

4 Likes