Lesson 4 - Rossmann (exclusion of zero Sales)

Part 1 (2018)
1

Hey everyone,
I want to ask you about this step in Rossmann, respectively it’s consequences for the DL model

joined = joined[joined.Sales!=0]

I thought it’s sort of a business decision whether to include zero sales or not and in the lecture these were excluded to be aligned with original approach.

For me zero dependent variable seems to be a valid value (especially in other datasets), but when I remove this line the model goes weird…

  • lr_find() doesn’t return error but also plot is empty
  • trn_loss and val_loss are nan in all epochs etc.

Have you tried this and do you get same results? Do you know why it should be happening?

Thanks a lot for any explanation :slight_smile:

3

Hi Stanislav,

Sorry about my earlier post (which i deleted), I misunderstood you.

So, I had the same problem as you did, mine was because of
yl = np.log(y), it returned a divide by zero warning when I ran it. I think we’ll have to avoid using log entirely according to one of Jeremy’s post which I can’t find unfortunately.

I had to update the metric function too, as its does exp(y).

I’ve updated my code but I still have issues with it, for one the training and validation losses are in the 10000 range, not sure if that matters too much as I’m not entirely sure about the calculation behind them.

Secondly, my rmspe function returns ‘inf’ and I think the reason to this is because the target can be 0 (when sales = 0) and when dividing by 0, it tends to infinity. I’m not quite certain how to handle this problem as yet.

Hope this helps, I’ll keep you posted :slight_smile:

4

Maybe add $0.01 to sales to avoid div/0.

5

I actually saw a post which suggested updating denominator of the RMSPE function to (y + 1). Yet to try :slight_smile:

6

Hi guys,
thanks for the responses. The problem is related both to the logs and defined metric (RMSPE)… obvious after reading the code more thoroughly :slight_smile:. Generally for zero-value problems different metric would need to be used, but still there would be the inf log problem (as logs are quite integral part of the solution). I’ve tried couple of quick tests based on some forum suggestions (setting zero sales to 0.01 or 1 etc.) but without any considerable results… I’ve tried also as proposed somewhere the log1p() function but the results were also invalid (likely due to other consequences of this function). So probably some conceptually different solution needs to be found.