Suspiciously Good Rossmann RSMPE

System: paperspace Jupyter notebook
Date: 2018-09-17

My Rossmann results from the ‘ALL’ section and the ‘TEST’ section seem suspiciously good. The video series and the Kaggle competition had an rmspe equal to 0.09x or 0.10x. However, returns exp_rmspe equal to or less than 0.02xxx for the examples using cycle_len=1…

I only made one modification to the Jupyter notebook. Namely, I had to wget the data, rossmann.tgz, b/c the paperspace fastai notebook doesn’t seem to include it

Any thoughts on why the exp_rsmpe is so low would be appreciated.

Yes, your exp_rmspe is indeed too good to be true. I believe that the Rossman notebook wasn’t written in a fashion that it could simply be run “end-to-end”, and that you will need to make some adjustments to achieve the “correct” 0.09/0.10 score.

IIRC I commented out cell 95, which sets val_idx=[0]. I may have made one or two other changes, but I’m afraid that I can’t recall what they were.

Obviously, you should also ensure that you are using the very latest version of the notebook, as quite a few amendments have been made to it over time.

1 Like

AndrewK: Thanks for the advice. I only needed to skip over the code you referenced in order to calculate the ‘correct’ answers. I must’ve forgotten or simply not noted why that particular code existed in the original when I watched the video.