Kaggle Alaska2 competition - poor training

fastai fastai dev
1

Hello,

I am trying to train a simple model for https://www.kaggle.com/c/alaska2-image-steganalysis and I expect around 0.8 weighted roc metric as a baseline. However, my metric goes to 0.1 already during or after the very first epoch.
I kept the bare minimum for data transforms and still don’t understand what’s happening. Other people successfully trained simple models with same augmentations and architectures.

The bare minimum is:

  • vertical\horizontal flips
  • no resize
  • resnet34 / resnet50 / effnet_b0
  • train/valid split is fixed. Now that I noticed classes contain the same picture, so I removed shuffling, but it didn’t change anything.

Here’s my notebook:
https://drive.google.com/file/d/1OUvk6ywHqym1kvKEo9RWx1D-0JShw2tn/view?usp=sharing

2

Hey @much_learner

Let me see if I got you. You want to ROC metric to be high. Such as 0.8 or above = good. Low like 0.1= bad.

I’m not sure if I can help. So just bouncing a few ideas off you.

  1. The bs =8. I would try bs=64 for more accuracy.
  2. Splits by index. I would go back to random splits because it seems simpler on images.
  3. learn.fit_one_cycle(1, lr). I would try 5 instead of 1 to see if it gets better each time.

Hope the competition goes well for you!

3

I want to figure out why using the same transforms and models I don’t get similar results as other participants. I.e. weighted roc 0.8

My bs is 165.
Random splitting makes little difference, and I think it’s better to have fixed splits to have the same picture with different labels across classes.

4

Ok, I found a bug in my metric this causes low number -

        if mask.sum() == 0:
           continue

I think that’s due normalization in code.

2 Likes
5

Well done. I wouldn’t have thought of that.

6

Well, the bug was there but no model trains better than 0.6 weighted roc.
The question stands.