Kaggle Histopathological Cancer Detection

[barashe]

It seems like everything is properly done.

Kaggle score = 0.9478, 4% lower than the AUROC calculated from my local validation set.

Is the Kaggle score calculated on your validation set? Or on the unseen test set?

[Pomo (Malcolm McLean)]

Kaggle score is from the test set of images. The mysteries are:

1. Kaggle AUC on test set is much lower than my AUC on my validation set.
2. Accuracy and AUC scores on my validation set are much different. (Should these be close?) Kaggle AUC score more closely resembles accuracy score on my validation set.

I was not confident of my code to create the Kaggle submission and even rewrote it to predict each image separately and write the submission file line by line. No difference.

[PPPW]

Hi @archit, I wonder did you use very small learning rate? After training for a while I found the performance is not improving and the lr_find gives a flat line (except at large lr), but my lr is already ~ 1e-8, which is pretty small…

[archit]

1e-8 is very small. Are you sure you have set up the neural network properly? Also, try to scale up the images to 256x256. That improves resnet accuracy very much.

[PPPW]

Thanks for the reply!
At the beginning (after unfreeze), the lr_find tells that I can use ~ 1e-3, I use 1cycle policy for 3 epochs, then decreased the lrs to ~ 1e-5 (as lr_find suggests) and run 1cycle policy for another 6 epochs, then I can only use very small lrs, otherwise the performance is not improving. Maybe I should do a large cycle rather than gradually increasing the cycle length… But I think I should be setting things correctly…
Scaling up is an interesting idea, I haven’t thought about that, thank you for the tips!

[Mauro]

I’m getting the same thing. Fastai score 99.4%. Kaggle 94.9%.

[Pomo (Malcolm McLean)]

Here’s some more investigation of the Test/Validation discrepancy, though I don’t know what to conclude from it.

1. Calculation of mean and standard deviations of the three image channels (pixel values) showed no significant differences between the full Training set and the Test set. The images have the same gross characteristics.

2. Calculation of cancer frequency by probability >.5…

Fraction of actual cancer in Training set = 0.40548801272582663
Fraction of predicted cancer in Validation set = 0.40247699125099423 (omitted from original post)
Fraction of predicted cancer in Test set = 0.33198162135820947

The latter Test prediction gets a Kaggle score = .9536, Validation AUC = .9923. Validation accuracy = .9659.

So it looks to me like the Kaggle Test and Training set are NOT uniform samplings of the same source image set, evidenced by the different prediction rates for cancer. The Test set is biased toward non-cancer.

All these test were done in a single session with a DataBunch created once. Therefore there’s no chance of leakage of Training images into Validation images.

Any ideas?

[Pomo (Malcolm McLean)]

This AUROC metric implementation is wrong. For the right way see…

Using AUC as metric in fastai

I faced this requirement a couple of weeks ago. Fastai computes metrics for each batch and then averaged across all batches, which makes sense for most metrics. However, AUROC can not be computed for individual batches, requiring to be computed on the entire dataset at once. So, I implemented a callback to compute the AUROC: from sklearn.metrics import roc_auc_score def auroc_score(input, target): input, target = input.cpu().numpy()[:,1], target.cpu().numpy() return roc_auc_score(tar…

[krash]

Can you please elabrate on this?