First of all, thanks for the great library and course, you made machine learning so very accessible.
I am trying to use the ResNet to classify cloud types and make some simple mobile app from it. The classifier works quite well, given decent accuracy. The problem, however, that it seems to fail to detect that there is NO cloud in the image (when I give it some random image of not sky :). Is there some approach to train the classifier so that it is able to tell that none of the classes are present in the image?
One option would be to use sigmoid instead of softmax as your activation layer, add some out of domain images with all the cloud categories set to 0, and then fine-tune threshold to predict none. See the twitter discussion around the two tweets I embedded below.
Another more experimental option would be to use Bayesian uncertainty [Article] Bayesian Neural Networks experiments using Fastai. (This article sparked the twitter discussion around the embedded tweets).
I think you could try both approaches or even use both : use a sigmoid for each class, and use MC Dropout during inference and look at the uncertainty of each class. If most images of your training dataset have clouds, then if you show it a picture without cloud it should also output higher uncertainty.
I don’t know what would be the result, but I could lend a hand if you need, it’s something that could be interesting to explore
Thanks for your replies, the idea of how dropout creates an ensemble of NNs is very interesting! I will eventually try both approaches, but am having trouble with simply creating DataBunch from CSV with missing labels (for the out-of-domain images), i.e:
image328722.png,CUMULUS STRATOCUMULUS CUMULONIMBUS
AssertionError: You have NaN values in column(s) 1 of your dataframe, please fix it.
Maybe you could give me a hint to how this is usually handled? Thanks!
For the out-of-domain images, just add a label such as
No clouds or
The classifier will then treat it just as any other category and everything should work just fine.