[Still Open] How to I tell my model to be honest and say "I don't know"

(Yogendra Joshi) #1

I have trained a model to detect tennis ball vs cricket ball. (Part1-Lecture 2)… now if I show it a picture of lets say a dog, I would like it to tell me … “I don’t know” instead of picking the “tallest amongst the midgets”.

0 Likes

Handle data that belongs to classes not seen in training or testing
(Koen van der Veen) #2

You could use sigmoid with a threshold instead of softmax.
(see https://github.com/fastai/course-v3/blob/master/nbs/dl1/lesson3-planet.ipynb)

1 Like

Handle data that belongs to classes not seen in training or testing
(Yogendra Joshi) #3

Thanks… just started off with Lesson 3… So I guess I asked the question too early!

0 Likes

(Yogendra Joshi) #4

experts, anyone willing to give a helping hand here. Even after going through Lesson 3 (which is multi class classification), I can’t find a way to come back to lesson 2 and my predictions above and make the system predict only if its really confident).

Someone suggested sigmoid with threshold however not sure how to change the lesson 2 code to do that.

I was hoping the predictions will give scores and they don’t!

this was the image I tested


and prediction is pretty confident (and wrong)

Is it possible that my training set is too small?

0 Likes

(Marcos Santana) #5

I was thinking the same and I believe even if you include all tennis and cricket balls in the world on your training set your model will still give a prediction - it was designed to give answers anyway.
But maybe there’s a way to modify your network to include the “I’m not sure” class and use a threshold. That would be a multiclass problem - Tennis ball, cricket ball and I’m not sure.
I’m new to fastai so you should take a look at the documentation of the activation functions to understand how they convert the activations from previous layers. Softmax and sigmoid probably use a 0.5 threshold. You could write your custom activation by changing this value.

1 Like

(Yogendra Joshi) #6

Will give it a try… if you find a way, let me know

0 Likes

(Koen van der Veen) #7

I think, the problem is that you dont have any examples like this in your train set. So you only learn what tennisballs and cricketballs are, given that all that you are looking at is images of these two objects. What you could do is include random images (maybe imagenet or something similar) that do not include either of the two balls and give them labels 0,0

1 Like

(Yogendra Joshi) #8

thanks… will try that

0 Likes

(Yogendra Joshi) #9

@jeremy: while going through lesson 10, I think I found the answer, but don’t have the skills currently to implement it.

I think the reason for the issue is that sigmoid will enlarge the probabilities and will result in skewed results. The way to handle it will be to use the last layer a “binomial” one instead of sigmoid and then run a if loop to say if one category is above 60% and all others are lower than 40% show that result else give multiple values and if all are really low then say “I don’t know”?

If the approach is correct, can you / someone guide me on how to implement it (way back in lesson 2 classification notebook :slight_smile: )

0 Likes