wgpubs
(WG)
November 6, 2018, 6:39am
1
So running this on my own dataset …
pred_class, pred_idx, outputs = learn.predict(img)
# outputs => tensor([15.9323, 0.1261])
The documentation describes outputs
as probabilities … but these don’t look like probabilities.
Am I missing something?
1 Like
ramesh
(Ramesh Sampath)
November 6, 2018, 12:43pm
2
From the code snippet of predict
it only looks like raw scores -
:param ps: dropout, can be a single float or a list for each layer."""
lin_ftrs = [nf, 512, nc] if lin_ftrs is None else [nf] + lin_ftrs + [nc]
ps = listify(ps)
if len(ps)==1: ps = [ps[0]/2] * (len(lin_ftrs)-2) + ps
actns = [nn.ReLU(inplace=True)] * (len(lin_ftrs)-2) + [None]
layers = [AdaptiveConcatPool2d(), Flatten()]
for ni,no,p,actn in zip(lin_ftrs[:-1],lin_ftrs[1:],ps,actns):
layers += bn_drop_lin(ni,no,True,p,actn)
return nn.Sequential(*layers)
class ClassificationLearner(Learner):
def predict(self, img:Image):
"Return prect class, label and probabilities for `img`."
ds = self.data.valid_ds
ds.set_item(img)
res = self.pred_batch()[0]
ds.clear_item()
pred_max = res.argmax()
return self.data.classes[pred_max],pred_max,res
def create_cnn(data:DataBunch, arch:Callable, cut:Union[int,Callable]=None, pretrained:bool=True,
You may need to do -
import torch.nn.functional as F
F.softmax(outputs)
To get probabilities. But I agree either the code in the predict function needs to be updated or the doc string in the function needs to be updated. Will wait for what others think or if I am missing something.
5 Likes
wgpubs
(WG)
November 6, 2018, 6:42pm
3
Here is where it gets more strange …
When I do fastai the hard way (FATHW), I get different probabilities than when I do it using the provided helper methods.
FATHW:
train_tfms, val_tfms = get_transforms()
img = apply_tfms(val_tfms.append(normalize_funcs(*imagenet_stats)), img, size=224)
backbone = create_body(models.resnet34(pretrained=False), -2)
head = create_head(num_features_model(backbone) * 2, 2)
m = nn.Sequential(backbone, head)
m.load_state_dict(torch.load(PATH/'models/stage-2-34.pth'))
m.eval()
log_probs = m(img.data.unsqueeze_(0))
print(log_probs.data)
# => tensor([[-1.9587, 2.3774]])
torch.softmax(log_probs.data, -1).numpy()
# => array([[0.012919, 0.987081]], dtype=float32)
… when I do it with the helper methods I see
classes = ['in', 'out']
data2 = ImageDataBunch.single_from_classes(PATH, classes, tfms=get_transforms(), size=224).normalize(imagenet_stats)
learn = create_cnn(data2, models.resnet34, pretrained=False)
learn.load('stage-2-34')
pred_class, pred_idx, outputs = learn.predict(img)
pred_class, pred_idx, outputs
# => ('out', tensor(1), tensor([2.0508e-02, 2.6639e+01]))
torch.softmax(outputs, -1)
# => tensor([2.7533e-12, 1.0000e+00])
Any ideas as to what is going on?
-wg
Lankinen
(Lankinen)
November 6, 2018, 7:25pm
4
Could it just be random weights? Like if others start closer to real result and that way converge faster in those steps.
KarlH
(Karl)
November 6, 2018, 9:32pm
5
The only thing I can think of is that the image is augmented differently each time it is passed to a model. The two cases you present got different augmented versions of the image.
You could check this by running the same image through the same model version several times.
wgpubs
(WG)
November 6, 2018, 9:33pm
6
But there shouldn’t be any augmentation at inference time … only during training.
wgpubs
(WG)
November 6, 2018, 10:56pm
7
There are no random weights … its using a fully trained model via learn.load
wgpubs
(WG)
November 7, 2018, 2:05am
8
ramesh:
You may need to do -
import torch.nn.functional as F
F.softmax(outputs)
To get probabilities. But I agree either the code in the predict function needs to be updated or the doc string in the function needs to be updated. Will wait for what others think or if I am missing something.
I do that but the different results still don’t make sense to me yet …
outputs
#=> tensor([3.1907e+02, 9.0539e-03])` ...
F.softmax(outputs, -1)
#=> tensor([1., 0.])
… but, when I run my custom code, I get values that look more sensible to me
m.eval()
log_probs = m(img.data.unsqueeze_(0))
#=> tensor([[ 4.1487, -3.3426]])
torch.softmax(log_probs.data, -1)
#=> tensor([9.994424e-01, 5.576377e-04])
Still can’t figure out why they are different or which to trust
balnazzar
(Andrea de Luca)
November 7, 2018, 3:02am
9
Not true: search for TTA()
(however it’s not the present case).
wgpubs:
pred_class, pred_idx, outputs = learn.predict(img)
# outputs => tensor([15.9323, 0.1261])
The documentation describes outputs
as probabilities … but these don’t look like probabilities.
As you guys correctly figured out, you were looking at logarthms of probabilities. No wonder that to get the actual probs, you need to exponentiate those values (softmax & co).
1 Like
wgpubs
(WG)
November 7, 2018, 7:05pm
10
At inference time you are predicting a single image, why would you apply any any transformation whatsoever? The goal of transforms is to provide more looks at your training examples during training .
balnazzar
(Andrea de Luca)
November 7, 2018, 7:08pm
11
Predicting over slight variations of the test image, and then average those predictions. It gives better results than a single prediction.
wyquek
(Qbiwan)
November 8, 2018, 12:40am
12
I’m interested to know if you have resolved the discrepancy between your custom code and the other code. Are you able to get the same answer?
wgpubs
(WG)
November 8, 2018, 12:44am
13
No I haven’t.
The results I’m seeing from my code look more like what I am expecting. Perhaps @jeremy can shed some light on what is going on and which is to be trusted.
jeremy
(Jeremy Howard)
November 8, 2018, 12:47am
14
As @balnazzar mentioned, I suspect you’re just missing a log().
wgpubs
(WG)
November 8, 2018, 12:56am
15
Yah I tried …
classes = ['in', 'out']
data2 = ImageDataBunch.single_from_classes(PATH, classes, tfms=get_transforms(), size=224).normalize(imagenet_stats)
learn = create_cnn(data2, models.resnet34, pretrained=False)
learn.load('stage-2-34')
pred_class, pred_idx, outputs = learn.predict(img)
torch.softmax(log(outputs), -1)
The results are still inconsistent (although much closer). My OCD self is driving me crazy because it just seems like either approach should yield the same result … but perhaps that isn’t the case for some reason unbeknownst to me.
2 Likes
Probably a silly question with a simple answer, but when I do learn.predict(img) on single images it calls up a pbar and I can’t seem to find how to prevent that. Is there a way?
cwerner
(Christian Werner)
November 8, 2018, 9:37pm
18
Oh wow. I think I missed the log() when using output and torch softmax, too! Now the results look much more what I would expect!
If have this now in my code:
pred_class, pred_idx, outputs = learner.predict(img)
probs = torch.nn.functional.softmax(np.log(outputs), dim=0)]
2 Likes
wyquek
(Qbiwan)
November 9, 2018, 4:52am
19
I don’t know, embarrassingly I’m not on fastai.v1 yet. You’re probably way ahead of me and I need to catch up