I feel doing
learn = cnn_learner(dls, resnet34, pretrained=True, metrics=[partial(accuracy_multi, thresh=0.95)], loss_func=BCEWithLogitsLossFlat(thresh=0.95)).to_fp16()
is dangerous in the sense that the model feels incentivized to do very sure predictions to minimize the loss during the training. I would like it during prediction only
Yes, so predict, when it works, focuses on the loss_func's decodes and activation in order to determine what category it is. For example, if you had no metric whatsoever, predict would still work exactly the same, and this is the reason why. Metrics != inference, as a metric still needs a y to work, doesnāt it?
So when we define our loss (ourselves, not by fastai) as such (for exactly where see here:
loss_func = BCEWithLogitsLossFlat(thresh=0.5)
0.5 is the default. So weād want to adjust this threshold to what we want to see. Also ideally youād make this and the metricās threshold to be exactly the same.
Why is this important? Remember that decodes and activation I brought up? When we do learn.predict, we take the raw output of our model, pass it through the activation of that function and then the decodes of that function as well. For BCE, they look like so:
def decodes(self, x): return x>self.thresh
def activation(self, x): return torch.sigmoid(x)
So we have our sigmoid activation followed by our threshold to scale on
Then adjust learn.loss_func.thresh to be what youād like, thatās all thatās needed 
Also, to expand on that predict I brought up, activation is always called, decodes is called when with_decoded=True
This help clear up some confusion?
Ah, thanks a lot! I am missing here one key part. When I look at the code of BCE (or the snipped you pasted) what are exactly the decodes and activations? I understand that we need a number between 0 and 1 (sigmoid takes care of this) and that we are only take it as valid if > thesh
but in the code I do not see how the 2 play together, it seems to me both return independently
I tried checking at the learn.predict?? code but I do not see how to access the gets_preds function inside it 
So letās walk through predict, specifically where it calls get_preds:
inp,preds,_,dec_preds = self.get_preds(dl=dl, with_input=True, with_decoded=True)
We see here we get back our input passed in, and with_decoded. This decoded is for the loss function only. decode_batch then decodes from our DataBlock.
For a different way to look at it, take my fastinference library Iāve been building. I rebuilt the get_preds function to make it a bit more efficient, but for all intensive purposes it still acts and behaves the same way the framework does:
for batch in dl:
with torch.no_grad():
...
if decoded_loss or fully_decoded:
out = x.model(*batch[:x.dls.n_inp])
raw.append(out)
dec_out.append(x.loss_func.decodes(out))
else:
raw.append(x.model(*batch[:x.dls.n_inp]))
This is how I get predictions (this is all hidden inside get_preds and the GatherPreds callback, so itās hard to figure out. Presume itās this with a bit more abstractness)
So we can see that if I want to decode my loss, I decode via the loss_func. At the end Iāll go through and get a result similar to predict here:
if not raw_outs:
try: outs.insert(0, x.loss_func.activation(tensor(raw)).numpy())
except: outs.insert(0, dec_out)
else:
outs.insert(0, raw)
if fully_decoded: outs = _fully_decode(x.dls, inps, outs, dec_out, is_multi)
if decoded_loss: outs = _decode_loss(x.dls.vocab, dec_out, outs)
return outs
(And Iām going to do a video walkthrough of this whole thing this weekend too, that may help some.)
Thatās a long explanation but does this help? To see _fully_decode and _decode_loss, see here, Iām going to go through those in the video, just a lot to explain here (but itās good to keep in mind because the fastai framework operates in this same way!)
Also, my version has an option return a fully decoded output similar to learn.predict, thatās why it will look different.
Thatās fantastic!! Pitty I can only give it a single 
Are you having any more videos sessions? Any fastai related study group link. I was quite busy over the past month (job change) but I would like to get fully back to it!!
On another note, for the ones reading this thread. This is how I solved the problem.
learn = cnn_learner(dls, resnet34, pretrained=True, metrics=[partial(accuracy_multi, thresh=0.95)], loss_func=BCEWithLogitsLossFlat(thresh=0.5)).to_fp16()
learn.loss_func = BCEWithLogitsLossFlat(thresh=0.95)
learn.predict('real-ai.jpg')
So basically, I am using for the metric the threshold I will use during evaluation to have an idea of how the model would behave. However, during training I use the 0.5 threshold to not incentivate extreme predictions. Hope this makes sense! Learning myself still.
Yes and no, theyāre unrelated to the course, and generally just YouTube uploads. I may make the next one live if people are interested, this weekend Iāll be doing a video on my fastinference library (where all that was taken from), so if people are interested in that being live let me know
I would be interested in both, really like your teaching style 
I just need to find a better way not to miss out on them!
Coming back to the question about how to add the test dataset in the test dataloaderā¦ thatās what I got so far. I tried several things but something is wrong
IIRC test_dl now is an is_labelled parameter. let me go see real quick
Edit: @mgloria yes, it has an with_labels parameter now. Pass True:
test_dl = data.test_dl(fnames, with_labels=True)
What is data in your code? In my case I have a folder called test with subfolders cat and dog which I would like to ignore. I edited my code with with_labels=True but I still missing something.
path = Path('../Cat_Dog_data/test')
items = get_image_files(path)
dsets = Datasets(items, tfms=[[PILImage.create], [parent_label, MultiCategoryBlock]])
dls = dsets.dataloaders(after_item=item_tfms, after_batch=batch_tfms, bs=64, with_labels=True)
~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/fastai2/data/transforms.py in __init__(self, vocab)
272 "Transform of one-hot encoded multi-category that decodes with `vocab`"
273 loss_func,order=BCEWithLogitsLossFlat(),1
--> 274 def __init__(self, vocab): self.vocab,self.c = vocab,len(vocab)
275 def encodes(self, o): return TensorMultiCategory(tensor(o).float())
276 def decodes(self, o): return MultiCategory (one_hot_decode(o, self.vocab))
TypeError: object of type 'NoneType' has no len()Your issue is youāre including with_labels in .dataloaders. Itās purely a dls.test_dl parameter. (Partially), let me see what else is going on
(Iām also recreating this in the pets dataset since itās the closest to what yoāre doing)
Edit: @mgloria use the Block methods on the DataBlock, and their interior on the Datasets. IE:
dsets = Datasets(items, tfms=[[PILImage.create], [parent_label, MultiCategorize()]])
Of course! that would be great. Your lectures gave me an in-depth understanding of what happens behind the screen.
Alright, letās go all in then. Weāll do my fastinference walkthrough on Sunday, time will be TBA but presume similar to how the time was for the course (I will announce a time in the next few minutes)
How does this combine with ```
dsets = Datasets(items, tfms=[[PILImage.create], [parent_label, MultiCategorize()]])
It doesnāt
So the process here is:
DatasetsDataLoaderstest_dlSo in code,
dsets = Datasets(items, tfms=[[PILImage.create], [parent_label, MultiCategorize()]])
dls = dsets.dataloaders(after_item, after_batch, etc)
test_dl = dls.test_dl(fnames)
yes, it is the pets datasetā¦ I thought about trying something easy but I cannot get the test part to work ā¦ 
!wget https://s3.amazonaws.com/content.udacity-data.com/nd089/Cat_Dog_data.zip
something is still missing but with this error message I find it very difficult to debug 
where does the with_label goes?
So, thereās a few bits youāre missing Our Blocks that we normally do add two super important transforms:
ToTensorIntToFloatTensorThese are very very important transforms and do just what they describe. Here is your full pipeline with that:
item_tfms = [Resize(460), ToTensor()]
batch_tfms = [IntToFloatTensor(), *aug_transforms(size=224), Normalize.from_stats(*imagenet_stats)]
dsets = Datasets(items, tfms=[[PILImage.create], [parent_label, MultiCategorize()]])
dls = dset.dataloaders(after_item = item_tfms, after_batch=batch_tfms, bs=64
From here building the test_dl is:
test_dl = dls.test_dl(get_image_files(path), with_labels=True)
(Also sorry I left out the with_labels part originally there)
@muellerzr Hi Zach, I have been following your videos and getting some very good info out of them! Thank you very much! In particular, I have been trying to follow the style transfer video and the reason is that Jeremy uses a similar approach in the Course V3 Lesson about No-GAN super-resolution using the same Feature preservation analogy you use in your video. I have an application where Iām trying to use UNETs to remove some noise from CT data and that approach worked great for me. However, Iām trying to follow what you did to do a FastAI v2 version of the same nb but Iām not very experienced and Iām getting lost in the details. I can see where all the basic blocks are but I canāt piece them together. Do you have any advice on how I can structure the FeatureLoss module to use on this application? Any advice is appreciated