I set up a tabular learner with with 30k samples, 20k of which I set aside for validation, giving me 10k for training. When I fit the model and return the training predictions I get a list with two tensors both with 9984 elements, instead of 10k as I’d expect. When I run it for the validation set I get 20k elements.
I’m using the following method calls to get the predictions for training and validation respectively:
I’m not sure what’s going on. I’m assuming that the learner is dropping rows for some reason – maybe as part of the preprocessing? I’m hoping that someone can lead me towards why this happening.
Any help would be appreciated.
The training dataloader drops the last batch if it doesn’t have bs elements. That’s because small batches lead to instability in training, particularly with BatchNorm. With a batch of size 1 we would even get an error from pytorch.
If you want all your training data, you can ask DataSet.Fix, which is the training set with
drop_last=False, ideal for evaluation.
Thanks for the knowledge! I’ll give this a shot.
Here’s some clarification for anyone else coming alter – DataSet.Fix is the training set and it already has shuffle=False and drop_last=False. I read the original comment incorrectly as I needed to set run DataSet.Fix and set shuffle=False and drop_last=False somewhere. Just run
) on your learner and you’re good to go!
TIL trying to compare unshuffled actuals to shuffled predictions gives you some really terrible results:sweat_smile:
Thanks! This is exactly what I needed as well. Does this work predict only on the data in the training set but also on the data in the valid/test sets?
The validation and test set aren’t shuffled.
yes, that makes a lot of sense. Thanks Sylvain!
I was able to run
learn.get_preds(ds_type=DatasetType.Train, with_loss=True) but changing to DatasetType.Fix generated AttributeError: Fix below. Could you help advise anything else I need to check?
preds,y,losses = learn.get_preds(ds_type=DatasetType.Fix)
AttributeError Traceback (most recent call last)
----> 1 preds,y,losses = learn.get_preds(ds_type=DatasetType.Fix)
/opt/anaconda3/lib/python3.7/enum.py in getattr(cls, name)
344 return cls.member_map[name]
345 except KeyError:
–> 346 raise AttributeError(name) from None
348 def getitem(cls, name):
Does this happen for test dataset? Particularly for scenario where I would load a learner with
load_learner and then adding a test folder with
test parameter in the
load_learner. Then getting the predictions with
preds, _ = learner.get_preds(ds_type=DatasetType.Test)
If it happens, how can I ensure that it does not happen for my Test set? I would need every row to be predicted in test set even if it overflows by 1 instance after batching up.
If you have this error, it probably means you don’t have the latest version of fastai. Just tried without any problem.
No, only for the training set.
Thank you for letting me know that my FastAI was not updated.
I am on GCP, and I finally found the issue. I have been updating my FastAI library using the command below, which in my newbie mind, would have updated all the libraries as confirmed by the message returned below.
Turns out that the FastAI library was only updated to version 1.0.30 (something like that), and I followed the instructions [here], (sudo /opt/anaconda3/bin/conda install fastai -c fastai -c pytorch -c conda-forge)(Update fastai with conda doesn't update to the latest version) to update to the latest and the
get_preds(ds_type= DatasetType.Fix `) works now.
jupyter@fastai-sg:~$ sudo /opt/anaconda3/bin/conda install -c fastai fastai
Collecting package metadata: done
Solving environment: done
All requested packages already installed.
@sgugger This happens for me for test dataset too. I use
load_learner and then
learner.data.add_test(). I have 288 examples in test set but when I use small batch sizes such as 2 or 4, the returned predictions from
learner.get_preds() is less than 288 but when I increase the batch size to 8, everything works fine. Do you know why this is happening?
By the way my fastai version is
Exactly same thing is happening with me. I’m getting correct predictions if use learn.predict(img) to get predicitons on each image 1 by 1. But if I use
"learn = load_learner(modelDir, test = test)
preds, out= learn.get_preds(ds_type=DatasetType.Test) "
to get batch wise predictions then I’m getting the same output for all images.
Had this problem as well. Solved it by replacing: