I set up a tabular learner with with 30k samples, 20k of which I set aside for validation, giving me 10k for training. When I fit the model and return the training predictions I get a list with two tensors both with 9984 elements, instead of 10k as I’d expect. When I run it for the validation set I get 20k elements.
I’m using the following method calls to get the predictions for training and validation respectively: learn.get_preds(ds_type=DatasetType.Train) learn.get_preds(ds_type=DatasetType.Valid)
I’m not sure what’s going on. I’m assuming that the learner is dropping rows for some reason – maybe as part of the preprocessing? I’m hoping that someone can lead me towards why this happening.
The training dataloader drops the last batch if it doesn’t have bs elements. That’s because small batches lead to instability in training, particularly with BatchNorm. With a batch of size 1 we would even get an error from pytorch.
If you want all your training data, you can ask DataSet.Fix, which is the training set with shuffle=False and drop_last=False, ideal for evaluation.
Here’s some clarification for anyone else coming alter – DataSet.Fix is the training set and it already has shuffle=False and drop_last=False. I read the original comment incorrectly as I needed to set run DataSet.Fix and set shuffle=False and drop_last=False somewhere. Just run get_preds(ds_type=DatasetType.Fix) on your learner and you’re good to go!
TIL trying to compare unshuffled actuals to shuffled predictions gives you some really terrible results:sweat_smile:
Thanks! This is exactly what I needed as well. Does this work predict only on the data in the training set but also on the data in the valid/test sets?
I was able to run
learn.get_preds(ds_type=DatasetType.Train, with_loss=True) but changing to DatasetType.Fix generated AttributeError: Fix below. Could you help advise anything else I need to check?
Does this happen for test dataset? Particularly for scenario where I would load a learner with load_learner and then adding a test folder with test parameter in the load_learner. Then getting the predictions with preds, _ = learner.get_preds(ds_type=DatasetType.Test)
If it happens, how can I ensure that it does not happen for my Test set? I would need every row to be predicted in test set even if it overflows by 1 instance after batching up.
Thank you for letting me know that my FastAI was not updated.
I am on GCP, and I finally found the issue. I have been updating my FastAI library using the command below, which in my newbie mind, would have updated all the libraries as confirmed by the message returned below.
Turns out that the FastAI library was only updated to version 1.0.30 (something like that), and I followed the instructions [here], (sudo /opt/anaconda3/bin/conda install fastai -c fastai -c pytorch -c conda-forge)(Update fastai with conda doesn't update to the latest version) to update to the latest and the get_preds(ds_type= DatasetType.Fix `) works now.
@sgugger This happens for me for test dataset too. I use load_learner and then learner.data.add_test(). I have 288 examples in test set but when I use small batch sizes such as 2 or 4, the returned predictions from learner.get_preds() is less than 288 but when I increase the batch size to 8, everything works fine. Do you know why this is happening?
By the way my fastai version is 1.0.53.post3
Exactly same thing is happening with me. I’m getting correct predictions if use learn.predict(img) to get predicitons on each image 1 by 1. But if I use
"learn = load_learner(modelDir, test = test) preds, out= learn.get_preds(ds_type=DatasetType.Test) "
to get batch wise predictions then I’m getting the same output for all images.