Get_preds returning less results than length of original dataset


(Tyler Morgan) #1

I set up a tabular learner with with 30k samples, 20k of which I set aside for validation, giving me 10k for training. When I fit the model and return the training predictions I get a list with two tensors both with 9984 elements, instead of 10k as I’d expect. When I run it for the validation set I get 20k elements.

I’m using the following method calls to get the predictions for training and validation respectively:
learn.get_preds(ds_type=DatasetType.Train) learn.get_preds(ds_type=DatasetType.Valid)

I’m not sure what’s going on. I’m assuming that the learner is dropping rows for some reason – maybe as part of the preprocessing? I’m hoping that someone can lead me towards why this happening.

Any help would be appreciated.


#2

The training dataloader drops the last batch if it doesn’t have bs elements. That’s because small batches lead to instability in training, particularly with BatchNorm. With a batch of size 1 we would even get an error from pytorch.

If you want all your training data, you can ask DataSet.Fix, which is the training set with shuffle=False and drop_last=False, ideal for evaluation.


NLP prediction - predict batch
(Tyler Morgan) #3

Thanks for the knowledge! I’ll give this a shot.


(Tyler Morgan) #4

Here’s some clarification for anyone else coming alter – DataSet.Fix is the training set and it already has shuffle=False and drop_last=False. I read the original comment incorrectly as I needed to set run DataSet.Fix and set shuffle=False and drop_last=False somewhere. Just run get_preds(ds_type=DatasetType.Fix) on your learner and you’re good to go!

TIL trying to compare unshuffled actuals to shuffled predictions gives you some really terrible results:sweat_smile:


(Avi A) #5

Thanks! This is exactly what I needed as well. Does this work predict only on the data in the training set but also on the data in the valid/test sets?


#6

The validation and test set aren’t shuffled.


(Avi A) #7

yes, that makes a lot of sense. Thanks Sylvain!


(Coral) #9

I was able to run
learn.get_preds(ds_type=DatasetType.Train, with_loss=True) but changing to DatasetType.Fix generated AttributeError: Fix below. Could you help advise anything else I need to check?

preds,y,losses = learn.get_preds(ds_type=DatasetType.Fix)


AttributeError Traceback (most recent call last)
in
----> 1 preds,y,losses = learn.get_preds(ds_type=DatasetType.Fix)

/opt/anaconda3/lib/python3.7/enum.py in getattr(cls, name)
344 return cls.member_map[name]
345 except KeyError:
–> 346 raise AttributeError(name) from None
347
348 def getitem(cls, name):

AttributeError: Fix


(Arkar Aung) #10

Does this happen for test dataset? Particularly for scenario where I would load a learner with load_learner and then adding a test folder with test parameter in the load_learner. Then getting the predictions with preds, _ = learner.get_preds(ds_type=DatasetType.Test)

If it happens, how can I ensure that it does not happen for my Test set? I would need every row to be predicted in test set even if it overflows by 1 instance after batching up.


#11

If you have this error, it probably means you don’t have the latest version of fastai. Just tried without any problem.

No, only for the training set.


(Coral) #12

Thank you for letting me know that my FastAI was not updated.

I am on GCP, and I finally found the issue. I have been updating my FastAI library using the command below, which in my newbie mind, would have updated all the libraries as confirmed by the message returned below.

Turns out that the FastAI library was only updated to version 1.0.30 (something like that), and I followed the instructions [here], (sudo /opt/anaconda3/bin/conda install fastai -c fastai -c pytorch -c conda-forge)(Update fastai with conda doesn't update to the latest version) to update to the latest and the get_preds(ds_type= DatasetType.Fix `) works now.

jupyter@fastai-sg:~$ sudo /opt/anaconda3/bin/conda install -c fastai fastai
Collecting package metadata: done
Solving environment: done

All requested packages already installed.