A Brief Guide to Test Sets in v2 (you can do labelled now too!)

fastai fastai dev
22

@sgugger I am really struggling with how to do that easily? I’m getting to a point where it’s looking like this:

split += L(range(len(train_imgs), len(train_imgs)+len(tst_imgs)), use_list=True)

But this still does not work as it’s not showing up as an L:
(#3931) [(#7220) [0,1,2,3,4,5,6,7,8,9...],(#1805) [7220,7221,7222,7223,7224,7225,7226,7227,7228,7229...],9025,9026,9027,9028,9029,9030,9031,9032...]
And I can’t use index splitter as it just operates with 1 validation set

It seems easier to just do a TfmdList to me for this case

I think I understand what the general idea is though, something like when we create the Dataset, we do:

dsrc = Datasets(train_imgs+tst_imgs, tfms=[[PILImage.create], [parent_label, Categorize]],
                splits = split)

But splitting it to a third with just tst_imgs I am struggling with

23

You should append it, not use +.

24

I was getting issues appending it to the split due to it being tuples, not a list (one moment and I’ll show code)

I think for KFold it’s a bit easier to just generate the TfmdList as we just care about the same test set and shuffle the training/validation

Here is the issue with split:

idxs = list(range(start_val, len(train_imgs)))
splits = IndexSplitter(idxs)
split = splits(train_imgs)
split.append(list(range(0,1)))
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-135-696258b95d3e> in <module>()
----> 1 split.append(list(range(0,1)))

AttributeError: 'tuple' object has no attribute 'append'

I got past this by just wrapping it all into a list, thanks @sgugger :slight_smile:

25

Hi @muellerzr! Have you managed to use a Test Set with Datasets and Dataloaders? I am struggling to make it work for text inference with ULMFIT.

26

Yes I have @fmobrj75 ! An example is this notebook here:

This is for vision but there are parallels

1 Like
27

Thank you, very much!

28

In fact, I need to infer on new datasets after training. I am trying to export the learner, but I am getting an error (ULMFIT):

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-71-d396129ee191> in <module>
----> 1 learn.export(fname='export.pkl')

/media/hdd3tb/data/fastai2/fastai2/learner.py in export(self, fname)
    574     if rank_distrib(): return # don't export if slave proc
    575     old_dbunch = self.dls
--> 576     self.dls = self.dls.new_empty()
    577     state = self.opt.state_dict()
    578     self.opt = None

/media/hdd3tb/data/fastai2/fastai2/data/core.py in new_empty(self)
    112     def __getitem__(self, i): return self.loaders[i]
    113     def new_empty(self):
--> 114         loaders = [dl.new(dl.dataset.new_empty()) for dl in self.loaders]
    115         return type(self)(*loaders, path=self.path, device=self.device)
    116 

/media/hdd3tb/data/fastai2/fastai2/data/core.py in <listcomp>(.0)
    112     def __getitem__(self, i): return self.loaders[i]
    113     def new_empty(self):
--> 114         loaders = [dl.new(dl.dataset.new_empty()) for dl in self.loaders]
    115         return type(self)(*loaders, path=self.path, device=self.device)
    116 

/media/hdd3tb/data/fastai2/fastai2/data/core.py in new(self, dataset, cls, **kwargs)
     53     @delegates(DataLoader.new)
     54     def new(self, dataset=None, cls=None, **kwargs):
---> 55         res = super().new(dataset, cls, **kwargs)
     56         if not hasattr(self, '_n_inp') or not hasattr(self, '_types'):
     57             self._one_pass()

/media/hdd3tb/data/fastai2/fastai2/data/load.py in new(self, dataset, cls, **kwargs)
    112                           bs=self.bs, shuffle=self.shuffle, drop_last=self.drop_last, indexed=self.indexed, device=self.device)
    113         for n in self._methods: cur_kwargs[n] = getattr(self, n)
--> 114         return cls(**merge(cur_kwargs, kwargs))
    115 
    116     @property

/media/hdd3tb/data/fastai2/fastai2/text/data.py in __init__(self, dataset, sort_func, res, **kwargs)
    146         if res is None and self.sort_func == _default_sort: res = _get_lengths(dataset)
    147         self.res = [self.sort_func(self.do_item(i)) for i in range_of(self.dataset)] if res is None else res
--> 148         self.idx_max = np.argmax(self.res)
    149 
    150     def get_idxs(self):

<__array_function__ internals> in argmax(*args, **kwargs)

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/numpy/core/fromnumeric.py in argmax(a, axis, out)
   1151 
   1152     """
-> 1153     return _wrapfunc(a, 'argmax', axis=axis, out=out)
   1154 
   1155 

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/numpy/core/fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     56     bound = getattr(obj, method, None)
     57     if bound is None:
---> 58         return _wrapit(obj, method, *args, **kwds)
     59 
     60     try:

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/numpy/core/fromnumeric.py in _wrapit(obj, method, *args, **kwds)
     45     except AttributeError:
     46         wrap = None
---> 47     result = getattr(asarray(obj), method)(*args, **kwds)
     48     if wrap:
     49         if not isinstance(result, mu.ndarray):

ValueError: attempt to get argmax of an empty sequence
29

@sgugger, in fastai v1 I could export a text_classifier_learner and then load it with a new databunch as this:

learn = load_learner(class_path,'model')
learn.data = data_clas
preds = learn.get_preds(ds_type=DatasetType.Test, ordered=True)

And bacth infer over thousands of records. Is it possible to do a similar procedure with v2?

For some reason I am facing an error while exporting the text_classifier_learner.

30

I can’t really help without seeing the error.

31

Sorry, @sgugger. My bad. It was on the previous post, but I forgot to flag you. I trained a text_classifier_learner, and when I try to export it, I get the following error: Here it is:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-44-cb712095abd8> in <module>
----> 1 learn.export('teste.pkl')

/media/hdd3tb/data/fastai2/fastai2/learner.py in export(self, fname)
    580     if rank_distrib(): return # don't export if slave proc
    581     old_dbunch = self.dls
--> 582     self.dls = self.dls.new_empty()
    583     state = self.opt.state_dict()
    584     self.opt = None

/media/hdd3tb/data/fastai2/fastai2/data/core.py in new_empty(self)
    117     def __getitem__(self, i): return self.loaders[i]
    118     def new_empty(self):
--> 119         loaders = [dl.new(dl.dataset.new_empty()) for dl in self.loaders]
    120         return type(self)(*loaders, path=self.path, device=self.device)
    121 

/media/hdd3tb/data/fastai2/fastai2/data/core.py in <listcomp>(.0)
    117     def __getitem__(self, i): return self.loaders[i]
    118     def new_empty(self):
--> 119         loaders = [dl.new(dl.dataset.new_empty()) for dl in self.loaders]
    120         return type(self)(*loaders, path=self.path, device=self.device)
    121 

/media/hdd3tb/data/fastai2/fastai2/data/core.py in new(self, dataset, cls, **kwargs)
     55     @delegates(DataLoader.new)
     56     def new(self, dataset=None, cls=None, **kwargs):
---> 57         res = super().new(dataset, cls, do_setup=False, **kwargs)
     58         if not hasattr(self, '_n_inp') or not hasattr(self, '_types'):
     59             try:

/media/hdd3tb/data/fastai2/fastai2/data/load.py in new(self, dataset, cls, **kwargs)
    112                           bs=self.bs, shuffle=self.shuffle, drop_last=self.drop_last, indexed=self.indexed, device=self.device)
    113         for n in self._methods: cur_kwargs[n] = getattr(self, n)
--> 114         return cls(**merge(cur_kwargs, kwargs))
    115 
    116     @property

/media/hdd3tb/data/fastai2/fastai2/text/data.py in __init__(self, dataset, sort_func, res, **kwargs)
    146         if res is None and self.sort_func == _default_sort: res = _get_lengths(dataset)
    147         self.res = [self.sort_func(self.do_item(i)) for i in range_of(self.dataset)] if res is None else res
--> 148         self.idx_max = np.argmax(self.res)
    149 
    150     def get_idxs(self):

<__array_function__ internals> in argmax(*args, **kwargs)

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/numpy/core/fromnumeric.py in argmax(a, axis, out)
   1151 
   1152     """
-> 1153     return _wrapfunc(a, 'argmax', axis=axis, out=out)
   1154 
   1155 

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/numpy/core/fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     56     bound = getattr(obj, method, None)
     57     if bound is None:
---> 58         return _wrapit(obj, method, *args, **kwds)
     59 
     60     try:

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/numpy/core/fromnumeric.py in _wrapit(obj, method, *args, **kwds)
     45     except AttributeError:
     46         wrap = None
---> 47     result = getattr(asarray(obj), method)(*args, **kwds)
     48     if wrap:
     49         if not isinstance(result, mu.ndarray):

ValueError: attempt to get argmax of an empty sequence
32

Are batch_tfms applied to the test_dl? Is there a possibility to choose which transform (if None) is applied to a test_dl?

33

Yes. You can do rm_type_tfms and play around with this. It usually accepts values between 0 and 2 for it. I haven’t played around with what each does, but there is somewhat of a way

34

The default value is None. Should it mean there is no transform by default? It doesnt seem so.

35

No, so it means by default it removes no transforms from your DataLoader

36

So what would 1 or 2 rm_type_tfms values do? This code is not too explicit for me: https://github.com/fastai/fastai2/blob/f3f8e6fe5ac5f345fdc14737d78f9cbbf2577bc6/fastai2/data/core.py#L317. And I dont find any documentation yet.

37

When creating a new test dl like following:

dl = test_dl(learn.dls, files, after_batch=[IntToFloatTensor, Normalize.from_stats(*imagenet_stats)])
dl.after_batch => Pipeline: IntToFloatTensor -> Normalize

I’m still getting augmentations (crops, resize, zooms, etc) on predictions:

inputs, preds, _, dec_preds = learn.get_preds(dl=dl, with_input=True, with_decoded=True)

(i.e: inputs & dec_preds image/mask are augmented)

How can I remove those augmentations?

@sgugger Am I doing something wrong?

38

What is your after_item transforms @lvaleriu?

39

Pipeline: AddMaskCodes -> Resize -> ToTensor

The resize will resize both image & mask to a square dimension (512x512).

40

I got it working this way:

In fact, once I noticed it was very easy. Very much like I was doing in fastai v1. I dont know if it is very ellegant, but it worked.

I created a new test dataframe with some dummy cases for the train split, and then:

test_splits = ColSplitter('is_valid')(df_test)
test_ds = Datasets(df_test, splits=test_splits, tfms=[x_tfms])
test_dl = test_ds.dataloaders(bs=tst_bs, cache=2048, num_workers=8, pin_memory=True, shuffle_train=True, before_batch=pad, seq_len=72)

learn.dls=test_dl
preds = learn.get_preds()
3 Likes
41

I have some confusion about the way get_preds works, at least for text.

Say I load my learner for inference:
learn = load_learner('albertlearner.pkl', cpu=False)

Create a simple test dataloader out of the first 2 rows of a dataframe,
dl = learn.dls.test_dl(df[0:2], device=torch.device('cuda'))

and get_preds from it:
preds = learn.get_preds(dl=dl)

preds[0] gives me the following:
tensor([[0.0995, 0.9005],
[0.0995, 0.9005]])

(which, incidentally, is consistent with what I get from learn.predict for these 2 items)

But if I grab 3 rows from my dataframe instead of 2 (df[0:3]):
dl = learn.dls.test_dl(df[0:3], device=torch.device('cuda'))

I instead get:

tensor([[0.4870, 0.5130],
[0.3985, 0.6015],
[0.3985, 0.6015]])

Results are identical if I feed the text from the dataframes in as lists. A similar problem occurs if I just make up my own list of strings and pass them in. I’m quite stumped about what would cause this dramatic shift in predictions. In V1, I’d have given up at this point and just made a loop that uses learn.predict. But in V2, learn.predict is several orders of magnitude slower.