AttributeError: 'MultiCategoryProcessor' object has no attribute 'one_hot'

Ankit_Doshi · December 28, 2018, 11:39am

Hi,
I m getting the error while running this line of code on Planet’s data.

> sd = LabelLists.load_empty(path).add_test_folder(‘train-jpg’)
path contains export.pkl and the folder

>     ---------------------------------------------------------------------------
    > AttributeError                            Traceback (most recent call last)
    > <ipython-input-24-daa621da3f2e> in <module>
    > ----> 1 sd = LabelLists.load_empty(path).add_test_folder('train-jpg')
    >       2 empty_data = sd.databunch()
    > 
    > /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in add_test_folder(self, test_folder, label)
    >     485         # note: labels will be ignored if available in the test dataset
    >     486         items = self.x.__class__.from_folder(self.path/test_folder)
    > --> 487         return self.add_test(items.items, label=label)
    >     488 
    >     489     @classmethod
    > 
    > /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in add_test(self, items, label)
    >     478         labels = [label] * len(items)
    >     479         if isinstance(items, ItemList): self.test = self.valid.new(items.items, labels, xtra=items.xtra)
    > --> 480         else: self.test = self.valid.new(items, labels)
    >     481         return self
    >     482 
    > 
    > /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in new(self, x, y, **kwargs)
    >     526             return self.__class__(x, y, tfms=self.tfms, tfm_y=self.tfm_y, **self.tfmargs)
    >     527         else:
    > --> 528             return self.new(self.x.new(x, **kwargs), self.y.new(y, **kwargs)).process()
    >     529 
    >     530     def __getattr__(self,k:str)->Any:
    > 
    > /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in process(self, xp, yp, filter_missing_y)
    >     584     def process(self, xp:PreProcessor=None, yp:PreProcessor=None, filter_missing_y:bool=False):
    >     585         "Launch the processing on `self.x` and `self.y` with `xp` and `yp`."
    > --> 586         self.y.process(yp)
    >     587         if filter_missing_y and (getattr(self.x, 'filter_missing_y', None)):
    >     588             filt = array([o is None for o in self.y])
    > 
    > /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in process(self, processor)
    >      66         if processor is not None: self.processor = processor
    >      67         self.processor = listify(self.processor)
    > ---> 68         for p in self.processor: p.process(self)
    >      69         return self
    >      70 
    > 
    > /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in process(self, ds)
    >     288         ds.classes = self.classes
    >     289         ds.c2i = self.c2i
    > --> 290         super().process(ds)
    >     291 
    >     292     def __getstate__(self): return {'classes':self.classes}
    > 
    > /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in process(self, ds)
    >      36     def __init__(self, ds:Collection=None):  self.ref_ds = ds
    >      37     def process_one(self, item:Any):         return item
    > ---> 38     def process(self, ds:Collection):        ds.items = array([self.process_one(item) for item in ds.items])
    >      39 
    >      40 class ItemList():
    > 
    > /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in <listcomp>(.0)
    >      36     def __init__(self, ds:Collection=None):  self.ref_ds = ds
    >      37     def process_one(self, item:Any):         return item
    > ---> 38     def process(self, ds:Collection):        ds.items = array([self.process_one(item) for item in ds.items])
    >      39 
    >      40 class ItemList():
    > 
    > /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in process_one(self, item)
    >     329 
    >     330     def process_one(self,item):
    > --> 331         if self.one_hot: return item
    >     332         return [super(MultiCategoryProcessor, self).process_one(o) for o in item]
    >     333 
    > 
    > AttributeError: 'MultiCategoryProcessor' object has no attribute 'one_hot'

This is the related Class and the function to this error -

class MultiCategoryProcessor(CategoryProcessor):
    "`PreProcessor` that create `classes` from `ds.items` and handle the mapping."
    def __init__(self, ds:ItemList, one_hot:bool=False): 
        super().__init__(ds)
        self.one_hot = one_hot
                
    def process_one(self,item): 
        if self.one_hot: return item
        return [super(MultiCategoryProcessor, self).process_one(o) for o in item]

I m on the current master branch.

Ankit_Doshi · December 28, 2018, 1:18pm

When I checked the sd object which has taken data from export.pkl. I think this is where the problem is

Capture

sgugger · December 28, 2018, 2:27pm

Are you sure it’s nor because your export file was before this change (you’d need to re-export)? I just tried the planet example and it’s running fine.

Ankit_Doshi · December 28, 2018, 4:21pm

Thanks for a quick one. Here is a snapshot as to what I m doing.
What am I doing wrong?

sgugger · December 28, 2018, 4:50pm

I’m sorry but I don’t see any bug in your code. What’s the problem?

Ankit_Doshi · December 28, 2018, 5:12pm

The idea is to deploy a model on different device and unseen incoming data. I learnt from this link https://docs.fast.ai/tutorial.inference.html that we require export.pkl along with the model.

I am solving a multilabel classification problem which is similar to planet data.

I m trying to find activations for images uploaded by client and compare them to the activations of the images in the catalog to find nearest neighbors.

If you can suggest any other way to approach this problem is also appreciated

Regards

sgugger · December 28, 2018, 5:16pm

The code you showed just created the empty DataBunch. It’s there, and you can normally create a Learner, load your model then feed it your new data.
If your problem is that you don’t have the old data, you shouldn’t use load_empty.

Ankit_Doshi · December 28, 2018, 6:48pm

The code you showed just created the empty DataBunch. It’s there, and you can normally create a Learner, load your model then feed it your new data.

Yes I can create a learner and load the model, but how do I add data to it?
I thought we first have to add data to the empty DataBunch object by doing something like add_test_folder() as mentioned in tutorial and than create learner to load model.
How do I add data after?

If your problem is that you don’t have the old data, you shouldn’t use load_empty .

The idea is to -

Store incoming images to a folder
Pass the folder as test data to the empty DataBunch object
Create learner, Load model
Get activations using get_actns func from widgets

Though the images which I have used to fine tune the model are the same looking like the ones I will receive in production but they are not required. Thats why I tried load_empty.

sgugger · December 28, 2018, 7:34pm

Then you can use add_test to add your data, like in the tutorial.

Ankit_Doshi · December 28, 2018, 8:37pm

Actually! that was the original post.

I was getting -

AttributeError: ‘MultiCategoryProcessor’ object has no attribute ‘one_hot’

when I used add_test_folder.
sd = LabelLists.load_empty(path).add_test_folder(‘train-jpg’)

If you intended to tell me to use add_test instead, isnt that same as add_test_folder?

sgugger · December 29, 2018, 8:35am

Now I understand. There was a bug indeed, just fixed it in this commit.

Ankit_Doshi · December 29, 2018, 10:58am

Hi,
Thanks, that solved the

> AttributeError: ‘MultiCategoryProcessor’ object has no attribute ‘one_hot’

But it raised a new error - 
> ---------------------------------------------------------------------------
> TypeError                                 Traceback (most recent call last)
> <ipython-input-26-61646fd53b00> in <module>
> ----> 1 sd = LabelLists.load_empty(path).add_test_folder('train-jpg')
>       2 #empty_data = sd.databunch()
> 
> /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in add_test_folder(self, test_folder, label)
>     492         # note: labels will be ignored if available in the test dataset
>     493         items = self.x.__class__.from_folder(self.path/test_folder)
> --> 494         return self.add_test(items.items, label=label)
>     495 
>     496     @classmethod
> 
> /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in add_test(self, items, label)
>     485         labels = [label] * len(items)
>     486         if isinstance(items, ItemList): self.test = self.valid.new(items.items, labels, xtra=items.xtra)
> --> 487         else: self.test = self.valid.new(items, labels)
>     488         return self
>     489 
> 
> /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in new(self, x, y, **kwargs)
>     533             return self.__class__(x, y, tfms=self.tfms, tfm_y=self.tfm_y, **self.tfmargs)
>     534         else:
> --> 535             return self.new(self.x.new(x, **kwargs), self.y.new(y, **kwargs)).process()
>     536 
>     537     def __getattr__(self,k:str)->Any:
> 
> /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in process(self, xp, yp, filter_missing_y)
>     591     def process(self, xp:PreProcessor=None, yp:PreProcessor=None, filter_missing_y:bool=False):
>     592         "Launch the processing on `self.x` and `self.y` with `xp` and `yp`."
> --> 593         self.y.process(yp)
>     594         if filter_missing_y and (getattr(self.x, 'filter_missing_y', None)):
>     595             filt = array([o is None for o in self.y])
> 
> /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in process(self, processor)
>      66         if processor is not None: self.processor = processor
>      67         self.processor = listify(self.processor)
> ---> 68         for p in self.processor: p.process(self)
>      69         return self
>      70 
> 
> /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in process(self, ds)
>     288         ds.classes = self.classes
>     289         ds.c2i = self.c2i
> --> 290         super().process(ds)
>     291 
>     292     def __getstate__(self): return {'classes':self.classes}
> 
> /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in process(self, ds)
>      36     def __init__(self, ds:Collection=None):  self.ref_ds = ds
>      37     def process_one(self, item:Any):         return item
> ---> 38     def process(self, ds:Collection):        ds.items = array([self.process_one(item) for item in ds.items])
>      39 
>      40 class ItemList():
> 
> /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in <listcomp>(.0)
>      36     def __init__(self, ds:Collection=None):  self.ref_ds = ds
>      37     def process_one(self, item:Any):         return item
> ---> 38     def process(self, ds:Collection):        ds.items = array([self.process_one(item) for item in ds.items])
>      39 
>      40 class ItemList():
> 
> /opt/anaconda3/lib/python3.7/site-packages/fastai/data_block.py in process_one(self, item)
>     337     def process_one(self,item):
>     338         if self.one_hot: return item
> --> 339         return [super(MultiCategoryProcessor, self).process_one(o) for o in item]
>     340 
>     341     def generate_classes(self, items):
> 
> TypeError: 'NoneType' object is not iterable

And I think this one is related to ‘export.pkl’ file.

Capture

Whats could be the reason to get this error?

sgugger · December 29, 2018, 1:56pm

That’s now linked to your test data: you have to pass a label in your add_test function since the default doesn’t seem to work and it can’t take the first one of the training set anymore (that’s what it does when the data isn’t empty). Just pass an array of zeros with the same number of classes as your regular target and you should be code.

Ankit_Doshi · December 29, 2018, 8:25pm

Problem Solved!

Instead of passing Zeros( which gave error) I passed the list of Labels and it worked-

I tried this before raising the issue here but it gave error. I think the bug that you fixed solved this error.

One last question-
Has the test data transformed with the arguments that we passed in get_transforms() during creating databunch object when we created our model?

sgugger · December 29, 2018, 9:29pm

Ah yes, it would need original labels, not 0s.
For your question, it’s using the transforms you passed for the validation set (so get_transforms()[1] if you used get_transforms).