Multiple outputs

Part 1 (2020)
1

I am trying to train a model with multiple outputs of RegressionBlocks.
The dataset is composed of multiple records each of which is a JSON object which looks like:

{'_index': 74,
 '_timestamp': 1587330810,
 'cam/image_array': '/home/rahulrav/Workspace/Donkey/Simulator/Run-1_2/data/tub_2/images/74_cam_image_array_.jpg',
 'track/lap': 0,
 'track/loc': 0,
 'user/angle': 0.0,
 'user/mode': 'user',
 'user/throttle': 0.886987566947937,
 '_image_base_path': '/home/rahulrav/Workspace/Donkey/Simulator/Run-1_2/data/tub_2/images'}

I am trying to build a model where the input is an image and the output is 2 RegressionBlocks. Here is the DataBlock I defined:

def get_records(*args, **kwargs):
    return records

def get_x(record):
    return record['cam/image_array']

def get_y_1(record):
    return record['user/angle']

def get_y_2(record):
    return record['user/throttle']
    

block = DataBlock(blocks=(ImageBlock, RegressionBlock, RegressionBlock),
    n_inp=1, 
    get_items=get_records, 
    splitter=RandomSplitter(),
    get_x = get_x,
    get_y = [get_y_1, get_y_2],
    item_tfms=Resize(160)
)

loader = block.dataloaders('.')

Next, i tried to load a batch of data.

batch = loader.one_batch()
print(batch[0].shape)
print(batch[1].shape)
print(batch[2].shape)

Gives me:

torch.Size([64, 3, 160, 160])
torch.Size([64])
torch.Size([64])

When I try and create a Learner for the dataloaders I run into the following problem:

learner = cnn_learner(loader, resnet34, loss_func=mse)

TypeError                                 Traceback (most recent call last)
<ipython-input-98-667b5ed52417> in <module>
----> 1 learner = cnn_learner(loader, resnet34, loss_func=mse)

~/Workspace/FastAi/fastcore/fastcore/utils.py in _f(*args, **kwargs)
    424         log_dict = {**func_args.arguments, **{f'{k} (not in signature)':v for k,v in xtra_kwargs.items()}}
    425         log = {f'{f.__qualname__}.{k}':v for k,v in log_dict.items() if k not in but}
--> 426         inst = f(*args, **kwargs) if to_return else args[0]
    427         init_args = getattr(inst, 'init_args', {})
    428         init_args.update(log)

~/Workspace/FastAi/fastai2/fastai2/vision/learner.py in cnn_learner(dls, arch, loss_func, pretrained, cut, splitter, y_range, config, n_out, normalize, **kwargs)
    174     if normalize: _add_norm(dls, meta, pretrained)
    175     if y_range is None and 'y_range' in config: y_range = config.pop('y_range')
--> 176     model = create_cnn_model(arch, n_out, ifnone(cut, meta['cut']), pretrained, y_range=y_range, **config)
    177     learn = Learner(dls, model, loss_func=loss_func, splitter=ifnone(splitter, meta['split']), **kwargs)
    178     if pretrained: learn.freeze()

~/Workspace/FastAi/fastai2/fastai2/vision/learner.py in create_cnn_model(arch, n_out, cut, pretrained, n_in, init, custom_head, concat_pool, **kwargs)
    102     if custom_head is None:
    103         nf = num_features_model(nn.Sequential(*body.children())) * (2 if concat_pool else 1)
--> 104         head = create_head(nf, n_out, concat_pool=concat_pool, **kwargs)
    105     else: head = custom_head
    106     model = nn.Sequential(body, head)

~/Workspace/FastAi/fastai2/fastai2/vision/learner.py in create_head(nf, n_out, lin_ftrs, ps, concat_pool, bn_final, lin_first, y_range)
     85     if lin_first: layers.append(nn.Dropout(ps.pop(0)))
     86     for ni,no,p,actn in zip(lin_ftrs[:-1], lin_ftrs[1:], ps, actns):
---> 87         layers += LinBnDrop(ni, no, bn=True, p=p, act=actn, lin_first=lin_first)
     88     if lin_first: layers.append(nn.Linear(lin_ftrs[-2], n_out))
     89     if bn_final: layers.append(nn.BatchNorm1d(lin_ftrs[-1], momentum=0.01))

~/Workspace/FastAi/fastai2/fastai2/layers.py in __init__(self, n_in, n_out, bn, p, act, lin_first)
    168         layers = [BatchNorm(n_out if lin_first else n_in, ndim=1)] if bn else []
    169         if p != 0: layers.append(nn.Dropout(p))
--> 170         lin = [nn.Linear(n_in, n_out, bias=not bn)]
    171         if act is not None: lin.append(act)
    172         layers = lin+layers if lin_first else layers+lin

~/.virtualenvs/torch/lib/python3.6/site-packages/torch/nn/modules/linear.py in __init__(self, in_features, out_features, bias)
     70         self.in_features = in_features
     71         self.out_features = out_features
---> 72         self.weight = Parameter(torch.Tensor(out_features, in_features))
     73         if bias:
     74             self.bias = Parameter(torch.Tensor(out_features))

TypeError: new() received an invalid combination of arguments - got (L, int), but expected one of:
 * (torch.device device)
 * (torch.Storage storage)
 * (Tensor other)
 * (tuple of ints size, torch.device device)
      didn't match because some of the arguments have invalid types: (L, int)
 * (object data, torch.device device)
      didn't match because some of the arguments have invalid types: (L, int)

Any ideas on what I might be doing wrong ?

1 Like
2

Are those ints or floats? Do a print(batch[1]) and a print(batch[2]) and make sure they are the later (if the are ints I believe fastai will default to assuming you are working on a classification problem, not a regression). You may need to add this transform: https://dev.fast.ai/data.transforms#IntToFloatTensor.

3

They are floats.

4

If you try doing a single regression, does it work?

If it does, I’m thinking you may have to write your own loss function that hands multiple regressions (I’m not sure if MSELossFlat is going to work out-of-the-box for you here).

5

I just tried a single regression, and that works as expected.

Usually a loss_fn has a signature that looks something like:

def loss_fun(predictions, actual):
   # ...

For the case of multiple outputs, are predictions and actual are going to a tuple of multiple values ?
Also, is there an API I should be using to get more information about the shape of the parameters being passed to the loss_fn ?

1 Like
6

I actually don’t think this is a problem with the loss_func. I think its failing to create the cnn heads because the initialization of the linear layers seems to be failing.

7

@sgugger Can you please help ?

8

You can’t use cnn_learner with data that has several targets: it produces a model that has one output only. Note that you don’t have to have two RegressionBlock as outputs, you can have just one like this: RegressionBlock(n_out=2)

6 Likes
9

That works as expected. Thank you @sgugger.

One thing I noticed is that I can no longer use learner.show_batch(...) or learner.show_results(...) anymore. I see this error:

--> 283     def decodes(self, o): return TitledFloat(o)
    284     def setups(self, dsets):
    285         if self.c is not None: return

ValueError: only one element tensors can be converted to Python scalars

My guess is I am missing a transform. Any suggestions on what I should be doing ?

10

That was a bug. This should be fixed now.

11

Thanks again !

12

I am also trying to train a model with multiple outputs but different to @rahulrav, in this case there is a CategoryBlock and a MaskBlock and I am getting a TypeError: 'L' object is not callable error during training.

get_msks = lambda o: path/'mask'/f'{o.ID}'

blocks = (ImageBlock,
          CategoryBlock,
          MaskBlock)
getters = [
           ColReader('ID', pref=path/'image'),
           ColReader('CATE'),
           get_msks
               ] 

vcvc = DataBlock(blocks=blocks,
                 getters=getters,
                 item_tfms=Resize(128),
                 batch_tfms=aug_transforms(size=224),
                 n_inp=1)
g = vcvc.dataloaders(labs, bs=16, num_workers=0)

I am able to view a batch and a summary

Final sample: (PILImage mode=RGB size=616x479, TensorCategory(0, dtype=torch.int32), PILMask mode=L size=616x479)

I am assuming that because there are 2 output blocks calling learn.loss_func gets 2 loss functions:
(#2) [FlattenedLoss of CrossEntropyLoss(),FlattenedLoss of CrossEntropyLoss()]

learn = unet_learner(g, resnet18, metrics=dice, n_out=2)

On training I get the following error TypeError: 'L' object is not callable

TypeError                                 Traceback (most recent call last)
<ipython-input-9-ca61b3aa75fc> in <module>
----> 1 learn.fine_tune(1)

c:\pillview\nih\fastcore\fastcore\utils.py in _f(*args, **kwargs)
    428         init_args.update(log)
    429         setattr(inst, 'init_args', init_args)
--> 430         return inst if to_return else f(*args, **kwargs)
    431     return _f
    432 

c:\pillview\nih\fastai2\fastai2\callback\schedule.py in fine_tune(self, epochs, base_lr, freeze_epochs, lr_mult, pct_start, div, **kwargs)
    159     "Fine tune with `freeze` for `freeze_epochs` then with `unfreeze` from `epochs` using discriminative LR"
    160     self.freeze()
--> 161     self.fit_one_cycle(freeze_epochs, slice(base_lr), pct_start=0.99, **kwargs)
    162     base_lr /= 2
    163     self.unfreeze()

c:\pillview\nih\fastcore\fastcore\utils.py in _f(*args, **kwargs)
    428         init_args.update(log)
    429         setattr(inst, 'init_args', init_args)
--> 430         return inst if to_return else f(*args, **kwargs)
    431     return _f
    432 

c:\pillview\nih\fastai2\fastai2\callback\schedule.py in fit_one_cycle(self, n_epoch, lr_max, div, div_final, pct_start, wd, moms, cbs, reset_opt)
    111     scheds = {'lr': combined_cos(pct_start, lr_max/div, lr_max, lr_max/div_final),
    112               'mom': combined_cos(pct_start, *(self.moms if moms is None else moms))}
--> 113     self.fit(n_epoch, cbs=ParamScheduler(scheds)+L(cbs), reset_opt=reset_opt, wd=wd)
    114 
    115 # Cell

c:\pillview\nih\fastcore\fastcore\utils.py in _f(*args, **kwargs)
    428         init_args.update(log)
    429         setattr(inst, 'init_args', init_args)
--> 430         return inst if to_return else f(*args, **kwargs)
    431     return _f
    432 

c:\pillview\nih\fastai2\fastai2\learner.py in fit(self, n_epoch, lr, wd, cbs, reset_opt)
    198                     try:
    199                         self.epoch=epoch;          self('begin_epoch')
--> 200                         self._do_epoch_train()
    201                         self._do_epoch_validate()
    202                     except CancelEpochException:   self('after_cancel_epoch')

c:\pillview\nih\fastai2\fastai2\learner.py in _do_epoch_train(self)
    173         try:
    174             self.dl = self.dls.train;                        self('begin_train')
--> 175             self.all_batches()
    176         except CancelTrainException:                         self('after_cancel_train')
    177         finally:                                             self('after_train')

c:\pillview\nih\fastai2\fastai2\learner.py in all_batches(self)
    151     def all_batches(self):
    152         self.n_iter = len(self.dl)
--> 153         for o in enumerate(self.dl): self.one_batch(*o)
    154 
    155     def one_batch(self, i, b):

c:\pillview\nih\fastai2\fastai2\learner.py in one_batch(self, i, b)
    159             self.pred = self.model(*self.xb);                self('after_pred')
    160             if len(self.yb) == 0: return
--> 161             self.loss = self.loss_func(self.pred, *self.yb); self('after_loss')
    162             if not self.training: return
    163             self.loss.backward();                            self('after_backward')

TypeError: 'L' object is not callable

Any ideas on how to correct this? Thanks

13

Combining the mask with anything requires special care IIRC, due to the order in which they’re in. MaskBlock and CategoryBlock weren’t designed to be in tangent. I attempted combining mask with bounding boxes which led to something similar. See the short discussion here:

What I can tell you is the order in which you pass the blocks is extremely important. So I’d start with simply reversing it (CategoryBlock then MaskBlock)

1 Like
14

I see, thanks for the link, Ill take a look at that and play around with the order. Thanks for the tip!

15

@sgugger I understand the resolution in the case when all outputs are regressors.
But how do we deal with a scenario where the outputs are let’s say classifiers. Or in the case when we have mixed outputs of let’s say regression and classification.
If cnn_learner does not support such multiple output cases, is there a workaround for this in v2?

1 Like
16

does vision_learner take multiple outputs (for category blocks?) @vaibhavgupta @sgugger