Error while using fbeta metric

I am working on tabular data for multi class classification. Using metrics = accuracy is showing no error but using metrics = fbeta is showing this error.

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-66-2a714d47a7ed> in <module>()
----> 1 learn.fit_one_cycle(2, 1e-2, wd=0.2)

/usr/local/lib/python3.6/dist-packages/fastai/train.py in fit_one_cycle(learn, cyc_len, max_lr, moms, div_factor, pct_start, wd, callbacks, **kwargs)
     20     callbacks.append(OneCycleScheduler(learn, max_lr, moms=moms, div_factor=div_factor,
     21                                         pct_start=pct_start, **kwargs))
---> 22     learn.fit(cyc_len, max_lr, wd=wd, callbacks=callbacks)
     23 
     24 def lr_find(learn:Learner, start_lr:Floats=1e-7, end_lr:Floats=10, num_it:int=100, stop_div:bool=True, **kwargs:Any):

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in fit(self, epochs, lr, wd, callbacks)
    170         callbacks = [cb(self) for cb in self.callback_fns] + listify(callbacks)
    171         fit(epochs, self.model, self.loss_func, opt=self.opt, data=self.data, metrics=self.metrics,
--> 172             callbacks=self.callbacks+callbacks)
    173 
    174     def create_opt(self, lr:Floats, wd:Floats=0.)->None:

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in fit(epochs, model, loss_func, opt, data, callbacks, metrics)
     92     except Exception as e:
     93         exception = e
---> 94         raise e
     95     finally: cb_handler.on_train_end(exception)
     96 

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in fit(epochs, model, loss_func, opt, data, callbacks, metrics)
     87             if not data.empty_val:
     88                 val_loss = validate(model, data.valid_dl, loss_func=loss_func,
---> 89                                        cb_handler=cb_handler, pbar=pbar)
     90             else: val_loss=None
     91             if cb_handler.on_epoch_end(val_loss): break

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in validate(model, dl, loss_func, cb_handler, pbar, average, n_batch)
     52             if not is_listy(yb): yb = [yb]
     53             nums.append(yb[0].shape[0])
---> 54             if cb_handler and cb_handler.on_batch_end(val_losses[-1]): break
     55             if n_batch and (len(nums)>=n_batch): break
     56         nums = np.array(nums, dtype=np.float32)

/usr/local/lib/python3.6/dist-packages/fastai/callback.py in on_batch_end(self, loss)
    237         "Handle end of processing one batch with `loss`."
    238         self.state_dict['last_loss'] = loss
--> 239         stop = np.any(self('batch_end', not self.state_dict['train']))
    240         if self.state_dict['train']:
    241             self.state_dict['iteration'] += 1

/usr/local/lib/python3.6/dist-packages/fastai/callback.py in __call__(self, cb_name, call_mets, **kwargs)
    185     def __call__(self, cb_name, call_mets=True, **kwargs)->None:
    186         "Call through to all of the `CallbakHandler` functions."
--> 187         if call_mets: [getattr(met, f'on_{cb_name}')(**self.state_dict, **kwargs) for met in self.metrics]
    188         return [getattr(cb, f'on_{cb_name}')(**self.state_dict, **kwargs) for cb in self.callbacks]
    189 

/usr/local/lib/python3.6/dist-packages/fastai/callback.py in <listcomp>(.0)
    185     def __call__(self, cb_name, call_mets=True, **kwargs)->None:
    186         "Call through to all of the `CallbakHandler` functions."
--> 187         if call_mets: [getattr(met, f'on_{cb_name}')(**self.state_dict, **kwargs) for met in self.metrics]
    188         return [getattr(cb, f'on_{cb_name}')(**self.state_dict, **kwargs) for cb in self.callbacks]
    189 

/usr/local/lib/python3.6/dist-packages/fastai/callback.py in on_batch_end(self, last_output, last_target, **kwargs)
    272         if not is_listy(last_target): last_target=[last_target]
    273         self.count += last_target[0].size(0)
--> 274         self.val += last_target[0].size(0) * self.func(last_output, *last_target).detach().cpu()
    275 
    276     def on_epoch_end(self, **kwargs):

/usr/local/lib/python3.6/dist-packages/fastai/metrics.py in fbeta(y_pred, y_true, thresh, beta, eps, sigmoid)
     16     y_pred = (y_pred>thresh).float()
     17     y_true = y_true.float()
---> 18     TP = (y_pred*y_true).sum(dim=1)
     19     prec = TP/(y_pred.sum(dim=1)+eps)
     20     rec = TP/(y_true.sum(dim=1)+eps)

RuntimeError: The size of tensor a (5) must match the size of tensor b (64) at non-singleton dimension 1
3 Likes

i am also facing similar issues , were you able to solve this problem. any guidance will be really helpful.

Also getting similar errors, wonder if maybe Sylvain could help on this ?

Linked issue

Couldn’t find any solution till now.

Same here - has there been a solution on running multiclass classification and using fbeta as the metric?

Did anyone solve this issue? Do we have to one hot encode the target variable? And if we do so, how do we pass it to the data bunch. @sgugger can you help?

As indicated in the docs, fbeta is for multi label classification. There is FBeta() for single label classification.

3 Likes

for my model the accuracy is 0.97 and fbeta(‘micro’) is 0.00
can anyone help me what could be the reason

Not sure it will be useful to you, but I once fitted for 6 cycles and first two of them had 0.000 fbeta value.
Later epochs had greater than zero values.

1 Like

Hi Guys,

I tried this and it worked for me: metrics = [accuracy, FBeta(‘macro’)]. For more details on f1 score micro, macro and weighted please visit wikipedia.

1 Like

This is what the document says. Isn’t it misleading? i thought it is just multi class, you say it is multi label

Note: This function is intended for one-hot-encoded targets (often in a multiclassification problem).

2 Likes

I too am getting this error and used

from fastai.metrics import error_rate

#acc_02 = partial(accuracy_thresh, thresh=0.2)
#F1 = MultiLabelFbeta(beta=1, average=“macro”) <------ this

learn = cnn_learner(data, models.resnet50 , metrics=[accuracy, error_rate, FBeta(‘macro’)], model_dir=’/kaggle/working/models’) <------ this

but none of them work. Any helpm would be greatly appreciated.

https://www.kaggle.com/vikranthkanumuru/naruto-hand-sign-detection-usin-fastai-diff-method/