Classifier error rate > 1?

fastai
1

I created a learner for testing performance on new data.
I load there the weights of a training with very good accuracy on train and valid set - error rate of 0.027 on train and 0.0045 on valid)

When I run validate on the new data, I get an error rate of 3.098.
How can it even be more than 1?

I created 2 test sets (one for each class) to inspect the predictions. And on them, I see that the outputs of the network are excellent… so this error rate of 3 is very confusing. Can someone help?

real_data_idan_val = ImageDataBunch.from_folder(path, ds_tfms=get_transforms(do_flip=False),                                   bs=bs,size=sz,train='TRAININGSET',valid='REAL_IDAN_VALID',resize_method=ResizeMethod.SQUISH)

`learn_for_real_idan_val = create_cnn(real_data_idan_val, models.resnet50, metrics=error_rate)`

learn_for_real_idan_val.load('fit_one_cycle_3epochs_00045error');

learn_for_real_idan_val.validate(real_data_idan_val.valid_dl)
Out:[3.098189, tensor(0.3611)]
1 Like
2

https://docs.fast.ai/basic_train.html#Learner.validate

It says:

Return the calculated loss and the metrics of the current model on the given data loader dl . The default data loader dl is the validation dataloader.

So I think the first value is the loss and the second is the “error rate”.

learn_for_real_idan_val.validate(real_data_idan_val.valid_dl)
["loss", "error rate"]
3

Yes. It is error rate. So error rate of 0.2 for instance means 80% accuracy.
Error rate of 3 doesn’t make sense to me.

4

It is interesting that at the end of training:

error rate of 0.027 on train and 0.0045 on valid

Usually train has worse accuracy.

5

Hi @Kalanit,

It seems that you are mistaking about what the validate method returns.

When you call learn.validate(data.valid_dl), it will return 2 things:

  • The first value is the value of your loss
  • The second value is the value of your error rate on this particular data

As you have said, having an error rate > 1 has no sense, and in your particular case, it is 0.3611

Note: you can pass a metrics argument to the validate method (e.g: learn.validate(data.valid_dl, metrics=[accuracy])) and it will return the accuracy directly

5 Likes
6

Ahhhh! right
I thought the loss was the second part. But now it makes sense.
Many thanks

1 Like
7

Hi @Kalanit thanks for posting this issue.

Hello @NathanHub thanks for your clear explanation on this. I was also struggling and get cleared it from your answer.

A huge help. :blush: