I created a learner for testing performance on new data.
I load there the weights of a training with very good accuracy on train and valid set - error rate of 0.027 on train and 0.0045 on valid)
When I run validate on the new data, I get an error rate of 3.098.
How can it even be more than 1?
I created 2 test sets (one for each class) to inspect the predictions. And on them, I see that the outputs of the network are excellent… so this error rate of 3 is very confusing. Can someone help?
real_data_idan_val = ImageDataBunch.from_folder(path, ds_tfms=get_transforms(do_flip=False), bs=bs,size=sz,train='TRAININGSET',valid='REAL_IDAN_VALID',resize_method=ResizeMethod.SQUISH)
`learn_for_real_idan_val = create_cnn(real_data_idan_val, models.resnet50, metrics=error_rate)`
Return the calculated loss and the metrics of the current model on the given data loader
dl . The default data loader
dl is the validation dataloader.
So I think the first value is the loss and the second is the “error rate”.
["loss", "error rate"]
Yes. It is error rate. So error rate of 0.2 for instance means 80% accuracy.
Error rate of 3 doesn’t make sense to me.
It is interesting that at the end of training:
error rate of 0.027 on train and 0.0045 on valid
Usually train has worse accuracy.
It seems that you are mistaking about what the
validate method returns.
When you call
learn.validate(data.valid_dl), it will return 2 things:
- The first value is the value of your loss
- The second value is the value of your error rate on this particular data
As you have said, having an error rate > 1 has no sense, and in your particular case, it is 0.3611
Note: you can pass a
metrics argument to the
validate method (e.g:
learn.validate(data.valid_dl, metrics=[accuracy])) and it will return the accuracy directly
I thought the loss was the second part. But now it makes sense.
Hi @Kalanit thanks for posting this issue.
Hello @NathanHub thanks for your clear explanation on this. I was also struggling and get cleared it from your answer.
A huge help.