Difference between train/val loss and error rate

Part 1 (2019) Advanced (Part 1 v3)
1

Hi everyone,
I am pretty new to the world of machine learning, I started fast.ai MOOCThis text will be hidden and for my “project” for lesson one I decided to differentiate lung problems from an X Ray. The dataset contains about 7000 images of normal lungs and bacterial or viral pneumonia infected ones, and the categories are: ‘normal’, ‘bacteria’, ‘virus’.
My biggest issue is the following:

Screen Shot 2020-05-25 at 03.34.28
Screen Shot 2020-05-25 at 03.34.281780×476 30.8 KB

I see my train_loss and valid_loss are quite high but my error rate is around 17%, what does that mean? If I were to test the model out in the world would it behave according to the valid_loss or the error_rate? And what’s the difference between those two?

I tried searching the forums and google but couldn’t come to an answer, hence this post :slight_smile:

As a side note: The problem has been attempted before and a 90% accuracy was achieved (it isn’t an imposible problem to tackle)

Sorry if this isn’t in the right category, I’ll move it if it doesn’t belong here

2

Loss value is decresing for both train and validation dataset, which means you can train for longer. Its tough to say if the loss is really high without comparing it with some other training metric. We generally do not use loss to decide wether the model can be used for production. Accuracy can be used if its a balanced dataset, but data from medical domains are rarely balanced. You can check at confusion matrix, which can give you more clarity on how good/bad the model is for each class.

1 Like
3

Thank you very much! This cleared a few doubts