Validation loss smaller than training loss

I have observed in training of datasets in Lesson 3 (like Camvid) that validation loss is generally lower than training loss in each epoch.

Doesn’t that mean my model has high bias? I expect training loss to be smaller as model is fitted to that data more closely.

60.00% [6/10 09:28<06:18]

epoch train_loss valid_loss acc_camvid
1 1.007837 0.906481 0.777404
2 0.745094 0.613785 0.846059
3 0.677771 0.646452 0.835799
4 0.619955 0.538954 0.848037
5 0.603284 0.462265 0.877266

Yes, you can train for longer and the accuracy should keep improving. When the accuracy starts to decrease then it will be a sign of overfitting, until then you can keep training. I believe the validation loss is lower because during the validation phase dropout is not active.

2 Likes

I don’t think the validation loss is lower because during the validation phase dropout is not active.
At 51m53s of the lesson 2 video, ps = 0 means turn off the dropout, but train_loss is still higher than valid_loss.


I can’t still figure out why.

3 Likes

I’m having the same problem with the MNIST dataset , using default LR, trying differents arquitetures (resnet18, resnet50), image size (16,32,64), splinting the validation with more data (0.3), but everytime I’m getting the training loss bigger than the validation:
am I doing something wrong?

mnist = untar_data(URLs.MNIST)
tfms = get_transforms(do_flip=False)

data = (ImageItemList.from_folder(mnist)
.random_split_by_pct(0.3)
.label_from_folder()
.transform(tfms, size=32)
.databunch()
.normalize())

learn.fit_one_cycle(3, slice(1e-5))

Total time: 05:34

epoch train_loss valid_loss accuracy
1 0.062921 0.029518 0.990286
2 0.064949 0.027088 0.991333
3 0.058849 0.026884 0.991238

learn.fit_one_cycle(3, slice(1e-5))

I find that it was needed to use regularization to solve that, so using it

learn = create_cnn(data, models.resnet18, metrics=accuracy, ps=0, wd=0)

the result was:

40 0.014386 0.082150 0.994571 01:59

I update the jupyter. more suggestions are welcome.

1 Like