Valid loss, Training Loss trends

Part 1 (2020)
1

Hi there,

What does it mean when training loss starts to increase after training a while while valid loss steadily decreases? See below:

Step Training Loss Validation Loss
500 3.481300 4.141750
1000 3.485500 4.134972
1500 3.459400 4.132983
2000 3.439300 4.134877
2500 3.452000 4.128098
3000 3.430700 4.093344
3500 3.435300 4.137605
4000 3.428000 4.141555
4500 3.428800 4.163394
5000 3.420900 4.140961
5500 3.411500 4.143158
6000 3.419400 4.081789
6500 3.408900 4.167723
7000 3.408000 4.066579
7500 3.375800 4.110569
8000 3.399500 4.108437
8500 3.375900 4.089203
9000 3.387700 4.192120
9500 3.377200 4.126762
10000 3.392600 4.059635
10500 3.357600 4.052819
11000 3.386600 4.060781
11500 3.412600 4.017525
12000 3.395900 4.029039
12500 3.393600 4.111553
13000 3.394700 3.988847
13500 3.405000 4.038273
14000 3.415400 4.018964
14500 3.416600 4.029273
15000 3.411500 3.992403
15500 3.426400 4.060863
16000 3.425900 3.985097
16500 3.436900 3.979489
17000 3.443700 3.919812
17500 3.430600 4.071402
18000 3.442700 3.964387
18500 3.453100 3.960077
19000 3.447500 3.879763
19500 3.459700 3.957340
20000 3.482900 3.959507
20500 3.482400 3.953536
21000 3.487800 3.907522
21500 3.488100 3.918010
22000 3.499200 3.875734
22500 3.530400 3.924965
23000 3.501300 3.870870
23500 3.527300 3.916741
24000 3.538900 3.906471
24500 3.556800 3.840249
25000 3.584500 3.921753
25500 3.584200 3.818825
26000 3.588900 3.782376
26500 3.587200 3.826088
27000 3.587700 3.914851
27500 3.623400 3.852811
28000 3.638700 3.798076
28500 3.650200 3.847332
29000 3.635400 3.771980
29500 3.643500 3.726057
30000 3.662700 3.762208
30500 3.678800 3.764087
31000 3.699900 3.752388
31500 3.710700 3.775933
32000 3.734900 3.745933
32500 3.752700 3.786586
33000 3.745400 3.763834
33500 3.755600 3.711195
34000 3.790200 3.713098
34500 3.778900 3.702047
35000 3.786700 3.788854
35500 3.819700 3.716982

In the case of overfitting, I believe that the training loss should continue to go down, while valid loss goes up. Is my model okay to keep training? Thank you for your help!

2

From the numbers I see, there is small difference between your training loss and validation loss. There should not be anything to worry about in my opinion (I might be wrong).

Usually if validation loss is way higher than training loss then that is a sign to worry about because that can be a sign of overfitting. Also,

I would suggest you to go through this post to find out more on this.

3

Hi Satyabrata,

Thank you for your reply. I have tried to train the model more and it turns out that it is overfit. (training loss ended up going down, and valid loss ended up going higher). I tried going back to the model before the weird trend occurred and continue training, however eventually, I always end up with the same trend above given more iterations.

Thank you for the post.

4

Ah! Now it might be overfitting. If you can then try to run the training with more data else try to make use of data augmentation.

One more thing you could try is to reduce the batch size just to check.