I'm overfitting (classifier of imdb) and running out of ideas

Part 2 & Alumni (2018)
#1

… So, I’m training an language model.
I have the encoder, and this is the loss/accuracy of the classification part:

epoch      trn_loss   val_loss   accuracy                      
0      0.237607   0.176866   0.933717  
1      0.232174   0.169032   0.936548                      
2      0.187602   0.175905   0.935893                      
3      0.165638   0.174451   0.938436                      
4      0.169487   0.263132   0.93259                       
5      0.147892   0.189482   0.936948                      
6      0.108331   0.212086   0.935653                      
7      0.090423   0.263828   0.933613                       
8      0.090065   0.233687   0.936148

as you can see, as time moves on, it’s getting horribler and horribler over fitted.

I have tried to jack up the learning rate 10 times and decrease it again, with no avail.

What else can I try?

#2

Increase dropout?

#3

what does the lr schedule plot look like?

#4

Try training the LM for longer before you do the classifier?

2 Likes
#5

Doing that right now, let’s see, let’s see.

#6

That means that I have to re-adjust the weights, so not too big on that. (too much work. )