Hello Community,
I am trying ULMFit first time. It looks very exciting and promising.
In my scenario, I am getting improved F1 score but it is not much higher than what I am getting from Logistic Regression. The validation loss is close to 0.28 and train loss around 0.14. If I train the model for more than 1 epoch (in the classification layer), it overfits. I have tried different drop out parameter value but cannot improve the F1 further.
Some details about the problem: It is a binary problem with pos to negative ratio 1:5. I have around 2k positives and 10k negatives in training data. Test data is around 600 positive and 1000 negative. Training data can be noisy. I have fine tuned the Language model on 50K unlabeled data and the final accuracy of LM is 44.
I am looking for tips on how to debug the model, what hyper-parameters to try etc. Also, is it possible to try different models in the classification layer like CNN, or simple NN? Finally, does fine tuning the LM on more unlabeled data helps?
Any resource or blog will also be helpful.
Thanks,
Javed