Hello everyone - I finished the part 1 course and some parts of the part 2 course as they relate to text data. The project I’m working on requires me to be able to classify medical documents. There are 3 classes, and each document has only one classification. I am using a fine-tuned language model as a backbone for the classifier head.
After initial processing of the data into csvs, I’ve used fastai v1
language_model to fine-tune a language model from pretrained wikitext-103 on a publicly available set of de-identified discharge summaries. I found a learning rate through
lr_find, and trained it for a couple epochs.
I then took some time to label a small proportion of the documents appropriately. I created a RNN classifier, also with the fastai v1 through
classifier, and was able to get it working. Unfortunately, my accuracy hovered around 50%. I’m sure a large part of this is due to the fact that I have only labeled a small portion of the dataset (80 examples for the ‘negative’ class, 20 examples each for two ‘positive’ classes).
I tried to spend some time reading through the ULMfit paper and working through the docs, but I feel a little overwhelmed at the moment. I would love some pointers to get me headed in the right direction. My most pressing questions are the following:
Is there a systematic way that I should approach adjusting the hyperparameters? If so, are there any articles or papers that I could read that would help me?
For those more well-versed in NLP, do you have a basic algorithm or recipe for tweaking your classifier models?
As it pertains to ULMfit in particular, how can I know if I’m overfitting the language model? I’m not sure if I can really compare train-valid loss in this case, because the real utility of the language model for my purposes is a downstream task.
I hope it doesn’t sound like I’m trying to have anyone solve my problems for me! I feel like there is a lot I can learn through this project and I just want to start to get a feel for a systematic approach to solving it.