Hello. I’m trying to create a classifier that categorizes different Yelp reviews about career coaching businesses.
I’ve manually classified about 100 reviews into 5 relatively even classes: career change, current job, interview, lost job, and resume.
I trained the language model, loaded the encoder, created the data using the following code with the datablock API:
data_clas = (TextList.from_df(df=clas_df, path=path)
.split_by_idx((range(0,24)))
.label_from_df(cols=1)
.databunch(bs=8))
Split by idx was just to try to get a nice, even validation set. I tried it with split_by_rand_pct as well.
For reference, here’s what the LR finder looks like:
I then trained the classifier using the code and numbers on the iMDB notebook. The results were very poor. It would often start out by making all its predictions in a single class (and the train_loss would often already be lower than the validation loss on the very first training cycle).
Beyond that, the train_loss continued to decrease while the valid_loss stayed the same or slightly increased/decreased and the accuracy stayed at around 15%-30%. A lot of the time, when the accuracy increased, the valid_loss had increased as well.
Is there something that I’m doing wrong here? Or is this just the result of having a small amount of data? I tried splitting each review in half to effectively double the sample size but was still only able to get up to 40%.
There’s obviously the possibility that there’s just not enough of a correlation with the text and the classes, but intuitively I’d think the model would do better than it’s doing. Career change reviews mention “career change”, “new career”, “transition”; resume reviews have the word “resume” many times, interview reviews have “interview” a lot, etc.
Here are some examples of out-of-sample reviews that would be classified as resume:
this company spent a lot of time with me throughout the resume drafting process. my resume hadn't been updated since my first job as a teacher so it needed a lot of help! they asked quite a few questions to determine which content was most important for my resume. i was very pleased with the draft they sent for my review. they were able to update my resume to a modern look-and-feel and highlight my top attributes. i will definitely use them again for other services!
Prediction: career change, 0.8088 (resume 0.0996)
after getting free resume critiques from several companies i went with executive drafts as theirs was spot on. i was with my first job after college for 15 years and had never had to do a resume before so i needed a lot of help. also, as the first employee of a startup my role was often not defined. hazel had her work cut out for her and she did not disappoint. my new resume got noticed immediately and i was able to land a job not long after. i have also referred several friends who had success with them as well.
Prediction: lost job, 0.4415 (resume 0.2103)
I feel like there has to be a way to make this better; these predictions seem way off.
Any advice would be much appreciated.
One side note: interp.plot_top_losses throws the error
AttributeError: ‘ClassificationInterpretation’ object has no attribute ‘plot_top_losses’
for some reason.