Hi everyone! I’m currently trying to use what I learned in lesson 8 to solve a different classification problem
Just for learning purposes, I decided to rewrite the target variables to “pos” and “neg”. Being 1 pos and 0 neg.
id keyword location text target
0 1 NaN NaN Our Deeds are the Reason of this #earthquake May ALLAH Forgive us all pos
1 4 NaN NaN Forest fire near La Ronge Sask. Canada pos
2 5 NaN NaN All residents asked to ‘shelter in place’ are being notified by officers. No other evacuation or shelter in place orders are expected pos
3 6 NaN NaN 13,000 people receive #wildfires evacuation orders in California pos
4 7 NaN NaN Just got sent this photo from Ruby #Alaska as smoke from #wildfires pours into a school pos
Creating the dataloaders
dls = TextDataLoaders.from_df(df, text_col='text', is_lm=True)
dls.show_batch(max_n=3)
example below:
xxbos i entered to # win the xxup entire set of xxunk xxmaj lip xxmaj xxunk via xxunk . - xxmaj go enter ! # xxunk http : / / t.co / xxunk xxbos xxmaj to fight bioterrorism sir . xxbos xxmaj emergency xxmaj response and xxmaj hazardous xxmaj chemical xxmaj management : xxmaj xxunk and xxmaj xxunk http : / / t.co / xxunk http : / / t.co / xxunk
I can notice a lot of xxunk tags, Im not sure if there is a way to reduce the amount. In the example for the movie reviews, it was much lower.
Then I create the learner
learn = language_model_learner(
dls, AWD_LSTM, drop_mult=0.3,
metrics=[accuracy, Perplexity()]
).to_fp16()
and fine_tune one epoch
learn.fine_tune(1, 2e-2)
The result was pretty good
|accuracy|
0.445406
Then I create the Datablock using this language model
class_data = DataBlock(
blocks=(TextBlock.from_df('text', vocab=dls.vocab), CategoryBlock),
get_x=ColReader('text'), get_y=ColReader('label'), splitter=RandomSplitter(0.2))
Now I only need to create the dataloaders
dls_class = class_data.dataloaders(df)
Create a new learner
learn2 = text_classifier_learner(dls_class, AWD_LSTM, drop_mult=0.5, metrics=accuracy).to_fp16()
And fit one cycle
learn2.fit_one_cycle(1, 0.01)
This gave me an accuracy of 0.772668
So I decided to continue and slowly unfreeze each layer of the model
Once i unfreezed the 3rd last layer I decided to unfreeze the hole thing
learn2.unfreeze()
learn2.fit_one_cycle(2, slice(1e-3/(2.6**4), 1e-3))
And the accuracy also got better 0.806833
But when I try to test it with unseen data is where the problem began
texts = list(df_test["text"])
preds = []
for text in texts:
preds.append(learn2.predict([text]))
[('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
('neg', tensor(0), tensor([0.9354, 0.0646])),
All the preds give me the same thing… Im not sure where im making the mistake.