I get an accuracy prediction of 99.3% on a 50k subset of the Quora dataset (kaggle competition). But, when ever I test it, predict something to see how it goes, it’s wrong. I looked at 20, it got all of the wrong and with 90%+ confidence …
Anyway has an idea ?
Thanks !
What are your validation losses like?
I’m having the same problem. Digging into it, I think that my problem is that when calling predict, the input text is not being tokenized. When I apply tokenization before, predict is giving much better results. Anyone can confirm this?
I think you are making the same mistake I have made before !
If you are checking your training or validation accuracy by “learn.get_preds()” then remember that for for training or validation set the index 1 of the returned of “learn.get_preds()” is the original targeted data which are predicting. And index 2 is our model’s predictions.
Whereas for test set the index 1 is our model’s predictions.
I am not really sure about the indices, because I haven’t practised fastai from last 2 months due to my another project.
I’m exporting the mode, loading with load_learner and using predict to obtain my predictions for single instances.
The thing is that I need to tokenize data manually before predicting to achieve the accuracy of my model while training and it seems weird to me. Not sure 100% tho.
Hi!
Can you provide just a few lines of code for that example ?
I also used the IMDB example to classify some text with two output classes and when I call :
learn.predict("This is my text")
I always get the wrong prediction, even with example text that was… in the training set!
By the way the NN was supposed to be excellent at predicting my two classes. Here are my final results :
accuracy : 0.981591
train_loss : 0.012486
valid_loss : 0.507916
Your validation loss too much higher than the training loss. I think you understand the cause of the problem and have some clue on how to fix it just by looking at this fact.
I may be more complicated than that as the validation_loss point does not explain why learn.predict does not predict right a text example that was present in the training set. If my NN is overfitting, it should predict the text right.
Don’t you think so ?
Your valid_loss is about 50 times higher than the training loss. I think it is not really about predicting text anymore. I guess it is the pattern of the training set. But let’s wait for other professional person comes here to explain because I am not professional.
I just loaded an older epoch back up with the following values :
train_loss : 0.018931
valid_loss : 0.001687
accuracy : 0.975050
Same wrong prediction.
Can we move forward to the real point here which is why text examples from the training set wouldn’t get predicted right ?
Yep let’s wait for the professionals.
Thanks for trying and for your time though. I appreciate it. 
My unbalanced dataset might be the problem, here ?
Let class1 and class2 my predicted classes :
class1 : 538 558
class2 : 12 064
I misprocessed the data initially.
I had much more class2 data.
In reality, I had :
class1 : 409369
class2 : 141253
I will test 2 options and report in this thread for the ones who might encounter this kind of problems :
Option1 : truncate class1 and training with :
class1 : 141253
class2 : 141253
Option2 : keep corrected but still unbalanced proportions and training with :
class1 : 409369
class2 : 141253
You all have a nice week-end!
Cheers,
Alexandre.
Just for people who get an unbalanced dataset and would like to see how truncating can help, here is my first path Option1 (I know there are ways to deal with an unbalanced dataset, especially “punishing” the NN a good deal when it fail to predict the class with low amount of related data)
Option1 : truncate class1 and training with :
class1 : 141253
class2 : 141253
Now, I have far better results.
I think the unbalanced was clearly the problem, here.
FYI, I’m still using a stligthly different version of the IMDB Jupyter notebook.
Learning rate :
I chose the value 1e-2 for my fit_one_cycle methode with 30 epochs :
Here are some of my first epochs from that same method :
As you can see my train_loss and valid_loss values wouldn’t converge to 0 even if my accuracy was pretty high.
At this point I told myself this training was getting nowhere (I don’t know how to play with weight decay and dropout yet! I will soon, don’t worry 
) and that I needed to pick one of my saved model from the best epoch and unfreeze another layer. I took epoch9.
After I unfroze the last 2 layers, I got :
After I unfroze the last 3 layers, I got :
At this point I copied my jupyter notebook, and used inference of the last backed up model to try and predict some texts. My class1 and class2 were predicted right!
learn.load('third-1e-2-epoch9-bouture');
learn.predict("my class1 text")
learn.predict("some class2 text")
Returning to my main notebook:
My last fit_one_cycle method with two epochs on a totally unfrozen model are now done.
The train_loss, valid_loss are still going lower and the accuracy higher.
Que demande le Peuple ?
Until I know more (in that field that keeps me awake at night ) :
Yes, truncating works!
Have a great week-end.
Stay safe and good luck.
Alexandre.
Thank You for sharing anyway. I think this is useful.