@jeremy
I thought transformation would be done on the validation on the fly, no? I thought (or at least it looks like) learn.TTA(…) is done on the testing set at the very end, to produce the inferences on the unknown set.
Reason I ask is that it seems like it would be better to do TTA on the fly on the validation, to get a better measure of model quality.
I don’t know the answer for sure, but I’d suspect that the error on TTA validation and validation should be highly correlated, and there’s a cost. You’re running your validation 5x more frequently with TTA on top of any costs to do data augmentation.
My guess is that the slight benefits of a better measure of model quality don’t outweigh those costs because the main use of validation loss is to ensure we’re not over or underfitting. I’m guessing the situations are few and far between where TTA validation would cause you to make a different decision about when to stop training the model.
Actually we use learn.TTA both on validation set and on test set when running predictions. The only difference is that we pass it an argument is_test=True when we are using it on the test set.
As far as the regular data augmentation/transformations, those are done on the fly during training but are only applied to the training set I believe. The difference with tfms and TTA is that with TTA we are predicting on augmented images and then taking the average of those predictions (either predicting on validation set or test set). With valid set we can see in the notebook that we usually get a bump in accuracy, with the test set we don’t know how it improves accuracy unless we submit it to Kaggle or something like that.
I’m here after searching for conversation about TTA, this looks like a good one!
So far I have found that TTA has not increased my accuracy, the results are typically not much better or even a little worse than my last epoch (see snapshot - validation error rate is better but accuracy is worse). So, apart from getting in idea of whether or not I am overfitting I don’t yet have any condidence in using it.
But the use of it confuses me - sorry about the number of questions!
@jeremy It was working last night until the last update from fastai.
lesson1-statoil-20171119.pdf (2.5 MB)
@gerardo can you try git pull again and see if that helps? Also, double-check that test_name is set, and that it’s pointing somewhere with data in.
When you ask your model to make a prediction on an unseen image, it basically looks at that image. It can be helpful to make it look at slightly transformed versions of the same image, since, in doing so, you lower the probability that your model can get confused.
TTA makes predictions over a group of transformed versions of the same image, then averages them, and present this average as its answer.
All I just said is my understanding and could very weel be wrong.
Training time augmentation is a different thing. Read the whole lesson1 nb.
No. It is up to you how you prepare your data for submission, as long as you follow their guidelines.
Not at all wrong 
Hi, why would learn.TTA(is_test=True) return both log_preds, y? Shouldn’t it not have a y available? I guess it’s just there for consistency?
Yes… y variable is already known through the data loader ImageClassifierData(). I believe its done for
convenience…
And also when the flag is_test=True is present, we are looking at the test set where the y’s are unknown, hence we can code as:
log_probs , _ = learn.TTA(is_test=True)
