Hi! All,
Just finish reading this paper https://arxiv.org/abs/1812.01187
Wondering if anyone have tried to implement those tricks in fastai/pytorch? especially some of training tricks like label smoothing, Knowledge distillation and Mixup Augmentation. If not, I am actually interested in giving it a shot.
Gen
Yes they’re pretty much all implemented in examples/train_imagenet_adv.py in fastai repo 
Adding Some links for future searchers:
Label Smoothing: loss_func = LabelSmoothingCrossEntropy()
Mixup Augmentation: .mixup(alpha=0.2)
As for Knowledge distillation: I am not sure the imagenette -> imagewoof does this. Would appreciate an explainer if it’s actually implemented.
It’s not. I don’t feel like that really counts as model training - it’s more like model pruning.
Jeremy, would the implementation of the LabelSmoothingCrossEntropy() be directly usable for multi-class multi-label classification ?
Nope…
Do I understand it correctly that loss_func = LabelSmoothingCrossEntropy() is all we have to do to implement label smoothing? I.e. we don’t have to additionally change the labels before training, right?
you are one of best teacher i have learned from thank you so much
Hi @muellerzr,
Cosine Learning Rate Decay from the paper can be implemented using general scheduler as shown here: https://github.com/fastai/fastai/blob/master/examples/train_imagenet.py#L50, right?
