Implement Bag of Tricks for Image Classification with Convolutional Neural Networks

Hi! All,

Just finish reading this paper https://arxiv.org/abs/1812.01187
Wondering if anyone have tried to implement those tricks in fastai/pytorch? especially some of training tricks like label smoothing, Knowledge distillation and Mixup Augmentation. If not, I am actually interested in giving it a shot.

Gen

Yes they’re pretty much all implemented in examples/train_imagenet_adv.py in fastai repo :slight_smile:

5 Likes

Adding Some links for future searchers:
Label Smoothing: loss_func = LabelSmoothingCrossEntropy()

Mixup Augmentation: .mixup(alpha=0.2)

As for Knowledge distillation: I am not sure the imagenette -> imagewoof does this. Would appreciate an explainer if it’s actually implemented.

3 Likes

It’s not. I don’t feel like that really counts as model training - it’s more like model pruning.

1 Like

Jeremy, would the implementation of the LabelSmoothingCrossEntropy() be directly usable for multi-class multi-label classification ?

1 Like

Nope…

Do I understand it correctly that loss_func = LabelSmoothingCrossEntropy() is all we have to do to implement label smoothing? I.e. we don’t have to additionally change the labels before training, right?

@howtodowtle nope! That is it :slight_smile:

Thanks, @muellerzr!