Implement Bag of Tricks for Image Classification with Convolutional Neural Networks

Hi! All,

Just finish reading this paper
Wondering if anyone have tried to implement those tricks in fastai/pytorch? especially some of training tricks like label smoothing, Knowledge distillation and Mixup Augmentation. If not, I am actually interested in giving it a shot.


Yes they’re pretty much all implemented in examples/ in fastai repo :slight_smile:


Adding Some links for future searchers:
Label Smoothing: loss_func = LabelSmoothingCrossEntropy()

Mixup Augmentation: .mixup(alpha=0.2)

As for Knowledge distillation: I am not sure the imagenette -> imagewoof does this. Would appreciate an explainer if it’s actually implemented.


It’s not. I don’t feel like that really counts as model training - it’s more like model pruning.

1 Like

Jeremy, would the implementation of the LabelSmoothingCrossEntropy() be directly usable for multi-class multi-label classification ?

1 Like


Do I understand it correctly that loss_func = LabelSmoothingCrossEntropy() is all we have to do to implement label smoothing? I.e. we don’t have to additionally change the labels before training, right?

@howtodowtle nope! That is it :slight_smile:

Thanks, @muellerzr!

you are one of best teacher i have learned from thank you so much

Hi @muellerzr,

Cosine Learning Rate Decay from the paper can be implemented using general scheduler as shown here:, right?