Two ways to improve accuracy and reduce training time

I have explained 2 new techniques to improve model accuracy and reduce training time here:


Input is appreciated
2 Likes

Nice write up. Thanks for introducing those training techniques.

This Hinton paper on Dropout introduces it as an efficient alternative to ensemble methods.

https://arxiv.org/abs/1207.0580

I know everyone uses dropout these days but it’s interesting in the comparison to the Snapshot Ensemble you cover in your article.

I wonder if the CLR (discussed in part 2 of the course) and Cyclic Cosine Annealing (discussed in your article) could be used somewhat interchangeably despite having different inputs.

As a follow up I’d love to hear if you had any success using these techniques in the wild.

2 Likes

Thanks for responding. Your input is appreciated!
I mentioned snapshot ensembles because its a recent paper. You pose a very interesting idea of using dropout regularisation to avoid ensembling. I am not sure if they are interchangeable in terms of accuracy.

The cyclic learning rates mentioned in part 2 of the course is similar to Cosine Annealing from my article, but in the snapshot ensemble paper, the authors lower the learning rate fairly early to reach a local minima.

Unfortunately I have not tried any of these methods yet, but I will hopefully writing about how to use both these concepts together.

Nice post! Thanks for sharing ^^