SGDR vs CLR

Part 1 (2017)
1

Hello fast.ai community,
What’s the difference between stochastic gradient descent with restarts and cyclical learning rate in part 1 lecture video 1?
As per my knowledge these both methods are used to find optimal learning rate. Which one is more efficient?
Kindly guide me through the differences of both.

Thanks.

2

Hi,

I had the same question in the past and the following articles helped me clarify:

  1. https://towardsdatascience.com/estimating-optimal-learning-rate-for-a-deep-neural-network-ce32f2556ce0
  2. http://teleported.in/posts/cyclic-learning-rate/
  3. https://techburst.io/improving-the-way-we-work-with-learning-rate-5e99554f163b

Cheers!

1 Like
3

Thanks @chho6822 This was really helpful.