Understanding cycle_len and cycle_mult

[himani]

In the code learn.fit(lr, 3, cycle_len=1, cycle_mult=2), can you please explain the function of cycle_len and cycle_mult?

Also, is 3 the number of epochs?

[amritv]

@himani, these notes might help to explain the difference

Cats and Dogs code notes

Hi all, I have been going through the videos and the forums with the aim of finding out exactly what every line of code dose. In the process I have been adding notes to improve my understanding especially as we are learning many new tricks! :grin:. Most of the notes are on what we have covered so far and there are still a number of gaps that I am hoping to complete soon. The process is helping me understand the code better so thought I would share in case anyone else might find this useful,…

[cqfd]

The cycle_len and cycle_mult parameters are used for doing a variation on stochastic gradient descent called “stochastic gradient descent with restarts” (SGDR).

This blog post by @mark-hoffmann gives a nice overview, but briefly, the idea is to start doing our usual minibatch gradient descent with a given learning rate (lr), while gradually decreasing it (the fast.ai library uses “cosine annealing”)… until we jump it back up to lr!

The cycle_len parameter governs how long we’re going to ride that cosine curve as we decrease… decrease… decrease… the learning rate. Cycles are measured in epochs, so cycle_len=1 by itself would mean to continually decrease the learning rate over the course of one epoch, and then jump it back up. The cycle_mult parameter says to multiply the length of a cycle by something (in this case, 2) as soon as you finish one.

So, here we’re going to do three cycles, of lengths (in epochs): 1, 2, and 4. So, 7 epochs in total, but our SGDR only restarts twice.

[himani]

Thank you so much, it is very helpful

[himani]

Thank you very much for sharing the notes

[ecdrid]

Prefer giving a search in the forum…
All most all the queries have answers already there…
Thanks…

[jeremy]

Prefer giving a search in the forum…

All most all the queries have answers already there…

Whilst that’s true, it’s important to note that some conceptual ideas are hard to search for and digest.

[ecdrid]

That’s true…
But I too learnt this search first and then ask concept quite helpful…
Learnt from this amazing forum itself…

Here’s the reference link…(Scrolling a bit below the answers my doubt too)

Why would I want to do a "learn.fit(lr, 3, cycle_len=3)" instead of a "learn.fit(lr, cycle_len=1, cycle_mult=2)"?

I’m working on figuring out exactly what these arguments do. I thought cycle_mult would just change how many epochs are run so my guess was that it would be ((1 epoch * 1 cycle_length) * 2 cycle_mult), but that doesn’t appear to be how it works. I am currently doing a learner.fit(3e-3, 20, wds=1e-6, cycle_len=1, cycle_mult=2) which I thought would be 40 total epochs so I was letting it run overnight. Currently I’m still training at 441. Anybody have insight on the cycle_mult parameter?

Image credit @Moody

[rob]

some conceptual ideas are hard to search for and digest

Totally agree. Speaking as someone who audited p1v1 and took p1v2 live, it was so much easier to make use of the forums while the course was live. Part of it was being involved, but also, searching for keywords over the entire course becomes tricky.

Video timings and wiki-ified stuff make it a lot easier

[himani]

I searched the forums, however could not quite understand the answers to a related (but different) question.

[boxabirds]

Super helpful. I wonder whether cycle_len be better named “num_epochs_per_cycle”?