Hi mods. May I know what is Jeremy’s typical process in debugging the issues that he encounters during his experimentation? Would love to see the process step by step.
I have a recollection that he mentioned this sometime before - but can’t find the reference somehow.
Why must the begin_fit method in the CallbackHandler class stop executing subsequent callbacks in the list if one of the prior ones returns False?
The execution will not stop, since the loop goes like for cb in self.cbs. We just record the fact that one of the functions returned False.
Also, I’m having trouble wrapping my mind around the concept of a Callback as a set of callbacks – I naturally think of “begin_fit”, “after_fit”, etc each as a separate callback. Why would you ever need to pass in two "begin_fit"s?
We may need to do more than one things at, say, begin_fit (i.e. when fit is called). Thus, you may want multiple callbacks per event.
That is the main reason we develop in Jupyter Notebook. Debugging or checking your outputs is super easy and that really boosts our productivity.
And the ipython debugger is great if needed!
I’m not sure about Jeremy … but one thing I do frequently is copy the pertinent fastai code into my own notebook, rename the methods/classes so it doesn’t conflict with the fastai code, and then I sprinkle pdb.set_trace() all over and look at the shape of things in particular as code runs.
He means that in the first batches of training, it’s crucial to have the beginning of training go well. That is why we begin with a low learning rate and warm up in 1cycle.
As far as I know, so far the schedules are set manually. Are you aware of any research that create a schedule based on the current state of the model? In order word, that determine the ‘phase’ the model is in by looking at it, and sets the annealing accordingly?
Would we use a callback if we want to be setting schedules that depend on current or recent values/statistics of different layers, or is that something that requires a different approach?
This lecture is the most dense lecture of all fastai courses. I reckon it will take weeks to get a good grasp of all practices taught in the lesson. Pure gold!