Lesson 9 Discussion & Wiki (2019)

Part 2 (2019)
201

Is there a simple way to do gradient accumulation in fastai? Or would the best way be to write a custom fit loop?

202

Wouldnā€™t the super resolution example that Jermey just mentioned still work if you combined the step and the zero into a single function? Does anyone else have another example where it wouldnā€™t work?

203

There is a Callback for that :wink:

4 Likes
204

You never know when someone might not want to zero the grad all the time, and the fastai training loop zeroes them for you, so thatā€™s not a real hassle :wink:

205

Do you have a good place to learn about callbacks? They seem so powerful, but I havenā€™t quite gotten my head wrapped around them.

2 Likes
206

I think I gave a talk about that recently :wink:

4 Likes
207

haha ā€¦ nice meme btw!

208

It was a great talk. I enjoyed it very much. :blush:

1 Like
209

I had done benchmarking of FP16 using fastai on 2080Ti Vs 1080Ti (With help from @Ekami) with a gentle intro to MP training.

Not sure if its good enough for wiki so Iā€™ll leave it here :slight_smile:

7 Likes
210

if(i%2) == 0 -> add callback :slight_smile:

211

callback is some thing at following points which are methods in Callbackā€¦

  1. on epoch begin
  2. on batch end,begin etc

You have to implement a new class for custom call back
Class a ( CallBack)

above methods defined

212

Did you guys create the callbacks or are they native to pytorch

213

No theyā€™re our own. PyTorch has hooks, but no callbcacks.

3 Likes
214

Watching Dataloaders impl reminded me of questiion about DataLoaders API design.
DataLoaders (and Datasets) in PyTorch return one element at the time before they are put together into a mini-batch. Iā€™m training a large model (similar to FAIRā€™s StarSpace) and itā€™s mostly about embedding lookups. The inputs to the model are embedding indices and I suspect the overhead of Python iteration over examples separately is so high that PyTorch canā€™t feed GPU with data fast enough. I get low GPU utilization as a result.

Is DataLoaders overhead (not being vectorized) something others have run into?
cc @jeremy (tried asking this q during the break but it was too long)

215

Was the GANModule in Part 1? Anyone remembers the lecture, might have missed it.

216

Are fastai callbacks synchronous or async?

4 Likes
217

sg,
what is diff between LearnerCb and TrackerCb ?

218

Most likely sync

219

Not presented, no. It was used behind the scenes.

1 Like
220

Synchronous.

3 Likes