I’ve just begun getting familiar with the Fastai library and programming in general. Please forgive any oversights on my part in failing to communicate clearly and effectively on what I intend to express .
I studied the fastai callback system in over the last 2 days and this is probably my favourite feature in the Fastai library. This was the first time I was making a few callbacks of my own and felt that there can be a repo separate from the main Fastai library containing a host of custom callbacks. These may implement a host of small changes to the training process which users find themselves doing all time, or implementations of the latest research papers to just plug in and play over any supported models. Any particular widely used callbacks can then maybe be merged into the main library.
Here’s how I came about this idea:
I was training a model that uses the perceptual loss from Lesson 7, Part 1, V3 to segment out roads from satellite imagery. I had to convert the VGG16 model to a two-channel model (by essentially removing one channel from the first kernel) so it can take in two channels as inputs: roads and the background. After doing that, all the noise in the segmentation was cleaned up almost instantly and the roads produced were much more continuous and smoother than they were before with a vanilla U-Net with a ResNet34 backbone). (Although these effects again vanished as I increased image size and decreased batch size from 224, 8 to 1024, 1 respectively).
But training this model would take a long time: 1.5 hours for a single epoch (image size:1024). Also, I couldn’t bump up the batch size above 1 (this was on a P100 GPU).
So I figured I needed to automatically save the model at batch intervals instead of epoch intervals as my Gcloud instance was preemptible. Also, sometimes I just wanted to train for a few batches instead of a few epochs to test out how things were working. The final thing I needed to implement was gradient accumulation over multiple batches. So I went ahead and implemented all of things in fastai callbacks and decided to put it up online: https://github.com/akashpalrecha/custom-fastai-callbacks
Some of my friends working on a different model made good use of these callbacks and that prompted me to post about it on this forum.
Maybe as a joint effort, people can contribute to this repo and add any of the amazing callbacks they’ve created to help them train models better. I’m sure a lot many people would find it very useful.
I’m not sure how the development process works in general and if it’s appropriate to make such an effort to create a repo intended just for callbacks. But I have a feeling that this can be very useful in the future. Also, I would like it if someone can see my code and suggest problems/improvements with it. I’m still a beginner and would love any sort of feedback on my work.
What are everyone’s thoughts on this?
Also, callback ideas I’ll be working on next:
- Enable training over a percentage subset of the data
- Showing results every
Nepochs or iterations