Wanted to bring up this very important, and somewhat neglected, area of DL research.
Given the importance fast.ai has put on transfer learning, and the incredible results we get out of it, would it make sense for an implementation of some version of knowledge distillation to the library?
Now I understand that doesn’t need to be a class or object doing this, but it would be pretty nice if we had a somewhat efficient coding method to apply KD. If that is asking too much, then perhaps some pointers as to how current iteration of fast.ai could be used for this would be very helpful for me!
Thanks for your thoughts.