@Leslie, @jeremy, @deanmark
I’ve been playing around with exactly that recently: I implemented custom sampler so that we could use less data to train quicker. My results so far indicate that this type of approach can speed up learning but it doesn’t converge to good end result. I am using CIFAR10 as a bencmark dataset and baseline is fast.ai dawnbench architecture.
e.g if our target is to find fastest time to 70-80% accuracy on CIFAR10 then small-sample training could win. But, if our target is 90+% accuracy, then small-sample training never achieves it. These are my results so far.
By the way, I modified fast.ai from here and there and my current fastest CIFAR-10 result is <11min to 93+% accuracy on my local machine (compared with <15 min to 94% acc using original fast.ai). I am trying to find this last 1% to push it over 94% 
@kcturgutlu, a bit related off-topic question, maybe you can help me out: ‘Wideresnet.py, line 37: multiply by 0.2’