Hi @danaludwig
Recently, there was an exciting progress in multi-node distributed data parallel training in fastai…
If you search the forum you will find some examples…
Particularly, see this Cifar training example that Jeremy, Sylvain, Andrew and Brett managed to smash everybody in April 2018 in the DAWNBench Stanford competition …
More about it here…
This forum thread related…
I am very excited about this to be already integrated in fastai 
Fastai is peerless!