How to use multiple gpus

fastai fastai dev
13

Thank you @sgugger . I did as you instructed and it worked. So just to recap (in case other people find it helpful), to train the RNNLearner.language_model with FastAI with multiple GPUs we do the following:

  1. Once we have our learn object, parallelize the model by executing learn.model = torch.nn.DataParallel(learn.model)
  2. Train as instructed in the docs
  3. Save the parallelized model learn.save('name')
  4. As instructed by @sgugger here, load the saved model, and strip out the word module from the keys of the dictionary and create and new state_dict
  5. Load the newly created state_dict into the model, which will now be unparallized
  6. Save both the model and encoder by executing learn.save('name-unpr') and learn.save_encoder('name').
14 Likes