I am trying to make use of either distributed or parallel training using fastai and SageMaker notebooks or training jobs (somewhat fixed on using this service based on my team). I am running code on a ml.p3.8xlarge with 4x V100, but I cannot get any speed ups with any of the approaches I have taken.
After spinning up the ml.p3.8xlarge notebook instance, here is the set up in my notebook using the pytorch env:
%%bash
pip install fastai==2.0.0 fastcore==1.0.0
sudo mkdir -p /opt/ml/input/data/collab
sudo chmod 777 /opt/ml/input/data/collab
Here is the code I am testing:
import fastai, fastcore, torch
print(f'fastai {fastai.__version__}')
print(f'fastcore {fastcore.__version__}')
print(f'torch {torch.__version__}')
from fastai.collab import *
from fastai.tabular.all import *
from fastai.distributed import *
path = untar_data(URLs.ML_100k, dest="/opt/ml/input/data/collab")
ratings = pd.read_csv(
path/'u.data',
delimiter='\t',
header=None,
names=['user','movie','rating','timestamp']
)
movies = pd.read_csv(
path/'u.item',
delimiter='|',
encoding='latin-1',
usecols=(0,1),
names=['movie','title'],
header=None,
)
ratings = ratings.merge(movies)
dls = CollabDataLoaders.from_df(ratings, item_name='title', bs=64)
n_users = len(dls.classes['user'])
n_movies = len(dls.classes['title'])
n_factors = 64
model = EmbeddingDotBias(n_factors, n_users, n_movies)
learn = Learner(dls, model, loss_func=MSELossFlat())
print(learn.model)
print("rank_distrib():", rank_distrib())
print("num_distrib():", num_distrib())
print("torch.cuda.device_count():", torch.cuda.device_count())
epochs, lr = 5, 5e-3
print('learn.fit_one_cycle')
learn.fit_one_cycle(epochs, lr)
print('with learn.distrib_ctx():')
with learn.distrib_ctx():
learn.fit_one_cycle(epochs, lr)
print('with learn.distrib_ctx(torch.cuda.device_count()-1):')
with learn.distrib_ctx(torch.cuda.device_count()-1):
learn.fit_one_cycle(epochs, lr)
print('with learn.parallel_ctx():')
with learn.parallel_ctx():
learn.fit_one_cycle(epochs, lr)
print('nn.DataParallel(learn.model)')
if torch.cuda.device_count() > 1:
learn.model = nn.DataParallel(learn.model)
learn.fit_one_cycle(epochs, lr)
Here is the output from running code as a script:
sh-4.2$ /home/ec2-user/anaconda3/envs/pytorch_p36/bin/python /home/ec2-user/SageMaker/cf.py
fastai 2.0.0
fastcore 0.1.39
torch 1.6.0
EmbeddingDotBias(
(u_weight): Embedding(944, 64)
(i_weight): Embedding(1665, 64)
(u_bias): Embedding(944, 1)
(i_bias): Embedding(1665, 1)
)
rank_distrib(): 0
num_distrib(): 0
torch.cuda.device_count(): 4
learn.fit_one_cycle
epoch train_loss valid_loss time
0 1.153435 1.154428 00:11
1 0.957201 0.954827 00:11
2 0.816548 0.878350 00:11
with learn.distrib_ctx():
epoch train_loss valid_loss time
0 0.999254 1.040871 00:11
1 0.821853 0.914921 00:11
2 0.658059 0.845227 00:11
with learn.distrib_ctx(torch.cuda.device_count()-1):
epoch train_loss valid_loss time
0 0.749317 0.997568 00:11
1 0.580846 0.912386 00:11
2 0.381058 0.878295 00:11
with learn.parallel_ctx():
epoch train_loss valid_loss time
0 0.514148 1.025872 00:25
1 0.383893 0.996381 00:18
2 0.204836 0.970403 00:18
nn.DataParallel(learn.model)
epoch train_loss valid_loss time
0 0.341708 1.103849 00:16
1 0.272570 1.067705 00:16
2 0.134262 1.055507 00:16
Using the command nvidia-smi dmon -s u to watch GPU usage, I can see that only the training with DataParallel (using with learn.parallel_ctx(): and nn.DataParallel(learn.model)) show GPU ids 1,2,3 being used. The problem is the data parallel is slower, even when I have tried increasing batch size or embedding size.
Any help with this would be appreciated. I have a much larger collaborative filtering model I would like to use that is experiencing the same issues as this movie example and I need to reduce the training time hopefully with the use of parallel/distributed training.