I’m working through the IMDB example under the
text application of the docs and I’ve run into some problem. Here is the notebook I created following the doc.
I start out by importing everything from
fastai.docs. Please note that I manually commented out the
from fastai.vision import * line because of the issue discussed here.
After following the documentation my notebook hangs on this line:
learn = RNNLearner.language_model(data_lm, pretrained_fnames=['lstm_wt103', 'itos_wt103'], drop_mult=0.5)
I checked the path where all the files are downloaded and made sure that everything is where it should it be. I would appreciate some help in resolving this issue and my objective is to eventually run this code on a different dataset. Please let me know if you require any other information.
Try running it with the debugger and step through that function - let us know which step it hangs at.
OK I ran the debugger and traced the bottleneck to this line in
self.model = self.model.to(self.data.device)
which essentially moves the model to the GPU. So this isn’t really a FastAI issue but more of an issue with either pytorch or the configuration of the machine I’m working on. Interestingly, I had reported here that running that snippet of code given by Jermey ran very fast. However, the same snippet of code runs very very slow now to point of patience running out. So I investigated further. I ran the following piece of code:
t = torch.randn(1)
t = t.cuda()
CPU times: user 4min 9s, sys: 5.67 s, total: 4min 15s
Wall time: 4min 16s
I didn’t have a separate timer but the
Wall time seemed way longer than 4m 16s. Anyway, a couple of observations:
- Even though the initial
cuda call takes very very long, subsequent GPU operations are very fast.
- A call back to
cpu followed by another call to
gpu on the same variable is very quick.
- In fact, except the very first
cuda call, all subsequent
cuda calls on all the variables are very fast.
After a dummy
cuda call, I even tried
from fast.vision import * and lo and behold it did not hang! I’m not sure why this is happening. A cursory search through the Pytorch forums did not get me an answer.
The moral of the story is that the very first
cuda call in my notebook takes a very long time. As of now, I’ll basically be working by just invoking a dummy
cuda call at the very beginning of my notebook, waiting for a few minutes for that call to finish and then proceeding with my work. Hopefully this might help others in the same situation until a more permanent solution can be realized.
Can you try updating to cuda92 both CUDA SDK and conda package (if you haven’t already) and installing the cuda92 version of pytorch-nightly?
I don’t have admin privileges on this machine, so unfortunately I don’t think I can unless I can do it without
root. Getting the admins to install software is a pain as it has to go through multiple chains of commands.