IMDB Example from text application

Hi,

I’m working through the IMDB example under the text application of the docs and I’ve run into some problem. Here is the notebook I created following the doc.

I start out by importing everything from fastai.docs. Please note that I manually commented out the from fastai.vision import * line because of the issue discussed here.

After following the documentation my notebook hangs on this line:
learn = RNNLearner.language_model(data_lm, pretrained_fnames=['lstm_wt103', 'itos_wt103'], drop_mult=0.5)

I checked the path where all the files are downloaded and made sure that everything is where it should it be. I would appreciate some help in resolving this issue and my objective is to eventually run this code on a different dataset. Please let me know if you require any other information.

Thank you!

Try running it with the debugger and step through that function - let us know which step it hangs at.

OK I ran the debugger and traced the bottleneck to this line in basic_train.py:

self.model = self.model.to(self.data.device)

which essentially moves the model to the GPU. So this isn’t really a FastAI issue but more of an issue with either pytorch or the configuration of the machine I’m working on. Interestingly, I had reported here that running that snippet of code given by Jermey ran very fast. However, the same snippet of code runs very very slow now to point of patience running out. So I investigated further. I ran the following piece of code:

%%time
t = torch.randn(1)
t = t.cuda()

CPU times: user 4min 9s, sys: 5.67 s, total: 4min 15s
Wall time: 4min 16s

I didn’t have a separate timer but the Wall time seemed way longer than 4m 16s. Anyway, a couple of observations:

  1. Even though the initial cuda call takes very very long, subsequent GPU operations are very fast.
  2. A call back to cpu followed by another call to gpu on the same variable is very quick.
  3. In fact, except the very first cuda call, all subsequent cuda calls on all the variables are very fast.

After a dummy cuda call, I even tried from fast.vision import * and lo and behold it did not hang! I’m not sure why this is happening. A cursory search through the Pytorch forums did not get me an answer.

The moral of the story is that the very first cuda call in my notebook takes a very long time. As of now, I’ll basically be working by just invoking a dummy cuda call at the very beginning of my notebook, waiting for a few minutes for that call to finish and then proceeding with my work. Hopefully this might help others in the same situation until a more permanent solution can be realized.

Can you try updating to cuda92 both CUDA SDK and conda package (if you haven’t already) and installing the cuda92 version of pytorch-nightly?

I don’t have admin privileges on this machine, so unfortunately I don’t think I can unless I can do it without root. Getting the admins to install software is a pain as it has to go through multiple chains of commands.