[Adv] Significant changes to fastai just pushed

Part 1 (2018)
25

@jeremy it looks like a few people have seen similar issues and it boils down to import order. Still not 100% sure why that is.

i.e. https://github.com/facebookresearch/ParlAI/issues/331 and https://github.com/pytorch/pytorch/issues/2083

In any case, I resolved it by importing torch before anything else. which seems to do the trick.

09 PM
Screen Shot 2017-11-13 at 1.23.09 PM.jpg1572×940 313 KB

2 Likes
26

Thanks @dillon - confirmed that works for me. Would be nice to find a better fix eventually!

27

definitely! I’m still digging in. It seems to be a very odd bug. Will post back to this thread with any findings. Thanks for the help :slight_smile:

28

@dillon when I run the first in lesson1 learn.fit line on paperspace it totally freezes for me. Does it work for you?

29

Hi, same problems here, (I just logged in and shift + entered to see if it worked).

Imports dont give errors with the fix but first chunk with learn.fit gives this error that was not before:

image
image.png1296×264 16.3 KB

30

@miguel_perez try deleting your tmp folder, git pull, and conda env update.

31

@jeremy did that now, + added source activate fastai to .bashrc (In AWS it was there already but not in my Paperspace).

result: error is gone, but learn.fit freezes forever, just as you described :frowning_face:

32

@dillon just a note to see above from @miguel_perez - we’re both finding the freezing issue on Paperspace.

33

@jeremy @miguel_perez For some reason I cannot repro on a new fast.ai instance. What I did:

  1. git pull
  2. conda env update
  3. source activate fastai
  4. add import torch above all other lines
    image
    image.jpg1107×158 62.8 KB
  5. Manually created a /cache/tmp directory and changed
os.makedirs('/cache/tmp', exist_ok=True)
!ln -fs /cache/tmp {PATH}

to

os.makedirs('./cache/tmp', exist_ok=True)
!ln -fs /cache/tmp {PATH}

(missing period on my system but maybe not relevant)

  1. run through cells.

image
image.jpg1375×844 389 KB

34

@dillon, reconected and tried again, even added the dot :grinning: …

but no luck, same as before, kernel freezes there, forever.

35

@dillon it seems it wasn’t really frozen forever - it was just taking a really long time to create the pytorch model for some reason. I’ve seen other mentions of that in this forum relating to paperspace. Eventually it finishes, and works fine. But restarting the notebook causes the extreme delay on model creation again.

I should take ~5 secs to create the model.

36

@dillon, any news about this problem? I would like to get used to Paperspace during the course, really like the environment… its a pitty that course weeks are passing without using it. :worried:

37

Hi @A_TF57, read your post as trying to do conda env update on Crestle. After git pull But weirdly,Anaconda doesn’t seem to be installed. Did you install it yourself?

And more in general, have you been succesfully using Crestle for lesson notebooks until now?

38

I did but seems I didn’t have to. If you read the FAQs, you can install extra packages using pip3 if you want to. Everything else is already setup for you to use.

Yes! I have been using it quite regularly. :slight_smile:

2 Likes
39

Understood. I was wanting to do “conda env update” but I guess Crestle manages the environment by itself…

Your answer saved me from switching to the “install eveything” mode. I will try pip3 and see if I also can make everything work, thanks a lot @A_TF57

1 Like
40

Hi guys, I’m not able to reproduce the issue 100% of the time but it does seem to be some weird interaction with import order. I have also not been able to reliably repro the issue of the freezing model creation.

I’m having the team here build out a new template to see if we can get a fresh base to work from.

Thanks for being so patient!

4 Likes
41

Many thanks for the update @dillon . It’s great to know that you’re working on it.

BTW pytorch 0.3 is about to come out. It has a massive improvement in group convolution speed, so you’ll definitely want to include it when you can!

42

OK, so I spent the last few days building out different versions of the template with different drivers/ base OS/etc.

  1. It seems that the dlopen: cannot load any more object with static TLS error only occurs on Ubuntu 14.04 and when I switched to a Ubuntu 16.04 template that issue went away entirely.
  2. That said, the new 16.04 template is now freezing on the learn.fit() consistently just as you mentioned. What I notice is that the CPU is getting hit really hard so I’m going to have to dig in to what this method is doing. From the outside it almost looks to me like there is a memory leak in the method. Virtual memory use explodes as CPU starts to really heat up.

@jeremy just as you mentioned it eventually works but something doesn’t look right and it can take 5 minutes to get going. This is on a brand new Ubuntu 16.04 template where I installed new drivers, fresh Conda install, etc.

image
image.jpg1290×781 441 KB

If anyone is interested in helping debug I am happy to share my new instance :slight_smile:

43

Sure. I can give it a try. Most likely later in the evening today. I am not using Paperspace at the moment, but used it in the past and will most likely return to it after a few months. So ping me if I can help.

44

@dillon I was seeing freezes on constructing learn, not on fitting. It might be worth doing a git pull and trying again, because I switched from ProcessPool to ThreadPool last night so if it was a multiprocessing issue this might well fix it.

We’re not seeing the problem on any other platform BTW, so it’s something specific to your config. Are you using docker? There are various threads around about docker and pytorch IIRC…