[Adv] Significant changes to fastai just pushed

Part 1 (2018)
21

Hi guys, let me look in to the template and see if can update for newest changes.

1 Like
22

@dillon Iā€™m seeing the same problem. Paperspace is mentioned in this thread: https://github.com/pytorch/pytorch/issues/2575 . Seems itā€™s not just a fastai template issue. To replicate, create new instance and:

git clone https://github.com/fastai/fastai.git
cd fastai
conda env update
source activate fastai
jupyter notebook

Then open courses/dl1/lesson1.ipynb and run the import cells.

1 Like
23

Was there a design discussion thread on why opencv had to be removed? I tried looking up on the forums but could not stumble upon one.

My understanding has always been that opencv and PIL should both co-exist ā€“ personally, opencv had helped me with bounding boxes on images quite easily with a couple of API calls, however if there is actual performance improvement in removing opencv, I am looking to dig deeper into this.

24

The basic issue was this one: https://github.com/pytorch/pytorch/issues/1838

1 Like
25

@jeremy it looks like a few people have seen similar issues and it boils down to import order. Still not 100% sure why that is.

i.e. https://github.com/facebookresearch/ParlAI/issues/331 and https://github.com/pytorch/pytorch/issues/2083

In any case, I resolved it by importing torch before anything else. which seems to do the trick.

09 PM
Screen Shot 2017-11-13 at 1.23.09 PM.jpg1572Ɨ940 313 KB

2 Likes
26

Thanks @dillon - confirmed that works for me. Would be nice to find a better fix eventually!

27

definitely! Iā€™m still digging in. It seems to be a very odd bug. Will post back to this thread with any findings. Thanks for the help :slight_smile:

28

@dillon when I run the first in lesson1 learn.fit line on paperspace it totally freezes for me. Does it work for you?

29

Hi, same problems here, (I just logged in and shift + entered to see if it worked).

Imports dont give errors with the fix but first chunk with learn.fit gives this error that was not before:

image
image.png1296Ɨ264 16.3 KB

30

@miguel_perez try deleting your tmp folder, git pull, and conda env update.

31

@jeremy did that now, + added source activate fastai to .bashrc (In AWS it was there already but not in my Paperspace).

result: error is gone, but learn.fit freezes forever, just as you described :frowning_face:

32

@dillon just a note to see above from @miguel_perez - weā€™re both finding the freezing issue on Paperspace.

33

@jeremy @miguel_perez For some reason I cannot repro on a new fast.ai instance. What I did:

  1. git pull
  2. conda env update
  3. source activate fastai
  4. add import torch above all other lines
    image
    image.jpg1107Ɨ158 62.8 KB
  5. Manually created a /cache/tmp directory and changed
os.makedirs('/cache/tmp', exist_ok=True)
!ln -fs /cache/tmp {PATH}

to

os.makedirs('./cache/tmp', exist_ok=True)
!ln -fs /cache/tmp {PATH}

(missing period on my system but maybe not relevant)

  1. run through cells.

image
image.jpg1375Ɨ844 389 KB

34

@dillon, reconected and tried again, even added the dot :grinning: ā€¦

but no luck, same as before, kernel freezes there, forever.

35

@dillon it seems it wasnā€™t really frozen forever - it was just taking a really long time to create the pytorch model for some reason. Iā€™ve seen other mentions of that in this forum relating to paperspace. Eventually it finishes, and works fine. But restarting the notebook causes the extreme delay on model creation again.

I should take ~5 secs to create the model.

36

@dillon, any news about this problem? I would like to get used to Paperspace during the course, really like the environmentā€¦ its a pitty that course weeks are passing without using it. :worried:

37

Hi @A_TF57, read your post as trying to do conda env update on Crestle. After git pull But weirdly,Anaconda doesnā€™t seem to be installed. Did you install it yourself?

And more in general, have you been succesfully using Crestle for lesson notebooks until now?

38

I did but seems I didnā€™t have to. If you read the FAQs, you can install extra packages using pip3 if you want to. Everything else is already setup for you to use.

Yes! I have been using it quite regularly. :slight_smile:

2 Likes
39

Understood. I was wanting to do ā€œconda env updateā€ but I guess Crestle manages the environment by itselfā€¦

Your answer saved me from switching to the ā€œinstall eveythingā€ mode. I will try pip3 and see if I also can make everything work, thanks a lot @A_TF57

1 Like
40

Hi guys, Iā€™m not able to reproduce the issue 100% of the time but it does seem to be some weird interaction with import order. I have also not been able to reliably repro the issue of the freezing model creation.

Iā€™m having the team here build out a new template to see if we can get a fresh base to work from.

Thanks for being so patient!

4 Likes