[Adv] Significant changes to fastai just pushed

Hi guys, let me look in to the template and see if can update for newest changes.

1 Like

@dillon I’m seeing the same problem. Paperspace is mentioned in this thread: https://github.com/pytorch/pytorch/issues/2575 . Seems it’s not just a fastai template issue. To replicate, create new instance and:

git clone https://github.com/fastai/fastai.git
cd fastai
conda env update
source activate fastai
jupyter notebook

Then open courses/dl1/lesson1.ipynb and run the import cells.

1 Like

Was there a design discussion thread on why opencv had to be removed? I tried looking up on the forums but could not stumble upon one.

My understanding has always been that opencv and PIL should both co-exist – personally, opencv had helped me with bounding boxes on images quite easily with a couple of API calls, however if there is actual performance improvement in removing opencv, I am looking to dig deeper into this.

The basic issue was this one: https://github.com/pytorch/pytorch/issues/1838

1 Like

@jeremy it looks like a few people have seen similar issues and it boils down to import order. Still not 100% sure why that is.

i.e. https://github.com/facebookresearch/ParlAI/issues/331 and https://github.com/pytorch/pytorch/issues/2083

In any case, I resolved it by importing torch before anything else. which seems to do the trick.

2 Likes

Thanks @dillon - confirmed that works for me. Would be nice to find a better fix eventually!

definitely! I’m still digging in. It seems to be a very odd bug. Will post back to this thread with any findings. Thanks for the help :slight_smile:

@dillon when I run the first in lesson1 learn.fit line on paperspace it totally freezes for me. Does it work for you?

Hi, same problems here, (I just logged in and shift + entered to see if it worked).

Imports dont give errors with the fix but first chunk with learn.fit gives this error that was not before:

@miguel_perez try deleting your tmp folder, git pull, and conda env update.

@jeremy did that now, + added source activate fastai to .bashrc (In AWS it was there already but not in my Paperspace).

result: error is gone, but learn.fit freezes forever, just as you described :frowning_face:

@dillon just a note to see above from @miguel_perez - we’re both finding the freezing issue on Paperspace.

@jeremy @miguel_perez For some reason I cannot repro on a new fast.ai instance. What I did:

  1. git pull
  2. conda env update
  3. source activate fastai
  4. add import torch above all other lines
  5. Manually created a /cache/tmp directory and changed
os.makedirs('/cache/tmp', exist_ok=True)
!ln -fs /cache/tmp {PATH}

to

os.makedirs('./cache/tmp', exist_ok=True)
!ln -fs /cache/tmp {PATH}

(missing period on my system but maybe not relevant)

  1. run through cells.

@dillon, reconected and tried again, even added the dot :grinning:

but no luck, same as before, kernel freezes there, forever.

@dillon it seems it wasn’t really frozen forever - it was just taking a really long time to create the pytorch model for some reason. I’ve seen other mentions of that in this forum relating to paperspace. Eventually it finishes, and works fine. But restarting the notebook causes the extreme delay on model creation again.

I should take ~5 secs to create the model.

@dillon, any news about this problem? I would like to get used to Paperspace during the course, really like the environment… its a pitty that course weeks are passing without using it. :worried:

Hi @A_TF57, read your post as trying to do conda env update on Crestle. After git pull But weirdly,Anaconda doesn’t seem to be installed. Did you install it yourself?

And more in general, have you been succesfully using Crestle for lesson notebooks until now?

I did but seems I didn’t have to. If you read the FAQs, you can install extra packages using pip3 if you want to. Everything else is already setup for you to use.

Yes! I have been using it quite regularly. :slight_smile:

2 Likes

Understood. I was wanting to do “conda env update” but I guess Crestle manages the environment by itself…

Your answer saved me from switching to the “install eveything” mode. I will try pip3 and see if I also can make everything work, thanks a lot @A_TF57

1 Like

Hi guys, I’m not able to reproduce the issue 100% of the time but it does seem to be some weird interaction with import order. I have also not been able to reliably repro the issue of the freezing model creation.

I’m having the team here build out a new template to see if we can get a fresh base to work from.

Thanks for being so patient!

4 Likes