Developer chat


#530

I’m getting the same thing.


#531

Ok found the problem. Needed to update fastprogress, as the way it’s being interacted with in the fast.ai code has changed. @MicPie


#532

Weird, how did you update your library? Normally conda would have updated fastprogress for you.


#533

I used pip to upgrade fastprogress directly.

I’m embarrassed to say, I don’t know how to resolve an impasse I currently have between jupyter_contrib_nbextensions and pytorch. Until I can get that sorted out, conda env update isn’t working.


(Michael) #534

I am using the developer install and I could fix it with conda install -c fastai fastprogress to get the newest fastprogress version. Thanks for the fast help. :slight_smile:


(Stas Bekman) #535

Not if you do git pull. I have experienced this problem several times.

I think the best solution is to assert for a specific fasprogress version in the fastai code (and not just package dependencies). That way regardless of how people get their fastai updates, they will know right away that they need a newer version of fastprogress.

It doesn’t look this is common in python to perform that at the code level. Or at least I couldn’t find almost any references on how to do it, or even people asking about how to do it.

I found this suggestion:

import pkg_resources
pkg_resources.require("fastprogress>=0.1.18")
import fastprogress

And if the version is insufficient, you’d get:

python -c 'import pkg_resources; pkg_resources.require("fastprogress>=0.1.19"); import fastprogress'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/stas/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 898, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/stas/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 789, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (fastprogress 0.1.18 (/home/stas/anaconda3/envs/pytorch-dev/lib/python3.6/site-packages), Requirement.parse('fastprogress>=0.1.19'))

(Stas Bekman) #536

fastai/environment.yml and environment-cpu.yml belong to the old fastai (0.7). conda env update is no longer the way to update your fastai-1.x environment. This is unfortunate, but we can’t remove these because the course-v2 video instructions rely on this setup. Eventually, once course-v3 p1 and p2 will be completed, they will probably be moved to where they belong - under old/.

instead do a normal install (even if you’re updating a previous install): conda install -c fastai fastai or pip install fastai and everything that needs to be updated will be updated.

It’s documented now here. Thank you for bringing up this issue, @ABertl.


#537

Thanks for the help, I’ll try this out in the next few days.

To give some detail about what brought me to this point, I followed the recommendations of the MOOC and got a paperspace machine with fast.ai preinstalled, and have been git pulling since with monthly-ish conda env update. (I eventually ran into the problem with nbextensions but really didn’t want to give it up!)

Somewhere in the middle of working my way through the MOOC I realized the lessons were being done with an old version of fast.ai, and every time I was git pulling, none of the updates were hitting what I was using. Eventually I decided to try out 1.0.

I’m figuring it out, but it wasn’t real clear to me at first how to make the transition.


(Stas Bekman) #538

It can be very confusing indeed, especially given that fastai-0.7 files aren’t all in the same place, due to historic reasons I explained above. If you’re following the course-v2 MOOC you better stick to fastai-0.7, since the API has dramatically changed and it’ll be very difficult to make sense of the course-v2 videos using fastai-1.x. And if you do then, yes, you still need to use conda env update exactly as in the past. But you won’t need fastprogress then.

It wasn’t clear from your original post that you were referring to the fastai-0.7 version, but we do know that now. And if you do have installation issues with fastai-0.7 - please post those here.


(Stas Bekman) #539

I was curious to see if i could get to generate a train_ds w/ labels using the new API, but it appears to be very rough at the edges and failing in many ways if the prescribed way of train/valid split is not followed to the point, e.g.:

train_ds = ImageItemList.from_folder(path).random_split_by_pct(0)
...
IndexError: index 0 is out of bounds for axis 0 with size 0

and it also has an edge problem of generating 1 item in valid ds, which shouldn’t be there, since train ds, already has the full number of available items.

and this fails too:

train_ds = ImageItemList.from_folder(path)
train_ds
train_ds = train_ds.label_from_folder()
train_ds
...
TypeError: 'NoneType' object is not subscriptable

Is this a work-in-progress, or is the new API set to not allow any non-fastai-way of creating _ds objects (i.e. w/o valid set together with train_ds)?

I was experimenting with fashion mnist, but mnist will do too if you’d like to try it (i.e. path = /path/to/mnist)

I was planning to feed train_ds to ImageDataBunch.create(train_ds...) as in nbs/dl1/lesson7-wgan.ipynb, but the example there has no labels and w/o labels it works.

Thank you.

p.s. the 2 ways of using the test dataset w/ labels to validate against are now documented here.


(Jeremy Howard (Admin)) #540

Neither. It simply means you found a bug.


(RobG) #541

It looks like v1.0 has been released https://github.com/pytorch/pytorch/releases


(Stas Bekman) #542

FYI, @sgugger implemented this - so future git pull updates will be fastprogress-progress aware.


(Stas Bekman) #543

FYI, a hotfix fastai-1.0.36.post1 pypi release was made (a fix in regex dependencies which was conflicting with spacy)

And it looks like pip behaves strangely with .postX releases. When I first run:

pip install fastai

it won’t install fastai-1.0.36.post1 (installed fastai-1.0.34 instead, due to conflicts in 1.0.36 I suppose). But when I did:

pip uninstall fastai -y
pip install fastai

it did install fastai-1.0.36.post1. That’s a very inconsistent behavior.


(Mikhail Grankin) #544

I think that I found a mistake in the simplest CV example. Normalisation was done with imagenet_stats instead of the actual dataset stats (in our case - mnist_stats). I’ve made a pull request which was rejected. Can somebody look at the PR and express the third opinion?


(Kaspar Lund) #545

When using a network pretrained on imagenet then you must use image-netstat to normalise - always


#546

On this, added a no_split method that will create a training set with everything and a validation set with nothing. Note that the empty validation set will still have a length of 1 otherwise the pytorch dataloader will complain (you have to test len(valid_ds.items) to check if it’s 0 or not.
Will change random_split_by_pct so that a 0 falls back to no_split, but this should be the method used.