Thanks Malcolm this helps a lot. I’ll try to get it working and then report what worked/didnt back here later. Cheers.
You’re correct, @Pomo.
If you’re using pip, this is just a straightforward:
pip install torch_nightly -f https://download.pytorch.org/whl/nightly/cu100/torch_nightly.html
# replace cu100 with cu90 if you’re on cuda-9.0
With conda it’s tricky. Installing pytorch-nightly via conda would be a problem with fastai updates which relies on pytorch (different package name), so when you update fastai it may force to re-install pytorch (not nightly) and you would lose pytorch-nightly
There are 3 ways you can go about it:
-
use a dedicated conda environment for the part2 lessons, which requires no
fastaiconda create -y python=3.7 --name fastai-part2 conda activate fastai-part2 conda install -y -c pytorch pytorch-nightly torchvisioni.e. don’t install
fastaiin that conda env. -
install
pytorch-nightlyvia pip into your normal fastai conda environment - conda won’t know that you did that and won’t overwrite itconda install -c pytorch -c fastai fastai pip install torch_nightly -f https://download.pytorch.org/whl/nightly/cu100/torch_nightly.html # replace cu100 with cu90 if you’re on cuda-9.0 -
always force reinstall
pytorch-nightlyafterpytorchgets installed/updated in condaconda install -c pytorch -c fastai fastai conda install -c pytorch pytorch-nightly --force-reinstall
I updated the first post with this info. Let me know if you have difficulties with any of them.
And should we keep updating pytorch-nightly using:
conda install -c pytorch pytorch-nightly
You probably won’t need to update it again, pytorch-nightly was required in this lesson due to some pytorch bug that has been fixed recently, so chances are that whatever pytorch-nightly you installed a few days ago will be just fine for the rest of the course. And soon pytorch-1.1.0 will be released, so fastai will require that instead.
To see how the pytorch and pytorch-nightly packages overwrite each other:
conda uninstall pytorch pytorch-nightly torchvision
conda install -c pytorch pytorch torchvision
python -c 'import sys, torch; print(torch.__version__, sys.modules["torch"])'
1.0.1.post2 <module 'torch' from '.../site-packages/torch/__init__.py'>
and once we get pytorch-nightly, it overwrites pytorch, yet conda thinks they are 2 totally different packages.
conda install -c pytorch pytorch-nightly
python -c 'import sys, torch; print(torch.__version__, sys.modules["torch"])'
1.0.0.dev20190405 <module 'torch' from '.../site-packages/torch/__init__.py'>
I trimmed the output path so that it fits into the width, but the purpose is to show that it loads the exact same path in both cases.
3 Likes
I’m curious, does anyone know which features necessitated use of nightly pytorch for this lesson? I guess I haven’t kept up to date enough on the changelog…
Batch size is the number of examples used compute the parameter updates. The larger the batch size, the less ‘noisy’ are the parameter updates. Batch size is a hyperparameter because it affects the model performance. You tune it by varying it till you get the best performance.
1 Like
Hi @tanyaroosta
Jeremy said that softmax is the correct way to compute class probabilities for “multi-class” problems, where every example belongs to one and only one class.
And Jeremy says that softmax should not be used to compute class probabilities for “multi-label” problems, where an example can belong to more than one class. An alternative method to compute class probabilities for the “multi-label” case is to apply a sigmoid followed by binary cross-entropy to the output for each label. This is method Jeremy advocates.
My own question is: why is it wrong to use softmax to compute class probabilities in the “multi-label” case?
It’s really just the var() taking multiple dims. We also hoped to get to some JIT in the lesson, but we’ve pushed that back a little. (Maybe next week.)
3 Likes
One idea I have (not sure if it works or not) is this, duplicate each training example 11 times, and append the numbers 0-10 respectively (representing all the classes except unknown) to the end of those training vectors. The network gives a binary output (therefore binary cross entropy) to show whether each training example corresponds to the class index appended.
During inference time, you would compare each word to all classes, and the one that gives the highest probability (apply softmax to the output) would be your predicted class. Or if the highest probability you get is less than some threshold, you should predict unknown. Not sure if this actually works or not, its just a thought.
The implementation of running batch norm looks like a form of meta learning? Thinking of it in this way: We could maybe add more meta learning techniques like calculating the hessian instead of the gradiant on the first epoch.
Any thoughts
one minor feedback i have regarding the callbacks and the way it is currently being implemented is that there is too much of data coupling across the runner class and callbacks and amongst the callbacks themselves. Looking through the code though it seems like individual callbacks must be independent, the callbacks for instance Learner and recorder sharing attributes such as lrs isn’t clean. Though this isn’t really a detriment to using this for now, it might get pretty hard to refactor later on, similar to the issue seen in a single “fit” function.
This is something I’m a little suspicious of myself. I have some experience with systems where everything was turned into a “plug-in” of some kind (similar to how callbacks are used here), and while it may seem clean in the beginning, eventually you’ll run into the problem that now these callbacks need a way to work together and that this work must be orchestrated somehow. And because all this logic is spread out across many different classes, it can be hard to understand exactly how they interact. Simplifying the training loop itself and moving all the complexity elsewhere doesn’t mean the complexity is gone – and may actually introduce additional complexity because stuff isn’t in one place anymore. I’m not sure if this will be a problem here, but it requires a careful design.
1 Like
It’s been tricky to get right, but after implementing dozens of callbacks we’ve found things are working out great. Complex stuff that requires big projects like GANs, mixed precision training, etc are just a few lines of code in fastai.
2 Likes
Hi
I am working on video link annotations for Lesson 10 notebooks. I noticed that the current version of 05a_foundations.ipynb does not match the version presented in the video. There are some parts missing. In particular, cells related to discussion about partials and entire “Callbacks as callable classes” section is missing. Is it possible to commit a version that Jeremy Howard used for the lecture to the repository so that annotated notebook would match the video content?
What’s __constants__ = ['eps'] in LayerNorm in 07_batchnorm.ipynb?
I don’t seem to find such feature in python - I only found some references to it in pytorch jit discussions.
I noticed that the current version of
05a_foundations.ipynbdoes not match the version presented in the video.
Perhaps you have an outdated checkout and you need to update it? git pull
It’s all there: https://github.com/fastai/fastai_docs/blob/master/dev_course/dl2/05a_foundations.ipynb
You can also check the history of modifications - I looked - none of them removed anything. https://github.com/fastai/fastai_docs/commits/master/dev_course/dl2/05a_foundations.ipynb
I will check again, but I did git pull right before I started working on annotation.
Do: git log --oneline
and check that you’re on origin master and not a forked branch. Should say at this moment:
5b26a49 (HEAD -> master, origin/master, origin/HEAD) Merge branch 'master' of github.com:fastai/fastai_docs
But most likely you’re not on upstream master. Chances are that you forked the fastai_docs repo before those files were updated and your forked master and branch are out of sync with upstream. So when you run git pull you’re pulling from your forked master, which is out of sync with upstream master. https://docs.fast.ai/dev/git.html#how-to-keep-your-feature-branch-up-to-date
So those bits you are missing were added after you did the fork, hence the idea that they were removed. You’re “back in the future” - they weren’t added yet 
torch tensor is throwing up some silly error, is anyone else seeing this as well ?
x.var((1,), keepdim=True)
Traceback (most recent call last):
File “”, line 1, in
TypeError: var(): argument ‘dim’ (position 1) must be int, not tuple
torch.version
‘1.0.1.post2’
I just did a fresh clone and the "Callbacks as callable classes " section is there, as it should be.
But when I check the log on the folder that I did a git pull on, I get exactly what I should get:
(base) serge@gpu:~/annotations/fastai_docs$ git log --oneline
5b26a49 (HEAD -> master, origin/master, origin/HEAD) Merge branch 'master' of github.com:fastai/fastai_docs
It is weird, isn’t it?
I can get right version from the freshly cloned repo folder, so we can call this issue closed, I guess. Thank you for the help!
Hmm, perhaps your repo is somehow misconfigured, what do you get when you run this in that folder?
echo $(git config --get remote.origin.url | sed 's|^.*//||; s/.*@//; s/[^:/]\+[:/]//; s/.git$//')/$(git branch | sed -n '/\* /s///p')
If you’re on bash, I highly recommend: https://docs.fast.ai/dev/git.html#bash-git-prompt which removes the guessing - it always tells you where you’re at. like cwd in your prompt, but git-wise.