So, Is this the equation?
Our architecture ~= Predefined architecture (No change) + second last layer (Calculating activations for this layer for the provided data?) + last layer (Output layer)
1 Like
Pretty close. We add a few layers to the end, but in practice what you describe is close enough 
1 Like
Questions -Original Q and Jeremy’s reply:
This is what we do:
- We find an optimal ‘lr’ and feed into ‘Adam’
- ‘Adam’ an adaptive optimizer will start with our ‘lr’ and reduce overall loss. The reason we are using ‘Adam’ because it has adaptive ‘lr’ technique?
- If statement 2 is correct, then why are we explicitly supplying 3 learning rates (when we unfreeze) in the code?
- If statement 2 is incorrect, then why not use SGD. What makes ‘Adam’ special apart from adaptive ‘lr’? Guess, it’ll be covered in future lectures?
We specify 3 learning rates for different layer groups and not for one layer. Different layer groups need different amount of fine tuning and hence different learning rates. Before unfreezing, we were only training the last layer and we only needed to supply one learning rate. After unfreezing, if we supply only learning rate, fastai library will use the same learning rate for all the layer groups and this may not be ideal.
So, ‘Adam’ when used without unfreezing will adapt learning rate over time for last layer?
How does Adam’s adaptive nature help just for the last layer?
Trying to test Dogs v Cats super-charged! ipynb getting weights not found exception. Where can I get these weights. Is there any pre-requisite to run these ipynb
FileNotFoundError: [Errno 2] No such file or directory: ‘…/fastai/courses/dl1/fastai/weights/resnext_50_32x4d.pth’
1 Like
Download the weights from http://files.fast.ai/models/weights.tgz and unzip it into that ‘fastai/courses/dl1/fastai/weights/’ folder.
6 Likes
Thanks a lot!
Adam Optimization vs SGDR
Adam optimization contains a momentum parameter that controls how quickly the optimizer reaches the global/local minimum.
The momentum parameter is in the range 0 - 1. Values close to 1 represent 'high` momentum.
The higher the momentum, the more likely the optimizer will overshoot the minimum.
Question: Can you simulate SGDR by manipulating the momentum parameter in Adam optimization (at specific times in the iteration cycle), without varying the actual learning rate?
I was among a few unfortunate people who do not have free $500 AWS access. Therefore i am trying to test everything in my local machine. When I try to run lesson1-sgd.ipynb. I am getting following error. I know this error is because model is trained in GPU that we are tying to run in CPU…But I do not know how to solve this any idea?
AssertionError: Torch not compiled with CUDA enabled
I am getting:
FileNotFoundError: [Errno 2] No such file or directory: 'wgts/resnext_50_32x4d.pth’
am i missing something?
i did git pull before I started
Please check this post
You probably have installed PyTorch with Cuda. If you pip uninstall torch and then reinstall using the non-cuda version it should not give that error. To install non-cuda version see the Getting Started section in http://pytorch.org/
But please note that you may not be able to run things to completion in your local (non-GPU) machine unless you wait for a very long time. The best course of action might be using Crestle or AMI. If you need financial help please post - Request or share AWS credits here and someone might be able to setup a AWS box using their credits and provide you access for limited hours / week.
Thanks Rashna
Download the weights from http://files.fast.ai/models/weights.tgz and unzip it into that ‘fastai/courses/dl1/fastai/weights/’ folder.
1 Like
Note that Adam does not change learning rates.
Find the algorithm in page 2 (alpha is the learning rate).
3 Likes
Yes, by default the library does center crop.
crop_type=CropType.NO
Would avoid that.
4 Likes
Thank you for confirming! This was very important in my case because the dataset I was working with has crucial information that cannot be left out of the image so cropping hurts the accuracy in that situation.
UPDATE:
@yinterian I just tried using crop_type=CropType.NO and it works when precompute=True but when I set precompute=False and unfreeze() I get the following error message. Any ideas what could be causing it?
TypeError: Traceback (most recent call last):
File "/home/james/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 40, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/james/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 40, in <listcomp>
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/james/fastai/courses/dl1/fastai/dataset.py", line 99, in __getitem__
return self.get(self.transform, x, y)
File "/home/james/fastai/courses/dl1/fastai/dataset.py", line 104, in get
return (x,y) if tfm is None else tfm(x,y)
File "/home/james/fastai/courses/dl1/fastai/transforms.py", line 488, in __call__
def __call__(self, im, y): return compose(im, y, self.tfms)
File "/home/james/fastai/courses/dl1/fastai/transforms.py", line 469, in compose
im, y =fn(im, y)
File "/home/james/fastai/courses/dl1/fastai/transforms.py", line 246, in __call__
x,y = ((self.transform(x),y) if self.tfm_y==TfmType.NO
File "/home/james/fastai/courses/dl1/fastai/transforms.py", line 254, in transform
x = self.do_transform(x)
File "/home/james/fastai/courses/dl1/fastai/transforms.py", line 330, in do_transform
return no_crop(x, self.sz)
File "/home/james/fastai/courses/dl1/fastai/transforms.py", line 53, in no_crop
r,c = im.size
TypeError: 'int' object is not iterable@yinterian one more follow up question, sorry!
I noticed in the example you provided tfms = tfms_from_model(resnet34, sz, crop_type=CropType.NO) that there is no aug_tfms being specified. Does this mean that no data augmentation i.e. flips, rotations, etc. are being applied in this case?
Let me look into this. Jeremy just did a big push.
1 Like