Part 1 (2020) - Weekly Beginner Only Review and Q&A

[Albertotono]

Thanks you so much

[harish3110 (Harish Vadlamani)]

I had a question regarding the max_card value set in the cont_cat_split method in the tabular lesson

Lesson 7 - Official topic

In the tabular chapter the cont_cat_split method is called with different max_cardinality parameters for the DT/RF model and the NN model as follows: #DT/RF cont,cat = cont_cat_split(df, 1, dep_var=dep_var) #NN cont_nn,cat_nn = cont_cat_split(df_nn_final, max_card=9000, dep_var=dep_var) The chapter does say that categorical columns are treated differently as it needs to create embeddings and indicates that embedding of size greater than 10k should not used and hence the 9k is used as max card…

Any thoughts?

[DrC]

What is it that is missing to make a dataloader?

data.train_dl isn’t there!

[marii (Molly Beavers)]

Are you on an old version of fastai2? I think databunch has been renamed to dataloaders.

Anyway, to me it looks like data.databunch is returning a ImageDataBunch object. So you would need to do something like:

db = data.databunch(bs=64)
db.train

Though again I am running on a different version of fastai, so I can’t test to make sure.

[wgpubs (WG)]

Just an FYI … we’ll be starting in < 15 mins. Zoom here: https://ucsd.zoom.us/j/93212231757

[pinaki]

Quick question about Object Detection. I was trying to run the coco dataset for object detection. I followed the instructions here to create the Dataloaders https://dev.fast.ai/tutorial.datablock#Bounding-boxes

coco = DataBlock(blocks=(ImageBlock, BBoxBlock, BBoxLblBlock),
                 get_items=get_image_files,
                 splitter=RandomSplitter(),
                 get_y=[lambda o: img2bbox[o.name][0], lambda o: img2bbox[o.name][1]], 
                 item_tfms=Resize(128),
                 batch_tfms=aug_transforms(),
                 n_inp=1

I am trying to define a leaner using the unet architecture

coco_learn = unet_learner(coco_dls, resnet18)

However, when I try to fit it coco_learn.fit_one_cycle(2) I am getting the following error.
Note that the get_y method is unique here. Has both bounding box and the label: get_y=[lambda o: img2bbox[o.name][0], lambda o: img2bbox[o.name][1]],
What am I getting wrong here ? Is unet_learner not suitable for object detection ? I have seen it being used for segmentation. Or am I missing something else
Here is the error

TypeError                                 Traceback (most recent call last)
<ipython-input-30-743c03b6f6bf> in <module>()
----> 1 coco_learn.fit_one_cycle(2)

6 frames
/usr/local/lib/python3.6/dist-packages/fastcore/utils.py in _f(*args, **kwargs)
    429         init_args.update(log)
    430         setattr(inst, 'init_args', init_args)
--> 431         return inst if to_return else f(*args, **kwargs)
    432     return _f
    433 

/usr/local/lib/python3.6/dist-packages/fastai2/callback/schedule.py in fit_one_cycle(self, n_epoch, lr_max, div, div_final, pct_start, wd, moms, cbs, reset_opt)
    111     scheds = {'lr': combined_cos(pct_start, lr_max/div, lr_max, lr_max/div_final),
    112               'mom': combined_cos(pct_start, *(self.moms if moms is None else moms))}
--> 113     self.fit(n_epoch, cbs=ParamScheduler(scheds)+L(cbs), reset_opt=reset_opt, wd=wd)
    114 
    115 # Cell

/usr/local/lib/python3.6/dist-packages/fastcore/utils.py in _f(*args, **kwargs)
    429         init_args.update(log)
    430         setattr(inst, 'init_args', init_args)
--> 431         return inst if to_return else f(*args, **kwargs)
    432     return _f
    433 

/usr/local/lib/python3.6/dist-packages/fastai2/learner.py in fit(self, n_epoch, lr, wd, cbs, reset_opt)
    198                     try:
    199                         self.epoch=epoch;          self('begin_epoch')
--> 200                         self._do_epoch_train()
    201                         self._do_epoch_validate()
    202                     except CancelEpochException:   self('after_cancel_epoch')

/usr/local/lib/python3.6/dist-packages/fastai2/learner.py in _do_epoch_train(self)
    173         try:
    174             self.dl = self.dls.train;                        self('begin_train')
--> 175             self.all_batches()
    176         except CancelTrainException:                         self('after_cancel_train')
    177         finally:                                             self('after_train')

/usr/local/lib/python3.6/dist-packages/fastai2/learner.py in all_batches(self)
    151     def all_batches(self):
    152         self.n_iter = len(self.dl)
--> 153         for o in enumerate(self.dl): self.one_batch(*o)
    154 
    155     def one_batch(self, i, b):

/usr/local/lib/python3.6/dist-packages/fastai2/learner.py in one_batch(self, i, b)
    159             self.pred = self.model(*self.xb);                self('after_pred')
    160             if len(self.yb) == 0: return
--> 161             self.loss = self.loss_func(self.pred, *self.yb); self('after_loss')
    162             if not self.training: return
    163             self.loss.backward();                            self('after_backward')

TypeError: __call__() takes 3 positional arguments but 4 were given

[butchland (Butch Landingin)]

It’s been great having you host the sessions, @wgpubs.

I really appreciate all the work you’ve done
to help us beginners understand the lessons!

Best regards,
Butch

[wgpubs (WG)]

My pleasure. It was a lot of fun.

Stay safe and sane for however your long your lock down continues. see ya in part 2.

[pierreguillou]

@wgpubs

Remember than you can define how the layers in a model are split up. In fastai, the body of a pret-trained model is usually divided into two layer/param groups while a 3rd is created for your custom head.

Hello @wgpubs. Do you know how to do this? (“you can define how the layers in a model are split up”)

For example, how can define 1 group = 1 layer and pass to lr_max an array of learning rates (one for each group, which means one for each layer in this case)?

[wgpubs (WG)]

Sure.

Take a look at how I do it with hugginface here: https://ohmeow.github.io/blurr/modeling-core/#hf_splitter

Lmk if you have any questions on it.

-wg

[florianl (Florian)]

Splitting the model is kind of easy if you build it with nn.Sequential and you are able to index into the model (model[0][5] for example).

You can define a splitter function

Group1: Layers 0…5 of the body (model[0][:6])
Group2: Layers 6…max of the body (model[0][6:])
Group3: Head of the model (model[1])

def  splitter(m):
    return L(m[0][:6], m[0][6:], m[1]).map(params)

And pass it to the learner:

learn = Learner(dls, model, metrics=accuracy, splitter=splitter, cbs=ShowGraphCallback())

See this notebook for the complete code:

floleuerer/fastai2_resnet_audio/blob/master/tutorial/tutorial.ipynb

[pierreguillou]

@wgpubs

How many “layer groups” are there?
Run len(learn.opt.param_groups) and it will tell you.

hi @wgpubs.

Do you know the differences between learn.opt.param_lists and learn.opt.param_groups?

len(learn.opt.param_groups) and len(learn.opt.param_lists) give the same results.

[pierreguillou]

@barnacl

if lr = 1e-3 -> [0.001,0.001,0.001]
if lr = slice(1e-3) -> [0.0001,0.0001,0.001]
if lr = slice(1e-5,1e-3) -> array([1.e-05, 1.e-04, 1.e-03]

Hello @wgpubs and @barnacl.

Do you know how to verify the Learning Rate value(s) by a Pytorch and/or fastai v2 function?

For example, learn.lr gives only the Learning Rate of the last layer (head of the model).

[wgpubs (WG)]

@pierreguillou

Do you know how to verify the Learning Rate value(s) by a Pytorch and/or fastai v2 function?

Not really. You can print out the param groups but its not intuitive as to what layers they correspond too (can’t remember the syntax exactly but I think its something like learn.opt.param_groups).

Here’s the documentation for how discriminative LRs are defined and applied in fastai.

[pierreguillou]

[ EDIT 05/25/2020 ] I found the function fastai2 that allows to get the Learning Rate values used by the optimizer. See my post about that.

---

Thank you but this does not answer my question (and you gave a link to the fastai v1 doc, not v2).

For example, if you use lr_max=slice(10e-6, 10e-4), how to get the Learning Rate values used by the optimizer? (I’m not talking about getting the parameters groups).

[pierreguillou]

@florianl

Splitting the model is kind of easy if you build it with nn.Sequential

Hi @florianl. You’re right about your affirmation. Then, I tested in this notebook (nbviewer version) how to create parameters groups for a model more complicated like resnet18.

As well, I tested all the methods to pass different Learning Rates (one by parameters group).
In fact, they are 4 possibilities, not 3.

For example with 3 parameters groups, you can do the following (for a Learner unfrozen and the use of learn.fit_one_cycle()):

1. if lr_max = 1e-3 -> [0.001,0.001,0.001]
2. if lr_max = slice(1e-3) -> [0.0001,0.0001,0.001]
3. if lr_max = slice(1e-5,1e-3) -> array([1.e-05, 1.e-04, 1.e-03]) #LRs evenly geometrically spaced
4. if lr_max = [1e-5, 1e-4, 1e-3] -> array([1.e-05, 1.e-04, 1.e-03]) #LRs evenly linearly spaced or not

Explications

1. All parameters groups will use the same Learning Rate (and the same Optimizer method like Adam + 1cycle policy for all).
2. The last layer group’s Learning Rate (max) value is setup to lr, and all previous parameters groups ones to lr/10.
3. Train the very first layers at a Learning Rate of 1e-5, the very last at 1e-3, and the Learning Rates of other parameters groups are evenly geometrically spaced between theses two values.
4. Train the very first layers at a Learning Rate of 1e-5, the very last at 1e-3, and the Learning Rates of other parameters groups are evenly linearly spaced between theses two values or you can pass as a list any Learning Rate values.

WARNING

Points 3 and 4 are not equivalent for a number of parameters groups greater than 3!!!

• point 3: Learning Rates are calculated geometrically.
• point 4: you can pass an array with the Learning Rate values you want.

List of Learning Rates: last values of cosine annealing for the 1cycle policy

In order to get the list of Learning Rates by parameters group that was passed to the Learner by the Optimizer during the training, you can display the hyperparameters by using learn.opt.hypers as following:

for i,h in enumerate(learn.opt.hypers):
    print(i,h)

[DanielLam]

Thanks, I’ve been wondering about all these details for awhile. But didn’t get the chance to look through. You saved me quite a bit of time.

[pierreguillou]

@DanielLam

You saved me quite a bit of time

My pleasure Daniel Thanks for your message.

[DrC]

Looking to assign different weights to different image datasets, similar to what we do for class imbalance but for datasets instead. Any example would be much appreciated! @sgugger @wgpubs

[golz]

I have a question on splitting tabular data. The example Jeremy used in fastbook ch 9 looks at time-series data (where he introduces np.where function). My question is how would you split this data if not using time series. In other words, my valid data should be in same pool as training data. This is from Ch 9…

In [ ]:
cond = (df.saleYear<2011) | (df.saleMonth<10)
train_idx = np.where( cond)[0]
valid_idx = np.where(~cond)[0]

splits = (list(train_idx),list(valid_idx))
TabularPandas needs to be told which columns are continuous and which are categorical. We can handle that automatically using the helper function cont_cat_split:

In [ ]:
cont,cat = cont_cat_split(df, 1, dep_var=dep_var)
In [ ]:
to = TabularPandas(df, procs, cat, cont, y_names=dep_var, splits=splits)