Fastai v2 Recipes (Tips and Tricks) - Wiki

Another trick I often use:
This is useful if you have a long running cell in Jupiter/Colab you would like to get notified when the cell finishes running. I figured out you can autoplay an audio file using IPython library.
For Colab just copy paste the below code snippet once, and simply put notify in the last line of the all the cells to play the notification audio.

import IPython
!wget https://notificationsounds.com/notification-sounds/eventually-590/download/mp3 -O notify.mp3
notify=IPython.display.Audio("./notify.mp3",autoplay=True)
notify

You can use any audio file you wish. :slight_smile:

10 Likes

Code to seed everything (taken from kaggle)

def seed_everything(seed):
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True

14 Likes

This is a really good thread! I am looking for a way to sample transformed data item from an instance of DataLoaders. Here is an example:

pets = DataBlock(blocks = (ImageBlock, CategoryBlock),
                 get_items=get_image_files, 
                 splitter=RandomSplitter(seed=42),
                 get_y=using_attr(RegexLabeller(r'(.+)_\d+.jpg$'), 'name'),
                 item_tfms=Resize(460),
                 batch_tfms=aug_transforms(size=224, min_scale=0.75))
dls = pets.dataloaders(path/"images")

instead of getting a transformed(item transformed + batch transformed) batch from dls, I want to get a transformed (item transformed only) data item from dls. Do any fellows know how to do that? (I tried dls.train_ds[0] but it seems it’s what I want because the output hasn’t gone through item transform)

Also, is there any way I can check from an attribute of dls what kind of transformations have been done on item / batch / training set / valid set?

The following answer is extracted from one of the tips listed above :wink:

As stated above, the summary() method will output the all transforms applied to both your input and label data.

I friendly suggest that we keep this thread dedicated to fastai2 Tips & Tricks that can be shared with the whole community, and post questions in the fastai2 chat thread. The objective is to keep a clean list of Tips & Tricks that will be easy to explore and discover :slightly_smiling_face:

thanks! @farid
and u a right, I should have posted the question on a separate post!

1 Like

No problem at all @riven314!

split_idx, or how to selectively apply a transform to train and valid (test) datasets?

First of all, I would like to credit @jeremy for sharing the 10 episodes of his fastai2 daily code walk-thrus, and @sgugger for answering questions on the forum as wall as @arora_aman, @akashpalrecha, and @init_27 for posting their code walk-thrus.

split_idx allows a given Transform to be applied to a specific data subset: train, valid and test datasets. For example, it enables a specific image augmentation to be applied to only the train dataset and not to the valid dataset.

By setting your transform split_idx flag, you can make your transform to be applied :

  • to both train and valid (test) datasets if you set (leave) your transform split_idx to None
  • or only to the train dataset (and not to the valid (test) datatset) if you set your transform split_idx to 0
  • or only to the valid (test) dataset (and not to the train datatset) if you set your transform split_idx to 1

In which files split_idx can be found?
split_idx can be found:
1- in fastai2.data.core.py: split_idx is used by both TfmLists, and Datasets classes (Datasets uses TfmLists objects) where:
○ the train dataset has a split_idx=0,
○ the valid dataset has a split_idx=1,
○ the test dataset also has a split_idx=1,
○ There is also the set_split_idx() method that sets a split_idx to a given dataset. That method is used in the “test time augmentattion” tta() method found in fast2.learner.py

2- in TfmDL: split_idx is used by the before_iter() method in order to set split_idx of each batch_tfms Pipeline objects to the same split_idx as the corresponding dataset (train and valid datasets)

3- in fastai2.vision.augment.py: split_idx is used by several Transform classes as shown further below (e.g. RandTransform, and Resize Transforms)

4- also in fast2.learner.py: it is used by the “test time augmentattion” tta() method

How does it work?
A Transform has a split_idx attribute and defines the following _call () method:

def _call(self, fn, x, split_idx=None, **kwargs):
        if split_idx!=self.split_idx and self.split_idx is not None: return x
        return self._do_call(getattr(self, fn), x, **kwargs)

As you might notice, we pass a split_idx argument to the _call() method. That split_idx argument is checked against the Transform self.split_idx in the if statement. The latter sets the behavior of the Transform as summarized here above.

We generally don’t explicitly call a Transform. A Pipeline which is a class that store a list of Transform objects is responsible of calling each one of its Transform objects _call() method.

Pipeline are used in the TfmLists class (and Datasets class because the latter uses TfmLists objects). Pipeline also store a split_idx as an attribute. Both Datasets and TfmLists generally have a train dataset (with a split_idx=0), a valid dataset (with a split_idx=1), and sometimes a test dataset (also with a split_idx=1). When a Pipeline of Transform is applied to one of the 3 datasets, the Pipeline call each of its Transform objects by passing the split_idx of the corresponding dataset that we are about to transform.

Therefore, if we are transforming a train dataset, the Pipeline passes split_idx=0 to each of its Transform objects _call() method. Similarly, for both valid dataset and test dataset, the Pipeline passes split_idx=1 to each of its Transform objects _call() method.

Now, back to our Transform _call() method. The latter will compare the passed argument (from the Pipeline being the dataset split_idx) to its self.split_idx (self is the Transform object), and decides either to ignore the call by returning the input x without any change, or apply the transform through the return self._do_call(getattr(self, fn), x, **kwargs) by following the rules mentioned here above.

Let’s check some Transform examples:

Transform examples in augment.py:

Resize Transform

class Resize(RandTransform):
    split_idx = None
    mode,mode_mask,order,final_size = Image.BILINEAR,Image.NEAREST,1,None
    "Resize image to `size` using `method`"

Resize has a split_idx=None meaning that it will be applied to both the train and valid (test) datasets.

RandTransform

class RandTransform(Transform):
    "A transform that before_call its state at each `__call__`"
    do,nm,supports,split_idx = True,None,[],0

RandTransform has a split_idx=0 meaning that it will only be applied to the train dataset. Be aware, if the transform is not applied to a given item it is because the transform is applied with a given probability p (meaning not all the train dataset items are transformed).

Test Time Augmentation tta() method
split_idx is also used in learner.py tta() method. Test time augmentation can significantly improves accuracy.

tta() combines predictions of several augmented images. To calculate the prediction of a given image, we follow the steps shown here below (assuming we are using the default n=4 value for the number of the augmented images):

1- First, we create 4 augmented images using the train dataset Pipeline Transforms. This is why the dl.dataset.set_split_idx(0) is called in order to make sure the Pipeline objects passes split_idx=0 to each Transform call. Each augmented image gets its prediction. The aug_preds, representing predictions of the 4 images, is then reduced using either the max or the mean value,

2- Then, we calculate only one prediction (preds) using the valid (test) dataset Pipeline of Transform. The use of dl.dataset.set_split_idx(1) ensures to apply only the Pipeline Transforms that is set for the valid (test) dataset: split_idx=1,

3- Finally, we combine aug_preds and preds using either the max or a linear interpolation function.

7 Likes

Thanks @farid
I’ve written a blog about this too: https://akashpalrecha.me/tutorials/blog/2020/03/27/split-transform.html

Maybe you could add to your post anything extra you find there!

1 Like

Hi @farid, thanks for putting together this page. One quick update:

Since PR#294, summary() now has a show_batch flag to display the batches at the end of the regular summary() output. It also passes whatever subsequent kwargs parameters to dls.show_batch().

Perhaps consider including it in your above summary of summary() ? :wink:

Cheers.

2 Likes

Thank you @philchu for your suggestion. I updated the summary() section using the information you provided :slightly_smiling_face:.

1 Like

In Inference prediction I get an error on categorize:

AttributeError: categorize

that is a really nice idea! :slight_smile:
Do you have tip how I can do that in a Jupyter Notebook on a Windows 10 machine?

Thanks!

This works on Jupyter notebook too, just download that audio file, or use your own audio file.

1 Like

all right, thank you!

Thank you very much for the post, it is really helpful. The link to the 50 Datablock examples is no longer valid. I think that the most relevant alternative is this: https://github.com/fastai/fastai/blob/master/nbs/50_tutorial.datablock.ipynb

1 Like

Thanks for nice examples! It tripped me up a bit because it is supposed to be with_labels - not with_label. The error message you will get for these missing s is the following for people who are searching:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/tmp/ipykernel_7970/1878872865.py in <module>
      3 learn.dls.loaders.append(test_dl)
      4 
----> 5 interp = ClassificationInterpretation.from_learner(learn, ds_idx=2)
      6 interp.plot_confusion_matrix()

~/.local/lib/python3.7/site-packages/fastai/interpret.py in from_learner(cls, learn, ds_idx, dl, act)
     39         if dl is None: dl = learn.dls[ds_idx].new(shuffle=False, drop_last=False)
     40         _,_,losses = learn.get_preds(dl=dl, with_input=False, with_loss=True, with_decoded=False,
---> 41                                      with_preds=False, with_targs=False, act=act)
     42         return cls(learn, dl, losses, act)
     43 

~/.local/lib/python3.7/site-packages/fastai/learner.py in get_preds(self, ds_idx, dl, with_input, with_decoded, with_loss, act, inner, reorder, cbs, **kwargs)
    253         if with_loss: ctx_mgrs.append(self.loss_not_reduced())
    254         with ContextManagers(ctx_mgrs):
--> 255             self._do_epoch_validate(dl=dl)
    256             if act is None: act = getattr(self.loss_func, 'activation', noop)
    257             res = cb.all_tensors()

~/.local/lib/python3.7/site-packages/fastai/learner.py in _do_epoch_validate(self, ds_idx, dl)
    201         if dl is None: dl = self.dls[ds_idx]
    202         self.dl = dl
--> 203         with torch.no_grad(): self._with_events(self.all_batches, 'validate', CancelValidException)
    204 
    205     def _do_epoch(self):

~/.local/lib/python3.7/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    161 
    162     def _with_events(self, f, event_type, ex, final=noop):
--> 163         try: self(f'before_{event_type}');  f()
    164         except ex: self(f'after_cancel_{event_type}')
    165         self(f'after_{event_type}');  final()

~/.local/lib/python3.7/site-packages/fastai/learner.py in all_batches(self)
    167     def all_batches(self):
    168         self.n_iter = len(self.dl)
--> 169         for o in enumerate(self.dl): self.one_batch(*o)
    170 
    171     def _do_one_batch(self):

~/.local/lib/python3.7/site-packages/fastai/learner.py in one_batch(self, i, b)
    192         b = self._set_device(b)
    193         self._split(b)
--> 194         self._with_events(self._do_one_batch, 'batch', CancelBatchException)
    195 
    196     def _do_epoch_train(self):

~/.local/lib/python3.7/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    163         try: self(f'before_{event_type}');  f()
    164         except ex: self(f'after_cancel_{event_type}')
--> 165         self(f'after_{event_type}');  final()
    166 
    167     def all_batches(self):

~/.local/lib/python3.7/site-packages/fastai/learner.py in __call__(self, event_name)
    139 
    140     def ordered_cbs(self, event): return [cb for cb in self.cbs.sorted('order') if hasattr(cb, event)]
--> 141     def __call__(self, event_name): L(event_name).map(self._call_one)
    142 
    143     def _call_one(self, event_name):

~/.local/lib/python3.7/site-packages/fastcore/foundation.py in map(self, f, gen, *args, **kwargs)
    153     def range(cls, a, b=None, step=None): return cls(range_of(a, b=b, step=step))
    154 
--> 155     def map(self, f, *args, gen=False, **kwargs): return self._new(map_ex(self, f, *args, gen=gen, **kwargs))
    156     def argwhere(self, f, negate=False, **kwargs): return self._new(argwhere(self, f, negate, **kwargs))
    157     def argfirst(self, f, negate=False): return first(i for i,o in self.enumerate() if f(o))

~/.local/lib/python3.7/site-packages/fastcore/basics.py in map_ex(iterable, f, gen, *args, **kwargs)
    777     res = map(g, iterable)
    778     if gen: return res
--> 779     return list(res)
    780 
    781 # Cell

~/.local/lib/python3.7/site-packages/fastcore/basics.py in __call__(self, *args, **kwargs)
    762             if isinstance(v,_Arg): kwargs[k] = args.pop(v.i)
    763         fargs = [args[x.i] if isinstance(x, _Arg) else x for x in self.pargs] + args[self.maxi+1:]
--> 764         return self.func(*fargs, **kwargs)
    765 
    766 # Cell

~/.local/lib/python3.7/site-packages/fastai/learner.py in _call_one(self, event_name)
    143     def _call_one(self, event_name):
    144         if not hasattr(event, event_name): raise Exception(f'missing {event_name}')
--> 145         for cb in self.cbs.sorted('order'): cb(event_name)
    146 
    147     def _bn_bias_state(self, with_bias): return norm_bias_params(self.model, with_bias).map(self.opt.state)

~/.local/lib/python3.7/site-packages/fastai/callback/core.py in __call__(self, event_name)
     55         res = None
     56         if self.run and _run:
---> 57             try: res = getattr(self, event_name, noop)()
     58             except (CancelBatchException, CancelEpochException, CancelFitException, CancelStepException, CancelTrainException, CancelValidException): raise
     59             except Exception as e:

~/.local/lib/python3.7/site-packages/fastai/callback/core.py in after_batch(self)
    135             torch.save(targs[0], self.save_targs/str(self.iter), pickle_protocol=self.pickle_protocol)
    136         if self.with_loss:
--> 137             bs = find_bs(self.yb)
    138             loss = self.loss if self.loss.numel() == bs else self.loss.view(bs,-1).mean(1)
    139             self.losses.append(self.learn.to_detach(loss))

~/.local/lib/python3.7/site-packages/fastai/torch_core.py in find_bs(b)
    568 def find_bs(b):
    569     "Recursively search the batch size of `b`."
--> 570     return item_find(b).shape[0]
    571 
    572 # Cell

~/.local/lib/python3.7/site-packages/fastai/torch_core.py in item_find(x, idx)
    554 def item_find(x, idx=0):
    555     "Recursively takes the `idx`-th element of `x`"
--> 556     if is_listy(x): return item_find(x[idx])
    557     if isinstance(x,dict):
    558         key = list(x.keys())[idx] if isinstance(idx, int) else idx

IndexError: Exception occured in `GatherPredsCallback` when calling event `after_batch`:
	tuple index out of range
1 Like

Do we call seed_everything() before creating the Learner or before calling fine_tune()?

Short answer:
Call seed_everything() in the very beginning of your code before you do anything else, so everything will be deterministic from that point.
You can check if everything works as intended, if you get the same loss for each run, then everything seems fine.

Longer answer:
The purpose of the seed_everything() method is to set the starting number (seed) for all random number generators (random, numpy, torch & underlying cuda too) + set cuDNN to deterministic mode (cuDNN is also used by Pytorch under the hood).

You want determinism from the beginning for repeatable experiments and because neural networks can have randomized parts (eg.: dropout), data sets & loaders usually also have randomized parts for every epoch, and so on, better to call seed_everything() before all of them.
(But you can call seed_everything() multiple times in your code if you want/need, it just sets the seed again and nothing wrong with that)

2 Likes

I tried out your suggestion, and it worked. (I think the trick is to call seed_everything() before creating the DataLoaders.)

Thanks for the answer.

1 Like

Wow! This saved me! I also got this error and have searched for the solution for quite some time…

1 Like