Unable to understand DataBunch code from Lesson 5

vishak · November 21, 2019, 9:41am

I was having a look, on Jeremy’s direction, of the create() method in the DataBunch Class

fastai/fastai/blob/4bc7e047cb3dcc45804d27d3c3659b4fffc7129b/fastai/basic_data.py#L61


@num_workers.setter
def num_workers(self,v): self.dl.num_workers = v


def add_tfm(self,tfm:Callable)->None:
    "Add `tfm` to `self.tfms`."
    self.tfms.append(tfm)
def remove_tfm(self,tfm:Callable)->None:
    "Remove `tfm` from `self.tfms`."
    if tfm in self.tfms: self.tfms.remove(tfm)


def new(self, **kwargs):
    "Create a new copy of `self` with `kwargs` replacing current values."
    new_kwargs = {**self.dl.init_kwargs, **kwargs}
    return DeviceDataLoader(self.dl.__class__(self.dl.dataset, **new_kwargs), self.device, self.tfms,
                            self.collate_fn)


def proc_batch(self,b:Tensor)->Tensor:
    "Process batch `b` of `TensorImage`."
    b = to_device(b, self.device)
    for f in listify(self.tfms): b = f(b)
    return b

A lot of things didn’t make sense to me though.

The first line is

datasets = cls._init_ds(train_ds, valid_ds, test_ds)

The definition of _Init_ds is right above the codeblock for DataBunch but I have no idea what it is doing at the level of code

The second line uses ifnone and I can’t find what that is

and the last two lines, well…
dls = [DataLoader(d, b, shuffle=s, drop_last=s, num_workers=num_workers, **dl_kwargs) for d,b,s in zip(datasets, (bs,val_bs,val_bs,val_bs), (True,False,False,False)) if d is not None]
return cls(*dls, path=path, device=device, dl_tfms=dl_tfms, collate_fn=collate_fn, no_check=no_check)

Any help would be highly appreciated