Unable to understand DataBunch code from Lesson 5

vishak · November 21, 2019, 8:37pm

Hey Guys

I was having a look at the create() method in the DataBunch Class

fastai/fastai/blob/4bc7e047cb3dcc45804d27d3c3659b4fffc7129b/fastai/basic_data.py#L61


@num_workers.setter
def num_workers(self,v): self.dl.num_workers = v


def add_tfm(self,tfm:Callable)->None:
    "Add `tfm` to `self.tfms`."
    self.tfms.append(tfm)
def remove_tfm(self,tfm:Callable)->None:
    "Remove `tfm` from `self.tfms`."
    if tfm in self.tfms: self.tfms.remove(tfm)


def new(self, **kwargs):
    "Create a new copy of `self` with `kwargs` replacing current values."
    new_kwargs = {**self.dl.init_kwargs, **kwargs}
    return DeviceDataLoader(self.dl.__class__(self.dl.dataset, **new_kwargs), self.device, self.tfms,
                            self.collate_fn)


def proc_batch(self,b:Tensor)->Tensor:
    "Process batch `b` of `TensorImage`."
    b = to_device(b, self.device)
    for f in listify(self.tfms): b = f(b)
    return b

A lot of things didn’t make sense to me though.

The first line is

datasets = cls._init_ds(train_ds, valid_ds, test_ds)

The definition of _Init_ds is right above the codeblock for DataBunch but I have no idea what it is doing at the level of code

The second line uses ifnone and I can’t find what that is

and the last two lines, well…
dls = [DataLoader(d, b, shuffle=s, drop_last=s, num_workers=num_workers, **dl_kwargs) for d,b,s in zip(datasets, (bs,val_bs,val_bs,val_bs), (True,False,False,False)) if d is not None]
return cls(*dls, path=path, device=device, dl_tfms=dl_tfms, collate_fn=collate_fn, no_check=no_check)

Any help would be highly appreciated

Honigtoast · November 22, 2019, 2:39pm

Hey, just as a quick hint: maybe the docs can help you answer some of your questions! I was a bit lost with all that code in the beginning as well but by navigating through the code and the docs AND testing/programming several things myself I learned a lot of how things are done in fastai

vishak · November 26, 2019, 4:28am

Hey, sorry for the late reply.

I’m going through the docs, but sometimes they’ve done things with the code that don’t really make it to the docs, so stuff like this is a little confusing