FastAI learner changes the dataloader

borundev · January 28, 2021, 10:09pm

The code below shows that passing a dataloader to a learner changes it. This seems like a very odd behavior. Why is it done this way? I have figured out how it is being done. The learner code goes and changes the method of the dataloader class. This is a very odd thing to do. What if the same dataloader is to be used for two learners to compare the effect of normalisation?

In general the whole monkey patching thing with a lack of clear documentation is very frustrating and every small change requires one to dive into the code. Is there any plan to document all this and maybe to rethink the places where monkey patching is done?

dls = ImageDataLoaders.from_name_func(path, get_image_files(path), valid_pct=0.2,
label_func=is_tb,item_tfms=Resize(224))


x,y=next(iter(dls[0]))
print(x.min(),x.max())

This gives 0 and 1 respectively. However, initiating a learner


learn = cnn_learner(dls, resnet34, metrics=[accuracy],n_out=2,loss_func=CrossEntropyLossFlat())

x,y=next(iter(dls[0]))
print(x.min(),x.max())

I get -2.11 and 2.64 respectively.

Pomo · January 30, 2021, 2:32am

Hi Borun,

I think the issue is to expect that the two next(iter(dls[0]) should give the same values. If the generator yields batches of samples in a random order, each time it is started the batches will be different.

So you could first try shuffle_train=False when creating the DataLoader.

seed is another optional parameter to ImageDataLoaders.from_name_func. You could try setting it to a fixed value to get shuffled batches reproducibly. You may need to set numworkers=0 in addition.

I have not tested either of these ideas, so please let me know what you discover.

HTH, Malcolm

P.S. I agree that the fastai code is difficult to understand and modify because of the numerous hidden changes to standard behaviors combined with sparse internal documentation.

borundev · January 30, 2021, 8:29am

Getting same values was one of the issue and thanks for clarifying that but it was not the main issue.

The main issue is that the learner changes a bound method of the dataloaders. This method changes the normalization

github.com

fastai/fastai/blob/ab66a69fccc2f632ee04982aff6655eea911691a/fastai/vision/learner.py#L153


    body = create_body(arch, n_in, pretrained, ifnone(cut, meta['cut']))
    if custom_head is None:
        nf = num_features_model(nn.Sequential(*body.children()))
        head = create_head(nf, n_out, concat_pool=concat_pool, **kwargs)
    else: head = custom_head
    model = nn.Sequential(body, head)
    if init is not None: apply_init(model[1], init)
    return model
# Cell
def _add_norm(dls, meta, pretrained):
    if not pretrained: return
    after_batch = dls.after_batch
    if first(o for o in after_batch.fs if isinstance(o,Normalize)): return
    stats = meta.get('stats')
    if stats is None: return
    after_batch.add(Normalize.from_stats(*stats))
# Cell
@delegates(create_cnn_model)
def cnn_learner(dls, arch, normalize=True, n_out=None, pretrained=True, config=None,

I would have not changed the original dataloader but instead either made a copy or wrapped it in a structure so that the learner receives the normalized values but the original is not changed. Changing the original without warning is not good coding standard in my opinion.

Pomo · January 30, 2021, 8:39pm

I’ll admit that I had a very hard time making sense of the fastai code in this area. It looks to me that the normalization is added to the DataLoaders object as a list of transforms, not to the individual DataLoader’s. Are you sure that dls[0] itself is altered?

In any case, I concur that altering the dls DataLoaders just because it was once used in a pretrained Learner seems likely to lead to bugs if dls is ever used again.

Maybe someone who understands this code could clarify what actually happens and the intention behind it.