How to use DataLoader?


(NGUYEN CONG MINH) #1

Hi everyone,

I want to use fastai’s DataLoader to load data from object.
I run it successfully when use torch.utils.data.DataLoader, but fail when use fastai’s DataLoader.
The code i run is: https://github.com/fxia22/pointnet.pytorch/blob/master/train_segmentation.py

Could anyone tell me how to use fastai’s DataLoader to load data? Then load some batch sample to show it out?

Thank you very much.


(Martin) #2

For the FastAI DataLoader you should be able to call:

next(iter(fastai_dataloader))

That should give you a batch. Does that work?

It is meant to be used as:

for batch_index, batch_data in enumerate(fastai_dataloader):

In the GitHub code, there is an extra argument for enumerate that is set to 0 but that is redundant because that is the default value.


(NGUYEN CONG MINH) #3

Thank @Hadus
I already tried it.
with torch.utils.data.DataLoader, both next(iter(…) and for batch in enumerate(…) are able to run correctly.
but with fastai’s DataLoader, both ways return error:

Traceback (most recent call last):
  File "/home/minhnc-lab/PROGRAMS/anaconda3/envs/fastai/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2910, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-6-5e4e60e93023>", line 1, in <module>
    batch = next(iter(fastai_dataloader))
  File "/home/minhnc-lab/WORKSPACES/Python/GRASP_DETECTION/PointNet/pointnet1_pytorch/fastai/dataloader.py", line 82, in __iter__
    for batch in map(self.get_batch, iter(self.batch_sampler)):
  File "/home/minhnc-lab/WORKSPACES/Python/GRASP_DETECTION/PointNet/pointnet1_pytorch/fastai/dataloader.py", line 75, in get_batch
    res = self.np_collate([self.dataset[i] for i in indices])
  File "/home/minhnc-lab/WORKSPACES/Python/GRASP_DETECTION/PointNet/pointnet1_pytorch/fastai/dataloader.py", line 71, in np_collate
    return [self.np_collate(samples) for samples in zip(*batch)]
  File "/home/minhnc-lab/WORKSPACES/Python/GRASP_DETECTION/PointNet/pointnet1_pytorch/fastai/dataloader.py", line 71, in <listcomp>
    return [self.np_collate(samples) for samples in zip(*batch)]
  File "/home/minhnc-lab/WORKSPACES/Python/GRASP_DETECTION/PointNet/pointnet1_pytorch/fastai/dataloader.py", line 72, in np_collate
    raise TypeError(("batch must contain numbers, dicts or lists; found {}".format(type(b))))
TypeError: batch must contain numbers, dicts or lists; found <class 'torch.Tensor'>

(Kevin L.) #4

I think that the fastai Dataloader automatically converts the contents of the batch into a tensor. Try modifying your Dataset class to return a numpy array as opposed to tensor.


(NGUYEN CONG MINH) #5

Thank @Hadus and @loftiskg,
I modified the dataset and it’s able to load data now.
In my own dataset loader, it return tensor. pytorch DataLoader accept all kind of fed data, but fastai not. When I change returned data to numpy array, fastai DataLoader is able to run.

Now I got problem that: HOW TO TRAIN MODEL?
I have my own model definition.
I can use DataLoader to get sample data.
But i don’t know how to:

  • Create learner from defined model
  • feed dataloader to learner

Does anyone have sample code of doing it?

Thank you very much.


(Martin) #6

Did you not watch the whole of course 1?? You should! :slight_smile:

One of the Jupyter notebooks from course 1 definitely has some code that you are looking for.


(NGUYEN CONG MINH) #7

Hi @Hadus
I checked all the samples of deep learning 1 lessons, but I couldn’t find out the sample that uses DataLoader.
Most of them use ImageClassifierData.from…
Could you tell me which lesson it is?


(Martin) #8

I was talking about not using the learner class at all. It looks like I was wrong and there isn’t a simple pytorch train function implementation there. Here is a basic example (UNTESTED CODE):

names = ["error"]
layout = "{!s:10} " * len(names)

epochs = 50
criterion = F.binary_cross_entropy
dataloader = 
net = 
optimizer = optim.Adam(net.parameters(), lr=0.001, betas=(0.9, 0.999)


def print_stats(epoch, values, decimals=6):
    layout = "{!s:^10}" + " {!s:10}" * len(values)
    values = [epoch] + list(np.round(values, decimals))
    print(layout.format(*values))

for epoch in tnrange(epochs, desc="Epoch"):
    t = tqdm(iter(dataloader), leave=False, total=len(dataloader))

    for i, batch in enumerate(t):

        xs = Variable(batch[0]).cuda()
        ys = Variable(batch[1]).cuda()

        optimizer.zero_grad()

        y_hats = net(xs)

        err = criterion(ys, y_hats)

        err.backward() 
        optimizer.step()

        t.set_postfix(err=to_np(err.mean()))

    if epoch == 0:
        print(f"\n{layout.format(*names)}")

    print_stats(epoch, [to_np(err.mean())])

I hope you can tailor it to be useful. Good luck :slight_smile:


(NGUYEN CONG MINH) #9

Hi @Hadus, Thank you for your kindness.
I did the similar way to you and I was able to train.
Now, I want to use learner to train. But I’m not sure what is the proper way.
Could you tell me how to feed model and dataloader to the learner?


(Martin) #10

The code from

uses the model PointNetDenseCls that returns a two values in its forward function. This could cause problems as the learner expects one value to be returned. The easy fix would be to edit the PointNetDenseCls so forward only returns x.

Try something like this: (ALSO NOT TESTED CODE)

# model data
path = "..."
trn_dl = DataLoader(...)
val_dl = DataLoader(...)
test_dl = None

model_data = ModelData(path, trn_dl, val_dl, test_dl=test_dl)

# model
model = PointNetDenseCls(...)

# optimizer
optimizer = optim.Adam(...)

# criterion
criterion = F.nll_loss

# learner
learner = Learner(model_data, BasicModel(to_gpu(model)), opt_fn=optimizer, crit=criterion)

where there is “…” just fill in with what you want.

Try this with and without editing the forward in the PointNetDenseCls class.

Also if this doesn’t throw an error, the learner.fit might.