Thank @Hadus
I already tried it.
with torch.utils.data.DataLoader, both next(iter(…) and for batch in enumerate(…) are able to run correctly.
but with fastai’s DataLoader, both ways return error:
Traceback (most recent call last):
File "/home/minhnc-lab/PROGRAMS/anaconda3/envs/fastai/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2910, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-6-5e4e60e93023>", line 1, in <module>
batch = next(iter(fastai_dataloader))
File "/home/minhnc-lab/WORKSPACES/Python/GRASP_DETECTION/PointNet/pointnet1_pytorch/fastai/dataloader.py", line 82, in __iter__
for batch in map(self.get_batch, iter(self.batch_sampler)):
File "/home/minhnc-lab/WORKSPACES/Python/GRASP_DETECTION/PointNet/pointnet1_pytorch/fastai/dataloader.py", line 75, in get_batch
res = self.np_collate([self.dataset[i] for i in indices])
File "/home/minhnc-lab/WORKSPACES/Python/GRASP_DETECTION/PointNet/pointnet1_pytorch/fastai/dataloader.py", line 71, in np_collate
return [self.np_collate(samples) for samples in zip(*batch)]
File "/home/minhnc-lab/WORKSPACES/Python/GRASP_DETECTION/PointNet/pointnet1_pytorch/fastai/dataloader.py", line 71, in <listcomp>
return [self.np_collate(samples) for samples in zip(*batch)]
File "/home/minhnc-lab/WORKSPACES/Python/GRASP_DETECTION/PointNet/pointnet1_pytorch/fastai/dataloader.py", line 72, in np_collate
raise TypeError(("batch must contain numbers, dicts or lists; found {}".format(type(b))))
TypeError: batch must contain numbers, dicts or lists; found <class 'torch.Tensor'>
I think that the fastai Dataloader automatically converts the contents of the batch into a tensor. Try modifying your Dataset class to return a numpy array as opposed to tensor.
Thank @Hadus and @loftiskg,
I modified the dataset and it’s able to load data now.
In my own dataset loader, it return tensor. pytorch DataLoader accept all kind of fed data, but fastai not. When I change returned data to numpy array, fastai DataLoader is able to run.
Now I got problem that: HOW TO TRAIN MODEL?
I have my own model definition.
I can use DataLoader to get sample data.
But i don’t know how to:
Hi @Hadus
I checked all the samples of deep learning 1 lessons, but I couldn’t find out the sample that uses DataLoader.
Most of them use ImageClassifierData.from…
Could you tell me which lesson it is?
I was talking about not using the learner class at all. It looks like I was wrong and there isn’t a simple pytorch train function implementation there. Here is a basic example (UNTESTED CODE):
Hi @Hadus, Thank you for your kindness.
I did the similar way to you and I was able to train.
Now, I want to use learner to train. But I’m not sure what is the proper way.
Could you tell me how to feed model and dataloader to the learner?
uses the model PointNetDenseCls that returns a two values in its forward function. This could cause problems as the learner expects one value to be returned. The easy fix would be to edit the PointNetDenseCls so forward only returns x.
Oh. Thank @Hadus and @pradla.
I already finished the program.
Thank you for your kindness. I am really grateful your helps.
I uploaded the code PointNet on fastai to github too. Hope it’s useful to someone.
Try to reduce your batch_size (that way less data will be loaded at the same time, so that it can fit in the available memory).
I also think num_workers=2 will spawn 2 processes, each with their own copy of the data in memory, doubling the memory usage. Setting num_workers=1 might help.