Very slow dataloader

(Rohit Singh) #1

I am creating an ObjectItemList and running into very slow loading of each batch.
x,y = next(iter(dl))
takes upwards of 1 minute for each batch, even when the batch size is small (16).

This slowness shows up in calls such as show_batch(), show_results() and training in each epoch. Am I missing something? Using 64GB RAM, i7 Extreme processor and GP100 GPU.

(Rohit Singh) #2

Digging further, it looks like it might be related to (Windows issue). However, in the fastai code num_workers is set to 0 before a batch is fetched:

Why is num_workers being set to 0?

(Rohit Singh) #3

I have a workaround for this issue. In num_workers needs to be set to 0 in the intercept_args function.