It seems that I can’t train the model on a huge dataset (17M images) on my local machine. I’ve created a post here describing my problem.
Could somebody advice what to do in this case? Do I need to enforce
num_workers=0? But in this case, the training speed becomes really low due to I/O bottleneck.
Is it a memory-leakage problem in PyTorch, or am I doing something wrong?