Hi, all of this pain is a combination of:
- windows
- multiprocessing
- jupyter
If you skip one these points, it’s quite easy:
- If no multiprocessing is fast enough, set num_workers=0 .
- If you don’t use jupyter, protect the multiprocessing parts with a
if __name__ == '__main__':
# create your dataloader here
block.
If you want all three, you also have to exclude your functions in another file. See https://stackoverflow.com/questions/47313732/jupyter-notebook-never-finishes-processing-using-multiprocessing-python-3
There is also a dynamic solution, but I haven’t tried it.