I’m going through Part 1 using Paperspace Gradient. I’ve started downloading some images for my own project from a large public dataset, resizing them, and storing them in my Gradient persistent storage. I got OK performance by using aiohttp to run the downloads in parallel, but it still takes a few hours to run my downloader – the first time I tried, my Free GPU session actually timed out in the middle of the process! I’d like to try parallelizing the image resizing and disk IO as well, and it seems like ipyparallel is the easiest way to do that.
I got ipyparallel through the Conda tab and started ipcluster in a terminal. The IPython Clusters tab is available to me now, but its list is empty. I’m not totally sure where to go from here – does anybody have any suggestions? Or an easier way to parallelize?