Transfering images to DL box

Deep Learning
1

I am wondering what methods you guys use to transfer all your images to a DL box.
Currently I am training a few models that each have about 1 million images spread among 400 classes. It is a very tedious task to transfer these images from my personal NAS to my DL box. Wondering if some of you have tips to speed this up.

I have all my images on a webserver too, which can be accessed by an URL. Using the URL might be a solution somehow.

Looking forward to some ideas.

2

Hi @GertjanBrouwer!

I’m wondering what method you’re currently using, and the type of times you’re getting?

Off the top of my head I would zip the files, probably using gzip from the terminal then I would scp the files to the remote machine and gunzip them there.

You could also scp them between your webserver and your DL box, if your internet is the bottleneck there might be some gains to be had.

Interested to hear your method!