Loving Floydhub so far.
I have a question about downloading data-sets from AWS S3. My PyTorch models are trained on huge amounts of data that we generate using a Spark process and dump to an AWS S3 location. My Python code loads files of data from there on-demand. Of course when I try to do that from the Floyd instance, it fails because it is not authorized to access my AWS S3 data.
How do you guys suggest getting around this? I think your data-set creation workflow is only for uploading data from my local computer. But it would be great to support downloading from an AWS S3 bucket, and if the bucket has access restrictions, there should be a way to supply the necessary credentials (secret/access key).
Once I know how to do this, I can switch my workflow to Floydhub