Is this thread closer to your use case? I have not tried this with v1 yet, so not sure how much of the code maps over, but the idea of a MatchedDataSet from files seems closer to your use case.
If this is the case, then it is likely you will benefit from using the built-in ImageDataBunch design that looks at paths or filenames stored in a pd.DataFrame Let me know if that gets you started down a path that is productive or if that is not the right direction.