Sorry for the late response. While improving GC is always in the headlights for the team behind GPUCompiler.jl, we do have a batched iterator called CuIterator in CUDA.jl. This iterator was specifically created for the case where all the data cannot fit in GPU memory, so it moves data to the GPU in batches and eagerly frees the batches to relieve memory pressure.
We are actually discussing the data loading pipeline on today’s call, and CuIterator is one piece of it. If you can’t make it, be sure to check out the minutes to see how it is used in practice.