How to train on tabular data in mini batches?

[ForBo7 (Salman Naqvi)]

This topic here may provide a few pointers.

Loading text data iteratively

I believe you should be able to use something like from_folder. For large datasets you can’t fit in RAM you shouldn’t use Dataframes. Those inherently load the whole dataset into RAM first which won’t work in your case. You should instead probably just have .txt files where each record is its own file and they are loaded on the fly. For such a large dataset it may be worth looking into writing a custom dataloader. Also you probably want to create a small subset to work with to get everything s…