I have a large amount of csv files (train_00001.csv, …, valid_00001.csv, …) with tabular data that I am trying to utilize the DataBlock for. I originally started by concatenating them all together and using TabularPandas and TabularDataLoaders, but the resulting dataframe is too large to fit into memory. I figure if my dataset can be the list of files, then I “transform” the file into a dataframe then tensor, it will be easier to train a model with DDP since the dataloader doesn’t have a large memory footprint.
The one problem I can’t figure out is how to read the csv into a dataframe only once and extract the Xs and Y columns from that object. It seems like I get the file name and read the csv with pandas both for Xs and Y columns. Any suggestions?