Im looking at the function Jeremy used but it seems you need to load the entire dataset into memory?
def save_array(fname, arr): c=bcolz.carray(arr, rootdir=fname, mode='w'); c.flush()
If I run my this code, it seems to take up all of my RAM memory and crash my computer:
train_data = np.concatenate([next(x) for _ in range(int(np.ceil(total_samples/r_batch_size)))]) save_array(os.path.join(save_dir, "train_data"), train_data)
Any tips as to how to save it without loading everything into memory?