I am doing the statefarm problem. Some code snippets are as follows:
test_batches, num_test_batches = get_batches('./data/test', gen=datagen, shuffle=False, batch_size=batch_size * 2) vgg_model = VGG16(include_top=False, weights='imagenet') bottleneck_test_data = vgg_model.predict_generator(test_batches, num_test_batches, verbose=1) np.savez('bottleneck/VGG16TestData', bottleneck_test_data=bottleneck_test_data)
While using transfer learning with the VGG16 model, one needs to precompute the data in the train, validation and test sets. Usually, when we precompute the data, the output of the VGG16 model is of the entire data and resides in the VRAM. Because it takes a lot of time, it is better to store it as an .npy file for easier retrieval. But when the dataset is extremely large running the np.savez command becomes unviable as the VRAM gets almost filled up ? I know the obvious method would be is to go for a higher configuration. But is there another alternative?