Why fit_generator is much slower than fit(keras2)?

Part 1 (2017)
1

Hi, I have a large dataset(around 100GB). so I tried bcolz_array_iterator.py discussed in the forum(chunklen is set to 16.) Batch size is set to 1024. The training time takes 2-3 hours for one epoch. Instead i tried fitting by first loading them all into memory, then using fit method. It takes only less 10 minutes for one epoch(I use the muti_gpu.py discussed here, but it seems not working with fit_generator).
I also tried tuning the number of workers parameter(from 2 to 16) as discussed in the forum, but it seems not working. I observed that the utility of gpu is very low(almost 0) for the fit_generator.

So why is it so slow to use fit_generator? how can I improve the speed using it( I have larger dataset which cannot fitted into the memory anyway). How do you deal with very very large dataset? Hope to get a more concrete solution. Thanks a lot!

2

You can try using queues in tensorflow . For more information check the below link.

https://www.tensorflow.org/performance/performance_guide

3

It sounds like your script is either a) not using the gpu or b) generating data too slowly to keep gpu occupied.

Hard to troubleshoot without more info.

1 Like
4

to check your gpu

from tensorflow.python.client import device_lib
print (device_lib.list_local_devices())