I’m running theano with the device set to cuda0, and when running models am having problems with the GPU memory filling.
How can I empty the GPU memory?
When I re-instantiate a model for configuration the memory does not empty, for example:
first model
model = sequential(...)
model.compile(...)
model.fit_generator
check gpu memory and it is 40%, then I continue by creating a new model variable
model = sequential(...)
model.compile(...)
model.fit_generator
check gpu memory and it is 80%
The old models never get written out of the GPU.
The only solution I have found is restarting the kernel which is not always a good option, is there a better way?
I have tried a few things such as, deleting the model, clearing the session, and running garbage collection.
del model
K.clear_session()
gc.collect()
As well as reading up on the forum about similar issues such as:
Surely there should be an easy way to do this, or maybe it is time to move over to tensorflow…