Has anyone found a way to run a garbage collection step on the GPU? I’m running into OOM errors when I run a series of models despite including del model
prior to instantiating the next model. I’ve tried a few things, like including a time.sleep()
prior to moving on to the next model, but the so far the only thing that works is restarting the notebook and running the next model, which isn’t ideal if I want to compare results across models?
Don’t know about theano, but for tf you can do a complete reset:
ops.reset_default_graph()
try: sess.close()
except: pass
sess = tf.InteractiveSession()
For keras with tf backend, you’ll need K.{get/set}_session or similar with a variant of the above.
@jeremy - thanks! Just to show an example for Keras with tf backend that appears to be working well now!
import tensorflow as tf
def clear_mem():
try: sess.close()
except: pass
sess = tf.InteractiveSession()
K.set_session(sess)
return
For the Theano there is a config option config.allow_gc = true or false. Please, see the doc.
Update - since I was using @jeremy’s limit_mem() function. I was stepping on my own feet. Thus I just wrote a little wrapper function to close the current session and then call limit mem:
def limit_mem():
cfg = K.tf.ConfigProto()
cfg.gpu_options.allow_growth = True
K.set_session(K.tf.Session(config=cfg))
def clear_mem():
sess = K.get_session()
sess.close()
limit_mem()
return
How about we just add an extra line to limit_mem()
?:
def limit_mem():
K.get_session().close()
cfg = K.tf.ConfigProto()
cfg.gpu_options.allow_growth = True
K.set_session(K.tf.Session(config=cfg))
Then we can just call that function again to reset.
I am inexperienced at programming, could you tell where to call this function and how to call ? Thanks