GPU Garbage Collection

Part 1 (2017)
1

Has anyone found a way to run a garbage collection step on the GPU? I’m running into OOM errors when I run a series of models despite including del model prior to instantiating the next model. I’ve tried a few things, like including a time.sleep() prior to moving on to the next model, but the so far the only thing that works is restarting the notebook and running the next model, which isn’t ideal if I want to compare results across models?

2

Don’t know about theano, but for tf you can do a complete reset:

ops.reset_default_graph()
try: sess.close()
except: pass
sess = tf.InteractiveSession()

For keras with tf backend, you’ll need K.{get/set}_session or similar with a variant of the above.

3

@jeremy - thanks! Just to show an example for Keras with tf backend that appears to be working well now!

import tensorflow as tf


def clear_mem():
    try: sess.close()
    except: pass
    sess = tf.InteractiveSession()
    K.set_session(sess)
    return
4

For the Theano there is a config option config.allow_gc = true or false. Please, see the doc.

5

Update - since I was using @jeremy’s limit_mem() function. I was stepping on my own feet. Thus I just wrote a little wrapper function to close the current session and then call limit mem:

def limit_mem():
    cfg = K.tf.ConfigProto()
    cfg.gpu_options.allow_growth = True
    K.set_session(K.tf.Session(config=cfg))

def clear_mem():
    sess = K.get_session()
    sess.close()
    limit_mem()
    return
1 Like
6

How about we just add an extra line to limit_mem()?:

def limit_mem():
    K.get_session().close()
    cfg = K.tf.ConfigProto()
    cfg.gpu_options.allow_growth = True
    K.set_session(K.tf.Session(config=cfg))

Then we can just call that function again to reset.

2 Likes
7

I am inexperienced at programming, could you tell where to call this function and how to call ? Thanks