According to this document https://www.tensorflow.org/tutorials/using_gpu#allowing_gpu_memory_growth :
Note that we do not release memory, since that can lead to even worse memory fragmentation.
What I think happens here: calling K.get_session() in the first line creates a session with default config, which uses all the memory. session.close() doesn’t release it, hence memory consumption stays the same as it was without calling limit_mem.
Just putting this block of code in the beginning of the notebook works for me:
from keras import backend as K
cfg = K.tf.ConfigProto()
cfg.gpu_options.allow_growth = True
K.set_session(K.tf.Session(config=cfg))