from keras import backend as K
(after you are done with the model)
K.clear_session()
solves the memory issue.