Yeah, I have the same issue with this dataset as well, fighting for a few weeks already 
Also, it seems that using plain PyTorch training loop (nightly version) shows the same memory leakage. I am not sure why the amount of the used RAM is linearly increasing during a single training epoch. Shouldn’t the samples be released after a batch is completed?