Kernels dies when using fastai library due to CPU memory issue


(Johannes Laute) #21

Hmm, can you share a full stack trace or provide a minimal code example that fails. I will try to fix it later.
cheers, Johannes


(Florian Peter) #22

Nevermind Johannes, looks like this was unrelated! Apologies. Will update later if I run into any troubles!


(WILLIAM PRIDE) #23

@j.laute 's fix worked for me as well - cheers, well done. Would support this being PRed into master


(Dhruv Jawalkar) #24

Hi there,
I’ve been facing the same issue too!

So, I’ve tried to do the TTA separately for ‘test-jpg’ and ‘test-jpg-additional’ folders. Created separate result csv’s and finally merged them using
‘result = pd.concat([test_df, addn_df])’

And it worked.
Hope this solves for you too.
(Just be watchful of the filenames while creating addn_df and ‘index=False’ during .to_csv())


(YIYU SHI) #25

Thanks for the advice, I will look into it!


(heisenburgzero) #26

I think its raising problem in other places too. Such as functions in metrics.py


(Johannes Laute) #27

Please share some code to reproduce if possible. Maybe we need to call to_tensor or something in the DataLoaderIter, I will investigate this weekend

Cheers, Johannes


(WILLIAM PRIDE) #28

@j.laute I’m running into the same issue reported by @heisenburgzero. I’ve uploaded my notebook here and a Gist of the stack trace I get here. Let me know if you have any thoughts.


(WILLIAM PRIDE) #29

Update, resolved this (at least locally) by re-creating the calls to get_tensor that look to have been lost in @j.laute’s original fix. Demonstrated here


Memory leak in dataloader?
(Caleb Everett) #30

Any suggestions on how to make a permanent fix? Is there anything that can be done to the ThreadPoolExecutor construction in dataloader.py?

https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor-example


(Johannes Laute) #31

Amazing! Will change that as well. Didn’t have time yet to submit a pr to master, hopefully tomorrow.

Cheers, Johannes


(Jeremy Howard) #32

This should now be fixed in git. Let me know if you see any further problems.


#33

I am still facing the same issue after the new update. Can anyone confirm if it works for them now after the update?


#34

I have the error as well. The code seems to have been updated in the dataloader.py file but solution is apparently not working for us.


#35

Git pull fixed the issues for me. Thank you!


#36

It has not been completely solved, although it is better. (Thanks for improving the code)

Indeed, fitting the data uses less RAM (although still a lot). After an epoch the memory gets freed, but when swap was needed only part of that memory is getting freed. So when too many epochs are run, the kernel might still die.

For the Amazon data set and the code I had the fix is good enough, but I think that in some cases this won’t solve the issue completely.


(Nick Johnson) #37

I’m getting this same error when running ImageClassifierData.from_csv against the yelp dataset (which is marked as “extra large”). I’ve verified that it is memory related using top.

Seems odd, since I wouldn’t expect this code to actually load the images into memory.

Note: I tried a git pull and a conda env update.


(Nick Johnson) #38

Ignore that. I had a bug in my data processing code. :frowning:


(Sufiyan Adhikari) #39

Similar problem. How did you fixed that? @harveynick
I am loading data from csv, for Landmark Recognition Challenge and still can’t get it work.
Kernel keeps dying. I am using Google Cloud with 26GB Ram and 1xK80 GPU


(Ankit) #40

@wnurmi , i actually did the changes mentioned in your PR but still Kernel is dying. I am working on same Google Image recognition and Kernel dies while using ImageClassifierData.from_csv. Did any solution worked for you? if so, could you let me know what changes can be done and how?