Kernel crashing consistently at learn.export()

Hi, I have modified the 02_production notebook for using my own dataset of hand gesture images.

Kernel is crashing always at learn.export().

On the terminal too, there is not much information -

[W 00:26:40.933 LabApp] 404 GET /static/lab/vendors~@jupyter-widgets/controls.5530d02956f7d73b04f7.js (127.0.0.1) 0.91ms referer=http://localhost:8888/lab?
[W 00:26:40.936 LabApp] 404 GET /static/lab/vendors~@jupyter-widgets/controls.5530d02956f7d73b04f7.js (127.0.0.1) 0.82ms referer=http://localhost:8888/lab?
[W 00:26:40.940 LabApp] 404 GET /static/lab/vendors~@jupyter-widgets/controls.5530d02956f7d73b04f7.js (127.0.0.1) 0.86ms referer=http://localhost:8888/lab?
[I 00:27:00.942 LabApp] Saving file at /Templates/tmp_programming/Projects/GestureRecognition/HandRecognition/02_production.ipynb
[I 00:27:11.966 LabApp] KernelRestarter: restarting kernel (1/5), keep random ports
kernel d1e4a435-6c82-4b9c-98eb-0e6d11f04828 restarted

Is there any way I can debug why the crash is happening at same point.

Thanks
Rahul

When I was using Keras, when the number of neural in hidden layer is too large, the kernel would die. I deliberately make the model very very large and complex to test my computer. Up till now I do not know what happened when the model is large and what cause the Jupyter kernel die. Do you use your own computer to run the program? Maybe the model is too complex for your computer to handle. How about change a platform?

1 Like

It worked. Few days back I was checking that issue, I noticed that somehow ImageClassifierCleaner just above started giving this issue too. At this function, CPU shoots up and suddenly jupyter freezes for few seconds and Kernel dies.

I commented this call and tried to train again. Now kernel is not crashing at learn.export().
I will experiment why ImageClassifierCleaner is messing with kernel. Its possible that issue may be something else. Need to check more.

1 Like

I am having similar issues as well.

RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 541065216 bytes. Error code 12 (Cannot allocate memory)

RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can’t allocate memory: you tried to allocate 541065216 bytes. Error code 12 (Cannot allocate memory)

I haven’t had any problem at export time, but when running the superres model in inference mode with large images, I experienced very similar issues. So far, I wasn’t able to find a better solution than reducing the model’s input image size :frowning:.

1 Like

It was due to the fact that in Paperspace I wasn’t with the P5000 but CPU before