The kernel appears to have died. Runnig notebook on large dataset of images

saidaspen · March 10, 2018, 4:05pm

I am trying to run the notebook from Lesson 1 and 2 on a very big data set of images. The notebook seems to die without any error logging when using ImageClassifierData.from_csv

This is what I run.

This part works fine:

tfms = tfms_from_model(resnet50, 199, aug_tfms=transforms_side_on, max_zoom=1.1)

But this part does not:

train_csv = PATH + "train.csv"
data = ImageClassifierData.from_csv(PATH, 'train', train_csv, test_name='test', num_workers=1, 
                  val_idxs=val_idxs, suffix='.jpg', tfms=tfms, bs=16)

After a minute or two I get this:

Nothing is ever written to the console.

Does anyone know whats going on? I believe it might be the fact that it is a very very large data-set of images (~150 Gb big).

skbisoi · March 21, 2018, 1:04am

I am getting often the same issues.

wspride · March 21, 2018, 2:27pm

I was able to resolve this issue by re-installing PyTorch from source as described here

Cheers,
Will

saidaspen · March 22, 2018, 2:06am

Tried that and updated the nvidia drivers as well as Cuda. Let’s see how it works

Ankit89 · May 16, 2018, 5:33am

Even i am getting the same issue, were you able to resolve the issue of “Kernels appears to have died.” ?

I am on AWS p2.xlarge and using the below code but even with small bs of 16 and sz of 65, it goes out of memory. Any suggestions on how to handle this?

shahnoza · November 19, 2018, 2:22pm

Hello @Ankit89, I have the same situation. Have you been able to solve it?