Reason for "Core dumped" error on learn.fit_one_cycle in lesson 1?

Part 1 (2019) Advanced (Part 1 v3)
1

I have looked on the forum and even though similar types of issues are discussed I could not find any definite solution. Perhaps someone has a suggestion?

I have a DL box at home w/ 1080Ti GPU and 16GB of CPU memory (I know this is small, but I don’t believe it’s an issue) running Ubuntu 16.04 w/ 384.145 nvidia driver, CUDA 9.0, cudNN=7.4.2, Python 3.6.8, PyTorch 1.x (I tried the most recent nightly 1.1.0.dev20190504 just now) and recent fastai=1.0.52

I get an Illegal instruction (core dumped) error in lesson1-pets.ipynb on line learn.fit_one_cycle(4) if I run the script from the Python command line.
It starts, goes for 5-10% on all 4 cores and then fails no matter what batch size is (I tried as small as bs=2). I have set num_workers=0. If I run it from Jupyter, then the error is “The kernel appears to have died. It will restart automatically.” It does not look like I am running out of either CPU or GPU memory - I am monitoring both of them as the script runs (25% is hardly consumed).

If I don’t set num_workers=0 in DataBunch, then I get a different error “RuntimeError: DataLoader worker (pid 2781) is killed by signal: Illegal instruction.” again after it is execulted for 5-10% on all CPU 4 cores - this has also been reported on this forum.

I get similar core dump error from Python if I run another notebook with large images from lesson 3 (I will post details shortly), but otherwise I have no problem executing any of the other notebooks or running any of the DL1 or DL2 notebooks from v2 of the class (2018) using fastai v0.7 on the same DL box in another conda environment.

From reading the forum it seems that the problem may be related to the Pytorch DataLoader, but I thought it was supposed to be fixed in PyTorch 1.0? What is it related to then? Do I need to update CUDA to v.10 or upgrade nVidia drivers? Anyone had a similar problem?

2

I also get the following error in lesson7-superres
when trying to resize images with
parallel(partial(resize_one, path=p, size=size), il.items)

Traceback (most recent call last):
File “”, line 1, in
File “”, line 34, in
File “/home/…/anaconda3/envs/fastai1/lib/python3.6/site-packages/fastai/core.py”, line 326, in parallel
futures = [ex.submit(func,o,i) for i,o in enumerate(arr)]
File “/home/…/anaconda3/envs/fastai1/lib/python3.6/site-packages/fastai/core.py”, line 326, in
futures = [ex.submit(func,o,i) for i,o in enumerate(arr)]
File “/home/…/anaconda3/envs/fastai1/lib/python3.6/concurrent/futures/process.py”, line 452, in submit
raise BrokenProcessPool('A child process terminated ’
concurrent.futures.process.BrokenProcessPool: A child process terminated abruptly, the process pool is not usable anymore

are the two errors related in any way?

3

Update: I upgraded to 418 Nvidia GPU driver and Cuda 10 and still get the same errors.

Is Ubuntu 18.04 preferred way to go?

Any thoughts?

4

Update #2:

Finally got it to work in the new conda environment by first installing Python 3.7 and then PyTorch and fastai using the fastai installation instructions:
conda install pytorch torchvision cudatoolkit=10.0 -c pytorch
conda install -c pytorch -c fastai fastai

I did not install the optimized jpeg encoder this time as optionally recommended - it seems like it screwed up my system last time…