How to tell if progress is being made by notebook?

Part 1 (2018) Beginner (2018)
1

Is it possible to determine if progress is being made by Jupyter notebook?

I have a database of 25000 1024x1024 images which is running on a i7 gen1 CPU with a gtx1050 gpu for 24 hours using the lesson1 notebook (cats Versus dogs), but I can’t tell if it is making progress or crashed. How can I tell?? The computer is responsive, and at one point firefox said a script wasn’t responding, but I clicked wait and continue and it eventually became responsive again. I seem the memory being used and it looks like less is being used than when I ran 2500 of the same images - which is why I’m concerned. How long should it take if 2500 took about 2 hours?

jupyternotebookemaybeprogressing
jupyternotebookemaybeprogressing.png791×391 24.7 KB

Thanks!

2

I have a screenshot of running nvidia-smi which I found when I googled something like top but for GPU on ubuntu (https://devtalk.nvidia.com/default/topic/486122/something-like-34-top-34-to-monitor-the-gpu-/).

IsItComputing

System load = 12%
Mem = 2734mb
swap = 1590mb
uptime = 3 days

Why would IPython not return or say something if it crashed? I see “err” in that, but the notebook didn’t terminate… but I also don’t see it listed as a GPU thread… Can someone please explain??