Edit: NVM, fixed. I had set up my Paperspace to automatically log out after 1 hour of inactivity. Unfortunately, it only counts activity in the terminal and not IPython activity which tripped it up after every hour of use. So I increased the auto logout timer to 8 hours in case I leave my machine running so not to get charged a fortune.
So I was going through and running this lesson and once I got to near the end, my paperspace session abruptly ended. I figure since the iPython notebook had commands to periodically save the model using learn.save(f'{sz}') I would be alright loading the one from where I left off.
Nope. In fact, I got the following error:
FileNotFoundError: [Errno 2] No such file or directory: ‘data/planet/models/256.h5’
And when I checked the data/planet/models directory, it was completely empty! And yet, I had never received any error messages whenever I was running the code fragments containing learn.save(f'{sz}').
Update: I ran Lesson 2 again and got disconnected near the end yet again here:
learn.unfreeze()
learn.fit(lrs, 3, cycle_len=1, cycle_mult=2)
learn.save(f'{sz}')
Update2: On my 3rd run, this happened before it even got to the end. Here’s the screenshot of the error, the code snippet, plus the intermediary result in the Jupyter notebok before it crashed:
What’s going on here? (I’m using a P5000) I’ve wasted several hours of my time AND computing resources only to have my instance crash right near the end every time so far!