Platform: Colab (Free; $10/month Pro)

ulat · June 9, 2020, 1:20pm

Automatically saving models to gdrive.

I was annoyed when colab lost the session during training and I had to start from scratch. Therefore I extended the SaveModel Callback to not only save the best model after each epoch but also tar and copy it to my gdrive (or any other bucket-store).

In case someone hast faced simliar problems I share the code snippet:

gist.github.com

https://gist.github.com/ulat/5658a863b578d2617654dabddd9dd1a7

SaveAndPersistModel.py

class SaveAndPersistModel(SaveModelCallback):
  def __init__(self , monitor='valid_loss', comp=None, min_delta=0., fname='model', every_epoch=False, add_save=None, with_opt=False, 
               persistence_path:str='./drive/My\ Drive/'):    
    super().__init__(monitor, comp, min_delta, fname, every_epoch, add_save, with_opt)
    self.persistence_path = persistence_path

  def after_epoch(self):  
    super().after_epoch()
    #print(str(self.path/f"{self.fname}.tar.gz"))
    #print(str(self.path/self.model_dir/f"{self.fname}.pth"))

This file has been truncated. show original