Hi,
I am using pretty large images for my training and each epoch in my training takes almost half an hour. So, I decided to run it behind VNC on a spot instance but I wanted to know how my training is going on. Getting notifications of 'end of training' on your mobile phone this post taught me about pushover and I used (obviously) @sgugger’s callback notebook (https://github.com/sgugger/Deep-Learning/blob/master/Using%20the%20callback%20system%20in%20fastai.ipynb) to write a small function that helps me log my training metrics. This notebook btw is very useful.
For those who’d like to have something like this, here’s the hacky tiny script. I’m using pushover for my notifications. They have a limit of 7500 requests per month per user - which is fine for my usecase.
def send_notification(msg_string):
'''
This function sends message to my mobile using Pushover.
'''
import requests
from datetime import datetime
url = "https://api.pushover.net/1/messages.json"
data = {
'user' : "<<YOUR_USER>",
'token' : "<<YOUR_TOKEN>>",
'sound' : "gamelan"
}
data['message'] = msg_string
data['message'] = data['message'] + "\n" + str(datetime.now())
r = requests.post(url = url, data = data)
class NotificationCallback(Callback):
def on_train_begin(self):
self.epoch = 0
def on_epoch_end(self, metrics):
val_loss, accuracy = metrics[0], metrics[1]
message = "Epoch: " + str(self.epoch) + " Val.Loss: " + str(val_loss[0])[0:7] + " Val.Acc: " + str(accuracy)[0:7]
send_notification(message)
self.epoch += 1
learn = ConvLearner.from_model_data(mod, data)
notif_cb = NotificationCallback()
learn.fit(lr, 1, wds=wd, cycle_len=10, use_clr_beta=(8,5,0.95,0.85), callbacks=[notif_cb])
and it works like a charm.
And now I can make tea while training without being anxious 
):
