Training metrics as notifications on mobile using Callbacks

(Phani Srikanth) #1

Hi,

I am using pretty large images for my training and each epoch in my training takes almost half an hour. So, I decided to run it behind VNC on a spot instance but I wanted to know how my training is going on. Getting notifications of 'end of training' on your mobile phone this post taught me about pushover and I used (obviously) @sgugger’s callback notebook (https://github.com/sgugger/Deep-Learning/blob/master/Using%20the%20callback%20system%20in%20fastai.ipynb) to write a small function that helps me log my training metrics. This notebook btw is very useful.

For those who’d like to have something like this, here’s the hacky tiny script. I’m using pushover for my notifications. They have a limit of 7500 requests per month per user - which is fine for my usecase.

def send_notification(msg_string):
    '''
        This function sends message to my mobile using Pushover.
    '''
    import requests
    from datetime import datetime

    url = "https://api.pushover.net/1/messages.json"
    data = {
        'user'  : "<<YOUR_USER>",
        'token' : "<<YOUR_TOKEN>>",
        'sound' : "gamelan"
    }
    data['message'] = msg_string
    data['message'] = data['message'] + "\n" + str(datetime.now())

    r = requests.post(url = url, data = data)

class NotificationCallback(Callback):
    def on_train_begin(self):
        self.epoch = 0
    def on_epoch_end(self, metrics):
        val_loss, accuracy = metrics[0], metrics[1]
        message = "Epoch: " + str(self.epoch) + " Val.Loss: " + str(val_loss[0])[0:7] + " Val.Acc: " + str(accuracy)[0:7]
        send_notification(message)
        self.epoch += 1

learn = ConvLearner.from_model_data(mod, data)
notif_cb = NotificationCallback()
learn.fit(lr, 1, wds=wd, cycle_len=10, use_clr_beta=(8,5,0.95,0.85), callbacks=[notif_cb])

and it works like a charm.

And now I can make tea while training without being anxious :wink:

48 Likes

Callbacks in fast.ai
#2

That’s pretty awesome!

0 Likes

(Amrit ) #3

This is super cool!

0 Likes

(Kaitlin Duck Sherwood) #4

Here’s the version which I use, which uses Telegram as the transport layer:

"""
Derived from Matt Potma's basic telegram bot usage script.

Utilizes this API Library:
   https://github.com/python-telegram-bot/python-telegram-bot
To install:
   pip install python-telegram-bot --upgrade

In ~/.telegram on the machine running the job, put

{"api_key": "462203107:<your API key>",
 "chat_id": "<your chat ID>"}

Here's how you get an API key:
   https://core.telegram.org/api/obtaining_api_id
Here's how you get your chat ID:
   https://stackoverflow.com/questions/32423837/telegram-bot-how-to-get-a-group-chat-id

"""

import telegram
import json
import os

def notify_me(message="Job's done!"):
    filename = os.environ['HOME'] + '/.telegram'
    with open(filename) as f:
        json_blob = f.read()
        credentials = json.loads(json_blob)

    # Initialize bot
    bot = telegram.Bot(token=credentials['api_key'])

    # Send message
    bot.send_message(chat_id=credentials['chat_id'], text=message)

I just use it to tell me the job is over (so I can turn off the instance and not keep racking up charges), but it would be easy to change to provide information about the run.

12 Likes

Callbacks in fast.ai
#5

Here is the code for Slack notification. I got it from using-python-slack-for-quick-and-easy-mobile-push-notifications

from slackclient import SlackClient
def slack_message(message, channel):
    token = 'xoxp-7067844xx-xxxxxxxx-xxxxxxxxxxxx'
    sc = SlackClient(token)
    sc.api_call('chat.postMessage', channel=channel, 
                text=message, username='My Sweet Bot',
                icon_emoji=':robot_face:')
    
slack_message('A greeting from a notebook', 'chanel-name')

To get the token:

3 Likes

(Dennis Sakva) #6

Here is my version of Slack Callback. But it is for Keras if anyone is still using it. Prepare to lose your ability to focus and do anything useful when your slack beeps at you. LOL :slight_smile:

1 Like

#8

Out of curiosity, how large are you images?

0 Likes

(Phani Srikanth) #9

Hi @AlexR,

Most of them are 2560x1920 while a few of them are ~3000x3000.

0 Likes

(ramin) #10

I did a similar thing using IFTTT (because it’s free). Didn’t want to use Pushover or other similar services because they start charging you after the trial period (7 days for Pushover, I believe).
Here is the step-by-step instructions to setup your ML notification system:


IMG_0492

5 Likes

(ramyjackson) #11

Custom callback objects have access to the current model and it’s training parameters via the following fields:

self$params

Named list with training parameters (eg. verbosity, batch size, number of epochs…).

self$model

Reference to the Keras model being trained.
https://downloader.vip/facebook/ https://downloader.vip/netflix/

0 Likes

(Haider Alwasiti) #12

If you are interested to apply this notification with callbacks in fastai v1, there has been few changes on how to use callbacks. Here is my code using Telegram to show metrics in my fitbit watch + phone inspired by @binga and @Ducky :

import telegram
import json
import os

def notify_me(message="Job's done!"):
    filename = os.environ['HOME'] + '/.telegram'
    with open(filename) as f:
        json_blob = f.read()
        credentials = json.loads(json_blob)

    # Initialize bot
    bot = telegram.Bot(token=credentials['api_key'])

    # Send message
    bot.send_message(chat_id=credentials['chat_id'], text=message)

Here is an example that can be used with fastai CNN model to send Telegram messages with every epoch:

@dataclass
class NotificationCallback(Callback):
        
    def on_train_begin(self, metrics_names: StrList, **kwargs: Any) -> None:
        notify_me("Epoch: train_loss , valid_loss , error_rate")

    def on_epoch_end(self,  epoch: int, smooth_loss: Tensor, last_metrics: MetricsList, **kwargs: Any) -> bool:
        super().on_epoch_end(**kwargs)
        val_loss, accuracy = last_metrics[0], last_metrics[1]
        message = str(epoch) + ": " + f"{smooth_loss.item():.4f}" + " , " + f"{val_loss:.4f}" + " , " + f"{accuracy:.4f}"
        notify_me(message)
        return False #  if return true it will stop training at this point

learn = create_cnn(data, models.resnet34, metrics=error_rate)
notif_cb = NotificationCallback()
learn.fit_one_cycle(4, callbacks=[notif_cb])

Here it is in my Telegram goup (me and Jupyter only :slight_smile: ):

I struggled to create a bot in Telegram and here are my rough thoughts and links that I used to understand how to do it:

Utilizes this API Library: https://github.com/python-telegram-bot/python-telegram-bot
To install: pip install python-telegram-bot --upgrade

To generate an Access Token, you have to talk to BotFather: https://t.me/botfather and follow a few simple steps (described here: https://core.telegram.org/bots#6-botfather ).

For a simple example: https://github.com/python-telegram-bot/python-telegram-bot/wiki/Introduction-to-the-API

‘TOKEN’ should be replaced by the API token you received from @BotFather:

In ~/.telegram on the machine running the job, put
example:
{“api_key”: “4424434344:abcdefddddd”, “chat_id”: “-344586144”}

Here’s how you get an API key: https://core.telegram.org/api/obtaining_api_id

Here’s how you get your chat ID: (https://stackoverflow.com/a/50661601/1970830

You should create an API in your Telegram account:
https://my.telegram.org/apps

I should mention that I found @devforfu notebooks quite useful in developing the Telegram notification callbacks. If you are interested in callbacks of fastai v1 you should definitely check his nb here and especially this.

9 Likes

RuntimeError: DataLoader worker is killed by signal
Callbacks in fast.ai
(Vijay Narayanan Parakimeethal) #13

This is great @hwasiti. Will use this for creating callbacks in my notebooks now.

0 Likes

(Cedric Chee) #14

I was talking and referring about this during our last fast.ai Asia study group meetup where you did a mini presentation on callbacks. An example from one of my notebook where I used the callback pattern to send push notifications to my phones:

But that’s for fastai v0.7.0.

1 Like

Callbacks in fast.ai
(Vijay Narayanan Parakimeethal) #15

Yes, I did the mini presentation on using callbacks. Will post the same onto the forums soon.

2 Likes

(Vijay Narayanan Parakimeethal) #16

Got it. Were you able to fit the caravana with the new SegmentationImageList on the v3 of fastai?

0 Likes

(Cedric Chee) #17

No, still working on it.

0 Likes

#18

Just for info, I developed an extension to a module that let you see the graphs of losses and metrics online using Weights & Biases. See here: Online logging, monitoring and comparison of models with W&B

1 Like

#19

@hwasiti I ran the exact code which you posted but for pushover (thanks @binga) with one little change wrt to latest release the on_epoch_end() should return -> None (actually an ItemList but setting it to none also works fine)
Here’s the gist also added a on_train_end() callback too!!

Pushover now supports Images too link

Any leads on how I can send the accuracy vs epoch plot at end of training onto mobile as push notification?

Update : Found a way to send loss plots to mobile as notification :grinning:

1 Like

(Jeffrey Fisher) #20

I made a callback to send text messages via Twilio:

# Goto https://www.twilio.com, sign up for a free account and enter
# your details in the variables below, and you can be texted
# after every epoch or after all training is complete, or both
twilio_account_sid = "YOUR_ACCOUNT_SID"
twilio_auth_token = "YOUR_AUTH_TOKEN"
mobile_number = "+YOUR_MOBILE_NUMBER"
twilio_number = "+YOUR_TWILIO_NUMBER"

# Get latest twilio library
! pip install twilio>=6.0.0

from twilio.rest import Client
client = Client(twilio_account_sid, twilio_auth_token)

# Helper function to send a text message to yourself
def textme(msg):
    client.messages.create(to=mobile_number, 
                       from_=twilio_number, 
                       body=msg)

# LearnerCallback where you can define what text to send
# on every epoch end and training end.
class SMSScheduler(LearnerCallback):
    "Sends texts to user during and after training"

    def __init__(self, learn:Learner):
        super().__init__(learn)
                
    # This def can be commented out if you don't want a text every epoch
    def on_epoch_end(self, **kwargs):
        "Send text to me on every epoch end"
        textme("Epoch end.")
    
    # This def can be commented out if you don't want a text on training end
    def on_train_end(self, **kwargs):
        "Send text to me after trainingn is complete"
        textme("Training Complete!")
        
        
# Usage:
# learn.fit_one_cycle(10, callbacks=[SMSScheduler(learn)])

I was thinking it would be cool if the on_epoch_end callback would send the “nth out of total m” callback numbers, so you could get an idea of how the progress is going and not have to remember how many epochs there are. Any ideas on how to add that?

1 Like