Training metrics as notifications on mobile using Callbacks

Part 2 & Alumni (2018)
12

If you are interested to apply this notification with callbacks in fastai v1, there has been few changes on how to use callbacks. Here is my code using Telegram to show metrics in my fitbit watch + phone inspired by @binga and @Ducky :

import telegram
import json
import os

def notify_me(message="Job's done!"):
    filename = os.environ['HOME'] + '/.telegram'
    with open(filename) as f:
        json_blob = f.read()
        credentials = json.loads(json_blob)

    # Initialize bot
    bot = telegram.Bot(token=credentials['api_key'])

    # Send message
    bot.send_message(chat_id=credentials['chat_id'], text=message)

Here is an example that can be used with fastai CNN model to send Telegram messages with every epoch:

@dataclass
class NotificationCallback(Callback):
        
    def on_train_begin(self, metrics_names: StrList, **kwargs: Any) -> None:
        notify_me("Epoch: train_loss , valid_loss , error_rate")

    def on_epoch_end(self,  epoch: int, smooth_loss: Tensor, last_metrics: MetricsList, **kwargs: Any) -> bool:
        super().on_epoch_end(**kwargs)
        val_loss, accuracy = last_metrics[0], last_metrics[1]
        message = str(epoch) + ": " + f"{smooth_loss.item():.4f}" + " , " + f"{val_loss:.4f}" + " , " + f"{accuracy:.4f}"
        notify_me(message)
        return False #  if return true it will stop training at this point

learn = create_cnn(data, models.resnet34, metrics=error_rate)
notif_cb = NotificationCallback()
learn.fit_one_cycle(4, callbacks=[notif_cb])

Here it is in my Telegram goup (me and Jupyter only :slight_smile: ):

Screenshot_20181128-052148_cropped
Screenshot_20181128-052148_cropped.jpg1075×1064 220 KB

I struggled to create a bot in Telegram and here are my rough thoughts and links that I used to understand how to do it:

Utilizes this API Library: https://github.com/python-telegram-bot/python-telegram-bot
To install: pip install python-telegram-bot --upgrade

To generate an Access Token, you have to talk to BotFather: https://t.me/botfather and follow a few simple steps (described here: https://core.telegram.org/bots#6-botfather ).

For a simple example: https://github.com/python-telegram-bot/python-telegram-bot/wiki/Introduction-to-the-API

‘TOKEN’ should be replaced by the API token you received from @BotFather:

In ~/.telegram on the machine running the job, put
example:
{“api_key”: “4424434344:abcdefddddd”, “chat_id”: “-344586144”}

Here’s how you get an API key: https://core.telegram.org/api/obtaining_api_id

Here’s how you get your chat ID: (https://stackoverflow.com/a/50661601/1970830

You should create an API in your Telegram account:
https://my.telegram.org/apps

I should mention that I found @devforfu notebooks quite useful in developing the Telegram notification callbacks. If you are interested in callbacks of fastai v1 you should definitely check his nb here and especially this.

9 Likes