Colorful result table

fastai fastai dev
1

I was exploring the idea of using pandas styling to highlight the best results for the metrics in the metrics table when you train. I did a quick custom callback to change the default table.

For example using background gradients to show visually the best metrics your neural network got during training. Not sure about the styling, but I include the code if you want to customize it for your own projects:

5ZLkhUP5tj

import ipywidgets as widgets
from IPython.display import display, HTML
from fastai2.callback.all import *
import pandas as pd
from functools import partial

def highlight(s, usemax=True):    
    if usemax:
        is_max = s == s.max()
    else:
        is_max = s == s.min()
    return ['text-decoration: underline;' if v else '' for v in is_max]

class ColorfulProgressCallback(ProgressCallback):
    def __init__(self):
        super()
        
    def begin_fit(self):
        super().begin_fit()
        if(hasattr(self, 'out')): delattr(self, 'out')
        for i, cb in enumerate(self.learn.cbs):
            if type(cb) == ProgressCallback:
                self.learn.cbs[i] = self
            
    def after_fit(self):
        super().after_fit()
        if(hasattr(self, 'all_log')): delattr(self, 'all_log')
        
    def _write_stats(self, log):
        if(not hasattr(self, 'all_log')):
            self.all_log = pd.DataFrame([], columns=log)
            return
        
        self.all_log.loc[len(self.all_log)] = [l if isinstance(l, float) else str(l) for l in log]
        for c in self.all_log.columns[1:-1]:
            self.all_log[c] = self.all_log[c].astype(float)
            
        s = self.all_log.style
        for c in self.all_log.columns[1:-1]:
            isR2 = 'R2' in c
            low = .8 if isR2 else .2
            high = .2 if isR2 else .8
            s = s.background_gradient(cmap=f'viridis{"_r" if not isR2 else ""}', subset=c, low=low, high=high)
            s.apply(partial(highlight, usemax=isR2), subset=[c])
        s = s.hide_index()
        
        if(hasattr(self, 'out')):
            self.out.update(s)
        else:
            self.out = display(s, display_id=True)

Then you can use it like so:

learner.fit_one_cycle(10, 1e-3, cbs=[ColorfulProgressCallback()])

11 Likes
2

Beautiful! I’d consider adding a PR to the library depending on feedback :smiley:

1 Like
3

Was about to echo the same. I love this! Combine it with a few callbacks to save the learner at each stage you want to keep it at and this can be super easy and useful to track

4

Hmm, its not working for me using fit_flat_cos. I’m setting

defaults.callbacks[-1] = ColorfulProgressCallback

But all the happens is that the metrics table is hidden both during trainining and when training completes…

Also, Is there a way to pass it to a learner or to a specific fit function instead of defaults? I tried passing it like below but I was getting the below error

learn.fit_flat_cos(5, lr=slice(1e-4, 1e-3), cbs=[ColorfulProgressCallback])

ERROR:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-25-c4200ec1ec10> in <module>
----> 1 eff_learn.fit_flat_cos(1, lr=1e-3, cbs=ColorfulProgressCallback) #0.564

~/fastai2/fastai2/callback/schedule.py in fit_flat_cos(self, n_epoch, lr, div_final, pct_start, wd, cbs, reset_opt)
    133     lr = np.array([h['lr'] for h in self.opt.hypers])
    134     scheds = {'lr': combined_cos(pct_start, lr, lr, lr/div_final)}
--> 135     self.fit(n_epoch, cbs=ParamScheduler(scheds)+L(cbs), reset_opt=reset_opt, wd=wd)
    136 
    137 # Cell

~/fastai2/fastai2/learner.py in fit(self, n_epoch, lr, wd, cbs, reset_opt)
    178 
    179     def fit(self, n_epoch, lr=None, wd=None, cbs=None, reset_opt=False):
--> 180         with self.added_cbs(cbs):
    181             if reset_opt or not self.opt: self.create_opt()
    182             if wd is None: wd = self.wd

~/anaconda3/envs/fastai2_me/lib/python3.7/contextlib.py in __enter__(self)
    110         del self.args, self.kwds, self.func
    111         try:
--> 112             return next(self.gen)
    113         except StopIteration:
    114             raise RuntimeError("generator didn't yield") from None

~/fastai2/fastai2/learner.py in added_cbs(self, cbs)
    109     @contextmanager
    110     def added_cbs(self, cbs):
--> 111         self.add_cbs(cbs)
    112         yield
    113         self.remove_cbs(cbs)

~/fastai2/fastai2/learner.py in add_cbs(self, cbs)
     92     def metrics(self,v): self._metrics = L(v).map(mk_metric)
     93 
---> 94     def add_cbs(self, cbs): L(cbs).map(self.add_cb)
     95     def remove_cbs(self, cbs): L(cbs).map(self.remove_cb)
     96     def add_cb(self, cb):

~/fastcore/fastcore/foundation.py in map(self, f, *args, **kwargs)
    360              else f.format if isinstance(f,str)
    361              else f.__getitem__)
--> 362         return self._new(map(g, self))
    363 
    364     def filter(self, f, negate=False, **kwargs):

~/fastcore/fastcore/foundation.py in _new(self, items, *args, **kwargs)
    313     @property
    314     def _xtra(self): return None
--> 315     def _new(self, items, *args, **kwargs): return type(self)(items, *args, use_list=None, **kwargs)
    316     def __getitem__(self, idx): return self._get(idx) if is_indexer(idx) else L(self._get(idx), use_list=None)
    317     def copy(self): return self._new(self.items.copy())

~/fastcore/fastcore/foundation.py in __call__(cls, x, *args, **kwargs)
     39             return x
     40 
---> 41         res = super().__call__(*((x,) + args), **kwargs)
     42         res._newchk = 0
     43         return res

~/fastcore/fastcore/foundation.py in __init__(self, items, use_list, match, *rest)
    304         if items is None: items = []
    305         if (use_list is not None) or not _is_array(items):
--> 306             items = list(items) if use_list else _listify(items)
    307         if match is not None:
    308             if is_coll(match): match = len(match)

~/fastcore/fastcore/foundation.py in _listify(o)
    240     if isinstance(o, list): return o
    241     if isinstance(o, str) or _is_array(o): return [o]
--> 242     if is_iter(o): return list(o)
    243     return [o]
    244 

~/fastcore/fastcore/foundation.py in __call__(self, *args, **kwargs)
    206             if isinstance(v,_Arg): kwargs[k] = args.pop(v.i)
    207         fargs = [args[x.i] if isinstance(x, _Arg) else x for x in self.pargs] + args[self.maxi+1:]
--> 208         return self.fn(*fargs, **kwargs)
    209 
    210 # Cell

~/fastai2/fastai2/learner.py in add_cb(self, cb)
     95     def remove_cbs(self, cbs): L(cbs).map(self.remove_cb)
     96     def add_cb(self, cb):
---> 97         old = getattr(self, cb.name, None)
     98         assert not old or isinstance(old, type(cb)), f"self.{cb.name} already registered"
     99         cb.learn = self

TypeError: getattr(): attribute name must be string
5

Hum, I replace defaults.callbacks[-1] because this is where ProgressCallback usually is and this is the callback responsible to update the progressbar and update the metrics table html. Mine simply inherit from it and override some key functions. I agree that it would be ideal to be able to pass it in cbs directly, but then you would have two ProgressCallback in your cbs, the default one and the colorful version.

Hum, I just tried it locally with fit_flat_cos and did not have any problem. Are you using colab or a normal jupyter notebook? I am using jupyter lab here. First time I play around with ipython widgets, so not too sure if there could be weird intteractions with the way collab vs jupyter handle widgets.

1 Like
6

On Jupyter here, I’ve haven’t pulled from fastai-v2 for a couple of weeks so will see if that fixes it

1 Like
7

Nice idea! I have never heard of pandas styling before. :slight_smile:

One thing you could try if the cb order can be different is to cycle through the cbs and use something like this:

for cb in cbs:
    if type(cb) == ProgressCallback:
        <do something>
8

Yes this could be interesting! Not too sure if we could modify the self.learn.cbs collection from ColorfulProgressCallback constructor though. Basically replacing ProgressCallback instance with ColorfulProgressCallback in the self.learn.cbs collection.

Hum just tried it and it works if I put this in the constructor. Then you can simply pass it in the cbs collection like normal. No need for default anymore @morgan :

for i, cb in enumerate(self.learn.cbs):
    if type(cb) == ProgressCallback:
        self.learn.cbs[i] = self

I will update the main post with this code.

1 Like
9

The error you had was because you did not construct ColorfulProgressCallback. You need to construct it before passing it to cbs:

learner.fit_one_cycle(10, 1e-3, cbs=[ColorfulProgressCallback()])

Also check the code again, I modified it to remove the need to modify the defaults variable. It will now replace ProgressCallback in the learner.

For anyone interested in fiddling with styling, here is a good tutorial on pandas styling:

https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html

2 Likes
10

Still can’t get this to work…have updated fastai2 and made sure ipywidgets is up to date. While training it just prints the progress bar, but no stats, this is half-way through the 2nd epoch, so the 1st epoch’s stats should have been printed:

Screenshot 2020-04-05 at 14.02.04
Screenshot 2020-04-05 at 14.02.041176×104 6.96 KB

Once training is finished the progress bar disappears and there is no output at all from the cell.

Debugging
I’m calling it with

learn.fit_one_cycle(2, 5e-3, cbs=[ColorfulProgressCallback()])

just using a resnext50 created with cnn_learner, nothing fancy.

I added a print statement to try and debug and it doesn’t seem like its entering the with self.widget: part in _write_stats at all…

@etremblay any ideas? I’m using Jupyter Notebook, are you using Jupyter Lab by any chance?

11

I use jupyter lab. I just tested on normal notebooks and indeed it doesn’t work. I think I found the culprit, I was using a widget.Output() from ipywidgets and normal notebooks doesn’t seem to like that. I edited my code in the top post with the fixed code.

2 Likes
12

inferno seems like a good gradient to use too:

image

1 Like
13

Absolutely love this @etremblay , no issues using it and it looks amazing with ‘inferno’

color

On a side note:

I am trying to resolve this issue; https://forums.fast.ai/t/issue-viewing-images-in-voila/68221, Is there any way you can share where in your code you had previously used widgets.Output() so I can how you fixed it.

Thanks!

1 Like
14

Glad you like it!

This is my first time working with widgets. I used it the same way you did in your code in your post (using a with block), but it was only working in JupyterLab and not normal notebooks. Since my scenario was pretty straightforward I just needed an updatable zone I went with

self.out = display(s, display_id=True)

So that I could use self.out.update(s) later.

I had used widgets.Output because this was one of the first stackoverflow post that I found about how to do this. But it did not work in normal notebooks for me, maybe I was doing something wrong.

Sorry I can`t be of more help.

1 Like
15

Thanks for the tip, unfortunately it did not work. It seems that viola does not like to display widget.Outputs() using ax. I had to customize show_image that displayed the image without using ax which worked. https://forums.fast.ai/t/issue-viewing-images-in-voila/68221/6

1 Like