Workflow to compare & monitor models using WandbCallback

Hi,

I’ve been working on WandbCallback for the past few months (with a lot of help from @sgugger) and I’m very excited to show how it works!

This is still in very active development so I’d love all the feedback you have regarding bugs or new features.

To use it:

import wandb
from fastai2.callback.wandb import *

# start logging a wandb run
wandb.init()  # optional -> wandb.init(project='my_project')

# just add WandbCallback to your learner
learn.fit(..., cbs=WandbCallback())

It let you:

  • quickly compare models -> I used it to debug and check GradientAccumulation
  • make lots of custom graphs or reports pulling data from your runs
  • watch long training runs on your phone
  • log automatically prediction samples

You can test it with your own project or this small demo notebook.

When you run it, you will have access to:

Now what I like most is that if you run the notebook several times trying different parameters (batch size, number of epochs, learning rate, GradientAccumulation callback…), then open your project page, you will see that more than 100 parameters have automatically been logged for you from all the fastai functions you used.

Press the little wand on the top right of your runs summary table, reorganize and hide columns you want, and you get a nice comparative summary.

You can easily create graphs to compare runs.

And finally you can use them to create cool reports where your results are fully traceable.

I’d love any feedback you may have and I’m here to help if you have any questions.

18 Likes

Thanks for all the work on this, that report functionality is a great example! I used the callback a little last week and enjoyed using it, especially for looking at the gradients in the model (although I still need to figure out a good rule of thumb of what is the most informative layer(s) I should be looking at when examining gradients :smiley: )

1 Like

One question I had is how to log slices?

Let’s assume learn.fit(lr=slice(5):

  • Option 1: string, ie {"learn.fit.lr": "slice(None, 5, None)"}: the issue is that we won’t be able to make graphs from it or use the parameter importance feature
  • Option 2: log all parts, ie
    {"learn.fit.lr.start" = None, "learn.fit.lr.stop" = 5, "learn.fit.lr.step" = None}
  • Option 3: same as option 2 but also log stop value within main parameter value, ie
    {"learn.fit.lr.start" = None, "learn.fit.lr.stop" = 5, "learn.fit.lr.step" = None, "learn.fit.lr" = 5}

I think option 2 probably makes the most sense and will be more convenient to make graphs, except when we alternate between using slices and single values.

The main impact I see is for the parameter importance feature (ex on this report). A random forest is run on all the parameters to see which ones are the most important and I’m not sure what makes the most sense in this instance but I think it would be the option 2.

1 Like

Also just learnt about an awesome new feature for semantic segmentation.
Basically images are logged so you can decide which masks to show as overlays and to which opacity.
Definitely want to add it to the callback.

See their example (and they are actually using fastai!).

Sounds reasonable. Though I think it should still be logged like this then post-processed at some point with a regex (e.g., not make log_args have special behavior on slices).

1 Like

Don’t see this in the docs … just wondering what the status of the callback is. Would love to give it a whirl.

@wgpubs: https://dev.fast.ai/callback.wandb

To my knowledge it’s fully integrated :slight_smile:

2 Likes

Correct it is already integrated. I think @sgugger has different plans for third party integrations which is probably why it’s not in the docs at the moment.

2 Likes

I just added a new feature: custom logging of semantic segmentation samples.

You can create custom graphs:

  • select which classes to display
  • add optionally input underneath the mask
  • set up opacity of mask and image

semantic_segmentation.png

You can even plot evolution of prediction over time.

See a sample run

And here I compiled a report showing all the current features of WandbCallback.

3 Likes