Looking for a graph I had seen in one of the lessons

fastai
1

I am looking for a graph Jeremy shown in a lecture (and I think I saw it on his twitter recently but could not find it) where he shows either the activations or the weights starting to increase and then collapsing, increasing and collapsing. The graph had kind of a blue background with very fine green lines… Not too sure if it is related to ActivationStats.

Thanks!

2

IIRC it is (if we’re talking fastai2). In fastaiv1 it was called the ColorfulDimension :slight_smile:

2 Likes
3

Oh thanks! There’s no example on how to use the graphs. It requires an idx parameter for the graph function and I am not sure what it is for!

4

I think it’s for the layer you want to use (I haven’t played with it but I’m pretty sure) (I also very much so need to)

1 Like
5

Turns out that it is pretty easy to use in v2.

stats = ActivationStats(with_hist=True)
learner = Learner(dls, model, cbs=[stats])

#... fit here...

layers = [m for m in flatten_model(model) if has_params(m)]

for i, layer in enumerate(layers):
    fig, ax = plt.subplots(figsize=(16, 32))
    ax.set_title(layer)
    stats.color_dim(i, ax=ax)

This will print the colorful graph for each of your layer.

Seems really helpful for me to find problematic layers:

image
image907×179 18 KB

or this one seems fishy:

image
image907×179 7.35 KB

1 Like
6

What’s the general guideline for reading them? (I only know of it) and then what do we do with this knowledge?

7

I found those two post last night:

1 Like