CNN activation visualizations

Part 2 (2017)
1

Here are some CNN activations for Starry Night. If you’ve visualized activations, please post some here so others can see.

You can click an image to zoom in. You can click the zoomed-in image to zoom in once more.

Input (three color channels)

input_1_higher_res.png5400×5093 1.41 MB

block1_conv1 (64 filters) (first conv layer)

L80XX5f.jpg2450×2379 935 KB

Each filter is latching on to different parts of the image.

block5_pool (512 filters) (last layer)

block5_pool_higher_res.png5380×5253 195 KB

A portion of block5_pool (40 out of 512 filters) (top left)

Screenshot from 2017-03-05 22-18-05.png1621×959 18 KB

These filter activations are very simple, yet each colored block corresponds to the activations of previous layers which had complicated-looking activations. I want to say, “each colored block is rich with information”, but I don’t know if that makes sense. I do wonder what each colored block means, though.

#Here’s a link to all the filter results on one page

Code

def plots(ims, figsize=(12,6), rows=1, cols=None, interp=None, titles=None, cmap=None):
    f = plt.figure(figsize=figsize)
    for i in range(len(ims)):
        sp = f.add_subplot(rows, cols, i+1)
        if titles is not None:
            sp.set_title(titles[i], fontsize=18)
        plt.imshow(ims[i], interpolation=interp, cmap=cmap)
        plt.axis('off') # New
        plt.subplots_adjust(hspace = 0.500) # New
    return f

# preprocess the image
style_image = Image.open('data/starry_night.jpg')
style_image = style_image.resize(np.divide(style_image.size, 3.5).astype('int32'))
style_arr = preproc(np.expand_dims(style_image ,0)[:,:,:,:3])
shp = style_arr.shape

# Get the activations
start = time()
model = VGG16_Avg(include_top=False, input_shape=shp[1:])
to_plot = []
for layer in model.layers:
    layer = layer.output
    layer_model = Model(model.input, layer)
    # get activations
    activations = layer_model.predict(style_arr)
    images = []
    h, w, f = activations[0].shape
    for k in range(f): # for each filter
        image = []
        for i in range(h): # for each row
            for j in range(w): # for each column
                pixel = activations[0][i][j][k]
                image.append(pixel)
        images.append(np.array(image).reshape(h, w))
    to_plot.append(images)
print(time() - start) # 24 seconds elapsed

# Plot the activations and save the plots
squares = np.array([i**2 for i in range(24)])
start = time()
for i, layer in enumerate(model.layers):
    start2 = time()
    try:
        images = to_plot[i]
        nb_imgs = len(images)
        nb_rows = np.sqrt([square for square in squares if square >= nb_imgs][0]).astype('int')
        fig = plots(images, rows=nb_rows, cols=nb_rows, figsize=(96,96))
        plt.savefig("data/starry_night/%s.png" % layer.name, bbox_inches='tight')
    except ValueError:
        print(i, nb_imgs, nb_rows)
        break
    print(layer.name+':', time() - start2)
print(time() - start) # 9 minutes, 21 seconds elapsed
5 Likes
2

Cool !
Could be useful.

3

Just a reminder about the DeepViz toolbox: http://yosinski.com/deepvis

Makes these kinds of studies easy and interactive.

6 Likes
4

I found it interesting to take the eigenvectors (principal components) of the gram matrix and then project them back across the vectorized images to see what parts of the images are generating the most variance (or attention) in the network:

First five layers, 3 ‘strongest’ eigenvectors:

eig1.png712×711 454 KB

Last three layers

eig4.png712×575 22.6 KB 

model = VGG16_Avg(include_top=False, input_shape=shp[1:])
i = 3
image_set = []
for layer in model.layers:
layer = layer.output
layer_model = Model(model.input, layer)
# get activations
activs = layer_model.predict(style_arr)
images = []
h, w, f = activs[0].shape

stacked_activs =np.reshape(activs, (h*w,f))                                            
covar = np.dot(np.transpose(stacked_activs),stacked_activs)/(h*w)

eigvals, eigvects = np.linalg.eig(covar)

eigimages = []
for j in range(min(i,f)):
    eigvect = eigvects[eigvals.argsort()[::-1][j]]
    eigimg = np.reshape(np.dot(activs, eigvect), (h,w))
    eigimages.append(eigimg)

image_set.append(eigimages)       

fig = plt.figure(figsize=(10,10))

for i in range(5):
for j in range(3):
    ax = fig.add_subplot(5,3, i*3 + j + 1)
    ax.imshow(image_set[i][j], cmap = 'bwr')
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.tight_layout()
5 Likes