CNN with large amount of categories

(Pavel Rybnicek) #1


I’m trying to train a convolutional network with large amount of classes (up to 30 000 classes) with resnet34.

My dataset consist of one image per class (and a few extra images for validation).

The learning goes quite well for about 200 classes, but I get strange graph from the recorder for larger sets. The train loss drops very fast at the start of the epoch and then it grows back. The result is a saw-tooth graph.

Why is this happening?

And the second question - is there any practical limit of classes amount for the specific network architecture?

1 Like

(Morgan McGuire) #2

Woah thats a lot of classes! And not a lot of training data per class… Are you doing lots of augmentations to increase the number of training examples? Your loss looks very high even at the end of your training run, you might just need more training examples. Also if you scaled this chart to 0 - 13 on the Y axis your sawtooth effect would be less dramatic and might just resemble “normal” fluctuations in loss…

Sorry I don’t have a theory to explain the sawtooth effect, apart from noticing that it gets worse as the LR increases, maybe its the result of large jumps around the loss space without being able to find a good place to settle…


(Pavel Rybnicek) #3

Yes, I do the augmentations - specifically rotation, resize and warp. And yes, the loss is quite high. The saw-tooth effect is present with the smaller amount of classes (2000), too, only not so strong. And yes, it gets worse with higher LR.
I don’t think it’s a normal fluctuation, as the individual teeth really match the epochs.


(Stefano Giomo) #4

Did you considered using a siamese network approach?
Should be useful in a situation like this, with few samples per class.

For reference, take a look at great @radek starter pack:


(Pavel Rybnicek) #5

I’m looking into it, thanks a lot. Looks it could be a solution (although the comparsion to 40 000 vectors can take some time - I need to try).

However I got stuck in the last notebook - cannot create similarity dict, fails with:
in create_similarity_dict(model, dataloader)
29 dists = {}
30 for i, (whale, _) in enumerate(dataloader.items):
—> 31 dists[whale] = torch.pairwise_distance(descs[i], descs).cpu().numpy()
33 return dists

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Any idea what I’m doing wrong? @radek?


(Pavel Rybnicek) #6

I’ll reply myself - it works fine on paperspace, I have probably some incompatible libraries on my local computer.


(Pavel Rybnicek) #7

Anyway, no one has an explanation for the saw-tooth effect? Even though this is probably not the way I’ll use, I’d really like to understand that.