Siamese Networks

(Harry Coultas Blum) #1

I thought it would be good to start a fresh thread about this particular network type. There are several posts in this forum about it but I thought it would be good to discuss approaches on how to use fastai to help create a dataset and module to get started with.

These resources were useful but if you have any others please share them and I’ll add them.

Resources

Description Type
Coursera Face Recognition Andrew Ng Video
One Shot Learning with Siamese Networks Medium Article
Object Tracking Paper
Siamese Neural Networks for One-shot Image Recognition Paper

Example

You’ll finds some classes to help create a siamese network.

In the notebook is how to:

  • Create a Siamese Dataset from an already labelled dataset.
  • Siamese Network Module

Features:

  • Currently it is generating pairs of items before training because I thought it would be to easier to evaluate bad data. Pairs could be generated on the fly?
  • The Siamese Network Module uses resnet34 by default as an encoder but could use any architecture as an encoder
  • You can choose classes to make the validation set. This validation set is hidden from training for better evaluation of the network.
  • Embedding Visualisation
  • I’ve used Hinge Loss but there are many other choices for loss functions. I’ll add some more and see what they do

Questions

  • Should we be freezing the encoder?
  • How close should our results be to 0 for us to accept them as a good guess
  • Do Siamese networks with heads improve results
  • The number of pairs we can train on is very large even for a small dataset like this. How should we allow the user to configure the usage of it while preserving good defaults (having an even balance of pairs for each class)
8 Likes

Siamese Network using custom classes
Deep Learning with Audio Thread
Model for queries response matching
Siamese Network DataBunch
How to build a Siamese Network with fastai-style code
Help needed in DataBunch construction
(Harry Coultas Blum) #2

After experimenting with a large dataset, I noticed that the current pair generation was too slow. I’ve begun implementing a much faster approach.

The idea is to generate a random diagonal matrix representing the all the unique pairings which can be stored as indices which will later be translated into actual files.

1 Like

(Harry Coultas Blum) #3

V2 coming out within a month and looks way cleaner than anything I had in mind:

I will hold off on finishign this for now!

2 Likes

(Haider Alwasiti) #4

The Humpback Whale Identification comp was very helpful to understand the approaches of Siamese networks and protonets (few shot learning).

There is this fastai v0.7 kernel that I highly recommend. The code is easy to understand with a lot of comments.

Other kernels like the following were very informative too:

  • this pytorch code: which used protonet (few shot learning), an interesting middle approach between one-shot learning like Siamese and classification). This is a very nice explanation on what it is and why it is a major trend in metric learning approaches.

  • this keras kernel: which was unbeatable due to the use of LAP in selecting the image pairs. LAP is Linear Assignment Problem which is a special type of linear programming problem which deals with the allocation of the various resources to the various activities on one to one basis. It does it in such a way that the cost or time involved in the process is minimum and profit or sale is maximum. It was very compute heavy approach to choose image pairs from the ~20K images, that it took around 50-70% of the total 3 days training time for CPU compute and the rest was training with GPU. But no other approaches could beat it.

If you are serious about Siamese and metric learning in general, I highly recommend to try running the above three kernels and play with them until you understand the concepts.

The way on how you are selecting the image pairs and how the code is increasing the difficulty of the negative image pairs were major keys in 20-30% variation in accuracy. So it is not just the Siamese network whether it works fine, but this essential minor details are crucial to make it on par or better performance than other classification approaches.

2 Likes

(Harry Coultas Blum) #5

Thank you this is really helpful.

I will investigate these notebooks and try to implement a small sub library to use the techniques with fastai

2 Likes

(Harry Coultas Blum) #6

A great article on n-shot learning

2 Likes

(Harry Coultas Blum) #7

Here is Jeremy showing how to create Siamese Network with the new V2 API

Youtube VIdeo

1 Like