Siamese Networks

Part 1 (2019)
1

I thought it would be good to start a fresh thread about this particular network type. There are several posts in this forum about it but I thought it would be good to discuss approaches on how to use fastai to help create a dataset and module to get started with.

These resources were useful but if you have any others please share them and I’ll add them.

Resources

Description Type
Coursera Face Recognition Andrew Ng Video
One Shot Learning with Siamese Networks Medium Article
Object Tracking Paper
Siamese Neural Networks for One-shot Image Recognition Paper

Example

You’ll finds some classes to help create a siamese network.

In the notebook is how to:

  • Create a Siamese Dataset from an already labelled dataset.
  • Siamese Network Module

Features:

  • Currently it is generating pairs of items before training because I thought it would be to easier to evaluate bad data. Pairs could be generated on the fly?
  • The Siamese Network Module uses resnet34 by default as an encoder but could use any architecture as an encoder
  • You can choose classes to make the validation set. This validation set is hidden from training for better evaluation of the network.
  • Embedding Visualisation
  • I’ve used Hinge Loss but there are many other choices for loss functions. I’ll add some more and see what they do

Questions

  • Should we be freezing the encoder?
  • How close should our results be to 0 for us to accept them as a good guess
  • Do Siamese networks with heads improve results
  • The number of pairs we can train on is very large even for a small dataset like this. How should we allow the user to configure the usage of it while preserving good defaults (having an even balance of pairs for each class)
9 Likes
2

After experimenting with a large dataset, I noticed that the current pair generation was too slow. I’ve begun implementing a much faster approach.

The idea is to generate a random diagonal matrix representing the all the unique pairings which can be stored as indices which will later be translated into actual files.

1 Like
3

V2 coming out within a month and looks way cleaner than anything I had in mind:

I will hold off on finishign this for now!

2 Likes
4

The Humpback Whale Identification comp was very helpful to understand the approaches of Siamese networks and protonets (few shot learning).

There is this fastai v0.7 kernel that I highly recommend. The code is easy to understand with a lot of comments.

Other kernels like the following were very informative too:

  • this pytorch code: which used protonet (few shot learning), an interesting middle approach between one-shot learning like Siamese and classification). This is a very nice explanation on what it is and why it is a major trend in metric learning approaches.

  • this keras kernel: which was unbeatable due to the use of LAP in selecting the image pairs. LAP is Linear Assignment Problem which is a special type of linear programming problem which deals with the allocation of the various resources to the various activities on one to one basis. It does it in such a way that the cost or time involved in the process is minimum and profit or sale is maximum. It was very compute heavy approach to choose image pairs from the ~20K images, that it took around 50-70% of the total 3 days training time for CPU compute and the rest was training with GPU. But no other approaches could beat it.

If you are serious about Siamese and metric learning in general, I highly recommend to try running the above three kernels and play with them until you understand the concepts.

The way on how you are selecting the image pairs and how the code is increasing the difficulty of the negative image pairs were major keys in 20-30% variation in accuracy. So it is not just the Siamese network whether it works fine, but this essential minor details are crucial to make it on par or better performance than other classification approaches.

5 Likes
5

Thank you this is really helpful.

I will investigate these notebooks and try to implement a small sub library to use the techniques with fastai

2 Likes
6

A great article on n-shot learning

2 Likes
7

Here is Jeremy showing how to create Siamese Network with the new V2 API

Youtube VIdeo

1 Like
8

Hello,
Is there a working example of Siamese Network for images with latest Fastai version?

2 Likes
9

@baz . Thanks for this post. Do you have the answer for your question that how close is the distance to be consider that similar or different ? I know that it should be some where from 0 to margin not sure how to choose properly :smiley:

10

Currently v2 is still under development and will make creating a Siamese pipeline much easier. Check out video just above that I posted that shows Jeremy explaining how to use the new api to do this. I will get to creating an example of siamese networks at some point and I’ll post it in here.

3 Likes
11

This is a great question. I did some exploration manually to see but didn’t see any glaringly obvious answer to it. I would re-run others experiments and see what answers they got and see how large they’re loss values got.

1 Like
12

Thanks for sharing this video! I managed to follow along and create a siamese pipeline, but don’t really know where to go from here as far as turning the pipeline into a databunch.

13

I will try and implement something when the v2 High level api is finalised :slight_smile:

2 Likes
14

Hi so I’ve been trying to get a siamese network working with fastai2.
I tried following the tutorial notebook on pets to get a siamese dataset but failing.

tfms = [[sp, OpenAndResize], [labeller, Categorize]]
dsets = Datasets(items, tfms, verbose=True)
t = dsets[0]
print(type(t[0]),type(t[1]))
x,y = dsets.decode(t)
print(x.shape,y)
dsets.show(t);
15

Hey,
Can you give more details into which line of code is failing and what’s the error it is throwing?
Thanks

16
17

I have been trying to implement Siamese Network on different dataset rather than PETS dataset as showed in tutorial. I have tried to implement on MNIST dataset, when i looked into the show_batch().
Their is a clear problem that they are showing disimilar images as similar. I dont know where is the problem is?

Screenshot_2020-09-25 Google Colaboratory
Screenshot_2020-09-25 Google Colaboratory1256×556 60.9 KB

colab notebook: - https://colab.research.google.com/gist/HarshaSatyavardhan/3b2c9ffa15a148d787e2ccb4f2cd61e4/siamese.ipynb

@muellerzr @radek @baz @hwasiti