Dealing with Missing KeyPoints - One Solution *kinda*

fastai
1

Hey everyone,

I have a problem where I need to know which keypoints were dropped for my model. One thing fastai will defaulty do is drop our keypoints and in investigating I found an imperfect way to get back your points.

Assumptions:

  • We are pre-generating our transformed images first, then training
  • The order of points matter as we want some point back for every class (and know which classes got zeroed out)
  • We want to not drop any points.

A notebook for a quick view is available here

We transform our keypoints twice. The order doesn’t matter, but we need to be sure not to transform our images.

For the first set of points, I will apply do_resolve=False and remove_out=True. We need do_resolve in order to keep our points linked:

img2 = img.apply_tfms(trans)
pts2 = ip.apply_tfms(trans, do_resolve=False, remove_out=True)
img2.show(y=pts2, figsize=(12,12))

The result is the following image:

StyImUp.png529×687

As we can see, we only have 5 points here. If we look at just the image points, I cannot tell who they belong to, as they look as such:

tensor([[ 0.4780, -0.5782],
        [ 0.5314, -0.3515],
        [ 0.9103, -0.2609],
        [ 0.5168,  0.0371],
        [ 0.1088,  0.4839]])

However if we apply remove_out=False, now this is our image:

NwOB27S.png475×683

Now we can see all the points! And the list, which is in order of how the csv is labeled with my points:

tensor([[-1.0230, -1.4979],
        [-1.0230, -1.4979],
        [-1.0230, -1.4979],
        [-1.0230, -1.4979],
        [-1.0230, -1.4979],
        [ 0.4780, -0.5782],
        [ 0.5314, -0.3515],
        [ 0.9103, -0.2609],
        [ 0.5168,  0.0371],
        [ 1.0463,  0.0111],
        [ 0.1088,  0.4839],
        [ 1.1726, -0.9021],
        [ 1.2503, -0.7013],
        [-1.0230, -1.4979],
        [ 1.2551,  0.1277],
        [-1.0230, -1.4979],
        [ 1.6534, -0.4487]])

Now I can easily go through and grade who is there and who is not really if I wanted to keep track of it.

I’m working on dealing with missing components next, but for now this is what I have. Let me know any thoughts on the topic, or what others have done :slight_smile:
Thanks!

Edit: Looks like if we just pass in the second, the databunch will automatically still not use the rest? Show_batch() did not show the other points outside the image