# Flipping and image regression with keypoints

Hi,

I want to augment my data with flipping. But I’ve realized that can have some issues when the keypoints have semantic meaning or an order to them. For instance, look at this picture of a cat. All the keypoints for ears are in the same direction: [left ear, right ear]. But after flipping, they are now opposite. So when the model makes a prediction, it will get a big error even though it was correct!

To make it even more clear:
I have two points for the ears: [(100, 150), (200, 150)]
Flipping the image around x=150 gives the Y: [(200, 150), (100, 150)]
Predictions from model: [(98, 149), (201, 151)]
Will give a huge error, since the model predicts left to right, but the points now are in the other direction.

How is this best solved? Fixing my MSELoss to first sort the points from Y-truth back to a left-to-right order? Or do it in some augmentation step to make sure they’re all in the same order?

I actually think this is the “bug” mentioned here, with flip and tensorpoints making it only predict in the middle: Flip with TensorPoint causes a bug · Issue #100 · fastai/fastai2 · GitHub
Of course it will, when the flips gives it a huge error so it regresses towards the middle.

Also might be the issue @muellerzr is seeing in his GH issue here? Major bug regarding Augmentation and TensorPoint · Issue #2628 · fastai/fastai · GitHub

Any thoughts on this?

How can I handle flipping augmentation with multiple keypoints when the loss is calculated looking at the order of the points?

The obvious answer is to fix it, modify your flipping augmentation to relabel the points as well. If the goal is to predict specifically the location of the left ear and the right ear in an ordered fashion this is the only real way.

For more general problems you don’t usually expect spatial ordering to matter. Do a minimum cost assignment between predicted points and possible targets, and then backprop only the best assignment.