Making the CNN classification model to not recognize faces and instead rely on other clues in the image

Part 1 (2019)
1

Hi all,

I built a gag classification app that tells if an image is a politician on stage or a comedian on stage. The purpose of it isn’t to actually classify politicians and comedians, but more to understand as an analogy to our brain what contextual clues the model uses to make that classification. In that sense, I’m more interested in misclassified images. Maybe those clues are the outfit, the stage set up, or the body language…

However, I’d like to train the model in such a way that it doesn’t memorize faces so that a picture of disheveled Romney in Comedy Cellar has a chance of being labeled comedian, instead of the model immediately recognizing Romney’s face from the training set and labeling him politician. It wouldn’t matter so much if I could control the predicted person to be outside the politicians and comedians used for the training set, but suppose I don’t.

It seems to me one (brute?) way of achieving this is by whiting out the faces in the training and validation sets using a face detection algorithm (or bruter, by doing it manually myself). This will make a portion of the image a white rectangle, but if the distribution of the locations and the sizes of these white rectangles is random, it shouldn’t matter for the prediction?

So my questions are:

  1. Does this approach make sense? Have you guys come across this topic of making CNN not detect the specifics of certain objects?
  2. If so, what face detection package do you recommend? A quick search brings me to Adam Geitgey’s Medium post, but it’s from 2016. Would also be nice to use fastai

Thank you for reading and offering advice!

2

There are several face swapping apps/models you could try instead of erasing face. But removing the face rectangle seems reasonable as well, haven’t tried making a model ignore certain features. With a quick search I only found https://www.quora.com/Can-a-neural-network-be-taught-to-conditionally-ignore-certain-inputs

3

I did this for coloring skin color picture that requires not to recognize the face, otherwise, it will only color the face, and it was pretty successful. My way was you need to make the model has a higher loss when the face becomes a factor in the prediction(I think you understand what I mean). My model did not even need to wipe out the face.

Another approach that I haven’t tried but I think it will work. You can have many similar faces but with a different gag to classifier, so your model will learn to annoy the gag.
However, I am not a professional, so I suggest you list to other people.

4

Thanks for the link! The first comment (“grass” one) suggests what I’m considering–the difficulty is zeroing out the grass in the input images. Attention makes sense to me… however, its goal is to fade out the irrelevant data in the input that is assumed to be the majority of the input? So operationally whiting out the face seems operationally easier for me to do.

I haven’t considered faceswapping. It’s interesting.

5

Elaborate, please! How did you make the model have a higher loss when the face becomes a factor? Doesn’t it require a face detection? Also, in my case every input has a face… so I’m not sure making them all have higher losses will work :frowning: