Part 3 – Background Removal with Robust PCA


(Samir Moussa) #1

Hey! Thank you so much for the course. I’m learning a lot!

I’ve found a neat way of separating the background and the foreground from the Human Activity video with very little to no noise. When you horizontally stack the frames to form the video matrix, you get those horizontal lines with some noise right?

You can get a perfect background without the people by calculating the mode of each row, without using matrix decomposition at all in the process. This results in a mode vector (which you can reshape into an image). Then for each frame:

  • background = mode vector
  • foreground = frame vector – mode vector

Then reshape to get the images. There will be some noise in the foreground but you can change values close to 0 or close to 255 to 0.

Here is my result in the end:

I’ve also recreated the video using just the foreground frames so you can see people walking in empty space. Although I’m having difficulty generating exporting the frames from Python using MoviePy. Any ideas?

:smile: Samir

Update

Code is here: https://github.com/smoussa/linear-algebra/tree/master

Video is here (download for better quality): https://drive.google.com/file/d/0B3q3JgN59rJhdXR2S1lfWU9XVDQ/view?usp=sharing


(Gertjan Brouwer) #2

Hey,

This looks very cool, thanks for posting.


(Rachel Thomas) #3

Clever idea about getting the mode-- I like it!

Matrix decompositions would still be useful in more complicated cases, such as if you have several camera angles.

I also like your idea about recreating the videos just using the foreground. I haven’t tried it, but are you using this approach: https://zulko.github.io/moviepy/getting_started/working_with_matplotlib.html

Feel free to share a link to your code, for using the mode and/or for recreating the video.


(alex) #4

Very cool!


(Samir Moussa) #5

Yes!

Mine would not work if the background is moving at all. Matrix decomposition is definitely needed here. I’ve used the mode to get the background and SVD to get the foreground.

Thanks for sharing the link, I’ve used it to create my final video. Apparently the ‘t’ argument is a float, not an integer index to be used to get a frame. It’s working now!

Code is here: https://github.com/smoussa/linear-algebra/tree/master

Video is here (download for better quality): https://drive.google.com/file/d/0B3q3JgN59rJhdXR2S1lfWU9XVDQ/view?usp=sharing


(Brendan Fortuner) #6

Neat! Maybe you can put your new skills to use in this new background removal kaggle competition?


(Rachel Thomas) #7

That’s great, thanks for sharing Samir!