Part 3 – Background Removal with Robust PCA

Computational Linear Algebra
1

Hey! Thank you so much for the course. I’m learning a lot!

I’ve found a neat way of separating the background and the foreground from the Human Activity video with very little to no noise. When you horizontally stack the frames to form the video matrix, you get those horizontal lines with some noise right?

video_matrix.png720×319 40.7 KB

You can get a perfect background without the people by calculating the mode of each row, without using matrix decomposition at all in the process. This results in a mode vector (which you can reshape into an image). Then for each frame:

  • background = mode vector
  • foreground = frame vector – mode vector

Then reshape to get the images. There will be some noise in the foreground but you can change values close to 0 or close to 255 to 0.

Here is my result in the end:

video_object_detection.png1454×388 215 KB

I’ve also recreated the video using just the foreground frames so you can see people walking in empty space. Although I’m having difficulty generating exporting the frames from Python using MoviePy. Any ideas?

:smile: Samir

Update

Code is here: https://github.com/smoussa/linear-algebra/tree/master

Video is here (download for better quality): https://drive.google.com/file/d/0B3q3JgN59rJhdXR2S1lfWU9XVDQ/view?usp=sharing

15 Likes
2

Hey,

This looks very cool, thanks for posting.

1 Like
3

Clever idea about getting the mode-- I like it!

Matrix decompositions would still be useful in more complicated cases, such as if you have several camera angles.

I also like your idea about recreating the videos just using the foreground. I haven’t tried it, but are you using this approach: https://zulko.github.io/moviepy/getting_started/working_with_matplotlib.html

Feel free to share a link to your code, for using the mode and/or for recreating the video.

3 Likes
4

Very cool!

1 Like
5

Yes!

Mine would not work if the background is moving at all. Matrix decomposition is definitely needed here. I’ve used the mode to get the background and SVD to get the foreground.

Thanks for sharing the link, I’ve used it to create my final video. Apparently the ‘t’ argument is a float, not an integer index to be used to get a frame. It’s working now!

Code is here: https://github.com/smoussa/linear-algebra/tree/master

Video is here (download for better quality): https://drive.google.com/file/d/0B3q3JgN59rJhdXR2S1lfWU9XVDQ/view?usp=sharing

2 Likes
6

Neat! Maybe you can put your new skills to use in this new background removal kaggle competition?

2 Likes
7

That’s great, thanks for sharing Samir!

1 Like
8

Hey. I too tried using Mode and was going to post about it when I found, of course, this had already been done. :slight_smile:

However, I used the background removal to create a mask over the original video to maintain the original colours. Then put Original, Mode, SVD and PCA together in a video: https://youtu.be/f2-dAuSNIfE

9

Thanks for the information. Keep posting such info. Looking for the same info.