New blog post - user classification by mouse movements

(Gleb Esman) #1

Cats and Dogs goes long way :slight_smile:

Thanks again Jeremy!
(Corporate blog post so it had to follow certain restrictions - but in essence this is DL, part 1!)

Architectures for a variable length signal classification problem
(Xinxin) #2

wow, what a nice read! congratulations @gesman !

I’m really impressed at the algorithm to “translate” the time-series mouse movement data into the mouse movement images. Is this specialized high contrast color encoding algorithm publicly available? If not, are there plans to publish this research? Thanks again for sharing this.

(Jeremy Howard) #3

That’s a great post - so happy to see your success @gesman and really interested to learn more about the approach. Translating mouse velocities into color is a nice idea :slight_smile:

(Suresh ) #4

This is absolutely brilliant and shows how thinking out of the box can lead to unique solutions and yet leveraging trained networks like vgg. Thanks for sharing and thereby motivating us.

(Vincent Marron) #5

Incredible work!

I’m curious - when you said: ‘We’ve devised a specialized high contrast color encoding algorithm to represent direction, speed, and acceleration of mouse movements in a consistent way.’ Was this done in order to get the specific problem to fit into the tried-and-true VGG architecture which handles digital images or did you already try calibrating a more general architecture on the data directly and the VGG approach was better? (not trying to poke holes, just generally curious)

Fascinating stuff.

(Thundering Typhoons) #6

Fascinating work. Curious to know more about how you represented direction and speed in the image…

(Gleb Esman) #7

Thank you.
I am pushing our corporate to make content available - and that includes complete source code + color encoding algorithms.
It might take a little while but I hopeful to get permission for complete disclosure by September - for our upcoming Splunk big user conference where I likely will be a speaker.


(Gleb Esman) #8

Thanks Jeremy.

Color encoding was important part of the process because we wanted to make CNN’s job as easy as possible - and essentially convey tiny behavior differences in big color contrasts so that algorithms would see more definitive patterns.


(Gleb Esman) #9

Thanks Vincent,

With mouse movements converted to images - there are not much of “image”-ish data to work with.
Unlike cat and dogs where furry creature occupies the whole frame - the whole mouse movement picture is about 5% of lines and 95% of empty space around.
And then we also needed somehow to convey speed, acceleration, direction etc… within that 5% of actual image data.
So it took some careful thinking and discoveries to devise the way to encode everything in a CNN-friendly way.

My past experience in professional photography helped too :slight_smile:


(Gleb Esman) #10

Hi, thanks.

As I mentioned in reply above - I will be looking to make the rest of approach available as soon as possible.


(Xinxin) #11

[quote=“gesman, post:7, topic:2595, full:true”]
I am pushing our corporate to make content available - and that includes complete source code + color encoding algorithms. [/quote]

Thank you! One quick followup on the actual vgg16 model, did you train a new VGG or is this a transfer learning using the imagenet weight and imagenet mean, std for normalization? Have you tried the vgg16 with batch normalization?

(Gleb Esman) #12

Batch norm was used, correct.
However i didn’t do transfer learning because mouse movement imagery are so much different from VGG16 natural sets.
And then - the training was so fast - so there were no need to go through extra troubles.

(David Gutman) #13

Very cool. Have you tried using a GAN?

(Xinxin) #14

David, have you seen research/implementation of using GANs to improve classification? I’m very curious and would appreciate it if you can point me to the research/examples. Thank you!

(Suresh ) #15

Hmm, for some reason I thought that you did use transfer learning… don’t know how I got that idea from the write-up. Intuitively, this makes sense as he images are different, but I thought that the first two layers would have still helped from VGG.

One of the papers I read ( mentioned that first few layers are very generic and can also be seen from the visualizations. If It is not too much trouble, can you also try it with freezing first two layers or so and see if it improves the results (or not freezing any layers, but using these initial weights from the trained network)?