Recognizing sprites in a 2d game

Part 1 (2017)
#1

Hey, So when I started thes course I had a project goal in mind. I’m trying to write a bot that plays video games. The actual control code will be hand written because the goal of the project is to iterate on strategies and stuff (not to do a full reinforcement learning game playing bot) but I want to use deep learning for the up front scene recognition.
Given a frame of the game (or more likely the last few frames of the game) I want to know what sprites are on screen, where they are and what direction and speed they are traveling at.

From what I’ve learned so far I think solving the problem is going to look like this.

  • Take some existing object recognition model (fast R-CNN looks like it might be a starting point) and finetune it to recognize the sprites in my game.

Is this the right approach?

Also once I have a model trained on my data set I really need it to execute FAST. (like 60fps if possible) Can you take a model trained in keras and “export” the weights and the architecture so that I could write some c code that just evaluates the model on new data as fast as possible? Is that a typical thing to do?

#2

https://blog.keras.io/keras-as-a-simplified-interface-to-tensorflow-tutorial.html

You can grab the TensorFlow graph and export it (if you are using that backend). That should be pretty good for speed depending on your hardware and model.

#3

Cool, that’s good to know. Thanks

Does anybody know of any models that are good at recognizing and segmenting a 2d scene? Picture a classic 2d game like Mario. Are there any models that are pretrained to be good at detecting the sprites in the scene? That output “This is Marioand these are question mark blocks” etc. I found something that seems promising in that it runs really quickly which is yolo but that is an object recognition algorithm and I think what I’m looking for is more of a scene segmentation problem.

Any guidance on where to look for such a model would be great! Thanks!