I Made an AI That Learns to Play Fallguys With FastAI

Part 1 (2019)
1

Thank you so much Jeremy and Rachel for creating some of the best educational content out there. They made it possible for me to learn enough to make this video possible!

12 Likes
2

Hey there, this is amazing! do you have a blog post on how you built the fast.ai part that learned to play the game? I know you have a little blurb in the video, but I’m looking for a bit more technical detail (just out of curiosity.)

3 Likes
3

This kind of long effort videos are worth making. Seeing someone putting this much effort is really inspiring. Thanks for creating and if you can open source the code, it will be useful for everyone

3 Likes
4

That’s great! I wonder how to send controls to the game and get feedback? Do you record the screen and also automate the controller?

3 Likes
5

Hey, I’m glad you liked it! Unfortunately the video is the closes thing I have to a blog post about it. I just posted the code in the video description including the google-colab file so you can follow along and see what I did.

2 Likes
6

Thank you! I just posted the code in the video description

1 Like
7

Thanks! The flow is like this: take screenshot, feed into pickle, pickle returns keyboard input, press keyboard input. I just posted the code so you can check it out if you want.

1 Like
8

Ah, ok, so you use native WinAPI to access window’s context. Makes sense, then it should be straightforward to interact with the game. Looks good!

2 Likes
9

Interesting approach, did you and Jacob Lower from Clarity Coders work on it together? I think I can use this approach for “driving around” my esp32 based car … thanks for Posting!

2 Likes
10

Nope, I just forked his code and went from there. And yeah, this is very similar to a self driving car I made a while back. Of course it’s not perfect, if I were to improve on this project, I might:

  • make it an RNN to recognize changing scenery
  • augment the data by reflecting every image by the y-axis and reversing the left-right classifications to essentially double the training data
  • get more varied training data to get the ai more used to different situations. Maybe of people playing other than me
  • if there was a way to lock the camera, that would help immensely with consistency in certain levels, but I don’t think this would help a car driving AI
1 Like