This tutorial series covers training an object detector using the IceVision library and implementing the trained model in a Unity game engine project using OpenVINO.
The tutorial uses a downscaled subsample of HaGRID (HAnd Gesture Recognition Image Dataset). HaGRID contains 552,992 annotated sample images for 18 distinct hand gestures and an additional no_gesture class to account for idle hands.
The original dataset is a massive 716GB, so I made smaller versions available on Kaggle for those following the tutorial. Both are small enough to fit in the free GPU tiers of Google Colab and Kaggle Notebooks. Although, you will probably need to lower the image and batch size to run the larger of the two versions on the free tiers.
Fastai to Unity Beginner Tutorial: This tutorial covers how to train an image classifier using the fastai library and implement it in a Unity game engine project using the Barracuda inference library.
I had someone with an AMD Ryzen CPU test the OpenVINO project, and it seems like OpenVINO only works for Intel hardware.
Since not everyone has an Intel CPU, and discrete Intel GPUs are not yet widely available, I made a follow-up tutorial showing how to use ONNX Runtime and DirectML instead of OpenVINO.
If you don’t know, DirectML is a hardware-accelerated DirectX 12 library for machine learning on Windows. It should work with basically any modern GPU.
This tutorial uses the ONNX models already generated in the training code from the original tutorial.
Blog Posts
Part1: covers creating a dynamic link library (DLL) file in Visual Studio to perform inference with ONNX Runtime and DirectML
Part 2: covers performing object detection in a Unity project with ONNX Runtime and DirectML
I have used unity a little bit a long time ago. I wonder you could make a video to recreate the demo of hand gesture recognition from the very first step of downloading unity to show us this is easily doable.
This will help more people to get started with both unity and Fastai.
Hi Daniel,
I have only done written content up to this point, so I would need to investigate the best approach for making a video tutorial. At the very least, the microphone integrated into my webcam is not ideal. I’ll look into it, though.