This is very nice! How did you deploy your fastai model on Flutter? You should write a tutorial/blog post on that!
2 Likes
Thanks @ilovescience!
I basically used the strategy that Jeremy suggested in the course. The fastai model lives on the server and is used at inference time only.
I will try to get the app in the Apple store as well. Perhaps after that I might write a tutorial / blog post.
5 Likes
I used fastai to train a model than can recognize Sign Language, then went ahead and created an app using opencv that runs inference on the video from my webcam.
Here are the relevant links:
Training the Model: https://jimmiemunyi.github.io/blog/tutorial/2021/01/20/Sign-Language-Classification-with-Deep-Learning.html
Creating the opencv inference app: https://jimmiemunyi.github.io/blog/tutorial/2021/01/21/Sign-Language-Inference-with-WebCam.html
Github: https://github.com/jimmiemunyi/Sign-Language-App
Youtube Output Video: https://youtu.be/-nggi8EwfOA
8 Likes
Toon-Me, A fun project to Toon portrait pictures. Please have a look at it.
8 Likes
FlightVision
A realtime solution to track dart targets with fastai and unets, using segmentations layers rather than classical regression.
x, y Coordinate Tracking
- Input frame passes through
unetmodel for segmentation mask inference. -
OpenCVblob detection identifies centroid location. -
OpenCVoverlay is applied to show the results
Full write up on Github FlightVision
Background
Flight Club is the name of a franchise in London that modernised the classic pub sport. It’s success comes from the gamified automated scoring system. It has a 3D vision system comprised of live cameras, and also includes a well-polished interface which runs on a screen above the board. This opens the traditional game up for improved entertainment with the inclusion of minigames like team elimination, or a snakes-and-ladders adaption – where the target of your dart throw is how far forward you move on the board.
As of 2020 it has expanded into 3 locations in the UK and one in Chicago, USA.
How it works
Powered by a “highly sophisticated 3D vision system”, the software brute-forces the solution using calibrated cameras positioned in a frame above the dart board.
A normal dart impact on the board triggers Flight Club Darts’ specially developed 3D fitting algorithms to identify, recognise and measure the precise position, pose and score of the dart to within a fraction of a millimetre. The software manipulates three virtual darts through millions of different orientations and angles until it finds what matches where the dart landed on the board. Using multiple cameras reduces obscuration effects.
A deep learning approach was attempted with a challenge to PhD students as part of a country-wide university competition around 2014. At that time no solution was found, and so more conventional computer vision techniques were used.
Bootleg Flightclub using fastai v2
Take your own pictures, find the x,y coordinates of the dart point using a unet segmentation layer.
Development
1. What Didn’t Work
In lesson 3 of the fastai deep learning course, it explains how regression can be used within computer vision. Unlike classification tasks, which are used to make a categorical predicions (i.e. cat or dog), regression is used to find a continuous numerical values. This fits for our example, because we are trying to determine the x, y pixel locations of the dart point.
I spent a fair amount of time trying to get this regression based cnn_learner working – but it simply failed to converge during training. I was surprised that the model couldn’t accurately predict the validation data given the simplicity of some of the samples. For example, the third image down was a picture of a dart, by itself, on a plain background – and it still couldn’t make a reasonable prediction.
I knew something must be wrong either with the training process and architecture, or there was a limitation with the dataset I had created. In order to isolate the problem I generated a new dataset.
2. Synthetic Dataset with Blender
Using blender I simulated a simplified version of the problem. With this I could determine whether it was my training database that was wrong, or whether it was my ML/Dl approach that was wrong. I downloaded a 3d model from grabcad and created a python script to automate rendering.
Blender allows for scripting with python. It is quick to learning the scripting API, because when you create an event with the UI the equivalent python command appears in the console, which you can then paste directly into a script. Then with python you can add a loop around it – specify the inputs/outputs, and the whole process is automated. My blender script randomised the rotation, zoom, light position, and centroid x, y location. With these in place I generated 15,000 renders in about half an hour.
Results
What I found was that even with this much simplified computer vision problem, the cnn_learner with regression head still couldn’t converge (and overfit) the training dataset. With that I was convinced the problem was with the architecture, so I moved on to try something else.
3. Learning with Unets
Unets are a segmentation method, where the output of the model is a per-pixel classification mask of the original image. These are commonly used in examples of self-driving car solutions online. My intuition here was that instead of using a multi-classification class of scenery objects (like tree, road, traffic_light etc), the target segmentation mask could simply be 0 or 1, where 0 is a pixel where the centroid isn’t, and 1 is a pixel where the centroid is.
17 Likes
And here it is on the App Store for the iOS peeps: https://apps.apple.com/de/app/yogapose/id1549738502?l=en
4 Likes
I have always had a problem in recognizing big cats, so I thought of building a classifier using FastAI as part of my learning from the FastAI book. ( I was surprised to see @dway8 has worked on a similar project)
An image classifier that predicts members of the genus Panthera family ( Classes include Cheetah, Cougar, Jaguar, Leopard, Lion, Snow leopard, Tiger ). Trained using FastAI v2 API and deployed on StreamLit.
Github Repo : https://github.com/dnaveenr/big_cat_classifier
Live Demo : https://share.streamlit.io/dnaveenr/big_cat_classifier/main/src/streamlit_deploy.py
Thank you.
4 Likes
That looks great - how easy was it to use flutter to make the app? Any good resources you could lnk to?
Cheers
Clive
1 Like
Glad you like it!
It was quite easy to use flutter. I liked the flutter tutorials by the net ninja on YouTube.
2 Likes
PredictionDynamics
Hi all,
I’d like to share with you something I have been using lately that I find pretty useful. It’s a callback that allows you to visualize predictions during training. The plot is actually updated with every epoch. This is the type of output you get:
The main idea is based on a blog post by Andrej Karpathy I read some time ago. One of the things he recommended was to:
visualize prediction dynamics: I like to visualize model predictions on a fixed test batch during the course of training. The “dynamics” of how these predictions move will give you incredibly good intuition for how the training progresses. Many times it is possible to feel the network “struggle” to fit your data if it wiggles too much in some way, revealing instabilities. Very low or very high learning rates are also easily noticeable in the amount of jitter." A. Karpathy
It’s pretty interesting to see how the model is trained, and what the impact of the model, loss function, different initialization schemes, on which classes does the model struggle, etc.
A notebook with a few examples is available in this gist.
As you’ll see it’s very easy to use. The only thing you need to do is upload the callback code and add PredictionDynamics() to your callbacks in your learner.
22 Likes
Great work, can’t wait to test this out!
Very very nice!
This is one of the reasons I really love the fastai library-- It’s just so easy to create a small really useful tool that just works with existing fastai code.
Just a small suggestion for improvement would be to stick the notebook into an nbdev repo and just add a few tags to turn it into a python package so people can just pip install it into their notebooks!
Again, thanks for this!
Butch
6 Likes
I tried a little project and wrote a tutorial about it. I wanted to replicate the “blur background” feature from apps like Zoom. I began with creating my own dataset (only about 40 images), trained a UNet with fastai, and used OpenCV to generate the images with blurred background.
It works quite well, but painfully slow at around 0.5 frames / second 
I’m not sure how to improve that. Maybe it could be faster with more efficient data processing, but maybe the UNet itself is just too heavy…
14 Likes
This is super cool!! Thank you very much for sharing. I second the suggestion to make a pypi package with nbdev, this would make integration to existing notebooks easier.
1 Like
Fantastic and a Fabulous tuttorial.
mrfabulous1
1 Like
Hi, I’ve just found your project.
I need to create a similar application for a research project. Many thanks for sharing.
1 Like
I published an older project which I recently picked up again. The task is detecting buildings in satellite images from a difficult dataset.
It’s a long post, covering data processing, visualization, undersampling, training, and presenting the results.
I didn’t write it as a tutorial, but maybe it’s helpful for someone getting started with segmentation tasks!
This was originally a challenge for an internship role (which didn’t work out in the end). The goal was to build a minimal solution, not over-optimizing a metric. It was more about how I’d approach the problem than the actual result. I’d be happy to hear some feedback from you: is this kind of presentation appropriate? Would you have done it differently?
6 Likes
I’ve written a little sublibrary called fastai_minima. As the name implies this is the minimal fastai needed to get the Callback system, Learner, and the param groups working with raw Pytorch. As a result the only real requirements are fastcore and fastprogress. Hopefully this can help some folks who are working with raw torch, or want a minimal version of the library to work off of
This project will forever be frozen on fastai v 2.2.6’s Learner/Callback/Optimizer code, unless some drastic and major changes arise
11 Likes
Hi Mueller, can you help on this:
I am trying to implement SRGAN in fastai, but I am getting complete black screen as output.
I do not know what is going wrong.I am able to start training, but output is always a black screen.
Also I am trying to include CRAFT for text regions localizations, so that model can focus on the text regions,
This is the notebook…
But I am not getting good results…