Running, Deploying Fastai successes: Links, Tutorials, Stories

[Brad_S]

I’m interested in learning ways fastai models can be deployed and run on various (edge) systems. Maybe this will be useful for many and turn into a wiki. I have properly done NONE of these, but I’m starting by collating under headings: forum posts (may link to repos and blog posts) and external links

Why at the Edge?
Your smartspeaker sits there using relatively little power running a ‘cheap’ microcontroller, locally inferring if the wake work is spoken (Alexa, Siri, Hey Google, Jeremy… if you’ve customized it). Your phone can help with text entry as you type, not a few seconds later. Low latency, lower data transfer (lower power) and increased privacy are just some of the reasons why neural net at the edge is of interest.

---

Topics

1. Fastai → Pytorch → ONNX
2. Fastai → Browser
3. Fastai → IOS
4. Fastai → Android
5. Fastai → Jetsons
6. Fastai → Microcontrollers
7. Fastai → ONNX elsewhere

Pytorch to ONNX

Fastai is a library built on Pytorch that contains lots of framework, tips and tricks for quickly and flexibly building and training models. The notebooks regularly run predictions or batch inference, but this is not the end environment where many models intend to be deployed.

What is ONNX? Open Neural Network Exchange
Website | GitHub | Wikipedia

ONNX is an open format built to represent machine learning models. ONNX defines a common set of operators - the building blocks of machine learning and deep learning models - and a common file format to enable AI developers to use models with a variety of frameworks, tools, runtimes, and compilers

In other words, ONNX is a shareable format for neural network models usable by

1. Frameworks e.g. Pytorch, Tensorflow can export models to ONNX
2. Hardware / developer environments e.g. mobile phones (android, iphone), TPUs, Jetsons etc
3. Languages e.g. Python, Java, C, C++, C# - APIs can call ONNX runtimes / API even if not every framework is supported directly

fastai tips/tricks/issues?

• Its important to realise ONNX only shares the model. If your pipeline relies on input transformations, these transformations will need to be ported to the deployed environment. e.g. this forum post
• is flatten() still an issue? Developer chat - #996 by davidpfahler

Pytorch to browser

Is tensorflow.js worth mentioning?
Forum Posts

How to run a fastai model in the browser

As a data protection lawyer, I am very aware of the privacy implications when it comes to model deployment. That’s what got me thinking: Could you train a model in fastai and run it in the browser? Turns out you can! All the pieces were already in place, but it was somewhat difficult to connect that dots. So I open-sourced my pet project and wrote a short blog post documenting the findings. If you find any mistakes that I made, please let me know! I want to thank everybody in the fastai commu…

Deploying Deep Learning Models On Web And Mobile (Apps)

By: Nidhin Pattaniyil (@npatta01) and Reshama Shaikh (@reshama) BLOG Deploying Deep Learning Models On Web And Mobile ABOUT Deploy deep learning models, with a production example for both web and mobile. Our example uses fastai. But, an output model file from any other deep learning library such as Keras or TensorFlow, etc. will work. WEB APP Heroku web app: food-img-classifier MOBILE APP Video of mobile app: 2-minute demo of Food classifier using fastai iOS Apple store app: Food Cl…

Pytorch to IOS

Forum Posts

Running a fastai model in iOS using CoreML

I recently wrote an article about running a fastai model in the browser. Today, I am sharing my next end-to-end example that trains a fastai vision classifier model, exports to CoreML (via ONNX) and build a react-native app around it. I hope you enjoy! See my article about it here. The code and notebook can be found in this repo: https://github.com/davidpfahler/react-native-ml-app I’m looking forward to your feedback and suggestions.

Deploying Deep Learning Models On Web And Mobile (Apps)

By: Nidhin Pattaniyil (@npatta01) and Reshama Shaikh (@reshama) BLOG Deploying Deep Learning Models On Web And Mobile ABOUT Deploy deep learning models, with a production example for both web and mobile. Our example uses fastai. But, an output model file from any other deep learning library such as Keras or TensorFlow, etc. will work. WEB APP Heroku web app: food-img-classifier MOBILE APP Video of mobile app: 2-minute demo of Food classifier using fastai iOS Apple store app: Food Cl…

Pytorch to Android

Forum Posts

Deploying Deep Learning Models On Web And Mobile (Apps)

By: Nidhin Pattaniyil (@npatta01) and Reshama Shaikh (@reshama) BLOG Deploying Deep Learning Models On Web And Mobile ABOUT Deploy deep learning models, with a production example for both web and mobile. Our example uses fastai. But, an output model file from any other deep learning library such as Keras or TensorFlow, etc. will work. WEB APP Heroku web app: food-img-classifier MOBILE APP Video of mobile app: 2-minute demo of Food classifier using fastai iOS Apple store app: Food Cl…

Pytorch to Jetsons

One thing to note with the Jetson is the Tx2 and Nano (and presumably the others) run on a AArch64 architecture which is not supported by Conda. So you want to be familiar with virtualenv and linux to use them
nano getting started | course
nvidia deep learning frameworks
Forum Posts
I havent found a recent forum post - don’t follow old ones!

Pytorch to Microcontrollers

It feels naughty… but TF lite (TensorFlow lite) is the only framework I know of.
But there’s a podcast of this being done on an Integer only ARM M0
Forum Posts

Fastai Model conversion to ONNX, TF, CoreML, Caffe2, TFlite

Here I would like to share a simple notebook as a walkthrough for model conversion. Some notes: TF to TFlite is not very mature when coming from PyTorch since sometimes operations can’t be expressed as native TF ops or TF lite only supports NHWC data format. Fix is to just add a permute() to beginning of your model for converting NHWC to NCHW which can be used by the actual PyTorch model. Other than TFLite - ONNX, TF, CoreML, Caffe2 seems to work fine. Probably swift would cross out all the a…

Pytorch to ONNX elsewhere

ONNX Runtime | Home | Blog

Run ONNX models with Amazon Elastic Inference | Amazon Web Services

Forum Posts

[Brad_S]

community - please add suggestions in the comments. As indicated, happy and hoping this could be a wiki for you to edit

[madhavajay]

Best resource for Custom Edge Inference in iOS would be:

Core ML Survival Guide

Tips on using Pytorch -> ONNX -> CoreML as well as creating custom converters, plus optimisations on CoreML / iOS side.

[ozgur (Özgür Şahin)]

I created a Google Colab sample for converting CycleGan generator to CoreML via Onnx.

[mhanan (hanan)]

hey,
i needed to train document image classification at work. so i used the fastai library to train the model, then export it using “jit”. (i didn’t install the fastai library in our deploy container cause it’s really heavy and have too much dependency).
for some reason there were a gap between the results i got when i loaded the image’s using fastai method’s, and when i did it manually.
after some investigation i found that the gap was between the “resize” function i used (from PIL library) and the way fastai method (using tensors).

[Brad_S]

thanks for this. I recall now that Jeremy at some point said Jit was unusable, but I did see him on Twitter later say he’d given it another go and got speed benefits. I’d forgotten

To include it as a ‘help to deploy’ information, what would we need here?
Torch Jit
torch.jit can be used to turn Pytorch/Fastai/Python code into serialized (faster) code that can be called away from python and other dependencies (e.g. in a C++ program)
… ???

[jayeshsaita]

I used Intel OpenVINO for faster inference of my mobilenetv2 on Intel CPU. Reduced my inference time from 200 ms (using naive learn.predict) to 25 ms! Best part, my model was still in FP32.

[muellerzr (Zachary Mueller)]

@Brad_S

Torch Jit
torch.jit can be used to turn Pytorch/Fastai/Python code into serialized (faster) code that can be called away from python and other dependencies (e.g. in a C++ program)

Look at how the fastai2 Mish activation function is done probably for a hint at an example. (It uses Jit)

[cudawarped]

For running inference on your GPU from C++ applications I would suggest using OpenCV once you have exported your ONNX model from Pytorch (OpenCV 4.2.0 has CUDA DNN backend).

[Brad_S]

turns out the fastbook draft chapter 2 talks about basic deployment using voila from jupyter and binder for free hosting

[Brad_S]

I’m pretty sure we can do this fastai esque in less than 3 years without all having phds

https://www.eetimes.eu/ide-brings-ai-training-to-mcus-for-the-first-time/

[rsomani95 (Rahul Somani)]

I had the same issue. Use torchvision.transforms.functional.resize, that gave me identical predictions as when using fastai’s resize

[rsomani95 (Rahul Somani)]

I just discovered that torchvision's resize is the same as PIL.resize((size), resample=2).

resample=2 resizes using “bilinear” interpolation whereas by default, resample is set to 3, which is “nearest” interpolation.