FastAI to CoreML

Part 1 (2019)
1

am experiencing mismatch in classification / prediction output of trained FastAI model after conversion to CoreML (via ONNX as interim step) on same test data.

Following a safari in South Africa during which we were lucky enough to have two leopard sightings, we struggled to identify gender of the animal. Following the pets example, trained a resnet34 based learner based on 100 downloaded images of female and male leopards each.

error rate after training is about 10%.

As I want to use model in iOS Swift based app, converted to Onnx then to CoreML.

Conversion:
import torch
import torch.nn as nn
import numpy as np
from onnx_coreml import convert
from torch.autograd import Variable
import torch.onnx
import torchvision
import onnx
model_name = “model_leopardGender”
learn.precompute = False
learn.model.cpu()
model = torch.save(learn.model, model_name + ‘.h5’)
dummy_input = Variable(torch.rand(1,3,224,224, device=‘cpu’))
torch.onnx.export(learn.model,dummy_input,model_name, input_names=[‘image’], output_names=[‘gender’], verbose=False)
mlmodel = convert(onnx.load(model_name), image_input_names = [‘image’], mode=‘classifier’, class_labels=[‘female’,‘male’])
mlmodel.author = ‘tpeter’
mlmodel.license = ‘MIT’
mlmodel.short_description = ‘This model takes a picture of a leopard and predicts its gender’
mlmodel.input_description[‘image’] = ‘Image of a leopard’
mlmodel.output_description[‘gender’] = ‘Confidence and label of predicted gender’
mlmodel.output_description[‘classLabel’] = ‘Label of predicted gender’
mlmodel.save(f’{model_name}.mlmodel’)

converts without error.

Then used Apples CoreML example app (https://developer.apple.com/documentation/vision/classifying_images_with_vision_and_core_ml?changes=_8) and replaced default coreml model with the above created.

Classification as well as confidence differs though vastly between fast.ai learner.predict and using the coreML model. (coreML model predictions are frequently way off)

Anybody experiencing the same? Any ideas on root cause? e.g. any pre-processing that needs to be done on image in swift to match?

2

Meanwhile I tried various scenarios to narrow down problem with no success:

  • used Pets example of lesson 1 (i.e. did not train own learner)
  • manually resized images in iOs app / Swift to same proportions as size in fast.ai training.
  • tried with ResNet50 and ResNet34
    Mismatches in classification between coreML and fastai inference result remain.
3

It’s been a while since I played with this but I think I remember one of the “tricks” I missed the first time around was that I forgot to add in the ImageNet normalization.

This was the tutorial with the steps that finally worked for me: https://medium.com/@hungminhnguyen/convert-fast-ai-trained-image-classification-model-to-ios-app-via-onnx-and-apple-core-ml-5fdb612379f1

2 Likes
4

Thank you Brad. Inclusion of the normalisation indeed makes classification now mostly match. Confidences still differ but I guess this is “by design” (also I could not include the softmax as per tutorial due to error message)

2 Likes
5

Hey guys,
I tried following the code from the tutorial using v3 of fasti but the when trying to save the onnx model, it won’t work unless I do nn.Softmax(dim = -1) instead of nn.Softmax()
In [10] final_model = [ImageScale()] + (list(learn.model.children())[:-1] + [nn.Softmax()])
And when I do that, the onnx model then can’t be converted into a mlmodel because it leads to Softmax[axis = 3] and that throws an error because the axis can only be 1.
The type error is: 'Error while converting op of type: Softmax. Error message: Unsupported axis 3 for softmax"
I also have 11 classes instead of two. It’ll work if I don’t include softmax. But any ideas how to fix it? Is there another way to predict a corml model without relying on softmax?

6

Did you ever get this working? Either I get weird prediction values over 100 with your code or the more generic message when I try the medium tutorial.
Failed to perform classification.
The VNCoreMLTransform request failed

EDIT: Got it working. Turns out Xcode caches your files so I was trying to run the app with my first model.
This code was also helpful

2 Likes