Deploying segmentation model to unity using barracuda

mvp354 · April 29, 2020, 4:42am

Hey! I have used the notebook for the camvid lesson and then trained it on my own data.
The notebook can be seen here

Hi. I have created a onnx file after building an image segmentation model that works to segment feet.
I am unable to figure out how to deploy this to unity.

The model is based on this notebook.

I used the same method and exported the model to onnx format.

Would you able to point me to a direction where I can do this ?

I need to use Barracuda for unity. This requires me to write how prediction should happen myself. Some examples on how classification was done is here

github.com

Syn-McJ/TFClassify-Unity-Barracuda/blob/master/Assets/Scripts/Classifier.cs

using System;
using Barracuda;
using System.Linq;
using UnityEngine;
using System.Collections;
using System.Collections.Generic;
using System.Text.RegularExpressions;

public class Classifier : MonoBehaviour
{
    public NNModel modelFile;
    public TextAsset labelsFile;

    public const int IMAGE_SIZE = 224;
    private const int IMAGE_MEAN = 127;
    private const float IMAGE_STD = 127.5f;
    private const string INPUT_NAME = "input";
    private const string OUTPUT_NAME = "MobilenetV2/Predictions/Reshape_1";

    private IWorker worker;

This file has been truncated. show original

Is there a python file that I can look at to do the same for the resnet model I have trained.
The model works as described in this picture to segment foot
foot

AnshulGoyal · June 14, 2020, 9:55am

Hey I have the same kind of challenge. Deploying a model on Unity. Did you get an idea on how to do it?