How to do Model conversion from PyTorch to ONNX / nGraph / TVM?


So after following the first couple lessons of the new FastAI course, I wanted to see how model conversion is from PyTorch to ONNX but ran into a couple issues using Resnet34;

Converting the PyTorch model to ONNX works:

from fastai.imports import *
from torch.autograd import Variable
import torchvision
import torch.onnx
import onnx

learn.precompute = False
model =, 'model_weights.h5')

dummy_input = Variable(torch.rand(1,3,224,224, device='cpu'))
torch.onnx.export(learn.model, dummy_input, "model.onnx", verbose=True, input_names["input_1"], output_names=["output_1"])

This succesfully saves the model to an ONNX file, which can be viewed using an ONNX viewer like Netron.

Problem 1: Running the model in Windows using C++ Windows::AI::MachineLearning
While the C++ Squeezenet tutorial works, replacing the ONNX file and changing the input / output dims so that they match does not seem to work and the program already crashes on the LearningModelSession at template <typename Class, typename Interface = Windows::Foundation::IActivationFactory, typename F> auto call_factory(F&& callback) in cppwinrt\winrt\base.h on line 7488.

Asking this here hoping someone has experienced this problem before or knows how to troubleshoot. I will also be posting this part on MSDN.

I noticed that the ONNX export from PyTorch is under version 9, yet the Sequeezenet example is version 7.

Problem 2: Importing the ONNX model using ngraph_onnx
Using previous mentioned code, we can add to it to import the model under ngraph, according to the documentation as follows:

onnx_model = onnx.load_model('model.onnx')
from ngraph_onnx.onn_importer.importer import import_onnx_model
ng_function = import_onnx_model(onnx_model)

However the nGraph library seems to throw an error related to the Gather op:

RuntimeError: Assertion 'unknown_operators.empty()' failed at /home/deeplearning/ngraph/ngraph/src/ngraph/frontend/onnx_import/core/graph.cpp:127:
nGraph does not support the following ONNX operations: Gather

So my question here is; How can I export the ONNX model excluding this op? When viewing the graph using Netron, there seem to be some exotic operators which I haven’t really seen before: //TODO add img

Problem 3: Invalid reshape using the nnvm.compiler
I have also looked into NNVM which seems to give me an error that might also be related to the Gather op, at least to me it seems to be near the end of the graph.

import nnvm
import tvm

sym, params = nnvm.frontend.from_onnx(onnx_model)
x = np.array(img)[np.newaxis] #Add batch axis
x.shape # (1, 224, 224, 3)
import nnvm.compiler
target = 'cuda'
input_name = sym.list_input_names()[0] # Its equal to 'input_1'
shape_dict = {input_name: x.shape}
with nnvm.compiler.build_config(opt_level=3):
    graph, lib, params =, target, shape_dict, params=params, dtype={'0':'float32'})

This will result into the following error:
nnvm/src/top/tensor/ Check failed: in_attrs->at(0).Size() == in_attrs->at(1).Size() (1024 vs. 2) Reshape inputs size should be compatible

A couple remarks:

  • PyTorch seems to export the ONNX model under ai.onnx.v9 whereas the SequeezeNet example on MSDN is using ai.onnx.v7. I am not sure if an API incompatibility is causing the error for problem 1.
  • Netron tells me that the Gather op is supported version 1 at support level common, so its not like it is anywhere near recent.
  • Ultimate goal is to run the model on CPU under Windows using as little libraries as needed, preferably on a single core only.

If anyone has any ideas, suggestions or tips let me know! Am quite new to PyTorch and all compilers / graph executors.

Thanks :slight_smile:

Sorry if this is in the wrong category.