Using a model in production

(Vishal ) #1

I have read a couple of posts in the forums which discuss similar topics but could not find anything conclusive . Can someone please tell me what would be the best way to do this? Can I convert model weights to a production ready library, like keras or pytorch itself?


How to productionize your models?
(Matthijs) #2

The weights are PyTorch weights since the model is a PyTorch model.

I don’t know what technologies there are to serve up PyTorch models, but it’s usually possible to convert the weights into any format you want. (But note that PyTorch and TensorFlow use different ways to do padding so just converting the weights does not always give identical results.)


(Saravanan Baskaran) #3

I have the same question, did some exploration. I guess we may have to use ONNX to convert between models. Check this link –

To deploy on mobile, caffe2 backend can be used throgh
There is an example at –


(Levan Tsinadze) #4

ONNX is really great library but it can not convert adaptive pooling layers (AdaptiveAvgPool2d and AdaptiveMaxPool2d)


(Matthijs) #5

It does have a GlobalAveragePool and GlobalMaxPool operator, which are similar to adaptive pooling, where the output will always be 1x1.

1 Like

(Vishal ) #6

Here is my attempt at using a fastai model from pytorch:

learn = ConvLearner.pretrained(arch, data1 )
model = learn.model()

`# pytorch code`,"./torch_model_v1")
the_model = torch.load("./torch_model_v1")
the_model.eval() # shows the entire network architecture

Based on the example shown here:, I understand that I need to write my own data loading class which will override some of the functions in the Dataset class. But what is not clear to me is the transformations that I need to apply at test time?

In fastai, the inference is very straightforward: learn.predict(). Does this internally normalize the images in the test folder? If yes, can someone please tell me how to do it in pytorch?

Another question: is my approach of saving and loading the model in pytorch fine? I read in the tutorial here: that the approach that I have used is not recommended. The reason is not clear though.


(Johannes Laute) #7

I posted an issue at pytorch for adding AdaptiveAvgPool2d and AdaptiveMaxPool2d support in ONNX. Right now you can just change in the fastai source, as these layers are only used in ways that are identical to GlobalPooling (at least right now).

1 Like

(Igor Kasianenko) #8

I’m also interested in this topic.
The only thing I know - from ONNX you may import it into MXNet - open source framework with server to serve models from cloud or local host. It is in active development, so they may not support some layers.
I’d be happy to have some native service to host models


(Jeremy Howard (Admin)) #9

Generally we recommend just using a flask endpoint with a standard pytorch model. Here are previous discussions on this topic:

(Thomas) #10

Hi, I’d like to convert a trained model to .onnx but wasn’t able to do so. Anyone some idea how to do this properly. Below please see my code. I get an error at the last line --> torch.onnx.export --> saying: AttributeError: ‘NoneType’ object has no attribute ‘state_dict’

from fastai.imports import *
from fastai.transforms import *
from fastai.conv_learner import *
from fastai.model import *
from fastai.dataset import *
from fastai.sgdr import *
from fastai.plots import *
from torch.autograd import Variable
import torch.onnx
import torchvision

PATH = “data/dogbreads/”

data = ImageClassifierData.from_paths(PATH, tfms=tfms_from_model(arch, sz))
learn = ConvLearner.pretrained(arch, data, precompute=True)

Model =, ‘Weights.h5’)
dummy_input = Variable(torch.randn(1, 3, 224, 224))
torch.onnx.export(Model, dummy_input, “/home/paperspace/OneDrive/Model.onnx”)



were you able to find a solution?


(Matthijs) #12

Shouldn’t that be:

torch.onnx.export(learn.model, dummy_input, “/home/paperspace/OneDrive/Model.onnx”)

Resnet34 to onnx for the MNIST dataset
(Thomas) #13

yes, thanks! it seems to be working until the point that:

–> “ONNX export failed: Couldn’t export Python operator

do you maybe also know how to handle that!?

I also read your post:
“I posted an issue at pytorch for adding AdaptiveAvgPool2d and AdaptiveMaxPool2d support in ONNX. Right now you can just change in the fastai source, as these layers are only used in ways that are identical to GlobalPooling (at least right now).”

–> I found it in the file of the fastai library … is it enough just to exchange AdaptiveAvgPool2d and AdaptiveMaxPool2d with GlobalPooling or would that be too easy??

thank you!


(atul) #14


Did anyone find the solution for the above problem as adaptivemaxpool2d and adaptiveavgpool2d are still not supported by ONNX while converting the model saved via fastai library?

Thank you!



@oxyd33 Where do you get the Weights at learn.load(“Weights”) ?
Also, did your change of the ‘Adaptive’ layers to ‘GlobalPooling’ eventually work?


(atul) #16

Actually Global layers are not present in fastai so even if you change it will not work and adaptive layers which are present in fastai are not supported by ONNX.

Please if anyone finds a solution for this porblem do reply.


(Jeong-Yoon Lee) #17

As @j.laute posted at pytorch, I replaced nn.AdaptiveAvgPool2d and nn.AdaptiveMaxPool2d with MyAdaptiveAvgPool2d and MyAdaptiveMaxPool2d in as follows:

class MyAdaptiveAvgPool2d(nn.Module):
    def __init__(self, sz=None):

    def forward(self, x): 
        inp_size = x.size()
        return nn.functional.avg_pool2d(input=x,
                  kernel_size= (inp_size[2], inp_size[3]))

class MyAdaptiveMaxPool2d(nn.Module):
    def __init__(self, sz=None):

    def forward(self, x): 
        inp_size = x.size()
        return nn.functional.max_pool2d(input=x,
                  kernel_size= (inp_size[2], inp_size[3]))

class AdaptiveConcatPool2d(nn.Module):
    def __init__(self, sz=None):
        sz = sz or (1,1)
        self.ap = MyAdaptiveAvgPool2d(sz) = MyAdaptiveMaxPool2d(sz)
    def forward(self, x): return[, self.ap(x)], 1)

Then, torch.onnx.export() worked.


(atul) #18

Thank you @jeongyoonlee.
Is it possible if you can tell me the reason behind it? Like I get it that nn.functional is being used but how is it helping adaptive layer in being readable by the onnx layer as adaptive layers are absent in onnx.
Instead of AdaptiveAvgPool2d and AdaptiveMaxpool2d I used other layers like AvgPool2D and MaxPool2D but it was giving an error of running_mean should contain 50176 elements not 1024. If you know anything about this error.
Thank you.


(Thomas) #19

@draz sorry for the late reply but i have not been online here for quite some time and just saw your post today.
the weights i got from training the model and then saving it.
how it’s done is explained in lesson 1.

no, i did not try to change it yet but i am gonna try it out as jeongyoonlee explained it.

@jeongyoonlee thanks for you explanation and code!



Hello, I have seen in this forum dozen of people requesting to easy export to onnx.
I dont see that feature in the doc, just export to (undocumented) pkl.
Any plan for fastai to officially support onnx export ?