Productionizing models thread

How we can do it for unet_learner?

    data_path = Path('/home/data/')
    codes = np.loadtxt(data_path/"codes.txt", dtype=str); codes
    label = data_path/"mask"
    image = data_path/"input"

    get_y_fn = lambda x: label/f'{x.stem}.jpg'

    size = (512,512)#src_size//2
    print(size)
    bs=8

    class SegLabelListCustom(SegmentationLabelList):
        def open(self, fn): return open_mask(fn, div=True)
    
    class SegItemListCustom(ImageList):
         _label_cls = SegLabelListCustom

    src = (SegItemListCustom.from_folder(image)
           .split_by_rand_pct(0.2)
           .label_from_func(get_y_fn, classes=codes) )

    data = (src.transform(get_transforms(), size=size, tfm_y=True )
             .databunch(bs=bs)
             .normalize(imagenet_stats))

    name2id = {v:k for k,v in enumerate(codes)}
    print(name2id)
    # void_code = name2id['Void']

    def acc_camvid(input, target):
        target = target.squeeze(1)
    #     mask = target != void_code
        return (input.argmax(dim=1)==target).float().mean()

    metrics=acc_camvid
    # metrics=accuracy

    wd=1e-2

    learn = unet_learner(data, models.resnet34, metrics=metrics, wd=wd)

    learn.load("./models/stage-2-big");

    img = open_image("test.jpg")

    prediction = learn.predict(img)[0]

    prediction #how to convert it to numpy image?

I have to all this for loading the learner for prediction in the flask application because of load_learner() from .pkl file is not working. Please provide an elegant way of doing this for a single image. After getting the prediction, I am not able to convert it to numpy or OpenCV image format. And If I am trying to save the image, it is saving it a blank image. Please help asap!!

My issue is that I never came to data science (and from there here) via a programming background. I do know a lot of data science programming but zero webstack or cloud. All the deployment guides seem to assume you ‘know’ webstack and cloud and work with it regularly. I don’t mind at all learning these things but I don’t want to learn them in their entirety - just “webstack/cloud for deep learning.” Tools to ‘go faster’ will help me nothing because as I learned when setting up Google cloud there are many things that always ‘go wrong’ or have changed etc and these roadblocks will be worse with an efficiency tool. Rather, I need to know the high level hoops I have to jump through and a list of the things I need to understand or try to jump through them. Thanks.

After scouring the web and forums for two days trying to figure out how to do this I have now discovered fastai have guides to put models into production, which I never noticed before. Doh!

Hi Michael !

Thanks a lot for your reply. I completely understand and share your perspective, as I also a maths background but almost 0 experience with web/cloud dev.

The Fastai tutorials you’ve just found are incredibly helpful for putting your first model in production, so hopefully it may help you solve some of your problems. But in my experience, the process to deploy is still tedious (especially if you want to use AWS lambda or Azure Functions), and many more advanced stuff is not covered by the tutorials (for example, deploying a web API, deploying on mobile, etc…).

Thanks,
Seb

I created a complete example of deployment fastai model in c++ application. There are some parts of the process in the forum. But I think it would be useful to have all of it in one place.

2 Likes

Have you found the solution? I’m also facing the same problem

I did not. I installed fastai and other packages on my customer’s machine. To make it easier, I created an environment.yml file.

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

All,

I’m attempting to follow the aws_lambda guide. https://course.fast.ai/deployment_aws_lambda.html Upon initial training, I have 3 models and data totaling about 800 megs compressed, and 1GB uncompressed.

One of the steps is to use Torchscript to simplify/streamline the model. The example they give is with a CNN. I’m attempting to deploy a variant of the IMDB ULMFiT implementation that they worked with in lessons 3 and 4. Clearly, the deployment guide is set to trace out a CNN.

I’m new to fast.ai, and discovered that torchscript existed yesterday when I started trying to follow the lambda guide described above. I’m still trying to get a sense of torchscript… the “gentle” guides are still over my head at this point. I’m honestly surprised that torchscript didn’t make the cut to be included in the 12 lessons of the fast.ai course. That feels like a bit of an oversight.

Can somebody point me in a good direction to begin understanding how to modify the CNN script for an RNN? A code sample would be amazing, but failing that, documentation/training would be great.

Thank you!

(error included below)

2 jit_model = torch.jit.trace(learn.model.float(), trace_input)
3 model_file=‘resnet50_jit.pth’
4 output_path = str(path_img/f’models/{model_file}’)
5 torch.jit.save(jit_model, output_path)

7 frames
/usr/local/lib/python3.6/dist-packages/fastai/text/learner.py in forward(self, input)
257
258 def forward(self, input:LongTensor)->Tuple[Tensor,Tensor]:
–> 259 bs,sl = input.size()
260 self.reset()
261 raw_outputs,outputs,masks = [],[],[]

ValueError: too many values to unpack (expected 2)

Ran into this recently and found a solution from this blog post:

You’ll need to specify fastprogress as a hiddenimport in your spec file, specify the location of your hook files, and then create a new hook file with the following content:

from PyInstaller.utils.hooks import copy_metadata
datas = copy_metadata('fastprogress')

Be sure to name it hook-fastprogress.py

For more information check out PyInstaller’s documentation regarding hooks and hidden imports.

1 Like

All,

I am looking for best practices in measuring concept drift . Can some body share how do you handle concept drift while dealing with text and images.

Any good papers that you can suggest. thanks.

how to do model conversion from pytorch to onnx ngraph
Sorry to be late, but this article helped me

Is there a version of this thread for beginners?

In lecture 2, Jeremy said to try to build a web app “This week”, and said that there are lots of tutorials out there. From the two days worth of tutorials I’ve watched so far, I have gathered that in order to build a web-app, you need baseline understanding in the following areas:

  1. A framework like Flask or Django
  2. JavaScript
  3. CSS
  4. HTML
  5. Template Engines
  6. Maybe SQL too…

That’s not happening “this week”, at least for me. I feel like I am missing something. Can someone point the dumb new person to the right place?

1 Like

To accelerate the inference on CPU we can convert the pytorch model to an “optimized model” Openvino model from Intel https://software.intel.com/en-us/openvino-toolkit (passing through an intermediate ONNX model ). I’ve just update a colab notebook (https://colab.research.google.com/drive/1z4GxsF6LaVl5cSeGgCbWTefuUIz78gfV) to summarize all the step to install the latest openvino (l_openvino_toolkit_p_2020.2.120) and run a resnet fastaiv1 model. The work is based on this notebook for old openvino version (https://colab.research.google.com/drive/1TdjV6bSrgSAL6RcGsYzCaWMdaqaqCYID).

I tested with 3 images on CPU:

  • Without Openvino: ~ 700ms
  • With Openvino: ~ 500ms

The gap is not so big but you can use some accelerators which are compatible with Openvino (Intel GPU, Neural Compute Stick), and normally, it is cheaper than NVIDIA GPU

Hi, I was just about to look through the code, but permission denied :joy:

Ah sorry. Does it work with this one ? https://colab.research.google.com/drive/1z4GxsF6LaVl5cSeGgCbWTefuUIz78gfV

1 Like

Laura, thanks for sharing this.

I’ve been trying to deploy a fastai model using lambda with no success.
As I was not able to use a public PyTorch layer, I’m building my own.
Applying all the recommendations, I ended up with dependencies totaling 437Mb. The compressed layer has 130Mb.

When trying to run the lambda function, I get the error ‘No space left on device’.

Below the most top dependency folders:
437M ./build/requirements
305M ./build/requirements/torch
272M ./build/requirements/torch/lib
68M ./build/requirements/numpy
29M ./build/requirements/numpy/.libs
23M ./build/requirements/numpy/core
23M ./build/requirements/pandas
17M ./build/requirements/pandas/_libs
17M ./build/requirements/torch/bin
12M ./build/requirements/torchvision
11M ./build/requirements/numpy/random
11M ./build/requirements/torch/include
8.6M ./build/requirements/matplotlib
8.0M ./build/requirements/caffe2
7.2M ./build/requirements/caffe2/python

Any hint/suggestion will be appreciated!

Hi Alejandro

Yes I see, the problem is the pytorch dependency, which is too big. It’s why I used the separate public layer for pytorch.

This post by @matt.mcclean explains how the pytorch layer was made, it might help:

AWS Lambda has a limit of 250 MB for the deployment package size including lamba layers. PyTorch plus its dependencies is more than this so we need to implement a trick to get around this limit. We will create a zipfile called .requirements.zip with all the PyTorch and associated packages. We will then add this zipfile to the Lambda Layer zipfile along with a python script called unzip_requirements.py . The python script will extract the zipfile .requirements.zip to the /tmp when the Lambda execution context is created.

Hope that helps!

Laura

1 Like

Thank you for this wonderful thread.
How can I deposit uploaded image files on the server?
For example, in OnRender example, the uploaded image file is not stored on the server.
However, it would be better to save the user’s input images to the server, and possibly update the model in the future with larger image database.
Anybody have some ideas how to stack the uploaded files?

@wcneill forking this repo is probably the easiest way to get started, and if you want to create a web app that doesn’t require uploading photos you can also fork this repo I made that works in conjunction with the render template I posted above.

Correct me if I’m wrong, but this method loads and uses the model for inference every time you call it right?