Productionizing models thread

@nok: this will happen if you somehow have your model.pth in your app directory (app/models) - (may be you ran the app locally for testing)

Make sure you delete that pth file before deploying (the server should download that file via the download url you provide in model_file_url)

1 Like

Thanks! That sounds promising, I only follow the tutorial and my impression is it will do that for me. I will come back soon when I get it working. :slight_smile:

I keep getting this error even I follow this instruction. I have a free domain name with xxxx.ml from https://my.freenom.com.

@arunoda Is that something you can help? Thanks you.

Please make sure that your nameservers point to zeit.world.
  Examples: (full list at https://zeit.world)
    a.zeit.world        96.45.80.1
    b.zeit.world        46.31.236.1
    c.zeit.world        43.247.170.1

  As an alternative, you can add following records to your DNS settings:

I’ve submitted a PR with my concept of LM inference API. Let me know if you like it (or not). https://github.com/fastai/fastai/pull/1128

3 Likes

Follow this guide: https://zeit.co/docs/v2/domains-and-aliases/adding-a-domain#3-using-a-custom-domain-dns-managed-by-now

1 Like

Thanks, I did follow the guide and the error message. I added the namesserver yet I still get the same error message.

image
image
image

Was trying to deploy segmentation model using zeit but unable to do so.

Step 7/9 : RUN python app/server.py

—> Running in 57b7b5612ce1
Traceback (most recent call last):
File “app/server.py”, line 37, in
learn = loop.run_until_complete(asyncio.gather(*tasks))[0]
File “/usr/local/lib/python3.6/asyncio/base_events.py”, line 473, in run_until_complete
return future.result()
File “app/server.py”, line 32, in setup_learner
learn.load(model_file_name)
File “/usr/local/lib/python3.6/site-packages/fastai/basic_train.py”, line 204, in load
self.model.load_state_dict(torch.load(self.path/self.model_dir/f’{name}.pth’, map_location=device))
File “/usr/local/lib/python3.6/site-packages/torch/serialization.py”, line 358, in load
return _load(f, map_location, pickle_module)
File “/usr/local/lib/python3.6/site-packages/torch/serialization.py”, line 519, in _load
magic_number = pickle_module.load(f)
_pickle.UnpicklingError: invalid load key, ‘<’.
The command ‘/bin/sh -c python app/server.py’ returned a non-zero code: 1

On running in zeid folder on Ubunto terminal and similar issue when running on paperspace machine.

sudo npm install -g now

I get the following error:

139 error Linux 4.4.0-17134-Microsoft 140 error argv “/usr/bin/nodejs” “/usr/bin/npm” “install” “-g” “now” 141 error node v4.2.6 142 error npm v3.5.2 143 error file sh 144 error code ELIFECYCLE 145 error errno ENOENT 146 error syscall spawn 147 error now@12.1.2 postinstall: node download/install.js 147 error spawn ENOENT 148 error Failed at the now@12.1.2 postinstall script ‘node download/install.js’. 148 error Make sure you have the latest version of node.js and npm installed. 148 error If you do, this is most likely a problem with the now package, 148 error not with npm itself. 148 error Tell the author that this fails on your system: 148 error node download/install.js 148 error You can get information on how to open an issue for this project with: 148 error npm bugs now 148 error Or if that isn’t available, you can get their info via: 148 error npm owner ls now 148 error There is likely additional logging output above.

npm -v 6.4.1
node -v 11.1.0

@vikashkmr: the download URL to your model is not correct

Make sure you do this step properly

Note: the download link is the one which starts the file download directly—and is normally different than the share link which presents you with a view to download the file (use https://rawdownload.now.sh/ if needed)

Refer:
https://forums.fast.ai/t/new-guide-for-easy-web-app-deployment/29616/15

Here is my little web app, thanks for everyone who helped.
It’s a little web app classifying type of a container.
https://jianyang.ml/

you don’t need to set real path in ImageDataBunch.single_from_classes example

Hi @simonw!
Thanks a lot for sharing this! I was able to create a MVP web app in no time thanks to this.
I have a question. Is this actually a full REST Api? Is it possible to call the API via curl?
The command curl -X POST -F image=@teddy-bear.jpg 'http://localhost/upload' gave me an Internal Server Error as response. I am asking this because I want to use your boilerplate to create a backend that could be called from an Android client.

By the way I slightly modified your code to work with the latest fastai version. Check it out here. The main issue was to use create_cnn instead of ConvLearner.

Thanks a lot for reading this! :smile:

@Andreas_Daiminger check my server boilerplate example https://github.com/IaroslavR/fastai-rest-server-template

1 Like

@elruso Thanks for sharing this. Looks cool. :raised_hands:
I built a REST API with Flask in the meantime and dockerized it. All based on suggestions from the fast.ai forum. Check it out here

I created a repository to test PyTorch models in Flask and then deploy them to AWS Lambda using Zappa.
Link to repository: https://github.com/pedrohbtp/pytorch-zappa-serverless

The Gif from above works from an AWS Lambda deployment of the pre-trained english Language Model available in Fastai. You can test it out here: https://pedrohbtp.github.io/language-model/

AWS Lambda has some size limitations, so I had to use fastai’s PyTorch models instead of installing the entire library.

If someone wants an introduction to Flask, I wrote a little about it here.

8 Likes

@Andreas_Daiminger pay attention to paragraph three of https://www.codacy.com/blog/five-ways-to-slim-your-docker-images/ this way you can minimize your image simple way

1 Like

Thanks @elruso! Very useful if you wanna become a docker power user!

It appears that the Zeit tarball linked to in the deployment tutorial on the website isn’t compatible with the most recent version of fastai. I tried modifying it to use ImageDataBunch.load_empty() but ran into some trouble.

It did deploy but it appears to not actually return the prediction. I hit Analyze, the button switches to Analyzing... and that’s it. Here’s my code, and I’d be super grateful if someone wants to look through it and try and pinpoint why this might be happening. Look at lines 14, 15, 33, 34, 35 for the changes.

P.S. I’ve never actually created a web app before so I might just be missing something silly.

2 Likes

Thanks for your link.

I was able to train my model and get something up and running more or less along your guidelines.

I had to do a lot to get everything to fit inside of lambda as the default deployment was way way too big. So much too big it makes me think you perhaps left out some of that in your repo?

I couldn’t find anyway to tree-shake in python “automatically” so I did myself a bit and was able to shrink enough.

I also had to load my model into memory directly from s3 rather than storing it locally as I couldn’t fit both the code and model in the scratch space.

My ‘deployed’ version is here: https://gdoteof.github.io/neuralnet_stuff/

I have no done a writeup (though all the code, complete with my fumblings is in the above repo).

If anyone ends up on this thread trying to figure out how to shrink their deployment to fit on lambda, take a look at my zappa config to see some things you can remove.

I am glad you also made it work. Great work there.
If there is interest I can try to explain a little better the deployment and create more documentation on the readme.

Indeed there is no auto tree-shaking. When I did it, I had to choose which of the libraries I would include. That is why I didn’t even include the actual fastai, but only Pytorch related dependencies. I apologize if there was any trouble. I am not sure if I left any unnecessary dependencies there. I can double check.

It is really not possible to fit the model and the code into the lambda space. We have to read it from s3 every time which might make loading a little bit slower.