Issues with deployment on google app engine example

(Arkar Aung) #1

I am having this error after following the example for deploying on google app engine.

This is after uploading the image and clicking ‘Analyze’

TypeError: Object of type 'Category' is not JSON serializable

Full Error Trace:

ERROR: Exception in ASGI application
2019-02-06 04:42:59 default[20190206t121532]  Traceback (most recent call last):  
File "/usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py", line 372, in run_asgi      result = await asgi(self.receive, self.send)   
File "/usr/local/lib/python3.6/site-packages/starlette/middleware/errors.py", line 125, in asgi      raise exc from None  
File "/usr/local/lib/python3.6/site-packages/starlette/middleware/errors.py", line 103, in asgi      await asgi(receive, _send)  
File "/usr/local/lib/python3.6/site-packages/starlette/exceptions.py", line 74, in app      raise exc from None    
File "/usr/local/lib/python3.6/site-packages/starlette/exceptions.py", line 63, in app      await instance(receive, sender)    
File "/usr/local/lib/python3.6/site-packages/starlette/routing.py", line 41, in awaitable      response = await func(request)    File "app/server.py", line 49, in analyze      return JSONResponse({'result': learn.predict(img)[0]})   
File "/usr/local/lib/python3.6/site-packages/starlette/responses.py", line 43, in __init__      self.body = self.render(content)  
File "/usr/local/lib/python3.6/site-packages/starlette/responses.py", line 183, in render      separators=(",", ":"),  
File "/usr/local/lib/python3.6/json/__init__.py", line 238, in dumps      **kw).encode(obj) 
File "/usr/local/lib/python3.6/json/encoder.py", line 199, in encode      chunks = self.iterencode(o, _one_shot=True)  
File "/usr/local/lib/python3.6/json/encoder.py", line 257, in iterencode      return _iterencode(o, 0) 
File "/usr/local/lib/python3.6/json/encoder.py", line 180, in default      o.__class__.__name__) 

TypeError: Object of type 'Category' is not JSON serializable```
0 Likes

(Arkar Aung) #2

I figured it out. The simple solution is to change the Category data type which is returned by learn.predict to String.

JSONResponse({'result':learn.predict(img)[0]} to
JSONResponse({'result':str(learn.predict(img)[0])}

5 Likes

Error Deploying on Google App Engine
Gcloud app engine does not work
#3

Have you managed to deploy an app on google cloud? My deployment crashes at:

Updating service [default] (this may take several minutes)...failed.

When I looked at the logs, I found that:

_check_disk_space.sh: Free disk space (792 MB) is lower than threshold value 954 MB. Reporting instance permanently unhealthy. Note: different instances will have a different threshold value.

0 Likes

(Arkar Aung) #4

Hey @endrju, I could successfully deploy on google app engine. From the looks of your error, I guess it has something to do with google app engine instance itself.

I had one hiccup when deploying on Google App Engine. I re-ran the command gcloud app deploy for the second time and it worked. What you could also do is delete the app from google app engine and deploy again.

0 Likes

(Akshai Rajendran) #5

Hey @ark_aung, I’m currently trying to deploy on Google App Engine and having difficult installing fastai due to what appears to be a memory error. I’m assuming your deployment involved installing the fastai module right? If so, how were you able to do that?

0 Likes

(Arkar Aung) #6

Hey @arajendran,

Do you mind attaching the memory error message that you are getting?

0 Likes

(Akshai Rajendran) #7

@ark_aung Thanks for the reply. When building with pip I just see “Killed” during the download/build of torch. When deploying I see “oserror cannot allocate memory” in the error log. I’ve switched from the Standard environment to Flexible and that seems to have solved the memory issue, though I’m still working through a few other errors. Since the runtime is custom however I’m currently trying to get logging to work.

0 Likes

(Arkar Aung) #8

When you create your instance, did you make sure that you choose the right image?

Here, the IMAGE_FAMILY makes sure that there are all the stuff we would need for Fast-AI Pytorch.

export IMAGE_FAMILY="pytorch-latest-gpu" # or "pytorch-latest-cpu" for non-GPU instances
export ZONE="us-west2-b" # budget: "us-west1-b"
export INSTANCE_NAME="my-fastai-instance"
export INSTANCE_TYPE="n1-highmem-8" # budget: "n1-highmem-4"

# budget: 'type=nvidia-tesla-k80,count=1'
gcloud compute instances create $INSTANCE_NAME \
        --zone=$ZONE \
        --image-family=$IMAGE_FAMILY \
        --image-project=deeplearning-platform-release \
        --maintenance-policy=TERMINATE \
        --accelerator="type=nvidia-tesla-p4,count=1" \
        --machine-type=$INSTANCE_TYPE \
        --boot-disk-size=200GB \
        --metadata="install-nvidia-driver=True" \
        --preemptible

Ref: https://course.fast.ai/start_gcp.html

0 Likes

(Akshai Rajendran) #9

Well I’m deploying this on Google App Engine which is separate from my Google Compute instance where I’m actually training the models, if I understand it correctly. I’ve actually been able to get it to work after making some changes to the code provided in the deployment guide git. I’m deploying via Flask which is the main reason I didn’t just follow the guide straight through. For future reference if it helps anyone, the primary change was changing the Dockerfile to the following (server is run from main.py located in app sub-folder):

FROM python:3.6-slim-stretch

RUN apt update

RUN apt install -y python3-dev gcc

ADD requirements.txt requirements.txt

RUN pip install -r requirements.txt

COPY app app/

EXPOSE 8080

CMD ["gunicorn", "-b", ":8080", "--chdir", "app/", "main:app"]
0 Likes

(Joaquin Maya) #10

Hi, I’m trying to deploy an app using google app engine and I am having this error

0 Likes

(David) #11

Did you ever get it to work?

0 Likes

(Derek) #12

Hey - where did you actually make this change in the code? I tried to change the response from learn.predict to a string but it still doesn’t seem to be working, maybe I am doing something incorrect.

0 Likes

(Arkar Aung) #13

This is the change that I made JSONResponse({'result':str(learn.predict(img)[0])}. Perhaps you did this and still things are not working. Show me your code and your error.

0 Likes

(Derek) #14

That’s exactly what I did, but I just realized I didn’t restart the server when testing (womp). Just restarted the server and it’s all working now, thanks for offering your help though.

0 Likes

(Amit) #15

Hi,

Kind of bit late but issue is pretty much the same that is mentioned here. So far have :

a) changed tfms to ds_tfms
b) changed learner to cnn_learner
c) updated return stmt in analyze to return JSONResponse({‘result’: learn.predict(img)[0].obj})

Now error is :

Traceback (most recent call last):
File “app/server.py”, line 36, in
learn = loop.run_until_complete(asyncio.gather(*tasks))[0]
File “/usr/local/lib/python3.6/asyncio/base_events.py”, line 484, in run_until_complete

Have shutdown and restored the GAE instance before the last change. Everytiem wiping out git directory and cloning after every update.

Am now going to try the docker file change suggested above.

Kindly help.

0 Likes

(Amit) #16

Made bit of progress.

No more build errors.

Now its complaining about time out:

Updating service [default] (this may take several minutes)…failed.
ERROR: (gcloud.app.deploy) Error Response: [4] Your deployment has failed to become healthy in the allotted time and therefore was rolled back. If you believe this was an error, try adjusting th
e ‘app_start_timeout_sec’ setting in the ‘readiness_check’ section.

Added with value of 600 but still same error.

Will keep checking…

Any ideas?

0 Likes

(Amit) #17

Hi,

Still no joy, tried with a clean project, enabled billing, and ran steps.

Error is still the same … timeout…

Updating service [default] (this may take several minutes)…failed.
ERROR: (gcloud.app.deploy) Error Response: [4] Your deployment has failed to become healthy in the allotted time and therefore was rolled back. If you believe this was an error, try adjusting
the ‘app_start_timeout_sec’ setting in the ‘readiness_check’ section.

Please help.

0 Likes

Problem deploying app using gcloud app engine