Issues with deployment on google app engine example

I am having this error after following the example for deploying on google app engine.

This is after uploading the image and clicking ‘Analyze’

TypeError: Object of type 'Category' is not JSON serializable

Full Error Trace:

ERROR: Exception in ASGI application
2019-02-06 04:42:59 default[20190206t121532]  Traceback (most recent call last):  
File "/usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py", line 372, in run_asgi      result = await asgi(self.receive, self.send)   
File "/usr/local/lib/python3.6/site-packages/starlette/middleware/errors.py", line 125, in asgi      raise exc from None  
File "/usr/local/lib/python3.6/site-packages/starlette/middleware/errors.py", line 103, in asgi      await asgi(receive, _send)  
File "/usr/local/lib/python3.6/site-packages/starlette/exceptions.py", line 74, in app      raise exc from None    
File "/usr/local/lib/python3.6/site-packages/starlette/exceptions.py", line 63, in app      await instance(receive, sender)    
File "/usr/local/lib/python3.6/site-packages/starlette/routing.py", line 41, in awaitable      response = await func(request)    File "app/server.py", line 49, in analyze      return JSONResponse({'result': learn.predict(img)[0]})   
File "/usr/local/lib/python3.6/site-packages/starlette/responses.py", line 43, in __init__      self.body = self.render(content)  
File "/usr/local/lib/python3.6/site-packages/starlette/responses.py", line 183, in render      separators=(",", ":"),  
File "/usr/local/lib/python3.6/json/__init__.py", line 238, in dumps      **kw).encode(obj) 
File "/usr/local/lib/python3.6/json/encoder.py", line 199, in encode      chunks = self.iterencode(o, _one_shot=True)  
File "/usr/local/lib/python3.6/json/encoder.py", line 257, in iterencode      return _iterencode(o, 0) 
File "/usr/local/lib/python3.6/json/encoder.py", line 180, in default      o.__class__.__name__) 

TypeError: Object of type 'Category' is not JSON serializable```

I figured it out. The simple solution is to change the Category data type which is returned by learn.predict to String.

JSONResponse({'result':learn.predict(img)[0]} to
JSONResponse({'result':str(learn.predict(img)[0])}

6 Likes

Have you managed to deploy an app on google cloud? My deployment crashes at:

Updating service [default] (this may take several minutes)...failed.

When I looked at the logs, I found that:

_check_disk_space.sh: Free disk space (792 MB) is lower than threshold value 954 MB. Reporting instance permanently unhealthy. Note: different instances will have a different threshold value.

Hey @endrju, I could successfully deploy on google app engine. From the looks of your error, I guess it has something to do with google app engine instance itself.

I had one hiccup when deploying on Google App Engine. I re-ran the command gcloud app deploy for the second time and it worked. What you could also do is delete the app from google app engine and deploy again.

Hey @ark_aung, I’m currently trying to deploy on Google App Engine and having difficult installing fastai due to what appears to be a memory error. I’m assuming your deployment involved installing the fastai module right? If so, how were you able to do that?

Hey @arajendran,

Do you mind attaching the memory error message that you are getting?

@ark_aung Thanks for the reply. When building with pip I just see “Killed” during the download/build of torch. When deploying I see “oserror cannot allocate memory” in the error log. I’ve switched from the Standard environment to Flexible and that seems to have solved the memory issue, though I’m still working through a few other errors. Since the runtime is custom however I’m currently trying to get logging to work.

When you create your instance, did you make sure that you choose the right image?

Here, the IMAGE_FAMILY makes sure that there are all the stuff we would need for Fast-AI Pytorch.

export IMAGE_FAMILY="pytorch-latest-gpu" # or "pytorch-latest-cpu" for non-GPU instances
export ZONE="us-west2-b" # budget: "us-west1-b"
export INSTANCE_NAME="my-fastai-instance"
export INSTANCE_TYPE="n1-highmem-8" # budget: "n1-highmem-4"

# budget: 'type=nvidia-tesla-k80,count=1'
gcloud compute instances create $INSTANCE_NAME \
        --zone=$ZONE \
        --image-family=$IMAGE_FAMILY \
        --image-project=deeplearning-platform-release \
        --maintenance-policy=TERMINATE \
        --accelerator="type=nvidia-tesla-p4,count=1" \
        --machine-type=$INSTANCE_TYPE \
        --boot-disk-size=200GB \
        --metadata="install-nvidia-driver=True" \
        --preemptible

Ref: https://course.fast.ai/start_gcp.html

Well I’m deploying this on Google App Engine which is separate from my Google Compute instance where I’m actually training the models, if I understand it correctly. I’ve actually been able to get it to work after making some changes to the code provided in the deployment guide git. I’m deploying via Flask which is the main reason I didn’t just follow the guide straight through. For future reference if it helps anyone, the primary change was changing the Dockerfile to the following (server is run from main.py located in app sub-folder):

FROM python:3.6-slim-stretch

RUN apt update

RUN apt install -y python3-dev gcc

ADD requirements.txt requirements.txt

RUN pip install -r requirements.txt

COPY app app/

EXPOSE 8080

CMD ["gunicorn", "-b", ":8080", "--chdir", "app/", "main:app"]
1 Like

Hi, I’m trying to deploy an app using google app engine and I am having this error

Did you ever get it to work?

Hey - where did you actually make this change in the code? I tried to change the response from learn.predict to a string but it still doesn’t seem to be working, maybe I am doing something incorrect.

This is the change that I made JSONResponse({'result':str(learn.predict(img)[0])}. Perhaps you did this and still things are not working. Show me your code and your error.

That’s exactly what I did, but I just realized I didn’t restart the server when testing (womp). Just restarted the server and it’s all working now, thanks for offering your help though.

Hi,

Kind of bit late but issue is pretty much the same that is mentioned here. So far have :

a) changed tfms to ds_tfms
b) changed learner to cnn_learner
c) updated return stmt in analyze to return JSONResponse({‘result’: learn.predict(img)[0].obj})

Now error is :

Traceback (most recent call last):
File “app/server.py”, line 36, in
learn = loop.run_until_complete(asyncio.gather(*tasks))[0]
File “/usr/local/lib/python3.6/asyncio/base_events.py”, line 484, in run_until_complete

Have shutdown and restored the GAE instance before the last change. Everytiem wiping out git directory and cloning after every update.

Am now going to try the docker file change suggested above.

Kindly help.

Made bit of progress.

No more build errors.

Now its complaining about time out:

Updating service [default] (this may take several minutes)…failed.
ERROR: (gcloud.app.deploy) Error Response: [4] Your deployment has failed to become healthy in the allotted time and therefore was rolled back. If you believe this was an error, try adjusting th
e ‘app_start_timeout_sec’ setting in the ‘readiness_check’ section.

Added with value of 600 but still same error.

Will keep checking…

Any ideas?

Hi,

Still no joy, tried with a clean project, enabled billing, and ran steps.

Error is still the same … timeout…

Updating service [default] (this may take several minutes)…failed.
ERROR: (gcloud.app.deploy) Error Response: [4] Your deployment has failed to become healthy in the allotted time and therefore was rolled back. If you believe this was an error, try adjusting
the ‘app_start_timeout_sec’ setting in the ‘readiness_check’ section.

Please help.

Using this git

 https://github.com/pankymathur/google-app-engine

and changing server.py inputs:
a) changed tfms to ds_tfms
b) updated return stmt in analyze to return JSONResponse({‘result’: learn.predict(img)[0].obj})
c) putting my own dropbox link for model
d) changing prediction classes to my models prediction classes…

and then firgured out my “size mismatch for 0.4.0.conv1.weight:” error was due to loading resnet34 instead of resnet 50 (which I trained), so I changed model download from resnet34 to resnet50, but now facing another issue/error:

Step 7/9 : RUN python app/server.py
—> Running in e16ff31a58db
Traceback (most recent call last):
File “app/server.py”, line 9, in
from fastai.vision import *
File “/usr/local/lib/python3.6/site-packages/fastai/vision/init.py”, line 5, in
from .data import *
File “/usr/local/lib/python3.6/site-packages/fastai/vision/data.py”, line 4, in
from .transform import *
File “/usr/local/lib/python3.6/site-packages/fastai/vision/transform.py”, line 233, in
_solve_func = getattr(torch, ‘solve’, torch.gesv)
AttributeError: module ‘torch’ has no attribute ‘gesv’
The command ‘/bin/sh -c python app/server.py’ returned a non-zero code: 1
ERROR
ERROR: build step 0 “gcr.io/cloud-builders/docker” failed: exit status 1

then changed dockefile per arajendran’s post to

FROM python:3.6-slim-stretch
RUN apt update
RUN apt install -y python3-dev gcc
ADD requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY app app/
EXPOSE 8080
CMD [“gunicorn”, “-b”, “:8080”, “–chdir”, “app/”, “main:app”]

but the app fails to deploy, due to error:

ERROR: (gcloud.app.deploy) Error Response: [4] Your deployment has failed to become healthy in the allotted time and therefore was rolled back. If you believe this was an error, try adjusting
the ‘app_start_timeout_sec’ setting in the ‘readiness_check’ section.

I looked into this error:

  1. https://stackoverflow.com/questions/46127236/are-updated-health-checks-causing-app-engine-deployment-to-fail

  2. https://stackoverflow.com/questions/46127236/are-updated-health-checks-causing-app-engine-deployment-to-fail

  3. https://issuetracker.google.com/issues/65500706

So i am not sure if the app actually deploys with Docker code specified above.
I think we have to run /add this line to docker:
RUN python app/server.py

But when we do, the following error.
I think the main error is this now:
Step 7/9 : RUN python app/server.py
—> Running in a9d5c8b2540e
Traceback (most recent call last):
File “app/server.py”, line 8, in
from fastai.vision import *
File “/usr/local/lib/python3.6/site-packages/fastai/vision/init.py”, line 5, in
from .data import *
File “/usr/local/lib/python3.6/site-packages/fastai/vision/data.py”, line 4, in
from .transform import *
File “/usr/local/lib/python3.6/site-packages/fastai/vision/transform.py”, line 233, in
_solve_func = getattr(torch, ‘solve’, torch.gesv)
AttributeError: module ‘torch’ has no attribute ‘gesv’
The command ‘/bin/sh -c python app/server.py’ returned a non-zero code: 1
ERROR
ERROR: build step 0 “gcr.io/cloud-builders/docker” failed: exit status 1

1 Like

Your problem is at line
_solve_func = getattr(torch, ‘solve’, torch.gesv)

torch.gesv has been deprecated. Try changing that to torch.solve

From: https://github.com/pytorch/pytorch/releases

Removed Use Instead
btrifact lu
btrifact_with_info lu with get_infos=True
btrisolve lu_solve
btriunpack lu_unpack
gesv solve
pstrf cholesky
potrf cholesky
potri cholesky_inverse
potrs cholesky_solve
trtrs triangular_solve
2 Likes

Thanks, I also had the same problem.
If I manually make the change, the error disappears.
However, I’m using fastai from anaconda and the latest version there is 1.0.55, which does not have the change in fastai/vision/transform.py (1.0.56 does).
Not sure if 1.0.56 is available in the app engine (didn’t try yet).

1 Like