FastAPI / Starlette issue: AttributeError: '_FakeLoader' object has no attribute 'noops'

Part 1 (2020) Non-beginner
1

So attempting to put the bears classifier behind a FastAPI/Starlette backend but have been struggling with this error for the past few hours. Any ideas???

@app.post("/predict")
async def predict(image: UploadFile=File(...)):
    img_bytes = await image.read()
    pred_class, pred_class_idx, probs = inf_learn.predict(img_bytes)

starts recursively throwing this error:

api-dev_1   |   Traceback (most recent call last):
api-dev_1   |   File "<string>", line 1, in <module>
api-dev_1   |   File "/usr/local/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
api-dev_1   |     exitcode = _main(fd)
api-dev_1   |   File "/usr/local/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
api-dev_1   |     self = reduction.pickle.load(from_parent)
api-dev_1   | AttributeError: '_FakeLoader' object has no attribute 'noops'

I’ve tried several ways to create the input for inf_learn.predict including converting img_bytes to a numpy array and a PILImage … all of them throw the same exception.

1 Like
2

So I pieced this out into a much simpler application … and while it’s still not working, I was able to get a better stack trace.

Any ideas?

Here again is the call:

@app.post("/predict")
async def predict(image: UploadFile=File(...)):
    img_bytes = await image.read()
    pred_class, pred_class_idx, probs = inf_learn.predict(img_bytes)

    return {
        "predicted_class": pred_class,
        "probability": classes[probs]
    }

… and after all the complaints about _FakeLoader not having a noops attribute, I get this:

INFO:     127.0.0.1:52607 - "POST /predict HTTP/1.1" 500 Internal Server Error                                       
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 385, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastapi/applications.py", line 149, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/starlette/applications.py", line 102, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/starlette/routing.py", line 550, in __call__
    await route.handle(scope, receive, send)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastapi/routing.py", line 166, in app
    dependant=dependant, values=values, is_coroutine=is_coroutine
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastapi/routing.py", line 119, in run_endpoint_function
    return await dependant.call(**values)
  File "./main.py", line 40, in predict
    pred_class, pred_class_idx, probs = inf_learn.predict(img_bytes)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastai2/learner.py", line 231, in predict
    inp,preds,_,dec_preds = self.get_preds(dl=dl, with_input=True, with_decoded=True)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastai2/learner.py", line 219, in get_preds
    self._do_epoch_validate(dl=dl)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastai2/learner.py", line 178, in _do_epoch_validate
    dl,*_ = change_attrs(dl, names, old, has);       self('after_validate')
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastai2/learner.py", line 124, in __call__
    def __call__(self, event_name): L(event_name).map(self._call_one)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastcore/foundation.py", line 372, in map
    return self._new(map(g, self))
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastcore/foundation.py", line 323, in _new
    def _new(self, items, *args, **kwargs): return type(self)(items, *args, use_list=None, **kwargs)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastcore/foundation.py", line 41, in __call__
    res = super().__call__(*((x,) + args), **kwargs)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastcore/foundation.py", line 314, in __init__
    items = list(items) if use_list else _listify(items)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastcore/foundation.py", line 250, in _listify
    if is_iter(o): return list(o)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastcore/foundation.py", line 216, in __call__
    return self.fn(*fargs, **kwargs)
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastai2/learner.py", line 127, in _call_one
    [cb(event_name) for cb in sort_by_run(self.cbs)]
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastai2/learner.py", line 127, in <listcomp>
    [cb(event_name) for cb in sort_by_run(self.cbs)]
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastai2/callback/core.py", line 24, in __call__
    if self.run and _run: getattr(self, event_name, noop)()
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastai2/callback/core.py", line 95, in after_validate
    if self.with_input:     self.inputs  = detuplify(to_concat(self.inputs, dim=self.concat_dim))
  File "/Users/admin/Development/_tools/anaconda/envs/fastapi_test_app/lib/python3.7/site-packages/fastai2/torch_core.py", line 213, in to_concat
    if is_listy(xs[0]): return type(xs[0])([to_concat([x[i] for x in xs], dim=dim) for i in range_of(xs[0])])
IndexError: list index out of range
1 Like
3

Hey I’m doing something similar* and have it working. Maybe it can help:
https://github.com/sutt/fastai2-dev/tree/master/chess-classification-hw/app

*: predserver.py loads a model pkl and serves predictions using Flask. camclient.py posts images to the server from a running webcam using OpenCV.
I need to run fastai2 in my linux vm (WSL) but need to run the client in windows to have hardware support for the webcam.

4

Nice setup there!

Unfortunately, I’m not getting everything back from learn.predict though the code is almost identical (I’m using FastAPI/Starlette).

from fastapi import FastAPI, File, UploadFile
from fastai2.vision.all import *

path = Path(__file__).parent
app = FastAPI()

inf_learn = load_learner(path/'export.pkl')
classes = list(inf_learn.dls.vocab)

@app.post("/predict")
async def predict(image: UploadFile=File(...)):
    img_bytes = await image.read()
    pred_class, pred_idx, probs = inf_learn.predict(img_bytes)

    return {
        "class": pred_class,
        "pred_idx": pred_idx,
        "probs": probs,
        "classes": classes
    }

Here’s the weird part.

It returns the pred_class just fine … but for everything else, it returns an empty dictionary:

{
    "class": "grizzly",
    "pred_idx": {},
    "probs": {},
    "classes": [
        "black",
        "grizzly",
        "teddy"
    ]
}

No errors … but no pred_idx or probs.

5

Hmm, I’d suggest trying to isolate the problem between the BytesIO aspect and fastai, by replicating my ugly process of bringing in numpy and cv2 to decode the request data. If that works, you know the problem is in encode/decode step, but if that doesn’t work, it points towards fastai’s learner.predict method being off.

image
image1468Ă—398 15 KB

6

If there was a problem with the input data (bytes in my case though I’ve tried others), why would I be getting the predicted class returned correctly?

It seems to point to something wrong with the learner.predict method

7

Can you try it in a notebook regularly?

8

Works just fine (just using the ch.2 code out of the box).

Screen Shot 2020-04-03 at 5.21.39 PM
Screen Shot 2020-04-03 at 5.21.39 PM1208Ă—754 53.9 KB

I’ve also tried passing a file path to my inference learner in my FastAPI app … I get the same results (like we do in the chapter 2 notebook)

9

SOLVED:

How this was crazy and weird.

So the problem was not with learner.predict but with how FastAPI (and maybe Starlette) behaves when attempting to return a type that is a pytorch tensor. For whatever reason, it doesn’t throw an error … it just says, “Oh I see you have something I’m not sure about … I’ll give you an empty dictionary”.

So the fix is this:

    return {
        "class": pred_class,
        "pred_idx": pred_idx.item(),
        "probs": probs.tolist(),
        "classes": classes
    }

pred_ix and probs are both tensor objects … and FastAPI doesn’t like them. Unfortunately, instead of throwing an exception, it just converts them to {}.

With the fix in place above all works. Here’s an example of what I get:

{
    "class": "teddy",
    "pred_idx": 2,
    "probs": [
        9.78011826191505e-07,
        3.243550509068882e-06,
        0.9999958276748657
    ],
    "classes": [
        "black",
        "grizzly",
        "teddy"
    ]
}
4 Likes