BENTOML + FAST.AI prediction problems

Deep Learning
1

Hi,
I am studying BentoML for the last few days, and found it very useful. Thanks !
I am not an advanced user of fast.ai and bentoml, but it seems that there is a problem with the predictions of FastaiModelArtifact both for fastai 1.0 and 2.0
I rechecked this many times, and I am pretty sure of the below described problem :

For FASTAI v1.0

following the original notebook at (https://github.com/bentoml/gallery/blob/master/fast-ai/salary-range-prediction/fast-ai-salary-range-prediction.ipynb) - using the same dataset.

A – Fast.ai predict -

In this version, the problem can be seen in the output of Line 11:
Line 11 - learn.predict(row)
Out[11]:
(Category tensor(1), tensor(1), tensor([0.4633, 0.5367]))

If you run learn.data.c2i , you can get the decodification for the output -
{’<50k’: 0, ‘>=50k’: 1} ---- therefore the prediction of 1 – corresponds to ‘>=50k’

B- Bentoml predict -

From the REST API Model Serving block, you can see that the prediction was {‘data’: ‘["<50k"]’…….
And this does not match the prediction obtained without BentoML

For FASTAI v2.0

– based on bentoml notebooks, I modified it to fastai V2.0, as below:
%%writefile tabular_csv.py

import bentoml,json
import pandas as pd
from bentoml.frameworks.fastai import FastaiModelArtifact
from bentoml.adapters import FileInput,DataframeInput
@bentoml.artifacts([FastaiModelArtifact(‘learner’)])
@bentoml.env(infer_pip_packages=True)
class FastaiTabularModel(bentoml.BentoService):

@bentoml.api(input=DataframeInput(), batch=True)  ## preisa ser True
def predict(self, df):
    dl = self.artifacts.learner.dls.test_dl(df.iloc[:1,:],rm_type_tfms=None, num_workers=0,with_input=True)
    print("dl.xs= ",dl.xs)
    result  = self.artifacts.learner.get_preds(dl=dl)
    return [[result]]
  1. import the custom BentoService defined above
    from tabular_csv import FastaiTabularModel
  2. pack it with required artifacts
    svc = FastaiTabularModel()
    svc.pack(‘learner’, learn)
  3. save your BentoSerivce
    saved_path = svc.save()

A – Fast.ai predict

dl = learn.dls.test_dl(df.iloc[:1], rm_type_tfms=None, num_workers=0)
learn.get_preds(dl=dl,with_input=True,with_decoded=True)
Out[ ]:
((tensor([[5, 8, 3, 0, 6, 5, 1]]), tensor([[ 0.7681, -0.8398, 0.7477]])),
tensor([[0.2092, 0.7908]]),
tensor([[1]], dtype=torch.int8),
tensor([1]))

get_preds predicts “1” and also gives its encoded input and output class probabilities

B- Bentoml predict – using bentoml serve

I get the following output from serve:

(…)
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.

  • Debug mode: on
  • Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
    dl.xs= workclass education marital-status … age fnlwgt education-num
    0 0 0 0 … 0.768098 -0.83979 0.747676

[1 rows x 10 columns]
[2021-03-05 13:27:04,337] INFO - {‘service_name’: ‘FastaiTabularModel’, ‘service_version’: ‘20210305132610_643B13’, ‘api’: ‘predict’, ‘task’: {‘data’: ‘[{\n “age”: 49,\n “workclass”: “Private”,\n “fnlwgt”: 101320,\n “education”: “Assoc-acdm”,\n “education-num”: 12.0,\n “marital-status”: “Married-civ-spouse”,\n “occupation”: “”,\n “relationship”: “Wift”,\n “race”: “White”,\n “sex”: “Female”,\n “capital-gain”: 0,\n “capital-loss”: 1902,\n “hours-per-week”: 40,\n “native-country”: “United-States”,\n “salary”: “>=50k”\n}]’, ‘task_id’: ‘15783505-fcff-46a1-a0d5-e6d75486885d’, ‘batch’: 1, ‘http_headers’: ((‘Host’, ‘127.0.0.1:5000’), (‘User-Agent’, ‘curl/7.69.1’), (‘Accept’, ‘/’), (‘Content-Type’, ‘application/json’), (‘Content-Length’, ‘370’))}, ‘result’: {‘data’: ‘[[[{“unknown_obj”: “tensor([[0.6520, 0.3480]])”}, {“unknown_obj”: “tensor([[1]], dtype=torch.int8)”}]]]’, ‘http_status’: 200, ‘http_headers’: ((‘Content-Type’, ‘application/json’),)}, ‘request_id’: ‘15783505-fcff-46a1-a0d5-e6d75486885d’}
127.0.0.1 - - [05/Mar/2021 13:27:04] “POST /predict HTTP/1.1” 200 -
^C

I printed , as above, the output of self.artifacts.learner.dls.test_dl -and you can see that dl.xs gives the same encoded input as the regular fast.ai learn .
However the self.artifacts.learner.get_preds gives completely different probabilities. The class prediction is correct “1” , but it does not match the probabilities - it should have decided by “0”

Again, Fastai + BentoML does not get the same results as Fastai alone.

As described, this happens with both fastai 1.0 and 2.0