Serving predictions on demand

I brought this up in another thread, but breaking it out on its own since I don’t see much discussion around it. I understand the course focuses on the details of deep learning, but I am struggling attempting to use these models in real-world scenarios. Essentially, I want a REST endpoint that can accept a image or image url, pass this image through the already trained model, and return a response containing the predictions. RIght now I am failing just to get the image predicted properly. Please see my current non-working example below.

Is there a different/standard way to do this? It seems like this should be a fairly common practice. Along these lines, when we save a model to an ‘h5’ file, is this model readable in other libraries? If so, should I consider the serving endpoint functionality with those?

Feedback is greatly appreciated.

Failing code below

from fastai.conv_learner import *
from planet import f2

PATH = 'data/shopstyle/'

f_model = resnet34
label_csv = f'{PATH}prod_train.csv'
n = len(list(open(label_csv)))-1
val_idxs = get_cv_idxs(n)

sz = 128

def get_data(sz):
    tfms = tfms_from_model(f_model, sz, aug_tfms=transforms_side_on, max_zoom=1.05)
    return ImageClassifierData.from_csv(PATH, 'train', label_csv, tfms=tfms, suffix='.jpg', val_idxs=val_idxs, test_name='test')

def print_list(list_or_iterator):
        return "[" + ", ".join( str(x) for x in list_or_iterator) + "]"

data = get_data(sz)

print("Loading model...")
learn = ConvLearner.pretrained(f_model, data, metrics=metrics)

trn_tfms, val_tfms = tfms_from_model(f_model, sz)
im = val_tfms(np.array('{PATH}valid/4500132.jpg')))
preds = learn.predict_array(im[None])
print("predictions = " + preds

The error

(fastai) ubuntu@ip-172-31-34-254:~/fastai/courses/dl1$ python
Loading model...
Traceback (most recent call last):
  File "", line 34, in <module>
    preds = learn.predict_array(im[None])
  File "/home/ubuntu/fastai/courses/dl1/fastai/", line 266, in predict_array
    def predict_array(self, arr): return to_np(self.model(V(T(arr).cuda())))
  File "/home/ubuntu/src/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/src/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/", line 67, in forward
    input = module(input)
  File "/home/ubuntu/src/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/src/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/", line 37, in forward, self.momentum, self.eps)
  File "/home/ubuntu/src/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/", line 1011, in batch_norm
    raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
ValueError: Expected more than 1 value per channel when training, got input size [1, 1024]

Not sure about standard way but you can put together a fairly vanilla flask app to do what you want. You can define routes eg /score_image that accepts image, path whatever. Afterwards, the reply to the request can be the output of your model (maybe in some pretty format).

I googled flask + tensorflow and this was the top result:

Hope that’s useful

Thank you. This is similar to what we have planned, but I think we’re looking at django-rest. What I really need to get past right now is hot to properly predict the single image.

1 Like

Since I am still stuck on this, I went a different route so I can at least get a prediction out of the model. So I placed a single image in the test folder and asked to get the test predictions. Then I just grab the single result. So far, I am happy with these results.

[('Shortsleeve', 0.33233735), ('Teens', 0.36527044), ('Tops', 0.65736127), ('Women', 0.4690754), ('Young_Women', 0.27735448)]