Error while multiple threads execute learner.get_preds()

fastai
1

Hi all, I have trained a text classification model and i have exported the model into a pickel file : ‘export.pkl’.
Now i want to deploy this model, and for that i’m using flask to get an endpoint.
So whenever anyone hits the flask endpoint, they send the text data that they want to be predicted in the form of a payload.
Input payload looks like: {“inputData” : [“Text-1”,“Text-2”,“Text-3”,“Text-4”]}
So , i use following statements to load the model and get the predictions.
> learner = load_learner(path=BASE)
> learner.data.add_test([“Text-1”,“Text-2”,“Text-3”,“Text-4”])
> preds,y = learner.get_preds(ds_type=DatasetType.Test)
> preds,y

This way i get the predictions.

But the problem comes, when concurrent users hit the flask end-point and triggers the model. Then the execution of learner.get_preds() breaks. Because when concurrent requests comes to flask, flask start a separate thread to execute a request.
So i’m not able to use learner.get_preds() in a multithreaded or multiprocessing environment.

Can anyone help me with this issue. Because i want learner.get_preds() to execute even when concurrent request comes. And also, i don’t want to store the request in a queue and execute one at a time because it will cause a latency issue.
Please Help!
Thanks in advance.

1 Like
2

How does it break? Can you show the error message?

3

This is the error I get when multiple threads execute learner.get_preds()

preds,y = self.learner.get_preds(ds_type=DatasetType.Test)
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/fastai/text/learner.py", line 87, in get_preds
    preds = super().get_preds(ds_type=ds_type, with_loss=with_loss, n_batch=n_batch, pbar=pbar)
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/fastai/basic_train.py", line 336, in get_preds
    activ=_loss_func2activ(self.loss_func), loss_func=lf, n_batch=n_batch, pbar=pbar)
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/fastai/basic_train.py", line 44, in get_preds
    zip(*validate(model, dl, cb_handler=cb_handler, pbar=pbar, average=False, n_batch=n_batch))]
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/fastai/basic_train.py", line 59, in validate
    val_loss = loss_batch(model, xb, yb, loss_func, cb_handler=cb_handler)
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/fastai/basic_train.py", line 26, in loss_batch
    out = model(*xb)
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/fastai/text/learner.py", line 263, in forward
    r, o = self.module(input[:,i: min(i+self.bptt, sl)])
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/fastai/text/models/awd_lstm.py", line 117, in forward
    raw_output, new_h = rnn(raw_output, self.hidden[l])
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/fastai/text/models/awd_lstm.py", line 52, in forward
    return self.module.forward(*args)
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 564, in forward
    return self.forward_tensor(input, hx)
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 543, in forward_tensor
    output, hidden = self.forward_impl(input, hx, batch_sizes, max_batch_size, sorted_indices)
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 523, in forward_impl
    self.check_forward_args(input, hx, batch_sizes)
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 500, in check_forward_args
    'Expected hidden[0] size {}, got {}')
  File "/home/vishal/Downloads/Testing/venv/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 166, in check_hidden_size
    raise RuntimeError(msg.format(expected_hidden_size, tuple(hx.size())))
RuntimeError: Expected hidden[0] size (1, 12, 400), got (1, 11, 400)