Error while predicting, shape of passed values is (1,130), indices imply (1, 127)

Deep Learning
1

Hello everyone,

I’m using a model deployed into production and i’m running into this issue when i call the function learn.predict(). The model is a TabularLearner that works fine in my Jupyter environment so I assume i’m doing something wrong in production, but i can’t get my finger on it.

This is the stack trace:
Traceback (most recent call last):
File “/project/venv/lib/python3.8/site-packages/fastai/tabular/core.py”, line 333, in decodes
try: df = pd.DataFrame(vals, columns=self.to.all_col_names)
File “/project/venv/lib/python3.8/site-packages/fastcore/foundation.py”, line 153, in getattr
if attr is not None: return getattr(attr,k)
File “/project/venv/lib/python3.8/site-packages/fastcore/transform.py”, line 202, in getattr
def getattr(self,k): return gather_attrs(self, k, ‘fs’)
File “/project/venv/lib/python3.8/site-packages/fastcore/transform.py”, line 165, in gather_attrs
if not res: raise AttributeError(k)
AttributeError: all_col_names

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/project/venv/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1654, in create_block_manager_from_blocks
    make_block(values=blocks[0], placement=slice(0, len(axes[0])))
  File "/project/venv/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 3049, in make_block
    return klass(values, ndim=ndim, placement=placement)
  File "/project/venv/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 124, in __init__
    raise ValueError(
ValueError: Wrong number of items passed 130, placement implies 127

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "calcul_prediccions_model_nou.py", line 366, in <module>
    print(learn.predict(df.iloc[5]))
  File "/project/venv/lib/python3.8/site-packages/fastai/tabular/learner.py", line 21, in predict
    full_dec = self.dls.decode(b)
  File "/project/venv/lib/python3.8/site-packages/fastai/data/core.py", line 80, in decode
    def decode(self, b): return self.before_batch.decode(to_cpu(self.after_batch.decode(self._retain_dl(b))))
  File "/project/venv/lib/python3.8/site-packages/fastcore/transform.py", line 206, in decode
    if full: return compose_tfms(o, tfms=self.fs, is_enc=False, reverse=True, split_idx=self.split_idx)
  File "/project/venv/lib/python3.8/site-packages/fastcore/transform.py", line 150, in compose_tfms
    x = f(x, **kwargs)
  File "/project/venv/lib/python3.8/site-packages/fastcore/transform.py", line 114, in decode
    def decode(self, x, **kwargs):   return self._call1(x, 'decode', **kwargs)
  File "/project/venv/lib/python3.8/site-packages/fastcore/transform.py", line 117, in _call1
    y = getattr(super(), name)(list(x), **kwargs)
  File "/project/venv/lib/python3.8/site-packages/fastcore/transform.py", line 74, in decode
    def decode  (self, x, **kwargs): return self._call('decodes', x, **kwargs)
  File "/project/venv/lib/python3.8/site-packages/fastcore/transform.py", line 83, in _call
    return self._do_call(getattr(self, fn), x, **kwargs)
  File "/project/venv/lib/python3.8/site-packages/fastcore/transform.py", line 89, in _do_call
    return retain_type(f(x, **kwargs), x, ret)
  File "/project/venv/lib/python3.8/site-packages/fastcore/dispatch.py", line 112, in __call__
    return f(*args, **kwargs)
  File "/project/venv/lib/python3.8/site-packages/fastai/tabular/core.py", line 334, in decodes
    except: df = pd.DataFrame(vals, columns=self.to.x_names)
  File "/project/venv/lib/python3.8/site-packages/fastai/torch_core.py", line 466, in __init__
    self._old_init(data, index=index, columns=columns, dtype=dtype, copy=copy)
  File "/project/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 464, in __init__
    mgr = init_ndarray(data, index, columns, dtype=dtype, copy=copy)
  File "/project/venv/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 210, in init_ndarray
    return create_block_manager_from_blocks(block_values, [columns, index])
  File "/project/venv/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1664, in create_block_manager_from_blocks
    construction_error(tot_items, blocks[0].shape[1:], axes, e)
  File "/project/venv/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1694, in construction_error
    raise ValueError(f"Shape of passed values is {passed}, indices imply {implied}")
ValueError: Shape of passed values is (1, 130), indices imply (1, 127)

So far i’ve worked around this problem by modifying the predict function in the TabularLearner class, (Commeting the line that caused the issue and modifying the return to just yield dec_preds and preds) but this is obviously not and ideal solution and i was hoping to find some help here.

Thanks everyone!

2

How many cat and cont variables do you have in your dataset?

3

107 cont variables and 20 cat variables if i remember correctly.

Thanks for replying!

4

I’ve been doing some tinkering and I managed to fix the issue by updating my pandas version from 1.0.4 to 1.1.0. Just wanted to share it in case this happens to someone else.

@muellerzr, don’t know if you looked into it but the problem appears to be fixed, thanks anyway! :smiley:

1 Like