For text or tabular data is there a way to speed up inference from using .predict() one by one?
Specific context for me:
My user-facing application involves inferring on a selection of text examples of the same class, and then taking a weighted average of those predictions as the final prediction. I do that about 100 times, each one taking about 0.5s.
The same application requires me to predict on tabular data, about 100 independent predictions with the same model. Each takes about 0.2s.
Would using pred_batch or something similar be faster? How can I turn a list of strings or a dataframe into the right type of dataset for pred_batch?
