which runs without an issue. As soon as I swap out the tok_func to SentencePieceTokenizer instead of spacy, learn.predict results in an error:
~/fastai/fastai2/fastai2/text/core.py in __call__(self, items)
371
372 def __call__(self, items):
--> 373 for t in items: yield self.tok.EncodeAsPieces(t)
374
375 # Cell
AttributeError: 'NoneType' object has no attribute 'EncodeAsPieces'
I do see that self.tok is indeed None. Has anybody come across this before?
The commit of fastcore and fastai2 that I am running:
fastcore: 4a2d5ea702d0dc4a6c34c4acefafd9b494d9e222
fastai2: bf455de9bc37c76f7f92b3c43227ef9d4779b614