ValueError: [E088] Text of length 70563384 exceeds maximum of 1000000

[zerotosingularity (Jan Van de Poel)]

This might also be helpful:

How can I increase the SpacyTokenizer max_length?

Hello everyone, I am new to NLP and fastai. I was trying to follow along with Chapter 10 in the fastai book, but instead of using the IMDB reviews, I wanted to use reddit data. The corpus itself is very large. When I try to create a dataloader, it gives me the following error: Process Process-1: Traceback (most recent call last): File "/home/jayanth/anaconda3/envs/fastai_nlp/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/jayanth/anaconda3/…