Using QRNN in Language Models


#21

This was very specific since there is a part of the model where we need to use C-accelerated libraries (here cupy) because the QRNN block isn’t implemented in pytorch. For the rest, I just took the pytorch existing implementation and adapted it inside the existing modules of fastai.

I never said this, I said there was no shared pretrained QRNN model yet. On my experiments, using a QRNN model pretrained on WT-103 then finetuned on imdb worked better, as with the LSTMs.


#22

@sgugger I’ve installed cupy-cuda91 with pip according to https://docs-cupy.chainer.org/en/stable/install.html and am using fastai 1.0.39. The package itself seem to be installed and import goes through, but when I run text_classifier_learner(data_clas, qrnn=True) I get FileNotFoundError: [Errno 2] No such file or directory: ‘/home/ubuntu/.local/share/virtualenvs/ubuntu-7Wf190Ea/lib/python3.6/site-packages/fastai/text/qrnn/forget_mult_cuda.cpp’
Not sure why the file is not installed, any suggestions would be most appreciated!


#23

You don’t need cupy anymore with fastai v1 since I rewrote the c extensions. I’m not sure why you don’t have those files with fastai 1.0.39, they are definitely in the repo. Maybe try a dev install?


#24

@sgugger I am getting an error when trying to run learn.lr_find() with QRNN:

`~/SageMaker/envs/fastai/lib/python3.7/site-packages/fastai/text/qrnn/forget_mult.py in compile(self)
107 if self.ptx is None:
108
–> 109 program = _NVRTCProgram(kernel.encode(), ‘recurrent_forget_mult.cu’.encode())
110 GPUForgetMult.ptx = program.compile()
111

NameError: name ‘_NVRTCProgram’ is not defined`

Any idea why this could be the case? I install cupy in the right environment and can import it in my notebook.


#25

This is the old code, and I don’t much about cupy, so I can’t help you on this one. Did you try the new version?