Even when the problem appears, i.e. the error message appears, it did not affect my workflow working with a vision project. Dataloading was successful, the model trained, and everything else worked as expected despite the error message.
If you do not want the error message, you can either-
!pip install torchtext==0.8.1
!pip install -U fastbook
as suggested by @muellerzr here,
or you could use light-the-torch-
!pip install light-the-torch
!ltt install torch torchvision
!pip install fastai --upgrade
also suggested by him.
You should know that not using IPython magic does not make the problem go away. It was mere chance.