Solved by compiling pyTorch from source. Here is how to do so in details:
1- Activate the virtual env:
source activate fastai
2- Uninstall pyTorch
conda uninstall pytorch
3- Clone pyTorch repo:
git clone --recursive https://github.com/pytorch/pytorch
4- Checkout to the latest branch (so that we install pytorch 0.3.1)
cd pytorch
git checkout v0.3.1
5- Sort out dependencies (I only needed cmake):
git submodule update --init
sudo apt install cmake
6- Install, this is were compiling takes place (took nearly 30 minutes):
python setup.py install