Ok solved by doing this in the nbs directory:
ln -s /home/marco/miniconda3/envs/TEST/lib/python3.7/site-packages/* .
And also modify .gitignore:
# Ignore the symlinks in the nbs folder, but include the notebooks
**/nbs/*
!**/nbs/*.ipynb
But then why does the build libs work? Is build docs called in a different way?
Anyway, for now this will be fine, now I can start building stuff using this awesome tool 
because one of the main reasons I want to use it is because I have never in the past managed to write proper documentation and it ends up all over the place