Everything seems to be ok, and still, pytorch uses the GPU just one time after every reboot, then falls back to the CPU.
2 Likes
what makes you think that it is using CPU over GPU? What times are you seeing for the learn.fit operations in lesson1 during data augmentation (3 epochs) and the fine tuning (7 epochs)?
During what I think to be GPU operation, it takes a fraction of second per iteration, while when using the cpu it takes an average of 1.8 secs per iteration. The entire process was taking a few seconds per epoch on the gpu, minutes on the cpu.
What about the wall time on your system, and what GPU are you using?
That should be fixed now. Sorry about that.
1 Like
The numbers that you show appear to be from a different area of the notebook than what I initially asked. I am more interested in the epochs later in the notebook in the corresponding sections of Data augmentation, and then fine tuning. I logged my results in this thread.
1 Like
I get the following experience when trying to get the spacy en installed - anyone had this and know how to work around it?
(D:\Anaconda3) D:\>python -m spacy download en
Collecting https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz
Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz (37.4MB)
100% |████████████████████████████████| 37.4MB 1.4MB/s
Requirement already satisfied (use --upgrade to upgrade): en-core-web-sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz in d:\anaconda3\lib\site-packages
Requirement already satisfied: spacy>=2.0.0a18 in d:\anaconda3\lib\site-packages (from en-core-web-sm==2.0.0)
Requirement already satisfied: numpy>=1.7 in d:\anaconda3\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: murmurhash<0.29,>=0.28 in d:\anaconda3\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: cymem<1.32,>=1.30 in d:\anaconda3\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Requirement already satisfied: preshed<2.0.0,>=1.0.0 in d:\anaconda3\lib\site-packages (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Collecting thinc<6.11.0,>=6.10.1 (from spacy>=2.0.0a18->en-core-web-sm==2.0.0)
Exception:
Traceback (most recent call last):
File "D:\Anaconda3\lib\site-packages\pip\basecommand.py", line 215, in main
status = self.run(options, args)
File "D:\Anaconda3\lib\site-packages\pip\commands\install.py", line 324, in run
requirement_set.prepare_files(finder)
File "D:\Anaconda3\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "D:\Anaconda3\lib\site-packages\pip\req\req_set.py", line 554, in _prepare_file
require_hashes
File "D:\Anaconda3\lib\site-packages\pip\req\req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "D:\Anaconda3\lib\site-packages\pip\index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "D:\Anaconda3\lib\site-packages\pip\index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "D:\Anaconda3\lib\site-packages\pip\index.py", line 568, in _get_pages
page = self._get_page(location)
File "D:\Anaconda3\lib\site-packages\pip\index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "D:\Anaconda3\lib\site-packages\pip\index.py", line 811, in get_page
inst = cls(resp.content, resp.url, resp.headers)
File "D:\Anaconda3\lib\site-packages\pip\index.py", line 731, in __init__
namespaceHTMLElements=False,
TypeError: parse() got an unexpected keyword argument 'transport_encoding'
Looks like your anaconda isn’t installed in your home dir. That may cause permissions problems - at the very least, it’s untested.
1 Like
Oh, OK, thank Jeremy.
I installed anaconda on a different drive as I am running out of space on my c: drive.
Can I force my D:\Anaconda3 directory to be my home directory?
When you install it gives you that option - it’s mentioned in the instructions in this post.
Hello and thanks again @jeremy for this 2018 course! I am from last year instalment as a remote student and have a perfectly working win 10 + GTX 1070 laptop, caveat being all packages are 6-8 months old (it was very hard to get them all up and running ehehe). So my humble question to you and fellow students is: how do I update my Anaconda & all the Python packages in the smoothest manner in order to work with the new 2018 course without destroying my environment? All the best, Gius.
Hi @jeremy
I had already installed Anaconda and for this user only, and specified the D:\Anaconda3 as my home directory. But when I interrogate this setting in Python this is the result (and yes I really didn’t change the default user name):
from pathlib import Path
str(Path.home())
'C:\\Users\\User'
Under that directory are all of the “dot” directories like ,anaconda,.conda, .ipython, .python, .jupyter, .keras etc.
But under Anaconda3 are directories like Lib, libs, pkgs, etc.
If I type sys.path then it does indeed show all of the Anaconda directories first, though I don’t know why some specific directories and .egg files are included.
['',
'D:\\Anaconda3\\python36.zip',
'D:\\Anaconda3\\DLLs',
'D:\\Anaconda3\\lib',
'D:\\Anaconda3',
'D:\\Anaconda3\\lib\\site-packages',
'D:\\Anaconda3\\lib\\site-packages\\Babel-2.5.0-py3.6.egg',
'D:\\Anaconda3\\lib\\site-packages\\Mako-1.0.7-py3.6.egg',
'D:\\Anaconda3\\lib\\site-packages\\pattern-2.6-py3.6.egg',
'D:\\Anaconda3\\lib\\site-packages\\cherrypy-13.1.0-py3.6.egg',
'D:\\Anaconda3\\lib\\site-packages\\python_docx-0.8.6-py3.6.egg',
'D:\\Anaconda3\\lib\\site-packages\\pdfminer.six-20170720-py3.6.egg',
'D:\\Anaconda3\\lib\\site-packages\\feedparser-5.2.1-py3.6.egg',
'D:\\Anaconda3\\lib\\site-packages\\mysqlclient-1.3.12-py3.6-win-amd64.egg',
'D:\\Anaconda3\\lib\\site-packages\\backports.csv-1.0.5-py3.6.egg',
'D:\\Anaconda3\\lib\\site-packages\\future-0.16.0-py3.6.egg',
'D:\\Anaconda3\\lib\\site-packages\\pypiwin32-220-py3.6-win-amd64.egg',
'D:\\Anaconda3\\lib\\site-packages\\portend-2.2-py3.6.egg',
'D:\\Anaconda3\\lib\\site-packages\\cheroot-6.0.0-py3.6.egg',
'D:\\Anaconda3\\lib\\site-packages\\pycryptodome-3.4.7-py3.6-win-amd64.egg',
'D:\\Anaconda3\\lib\\site-packages\\tempora-1.10-py3.6.egg',
'D:\\Anaconda3\\lib\\site-packages\\more_itertools-4.0.1-py3.6.egg',
'D:\\Anaconda3\\lib\\site-packages\\win32',
'D:\\Anaconda3\\lib\\site-packages\\win32\\lib',
'D:\\Anaconda3\\lib\\site-packages\\Pythonwin',
'D:\\Anaconda3\\lib\\site-packages\\IPython\\extensions',
'C:\\Users\\User\\.ipython']
I suspect that when trying to compile the english dictionary support that spacy is making an assumption about the location of files in relation to the running python.
I cannot really believe it. I queried the GPU just before running the model, and it responds it’s all ok. Still, it has something like a 10 minutes ETA, while it takes less than 15 seconds when the GPU is properly initialized.
Any idea about what’s happening?
3 Likes
I’ve created an “Anaconda Prompt (fastai)” shortcut. Might be something to create when installing fastai (or the option)?
note this works after activating fastai. otherwise I got a spacy not found kindof message
1 Like
Please help diagnose my “torch.cuda.is_available()” -? false.
cudnn is true
I tried a reboot. no change.
I looked into source - no clues there, calls torch._C._cuda_isDriverSufficient() which is false, but its builtin - no code
looking at some of the suggestions here - path in my anaconda fastai cmd has ~6 anaconda fastai directories first
C:\Users\Brad\Anaconda2\pkgs has cuda90-1.0-h4c72538_0
my anaconda fastai cmd does not show any CUDA enviroment variables (via “set”)
the only other thing I can think of is I did most of the installs last night, but put a NVIDIA GPU in the PC today. conda env update has run
@balnazzar maybe not just you 
thanks for any suggestions / ideas
if this happens to you… try a 2nd reboot. I searched fruitlessly for ages for something to try - nothing. turned the computer off. try again.
“torch.cuda.is_available()” -? true.
It happens. In my case, reboots didn’t help, a fresh installation of (systemwide) cuda 9 solved the issue.
I think it wasn’t cuda 9 in itself. Rather, it straightened my env vars or kind of.
1 Like
I had gone through a pretty tedious/painful installation last year as well (following this very helpful guide: https://github.com/philferriere/dlwin), so I had the exact same fear. However, I found that following the instructions exactly as written here worked like a charm. Good luck!
1 Like
yep. it was a pretty good feeling when it all works, so I’d encourage people not to be afraid of this on Windows now. Thanks again fastai team!
I created a little batch to popup a window showing the nvidia stats which is useful to confirm power loads, fan and temperature etc - especially if a new pc / gpu
Got this working on Windows 7, with some fixes required.
-
defaults channel had to be disabled, otherwise ipykernel cannot be installed properly due to an error related to defaults::ipykernel-4.8.0-py36_0 (https://github.com/ContinuumIO/anaconda-issues/issues/8087)
conda update conda
conda update ipykernel
conda env update # after commenting out defaults in environment.yml -
install the VC2017/VC2015 redistributables or runtimes
-
One machine repeatedly crashes at the opencv commands. This only went away after applying all available Windows 7 updates (including .net framework 4.6.1), and re-installing opencv-python. Exact cause is still unknown.
3 Likes