If you pass an absolute path as dest it should work.
Thank you Jeremy for your fast reply. Following your suggesting I tried these two options:
Option 1 : set dest to the fully qualified path. It stores the uncompressed files in the right folder but it returns the wrong path:
dsname = 'NATOPS' #'NATOPS', 'LSST', 'Wine', 'Epilepsy', 'HandMovementDirection'
path_data = Config().data
dest = path_data/dsname
dest
Path('C:/Users/fh/.fastai/data/NATOPS')
url = 'http://www.timeseriesclassification.com/Downloads/NATOPS.zip'
path = untar_data(url, dest=dest)
path
Path('C:/Users/fh/.fastai/data/NATOPS/NATOPS') <--- doesn't exist
Option 2 : set dest to the .fastai/data path. It doesn’t store the uncompressed files in a separate folder but it returns the intended path:
For both options, the returned paths don’t exist (see <--- arrows in the code snippets)
dsname = 'NATOPS' #'NATOPS', 'LSST', 'Wine', 'Epilepsy', 'HandMovementDirection'
path_data = Config().data
dest = path_data
dest
Path('C:/Users/fh/.fastai/data')
url = 'http://www.timeseriesclassification.com/Downloads/NATOPS.zip'
path = untar_data(url, dest=dest)
path
Path('C:/Users/fh/.fastai/data/NATOPS') <--- doesn't exist either
I think this comes down to the fact that the timeseries zip files don’t have an inside folder and untar_data() assumes there is one.
Thank you for your help anyway. If it isn’t possible to modify untar_data(), it’s fine; I will continue using a method that I called unzip_data() that looks like untar_data()
Yeah unless we actually look inside the file we can’t really know where it will extract to, so we have to make an assumption about what to return.
1 Like
Hi there.
Been trying tirelessly to get fastai working under Windows with GPU. It’s installed, working perfectly fine on CPU, but absolutely refuses to use GPU seemingly whatever I try. I have tried using what I assume are “up to date” conda versions of things and also going back to pytorch 1.0.0 with cuda 9.0 and cudnn 7005. Also tried cuda 10.1 and 10.2 (and pytorch 1.4.0 I think it was…?). GPU driver is 441.66 - this is the constant I haven’t changed essentially because I wouldn’t know what version to try going backwards.
I don’t get any obvious errors that are exposed via the Anaconda console or in the notebook itself. I just get CPU activity with zilch happening on the GPU. “torch.cuda.is_available()” returns true, my GPU model is returned when I do “torch.cuda.get_device_name(0)”. I’m kind of at a loss at the moment.
Is this a known issue or a me related weirdness?
I had this running on Windows a while back @Russbo, but have been running more recently in Linux. I was planning on trying Windows again before the course starts (and do a write up of anything special needed) - so I’ll let you know how it goes. I seem to remember the GPU use was very spiky and overall slower than Linux, but for what I’d expect we’d be doing on the course it should be ok.
I had the same problem while developing the audio module, where the files in some datasets are directly at the root of the tar file. The solution that I found was to modify the extract function, so that it extracts the contents to a folder with the same name as the compressed file.
The modified function is:
def tar_extract_at_filename(fname, dest):
"Extract `fname` to `dest`/`fname`.name folder using `tarfile`"
dest = Path(dest)/Path(fname).with_suffix('').name
tarfile.open(fname, 'r:gz').extractall(dest)
And I use it like:
url = "https://public-datasets.fra1.digitaloceanspaces.com/250-speakers.tar"
path = untar_data(url, extract_func=tar_extract_at_filename)
4 Likes
An update on this problem:
After reducing the number of workers for document tokenization to half the number of CPUs it almost worked (things broke just 10.000 docs shy of the 3.7 million goal).
After that I hacked a bit the code.
First I forced the method from_folder (fastai2/fastai2/text/ core.py) to always do the line tokenize_folder(path, rules=rules, **kwargs) which is now skipped if the target folder already exists (I really should have changed the skip condition to the lengths and counts files existing).
Then I changed the code in tokenize_folder. As it is, files in a source folder are read and a tokenized version is saved for each. I made it so that if the target file exists already, then it reads it and gets tokens from it in a list. Note! Don’t forget about ‘\n’ (for some reason my method has a very minor mismatch in the count of ‘\n’, but I don’t think it’s an issue).
This is the code. It’s not really very good, since this should be parallelized, but even for 3.7 million documents (with only around 10k missing tokenization) the solution was fast enough.
if os.path.isfile(out):
text = out.read()
tok = re.findall(r'\S+|\n',text)
else:
tok = tokenize1(file, tok_func=SpacyTokenizer, rules=rules,
post_rules=None, **tok_kwargs)
out.write(' '.join(tok))
So now I have my lengths, counter and lm_data files 
I am moving things to the big GPU machine to continue training the language model, I hope things work fine from now on, although I’m a bit worried about the step of creating dataloaders for classification. Well, one step at a time, I guess!
Update: Fastai does two funny things in tokenization that complicate things (and make my lengths list not match its expectations).
- New line + space = ‘\n▁’
- Repeated new lines get concatenated: \n \n --> \n\n
It’s unfortunate since postprocessing lists really is a bit of a hassle, but it seems like this can be solved by… well, by applying these operations to the tokenized lists (when reading from file).
3 Likes
Fastai v2 is quite different it turns out. My v1 notebook is basically a modified version of class 3 planets from 2019. Transforms works differently, looks like ImageList might be gone.
Would rather just use v1 for the minute…
EDIT:
Ah - it seems it is working, but pretty poorly. With image size 256 from the plane dataset I’m using, I’m still getting 60% CPU utilisation, but 33% spikes of CUDA. GPU memory is also ~10GB/11GB. So it’s doing something.
Perhaps this is what @brismith was alluding to in terms of spiky utilisation.
I had some problems with GPU utilization, but now I managed to stay above 90% most of the time. Reading from text files or from a Dataframe.
x_bwd_tfms = [attrgetter("lm_relatorio"), rules, Tokenizer(tokenizer=sp), Numericalize(vocab), backwards]
clas_splits = RandomSplitter(valid_pct=0.1, seed=42)(df)
bwd_dsrc_clas = Datasets(df, splits=clas_splits, tfms=[x_bwd_tfms, [attrgetter("pred_resultado"), Categorize()]], dl_type=SortedDL)
bwd_dbunch_clas = bwd_dsrc_clas.dataloaders(bs=bs, cache=2048, num_workers=8, pin_memory=True, shuffle_train=True, before_batch=pad, seq_len=72)
These are the settings I have been using with very good results in terms of GPU performance.
For eg.: I have a large Dataset (20MM) texts in text files, and each epoch takes about 5 hours with GPU utilization around 95% most of the time. The painful part is that the Dataloader takes another 5 hours to parse the texts. But the model runs very smoothly. For the very large Dataset, the shuffle_train=False is also essential. Code:
def read_file(f): return L(SPieceTokenizer(f.read(),sp))
tfms = [read_file,Numericalize(vocab)]
splits = RandomSplitter(valid_pct=0.1, seed=42)(texts)
%time dsrc = DataSource(texts, [tfms], splits=splits, dl_type=LMDataLoader)
dbunch_lm = dsrc.databunch(bs=bs, seq_len=sl, val_bs=bs, num_workers=2, pin_memory=True, shuffle_train=False)
2 Likes
With the latest version of fastai 0.0.11 and fastcore (0.1.13), If I try to use to_fp16() I get always an error when finishing the 2nd epoch during training. Not using fp16() avoids the error.
Error message:
RuntimeError: param_from.type() == param_to.type() INTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/ATen/native/cudnn/RNN.cpp:541, please report a bug to PyTorch. parameter types mismatch
At the bottom of the doc created by nbdev it shows ©2020 Your Name or Company Name. All rights reserved. I uncommented and modified in settings.ini the copyright but nothing changes. I think that the settings.ini is fixed and when I ran nbdev_build it won’t reload the settings. How can I fix it ? thanks
Thanks for sharing! I’m very interested in this
do you have any idea how relevant this is?
In my tests, for the massive Dataset (20MM) without shuffle_train=False, the performance drops to unbearable levels (too much time without GPU processing - 0% and much bigger training time).
Using the low level API, some decisions impact the processing time very abruptly.
For eg: testing the imdb notebook from the fastai course, the running times can be very different depending on the dataloaders.
Using the factory TextDataLoaders class, the running time is 9 s for each epoch:
dls = TextDataLoaders.from_df(df, text_col='text', is_lm=True, valid_col='is_valid')
Using the low level api, it runs in 19 s:
splits = ColSplitter()(df)
tfms = [attrgetter("text"), Tokenizer(tokenizer=SpacyTokenizer()), Numericalize()]
dsets = Datasets(df, [tfms], splits=splits, dl_type=LMDataLoader)
dls = dsets.dataloaders(bs=64, seq_len=72)
Changing some dataloaders parameters, we can achieve between 8 and 9 s per epoch using the low level API:
dls = dsets.dataloaders(bs=64, seq_len=72, num_workers=2, pin_memory=True)
1 Like
I am not sure that the LMDataLoader can safely handle num_workers > 1`, FYI.
1 Like
Really? I have been using it for some time, and have had good results with fastai2. But I will keep my eyes open.
Well, that’s exactly the kind of trouble I have had. A couple of times it has almost worked, though.
I’ve moved some posts about fastai2 on Windows to a new topic:
2 Likes
I just saw the update on the main website and the arxiv paper on fastai v2. I’m very excited, especially for the mid-level APIs, two-way callbacks, and “refactored layer classes” thingy. The article mentioned that fastai v2 is feature complete and just lacking documentation and a youtube course, is that the current status?
How do users who have migrated to v2 feel about the current state? Do you find it stable enough relative to v1? Has it been easier to implement custom models? It seems with the mid-level APIs the code will be more flexible, and I’m really excited if that’s the case.
It’s relatively stable, (I’ve been doing all my work in v2 right now) see my comment here:
(which also answers your other questions somewhat). The study group I run shows many examples for custom models and mid-level api.
Said study group: A walk with fastai2 - Study Group and Online Lectures Megathread
I’ll also add I’m in the middle of implementing Mask RCNN, and getting the databunch working was trivial
1 Like