Any idea about this?
Hi,
I reinstalled the fastai2 yesterday on WSL environment.
Iām trying to go over the library nbs, but when I just open ā08_vision.dataā I get error -
āCould not find a kernel matching env37ā.
I tried to open other notebooks (07_vision.core) and had no issue.
I used the editable install - > > pip install -e ".[dev]" >
my current python version is 3.7 (according conda list).
running āwhich jupyter notebookā shows it runs the correct version from the fastai2 environment.
might it make sense to hide the fact that pillow v7 removed PILLOW_VERSION from fastai users?
I put a some ideas together here; https://github.com/pete88b/data-science/blob/master/fix-PILLOW_VERSION.ipynb
I am using untar_data()
to download timeseries zipped files. Those files contain the zipped files stored at the root folder (i.e. there isnāt any folder inside as opposed to planet_tiny.tgz for example). When using untar_data()
, the timeseries uncompressed files are store in .fastai/data
folder and therefore polluting the data folder. I tried to use dest
argument but that ended up storing the uncompressed files in my current folder. Is there an option to uncompress files in a separate folder under .fastai/data
?
Just for testing and since I am using the editable version fastai v2, I temporarily replaced dest.parent
by dest
(see commented line at the end of the code snippet here below) and that fixed the problem (files are stored in their own folder i.e. name of the zip file). I was wondering if it would be possible to add an argument to untar_data()
(use_parent_dest=True
for example, used as default in order to stay compatible with current use, and False in my case).
def untar_data(url, fname=None, dest=None, c_key='data', force_download=False, extract_func=file_extract):
"Download `url` to `fname` if `dest` doesn't exist, and un-tgz to folder `dest`."
default_dest = URLs.path(url, c_key=c_key).with_suffix('')
dest = default_dest if dest is None else Path(dest)/default_dest.name
fname = Path(fname or URLs.path(url))
if fname.exists() and _get_check(url) and _check_file(fname) != _get_check(url):
print("A new version of this dataset is available, downloading...")
force_download = True
if force_download:
if fname.exists(): os.remove(fname)
if dest.exists(): shutil.rmtree(dest)
if not dest.exists():
fname = download_data(url, fname=fname, c_key=c_key)
if _get_check(url) and _check_file(fname) != _get_check(url):
print(f"File downloaded is broken. Remove {fname} and try again.")
# extract_func(fname, dest.parent)
extract_func(fname, dest)
return dest
If you pass an absolute path as dest
it should work.
Thank you Jeremy for your fast reply. Following your suggesting I tried these two options:
Option 1 : set dest
to the fully qualified path. It stores the uncompressed files in the right folder but it returns the wrong path:
dsname = 'NATOPS' #'NATOPS', 'LSST', 'Wine', 'Epilepsy', 'HandMovementDirection'
path_data = Config().data
dest = path_data/dsname
dest
Path('C:/Users/fh/.fastai/data/NATOPS')
url = 'http://www.timeseriesclassification.com/Downloads/NATOPS.zip'
path = untar_data(url, dest=dest)
path
Path('C:/Users/fh/.fastai/data/NATOPS/NATOPS') <--- doesn't exist
Option 2 : set dest
to the .fastai/data
path. It doesnāt store the uncompressed files in a separate folder but it returns the intended path:
For both options, the returned paths donāt exist (see <---
arrows in the code snippets)
dsname = 'NATOPS' #'NATOPS', 'LSST', 'Wine', 'Epilepsy', 'HandMovementDirection'
path_data = Config().data
dest = path_data
dest
Path('C:/Users/fh/.fastai/data')
url = 'http://www.timeseriesclassification.com/Downloads/NATOPS.zip'
path = untar_data(url, dest=dest)
path
Path('C:/Users/fh/.fastai/data/NATOPS') <--- doesn't exist either
I think this comes down to the fact that the timeseries zip files donāt have an inside folder and untar_data()
assumes there is one.
Thank you for your help anyway. If it isnāt possible to modify untar_data()
, itās fine; I will continue using a method that I called unzip_data()
that looks like untar_data()
Yeah unless we actually look inside the file we canāt really know where it will extract to, so we have to make an assumption about what to return.
Hi there.
Been trying tirelessly to get fastai working under Windows with GPU. Itās installed, working perfectly fine on CPU, but absolutely refuses to use GPU seemingly whatever I try. I have tried using what I assume are āup to dateā conda versions of things and also going back to pytorch 1.0.0 with cuda 9.0 and cudnn 7005. Also tried cuda 10.1 and 10.2 (and pytorch 1.4.0 I think it wasā¦?). GPU driver is 441.66 - this is the constant I havenāt changed essentially because I wouldnāt know what version to try going backwards.
I donāt get any obvious errors that are exposed via the Anaconda console or in the notebook itself. I just get CPU activity with zilch happening on the GPU. ātorch.cuda.is_available()ā returns true, my GPU model is returned when I do ātorch.cuda.get_device_name(0)ā. Iām kind of at a loss at the moment.
Is this a known issue or a me related weirdness?
I had this running on Windows a while back @Russbo, but have been running more recently in Linux. I was planning on trying Windows again before the course starts (and do a write up of anything special needed) - so Iāll let you know how it goes. I seem to remember the GPU use was very spiky and overall slower than Linux, but for what Iād expect weād be doing on the course it should be ok.
I had the same problem while developing the audio module, where the files in some datasets are directly at the root of the tar file. The solution that I found was to modify the extract function, so that it extracts the contents to a folder with the same name as the compressed file.
The modified function is:
def tar_extract_at_filename(fname, dest):
"Extract `fname` to `dest`/`fname`.name folder using `tarfile`"
dest = Path(dest)/Path(fname).with_suffix('').name
tarfile.open(fname, 'r:gz').extractall(dest)
And I use it like:
url = "https://public-datasets.fra1.digitaloceanspaces.com/250-speakers.tar"
path = untar_data(url, extract_func=tar_extract_at_filename)
An update on this problem:
After reducing the number of workers for document tokenization to half the number of CPUs it almost worked (things broke just 10.000 docs shy of the 3.7 million goal).
After that I hacked a bit the code.
First I forced the method from_folder
(fastai2/fastai2/text/ core.py) to always do the line tokenize_folder(path, rules=rules, **kwargs)
which is now skipped if the target folder already exists (I really should have changed the skip condition to the lengths and counts files existing).
Then I changed the code in tokenize_folder
. As it is, files in a source folder are read and a tokenized version is saved for each. I made it so that if the target file exists already, then it reads it and gets tokens from it in a list. Note! Donāt forget about ā\nā (for some reason my method has a very minor mismatch in the count of ā\nā, but I donāt think itās an issue).
This is the code. Itās not really very good, since this should be parallelized, but even for 3.7 million documents (with only around 10k missing tokenization) the solution was fast enough.
if os.path.isfile(out):
text = out.read()
tok = re.findall(r'\S+|\n',text)
else:
tok = tokenize1(file, tok_func=SpacyTokenizer, rules=rules,
post_rules=None, **tok_kwargs)
out.write(' '.join(tok))
So now I have my lengths, counter and lm_data files I am moving things to the big GPU machine to continue training the language model, I hope things work fine from now on, although Iām a bit worried about the step of creating dataloaders for classification. Well, one step at a time, I guess!
Update: Fastai does two funny things in tokenization that complicate things (and make my lengths list not match its expectations).
- New line + space = ā\nāā
- Repeated new lines get concatenated: \n \n --> \n\n
Itās unfortunate since postprocessing lists really is a bit of a hassle, but it seems like this can be solved byā¦ well, by applying these operations to the tokenized lists (when reading from file).
Fastai v2 is quite different it turns out. My v1 notebook is basically a modified version of class 3 planets from 2019. Transforms works differently, looks like ImageList might be gone.
Would rather just use v1 for the minuteā¦
EDIT:
Ah - it seems it is working, but pretty poorly. With image size 256 from the plane dataset Iām using, Iām still getting 60% CPU utilisation, but 33% spikes of CUDA. GPU memory is also ~10GB/11GB. So itās doing something.
Perhaps this is what @brismith was alluding to in terms of spiky utilisation.
I had some problems with GPU utilization, but now I managed to stay above 90% most of the time. Reading from text files or from a Dataframe.
x_bwd_tfms = [attrgetter("lm_relatorio"), rules, Tokenizer(tokenizer=sp), Numericalize(vocab), backwards]
clas_splits = RandomSplitter(valid_pct=0.1, seed=42)(df)
bwd_dsrc_clas = Datasets(df, splits=clas_splits, tfms=[x_bwd_tfms, [attrgetter("pred_resultado"), Categorize()]], dl_type=SortedDL)
bwd_dbunch_clas = bwd_dsrc_clas.dataloaders(bs=bs, cache=2048, num_workers=8, pin_memory=True, shuffle_train=True, before_batch=pad, seq_len=72)
These are the settings I have been using with very good results in terms of GPU performance.
For eg.: I have a large Dataset (20MM) texts in text files, and each epoch takes about 5 hours with GPU utilization around 95% most of the time. The painful part is that the Dataloader takes another 5 hours to parse the texts. But the model runs very smoothly. For the very large Dataset, the shuffle_train=False is also essential. Code:
def read_file(f): return L(SPieceTokenizer(f.read(),sp))
tfms = [read_file,Numericalize(vocab)]
splits = RandomSplitter(valid_pct=0.1, seed=42)(texts)
%time dsrc = DataSource(texts, [tfms], splits=splits, dl_type=LMDataLoader)
dbunch_lm = dsrc.databunch(bs=bs, seq_len=sl, val_bs=bs, num_workers=2, pin_memory=True, shuffle_train=False)
With the latest version of fastai 0.0.11 and fastcore (0.1.13), If I try to use to_fp16() I get always an error when finishing the 2nd epoch during training. Not using fp16() avoids the error.
Error message:
RuntimeError: param_from.type() == param_to.type() INTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/ATen/native/cudnn/RNN.cpp:541, please report a bug to PyTorch. parameter types mismatch
At the bottom of the doc created by nbdev it shows Ā©2020 Your Name or Company Name. All rights reserved. I uncommented and modified in settings.ini the copyright but nothing changes. I think that the settings.ini is fixed and when I ran nbdev_build it wonāt reload the settings. How can I fix it ? thanks
Thanks for sharing! Iām very interested in this
do you have any idea how relevant this is?
In my tests, for the massive Dataset (20MM) without shuffle_train=False, the performance drops to unbearable levels (too much time without GPU processing - 0% and much bigger training time).
Using the low level API, some decisions impact the processing time very abruptly.
For eg: testing the imdb notebook from the fastai course, the running times can be very different depending on the dataloaders.
Using the factory TextDataLoaders class, the running time is 9 s for each epoch:
dls = TextDataLoaders.from_df(df, text_col='text', is_lm=True, valid_col='is_valid')
Using the low level api, it runs in 19 s:
splits = ColSplitter()(df)
tfms = [attrgetter("text"), Tokenizer(tokenizer=SpacyTokenizer()), Numericalize()]
dsets = Datasets(df, [tfms], splits=splits, dl_type=LMDataLoader)
dls = dsets.dataloaders(bs=64, seq_len=72)
Changing some dataloaders parameters, we can achieve between 8 and 9 s per epoch using the low level API:
dls = dsets.dataloaders(bs=64, seq_len=72, num_workers=2, pin_memory=True)
I am not sure that the LMDataLoader can safely handle
num_workers > 1`, FYI.
Really? I have been using it for some time, and have had good results with fastai2. But I will keep my eyes open.