Fastai v2 chat

Any idea about this?

Hi,
I reinstalled the fastai2 yesterday on WSL environment.
Iā€™m trying to go over the library nbs, but when I just open ā€œ08_vision.dataā€ I get error -
ā€œCould not find a kernel matching env37ā€.
I tried to open other notebooks (07_vision.core) and had no issue.

I used the editable install - > > pip install -e ".[dev]" >

my current python version is 3.7 (according conda list).
running ā€œwhich jupyter notebookā€ shows it runs the correct version from the fastai2 environment.

might it make sense to hide the fact that pillow v7 removed PILLOW_VERSION from fastai users?

I put a some ideas together here; https://github.com/pete88b/data-science/blob/master/fix-PILLOW_VERSION.ipynb

I am using untar_data() to download timeseries zipped files. Those files contain the zipped files stored at the root folder (i.e. there isnā€™t any folder inside as opposed to planet_tiny.tgz for example). When using untar_data(), the timeseries uncompressed files are store in .fastai/data folder and therefore polluting the data folder. I tried to use dest argument but that ended up storing the uncompressed files in my current folder. Is there an option to uncompress files in a separate folder under .fastai/data?

Just for testing and since I am using the editable version fastai v2, I temporarily replaced dest.parent by dest (see commented line at the end of the code snippet here below) and that fixed the problem (files are stored in their own folder i.e. name of the zip file). I was wondering if it would be possible to add an argument to untar_data() (use_parent_dest=True for example, used as default in order to stay compatible with current use, and False in my case).

def untar_data(url, fname=None, dest=None, c_key='data', force_download=False, extract_func=file_extract):
    "Download `url` to `fname` if `dest` doesn't exist, and un-tgz to folder `dest`."
    default_dest = URLs.path(url, c_key=c_key).with_suffix('')
    dest = default_dest if dest is None else Path(dest)/default_dest.name
    fname = Path(fname or URLs.path(url))
    if fname.exists() and _get_check(url) and _check_file(fname) != _get_check(url):
        print("A new version of this dataset is available, downloading...")
        force_download = True
    if force_download:
        if fname.exists(): os.remove(fname)
        if dest.exists(): shutil.rmtree(dest)
    if not dest.exists():
        fname = download_data(url, fname=fname, c_key=c_key)
        if _get_check(url) and _check_file(fname) != _get_check(url):
            print(f"File downloaded is broken. Remove {fname} and try again.")
        # extract_func(fname, dest.parent)
        extract_func(fname, dest)
    return dest

If you pass an absolute path as dest it should work.

Thank you Jeremy for your fast reply. Following your suggesting I tried these two options:

Option 1 : set dest to the fully qualified path. It stores the uncompressed files in the right folder but it returns the wrong path:

dsname =  'NATOPS' #'NATOPS', 'LSST', 'Wine', 'Epilepsy', 'HandMovementDirection'
path_data = Config().data
dest = path_data/dsname
dest

Path('C:/Users/fh/.fastai/data/NATOPS')
url = 'http://www.timeseriesclassification.com/Downloads/NATOPS.zip'
path = untar_data(url, dest=dest)
path

Path('C:/Users/fh/.fastai/data/NATOPS/NATOPS') <--- doesn't exist

Option 2 : set dest to the .fastai/data path. It doesnā€™t store the uncompressed files in a separate folder but it returns the intended path:

For both options, the returned paths donā€™t exist (see <--- arrows in the code snippets)

dsname =  'NATOPS' #'NATOPS', 'LSST', 'Wine', 'Epilepsy', 'HandMovementDirection'
path_data = Config().data
dest = path_data
dest

Path('C:/Users/fh/.fastai/data')
url = 'http://www.timeseriesclassification.com/Downloads/NATOPS.zip'
path = untar_data(url, dest=dest)
path

Path('C:/Users/fh/.fastai/data/NATOPS') <--- doesn't exist either

I think this comes down to the fact that the timeseries zip files donā€™t have an inside folder and untar_data() assumes there is one.

Thank you for your help anyway. If it isnā€™t possible to modify untar_data(), itā€™s fine; I will continue using a method that I called unzip_data() that looks like untar_data()

Yeah unless we actually look inside the file we canā€™t really know where it will extract to, so we have to make an assumption about what to return.

1 Like

Hi there.

Been trying tirelessly to get fastai working under Windows with GPU. Itā€™s installed, working perfectly fine on CPU, but absolutely refuses to use GPU seemingly whatever I try. I have tried using what I assume are ā€œup to dateā€ conda versions of things and also going back to pytorch 1.0.0 with cuda 9.0 and cudnn 7005. Also tried cuda 10.1 and 10.2 (and pytorch 1.4.0 I think it wasā€¦?). GPU driver is 441.66 - this is the constant I havenā€™t changed essentially because I wouldnā€™t know what version to try going backwards.

I donā€™t get any obvious errors that are exposed via the Anaconda console or in the notebook itself. I just get CPU activity with zilch happening on the GPU. ā€œtorch.cuda.is_available()ā€ returns true, my GPU model is returned when I do ā€œtorch.cuda.get_device_name(0)ā€. Iā€™m kind of at a loss at the moment.

Is this a known issue or a me related weirdness?

I had this running on Windows a while back @Russbo, but have been running more recently in Linux. I was planning on trying Windows again before the course starts (and do a write up of anything special needed) - so Iā€™ll let you know how it goes. I seem to remember the GPU use was very spiky and overall slower than Linux, but for what Iā€™d expect weā€™d be doing on the course it should be ok.

6 posts were merged into an existing topic: Fastai2 on Windows

I had the same problem while developing the audio module, where the files in some datasets are directly at the root of the tar file. The solution that I found was to modify the extract function, so that it extracts the contents to a folder with the same name as the compressed file.

The modified function is:

def tar_extract_at_filename(fname, dest):
    "Extract `fname` to `dest`/`fname`.name folder using `tarfile`"
    dest = Path(dest)/Path(fname).with_suffix('').name
    tarfile.open(fname, 'r:gz').extractall(dest)

And I use it like:

url = "https://public-datasets.fra1.digitaloceanspaces.com/250-speakers.tar"
path = untar_data(url, extract_func=tar_extract_at_filename)
4 Likes

An update on this problem:

After reducing the number of workers for document tokenization to half the number of CPUs it almost worked (things broke just 10.000 docs shy of the 3.7 million goal).

After that I hacked a bit the code.

First I forced the method from_folder (fastai2/fastai2/text/ core.py) to always do the line tokenize_folder(path, rules=rules, **kwargs) which is now skipped if the target folder already exists (I really should have changed the skip condition to the lengths and counts files existing).

Then I changed the code in tokenize_folder. As it is, files in a source folder are read and a tokenized version is saved for each. I made it so that if the target file exists already, then it reads it and gets tokens from it in a list. Note! Donā€™t forget about ā€˜\nā€™ (for some reason my method has a very minor mismatch in the count of ā€˜\nā€™, but I donā€™t think itā€™s an issue).

This is the code. Itā€™s not really very good, since this should be parallelized, but even for 3.7 million documents (with only around 10k missing tokenization) the solution was fast enough.

if os.path.isfile(out):
    text = out.read()
    tok = re.findall(r'\S+|\n',text)
else:
    tok = tokenize1(file, tok_func=SpacyTokenizer, rules=rules, 
                    post_rules=None, **tok_kwargs)
    out.write(' '.join(tok))

So now I have my lengths, counter and lm_data files :slight_smile: I am moving things to the big GPU machine to continue training the language model, I hope things work fine from now on, although Iā€™m a bit worried about the step of creating dataloaders for classification. Well, one step at a time, I guess!

Update: Fastai does two funny things in tokenization that complicate things (and make my lengths list not match its expectations).

  1. New line + space = ā€˜\nā–ā€™
  2. Repeated new lines get concatenated: \n \n --> \n\n

Itā€™s unfortunate since postprocessing lists really is a bit of a hassle, but it seems like this can be solved byā€¦ well, by applying these operations to the tokenized lists (when reading from file).

3 Likes

Fastai v2 is quite different it turns out. My v1 notebook is basically a modified version of class 3 planets from 2019. Transforms works differently, looks like ImageList might be gone.

Would rather just use v1 for the minuteā€¦

EDIT:

Ah - it seems it is working, but pretty poorly. With image size 256 from the plane dataset Iā€™m using, Iā€™m still getting 60% CPU utilisation, but 33% spikes of CUDA. GPU memory is also ~10GB/11GB. So itā€™s doing something.

Perhaps this is what @brismith was alluding to in terms of spiky utilisation.

I had some problems with GPU utilization, but now I managed to stay above 90% most of the time. Reading from text files or from a Dataframe.

x_bwd_tfms = [attrgetter("lm_relatorio"), rules, Tokenizer(tokenizer=sp), Numericalize(vocab), backwards]
clas_splits = RandomSplitter(valid_pct=0.1, seed=42)(df)

bwd_dsrc_clas = Datasets(df, splits=clas_splits, tfms=[x_bwd_tfms, [attrgetter("pred_resultado"), Categorize()]], dl_type=SortedDL)

bwd_dbunch_clas = bwd_dsrc_clas.dataloaders(bs=bs, cache=2048, num_workers=8, pin_memory=True, shuffle_train=True, before_batch=pad, seq_len=72)

These are the settings I have been using with very good results in terms of GPU performance.

For eg.: I have a large Dataset (20MM) texts in text files, and each epoch takes about 5 hours with GPU utilization around 95% most of the time. The painful part is that the Dataloader takes another 5 hours to parse the texts. But the model runs very smoothly. For the very large Dataset, the shuffle_train=False is also essential. Code:

def read_file(f): return L(SPieceTokenizer(f.read(),sp))

tfms = [read_file,Numericalize(vocab)]

splits = RandomSplitter(valid_pct=0.1, seed=42)(texts)
%time dsrc = DataSource(texts, [tfms], splits=splits, dl_type=LMDataLoader)

dbunch_lm = dsrc.databunch(bs=bs, seq_len=sl, val_bs=bs, num_workers=2, pin_memory=True, shuffle_train=False)
2 Likes

With the latest version of fastai 0.0.11 and fastcore (0.1.13), If I try to use to_fp16() I get always an error when finishing the 2nd epoch during training. Not using fp16() avoids the error.

Error message:

RuntimeError: param_from.type() == param_to.type() INTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/ATen/native/cudnn/RNN.cpp:541, please report a bug to PyTorch. parameter types mismatch

At the bottom of the doc created by nbdev it shows Ā©2020 Your Name or Company Name. All rights reserved. I uncommented and modified in settings.ini the copyright but nothing changes. I think that the settings.ini is fixed and when I ran nbdev_build it wonā€™t reload the settings. How can I fix it ? thanks

Thanks for sharing! Iā€™m very interested in this

do you have any idea how relevant this is?

In my tests, for the massive Dataset (20MM) without shuffle_train=False, the performance drops to unbearable levels (too much time without GPU processing - 0% and much bigger training time).

Using the low level API, some decisions impact the processing time very abruptly.

For eg: testing the imdb notebook from the fastai course, the running times can be very different depending on the dataloaders.

Using the factory TextDataLoaders class, the running time is 9 s for each epoch:
dls = TextDataLoaders.from_df(df, text_col='text', is_lm=True, valid_col='is_valid')

Using the low level api, it runs in 19 s:

splits = ColSplitter()(df)
tfms = [attrgetter("text"), Tokenizer(tokenizer=SpacyTokenizer()), Numericalize()]
dsets = Datasets(df, [tfms], splits=splits, dl_type=LMDataLoader)
dls = dsets.dataloaders(bs=64, seq_len=72)

Changing some dataloaders parameters, we can achieve between 8 and 9 s per epoch using the low level API:

dls = dsets.dataloaders(bs=64, seq_len=72, num_workers=2, pin_memory=True)

1 Like

I am not sure that the LMDataLoader can safely handle num_workers > 1`, FYI.

1 Like

Really? I have been using it for some time, and have had good results with fastai2. But I will keep my eyes open.