…and the reason for this is that PyTorch datasets are expected to create tuples, and DataLoader batches are tuples.
3 Likes
I am experiencing a problem with the DataBlock.datasets() using a list of tuples (numpy.ndarray, str) as a source. Using fastai2 v0.0.10, I was able to create dataloaders using both Datasets and DataBlock classes. When I updraded to v.0.0.11 this issue appeared.
I updated all my transforms that were using as_item=True to ItemTransform. By doing so, I was able to create a dataset using Datasets class. Datasets.vocab works fine and I can plot a timeseries from my dataset using show_at.
My items are basically a list of tuples. The tuples are numpy.ndarray and string. The error occur in the Categorize.setups() (in fastai2.data.transforms.py). It says TypeError: unhashable type: 'numpy.ndarray'. It looks like the numpy array is passed in instead of the strings. I suspect that it has to do with the DataBlock.getters() but all my attempts were not successful.
Source code and the error stack are here below
class TensorTS(TensorBase):
"Transform a 2D array into a Tensor"
def show(self, ctx=None, title=None, chs=None, leg=True, **kwargs):
....
class ToTensorTS(ItemTransform):
"xy : tuple representing (2D numpy array, Label)"
def encodes(self, xy): x,y=xy; return TensorTS(x)
def TSBlock():
"`TransformBlock` for timeseries : Transform np array to TensorTS type"
return TransformBlock(type_tfms=ToTensorTS())
class LabelTS(ItemTransform):
"x : tuple representing (2D numpy array, Label)"
def encodes(self, xy): x,y=xy; return y
getters = [lambda xy: xy[0], lambda xy: xy[1]]
tsdb = DataBlock(blocks=(TSBlock, CategoryBlock),
get_items=get_ts_items,
getters=getters,
splitter=RandomSplitter(seed=seed),
batch_tfms = batch_tfms)
tsdb.datasets(fnames, verbose=True)
=======================
Collecting items from [Path('C:/Users/fh/.fastai/data/NATOPS/NATOPS_TRAIN.arff'), Path('C:/Users/fh/.fastai/data/NATOPS/NATOPS_TEST.arff')]
Found 360 items
2 datasets of sizes 288,72
Setting up Pipeline: <lambda> -> ToTensorTSBlock
TYPE of xy : <class 'list'>
LENTGH of xy : 2
[array([-0.540579, -0.54101 , -0.540603, -0.540807, -0.540564, -0.540681,
-0.540665, -0.541065, -0.540593, -0.540723, -0.54044 , -0.540232,
-0.540191, -0.540209, -0.53981 , -0.539904, -0.540259, -0.540194,
-0.54002 , -0.54033 , -0.546852, -0.551316, -0.553834, -0.558104,
-0.560738, -0.558948, -0.559712, -0.565038, -0.560855, -0.55511 ,
-0.553586, -0.55047 , -0.555965, -0.554922, -0.55378 , -0.557211,
-0.556262, -0.558439, -0.560581, -0.560134, -0.55657 , -0.564141,
-0.56727 , -0.568937, -0.572611, -0.570396, -0.569147, -0.564811,
-0.56305 , -0.566314, -0.553712], dtype=float32), '2']
TYPE of x : <class 'numpy.ndarray'>
TYPE of y : <class 'str'>
x : [-0.540579 -0.54101 -0.540603 -0.540807 -0.540564 -0.540681 -0.540665
-0.541065 -0.540593 -0.540723 -0.54044 -0.540232 -0.540191 -0.540209
-0.53981 -0.539904 -0.540259 -0.540194 -0.54002 -0.54033 -0.546852
-0.551316 -0.553834 -0.558104 -0.560738 -0.558948 -0.559712 -0.565038
-0.560855 -0.55511 -0.553586 -0.55047 -0.555965 -0.554922 -0.55378
-0.557211 -0.556262 -0.558439 -0.560581 -0.560134 -0.55657 -0.564141
-0.56727 -0.568937 -0.572611 -0.570396 -0.569147 -0.564811 -0.56305
-0.566314 -0.553712]
y : 2
Setting up Pipeline: <lambda> -> Categorize
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
in
----> 1 tsdb.datasets(fnames, verbose=True)
c:\users\fh\dev\fastai2\fastai2\fastai2\data\block.py in datasets(self, source, verbose)
93 splits = (self.splitter or RandomSplitter())(items)
94 pv(f"{len(splits)} datasets of sizes {','.join([str(len(s)) for s in splits])}", verbose)
---> 95 return Datasets(items, tfms=self._combine_type_tfms(), splits=splits, dl_type=self.dl_type, n_inp=self.n_inp, verbose=verbose)
96
97 def dataloaders(self, source, path='.', verbose=False, **kwargs):
c:\users\fh\dev\fastai2\fastai2\fastai2\data\core.py in __init__(self, items, tfms, tls, n_inp, dl_type, **kwargs)
259 def __init__(self, items=None, tfms=None, tls=None, n_inp=None, dl_type=None, **kwargs):
260 super().__init__(dl_type=dl_type)
--> 261 self.tls = L(tls if tls else [TfmdLists(items, t, **kwargs) for t in L(ifnone(tfms,[None]))])
262 self.n_inp = (1 if len(self.tls)==1 else len(self.tls)-1) if n_inp is None else n_inp
263
c:\users\fh\dev\fastai2\fastai2\fastai2\data\core.py in (.0)
259 def __init__(self, items=None, tfms=None, tls=None, n_inp=None, dl_type=None, **kwargs):
260 super().__init__(dl_type=dl_type)
--> 261 self.tls = L(tls if tls else [TfmdLists(items, t, **kwargs) for t in L(ifnone(tfms,[None]))])
262 self.n_inp = (1 if len(self.tls)==1 else len(self.tls)-1) if n_inp is None else n_inp
263
c:\users\fh\dev\fastai2\fastcore\fastcore\foundation.py in __call__(cls, x, *args, **kwargs)
39 return x
40
---> 41 res = super().__call__(*((x,) + args), **kwargs)
42 res._newchk = 0
43 return res
c:\users\fh\dev\fastai2\fastai2\fastai2\data\core.py in __init__(self, items, tfms, use_list, do_setup, split_idx, train_setup, splits, types, verbose)
200 if do_setup:
201 pv(f"Setting up {self.tfms}", verbose)
--> 202 self.setup(train_setup=train_setup)
203
204 def _new(self, items, **kwargs): return super()._new(items, tfms=self.tfms, do_setup=False, types=self.types, **kwargs)
c:\users\fh\dev\fastai2\fastai2\fastai2\data\core.py in setup(self, train_setup)
213
214 def setup(self, train_setup=True):
--> 215 self.tfms.setup(self, train_setup)
216 if len(self) != 0:
217 x = super().__getitem__(0) if self.splits is None else super().__getitem__(self.splits[0])[0]
c:\users\fh\dev\fastai2\fastcore\fastcore\transform.py in setup(self, items, train_setup)
177 tfms = self.fs[:]
178 self.fs.clear()
--> 179 for t in tfms: self.add(t,items, train_setup)
180
181 def add(self,t, items=None, train_setup=False):
c:\users\fh\dev\fastai2\fastcore\fastcore\transform.py in add(self, t, items, train_setup)
180
181 def add(self,t, items=None, train_setup=False):
--> 182 t.setup(items, train_setup)
183 self.fs.append(t)
184
c:\users\fh\dev\fastai2\fastcore\fastcore\transform.py in setup(self, items, train_setup)
76 def setup(self, items=None, train_setup=False):
77 train_setup = train_setup if self.train_setup is None else self.train_setup
---> 78 return self.setups(getattr(items, 'train', items) if train_setup else items)
79
80 def _call(self, fn, x, split_idx=None, **kwargs):
c:\users\fh\dev\fastai2\fastcore\fastcore\dispatch.py in __call__(self, *args, **kwargs)
96 if not f: return args[0]
97 if self.inst is not None: f = MethodType(f, self.inst)
---> 98 return f(*args, **kwargs)
99
100 def __get__(self, inst, owner):
c:\users\fh\dev\fastai2\fastai2\fastai2\data\transforms.py in setups(self, dsets)
184
185 def setups(self, dsets):
--> 186 if self.vocab is None and dsets is not None: self.vocab = CategoryMap(dsets, add_na=self.add_na)
187 self.c = len(self.vocab)
188
c:\users\fh\dev\fastai2\fastai2\fastai2\data\transforms.py in __init__(self, col, sort, add_na)
169 if not hasattr(col,'unique'): col = L(col, use_list=True)
170 # `o==o` is the generalized definition of non-NaN used by Pandas
--> 171 items = L(o for o in col.unique() if o==o)
172 if sort: items = items.sorted()
173 self.items = '#na#' + items if add_na else items
c:\users\fh\dev\fastai2\fastcore\fastcore\foundation.py in unique(self)
372 return self._new(i for i,o in enumerate(self) if f(o))
373
--> 374 def unique(self): return L(dict.fromkeys(self).keys())
375 def enumerate(self): return L(enumerate(self))
376 def val2idx(self): return {v:k for k,v in self.enumerate()}
TypeError: unhashable type: 'numpy.ndarray'
======================
Thank you in advance for your help
You should replace the two lambdas in the getters list by ItemGetter(0) and ItemGetter(1).
1 Like
@sgugger can you please see if this is how one must evaluate when to use get_items, get_x, get_y?
I am having a hard time coming up with rules when to use them. thank you.
-
if you have paths always use
get_items. You cannot useget_xto pass in the x.get_ycan be used to get the labels -
if we are working with a df:
2a. you can passget_itemslike -
def _planet_items(x): return (
f'{planet_source}/train/'+x.image_name+'.jpg', x.tags.str.split())
so you can pass in both the x and y with just get_items.
2b. or you can use get_x and get_y to get the column names.
Basically:
-
get_yis some way to get the labels from theitems(paths) or can be a col name in case of a df. - you cannot pass
get_xandget_yinstead ofget_itemsand expect it to become a tuple “(get_x, get_y)” for paths it works for df.
The most common situation is have seen is use get_items to get your paths and then use get_y to extract the label from it.
This is the first example i have seen with getters need to figure this still(any tips on when to use this is much appreciated)
get_items is completely decoupled from get_x and get_y: it is there to return all your items from the source. You can pass get_x and get_y (or a list of getters) to explain how to get your x and y from the result of get_items and they both default to noop (which is why when get_items return filenames, we don’t pass a get_x).
3 Likes
Any idea about this?
Hi,
I reinstalled the fastai2 yesterday on WSL environment.
I’m trying to go over the library nbs, but when I just open “08_vision.data” I get error -
“Could not find a kernel matching env37”.
I tried to open other notebooks (07_vision.core) and had no issue.
I used the editable install - > > pip install -e ".[dev]" >
my current python version is 3.7 (according conda list).
running “which jupyter notebook” shows it runs the correct version from the fastai2 environment.
might it make sense to hide the fact that pillow v7 removed PILLOW_VERSION from fastai users?
I put a some ideas together here; https://github.com/pete88b/data-science/blob/master/fix-PILLOW_VERSION.ipynb
I am using untar_data() to download timeseries zipped files. Those files contain the zipped files stored at the root folder (i.e. there isn’t any folder inside as opposed to planet_tiny.tgz for example). When using untar_data(), the timeseries uncompressed files are store in .fastai/data folder and therefore polluting the data folder. I tried to use dest argument but that ended up storing the uncompressed files in my current folder. Is there an option to uncompress files in a separate folder under .fastai/data?
Just for testing and since I am using the editable version fastai v2, I temporarily replaced dest.parent by dest (see commented line at the end of the code snippet here below) and that fixed the problem (files are stored in their own folder i.e. name of the zip file). I was wondering if it would be possible to add an argument to untar_data() (use_parent_dest=True for example, used as default in order to stay compatible with current use, and False in my case).
def untar_data(url, fname=None, dest=None, c_key='data', force_download=False, extract_func=file_extract):
"Download `url` to `fname` if `dest` doesn't exist, and un-tgz to folder `dest`."
default_dest = URLs.path(url, c_key=c_key).with_suffix('')
dest = default_dest if dest is None else Path(dest)/default_dest.name
fname = Path(fname or URLs.path(url))
if fname.exists() and _get_check(url) and _check_file(fname) != _get_check(url):
print("A new version of this dataset is available, downloading...")
force_download = True
if force_download:
if fname.exists(): os.remove(fname)
if dest.exists(): shutil.rmtree(dest)
if not dest.exists():
fname = download_data(url, fname=fname, c_key=c_key)
if _get_check(url) and _check_file(fname) != _get_check(url):
print(f"File downloaded is broken. Remove {fname} and try again.")
# extract_func(fname, dest.parent)
extract_func(fname, dest)
return dest
If you pass an absolute path as dest it should work.
Thank you Jeremy for your fast reply. Following your suggesting I tried these two options:
Option 1 : set dest to the fully qualified path. It stores the uncompressed files in the right folder but it returns the wrong path:
dsname = 'NATOPS' #'NATOPS', 'LSST', 'Wine', 'Epilepsy', 'HandMovementDirection'
path_data = Config().data
dest = path_data/dsname
dest
Path('C:/Users/fh/.fastai/data/NATOPS')
url = 'http://www.timeseriesclassification.com/Downloads/NATOPS.zip'
path = untar_data(url, dest=dest)
path
Path('C:/Users/fh/.fastai/data/NATOPS/NATOPS') <--- doesn't exist
Option 2 : set dest to the .fastai/data path. It doesn’t store the uncompressed files in a separate folder but it returns the intended path:
For both options, the returned paths don’t exist (see <--- arrows in the code snippets)
dsname = 'NATOPS' #'NATOPS', 'LSST', 'Wine', 'Epilepsy', 'HandMovementDirection'
path_data = Config().data
dest = path_data
dest
Path('C:/Users/fh/.fastai/data')
url = 'http://www.timeseriesclassification.com/Downloads/NATOPS.zip'
path = untar_data(url, dest=dest)
path
Path('C:/Users/fh/.fastai/data/NATOPS') <--- doesn't exist either
I think this comes down to the fact that the timeseries zip files don’t have an inside folder and untar_data() assumes there is one.
Thank you for your help anyway. If it isn’t possible to modify untar_data(), it’s fine; I will continue using a method that I called unzip_data() that looks like untar_data()
Yeah unless we actually look inside the file we can’t really know where it will extract to, so we have to make an assumption about what to return.
1 Like
Hi there.
Been trying tirelessly to get fastai working under Windows with GPU. It’s installed, working perfectly fine on CPU, but absolutely refuses to use GPU seemingly whatever I try. I have tried using what I assume are “up to date” conda versions of things and also going back to pytorch 1.0.0 with cuda 9.0 and cudnn 7005. Also tried cuda 10.1 and 10.2 (and pytorch 1.4.0 I think it was…?). GPU driver is 441.66 - this is the constant I haven’t changed essentially because I wouldn’t know what version to try going backwards.
I don’t get any obvious errors that are exposed via the Anaconda console or in the notebook itself. I just get CPU activity with zilch happening on the GPU. “torch.cuda.is_available()” returns true, my GPU model is returned when I do “torch.cuda.get_device_name(0)”. I’m kind of at a loss at the moment.
Is this a known issue or a me related weirdness?
I had this running on Windows a while back @Russbo, but have been running more recently in Linux. I was planning on trying Windows again before the course starts (and do a write up of anything special needed) - so I’ll let you know how it goes. I seem to remember the GPU use was very spiky and overall slower than Linux, but for what I’d expect we’d be doing on the course it should be ok.
I had the same problem while developing the audio module, where the files in some datasets are directly at the root of the tar file. The solution that I found was to modify the extract function, so that it extracts the contents to a folder with the same name as the compressed file.
The modified function is:
def tar_extract_at_filename(fname, dest):
"Extract `fname` to `dest`/`fname`.name folder using `tarfile`"
dest = Path(dest)/Path(fname).with_suffix('').name
tarfile.open(fname, 'r:gz').extractall(dest)
And I use it like:
url = "https://public-datasets.fra1.digitaloceanspaces.com/250-speakers.tar"
path = untar_data(url, extract_func=tar_extract_at_filename)
4 Likes
An update on this problem:
After reducing the number of workers for document tokenization to half the number of CPUs it almost worked (things broke just 10.000 docs shy of the 3.7 million goal).
After that I hacked a bit the code.
First I forced the method from_folder (fastai2/fastai2/text/ core.py) to always do the line tokenize_folder(path, rules=rules, **kwargs) which is now skipped if the target folder already exists (I really should have changed the skip condition to the lengths and counts files existing).
Then I changed the code in tokenize_folder. As it is, files in a source folder are read and a tokenized version is saved for each. I made it so that if the target file exists already, then it reads it and gets tokens from it in a list. Note! Don’t forget about ‘\n’ (for some reason my method has a very minor mismatch in the count of ‘\n’, but I don’t think it’s an issue).
This is the code. It’s not really very good, since this should be parallelized, but even for 3.7 million documents (with only around 10k missing tokenization) the solution was fast enough.
if os.path.isfile(out):
text = out.read()
tok = re.findall(r'\S+|\n',text)
else:
tok = tokenize1(file, tok_func=SpacyTokenizer, rules=rules,
post_rules=None, **tok_kwargs)
out.write(' '.join(tok))
So now I have my lengths, counter and lm_data files 
I am moving things to the big GPU machine to continue training the language model, I hope things work fine from now on, although I’m a bit worried about the step of creating dataloaders for classification. Well, one step at a time, I guess!
Update: Fastai does two funny things in tokenization that complicate things (and make my lengths list not match its expectations).
- New line + space = ‘\n▁’
- Repeated new lines get concatenated: \n \n --> \n\n
It’s unfortunate since postprocessing lists really is a bit of a hassle, but it seems like this can be solved by… well, by applying these operations to the tokenized lists (when reading from file).
3 Likes
Fastai v2 is quite different it turns out. My v1 notebook is basically a modified version of class 3 planets from 2019. Transforms works differently, looks like ImageList might be gone.
Would rather just use v1 for the minute…
EDIT:
Ah - it seems it is working, but pretty poorly. With image size 256 from the plane dataset I’m using, I’m still getting 60% CPU utilisation, but 33% spikes of CUDA. GPU memory is also ~10GB/11GB. So it’s doing something.
Perhaps this is what @brismith was alluding to in terms of spiky utilisation.
I had some problems with GPU utilization, but now I managed to stay above 90% most of the time. Reading from text files or from a Dataframe.
x_bwd_tfms = [attrgetter("lm_relatorio"), rules, Tokenizer(tokenizer=sp), Numericalize(vocab), backwards]
clas_splits = RandomSplitter(valid_pct=0.1, seed=42)(df)
bwd_dsrc_clas = Datasets(df, splits=clas_splits, tfms=[x_bwd_tfms, [attrgetter("pred_resultado"), Categorize()]], dl_type=SortedDL)
bwd_dbunch_clas = bwd_dsrc_clas.dataloaders(bs=bs, cache=2048, num_workers=8, pin_memory=True, shuffle_train=True, before_batch=pad, seq_len=72)
These are the settings I have been using with very good results in terms of GPU performance.
For eg.: I have a large Dataset (20MM) texts in text files, and each epoch takes about 5 hours with GPU utilization around 95% most of the time. The painful part is that the Dataloader takes another 5 hours to parse the texts. But the model runs very smoothly. For the very large Dataset, the shuffle_train=False is also essential. Code:
def read_file(f): return L(SPieceTokenizer(f.read(),sp))
tfms = [read_file,Numericalize(vocab)]
splits = RandomSplitter(valid_pct=0.1, seed=42)(texts)
%time dsrc = DataSource(texts, [tfms], splits=splits, dl_type=LMDataLoader)
dbunch_lm = dsrc.databunch(bs=bs, seq_len=sl, val_bs=bs, num_workers=2, pin_memory=True, shuffle_train=False)
2 Likes