I would like to use @patch for overriding fastai2.learner.Recorder.after_batch method easily.
If I create a new Recorder class I need to modify the line where learner extracts Callbacks array. So, subclassing Learner is not possibble!
If I create a subclass of fastai2.data.load.DataLoader(GetAttr) for changing this line of code and add a collate that doesn’t stack items:
def create_batch(self, b): return (fa_collate,fa_convert)[self.prebatched](b)
Is it possible to link DataBlock.dataloaders to this new subclass? Or do I need to create new DataBlock subclass linked with Dataset subclass and with DataLoaders subclass?
However, I see something strange DataLoader is defined in data.core and data.load
I hope this is the right place for a short question:
I need to display data in a form: x,y = ((tensor, tensor), (tensor)). I generate it as shown in the previous expression, the whole item at a time. I @typedispatch show_batch() and I get the following error: AssertionError: Match length mismatch (full stacktrace at the end of the post).
I declared my own tuple (that does nothing):
class HeatingTuple(Tuple):
pass
and tried to write the show_batch() (and show_results() for that matter):
def show_results(x, y, samples:HeatingTuple, . . .
def show_results(x:HeatingTuple, y, samples, . . .
to no avail. Calling dls.show_batch() gives me the error. Manually running the show_batch works. (after I remove the annotation)
DataBlock is using an ItemTransform.
DataBlock(item_tfms=HeatingItemiser(future_len))
HeatingItemiser.encodes() returns a tuple and decodes() a HeatingTuple()
This is the error from dls.show_batch():
AssertionError Traceback (most recent call last)
<ipython-input-12-785270320dfd> in <module>
56 seq_dloaders = seq_block.dataloaders(samples, bs=8)
57 # onebatch = seq_dloaders.one_batch()
---> 58 seq_dloaders.show_batch()
59 # show_batch(None, None, samples=onebatch)
60 # print(f"Data shapes: x[0] {onebatch[0][0].shape}, x[1] {onebatch[0][1].shape} y{onebatch[1].shape}")
~/work/installs/fastai2/fastai2/data/core.py in show_batch(self, b, max_n, ctxs, show, unique, **kwargs)
97 if b is None: b = self.one_batch()
98 if not show: return self._pre_show_batch(b, max_n=max_n)
---> 99 show_batch(*self._pre_show_batch(b, max_n=max_n), ctxs=ctxs, max_n=max_n, **kwargs)
100 if unique: self.get_idxs = old_get_idxs
101
~/work/installs/fastai2/fastai2/data/core.py in _pre_show_batch(self, b, max_n)
87 b = self.decode(b)
88 if hasattr(b, 'show'): return b,None,None
---> 89 its = self._decode_batch(b, max_n, full=False)
90 if not is_listy(b): b,its = [b],L((o,) for o in its)
91 return detuplify(b[:self.n_inp]),detuplify(b[self.n_inp:]),its
~/work/installs/fastai2/fastai2/data/core.py in _decode_batch(self, b, max_n, full)
81 f = self.after_item.decode
82 f = compose(f, partial(getattr(self.dataset,'decode',noop), full = full))
---> 83 return L(batch_to_samples(b, max_n=max_n)).map(f)
84
85 def _pre_show_batch(self, b, max_n=9):
~/work/installs/fastcore/fastcore/foundation.py in map(self, f, *args, **kwargs)
373 else f.format if isinstance(f,str)
374 else f.__getitem__)
--> 375 return self._new(map(g, self))
376
377 def filter(self, f, negate=False, **kwargs):
~/work/installs/fastcore/fastcore/foundation.py in _new(self, items, *args, **kwargs)
324 @property
325 def _xtra(self): return None
--> 326 def _new(self, items, *args, **kwargs): return type(self)(items, *args, use_list=None, **kwargs)
327 def __getitem__(self, idx): return self._get(idx) if is_indexer(idx) else L(self._get(idx), use_list=None)
328 def copy(self): return self._new(self.items.copy())
~/work/installs/fastcore/fastcore/foundation.py in __call__(cls, x, *args, **kwargs)
39 return x
40
---> 41 res = super().__call__(*((x,) + args), **kwargs)
42 res._newchk = 0
43 return res
~/work/installs/fastcore/fastcore/foundation.py in __init__(self, items, use_list, match, *rest)
315 if items is None: items = []
316 if (use_list is not None) or not _is_array(items):
--> 317 items = list(items) if use_list else _listify(items)
318 if match is not None:
319 if is_coll(match): match = len(match)
~/work/installs/fastcore/fastcore/foundation.py in _listify(o)
251 if isinstance(o, list): return o
252 if isinstance(o, str) or _is_array(o): return [o]
--> 253 if is_iter(o): return list(o)
254 return [o]
255
~/work/installs/fastcore/fastcore/foundation.py in __call__(self, *args, **kwargs)
217 if isinstance(v,_Arg): kwargs[k] = args.pop(v.i)
218 fargs = [args[x.i] if isinstance(x, _Arg) else x for x in self.pargs] + args[self.maxi+1:]
--> 219 return self.fn(*fargs, **kwargs)
220
221 # Cell
~/work/installs/fastcore/fastcore/utils.py in _inner(x, *args, **kwargs)
346 if order is not None: funcs = funcs.sorted(order)
347 def _inner(x, *args, **kwargs):
--> 348 for f in L(funcs): x = f(x, *args, **kwargs)
349 return x
350 return _inner
~/work/installs/fastai2/fastai2/data/core.py in decode(self, o, full)
294 def __iter__(self): return (self[i] for i in range(len(self)))
295 def __repr__(self): return coll_repr(self)
--> 296 def decode(self, o, full=True): return tuple(tl.decode(o_, full=full) for o_,tl in zip(o,tuplify(self.tls, match=o)))
297 def subset(self, i): return type(self)(tls=L(tl.subset(i) for tl in self.tls), n_inp=self.n_inp)
298 def _new(self, items, *args, **kwargs): return super()._new(items, tfms=self.tfms, do_setup=False, **kwargs)
~/work/installs/fastcore/fastcore/utils.py in tuplify(o, use_list, match)
132 def tuplify(o, use_list=False, match=None):
133 "Make `o` a tuple"
--> 134 return tuple(L(o, use_list=use_list, match=match))
135
136 # Cell
~/work/installs/fastcore/fastcore/foundation.py in __call__(cls, x, *args, **kwargs)
39 return x
40
---> 41 res = super().__call__(*((x,) + args), **kwargs)
42 res._newchk = 0
43 return res
~/work/installs/fastcore/fastcore/foundation.py in __init__(self, items, use_list, match, *rest)
319 if is_coll(match): match = len(match)
320 if len(items)==1: items = items*match
--> 321 else: assert len(items)==match, 'Match length mismatch'
322 super().__init__(items)
323
AssertionError: Match length mismatch
Thank you!
1 Like
it seems that a fresh install doesnt necesarily install correct cuda version
I installed as http://dev.fast.ai/#Installing
but later I see that torch.cuda.is_available() returned false, but I have cuda… so I went to the page of pytorch and grab their command line
conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /home/tyoc213/miniconda3/envs/fastai2
added / updated specs:
- cudatoolkit=10.1
- pytorch
- torchvision
The following packages will be downloaded:
package | build
---------------------------|-----------------
cudatoolkit-10.1.243 | h6bb024c_0 347.4 MB
pytorch-1.5.0 |py3.7_cuda10.1.243_cudnn7.6.3_0 399.5 MB pytorch
torchvision-0.6.0 | py37_cu101 11.8 MB pytorch
------------------------------------------------------------
Total: 758.7 MB
The following packages will be DOWNGRADED:
cudatoolkit 10.2.89-hfd86e86_1 --> 10.1.243-h6bb024c_0
pytorch 1.5.0-py3.7_cuda10.2.89_cudnn7.6.5_0 --> 1.5.0-py3.7_cuda10.1.243_cudnn7.6.3_0
torchvision 0.6.0-py37_cu102 --> 0.6.0-py37_cu101
Proceed ([y]/n)? y
Downloading and Extracting Packages
pytorch-1.5.0 | 399.5 MB | ########################################################### | 100%
cudatoolkit-10.1.243 | 347.4 MB | ########################################################### | 100%
torchvision-0.6.0 | 11.8 MB | ########################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Like you see it installed incorrect cuda version, here it is 10.1 not 102, now torch.cuda.is_available() is true.
I dont know if this was just a failure or why it installed those other libs. Or is there a way that it dectects the correct ones on install? if not, ppl will think some things are “slow” because using CPU.
Hi everyone,
In some problems, for efficient storing, we can use lmdb to store our dataset with e.g. millions of images in just 1 or 2 files, like in Pytorch LSUN dataset or in most Scene text recognition problems.
I’m facing an issue when using a lmdb dataset with fastai2. Everything works fine except the export of the trained Learner, like learner.export(). The problem is when we use lmdb datasets, it opens an environment and then return indexed image, labels,… from that as arrays of bytes (see this example in Pytorch LSUN dataset). When a Learner uses DataLoaders created from a lmdb dataset, it can’t be exported, and gives typeError: can't pickle Environment objects, which refers to the environment that was opened.
Is there anything I can do to make the .export() work? Thanks.
Hi,
I am continuing to explore Siamese network in fastai2. I am trying to ‘hook’ in GradCAM for a Siamese model, at the ‘encoder’ layer, but I am not sure how to extract the activations output and gradient for both ‘passes’ of the two images that form the Siamese image pair…?
From the notebook, as per definition the Siamese model’s forward pass calls the same ‘encoder’ twice, on the two images, and concatenate their output together before calling ‘head’.
I have defined my hooks:
class Hook():
def __init__(self, m):
self.hook = m.register_forward_hook(self.hook_func)
def hook_func(self, m, i, o): self.stored = o.detach().clone()
def __enter__(self, *args): return self
def __exit__(self, *args): self.hook.remove()
class HookBwd():
def __init__(self, m):
self.hook = m.register_backward_hook(self.hook_func)
def hook_func(self, m, gi, go): self.stored = go[0].detach().clone()
def __enter__(self, *args): return self
def __exit__(self, *args): self.hook.remove()
And I created a test image-pair, and applied the transforms using the defined dataloaders:
img1 = PILImage.create(Path('/path/to/image1.jpg'))
img2 = PILImage.create(Path('/path/to/image2.jpg'))
siamtest = SiameseImage(img1, img2)
tdl = learn.dls.test_dl([siamtest])
x = tdl.one_batch()
And then I called the hooks and eval the model:
cls = 1
with HookBwd(learn.model.encoder) as hookg:
with Hook(learn.model.encoder) as hook:
output = learn.model.eval()(x[0],x[1])
act = hook.stored[0].cpu()
output[0,cls].backward()
grad = hookg.stored[0].cpu()
But this will only get me 1x act for the activations and 1x grad for the gradients, presumably from the second (i.e. final) call of encoder on the second image x[1]. I am not sure how to make it output two different sets of act and grad, for the encoder pass of x[0] and x[1] respectively.
Thoughts and comments welcome. Thank you.
Regards,
Yijin
Is it better to early stopping at loss or at precission metric?
What do you think of next approach:
Compare the valid loss between 2 models at validation and selecting the best hyperparameters based upon this.
With this hyperparameters train a model that early stop when accuracy is decreasing.
How to create dataloader for multiple outputs?
@Transform
def get_x(x): return f"{path}\\train_images\\{x[4]}"
@Transform
def get_arrays(x):
return [rle_decode(x[0],(xs,ys)),
rle_decode(x[1],(xs,ys)),
rle_decode(x[2],(xs,ys)),
rle_decode(x[3],(xs,ys))]
@Transform
def array_to_mask(x):
return (PILMask.create(x[0]),PILMask.create(x[1]),PILMask.create(x[2]),PILMask.create(x[3]))
batch_tfms=[*aug_transforms(size=(xs//7,ys//7)), Normalize.from_stats(*imagenet_stats)]
dsets=Datasets(train_df_t,tfms=[[get_x,PILImage.create],[get_arrays,array_to_mask]],splits=splits)``
dsets.valid[0]
gives me
(PILImage mode=RGB size=2100x1400,
(PILMask mode=I size=2100x1400,
PILMask mode=I size=2100x1400,
PILMask mode=I size=2100x1400,
PILMask mode=I size=2100x1400))
but
dls = dsets.dataloaders(bs=12,after_item=[ToTensor],before_batch=[IntToFloatTensor, Normalize.from_stats(*imagenet_stats)])
b=dls.one_batch()
len(b),len(b[0]),len(b[1])
gives me
(2, 12, 4)
I expect (2,12,12) but y has only one sample instead of a batch of sample.
Are those 4 masks you getting in b[1] of only one sample ?
Considering those are PILMasks, don’t you want to stack them up in a single Tensor? like a Tensor of shape (4,2100,1400) ?
There’s one parameter n_inp for Datasets, have you tried setting that to 1 explicitly?
1 Like
Do setups in Transform is called only once in case of DataLoaders? Normalize uses setups to calculate mean and std if they’re None. Does this mean we only calculate mean and std of single batch and use that for the entire dataset?
Yes, it does. But the above only works on a TfmdDL I believe. I didn’t see this behavior with a regular DataLoader.
I did calculate the entire dataset though, see my ‘hack’ here: https://www.kaggle.com/muellerzr/plant-pathology-fastai2-exploration
Also, Jeremy and Sylvain found that in general, the one batch is normally enough. If you look at my kernel you’ll see that the full vs one batch is extremely close
1 Like
This looks great idea! but doesn’t it cause MemoryError as you’re loading whole dataset at once?
I have also worked around to calculate the dataset stats, but I use this snippet for calculations
ds = Datasets(fnames, tfms=Pipeline([PILImage.create, Resize(320), ToTensor]))
dl = TfmdDL(ds, bs=32,after_batch=[IntToFloatTensor],drop_last=True)
mean, std = 0., 0.
for b in tqdm(dl,total=len(dl)):
mean += b[0].mean((0,2,3))
std += b[0].std((0,2,3))
mean /= len(dl)
std /= len(dl)
It actually doesn’t (I’ve found so far) 
I ran mine with 4g of memory just fine on the entire plant pathology dataset. Took quite a bit of time (a minute or two). However your way works as well I believe
@muellerzr do you have any idea about this?
I am using sigmoid in decodes as workaround for show_batch as matplotlib requires float values to be in [0,1] range and if I don’t scale them as required, matplotlib just clamps the Tensor, which is definitely loss of information.
I agree we should reverse the procedure done in the encodes to undo the effect, but what if I want to look at the image with change made in the encodes (in case of pre-processing) ?
Typically you want optimal metrics so better to look at this.
Thanks for replying, after I posted this I dig deeper and found b[1] is a tuple, b[1][0] had 12 masks of 1st channel and so on.
I tried n_inp and it didn’t help.
and for the stacking, I wanted to keep them separated, even If I wanted it to be stacked I don’t know how to stack PILMasks as a single tensor. 
I’ve worked with image+masks input recently. I’ll post a tutorial soon but this is how I did it
class MultiMask(Tuple):
...
def stack(self):
# To be used with batch only
return L(self).stack(dim=1)
To make the stack work for you, you need all of your masks be of same size and maybe square, I’m not sure about the square though. So in case, the stack fails, just put Resize(<desired size>) before calling ToTensor as you might be customizing ToTensor for you:
@ToTensor
def encodes(self, o:MultiMask): return TensorMask(o.stack())
You can simply pass in a list of PILMasks to this tuple, so your array_to_mask could be modified like so:
@Transform
def array_to_mask(x):
return MutliMask([PILMask.create(o) for o in x[:4]])
This will give you a single TensorMask object with all your masks, expecting the dim to be (4,size,size).
You won’t be able to show_batch anymore, as TensorMask isn’t designed to show multiple masks at once. This requires you to @typedispatch the show_batch for your use-case. I’ll share a tutorial soon explaining the whole process.
1 Like
Does this apply to Singular Value Decomposition as well? covariance matrix of one batch vs full dataset?
EDIT: I suppose it’s not the covariance matrix that make difference, it’s their decomposition.
Hi all
Apologies if this info is available somewhere already. Does anyone know the current state of pretrained xresnets? Back in January, it seems that they weren’t yet ready. I’m seeing better performance with old pretrained resnets so I’m assuming this is still the case?
Cheers!