However also PointScaler is done before resize, so how do we explain the > 1 seen on the DataLoader?
That is what I keep saying. It will not take 224 as a size since Resize happens after PointScaler, so the size used at this stage will be (1025, 721).
1 Like
Got it, that just clicked. My apologies @sgugger 
So how would this then explain the yās being above or below 1, -1 when outputted from the DataLoader? As I can verify all input coordinates were in fact on the image when it came in.
I canāt say, you need to debug your data pipeline step by step. Just explaining why the order of the transforms need to be this way 
1 Like
Iāll look into it and report back
Thanks for your guidance, itās greatly appreciated
Would the new dblock.summary be able to provide this? (just curious)
Try it and see what the output is
1 Like
Will do!
@sgugger I get an issue on dblock.summary():
Setting-up type transforms pipelines
Collecting items from
Found 9888 items
2 datasets of sizes 7911,1977
Setting up Pipeline: PILBase.create
Setting up Pipeline: get_y -> TensorPoint.create
Building one sample
Pipeline: PILBase.create
starting from
cats/00001249_005.jpg
applying PILBase.create gives
<fastai2.vision.core.PILImage image mode=RGB size=500x394 at 0x7FCABA492A90>
Pipeline: get_y -> TensorPoint.create
starting from
cats/00001249_005.jpg
applying get_y gives
Tensor of size 9x2
applying TensorPoint.create gives
TensorPoint of size 9x2
Final sample: (<fastai2.vision.core.PILImage image mode=RGB size=500x394 at 0x7FCABA4F4FD0>, TensorPoint([[383., 107.],
[393., 123.],
[376., 123.],
[383., 95.],
[396., 80.],
[398., 97.],
[406., 110.],
[420., 115.],
[406., 128.]]))
Setting up after_item: Pipeline: PointScaler -> Resize -> ToTensor
Setting up before_batch: Pipeline:
Setting up after_batch: Pipeline: IntToFloatTensor -> AffineCoordTfm -> Normalize
Building one batch
Applying item_tfms to the first sample:
Pipeline: PointScaler -> Resize -> ToTensor
starting from
(<fastai2.vision.core.PILImage image mode=RGB size=500x394 at 0x7FCABA4F4CC0>, TensorPoint of size 9x2)
applying PointScaler gives
(<fastai2.vision.core.PILImage image mode=RGB size=500x394 at 0x7FCABA4F4CC0>, TensorPoint of size 9x2)
applying Resize gives
(<fastai2.vision.core.PILImage image mode=RGB size=448x448 at 0x7FCABA4EC9E8>, TensorPoint of size 9x2)
applying ToTensor gives
(TensorImage of size 3x448x448, TensorPoint of size 9x2)
Adding the next 3 samples
No before_batch transform to apply
Collating items in a batch
Applying batch_tfms to the batch built
Pipeline: IntToFloatTensor -> AffineCoordTfm -> Normalize
starting from
(TensorImage of size 4x3x448x448, TensorPoint of size 4x9x2)
applying IntToFloatTensor gives
(TensorImage of size 4x3x448x448, TensorPoint of size 4x9x2)
applying AffineCoordTfm gives
(TensorImage of size 4x3x448x448, TensorPoint of size 4x9x2)
applying Normalize failed.
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-46-1d2738086ea4> in <module>()
----> 1 dblock.summary('')
9 frames
/usr/local/lib/python3.6/dist-packages/fastai2/data/block.py in summary(self, source, bs, **kwargs)
173 if len([f for f in dls.train.after_batch.fs if f.name != 'noop'])!=0:
174 print("\nApplying batch_tfms to the batch built")
--> 175 b = _apply_pipeline(dls.train.after_batch, b)
176 else: print("\nNo batch_tfms to apply")
/usr/local/lib/python3.6/dist-packages/fastai2/data/block.py in _apply_pipeline(p, x)
120 except Exception as e:
121 print(f" applying {name} failed.")
--> 122 raise e
123 return x
124
/usr/local/lib/python3.6/dist-packages/fastai2/data/block.py in _apply_pipeline(p, x)
116 name = f.name
117 try:
--> 118 x = f(x)
119 if name != "noop": print(f" applying {name} gives\n {_short_repr(x)}")
120 except Exception as e:
/usr/local/lib/python3.6/dist-packages/fastcore/transform.py in __call__(self, x, **kwargs)
69 @property
70 def name(self): return getattr(self, '_name', _get_name(self))
---> 71 def __call__(self, x, **kwargs): return self._call('encodes', x, **kwargs)
72 def decode (self, x, **kwargs): return self._call('decodes', x, **kwargs)
73 def __repr__(self): return f'{self.name}: {self.use_as_item} {self.encodes} {self.decodes}'
/usr/local/lib/python3.6/dist-packages/fastcore/transform.py in _call(self, fn, x, split_idx, **kwargs)
81 f = getattr(self, fn)
82 if self.use_as_item or not is_listy(x): return self._do_call(f, x, **kwargs)
---> 83 res = tuple(self._do_call(f, x_, **kwargs) for x_ in x)
84 return retain_type(res, x)
85
/usr/local/lib/python3.6/dist-packages/fastcore/transform.py in <genexpr>(.0)
81 f = getattr(self, fn)
82 if self.use_as_item or not is_listy(x): return self._do_call(f, x, **kwargs)
---> 83 res = tuple(self._do_call(f, x_, **kwargs) for x_ in x)
84 return retain_type(res, x)
85
/usr/local/lib/python3.6/dist-packages/fastcore/transform.py in _do_call(self, f, x, **kwargs)
85
86 def _do_call(self, f, x, **kwargs):
---> 87 return x if f is None else retain_type(f(x, **kwargs), x, f.returns_none(x))
88
89 add_docs(Transform, decode="Delegate to `decodes` to undo transform", setup="Delegate to `setups` to set up transform")
/usr/local/lib/python3.6/dist-packages/fastcore/dispatch.py in __call__(self, *args, **kwargs)
96 if not f: return args[0]
97 if self.inst is not None: f = MethodType(f, self.inst)
---> 98 return f(*args, **kwargs)
99
100 def __get__(self, inst, owner):
/usr/local/lib/python3.6/dist-packages/fastai2/data/transforms.py in encodes(self, x)
291 self.mean,self.std = x.mean(self.axes, keepdim=True),x.std(self.axes, keepdim=True)+1e-7
292
--> 293 def encodes(self, x:TensorImage): return (x-self.mean) / self.std
294 def decodes(self, x:TensorImage):
295 f = to_cpu if x.device.type=='cpu' else noop
/usr/local/lib/python3.6/dist-packages/fastai2/torch_core.py in _f(self, *args, **kwargs)
270 def _f(self, *args, **kwargs):
271 cls = self.__class__
--> 272 res = getattr(super(TensorBase, self), fn)(*args, **kwargs)
273 return retain_type(res, self)
274 return _f
RuntimeError: expected device cpu but got device cuda:0
Looks to be like Normalize wonāt default to the present device on it? (investigating further)
Also, it would be nice (if possible) if we could see these points:
Applying item_tfms to the first sample:
Pipeline: PointScaler -> Resize -> ToTensor
starting from
(<fastai2.vision.core.PILImage image mode=RGB size=500x375 at 0x7FCABA586390>, TensorPoint of size 9x2)
applying PointScaler gives
(<fastai2.vision.core.PILImage image mode=RGB size=500x375 at 0x7FCABA586390>, TensorPoint of size 9x2)
applying Resize gives
(<fastai2.vision.core.PILImage image mode=RGB size=448x448 at 0x7FCABA43F550>, TensorPoint of size 9x2)
applying ToTensor gives
(TensorImage of size 3x448x448, TensorPoint of size 9x2)
Mabye an option to return_tfmd_batch or something similar off the summary?
After modifying the code a bit, (just to apply it), here is what I discovered @sgugger
Before Resize is applied, all points are between -1, 1 as expected. But the moment Resize is applied, theyāre not!
For example:
x[1] = TensorPoint([[ 53., 165.],
[ 75., 171.],
[ 61., 184.],
[ 38., 157.],
[ 47., 131.],
[ 58., 148.],
[ 79., 153.],
[ 98., 146.],
[ 90., 171.]])
Assume our input. Now letās run through our after_item transforms like so:
for f in dls.train.after_item:
name = f.name
x = f(x)
print(name, x[1])
Which results in the following:
PointScaler TensorPoint([[-0.7880, 0.1786],
[-0.7000, 0.2214],
[-0.7560, 0.3143],
[-0.8480, 0.1214],
[-0.8120, -0.0643],
[-0.7680, 0.0571],
[-0.6840, 0.0929],
[-0.6080, 0.0429],
[-0.6400, 0.2214]])
Resize TensorPoint([[-2.0714, 0.1786],
[-1.9143, 0.2214],
[-2.0143, 0.3143],
[-2.1786, 0.1214],
[-2.1143, -0.0643],
[-2.0357, 0.0571],
[-1.8857, 0.0929],
[-1.7500, 0.0429],
[-1.8071, 0.2214]])
ToTensor TensorPoint([[-2.0714, 0.1786],
[-1.9143, 0.2214],
[-2.0143, 0.3143],
[-2.1786, 0.1214],
[-2.1143, -0.0643],
[-2.0357, 0.0571],
[-1.8857, 0.0929],
[-1.7500, 0.0429],
[-1.8071, 0.2214]])
So now @sgugger the question is are we expected to no longer have -1,1 after doing a Resize on our tensor points?
1 Like
Sounds like there is a bug in Resize and not PointScaler. Let me look into it: there is an implementation of Resize for TensorPoint since you can pad or crop depending on the resize method chosen, which would change the points. But itās not supposed to change them that much! If you squish for instance, itās not supposed to change them at all.
2 Likes
Thanks! I knew I wasnāt going too crazy I was using the default, which is crop
1 Like
While you wait for the fix, squish does not have the bug. Iām still investigating this further and able to reproduce.
Edit Actually you are cropping, so itās normal points that were in the image get outside of it, hence the coordinates being bigger than 1. You should pad or squish in your resize. Does show_batch look weird?
1 Like
Yes actually it did, I just assumed that with the transforms and going back as much as it could some parts couldnāt be āstepped backā. But itās sounding like thatās not the case
Is it possible to put guards in place for this? Such as if it goes off, reset the point to (-1,-1)? (Something commonly done for these type of problems if the point doesnāt show up)
You can add a transform that clips to -1,1, but this is intentionally left as is. I looked back at Resize and all the computations are correct in the case of Crop method. It does make coordinates go outside of -1,1 but thatās only when the points are in a part of the image that ends up being cropped.
2 Likes
This explains so much. Thanks @sgugger. Iāll look into the transform as for these type of problems thatās definitely something that should help. This would then also explain how my model could seemingly plot points on images of cats when the point isnāt even really there! (Which as a side note I find facinating).
Yesterday I was debugging this together with @muellerzr but I was using squish, and the problem were still there. In the possibility of being crazy I just double checked an ran the experiment again, and found a range from -2.24 to 0.996
Iām making sure none of the images contains any points outside the image and the only relevant transform being applied is Resize with squish. Since mostly probably Iām doing something very wrong Iām providing a āminimumā working example here
Hereās said transform:
class ClampPoints(Transform):
"Clamp points to a minimum and maximum"
order = 4
def __init__(self, min=-1, max=1, **kwargs):
super().__init__(**kwargs)
self.min, self.max = min, max
def encodes(self, x:(TensorPoint)):
for pnt in x:
cpnt = torch.clamp(x, self.min, self.max)
for i, t in enumerate(cpnt):
if any(t==1) or any(t==-1):
x[i] = tensor([-1,-1])
return x
This should ideally only be needed if you use Resize with a crop method
Itās a dataset issue Thanks for the help @sgugger. What wound up happening was we were checking for images whoās points were above the resolution size, but we did not realize on this dataset they also labeled them in the negatives too. This is why we were getting the local minima. The change results in a minima of -1
1 Like
Yeahā¦ Sorry for the trouble 
As @muellerzr said, itās very obvious when you open your eyes and see that the only problem was the negative part of the range ā-2.24 to 0.996ā e.e
1 Like