Sadly it didn’t work… It seemed like the __call__ wasn’t even called for.
I tried to get smarter with this tutorial:
Where it kinda explains crucial things like how to apply a transform only on the train dataset, but not the valid dataset.
But it doesn’t really explain applying a transform only on the input. 
I also found this guy’s advice:
Followed what you said:
I think the problem is that the transform is being type-dispatched, so since both x and y are images, it is being applied to both.
Seems that you were right, and the other guy found a way to solve that by improvising on that basis.
I also found this topic:
I think that I got inspired by them for a good solution.
My idea is to add some flag on my TensorRawImage that holds either input or target. The transform later would check out whether it’s input or target, just like it would have checked on this whether it’s a train or valid (with the flat split_idx).
I will post here a nice solution if it works.
Edit: Wow, almost made it. It was very challenging 
But I got some problem but well, can’t solve this much