Define mixed label types for multitask? (e.g., segmentation + "per image" classification)

fastai
1

Hi,
Hope I’m not duplicating - couldn’t find a clear solution anywhere:
I have a segmentation problem, to which I want to apply an additional classification task.
I assume I should be able to do some customization on top of the following definitions for the segmentation alone:

class SegmentationLabelList(SegmentationLabelList):
def open(self, fn): return open_mask(fn, div=True)
class SegmentationItemList(SegmentationItemList):
_label_cls = SegmentationLabelList

It unsurprisingly fails to replace “return open_mask(fn, div=True)”, with “return [open_mask(fn, div=True), <some_scalar_target> ]” …

I would really appreciate some hints of how to customize this to treat both types of labels (segmentation mask + some classification “scalar” value").

Thanks a lot!

1 Like
2

Solved. While @denisvlr has developed a more general solution for such cases, he also pointed me tozhang_yang’s article and notebook which was also easy enough for a noob like me to follow, and well, “implement” :).
Thanks guys.

2 Likes
3

If you’d still want to use DataBlock API, you would need to create a totally custom ItemList imo, with probably a custom ItemBase as well that would wrap multiple labels for one item. I’ll probably look into it at some point as I might try to to some multitask learning for my work.