Segmentation Data with fastai.__version__ == '2.0.15'

Deep Learning
1

I am having trouble finding an effective tutorial for using the latest Data Block API for Segmentation Data.

I am not clear exactly what I need to do to my masks to get them loaded.

It appears that open_mask is no longer available, because I get NameError: name 'open_mask' is not defined or module not found when I try to call it.

I am using the Mut1ny face dataset which provides a color mask like this:

mutiny_labels = [
    (0,0,0,'Background/undefined'),
    (255,0,0,'Lips'),
    (0,255,0,'Eyes'),
    (0,0,255,'Nose'),
    (255,255,0,'Hair'),
    (0,255,255,'Ears'),
    (255,0,255,'Eyebrows'),
    (255,255,255,'Teeth'),
    (128,128,128,'General face'),
    (255,192,192,'Facial hair'),
    (0,128,128,'Specs/sunglasses')
]

Can anyone point me to a segmentation tutorial for fastai v2?

1 Like
2

OK, I keep hammering away at this. For any watchers, I found this tutorial.

I have pretty much concluded that my masks are not encoded like the other masks, or I am not good enough at reading the fastai documentation to figure out how to use the PILMask API to use the Mut1ny masks.

So my plan is to convert my files specified in the mutiny_labels object found in my post above, to integers representing each class. The part I got thrown off by was that the label file is saved as a TIF (or other image file). My base image files were shape=(3,x,y), so I thought that I needed to make my mask that same shape, which didn’t make sense because there were no longer any R,G,B values, just the labels. However, that was not the case. You just need to get a matrix (x,y) and that can also be saved as an image, which can be loaded by the PILMask api just as was in the Jeremy Segmentation tutorials from 2019. (see image below)

Screen Shot 2020-10-04 at 11.58.15 AM
Screen Shot 2020-10-04 at 11.58.15 AM1362Ă—778 148 KB

I will post code once it’s tested and finished.

And I believe that the tutorial I linked above from WaterKnight1998 doesn’t show the step that I mentioned above, but he did it in the section marked “Manual”. Even though his problem is not multi-class, it’s just binary

1 Like
3

By the way, here is the error that I was getting after trying to run

learn = unet_learner(dls, resnet34)
learn.fine_tune(8)

using the mask as provided by Mut1ny. There were many iterations, and this was the clearest error message. Lot’s of non-descript CUDA errors then I turned off GPU compute and got something more clear.

I believe these errors are existing because the data bunch is not configured correctly.

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-23-f067e6d6d342> in <module>
      1 learn = unet_learner(dls, resnet34)
----> 2 learn.fine_tune(8)

/opt/conda/lib/python3.6/site-packages/fastcore/logargs.py in _f(*args, **kwargs)
     54         init_args.update(log)
     55         setattr(inst, 'init_args', init_args)
---> 56         return inst if to_return else f(*args, **kwargs)
     57     return _f

/opt/conda/lib/python3.6/site-packages/fastai/callback/schedule.py in fine_tune(self, epochs, base_lr, freeze_epochs, lr_mult, pct_start, div, **kwargs)
    159     "Fine tune with `freeze` for `freeze_epochs` then with `unfreeze` from `epochs` using discriminative LR"
    160     self.freeze()
--> 161     self.fit_one_cycle(freeze_epochs, slice(base_lr), pct_start=0.99, **kwargs)
    162     base_lr /= 2
    163     self.unfreeze()

/opt/conda/lib/python3.6/site-packages/fastcore/logargs.py in _f(*args, **kwargs)
     54         init_args.update(log)
     55         setattr(inst, 'init_args', init_args)
---> 56         return inst if to_return else f(*args, **kwargs)
     57     return _f

/opt/conda/lib/python3.6/site-packages/fastai/callback/schedule.py in fit_one_cycle(self, n_epoch, lr_max, div, div_final, pct_start, wd, moms, cbs, reset_opt)
    111     scheds = {'lr': combined_cos(pct_start, lr_max/div, lr_max, lr_max/div_final),
    112               'mom': combined_cos(pct_start, *(self.moms if moms is None else moms))}
--> 113     self.fit(n_epoch, cbs=ParamScheduler(scheds)+L(cbs), reset_opt=reset_opt, wd=wd)
    114 
    115 # Cell

/opt/conda/lib/python3.6/site-packages/fastcore/logargs.py in _f(*args, **kwargs)
     54         init_args.update(log)
     55         setattr(inst, 'init_args', init_args)
---> 56         return inst if to_return else f(*args, **kwargs)
     57     return _f

/opt/conda/lib/python3.6/site-packages/fastai/learner.py in fit(self, n_epoch, lr, wd, cbs, reset_opt)
    205             self.opt.set_hypers(lr=self.lr if lr is None else lr)
    206             self.n_epoch = n_epoch
--> 207             self._with_events(self._do_fit, 'fit', CancelFitException, self._end_cleanup)
    208 
    209     def _end_cleanup(self): self.dl,self.xb,self.yb,self.pred,self.loss = None,(None,),(None,),None,None

/opt/conda/lib/python3.6/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    153 
    154     def _with_events(self, f, event_type, ex, final=noop):
--> 155         try:       self(f'before_{event_type}')       ;f()
    156         except ex: self(f'after_cancel_{event_type}')
    157         finally:   self(f'after_{event_type}')        ;final()

/opt/conda/lib/python3.6/site-packages/fastai/learner.py in _do_fit(self)
    195         for epoch in range(self.n_epoch):
    196             self.epoch=epoch
--> 197             self._with_events(self._do_epoch, 'epoch', CancelEpochException)
    198 
    199     @log_args(but='cbs')

/opt/conda/lib/python3.6/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    153 
    154     def _with_events(self, f, event_type, ex, final=noop):
--> 155         try:       self(f'before_{event_type}')       ;f()
    156         except ex: self(f'after_cancel_{event_type}')
    157         finally:   self(f'after_{event_type}')        ;final()

/opt/conda/lib/python3.6/site-packages/fastai/learner.py in _do_epoch(self)
    189 
    190     def _do_epoch(self):
--> 191         self._do_epoch_train()
    192         self._do_epoch_validate()
    193 

/opt/conda/lib/python3.6/site-packages/fastai/learner.py in _do_epoch_train(self)
    181     def _do_epoch_train(self):
    182         self.dl = self.dls.train
--> 183         self._with_events(self.all_batches, 'train', CancelTrainException)
    184 
    185     def _do_epoch_validate(self, ds_idx=1, dl=None):

/opt/conda/lib/python3.6/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    153 
    154     def _with_events(self, f, event_type, ex, final=noop):
--> 155         try:       self(f'before_{event_type}')       ;f()
    156         except ex: self(f'after_cancel_{event_type}')
    157         finally:   self(f'after_{event_type}')        ;final()

/opt/conda/lib/python3.6/site-packages/fastai/learner.py in all_batches(self)
    159     def all_batches(self):
    160         self.n_iter = len(self.dl)
--> 161         for o in enumerate(self.dl): self.one_batch(*o)
    162 
    163     def _do_one_batch(self):

/opt/conda/lib/python3.6/site-packages/fastai/learner.py in one_batch(self, i, b)
    177         self.iter = i
    178         self._split(b)
--> 179         self._with_events(self._do_one_batch, 'batch', CancelBatchException)
    180 
    181     def _do_epoch_train(self):

/opt/conda/lib/python3.6/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    153 
    154     def _with_events(self, f, event_type, ex, final=noop):
--> 155         try:       self(f'before_{event_type}')       ;f()
    156         except ex: self(f'after_cancel_{event_type}')
    157         finally:   self(f'after_{event_type}')        ;final()

/opt/conda/lib/python3.6/site-packages/fastai/learner.py in _do_one_batch(self)
    164         self.pred = self.model(*self.xb)
    165         self('after_pred')
--> 166         if len(self.yb): self.loss = self.loss_func(self.pred, *self.yb)
    167         self('after_loss')
    168         if not self.training or not len(self.yb): return

/opt/conda/lib/python3.6/site-packages/fastai/losses.py in __call__(self, inp, targ, **kwargs)
     32         if targ.dtype in [torch.int8, torch.int16, torch.int32]: targ = targ.long()
     33         if self.flatten: inp = inp.view(-1,inp.shape[-1]) if self.is_2d else inp.view(-1)
---> 34         return self.func.__call__(inp, targ.view(-1) if self.flatten else targ, **kwargs)
     35 
     36 # Cell

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    946     def forward(self, input: Tensor, target: Tensor) -> Tensor:
    947         return F.cross_entropy(input, target, weight=self.weight,
--> 948                                ignore_index=self.ignore_index, reduction=self.reduction)
    949 
    950 

/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2420     if size_average is not None or reduce is not None:
   2421         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2422     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2423 
   2424 

/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2216                          .format(input.size(0), target.size(0)))
   2217     if dim == 2:
-> 2218         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2219     elif dim == 4:
   2220         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

IndexError: Target 128 is out of bounds.
4

Here’s what I ended up doing that seems to have fixed it.

The notebook sets up the labels as integer masks so that the dataset can be set up as follows:

defaults.use_cuda = True

dls = SegmentationDataLoaders.from_label_func(
    path, 
    bs=1, 
    fnames=fnames,
    label_func=get_y_fn,
    codes=codes, 
    item_tfms=[Resize((size,size))],
    batch_tfms=[Normalize.from_stats(*imagenet_stats)]
)

Now my machine can dream in parameters again. See below for the data pipeline I used:

fnames = [data_root/pair['srcimg'] for pair in pairs]
lnames = [data_root/pair['labelimg'] for pair in pairs]
img_to_l = {fname: lnames[i] for i, fname in enumerate(fnames)}

# Source: https://www.mut1ny.com/face-headsegmentation-dataset
header = ('R','G','B','L')
mutiny_labels = [
    (0,0,0,'Background/undefined'),
    (255,0,0,'Lips'),
    (0,255,0,'Eyes'),
    (0,0,255,'Nose'),
    (255,255,0,'Hair'),
    (0,255,255,'Ears'),
    (255,0,255,'Eyebrows'),
    (255,255,255,'Teeth'),
    (128,128,128,'General face'),
    (255,192,192,'Facial hair'),
    (0,128,128,'Specs/sunglasses')
]
mutiny_labels = pd.DataFrame(mutiny_labels, columns=header)
mutiny_labels['I'] = mutiny_labels.index
label_map = {
    (rec['R'], rec['G'], rec['B']): rec['I'] 
    for rec in mutiny_labels.to_dict('records')
}
int_to_label = {
    rec['I']: rec['L']
    for rec in mutiny_labels.to_dict('records')
}

codes = mutiny_labels.L.values
# codes = np.append(codes, ['Error'])

import ray

@ray.remote(num_returns=1)
def label_color_to_int(fn, label_map):
    """
    For RGB labels.
    """
    img = PILImage.create(fn)
    dat = TensorImage(image2tensor(img))
    frame_labels = np.zeros(
        (int(dat.shape[1]), int(dat.shape[2])),
        dtype=int
    )
    for i in range(dat.shape[1]):
        for j in range(dat.shape[2]):
            R = int(dat[0, i, j])
            G = int(dat[1, i, j])
            B = int(dat[2, i, j])
            if (R,G,B) not in label_map:
                # undefined pixel identified
                print('PixelMapError', fn, (R,G,B))
                frame_labels[i,j] = len(label_map)
                continue
            label = label_map[(R,G,B)]
            frame_labels[i,j] = label
    return frame_labels

import traceback

ray.init(address='auto', _redis_password='5241590000000000')

int_label_dir = data_root/'labels_int'
futures = []
names = []
for lname in lnames:
    ext = '/'.join(lname.parts[-2:])
    outfp = int_label_dir/ext.replace('png', 'tif')
    # if outfp.exists(): continue
    names.append(outfp)
    futures.append(
        label_color_to_int.remote(
            fn=lname, label_map=label_map
        )
    )
print('Starting...')   
for i, future in enumerate(futures):
    try:
        label_mat = ray.get(future)
    except FileNotFoundError: 
        traceback.print_exc()
        continue
    outfp = names[i]
    outfp.parent.mkdir(parents=True, exist_ok=True)
    im = Image.fromarray(
        label_mat.astype(np.uint8)
    )
    im.save(outfp)

def get_y_fn(fp):
    l_str = str(img_to_l[fp])
    out = l_str \
        .replace('labels', 'labels_int') \
        .replace('png', 'tif')
    return out

## Proof

# open and show image
img_f = fnames[0]
print(img_f)
img = PILImage.create(img_f)
img.show(figsize=(5, 5), alpha=1)

mask = PILMask.create(get_y_fn(img_f))
print(get_y_fn(img_f))
mask.show(figsize=(5, 5), alpha=1)

src_size = np.array(mask.shape)
print(src_size, np.array(mask))

The ray stuff is just so it runs on all my cpu cores. This is meant to be run on a notebook, but I don’t have this in a public repo right now.

1 Like
5

Good effort.

mrfabulous1 :smiley: :smiley:

1 Like
6

Hey Aaron and mrfabulous1,
this is a great effort and it seems in line with my problem: I also went in and digged into the Pillow encoding of the colors. I use images and man-made segmentation masks to train the algorithm. My present aim is to get to fit. Because when I try to fit or an lr_find it replies:
IndexError: Target 199 is out of bounds.
where 199 is one of the L colors from the Pillow mask in which the original hex colors are re-encoded. I put codes and colors in a dictionary, and tried to insert them but it seems not to find them.
I use Unet learner and opt = ranger . Do you have a suggestion?