Proper splitting and mIOU for CAMVID

fastai
1

A little FYI, the fastai split for CamVid is inconsistant with papers, as we train on the test set as well. Below is a function which will prepare the dataset properly. You should evaluate on the test folder when wanting to compare benchmarks (and note that most use mIOU, not accuracy. Below is also a mIOU formula, and Jeremy is aware of this and formulating an adjustment for the dataset. This is for the interim period :slight_smile: )

The txt's are located here

def prepare_camvid(path):
    "Splits CamVid dataset to make it consistant with papers"
    dsets = 'train,valid,test'.split(',')
    mb = master_bar(range(3))
    for i in mb:
        folder = path/dsets[i]
        os.makedirs(folder, exist_ok=True)
        fnames = np.loadtxt(f'{dsets[i]}.txt', dtype=str)
        for j in progress_bar(range(len(fnames)), parent=mb):
            shutil.move(path/'images'/fnames[j], folder/fnames[j])

To use, do the following:

path = untar_data(URLs.CAMVID)
prepare_camvid(path)

Do also note that now you should use a GrandparentSplitter when generating your DataBlock, as the dataset will be split up between a train, valid, and test folder.

Here is a mIOU code snippet for you to use as well by @juvian:

import numpy as np

def get_confusion_matrix(label, pred, size, num_class, ignore=-1):
    "Calcute the confusion matrix by given label and pred"
    output = pred.cpu().numpy().transpose(0, 2, 3, 1)
    seg_pred = np.asarray(np.argmax(output, axis=3), dtype=np.uint8)
    seg_gt = np.asarray(
    label.cpu().numpy()[:, :size[-2], :size[-1]], dtype=np.int)

    ignore_index = seg_gt != ignore
    seg_gt = seg_gt[ignore_index]
    seg_pred = seg_pred[ignore_index]

    index = (seg_gt * num_class + seg_pred).astype('int32')
    label_count = np.bincount(index)
    confusion_matrix = np.zeros((num_class, num_class))

    for i_label in range(num_class):
        for i_pred in range(num_class):
            cur_index = i_label * num_class + i_pred
            if cur_index < len(label_count):
                confusion_matrix[i_label,
                                 i_pred] = label_count[cur_index]
    return confusion_matrix

def mIOU(pred: torch.Tensor, truth: torch.Tensor, name2id:dict, ignore_index:int=-1):
    confusion_matrix = get_confusion_matrix(truth, pred, np.array(pred.shape), len(name2id), ignore_index)
    pos = confusion_matrix.sum(1)
    res = confusion_matrix.sum(0)
    tp = np.diag(confusion_matrix)
    IoU_array = (tp / np.maximum(1.0, pos + res - tp))
    mean_IoU = IoU_array.mean()
    return mean_IoU

To use on CamVid, do:

iou = partial(mIOU, name2id=name2id, ignore_index=void_code)
metrics = [iou]

It will also then require a different splitter to go by the parent folder. I’ve modified GrandparentSplitter to make a ParentSplitter

def _parent_idxs(items, name):
    def _inner(items, name): return mask2idxs(Path(o).parent.name == name for o in items)
    return [i for n in L(name) for i in _inner(items,n)]

def ParentSplitter(train_name='train', valid_name='valid'):
    "Split `items` from the parent folder names (`train_name` and `valid_name`)."
    def _inner(o):
        return _parent_idxs(o, train_name),_parent_idxs(o, valid_name)
    return _inner

Example DataBlock:

camvid = DataBlock(blocks=(ImageBlock, MaskBlock(codes)),
                   get_items=get_image_files,
                   splitter=ParentSplitter(),
                   get_y=get_msk,
                   batch_tfms=[*aug_transforms(size=half), Normalize.from_stats(*imagenet_stats)])
dls = camvid.dataloaders(path, bs=8)
4 Likes