I am wondering what the recommended way to ingest the dataset provided in the recent Kaggle competition, https://www.kaggle.com/c/severstal-steel-defect-detection is.
The provided csv contains
[imgname].jpg_[class] in the first column and an RLE string (or NaN) in the second.
So for each image, I have:
I recognize that fastai provides a open_mask_rle function, however the discussions I can find are outdated for the current DataBunch api and it is unclear to me how to construct my databunch.
Being new to python and ML, I agree with Jeremy’s statement in lesson 3 that figuring out how to actually get the data in is the most difficult part for me.
Are any others here looking at this competition? How are you planning to work with this data?
My current attempt looks like this:
isna = df_train.isna() def get_y_fn(x): idxs = df_train.index[df_train['ImageId_ClassId'].str.contains(str(x.name)) == True] masks =  for i, v in enumerate(idxs): cell = df_train.iloc[v] if isna.iloc[v] == False: mask = open_mask_rle(df_train.iloc[v], size) mask.data[mask.data!=(i+1)] = i+1 masks.append(mask) else: mask = ImageSegment(torch.zeros(1, 1600, 256)) masks.append(mask) return masks codes = ['0', '1', '2', '3'] class SegLabelList(SegmentationLabelList): def open(self, fn): return open_mask_rle(fn) class SegItemList(SegmentationItemList): _label_cls,_square_show_res = SegLabelList,False data = (SegmentationItemList.from_folder(path/'train_images') .use_partial_data() .split_by_rand_pct() .label_from_func(get_y_fn, classes=codes) .transform(get_transforms(), size=size, tfm_y=True) .databunch(bs=bs) .normalize(imagenet_stats))
Which returns an error
AttributeError: 'list' object has no attribute 'read' (I assume I can’t return this “masks” list).
I am thinking I could combine these separate masks, which contain zeros and [class_num] as values, into one mask. Is this a good way to go? How would I do that? I also do not know if this dataset has overlapping masks.