Can't do Segmentation: CUDA error: device-side assert triggered

(Renato Hermoza) #4

When defining the DataBunch, have you specified the classes?

0 Likes

(Imanol Pinto) #5

I have found the solution. The problem is that my masks have the following values: [0, 255], and this is not supported by default by fast.ai. The problem is that the fast.ai function open_mask works with small mask pixel values like 0,1,2,3 by default. This function is called by the SegmentationLabelList class. To work with 255 values we should call the function open_mask with div=True: open_mask(fn, div=True). It divides the mask pixel values by 255.

In order to change the default behaviour of open_mask, I did the following:

class SegLabelListCustom(SegmentationLabelList):
    def open(self, fn): return open_mask(fn, div=True)

class SegItemListCustom(SegmentationItemList):
    _label_cls = SegLabelListCustom

codes = ['0','1']
src = (SegItemListCustom.from_folder(path_img)
       .random_split_by_pct(valid_pct=0.2, seed=33)
       .label_from_func(get_y_fn, classes=codes))

data = (src.transform(get_transforms(), size=size, tfm_y=True)
        .databunch(bs=bs)
        .normalize(imagenet_stats))

Hope this helps. Thank you for your replies!

12 Likes

RuntimeError: CUDA error: device-side assert triggered when using unet_learner
#6

Hi, I used your codes, the error gone, but now by viewing data.show_batch, I found there is no mask. And if try" lr_find(learn);learn.recorder.plot()" after the learner created, the loss is 0, which confirm that the mask is gone…
How to fix that? Thanks

2 Likes

#7

I used “get_y_fn = lambda x: pathmask_train/f’{x.stem}_json.png’” before modification

0 Likes

(hasif) #8

Thanks @imanol, it works!

1 Like

(Danielh Carranza) #9

I have the same issue, Did you solve it?

1 Like

#10

Hey, guys! I’ve just partly fixed the problem(fix the problem using COCO dataset). Now, I can use lr_find(learn) and learn.recorder.plot() without error. Also see this link:Image Segmentation on COCO dataset - summary, questions and suggestions.

I think there are 2 problems in this error: 1)The value of the mask array should be {0,1}, not {0,255}or some other stuff, that’s what many other people said. The link above find ways to generate {0,1} mask. But how to fix it by not transforming data into COCO form, I haven’t tried; 2)The class list. Mine is a bynary segmentation project(to segment “row” ), so CATEGORY_NAMES=[0, ‘row’]. If CATEGORY_NAMES=[‘row’], I found the loss would always be 0, when using lr_find(learn) and learn.recorder.plot().
I think there still are something more to figure out, but for now, at leat I can use unet and lr_find(learn).

0 Likes

#11

I transformed data to COCO, and added 0 to class list(CATEGORY_NAMES=[0, ‘row’], instead[‘row’], e.g.), and it works.
I also updated fastai to 1.0.48, then the data.show_batch error disappeared, though I don’t know why.

0 Likes

(Simone Massaro) #12

It works! thank you

0 Likes

(Divyanshu Sharma) #13

It works!
Should’t it be reported as a bug ?

0 Likes

(Patrick Trampert) #14

Have a look here for those that still have problems:

0 Likes

(Nithin Varghese) #15

Thanks so much, much needed one

0 Likes

#16
class SegLabelListCustom(SegmentationLabelList):
    def open(self, fn): return open_mask(fn, div=True)

class SegItemListCustom(SegmentationItemList):
    _label_cls = SegLabelListCustom

codes = ['0','1']
src = (SegItemListCustom.from_folder(path_img)
       .random_split_by_pct(valid_pct=0.2, seed=33)
       .label_from_func(get_y_fn, classes=codes))

data = (src.transform(get_transforms(), size=size, tfm_y=True)
        .databunch(bs=bs)
        .normalize(imagenet_stats))

Is it normal for this code to take quite a while to run? I’ve had it running for well over an hour with just 5000 128x128 images in the dataset. Hoping it will work but I’m not sure what’s going on. I only changed the paths.

0 Likes

Create databunch with multiple segmentation mask as label
(moein hasani) #17

Did u manage to fix the show batch problem ?

0 Likes

#18

Just update fastai, then show batch will be ok

0 Likes

(moein hasani) #19

I updated the fastai to 1.0.55 but still no mask is shown

0 Likes

#20

Sorry, I didn’t figure out why it worked, but it worked when I was using kaggle kernel

1 Like

#21

I had the same problem. Turned out I had saved the modified images in the wrong format with floats

0 Likes

(MassimilianoG) #22

Hi, I’m trying to segmet some images where masks have data between [0,255]; after a lot of searches I tried your code that seems to solve my problem, but when I try do display data:

data.show_batch(2, figsize=(10,7))

I got this error:

BrokenPipeError: [Errno 32] Broken pipe

Any ideas? Thanks for your help.

(fastai version: 1.0.55)

0 Likes

(Cesar Calderon Muro) #23

Thanks @imanol this fixed my problem!

0 Likes