ImageData.get_ds and test dataset

Part 2 & Alumni (2018)
1

I have a problem similar to Lesson 14 (Carvana Segmentation). I am following Carvana’s notebook but I need to use a test dataset.

Unfortunately, get_ds code has no comments and I can’t understand how to add a test dataset.

Here is what I tried:

    datasets = ImageData.get_ds(MatchedFilesDataset, (trn_x,trn_y), (val_x,val_y), tfms, path=PATH, test=testFilenames)
    md = ImageData(PATH, datasets, bs, num_workers=8, classes=None)

where:

['Validation_Balanced_256/ISIC_0012585.jpg',
 'Validation_Balanced_256/ISIC_0022029.jpg',
 'Validation_Balanced_256/ISIC_0022192.jpg']

Can someone help?

1 Like
2

i also have this problem , If you solve it, please help me

3

I solved and forgot to mention. Create a tuple test=(testfilenames, testfilenames) where testfilenames is a list of all files in test dataset (with path).

Then:
datasets = ImageData.get_ds(MatchedFilesDataset, (trn_x,trn_y), (val_x,val_y), tfms, path=PATH, test=test)

7 Likes
4

Thanks! That’s very helpful. But I was confused by another question.
When submitted in the kaggle competition, how to convert the last prediction of the network to rle_mask.

5

No idea. I was just applying Lesson’s 14 implementation for the Carvana challenge to another segmentation problem.

6

Thanks~
If you are interested, you can check out the latest split games in kaggle.

7

Hello guys,

I still have problems regarding read in test data. Can you guys help me out with the code below please?

TESTPATH = Path("../data/tgs/test")
test = (testname, TESTPATH) #testname is a list of the files in test folder.
tfms = tfms_from_model(resnet34, sz, crop_type=CropType.NO, tfm_y=TfmType.CLASS, aug_tfms=aug_tfms)
datasets = ImageData.get_ds(MatchedFilesDataset, (trn_x,trn_y), (val_x,val_y), tfms, test = test, path= PATH)
md = ImageData(PATH, datasets, bs, num_workers=16, classes=None)
denorm = md.trn_ds.denorm

Also, should I change the parameters in MatchedFilesDataset as well?

Many thanks!

8

test = (TESTPATH/testname,TESTPATH/testname)
try this

10

Hi Charm,

Thanks a lot! I think it works.

But I got another problem here, should I change MatchedFilesDataset function as well? as this error comes out.

class  MatchedFilesDataset(FilesDataset):
    def __init__(self, fnames, y, transform, path):
        self.y=y
        assert(len(fnames)==len(y))
        super().__init__(fnames, transform, path)
    def get_y(self, i): return open_image(os.path.join(self.path, self.y[i]))
    def get_c(self): return 0

import pandas as pd
testname = pd.read_csv("filename.csv")
testname = ''.join(testname['0'].values.tolist())

TESTPATH = Path("../data/tgs/test")
test = (TESTPATH/testname, TESTPATH/testname) #testname is a list of the files in test folder.
tfms = tfms_from_model(resnet34, sz, crop_type=CropType.NO, tfm_y=TfmType.CLASS, aug_tfms=aug_tfms)
datasets = ImageData.get_ds(MatchedFilesDataset, (trn_x,trn_y), (val_x,val_y), tfms, test = test, path= PATH)
md = ImageData(PATH, datasets, bs, num_workers=16, classes=None)
denorm = md.trn_ds.denorm

<ipython-input-21-055d0623841d> in __init__(self, fnames, y, transform, path)
      2     def __init__(self, fnames, y, transform, path):
      3         self.y=y
----> 4         assert(len(fnames)==len(y))
      5         super().__init__(fnames, transform, path)
      6     def get_y(self, i): return open_image(os.path.join(self.path, self.y[i]))

TypeError: object of type 'PosixPath' has no len()
11

x_names = np.array([Path(TRAIN_DN)/o for o in masks_csv[‘img’]])
test_name = (x_names ,x_names )
The format should be the same as trn_x in the training set

12

right…lol thanks!

13

congratulations, do you konw how to convert the last prediction of the network to rle_mask,when submitted in the kaggle competition?

14

I am not there yet, I need to finish IoU part first. I will let you know when I work it out.

By the way, do you have a problem running the code below?

x,y = next(iter(md.trn_dl))

It is so weird that it fails to run on colab, but it runs perfectly well on my laptop …

15

I just runs on my laptop , it’s no problem.

16

No worries, false alarm, just a batch size problem. thanks for the help!

17

@charming Thanks for this, I was stuck here too. :slightly_smiling_face:

1 Like
18

Thank you! Now I’m also looking for how to convert to rle_mask.

1 Like