Filter_by_func on ImageList

sinsji · February 28, 2019, 2:53pm

For a project I want to apply a filter to the image data.
The filter should select only images with certain labels in the accompanied .csv file.
How can I achieve this?

Before filtering I retrieved the data like this:

data = (ImageList.from_csv(PATH, folder='train', csv_name='labels.csv', cols='image_id')
        .use_partial_data(sample_pct = .1, seed = 31)
        .random_split_by_pct(valid_pct=0.25, seed = 29)
        .label_from_df(cols='dx')
        .transform(tfms, size = 224)
        .databunch(bs=64)).normalize(imagenet_stats)

My current idea is to list all needed files with Pandas. Then compare for each item in data.items if the item is present in the earlier created list. Somehow I think there is an easier way using either filter_by_func or label_cls. Any tips?

ucalyptus · May 30, 2019, 1:34pm

did you get a solution?

ste · May 30, 2019, 7:59pm

ImageList.from_csv use ImageList.from_df behind the scene.

github.com

fastai/fastai/blob/7895034df91c526de3e5c98a9deb9566b1fe192e/fastai/vision/data.py#L122




@classmethod
def from_df(cls, path:PathOrStr, df:pd.DataFrame, folder:PathOrStr=None, label_delim:str=None, valid_pct:float=0.2,
            fn_col:IntsOrStrs=0, label_col:IntsOrStrs=1, suffix:str='', **kwargs:Any)->'ImageDataBunch':
    "Create from a `DataFrame` `df`."
    src = (ImageList.from_df(df, path=path, folder=folder, suffix=suffix, cols=fn_col)
            .split_by_rand_pct(valid_pct)
            .label_from_df(label_delim=label_delim, cols=label_col))
    return cls.create_from_ll(src, **kwargs)


@classmethod
def from_csv(cls, path:PathOrStr, folder:PathOrStr=None, label_delim:str=None, csv_labels:PathOrStr='labels.csv',
             valid_pct:float=0.2, fn_col:int=0, label_col:int=1, suffix:str='', delimiter:str=None,
             header:Optional[Union[int,str]]='infer', **kwargs:Any)->'ImageDataBunch':
    "Create from a csv file in `path/csv_labels`."
    path = Path(path)
    df = pd.read_csv(path/csv_labels, header=header, delimiter=delimiter)
    return cls.from_df(path, df, folder=folder, label_delim=label_delim, valid_pct=valid_pct,
            fn_col=fn_col, label_col=label_col, suffix=suffix, **kwargs)


@classmethod

So my suggestion is to read your data in a DataFrame and filter them using standard pandas code then use ImageList.from_df

dipam7 · May 31, 2019, 5:47am

Hey, if I understand the question correctly, why don’t you subset your df using pandas to select the labels you want. You can store the results in a new df or new csv and then use that to create a data bunch. Suppose you have 5 classes and you don’t want classes numbered 2 and 4 you can do it as follows:

df_sub = df[(df['label'] != 2) & (df['label'] != 4)]

Hope this helps

Matthieu · August 27, 2019, 7:38am

Thanks for the idea Stephano!