Custom an ensemble metric

fastai
1

Hi, I am trying to design a custom metric during training. So here are some information:

  1. I am training a classification model with images, which the input is stored in a dataframe.
  2. for each image I have a label and the trained model will give a prediction per image.
  3. in the dataframe, I also have another column named address, one address might has several images. If the images have same address, than they will have same labels.
  4. Instead of the traditional accuracy, I want to create a custom accuracy which we will ensemble same address’ images’ predictions, and then calculating the average probabilities for each class, and select the class with the highest probability as the answer for address.
    In this case, I wonder if it is possible to create a custom accuracy in fastai Metrics/Callback?
    Any suggestions will be super helpful. Thank you!
    My step 1 is creating an DataBlock
db = DataBlock(
    blocks=(ImageBlock, CategoryBlock),
    splitter=ColSplitter(col="valid"),
    get_x=partial(get_x_from_df_row, img_col="images"),
    get_y=partial(
        get_y_from_df_row,
        label_col=label_name,
        mask=False if model_type == "cnn" else True,
    ),
    item_tfms=Resize(img_size, method="squish"),
    batch_tfms=tfms,
)

my step 2:

dls = db.dataloaders(df, bs=BATCH_SIZE, num_workers=num_workers)

my step3:

learner = vision_learner(
    dls,
    model_architecture,     
    pretrained=pretrained,
    path=model_path,
    metrics=[accuracy, FBeta(beta=1)]
).to_fp16()

Now my question is if I can create a custom metric/callback?

2

I think you will find Jeremy’s notebook on multi-target classification helpful for this problem. He also walks through his approach to multi-classification in this live coding video.