How to use SaveModelCallback with Custom Metrics?

fastai
1

Trying to use IceVision together with FastAI for my object detection project using labeled bounding boxes.

IceVision has a COCOMetric Wrapper around COCO API to capture one of the metrics and returns a dictionary i.e. { “mAP”: mAP_score }

I thought I can just use SaveModelCallback(monitor=‘COCOMetric’) or SaveModelCallback(monitor=‘mAP’) to save the best performing model but I encounter these exception:

== when using monitor=‘COCOMetric’ ==

fastai/callback/tracker.py...
---> 40         if self.comp(val - self.min_delta, self.best): 
...
TypeError: unsupported operand type(s) for -: 'dict' and 'float'

== when using monitor=‘mAP’ ==

fastai/callback/tracker.py in before_fit(self)
...
---> 34         assert self.monitor in self.recorder.metric_names[1:]
...
AssertionError:

How should I get at the mAP number inside the COCOMetric dict? Is my only option to write a wrapper metric around COCOMetric e.g. COCOmAPMetric which just retrieve and return the value from inside the dict?

2

Hey Brian,

As logic would dictate and as the error says, you need to be able to compare the results of the metric function to each other. If you your metric function returns a dict or anything else that does not have built-in comparison support in Python then indeed you need to implement it yourself, such as by inheriting from the metric function (if it’s defined in a class) or just copying+pasting it otherwise and adding comparison support. It shouldn’t be more than a few lines of code, let me know if you need help :slight_smile:

3

Hi,

I tried using IceVision for object detection using the framework provided in the PlantDoc tutorial notebook. My data is in csv format. However I am getting " BBox values out of range [0.0,1.0]. Apperently this error is due to Albumentation package. How did you overcome this issue?