Retrieve data from original Item in metric function

fastai
1

I’m quite new to fastai, and while toying around with some projects I keep finding myself in the position to want to have access to the original data in a metric function, rather than just the prediction and target tensors. Some examples:

  • I was playing around with a model that predicts some points using a heat map (it’s a bit more involved than this but that doesn’t matter for the example). The original label data is a set of coordinates, which is converted to a heat map, which is then to be predicted from the input data by some conv net. The operation turning the heat map back into points is not super expensive, but it does slow things down and is kind of wasteful given that the target points are known information.
  • In a regression model I want to print out the percentage of predictions being within some epsilon value of a target value. Due to normalization, both the target value and the predicted value are scaled by a factor present in the original dataframe, so the epsilon no longer makes sense. Access to the original dataframe in the metric would tell me what to scale it by (or let me scale the predictions / targets).

I’ve been going through the fastai code for the past few hours, but I can only figure out how to access the fixed items that the CallbackHandler puts in the state_dict from any metric callback and I don’t see anything that’s immediately useful to this problem there. Given how often this comes up for me, I was wondering if somebody has encountered this problem and came up with a solution.

2

For now I’ve settled on a hack that I probably wouldn’t recommend if you care about the stability and maintainability of your code, and probably doesn’t solve the scenario with performance concerns… but does part of the job anyway, so I’ll share it here. It involves wrapping metrics in a different Callback class that you also pass your data:

class AugmentedAverageMetric(AverageMetric):
    def __init__(self, func, data):
        super().__init__(func)
        self.data = data
        self.idx_start = 0
    
    def on_epoch_begin(self, *args, **kwargs):
        super().on_epoch_begin(*args, **kwargs)
        self.idx_start = 0
    
    def on_batch_end(self, last_output, last_target, **kwargs):
        idx,bs = self.idx_start,self.data.batch_size
        batch_items = self.data.valid_ds[idx:idx+bs]
        
        self.idx_start += bs

        # HACK We know `AverageMetric.on_batch_end()` expands the target
        # as *last_target, and so we pass a list with the target and batch items
        # to get the batch items as the third argument to our metric function
        super().on_batch_end(last_output, [last_target,batch_items], **kwargs)

Now I can define a metric like:

def my_metric(y_pred, y_target, batch_items):
    return whatever_you_must_do_here()

And pass it to a learner like:

tabular_learner(data, layers=[100, 50], metrics=[AugmentedAverageMetric(my_metric, data)])

And as a third argument it will receive a slice from the validation data loader with labels and values. Use at your own risk ;).

3

But I don’t believe you normalize your y’s. (Not even Rossmann did this). Plus your model (if I assume tabular here) has a y_range to ensure your results are in the expected range (which is not 0,1 always with regression.

4

@muellerzr You are correct, the y’s aren’t normalized. This example is unfortunately also a bit more involved than that, in that it’s comparing to some other value in the data frame that is. I’ll see if I can clarify a bit later (I have to leave in 5 minutes so trying that now will guarantee I’m late to my next meeting ;)). Generally though I’ve found myself wanting access to the general dataframe to create metrics that don’t immediately follow from the exact tensors that come out of the model.

5

Hello @ElteHupkes ,
Here Andrea.
I have a similar problem. I want to run the model many many times and extract for each run the accuracy/metrics at which it converges, for many datasets. Saving it into a dataframe, with also meanwhile keeping the datasets and understand what in the data is creating this behavior of the accuracy.
I need to extract the value for the accuracy after it converges at the learning rate chosen (can I also extract it?). Do you know how can I interrogate the learner which is like:
#A TrackerCallback that terminates training when monitored quantity stops improving.
learn1 = vision_learner(ClofRBCs, ResN[v], metrics=[accuracy,Precision(average=None),Recall(average=None)], cbs= EarlyStoppingCallback(monitor=‘accuracy’, min_delta=0.001, patience=4, reset_on_fit=True)])

Thanks!