Calculating a Specificity Metric

madhavajay · September 30, 2019, 4:21pm

I need to calculate the Sensitivity and Specificity.
According to these:

Sensitivity is the same as Recall, so that one is done.

Recall = tp / tp + fn
Sensitivity = Recall
Precision = tp / tp + fp

All of these are in the fast.ai metrics already:
https://docs.fast.ai/metrics.html

Specificity is defined as:
Specificity = tn / tn + fp

I don’t see it in the metrics.
I also don’t see any way to calculate the TN required for specificity:

github.com

fastai/fastai/blob/master/fastai/metrics.py

"Implements various metrics to measure training accuracy"
from .torch_core import *
from .callback import *
from .layers import *
from .basic_train import LearnerCallback

__all__ = ['error_rate', 'accuracy', 'accuracy_thresh', 'dice', 'exp_rmspe', 'fbeta','FBeta', 'mse', 'mean_squared_error',
            'mae', 'mean_absolute_error', 'rmse', 'root_mean_squared_error', 'msle', 'mean_squared_logarithmic_error',
            'explained_variance', 'r2_score', 'top_k_accuracy', 'KappaScore', 'ConfusionMatrix', 'MatthewsCorreff',
            'Precision', 'Recall', 'R2Score', 'ExplainedVariance', 'ExpRMSPE', 'RMSE', 'Perplexity', 'AUROC', 'auc_roc_score', 
            'roc_curve', 'MultiLabelFbeta', 'foreground_acc']

def fbeta(y_pred:Tensor, y_true:Tensor, thresh:float=0.2, beta:float=2, eps:float=1e-9, sigmoid:bool=True)->Rank0Tensor:
    "Computes the f_beta between `preds` and `targets`"
    beta2 = beta ** 2
    if sigmoid: y_pred = y_pred.sigmoid()
    y_pred = (y_pred>thresh).float()
    y_true = y_true.float()
    TP = (y_pred*y_true).sum(dim=1)
    prec = TP/(y_pred.sum(dim=1)+eps)

This file has been truncated. show original

So we need tn and fp.
fp is in Precision:

def _precision(self):
    # Precision = tp / tp + fp
    prec = torch.diag(self.cm) / self.cm.sum(dim=0)
    if self.average is None: return prec
    else:
        weights = self._weights(avg=self.average)
        return (prec * weights).sum()

So that means that:

Precision = tp / tp + fp = torch.diag(self.cm) / self.cm.sum(dim=0)

TP = torch.diag(self.cm)
TP + FP = self.cm.sum(dim=0)
So to rearrange:
FP = (TP + FP) - TP = self.cm.sum(dim=0) - torch.diag(self.cm)
I also assume theres a better way since this seems barbaric.

However for TN, I don’t see any examples.
I can see in recall we could do the same trickery to get fn by subtracting tp from self.cm.sum(dim=1)

rec = torch.diag(self.cm) / self.cm.sum(dim=1)

So seems that there are two ways to sum the confusion matrix by row or column which result in the fp or the fn of the denominator. Is there a way to get the tn to calculate the specificity, or perhaps a way to get the total results to subtract the other values from it?

I assume the weighting matters for non binary classification scoring but will it change in calculating specificity?

Is it just me or are these literally the worst names for these things in the world. My human brain doesn’t seem to handle binary logic AND its synonyms very well as double negatives and double positives?

Also is there a reason why specificity is not included, it seems popular when discussing medical results, but maybe its fallen out of favour for some reason?

Marcola · September 23, 2020, 11:59pm

Can anybody help me? I’m going through the same challenge…

stochastic · September 28, 2020, 8:46am

Same here! Would appreciate the help.

stochastic · September 29, 2020, 8:33am

For a temporary solution, I found the following:

interp = ClassificationInterpretation.from_learner(learn)
upp, low = interp.confusion_matrix()
tn, fp = upp[0], upp[1]
fn, tp = low[0], low[1]
specificity = tn/(fp + tn)