How to use TTA make prediction

Hi, I am using fastai 1.0.22, and I trained my vision model, this is my code snippet:

data = ImageDataBunch.from_csv('.', folde, test, tfms, ...)

learner = create_cnn(data=data, arch=arch, 
                     metrics=[metrics.accuracy_thresh, partial(metrics.fbeta, beta=1)], 
                     pretrained=False, 
                     callback_fns=[
                         ShowGraph, 
                         partial(callbacks.tracker.EarlyStoppingCallback, monitor='fbeta', mode='max', min_delta=1e-4, patience=3),
                         partial(callbacks.tracker.SaveModelCallback, monitor='fbeta', mode='max', name='resnet50_1118'),
#                          partial(callbacks.tracker.ReduceLROnPlateauCallback, patience=1)
                     ],
                     path='./',
                     model_dir='models'
                    )

and I got:

epoch  train_loss  valid_loss  accuracy_thresh  fbeta  
... 
46     0.075248    0.073362    0.974229         0.751264  (31:51)
47     0.073365    0.073218    0.974142         0.758855  (29:33)
48     0.076841    0.073179    0.974085         0.759473  (26:55)
49     0.080939    0.073682    0.974136         0.759239  (28:10)
50     0.075040    0.073418    0.973935         0.762767  (28:38)

but when I do this:

train_preds, train_ys = learner.TTA(ds_type=basic_data.DatasetType.Train)
valid_preds, valid_ys = learner.TTA(ds_type=basic_data.DatasetType.Valid)
metrics.fbeta(train_preds, train_ys, beta=1)
metrics.fbeta(valid_preds, valid_ys, beta=1)
# I got 
tensor(0.1093)
tensor(0.1090)

It doesn’t fit the training result, and I am wondering what are TTA returns? What are they?(train_preds, train_ys, valid_preds, valid_ys)

and I also tried:

metrics.fbeta(train_preds, train_ys, beta=1, sigmoid=False)
metrics.fbeta(valid_preds, valid_ys, beta=1, sigmoid=False)
# I got
tensor(0.6578)
tensor(0.7648)

Is it the right way to use metrics after training?

And I tried:

train_preds[:2], train_ys[:2], data.train_ds.ds.y[:2], data.train_ds.ds.class2idx
# I got
(tensor([[7.3867e-03, 3.6480e-01, 7.3652e-03, 2.9591e-03, 3.8979e-01, 1.5940e-02,
          8.2336e-03, 4.3327e-01, 1.3214e-02, 1.8333e-03, 5.7280e-02, 5.5079e-02,
          3.4562e-03, 8.0098e-03, 2.3170e-03, 9.1056e-03, 1.8765e-01, 5.9625e-04,
          1.4326e-02, 2.8856e-03, 6.8006e-03, 6.9944e-05, 2.4585e-05, 1.2802e-05,
          5.5607e-02, 2.9375e-02, 8.4388e-06, 3.0360e-05],
         [4.4335e-02, 3.5788e-01, 4.5961e-03, 7.5446e-02, 1.7432e-01, 1.5085e-01,
          1.2439e-02, 4.5668e-01, 5.8530e-03, 3.4558e-03, 2.7315e-02, 1.5847e-02,
          2.5153e-02, 3.4241e-03, 1.0362e-02, 3.8466e-03, 9.6281e-02, 1.2357e-03,
          3.4179e-02, 7.7574e-03, 8.5139e-02, 2.2935e-04, 1.2956e-04, 7.5721e-05,
          1.9417e-02, 3.3238e-03, 6.3162e-05, 2.2167e-04]]),
 tensor([[0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.,
          0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
          0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]),
 [array([0, 1]), array([2, 3, 4, 1])],
 {'16': 0,   '0': 1,   '7': 2,   '1': 3,   '2': 4,   '5': 5,   '18': 6,   '25': 7,   '23': 8,   '24': 9,   '6': 10,   '11': 11,    '3': 12,   '12': 13,   '13': 14,   '14': 15,   '21': 16,   '20': 17,   '22': 18,   '17': 19,   '4': 20,   '8': 21,   '9': 22,   '10': 23,   '19': 24,   '26': 25,   '27': 26,   '15': 27})

Why is that, ys and preds are not fit? And I also use get_preds, and it makes more sense, but still not clear:

(tensor([[2.3324e-07, 2.8542e-02, 3.7166e-04, 1.4265e-07, 2.4823e-03, 1.5732e-05,
          2.9932e-10, 2.0137e-03, 1.5571e-04, 2.6469e-11, 1.0999e-06, 1.7187e-08,
          4.4930e-05, 3.3567e-05, 7.1373e-07, 9.0715e-01, 1.6405e-05, 1.1514e-12,
          7.4283e-07, 1.9908e-11, 1.1144e-07, 4.3902e-13, 7.7824e-17, 1.0231e-16,
          1.0996e-07, 8.4115e-12, 6.8313e-17, 3.0076e-15],
         [1.4445e-03, 7.6775e-01, 3.6843e-02, 1.1560e-03, 2.4584e-04, 2.2187e-03,
          1.1103e-03, 2.8304e-02, 6.5590e-03, 1.5433e-05, 1.6795e-04, 1.2466e-03,
          3.4120e-04, 6.0435e-03, 1.1569e-03, 3.4924e-05, 2.2100e-02, 9.4374e-05,
          6.8627e-04, 1.7658e-04, 3.4890e-01, 3.1008e-06, 2.2302e-06, 9.7411e-07,
          1.7396e-03, 2.1731e-03, 7.6317e-07, 1.6299e-06]]),
 tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.,
          0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
          0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]))

So, how to use TTA result to make prediction?, what are they mean?

THANKS !

1 Like

TTA prediction return probabilities, not the last activations (because they took the average of the probabilities with different data augmentation), you should use fbeta with the option sigmoid=False to match your training results.

thanks for reply, and what does ys mean?

ys is your target, the y of your dataset. It’s one-hot encoded in the case of a multi-class problem.

Hello,
I use fast.ai v.1.0.

I learn and save my model, and try load model and do .TTA(), get error!

learn.load(‘my_model’)
log_preds,y = learn.TTA()
probs = np.mean(np.exp(log_preds),0)

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 223 and 224 in dimension 2 at /pytorch/aten/src/TH/generic/THTensorMoreMath.cpp:1333

Also tryed call:
log_preds,y = learn.TTA(ds_type=fastai.basic_data.DatasetType.Valid)

And I get the same error.

Method doesn’t work in fast.ai v.1.0?

This was a bug that has been fixed in 1.0.40, just udpate your library and you should be fine.

2 Likes

Error:
1 # log_preds,y = learn.TTA()
2 log_preds,y = learn.TTA(ds_type=fastai.basic_data.DatasetType.Valid)
----> 3 probs = np.mean(np.exp(log_preds),0)

~/.pyenv/lib/python3.6/site-packages/numpy/core/fromnumeric.py in mean(a, axis, dtype, out, keepdims)
2915 pass
2916 else:
-> 2917 return mean(axis=axis, dtype=dtype, out=out, **kwargs)
2918
2919 return _methods._mean(a, axis=axis, dtype=dtype,

TypeError: mean() missing 3 required positional argument: “dim”, “keepdim”, “dtype”

Whether you need to retrain the model 1.0.39 on the new version fast.ai 1.0.40 ?

I have the same error. Does anyone know how to solve this?

1 Like

I did this:

(The dataset has 10 classes)

Shape of log_preds is (10000, 10) (test samples = 10000)

log_preds, y = learner.to_fp32().TTA(ds_type=DatasetType.Test)
predIdxs = np.argmax(log_preds.numpy(), axis=1)
2 Likes