Hi, I am using fastai 1.0.22, and I trained my vision model, this is my code snippet:
data = ImageDataBunch.from_csv('.', folde, test, tfms, ...)
learner = create_cnn(data=data, arch=arch,
metrics=[metrics.accuracy_thresh, partial(metrics.fbeta, beta=1)],
pretrained=False,
callback_fns=[
ShowGraph,
partial(callbacks.tracker.EarlyStoppingCallback, monitor='fbeta', mode='max', min_delta=1e-4, patience=3),
partial(callbacks.tracker.SaveModelCallback, monitor='fbeta', mode='max', name='resnet50_1118'),
# partial(callbacks.tracker.ReduceLROnPlateauCallback, patience=1)
],
path='./',
model_dir='models'
)
and I got:
epoch train_loss valid_loss accuracy_thresh fbeta
...
46 0.075248 0.073362 0.974229 0.751264 (31:51)
47 0.073365 0.073218 0.974142 0.758855 (29:33)
48 0.076841 0.073179 0.974085 0.759473 (26:55)
49 0.080939 0.073682 0.974136 0.759239 (28:10)
50 0.075040 0.073418 0.973935 0.762767 (28:38)
but when I do this:
train_preds, train_ys = learner.TTA(ds_type=basic_data.DatasetType.Train)
valid_preds, valid_ys = learner.TTA(ds_type=basic_data.DatasetType.Valid)
metrics.fbeta(train_preds, train_ys, beta=1)
metrics.fbeta(valid_preds, valid_ys, beta=1)
# I got
tensor(0.1093)
tensor(0.1090)
It doesnât fit the training result, and I am wondering what are TTA returns? What are they?(train_preds, train_ys, valid_preds, valid_ys)
and I also tried:
metrics.fbeta(train_preds, train_ys, beta=1, sigmoid=False)
metrics.fbeta(valid_preds, valid_ys, beta=1, sigmoid=False)
# I got
tensor(0.6578)
tensor(0.7648)
Is it the right way to use metrics after training?
And I tried:
train_preds[:2], train_ys[:2], data.train_ds.ds.y[:2], data.train_ds.ds.class2idx
# I got
(tensor([[7.3867e-03, 3.6480e-01, 7.3652e-03, 2.9591e-03, 3.8979e-01, 1.5940e-02,
8.2336e-03, 4.3327e-01, 1.3214e-02, 1.8333e-03, 5.7280e-02, 5.5079e-02,
3.4562e-03, 8.0098e-03, 2.3170e-03, 9.1056e-03, 1.8765e-01, 5.9625e-04,
1.4326e-02, 2.8856e-03, 6.8006e-03, 6.9944e-05, 2.4585e-05, 1.2802e-05,
5.5607e-02, 2.9375e-02, 8.4388e-06, 3.0360e-05],
[4.4335e-02, 3.5788e-01, 4.5961e-03, 7.5446e-02, 1.7432e-01, 1.5085e-01,
1.2439e-02, 4.5668e-01, 5.8530e-03, 3.4558e-03, 2.7315e-02, 1.5847e-02,
2.5153e-02, 3.4241e-03, 1.0362e-02, 3.8466e-03, 9.6281e-02, 1.2357e-03,
3.4179e-02, 7.7574e-03, 8.5139e-02, 2.2935e-04, 1.2956e-04, 7.5721e-05,
1.9417e-02, 3.3238e-03, 6.3162e-05, 2.2167e-04]]),
tensor([[0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]),
[array([0, 1]), array([2, 3, 4, 1])],
{'16': 0, '0': 1, '7': 2, '1': 3, '2': 4, '5': 5, '18': 6, '25': 7, '23': 8, '24': 9, '6': 10, '11': 11, '3': 12, '12': 13, '13': 14, '14': 15, '21': 16, '20': 17, '22': 18, '17': 19, '4': 20, '8': 21, '9': 22, '10': 23, '19': 24, '26': 25, '27': 26, '15': 27})
Why is that, ys and preds are not fit? And I also use get_preds
, and it makes more sense, but still not clear:
(tensor([[2.3324e-07, 2.8542e-02, 3.7166e-04, 1.4265e-07, 2.4823e-03, 1.5732e-05,
2.9932e-10, 2.0137e-03, 1.5571e-04, 2.6469e-11, 1.0999e-06, 1.7187e-08,
4.4930e-05, 3.3567e-05, 7.1373e-07, 9.0715e-01, 1.6405e-05, 1.1514e-12,
7.4283e-07, 1.9908e-11, 1.1144e-07, 4.3902e-13, 7.7824e-17, 1.0231e-16,
1.0996e-07, 8.4115e-12, 6.8313e-17, 3.0076e-15],
[1.4445e-03, 7.6775e-01, 3.6843e-02, 1.1560e-03, 2.4584e-04, 2.2187e-03,
1.1103e-03, 2.8304e-02, 6.5590e-03, 1.5433e-05, 1.6795e-04, 1.2466e-03,
3.4120e-04, 6.0435e-03, 1.1569e-03, 3.4924e-05, 2.2100e-02, 9.4374e-05,
6.8627e-04, 1.7658e-04, 3.4890e-01, 3.1008e-06, 2.2302e-06, 9.7411e-07,
1.7396e-03, 2.1731e-03, 7.6317e-07, 1.6299e-06]]),
tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]))
So, how to use TTA result to make prediction?, what are they mean?
THANKS !