Cohen's Kappa score

Part 1 (2019)
1

Hello all,

I would like to use the Cohen’s Kappa score as a metric (playing with the Diabetic Retinopathy dataset)

I tried the following:

learn = create_cnn(data, models.resnet34, metrics=[error_rate, KappaScore()])

but at the end of the epoch I got:

RuntimeError: Expected object of backend CUDA but got backend CPU for argument #2 'other'

How can I get the Cohen’s Kappa score to work?

2

Try the following :

learn = create_cnn(data, models.resnet34, metrics=[error_rate, KappaScore])

You want to pass reference to the function, or in this case the class.

3

I get this:

TypeError: __init__() takes from 1 to 2 positional arguments but 3 were given

4

Where is that error coming from ? The KappaScore function ?

5

This is when I run learn.fit.one_cycle(4)

Note that KappaScore is a class, not a function…

Here’s a link for it: https://docs.fast.ai/metrics.html#KappaScore

6

This error means something involved in your function is on the CPU when it should be on the GPU.

7

How can this be fixed? This is a fastai function so is there a bug with the fastai function?

8

This was fixed by upgrading the version of fastai in Kaggle Kernels (the platform that I am using)… turns out that fastai 1.0.39 has this bug, which I guess has to do with the confusion matrix because this problem happened with FBeta as well which depends on the ConfusionMatrix class like KappaScore… but fastai 1.0.40 has it fixed…

.

9

The code I use (and which is working well) is (fastai 1.0.50.post1) :

kappa = KappaScore()
kappa.weights = "quadratic"
learn = cnn_learner(data, models.resnet34, metrics=[error_rate, kappa])
10

hi all… I have fastai ‘1.0.53.post2’ version but still i have the same problem with KappaScore as metrics
TypeError: __init__() takes from 1 to 2 positional arguments but 3 were given

so i cannot figure out what is going wrong.

11

Some code below:
kappa = KappaScore()
kappa.weights = “quadratic”
learn = cnn_learner(data, models.resnet50, metrics=kappa)

I got a RuntimeError:Expected object of scalar type Float but got scalar type Long for argument #2 ‘other’
How can i solved it?

1 Like
12

I have the same error. any advice please?

13

I have the same error message but it is due to an error creating the ImageList