{ "id": 12702, "title": "Local evaluation method?", "url": "/competitions/diabetic-retinopathy-detection/discussion/12702", "author_name": "", "post_date": "2015-03-06T00:42:09.340Z", "votes": 2, "comment_count": 7, "views": 2971, "content": "

Hi, I'm wondering what local evaluation method or metric is suitable for approximating weighted kappa. I'm using mean square error. however it is not consistent with LB.

\n

for example, cv error (lower the better) 1.82, LB kappa 0.091; cv error 1.77, LB kappa 0.076

\n

Any comments are appreciated.

", "messages": [ { "id": "65555", "postDate": "03/06/2015 00:42:09", "content": "

Hi, I'm wondering what local evaluation method or metric is suitable for approximating weighted kappa. I'm using mean square error. however it is not consistent with LB.

\n

for example, cv error (lower the better) 1.82, LB kappa 0.091; cv error 1.77, LB kappa 0.076

\n

Any comments are appreciated.

", "rawMarkdown": "", "votes": null }, { "id": "65579", "postDate": "03/06/2015 09:01:36", "content": "

Why don't you use ... weighted kappa?

", "rawMarkdown": "", "votes": null }, { "id": "65612", "postDate": "03/06/2015 16:39:03", "content": "

[quote=Tsakalis Kostas;65579]

\n

Why don't you use ... weighted kappa?

\n

[/quote]

\n

Thank you. I foolishly thought it depends on some unknown cost. Never mind.

\n

I found this script, is it the same with what we use here?

", "rawMarkdown": "", "votes": null }, { "id": "65629", "postDate": "03/06/2015 19:57:22", "content": "

Weighted Kappa would add a little more accuracy, however the prevalence of greater than mild DR is about 40%, and greater than moderate is about 25% so the difference should not big.

", "rawMarkdown": "", "votes": null }, { "id": "65644", "postDate": "03/06/2015 22:31:45", "content": "

Jorge: In the training set, the number of cases greater than mild is 17%, and greater than moderate is 4%. You estimate these proportions are about 40% and 25% respectively. Given the random nature of the file naming IDs, I would expect the training and test datasets to have very similar proportions - a true random test sample suitable for evaluation.

\n\n

Thanks, Alastair

\n

Cases     Level                                  Proportion

\n

25810  0 - No DR             73%
 2443  1 - Mild               7%
 5292  2 - Moderate          15%
  873  3 - Severe             2%
  708  4 - Proliferative DR   2%
35126  Total

", "rawMarkdown": "", "votes": null }, { "id": "65647", "postDate": "03/06/2015 23:00:57", "content": "

Alastair:

\n

I was speaking generally and I'm not able to comment on the competition dataset.

\n

-Jorge

", "rawMarkdown": "", "votes": null }, { "id": "80141", "postDate": "05/27/2015 14:32:42", "content": "

Should I use like this: quadratic_weighted_kappa(y_true, y_pred) ?

\n\n

Regards,

\n

Emilio

", "rawMarkdown": "", "votes": null }, { "id": "80212", "postDate": "05/28/2015 03:22:59", "content": "

@Emilio Yes. That script has worked for me. I never bothered checking if it's 100% correct though ^^

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 65579, "author_name": "epinephelus", "author_url": "", "post_date": "03/06/2015 09:01:36", "content": "

Why don't you use ... weighted kappa?

", "votes": null, "replies": [] }, { "id": 65612, "author_name": "jiweiliu", "author_url": "", "post_date": "03/06/2015 16:39:03", "content": "

[quote=Tsakalis Kostas;65579]

\n

Why don't you use ... weighted kappa?

\n

[/quote]

\n

Thank you. I foolishly thought it depends on some unknown cost. Never mind.

\n

I found this script, is it the same with what we use here?

", "votes": null, "replies": [] }, { "id": 65629, "author_name": "jorge9", "author_url": "", "post_date": "03/06/2015 19:57:22", "content": "

Weighted Kappa would add a little more accuracy, however the prevalence of greater than mild DR is about 40%, and greater than moderate is about 25% so the difference should not big.

", "votes": null, "replies": [] }, { "id": 65644, "author_name": "alastairk", "author_url": "", "post_date": "03/06/2015 22:31:45", "content": "

Jorge: In the training set, the number of cases greater than mild is 17%, and greater than moderate is 4%. You estimate these proportions are about 40% and 25% respectively. Given the random nature of the file naming IDs, I would expect the training and test datasets to have very similar proportions - a true random test sample suitable for evaluation.

\n\n

Thanks, Alastair

\n

Cases     Level                                  Proportion

\n

25810  0 - No DR             73%
 2443  1 - Mild               7%
 5292  2 - Moderate          15%
  873  3 - Severe             2%
  708  4 - Proliferative DR   2%
35126  Total

", "votes": null, "replies": [] }, { "id": 65647, "author_name": "jorge9", "author_url": "", "post_date": "03/06/2015 23:00:57", "content": "

Alastair:

\n

I was speaking generally and I'm not able to comment on the competition dataset.

\n

-Jorge

", "votes": null, "replies": [] }, { "id": 80141, "author_name": "emiliocs", "author_url": "", "post_date": "05/27/2015 14:32:42", "content": "

Should I use like this: quadratic_weighted_kappa(y_true, y_pred) ?

\n\n

Regards,

\n

Emilio

", "votes": null, "replies": [] }, { "id": 80212, "author_name": "sveitser", "author_url": "", "post_date": "05/28/2015 03:22:59", "content": "

@Emilio Yes. That script has worked for me. I never bothered checking if it's 100% correct though ^^

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "65555": "", "65579": "", "65612": "", "65629": "", "65644": "", "65647": "", "80141": "", "80212": "" }, "source": "meta" }