{ "id": 13115, "title": "Paper on Using ANN for Ordinal Problems", "url": "/competitions/diabetic-retinopathy-detection/discussion/13115", "author_name": "", "post_date": "2015-03-29T07:45:20.330Z", "votes": 19, "comment_count": 12, "views": 4036, "content": "

As has already been pointed out, this is an ordinal problem, rather than a classification problem. I found the following paper, which describes a fairly simple way to set up ordinal problems on a neural net, helpful in setting up the inputs and outputs to my net. Enjoy:

\n

https://web.missouri.edu/~zwyw6/files/rank.pdf

", "messages": [ { "id": "68823", "postDate": "03/29/2015 07:45:20", "content": "

As has already been pointed out, this is an ordinal problem, rather than a classification problem. I found the following paper, which describes a fairly simple way to set up ordinal problems on a neural net, helpful in setting up the inputs and outputs to my net. Enjoy:

\n

https://web.missouri.edu/~zwyw6/files/rank.pdf

", "rawMarkdown": "", "votes": null }, { "id": "72642", "postDate": "04/20/2015 14:57:43", "content": "

I was also thinking about ordinal classification. Have you tried this approach?

", "rawMarkdown": "", "votes": null }, { "id": "72668", "postDate": "04/20/2015 16:49:01", "content": "

Yes. I started off using this approach exactly. Since then I'm made some super-secret modifications to the way the output is decoded that improve Kappa, but general idea is the same.  This has been good for a kappa of  ~0.63 with a single net. If I decoded the output of this same net through the decoding method they propose in the paper it's only good for a kappa of ~0.55. So there is considerable room for improvement on the decoding side.  

", "rawMarkdown": "", "votes": null }, { "id": "72672", "postDate": "04/20/2015 17:05:09", "content": "

I'm sorry for stupid question, but what is kappa?

", "rawMarkdown": "", "votes": null }, { "id": "72673", "postDate": "04/20/2015 17:10:40", "content": "

No worries. Kappa is how the contest is being scored.  Specifically quadratic kappa (https://www.kaggle.com/c/diabetic-retinopathy-detection/details/evaluation).  As it turns out, optimizing against kappa tends to reduce the accuracy as bit since you end up emphasizing the rare cases more.  If you are using Python, you can install skll and use the kappa function from there (http://skll.readthedocs.org/en/latest/api/skll.html?highlight=kappa#skll.kappa), this is what I've been doing.  If you are using some other environment, I can't help you, but I imagine there is something out there.

", "rawMarkdown": "", "votes": null }, { "id": "72700", "postDate": "04/20/2015 18:20:44", "content": "

cool, thanks!

", "rawMarkdown": "", "votes": null }, { "id": "75579", "postDate": "04/30/2015 07:19:09", "content": "

Hi Tim,

\n

Firstly congrats for your new achievement! :)

\n

And thank you for both link you provide to us.Just to clarify: under decoding you meant manipulating the output of CNN?

", "rawMarkdown": "", "votes": null }, { "id": "75621", "postDate": "04/30/2015 12:45:17", "content": "

[quote=George Solymosi;75579]

\n

Hi Tim,

\n

Firstly congrats for your new achievement! :)

\n

[/quote]

\n

Thanks!

\n

[quote=George Solymosi;75579]

\n

And thank you for both link you provide to us.Just to clarify: under decoding you meant manipulating the output of CNN?

\n

[/quote]

\n

The output of the CNN is just some number of floating point numbers.  In the paper that I referenced they use a CNN with 4 outputs and map the five cases map to "0000", "0001", "0011", "0111" and "1111".  However the output values aren't going to be exactly 0 and 1, although they use a sigmoid nonlinearity on the output to force them to be between 0 and 1 . So the problem of decoding the output, in the case of this paper,  is one of mapping four floating point values between, 0 and 1, onto the integers 0 to 4.  The paper proposes a very simple and easy to understand scheme for doing this, which I'll let you read for yourself, but it turns out that one can do better.

\n

*EDIT*

\n

When I said above that in the paper they use a CNN with four outputs, etc, etc. I meant that is what one would do if applying that paper to THIS problem. In general the number of outputs would vary on the number of ordinal categories. Specifically there would be one fewer outputs than categories. Sorry for any confusion.

\n

*EDIT*

\n

One way to think of the way they set up the mapping in the paper is that four bits correspond to "at least case 1", "at least case 2", "at least case 3" and "at least case 4".  This is only slightly different than using a standard categorical setup were there would be 5 outputs "00001", "00010", "00100", "01000" and "10000" which correspond to "exactly case 0", "exactly case 1", etc. However, in the categorical case, a softmax function is typically applied to the output to force one of the values to dominate and decoding is typically just a manner of applying argmax to the outputs. 

\n

I hope that helps.

", "rawMarkdown": "", "votes": null }, { "id": "132360", "postDate": "08/24/2016 21:33:33", "content": "

A current link for A Neural Network Approach to Ordinal Regression:

\n\n

http://orca.st.usm.edu/~zwang/files/rank.pdf

", "rawMarkdown": "A current link for _A Neural Network Approach to Ordinal Regression_:\r\n \r\nhttp://orca.st.usm.edu/~zwang/files/rank.pdf", "votes": null }, { "id": "554983", "postDate": "06/18/2019 09:25:23", "content": "

Hi @bitsofbits ,

\n\n

I assume that this competition is over. I am trying to apply ordinal classification for my data for age classification. Could you please share the super secret modification you mentioned.

\n\n

Srini

", "rawMarkdown": "Hi @bitsofbits ,\n\nI assume that this competition is over. I am trying to apply ordinal classification for my data for age classification. Could you please share the super secret modification you mentioned.\n\nSrini", "votes": null }, { "id": "638305", "postDate": "10/01/2019 18:11:21", "content": "

Thanks for the idea. Would you mind to let me know if there is an existing python package function to perform such encoding? I didn't find the exactly same function from the latest sklearn.

\n\n

Thanks again!

", "rawMarkdown": "Thanks for the idea. Would you mind to let me know if there is an existing python package function to perform such encoding? I didn't find the exactly same function from the latest sklearn.\n\nThanks again!", "votes": null }, { "id": "641627", "postDate": "10/04/2019 21:26:20", "content": "

@yantinghuang, not that I know of. It's been a while since I looked at this, but you should be able to get the correct effect using one-hot encoding followed by something involving cumsum. Maybe '1 - cumsum(one_hot(x))'? But check that before you try it.

", "rawMarkdown": "yantinghuang, not that I know of. It's been a while since I looked at this, but you should be able to get the correct effect using one-hot encoding followed by something involving cumsum. Maybe '1 - cumsum(one_hot(x))'? But check that before you try it.", "votes": null }, { "id": "2327149", "postDate": "07/02/2023 17:24:41", "content": "

I'm trying to apply this to a problem of my own and I really like the approach. I know this thread is way old but I'd really appreciate some advice on your superior decoding method for the ordinal encoded output! Thanks.

", "rawMarkdown": "I'm trying to apply this to a problem of my own and I really like the approach. I know this thread is way old but I'd really appreciate some advice on your superior decoding method for the ordinal encoded output! Thanks.", "votes": null } ], "comments": [ { "id": 72642, "author_name": "", "author_url": "", "post_date": "04/20/2015 14:57:43", "content": "

I was also thinking about ordinal classification. Have you tried this approach?

", "votes": null, "replies": [] }, { "id": 72668, "author_name": "bitsofbits", "author_url": "", "post_date": "04/20/2015 16:49:01", "content": "

Yes. I started off using this approach exactly. Since then I'm made some super-secret modifications to the way the output is decoded that improve Kappa, but general idea is the same.  This has been good for a kappa of  ~0.63 with a single net. If I decoded the output of this same net through the decoding method they propose in the paper it's only good for a kappa of ~0.55. So there is considerable room for improvement on the decoding side.  

", "votes": null, "replies": [ { "id": 554983, "author_name": "sreenivasaupadhyaya", "author_url": "", "post_date": "06/18/2019 09:25:23", "content": "

Hi @bitsofbits ,

\n\n

I assume that this competition is over. I am trying to apply ordinal classification for my data for age classification. Could you please share the super secret modification you mentioned.

\n\n

Srini

", "votes": null, "replies": [] }, { "id": 2327149, "author_name": "abalter85", "author_url": "", "post_date": "07/02/2023 17:24:41", "content": "

I'm trying to apply this to a problem of my own and I really like the approach. I know this thread is way old but I'd really appreciate some advice on your superior decoding method for the ordinal encoded output! Thanks.

", "votes": null, "replies": [] } ] }, { "id": 72672, "author_name": "", "author_url": "", "post_date": "04/20/2015 17:05:09", "content": "

I'm sorry for stupid question, but what is kappa?

", "votes": null, "replies": [] }, { "id": 72673, "author_name": "bitsofbits", "author_url": "", "post_date": "04/20/2015 17:10:40", "content": "

No worries. Kappa is how the contest is being scored.  Specifically quadratic kappa (https://www.kaggle.com/c/diabetic-retinopathy-detection/details/evaluation).  As it turns out, optimizing against kappa tends to reduce the accuracy as bit since you end up emphasizing the rare cases more.  If you are using Python, you can install skll and use the kappa function from there (http://skll.readthedocs.org/en/latest/api/skll.html?highlight=kappa#skll.kappa), this is what I've been doing.  If you are using some other environment, I can't help you, but I imagine there is something out there.

", "votes": null, "replies": [] }, { "id": 72700, "author_name": "", "author_url": "", "post_date": "04/20/2015 18:20:44", "content": "

cool, thanks!

", "votes": null, "replies": [] }, { "id": 75579, "author_name": "", "author_url": "", "post_date": "04/30/2015 07:19:09", "content": "

Hi Tim,

\n

Firstly congrats for your new achievement! :)

\n

And thank you for both link you provide to us.Just to clarify: under decoding you meant manipulating the output of CNN?

", "votes": null, "replies": [] }, { "id": 75621, "author_name": "bitsofbits", "author_url": "", "post_date": "04/30/2015 12:45:17", "content": "

[quote=George Solymosi;75579]

\n

Hi Tim,

\n

Firstly congrats for your new achievement! :)

\n

[/quote]

\n

Thanks!

\n

[quote=George Solymosi;75579]

\n

And thank you for both link you provide to us.Just to clarify: under decoding you meant manipulating the output of CNN?

\n

[/quote]

\n

The output of the CNN is just some number of floating point numbers.  In the paper that I referenced they use a CNN with 4 outputs and map the five cases map to "0000", "0001", "0011", "0111" and "1111".  However the output values aren't going to be exactly 0 and 1, although they use a sigmoid nonlinearity on the output to force them to be between 0 and 1 . So the problem of decoding the output, in the case of this paper,  is one of mapping four floating point values between, 0 and 1, onto the integers 0 to 4.  The paper proposes a very simple and easy to understand scheme for doing this, which I'll let you read for yourself, but it turns out that one can do better.

\n

*EDIT*

\n

When I said above that in the paper they use a CNN with four outputs, etc, etc. I meant that is what one would do if applying that paper to THIS problem. In general the number of outputs would vary on the number of ordinal categories. Specifically there would be one fewer outputs than categories. Sorry for any confusion.

\n

*EDIT*

\n

One way to think of the way they set up the mapping in the paper is that four bits correspond to "at least case 1", "at least case 2", "at least case 3" and "at least case 4".  This is only slightly different than using a standard categorical setup were there would be 5 outputs "00001", "00010", "00100", "01000" and "10000" which correspond to "exactly case 0", "exactly case 1", etc. However, in the categorical case, a softmax function is typically applied to the output to force one of the values to dominate and decoding is typically just a manner of applying argmax to the outputs. 

\n

I hope that helps.

", "votes": null, "replies": [] }, { "id": 132360, "author_name": "zygmunt", "author_url": "", "post_date": "08/24/2016 21:33:33", "content": "

A current link for A Neural Network Approach to Ordinal Regression:

\n\n

http://orca.st.usm.edu/~zwang/files/rank.pdf

", "votes": null, "replies": [] }, { "id": 638305, "author_name": "yantinghuang", "author_url": "", "post_date": "10/01/2019 18:11:21", "content": "

Thanks for the idea. Would you mind to let me know if there is an existing python package function to perform such encoding? I didn't find the exactly same function from the latest sklearn.

\n\n

Thanks again!

", "votes": null, "replies": [] }, { "id": 641627, "author_name": "bitsofbits", "author_url": "", "post_date": "10/04/2019 21:26:20", "content": "

@yantinghuang, not that I know of. It's been a while since I looked at this, but you should be able to get the correct effect using one-hot encoding followed by something involving cumsum. Maybe '1 - cumsum(one_hot(x))'? But check that before you try it.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "68823": "", "72642": "", "72668": "", "72672": "", "72673": "", "72700": "", "75579": "", "75621": "", "132360": "A current link for _A Neural Network Approach to Ordinal Regression_:\r\n \r\nhttp://orca.st.usm.edu/~zwang/files/rank.pdf", "554983": "Hi @bitsofbits ,\n\nI assume that this competition is over. I am trying to apply ordinal classification for my data for age classification. Could you please share the super secret modification you mentioned.\n\nSrini", "638305": "Thanks for the idea. Would you mind to let me know if there is an existing python package function to perform such encoding? I didn't find the exactly same function from the latest sklearn.\n\nThanks again!", "641627": "yantinghuang, not that I know of. It's been a while since I looked at this, but you should be able to get the correct effect using one-hot encoding followed by something involving cumsum. Maybe '1 - cumsum(one_hot(x))'? But check that before you try it.", "2327149": "I'm trying to apply this to a problem of my own and I really like the approach. I know this thread is way old but I'd really appreciate some advice on your superior decoding method for the ordinal encoded output! Thanks." }, "source": "meta" }