{
  "id": 13354,
  "title": "Cant Classify the images of the Class 1",
  "url": "/competitions/diabetic-retinopathy-detection/discussion/13354",
  "author_name": "",
  "post_date": "2015-04-14T07:25:30.513Z",
  "votes": null,
  "comment_count": 6,
  "views": 2161,
  "content": "<p>Hello&nbsp;</p>\n<p>The Models that I tried, had always very low 1 Classification... almost none of the ~52000 images... Anyone got a hint for that problem?</p>",
  "messages": [
    {
      "id": "71456",
      "postDate": "04/14/2015 07:25:30",
      "content": "<p>Hello&nbsp;</p>\n<p>The Models that I tried, had always very low 1 Classification... almost none of the ~52000 images... Anyone got a hint for that problem?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "71457",
      "postDate": "04/14/2015 07:31:14",
      "content": "<p>And here is a confusion matrix for a little test set:</p>\n<p>&nbsp; &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 1&nbsp;&nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; 4</p>\n<p>0 5137 470 892 &nbsp; 106 &nbsp; 97</p>\n<p>2 58 &nbsp; &nbsp; &nbsp;11 &nbsp; 110 &nbsp; 39 &nbsp; 31</p>\n<p>3 &nbsp;2 &nbsp; &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;6 &nbsp; &nbsp; 12 &nbsp; &nbsp;3</p>\n<p>4 &nbsp;9 &nbsp; &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 14 &nbsp; &nbsp; 8 &nbsp; &nbsp;20</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "71470",
      "postDate": "04/14/2015 09:57:29",
      "content": "<p>Hirts -&nbsp;There is no 1 row in your matrix?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "71476",
      "postDate": "04/14/2015 10:38:56",
      "content": "<p>[quote=Alexander Izvorski;71470]</p>\n<p>Hirts -&nbsp;There is no 1 row in your matrix?</p>\n<p>[/quote]</p>\n\n<p>Hi Alexander</p>\n\n<p>Yes, We didnt classify any of the images as 1, thats the problem :-)</p>\n<p>the 1 row would be :&nbsp;</p>\n<p>&nbsp; &nbsp; 0 &nbsp;1 &nbsp;2 &nbsp;3 &nbsp;4&nbsp;</p>\n<p>1 &nbsp;0 &nbsp;0 &nbsp;0 &nbsp;0 &nbsp;0&nbsp;</p>\n\n<p>but there are 482 images with the class 1 in the testset</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "71478",
      "postDate": "04/14/2015 10:46:29",
      "content": "<p>I see, your confusion matrix is transposed relative to what I'm used to. &nbsp;Normally rows=actual columns=predicted.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "82426",
      "postDate": "06/19/2015 17:51:20",
      "content": "<p>Yes I'm seeing that too. My confusion matrix for a ~0.5 kappa:</p>\n<p><code>T\\P | 0 &nbsp; &nbsp; &nbsp;| 1 &nbsp; | 2 &nbsp; &nbsp;| 3 &nbsp;| 4 &nbsp;|</code><code><br>0 &nbsp; | 3243 &nbsp; | 14 &nbsp;| 249 &nbsp;| 25 | 45 |<br> 1 &nbsp; | 285 &nbsp; &nbsp;| 5 &nbsp; | 35 &nbsp; | 8 &nbsp;| 5 &nbsp;|<br> 2 &nbsp; | 393 &nbsp; &nbsp;| 4 &nbsp; | 251 &nbsp;| 62 | 22 |<br> 3 &nbsp; | 31 &nbsp; &nbsp; | 0 &nbsp; | 40 &nbsp; | 39 | 10 |<br> 4 &nbsp; | 12 &nbsp; &nbsp; | 0 &nbsp; | 25 &nbsp; | 20 | 41 |<br>&nbsp;</code></p>\n\n<p><code>&nbsp;</code></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "82876",
      "postDate": "06/28/2015 22:12:50",
      "content": "<p>I can say that following the conversation on this <a href=\"https://www.kaggle.com/c/diabetic-retinopathy-detection/forums/t/13115/paper-on-using-ann-for-ordinal-problems\">forum page</a>&nbsp;about ordinal regression led me a little away from this problem. &nbsp;</p>\n<p>Here is a confusion matrix where the only thing I changed between the run that generated the confusion matrix<a href=\"https://www.kaggle.com/c/diabetic-retinopathy-detection/forums/t/13354/cant-classify-the-images-of-the-class-1/82426#post82426\">&nbsp;I posted above</a>&nbsp;and the run that generated this one&nbsp;is&nbsp;the error function. Here, its taken directly from the paper mentioned in the forum&nbsp;(which is a little different than what the posters talk about). Specifically I used relative&nbsp;entropy as in the paper (instead of <a href=\"http://deeplearning.net/software/theano/library/tensor/nnet/nnet.html#tensor.nnet.categorical_crossentropy\">categorical cross entropy</a>&nbsp;in the forum) with 5&nbsp;different target vectors, each of length k=5, just like the paper specifies (which is different than what's mentioned in the forum). This gets ~0.53 kappa.</p>\n<p><code>Confusion Matrix<br>T\\P| 0 &nbsp; &nbsp;| 1 &nbsp;| 2 &nbsp; | 3 &nbsp;| 4 &nbsp;|<br> 0 &nbsp;| 3314 | 65 | 168 | 12 | 17 |<br> 1 &nbsp;| 293 &nbsp;| 18 | 23 &nbsp;| 1 &nbsp;| 3 &nbsp;|<br> 2 &nbsp;| 437 &nbsp;| 28 | 197 | 44 | 26 |<br> 3 &nbsp;| 22 &nbsp; | 5 &nbsp;| 34 &nbsp;| 33 | 26 |<br> 4 &nbsp;| 16 &nbsp; | 5 &nbsp;| 22 &nbsp;| 13 | 42 |</code></p>\n\n<p>EDIT:</p>\n<p>---</p>\n<p>After more experiments I can say that this problem goes away for me when I opt from a softmax+categorical crossentropy to sigmoid+relative entropy final layer. The ordinal regression helps further on top of that.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 71457,
      "author_name": "hirtste1",
      "author_url": "",
      "post_date": "04/14/2015 07:31:14",
      "content": "<p>And here is a confusion matrix for a little test set:</p>\n<p>&nbsp; &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 1&nbsp;&nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; 4</p>\n<p>0 5137 470 892 &nbsp; 106 &nbsp; 97</p>\n<p>2 58 &nbsp; &nbsp; &nbsp;11 &nbsp; 110 &nbsp; 39 &nbsp; 31</p>\n<p>3 &nbsp;2 &nbsp; &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;6 &nbsp; &nbsp; 12 &nbsp; &nbsp;3</p>\n<p>4 &nbsp;9 &nbsp; &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 14 &nbsp; &nbsp; 8 &nbsp; &nbsp;20</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 71470,
      "author_name": "aizvorski",
      "author_url": "",
      "post_date": "04/14/2015 09:57:29",
      "content": "<p>Hirts -&nbsp;There is no 1 row in your matrix?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 71476,
      "author_name": "elvism",
      "author_url": "",
      "post_date": "04/14/2015 10:38:56",
      "content": "<p>[quote=Alexander Izvorski;71470]</p>\n<p>Hirts -&nbsp;There is no 1 row in your matrix?</p>\n<p>[/quote]</p>\n\n<p>Hi Alexander</p>\n\n<p>Yes, We didnt classify any of the images as 1, thats the problem :-)</p>\n<p>the 1 row would be :&nbsp;</p>\n<p>&nbsp; &nbsp; 0 &nbsp;1 &nbsp;2 &nbsp;3 &nbsp;4&nbsp;</p>\n<p>1 &nbsp;0 &nbsp;0 &nbsp;0 &nbsp;0 &nbsp;0&nbsp;</p>\n\n<p>but there are 482 images with the class 1 in the testset</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 71478,
      "author_name": "aizvorski",
      "author_url": "",
      "post_date": "04/14/2015 10:46:29",
      "content": "<p>I see, your confusion matrix is transposed relative to what I'm used to. &nbsp;Normally rows=actual columns=predicted.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 82426,
      "author_name": "ilyakava",
      "author_url": "",
      "post_date": "06/19/2015 17:51:20",
      "content": "<p>Yes I'm seeing that too. My confusion matrix for a ~0.5 kappa:</p>\n<p><code>T\\P | 0 &nbsp; &nbsp; &nbsp;| 1 &nbsp; | 2 &nbsp; &nbsp;| 3 &nbsp;| 4 &nbsp;|</code><code><br>0 &nbsp; | 3243 &nbsp; | 14 &nbsp;| 249 &nbsp;| 25 | 45 |<br> 1 &nbsp; | 285 &nbsp; &nbsp;| 5 &nbsp; | 35 &nbsp; | 8 &nbsp;| 5 &nbsp;|<br> 2 &nbsp; | 393 &nbsp; &nbsp;| 4 &nbsp; | 251 &nbsp;| 62 | 22 |<br> 3 &nbsp; | 31 &nbsp; &nbsp; | 0 &nbsp; | 40 &nbsp; | 39 | 10 |<br> 4 &nbsp; | 12 &nbsp; &nbsp; | 0 &nbsp; | 25 &nbsp; | 20 | 41 |<br>&nbsp;</code></p>\n\n<p><code>&nbsp;</code></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 82876,
      "author_name": "ilyakava",
      "author_url": "",
      "post_date": "06/28/2015 22:12:50",
      "content": "<p>I can say that following the conversation on this <a href=\"https://www.kaggle.com/c/diabetic-retinopathy-detection/forums/t/13115/paper-on-using-ann-for-ordinal-problems\">forum page</a>&nbsp;about ordinal regression led me a little away from this problem. &nbsp;</p>\n<p>Here is a confusion matrix where the only thing I changed between the run that generated the confusion matrix<a href=\"https://www.kaggle.com/c/diabetic-retinopathy-detection/forums/t/13354/cant-classify-the-images-of-the-class-1/82426#post82426\">&nbsp;I posted above</a>&nbsp;and the run that generated this one&nbsp;is&nbsp;the error function. Here, its taken directly from the paper mentioned in the forum&nbsp;(which is a little different than what the posters talk about). Specifically I used relative&nbsp;entropy as in the paper (instead of <a href=\"http://deeplearning.net/software/theano/library/tensor/nnet/nnet.html#tensor.nnet.categorical_crossentropy\">categorical cross entropy</a>&nbsp;in the forum) with 5&nbsp;different target vectors, each of length k=5, just like the paper specifies (which is different than what's mentioned in the forum). This gets ~0.53 kappa.</p>\n<p><code>Confusion Matrix<br>T\\P| 0 &nbsp; &nbsp;| 1 &nbsp;| 2 &nbsp; | 3 &nbsp;| 4 &nbsp;|<br> 0 &nbsp;| 3314 | 65 | 168 | 12 | 17 |<br> 1 &nbsp;| 293 &nbsp;| 18 | 23 &nbsp;| 1 &nbsp;| 3 &nbsp;|<br> 2 &nbsp;| 437 &nbsp;| 28 | 197 | 44 | 26 |<br> 3 &nbsp;| 22 &nbsp; | 5 &nbsp;| 34 &nbsp;| 33 | 26 |<br> 4 &nbsp;| 16 &nbsp; | 5 &nbsp;| 22 &nbsp;| 13 | 42 |</code></p>\n\n<p>EDIT:</p>\n<p>---</p>\n<p>After more experiments I can say that this problem goes away for me when I opt from a softmax+categorical crossentropy to sigmoid+relative entropy final layer. The ordinal regression helps further on top of that.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "71456": "",
    "71457": "",
    "71470": "",
    "71476": "",
    "71478": "",
    "82426": "",
    "82876": ""
  },
  "source": "meta"
}