{
  "id": 100680,
  "title": "The score is negative :(",
  "url": "/competitions/aptos2019-blindness-detection/discussion/100680",
  "author_name": "",
  "post_date": "2019-07-20T05:34:59.240357600Z",
  "votes": 1,
  "comment_count": 7,
  "views": 0,
  "content": "<p>The score of my submission is negative....</p>\n\n<p>The notent of submission is like below</p>\n\n<p>1   id_code diagnosis\n2   0005cfc8afb6    2\n3   003f0afdcd15    2\n4   006efc72b638    2\n5   00836aaacf06    2\n...</p>\n\n<p>Do I make any mistake?\nPlease help me!</p>",
  "messages": [
    {
      "id": "580394",
      "postDate": "07/20/2019 05:34:59",
      "content": "<p>The score of my submission is negative....</p>\n\n<p>The notent of submission is like below</p>\n\n<p>1   id_code diagnosis\n2   0005cfc8afb6    2\n3   003f0afdcd15    2\n4   006efc72b638    2\n5   00836aaacf06    2\n...</p>\n\n<p>Do I make any mistake?\nPlease help me!</p>",
      "rawMarkdown": "The score of my submission is negative....\n\nThe notent of submission is like below\n\n1\tid_code\tdiagnosis\n2\t0005cfc8afb6\t2\n3\t003f0afdcd15\t2\n4\t006efc72b638\t2\n5\t00836aaacf06\t2\n...\n\nDo I make any mistake?\nPlease help me!",
      "votes": null
    },
    {
      "id": "580397",
      "postDate": "07/20/2019 05:37:38",
      "content": "<p>A couple of questions.\n- Do you have CV score?\n- What framework are you using? Keras? Torch?</p>",
      "rawMarkdown": "A couple of questions.\n- Do you have CV score?\n- What framework are you using? Keras? Torch?",
      "votes": null
    },
    {
      "id": "580400",
      "postDate": "07/20/2019 05:43:33",
      "content": "<p>Thanks for reply!</p>\n\n<blockquote>\n  <blockquote>\n    <p>Do you have CV score?\n    No. I computed online.\n    In evaluation description, it notes \"Submissions are scored based on the quadratic weighted kappa, which measures the agreement between two ratings. This metric typically varies from 0 (random agreement between raters) to 1 (complete agreement between raters). \"\n    , but my score is negative like -0.20.\n    I have no idea why this happened.</p>\n    \n    <p>What framework are you using? Keras? Torch?\n    I used pytorch and simple CNN.</p>\n  </blockquote>\n</blockquote>",
      "rawMarkdown": "Thanks for reply!\n\n&gt;&gt; Do you have CV score?\nNo. I computed online.\nIn evaluation description, it notes \"Submissions are scored based on the quadratic weighted kappa, which measures the agreement between two ratings. This metric typically varies from 0 (random agreement between raters) to 1 (complete agreement between raters). \"\n, but my score is negative like -0.20.\nI have no idea why this happened.\n\n&gt;&gt;What framework are you using? Keras? Torch?\nI used pytorch and simple CNN.",
      "votes": null
    },
    {
      "id": "580402",
      "postDate": "07/20/2019 05:48:01",
      "content": "<p>You forgot to read the next line. It says:</p>\n\n<blockquote>\n  <p>In the event that there is less agreement between the raters than expected by chance, this metric may go below 0</p>\n</blockquote>",
      "rawMarkdown": "You forgot to read the next line. It says:\n&gt; In the event that there is less agreement between the raters than expected by chance, this metric may go below 0",
      "votes": null
    },
    {
      "id": "580495",
      "postDate": "07/20/2019 08:37:28",
      "content": "<p>You can have a negative score with Cohen's Kappa.\nThe classification accuracy of your predictions is actually lower than what you would expect by chance. </p>\n\n<p>The formula is : Kappa = [Prob(baseline) - acc(your_predictions)] / [1 - acc(your_predictions)] </p>\n\n<p>Check this video (13min+) and i'm sure you will fully understand this metric : <a href=\"https://www.youtube.com/watch?v=oq2ZRgJdYlY\">https://www.youtube.com/watch?v=oq2ZRgJdYlY</a></p>",
      "rawMarkdown": "You can have a negative score with Cohen's Kappa.\nThe classification accuracy of your predictions is actually lower than what you would expect by chance. \n\nThe formula is : Kappa = [Prob(baseline) - acc(your_predictions)] / [1 - acc(your_predictions)] \n\nCheck this video (13min+) and i'm sure you will fully understand this metric : https://www.youtube.com/watch?v=oq2ZRgJdYlY",
      "votes": null
    },
    {
      "id": "580658",
      "postDate": "07/20/2019 14:23:12",
      "content": "<p>can someone explain why in layman terms?</p>\n\n<p>I'm using keras with simple CNN.\nTrying to create a baseline here, I get 71% acc on validation.</p>",
      "rawMarkdown": "can someone explain why in layman terms?\n\nI'm using keras with simple CNN.\nTrying to create a baseline here, I get 71% acc on validation.",
      "votes": null
    },
    {
      "id": "580856",
      "postDate": "07/20/2019 22:52:51",
      "content": "<p>you're probably somehow overfitting? Are you training on validation?\nAlso a simple CNN will not be that great since there are only 2000 training images.\nTransfer learning(with vgg, resnet, efficientnet) works best.</p>",
      "rawMarkdown": "you're probably somehow overfitting? Are you training on validation?\nAlso a simple CNN will not be that great since there are only 2000 training images.\nTransfer learning(with vgg, resnet, efficientnet) works best.",
      "votes": null
    },
    {
      "id": "580879",
      "postDate": "07/21/2019 00:33:52",
      "content": "<p>Thanks, everyone!\nI understood the cause is my worse prediction.</p>",
      "rawMarkdown": "Thanks, everyone!\nI understood the cause is my worse prediction.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 580397,
      "author_name": "higepon",
      "author_url": "",
      "post_date": "07/20/2019 05:37:38",
      "content": "<p>A couple of questions.\n- Do you have CV score?\n- What framework are you using? Keras? Torch?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 580400,
      "author_name": "yoshito",
      "author_url": "",
      "post_date": "07/20/2019 05:43:33",
      "content": "<p>Thanks for reply!</p>\n\n<blockquote>\n  <blockquote>\n    <p>Do you have CV score?\n    No. I computed online.\n    In evaluation description, it notes \"Submissions are scored based on the quadratic weighted kappa, which measures the agreement between two ratings. This metric typically varies from 0 (random agreement between raters) to 1 (complete agreement between raters). \"\n    , but my score is negative like -0.20.\n    I have no idea why this happened.</p>\n    \n    <p>What framework are you using? Keras? Torch?\n    I used pytorch and simple CNN.</p>\n  </blockquote>\n</blockquote>",
      "votes": null,
      "replies": [
        {
          "id": 580402,
          "author_name": "rishabhiitbhu",
          "author_url": "",
          "post_date": "07/20/2019 05:48:01",
          "content": "<p>You forgot to read the next line. It says:</p>\n\n<blockquote>\n  <p>In the event that there is less agreement between the raters than expected by chance, this metric may go below 0</p>\n</blockquote>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 580495,
      "author_name": "seif95",
      "author_url": "",
      "post_date": "07/20/2019 08:37:28",
      "content": "<p>You can have a negative score with Cohen's Kappa.\nThe classification accuracy of your predictions is actually lower than what you would expect by chance. </p>\n\n<p>The formula is : Kappa = [Prob(baseline) - acc(your_predictions)] / [1 - acc(your_predictions)] </p>\n\n<p>Check this video (13min+) and i'm sure you will fully understand this metric : <a href=\"https://www.youtube.com/watch?v=oq2ZRgJdYlY\">https://www.youtube.com/watch?v=oq2ZRgJdYlY</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 580658,
      "author_name": "jagadeeshkotra",
      "author_url": "",
      "post_date": "07/20/2019 14:23:12",
      "content": "<p>can someone explain why in layman terms?</p>\n\n<p>I'm using keras with simple CNN.\nTrying to create a baseline here, I get 71% acc on validation.</p>",
      "votes": null,
      "replies": [
        {
          "id": 580856,
          "author_name": "sidhanthholalkere",
          "author_url": "",
          "post_date": "07/20/2019 22:52:51",
          "content": "<p>you're probably somehow overfitting? Are you training on validation?\nAlso a simple CNN will not be that great since there are only 2000 training images.\nTransfer learning(with vgg, resnet, efficientnet) works best.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 580879,
      "author_name": "yoshito",
      "author_url": "",
      "post_date": "07/21/2019 00:33:52",
      "content": "<p>Thanks, everyone!\nI understood the cause is my worse prediction.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "580394": "The score of my submission is negative....\n\nThe notent of submission is like below\n\n1\tid_code\tdiagnosis\n2\t0005cfc8afb6\t2\n3\t003f0afdcd15\t2\n4\t006efc72b638\t2\n5\t00836aaacf06\t2\n...\n\nDo I make any mistake?\nPlease help me!",
    "580397": "A couple of questions.\n- Do you have CV score?\n- What framework are you using? Keras? Torch?",
    "580400": "Thanks for reply!\n\n&gt;&gt; Do you have CV score?\nNo. I computed online.\nIn evaluation description, it notes \"Submissions are scored based on the quadratic weighted kappa, which measures the agreement between two ratings. This metric typically varies from 0 (random agreement between raters) to 1 (complete agreement between raters). \"\n, but my score is negative like -0.20.\nI have no idea why this happened.\n\n&gt;&gt;What framework are you using? Keras? Torch?\nI used pytorch and simple CNN.",
    "580402": "You forgot to read the next line. It says:\n&gt; In the event that there is less agreement between the raters than expected by chance, this metric may go below 0",
    "580495": "You can have a negative score with Cohen's Kappa.\nThe classification accuracy of your predictions is actually lower than what you would expect by chance. \n\nThe formula is : Kappa = [Prob(baseline) - acc(your_predictions)] / [1 - acc(your_predictions)] \n\nCheck this video (13min+) and i'm sure you will fully understand this metric : https://www.youtube.com/watch?v=oq2ZRgJdYlY",
    "580658": "can someone explain why in layman terms?\n\nI'm using keras with simple CNN.\nTrying to create a baseline here, I get 71% acc on validation.",
    "580856": "you're probably somehow overfitting? Are you training on validation?\nAlso a simple CNN will not be that great since there are only 2000 training images.\nTransfer learning(with vgg, resnet, efficientnet) works best.",
    "580879": "Thanks, everyone!\nI understood the cause is my worse prediction."
  },
  "source": "meta"
}