{
  "id": 246536,
  "title": "dark magic",
  "url": "/competitions/siim-covid19-detection/discussion/246536",
  "author_name": "hengck23",
  "post_date": "2021-06-15T18:40:58.564000",
  "votes": 16,
  "comment_count": 6,
  "views": 0,
  "content": "<p>this is due to the flaw in the evaluation metrics and process:</p>\n<pre><code>- remove all 'none 1 0 0 1 1' in xxx_image\n- add to ALL image prediction : 'none score 0 0 1 1' where score is the one you have in the corresponding yyy_study\n\ne.g.\n\nid,PredictionString\n557a70442928_image,none 0.874495 0 0 1 1 \n36141cda67ad_image,none 0.824778 0 0 1 1 \n2413a23a5477_image,none 0.176755 0 0 1 1 opacity 0.469323 443 653 1095 1585 opacity 0.496004 1660 770 2400 1694\nc263b1e9aa64_image,none 0.256808 0 0 1 1 opacity 0.300391 438 1148 1050 1860\n4fe0444d7fc5_image,none 0.433021 0 0 1 1 opacity 0.282367 2037 953 2801 1579\n1ff307ae0df6_image,none 0.674902 0 0 1 1 \nb5e287db5781_image,none 0.855466 0 0 1 1 \n07c552c36da3_image,none 0.887609 0 0 1 1 \nc26e3ae08056_image,none 0.929014 0 0 1 1 \n</code></pre>\n<p>the assumption is that you have very accurate study level prediction.</p>\n<p>there should also be some smart way to post-process and optimize scores between study level  \"typical\", \"indeterminate\", \"atypical\" and image level \"opacity\". This is left as an exercise for you.</p>\n<hr>\n<p>the trick in mAP is to that some prediction is better than no prediction<br>\ne.g. \"none 0.9999 0 0 1 1      opacity 0.0001 x1 y1 x2 y2\" may be better than just \"none 0.9999 0 0 1 1\". </p>\n<p>you have to determine some prior values \"0.0001\",  \"x1 y1 x2 y2\" (e.g. mean box or a set of mean boxes) that work best for you. this can improve (or worsen?) your leaderboard score by some (small or big?) amount.</p>",
  "messages": [
    {
      "id": 1350780,
      "postDate": "2021-06-15T18:40:58.563Z",
      "content": "<p>this is due to the flaw in the evaluation metrics and process:</p>\n<pre><code>- remove all 'none 1 0 0 1 1' in xxx_image\n- add to ALL image prediction : 'none score 0 0 1 1' where score is the one you have in the corresponding yyy_study\n\ne.g.\n\nid,PredictionString\n557a70442928_image,none 0.874495 0 0 1 1 \n36141cda67ad_image,none 0.824778 0 0 1 1 \n2413a23a5477_image,none 0.176755 0 0 1 1 opacity 0.469323 443 653 1095 1585 opacity 0.496004 1660 770 2400 1694\nc263b1e9aa64_image,none 0.256808 0 0 1 1 opacity 0.300391 438 1148 1050 1860\n4fe0444d7fc5_image,none 0.433021 0 0 1 1 opacity 0.282367 2037 953 2801 1579\n1ff307ae0df6_image,none 0.674902 0 0 1 1 \nb5e287db5781_image,none 0.855466 0 0 1 1 \n07c552c36da3_image,none 0.887609 0 0 1 1 \nc26e3ae08056_image,none 0.929014 0 0 1 1 \n</code></pre>\n<p>the assumption is that you have very accurate study level prediction.</p>\n<p>there should also be some smart way to post-process and optimize scores between study level  \"typical\", \"indeterminate\", \"atypical\" and image level \"opacity\". This is left as an exercise for you.</p>\n<hr>\n<p>the trick in mAP is to that some prediction is better than no prediction<br>\ne.g. \"none 0.9999 0 0 1 1      opacity 0.0001 x1 y1 x2 y2\" may be better than just \"none 0.9999 0 0 1 1\". </p>\n<p>you have to determine some prior values \"0.0001\",  \"x1 y1 x2 y2\" (e.g. mean box or a set of mean boxes) that work best for you. this can improve (or worsen?) your leaderboard score by some (small or big?) amount.</p>",
      "rawMarkdown": "this is due to the flaw in the evaluation metrics and process:\n\n```\n- remove all 'none 1 0 0 1 1' in xxx_image\n- add to ALL image prediction : 'none score 0 0 1 1' where score is the one you have in the corresponding yyy_study\n\ne.g.\n\nid,PredictionString\n557a70442928_image,none 0.874495 0 0 1 1 \n36141cda67ad_image,none 0.824778 0 0 1 1 \n2413a23a5477_image,none 0.176755 0 0 1 1 opacity 0.469323 443 653 1095 1585 opacity 0.496004 1660 770 2400 1694\nc263b1e9aa64_image,none 0.256808 0 0 1 1 opacity 0.300391 438 1148 1050 1860\n4fe0444d7fc5_image,none 0.433021 0 0 1 1 opacity 0.282367 2037 953 2801 1579\n1ff307ae0df6_image,none 0.674902 0 0 1 1 \nb5e287db5781_image,none 0.855466 0 0 1 1 \n07c552c36da3_image,none 0.887609 0 0 1 1 \nc26e3ae08056_image,none 0.929014 0 0 1 1 \n\n```\n\nthe assumption is that you have very accurate study level prediction.\n\nthere should also be some smart way to post-process and optimize scores between study level  \"typical\", \"indeterminate\", \"atypical\" and image level \"opacity\". This is left as an exercise for you.\n\n---\n\nthe trick in mAP is to that some prediction is better than no prediction\ne.g. \"none 0.9999 0 0 1 1      opacity 0.0001 x1 y1 x2 y2\" may be better than just \"none 0.9999 0 0 1 1\". \n \nyou have to determine some prior values \"0.0001\",  \"x1 y1 x2 y2\" (e.g. mean box or a set of mean boxes) that work best for you. this can improve (or worsen?) your leaderboard score by some (small or big?) amount.\n",
      "votes": 16
    },
    {
      "id": 1350785,
      "postDate": "2021-06-15T18:50:21.980Z",
      "content": "<p>What does this do?</p>",
      "rawMarkdown": "What does this do?",
      "votes": 1,
      "replies": [
        {
          "id": 1350790,
          "postDate": "2021-06-15T18:54:38.353Z",
          "content": "<p>'none score 0 0 1 1' is likely to score better than 'none 1 0 0 1 1'.</p>\n<p>It also depends on your strategy. Some people train separate image classification and object detection models. </p>\n<p>Others might treat it as a single end-to-end model with joint classification and detection joint task </p>",
          "rawMarkdown": "'none score 0 0 1 1' is likely to score better than 'none 1 0 0 1 1'.\n\nIt also depends on your strategy. Some people train separate image classification and object detection models. \n\nOthers might treat it as a single end-to-end model with joint classification and detection joint task ",
          "votes": 3
        },
        {
          "id": 1350795,
          "postDate": "2021-06-15T19:01:24.500Z",
          "content": "<p>Okay, thanks!</p>",
          "rawMarkdown": "Okay, thanks!"
        },
        {
          "id": 1356342,
          "postDate": "2021-06-19T00:16:43.967Z",
          "content": "<p><a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> why would 'none score 0 0 1 1' be better than 'none 1 0 0 1 1' if there is only one none prediction per prediction string?</p>",
          "rawMarkdown": "@hengck23 why would 'none score 0 0 1 1' be better than 'none 1 0 0 1 1' if there is only one none prediction per prediction string?"
        }
      ]
    },
    {
      "id": 1355006,
      "postDate": "2021-06-18T03:50:02.900Z",
      "content": "<p>What does it mean <code>where score is the one you have in the corresponding yyy_study</code> ? In study level does not have \"none\" class isn't it</p>",
      "rawMarkdown": "What does it mean `where score is the one you have in the corresponding yyy_study` ? In study level does not have \"none\" class isn't it\n"
    },
    {
      "id": 1351660,
      "postDate": "2021-06-16T14:00:48.450Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1350785,
      "author_name": "Joseph AMIGO",
      "author_url": "",
      "post_date": "2021-06-15T18:50:21.980000",
      "content": "<p>What does this do?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1350790,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2021-06-15T18:54:38.353000",
          "content": "<p>'none score 0 0 1 1' is likely to score better than 'none 1 0 0 1 1'.</p>\n<p>It also depends on your strategy. Some people train separate image classification and object detection models. </p>\n<p>Others might treat it as a single end-to-end model with joint classification and detection joint task </p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 1350795,
          "author_name": "Joseph AMIGO",
          "author_url": "",
          "post_date": "2021-06-15T19:01:24.500000",
          "content": "<p>Okay, thanks!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1356342,
          "author_name": "Yousef Rabi",
          "author_url": "",
          "post_date": "2021-06-19T00:16:43.967000",
          "content": "<p><a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> why would 'none score 0 0 1 1' be better than 'none 1 0 0 1 1' if there is only one none prediction per prediction string?</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1355006,
      "author_name": "kaihong",
      "author_url": "",
      "post_date": "2021-06-18T03:50:02.900000",
      "content": "<p>What does it mean <code>where score is the one you have in the corresponding yyy_study</code> ? In study level does not have \"none\" class isn't it</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1351660,
      "author_name": "",
      "author_url": "",
      "post_date": "2021-06-16T14:00:48.450000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1350780": "this is due to the flaw in the evaluation metrics and process:\n\n```\n- remove all 'none 1 0 0 1 1' in xxx_image\n- add to ALL image prediction : 'none score 0 0 1 1' where score is the one you have in the corresponding yyy_study\n\ne.g.\n\nid,PredictionString\n557a70442928_image,none 0.874495 0 0 1 1 \n36141cda67ad_image,none 0.824778 0 0 1 1 \n2413a23a5477_image,none 0.176755 0 0 1 1 opacity 0.469323 443 653 1095 1585 opacity 0.496004 1660 770 2400 1694\nc263b1e9aa64_image,none 0.256808 0 0 1 1 opacity 0.300391 438 1148 1050 1860\n4fe0444d7fc5_image,none 0.433021 0 0 1 1 opacity 0.282367 2037 953 2801 1579\n1ff307ae0df6_image,none 0.674902 0 0 1 1 \nb5e287db5781_image,none 0.855466 0 0 1 1 \n07c552c36da3_image,none 0.887609 0 0 1 1 \nc26e3ae08056_image,none 0.929014 0 0 1 1 \n\n```\n\nthe assumption is that you have very accurate study level prediction.\n\nthere should also be some smart way to post-process and optimize scores between study level  \"typical\", \"indeterminate\", \"atypical\" and image level \"opacity\". This is left as an exercise for you.\n\n---\n\nthe trick in mAP is to that some prediction is better than no prediction\ne.g. \"none 0.9999 0 0 1 1      opacity 0.0001 x1 y1 x2 y2\" may be better than just \"none 0.9999 0 0 1 1\". \n \nyou have to determine some prior values \"0.0001\",  \"x1 y1 x2 y2\" (e.g. mean box or a set of mean boxes) that work best for you. this can improve (or worsen?) your leaderboard score by some (small or big?) amount.\n",
    "1350785": "What does this do?",
    "1355006": "What does it mean `where score is the one you have in the corresponding yyy_study` ? In study level does not have \"none\" class isn't it\n",
    "1351660": ""
  }
}