{
  "id": 251762,
  "title": "Image Level + Study Level CV Scores Notebook (mAP-0.5)",
  "url": "/competitions/siim-covid19-detection/discussion/251762",
  "author_name": "",
  "post_date": "2021-07-08T17:22:26.955175600Z",
  "votes": 14,
  "comment_count": 12,
  "views": 0,
  "content": "<p>Anyone looking for a notebook to calculate the competition metric for image-level and study level, can use the notebook below</p>\n<p><a href=\"https://www.kaggle.com/varundutt9213/covid-competition-metric-image-level-map\" target=\"_blank\">https://www.kaggle.com/varundutt9213/covid-competition-metric-image-level-map</a></p>\n<p>Note : To get the overall score you need to manually enter the CV scores for the study level task</p>",
  "messages": [
    {
      "id": "1381170",
      "postDate": "07/08/2021 17:22:26",
      "content": "<p>Anyone looking for a notebook to calculate the competition metric for image-level and study level, can use the notebook below</p>\n<p><a href=\"https://www.kaggle.com/varundutt9213/covid-competition-metric-image-level-map\" target=\"_blank\">https://www.kaggle.com/varundutt9213/covid-competition-metric-image-level-map</a></p>\n<p>Note : To get the overall score you need to manually enter the CV scores for the study level task</p>",
      "rawMarkdown": "Anyone looking for a notebook to calculate the competition metric for image-level and study level, can use the notebook below\n\nhttps://www.kaggle.com/varundutt9213/covid-competition-metric-image-level-map\n\nNote : To get the overall score you need to manually enter the CV scores for the study level task",
      "votes": null
    },
    {
      "id": "1382365",
      "postDate": "07/09/2021 18:23:24",
      "content": "<p>Thank you for sharing, upvoted.👍</p>",
      "rawMarkdown": "Thank you for sharing, upvoted.👍",
      "votes": null
    },
    {
      "id": "1382369",
      "postDate": "07/09/2021 18:29:13",
      "content": "<p>Thank you for sharing! +1<br>\nCan you also let us know what is the corresponding LB Study ONLY score of the <code>study scores</code> attached ? In case you have tested</p>\n<pre><code>study_scores=[\n0.39388390121767813,\n0.39234564782357945,\n0.3847761244542205,\n0.37818329809109086,\n0.38300730435642993]\n</code></pre>",
      "rawMarkdown": "Thank you for sharing! +1\nCan you also let us know what is the corresponding LB Study ONLY score of the `study scores` attached ? In case you have tested\n\n```\nstudy_scores=[\n0.39388390121767813,\n0.39234564782357945,\n0.3847761244542205,\n0.37818329809109086,\n0.38300730435642993]\n```",
      "votes": null
    },
    {
      "id": "1382375",
      "postDate": "07/09/2021 18:42:43",
      "content": "<p>LB score mean of all folds is 45.6 with image level labels = 'none 1 0 0 1 1'</p>",
      "rawMarkdown": "LB score mean of all folds is 45.6 with image level labels = 'none 1 0 0 1 1'",
      "votes": null
    },
    {
      "id": "1382402",
      "postDate": "07/09/2021 19:40:05",
      "content": "<p>Thank You 😄</p>",
      "rawMarkdown": "Thank You 😄",
      "votes": null
    },
    {
      "id": "1388243",
      "postDate": "07/14/2021 18:38:22",
      "content": "<p>How do you calculate the study scores?</p>",
      "rawMarkdown": "How do you calculate the study scores?",
      "votes": null
    },
    {
      "id": "1388498",
      "postDate": "07/15/2021 02:40:25",
      "content": "<p>Just used average_precision_score from sklearn.</p>",
      "rawMarkdown": "Just used average_precision_score from sklearn.",
      "votes": null
    },
    {
      "id": "1388731",
      "postDate": "07/15/2021 07:18:29",
      "content": "<p>When I use average_precision_score my validation scores are around 0.5 is there anything I have to configure?</p>",
      "rawMarkdown": "When I use average_precision_score my validation scores are around 0.5 is there anything I have to configure?",
      "votes": null
    },
    {
      "id": "1399710",
      "postDate": "07/25/2021 15:41:57",
      "content": "<p><a href=\"https://www.kaggle.com/varundutt9213\" target=\"_blank\">@varundutt9213</a> thanks for sharing! The idea looks interesting.<br>\nOne question - is there any reason why you are using the metric average_precision_score to evaluate your predictions on the study level?<br>\nAs I understand, this function is not monotonically decreasing, as opposed to mAP VOC 2010</p>",
      "rawMarkdown": "varundutt9213 thanks for sharing! The idea looks interesting.\nOne question - is there any reason why you are using the metric average_precision_score to evaluate your predictions on the study level?\nAs I understand, this function is not monotonically decreasing, as opposed to mAP VOC 2010",
      "votes": null
    },
    {
      "id": "1399778",
      "postDate": "07/25/2021 17:06:54",
      "content": "<p>Yes if remember sklearn does not interpolate the PR curve so I believe average_precision_score might be underestimating the CV scores but to be honest I was too lazy to calculate mAP for study level using pycoco xd! Also, because sklearn should underestimate the score it can be used and in practice seems stable for this competition.</p>",
      "rawMarkdown": "Yes if remember sklearn does not interpolate the PR curve so I believe average_precision_score might be underestimating the CV scores but to be honest I was too lazy to calculate mAP for study level using pycoco xd! Also, because sklearn should underestimate the score it can be used and in practice seems stable for this competition.",
      "votes": null
    },
    {
      "id": "1399987",
      "postDate": "07/25/2021 22:12:02",
      "content": "<p>Actually indeed after running some tests I can see that the deviation from VOC is small enough to use it as an approximation. Thanks!</p>",
      "rawMarkdown": "Actually indeed after running some tests I can see that the deviation from VOC is small enough to use it as an approximation. Thanks!",
      "votes": null
    },
    {
      "id": "1400000",
      "postDate": "07/25/2021 22:29:55",
      "content": "<p>Did you find the sklearn cv scores underestimating or overestimating the voc 2010 scores?</p>",
      "rawMarkdown": "Did you find the sklearn cv scores underestimating or overestimating the voc 2010 scores?",
      "votes": null
    },
    {
      "id": "1400371",
      "postDate": "07/26/2021 08:32:04",
      "content": "<p>As expected, in my results sklearn underestimates VOC. The difference is around 1-1.5% (approximately 1 p.p.)</p>",
      "rawMarkdown": "As expected, in my results sklearn underestimates VOC. The difference is around 1-1.5% (approximately 1 p.p.)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1382365,
      "author_name": "minhtien1405",
      "author_url": "",
      "post_date": "07/09/2021 18:23:24",
      "content": "<p>Thank you for sharing, upvoted.👍</p>",
      "votes": null,
      "replies": [
        {
          "id": 1382402,
          "author_name": "varundutt9213",
          "author_url": "",
          "post_date": "07/09/2021 19:40:05",
          "content": "<p>Thank You 😄</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1382369,
      "author_name": "imeintanis",
      "author_url": "",
      "post_date": "07/09/2021 18:29:13",
      "content": "<p>Thank you for sharing! +1<br>\nCan you also let us know what is the corresponding LB Study ONLY score of the <code>study scores</code> attached ? In case you have tested</p>\n<pre><code>study_scores=[\n0.39388390121767813,\n0.39234564782357945,\n0.3847761244542205,\n0.37818329809109086,\n0.38300730435642993]\n</code></pre>",
      "votes": null,
      "replies": [
        {
          "id": 1382375,
          "author_name": "varundutt9213",
          "author_url": "",
          "post_date": "07/09/2021 18:42:43",
          "content": "<p>LB score mean of all folds is 45.6 with image level labels = 'none 1 0 0 1 1'</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1388243,
      "author_name": "simon111",
      "author_url": "",
      "post_date": "07/14/2021 18:38:22",
      "content": "<p>How do you calculate the study scores?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1388498,
          "author_name": "varundutt9213",
          "author_url": "",
          "post_date": "07/15/2021 02:40:25",
          "content": "<p>Just used average_precision_score from sklearn.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1388731,
          "author_name": "simon111",
          "author_url": "",
          "post_date": "07/15/2021 07:18:29",
          "content": "<p>When I use average_precision_score my validation scores are around 0.5 is there anything I have to configure?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1399710,
      "author_name": "mikecho",
      "author_url": "",
      "post_date": "07/25/2021 15:41:57",
      "content": "<p><a href=\"https://www.kaggle.com/varundutt9213\" target=\"_blank\">@varundutt9213</a> thanks for sharing! The idea looks interesting.<br>\nOne question - is there any reason why you are using the metric average_precision_score to evaluate your predictions on the study level?<br>\nAs I understand, this function is not monotonically decreasing, as opposed to mAP VOC 2010</p>",
      "votes": null,
      "replies": [
        {
          "id": 1399778,
          "author_name": "varundutt9213",
          "author_url": "",
          "post_date": "07/25/2021 17:06:54",
          "content": "<p>Yes if remember sklearn does not interpolate the PR curve so I believe average_precision_score might be underestimating the CV scores but to be honest I was too lazy to calculate mAP for study level using pycoco xd! Also, because sklearn should underestimate the score it can be used and in practice seems stable for this competition.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1399987,
          "author_name": "mikecho",
          "author_url": "",
          "post_date": "07/25/2021 22:12:02",
          "content": "<p>Actually indeed after running some tests I can see that the deviation from VOC is small enough to use it as an approximation. Thanks!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1400000,
          "author_name": "varundutt9213",
          "author_url": "",
          "post_date": "07/25/2021 22:29:55",
          "content": "<p>Did you find the sklearn cv scores underestimating or overestimating the voc 2010 scores?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1400371,
          "author_name": "mikecho",
          "author_url": "",
          "post_date": "07/26/2021 08:32:04",
          "content": "<p>As expected, in my results sklearn underestimates VOC. The difference is around 1-1.5% (approximately 1 p.p.)</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1381170": "Anyone looking for a notebook to calculate the competition metric for image-level and study level, can use the notebook below\n\nhttps://www.kaggle.com/varundutt9213/covid-competition-metric-image-level-map\n\nNote : To get the overall score you need to manually enter the CV scores for the study level task",
    "1382365": "Thank you for sharing, upvoted.👍",
    "1382369": "Thank you for sharing! +1\nCan you also let us know what is the corresponding LB Study ONLY score of the `study scores` attached ? In case you have tested\n\n```\nstudy_scores=[\n0.39388390121767813,\n0.39234564782357945,\n0.3847761244542205,\n0.37818329809109086,\n0.38300730435642993]\n```",
    "1382375": "LB score mean of all folds is 45.6 with image level labels = 'none 1 0 0 1 1'",
    "1382402": "Thank You 😄",
    "1388243": "How do you calculate the study scores?",
    "1388498": "Just used average_precision_score from sklearn.",
    "1388731": "When I use average_precision_score my validation scores are around 0.5 is there anything I have to configure?",
    "1399710": "varundutt9213 thanks for sharing! The idea looks interesting.\nOne question - is there any reason why you are using the metric average_precision_score to evaluate your predictions on the study level?\nAs I understand, this function is not monotonically decreasing, as opposed to mAP VOC 2010",
    "1399778": "Yes if remember sklearn does not interpolate the PR curve so I believe average_precision_score might be underestimating the CV scores but to be honest I was too lazy to calculate mAP for study level using pycoco xd! Also, because sklearn should underestimate the score it can be used and in practice seems stable for this competition.",
    "1399987": "Actually indeed after running some tests I can see that the deviation from VOC is small enough to use it as an approximation. Thanks!",
    "1400000": "Did you find the sklearn cv scores underestimating or overestimating the voc 2010 scores?",
    "1400371": "As expected, in my results sklearn underestimates VOC. The difference is around 1-1.5% (approximately 1 p.p.)"
  },
  "source": "meta"
}