{
  "id": 241300,
  "title": "CV / LB (Study Level)",
  "url": "/competitions/siim-covid19-detection/discussion/241300",
  "author_name": "",
  "post_date": "2021-05-24T01:39:02.514758700Z",
  "votes": 37,
  "comment_count": 22,
  "views": 0,
  "content": "<p>EfficientNetB0 single fold for experiment (multilabel stratified group kfold, NFOLDS=5)<br>\nImage Level for submission: all \"none 1 0 0 1 1\"</p>\n<p>---CV / LB---<br>\n0.166 / 0.176 (all labels 1 0 0 1 1)<br>\n0.176 / 0.180 (only Indeterminate predicted)<br>\n0.190 / 0.204 (only Atypical predicted)<br>\n0.221 / 0.237 (only Typical predicted)<br>\n0.252 / 0.252 (only Negative predicted)<br>\n0.327 / 0.332<br>\n0.336 / 0.350<br>\n0.339 / 0.345<br>\n0.345 / 0.352<br>\n0.346 / 0.341<br>\n0.359 / 0.346<br>\n0.364 / 0.363<br>\n0.365 / 0.352</p>\n<p>CV is calculated as; study-level mAP * 2/3<br>\nWhat is your splitting method and scores?<br>\nI guess stratified group kfold is another choice.</p>",
  "messages": [
    {
      "id": "1320301",
      "postDate": "05/24/2021 01:39:02",
      "content": "<p>EfficientNetB0 single fold for experiment (multilabel stratified group kfold, NFOLDS=5)<br>\nImage Level for submission: all \"none 1 0 0 1 1\"</p>\n<p>---CV / LB---<br>\n0.166 / 0.176 (all labels 1 0 0 1 1)<br>\n0.176 / 0.180 (only Indeterminate predicted)<br>\n0.190 / 0.204 (only Atypical predicted)<br>\n0.221 / 0.237 (only Typical predicted)<br>\n0.252 / 0.252 (only Negative predicted)<br>\n0.327 / 0.332<br>\n0.336 / 0.350<br>\n0.339 / 0.345<br>\n0.345 / 0.352<br>\n0.346 / 0.341<br>\n0.359 / 0.346<br>\n0.364 / 0.363<br>\n0.365 / 0.352</p>\n<p>CV is calculated as; study-level mAP * 2/3<br>\nWhat is your splitting method and scores?<br>\nI guess stratified group kfold is another choice.</p>",
      "rawMarkdown": "EfficientNetB0 single fold for experiment (multilabel stratified group kfold, NFOLDS=5)\nImage Level for submission: all \"none 1 0 0 1 1\"\n\n---CV / LB---\n0.166 / 0.176 (all labels 1 0 0 1 1)\n0.176 / 0.180 (only Indeterminate predicted)\n0.190 / 0.204 (only Atypical predicted)\n0.221 / 0.237 (only Typical predicted)\n0.252 / 0.252 (only Negative predicted)\n0.327 / 0.332\n0.336 / 0.350\n0.339 / 0.345\n0.345 / 0.352\n0.346 / 0.341\n0.359 / 0.346\n0.364 / 0.363\n0.365 / 0.352\n\nCV is calculated as; study-level mAP * 2/3\nWhat is your splitting method and scores?\nI guess stratified group kfold is another choice.",
      "votes": null
    },
    {
      "id": "1320309",
      "postDate": "05/24/2021 01:54:55",
      "content": "<p>New to kaggle and quick question- Is there any way to get your LB score without physically submitting to the leaderboard? Also, others have said they get .001 for image_level on LB, so is there a way to know the breakdown of your LB score from each of study and image level? </p>",
      "rawMarkdown": "New to kaggle and quick question- Is there any way to get your LB score without physically submitting to the leaderboard? Also, others have said they get .001 for image_level on LB, so is there a way to know the breakdown of your LB score from each of study and image level?",
      "votes": null
    },
    {
      "id": "1320330",
      "postDate": "05/24/2021 02:28:42",
      "content": "<p>What do you mean by physical submission?</p>",
      "rawMarkdown": "What do you mean by physical submission?",
      "votes": null
    },
    {
      "id": "1320363",
      "postDate": "05/24/2021 03:02:15",
      "content": "<blockquote>\n  <p>Is there any way to get your LB score without physically submitting to the leaderboard?</p>\n</blockquote>\n<p>If 'physically submitting' means pressing the submit button that appears on the kaggle page, The answer is no. However, it can be estimated from the validation score.</p>\n<blockquote>\n  <p>is there a way to know the breakdown of your LB score from each of study and image level?</p>\n</blockquote>\n<p>If you send one of them with a score of 0, you will know the score of the other one.</p>",
      "rawMarkdown": "> Is there any way to get your LB score without physically submitting to the leaderboard?\n\nIf 'physically submitting' means pressing the submit button that appears on the kaggle page, The answer is no. However, it can be estimated from the validation score.\n> is there a way to know the breakdown of your LB score from each of study and image level?\n\nIf you send one of them with a score of 0, you will know the score of the other one.",
      "votes": null
    },
    {
      "id": "1320407",
      "postDate": "05/24/2021 04:27:29",
      "content": "<p>Gotcha, yep by physical submission I just meant pressing the submit button. Thanks so much! </p>",
      "rawMarkdown": "Gotcha, yep by physical submission I just meant pressing the submit button. Thanks so much!",
      "votes": null
    },
    {
      "id": "1322505",
      "postDate": "05/25/2021 13:31:48",
      "content": "<p>I would suggest Stratified Shuffle Split it will shuffle as well as stratify it </p>",
      "rawMarkdown": "I would suggest Stratified Shuffle Split it will shuffle as well as stratify it",
      "votes": null
    },
    {
      "id": "1323102",
      "postDate": "05/26/2021 02:10:26",
      "content": "<p>There is no way to get LB score without submitting and there is a limit of submissions you can make in one day on Kaggle. I remember there were many fake accounts on Kaggle created only to submit additional results and check the LB, you can see it often on LB (high score, unknown account) and very often people are banned because that.</p>",
      "rawMarkdown": "There is no way to get LB score without submitting and there is a limit of submissions you can make in one day on Kaggle. I remember there were many fake accounts on Kaggle created only to submit additional results and check the LB, you can see it often on LB (high score, unknown account) and very often people are banned because that.",
      "votes": null
    },
    {
      "id": "1323169",
      "postDate": "05/26/2021 04:00:46",
      "content": "<p>Updated. Bad correlation…😂</p>",
      "rawMarkdown": "Updated. Bad correlation...😂",
      "votes": null
    },
    {
      "id": "1323180",
      "postDate": "05/26/2021 04:20:23",
      "content": "<p>The expected number of atypical studies in a public test that can be calculated from the train data is less than 100, so relying on LB may be a bit risky.</p>",
      "rawMarkdown": "The expected number of atypical studies in a public test that can be calculated from the train data is less than 100, so relying on LB may be a bit risky.",
      "votes": null
    },
    {
      "id": "1323183",
      "postDate": "05/26/2021 04:22:38",
      "content": "<p>Yeah, besides there are only 4 labels.</p>",
      "rawMarkdown": "Yeah, besides there are only 4 labels.",
      "votes": null
    },
    {
      "id": "1332603",
      "postDate": "06/02/2021 07:55:49",
      "content": "<p>can you share your mAP function you are using?</p>",
      "rawMarkdown": "can you share your mAP function you are using?",
      "votes": null
    },
    {
      "id": "1332608",
      "postDate": "06/02/2021 07:58:54",
      "content": "<p>Hi. I just used \"average_precision_score\" from sklearn for the classification CV.</p>",
      "rawMarkdown": "Hi. I just used \"average_precision_score\" from sklearn for the classification CV.",
      "votes": null
    },
    {
      "id": "1332647",
      "postDate": "06/02/2021 08:23:06",
      "content": "<p>okay, so you are just giving classification mAP scores?</p>",
      "rawMarkdown": "okay, so you are just giving classification mAP scores?",
      "votes": null
    },
    {
      "id": "1334223",
      "postDate": "06/03/2021 10:43:20",
      "content": "<p>Hello,did u use detection network?</p>",
      "rawMarkdown": "Hello,did u use detection network?",
      "votes": null
    },
    {
      "id": "1335124",
      "postDate": "06/04/2021 03:03:10",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/drtausamaru\" target=\"_blank\">@drtausamaru</a>, why CV is 2/3 * study-level mAP?</p>",
      "rawMarkdown": "Hi @drtausamaru, why CV is 2/3 * study-level mAP?",
      "votes": null
    },
    {
      "id": "1335129",
      "postDate": "06/04/2021 03:07:01",
      "content": "<p>Because study level has 4 labels and image level has 2 labels, there are 6 labels in total.<br>\nAm I wrong? 😨</p>",
      "rawMarkdown": "Because study level has 4 labels and image level has 2 labels, there are 6 labels in total.\nAm I wrong? 😨",
      "votes": null
    },
    {
      "id": "1335215",
      "postDate": "06/04/2021 05:00:14",
      "content": "<p>stratified kfold<br>\nefficientnet-b3, 5fold<br>\ncv 0.398, lb 0.394</p>",
      "rawMarkdown": "stratified kfold\nefficientnet-b3, 5fold\ncv 0.398, lb 0.394",
      "votes": null
    },
    {
      "id": "1335229",
      "postDate": "06/04/2021 05:25:04",
      "content": "<p>Thanks for the info!!</p>",
      "rawMarkdown": "Thanks for the info!!",
      "votes": null
    },
    {
      "id": "1335252",
      "postDate": "06/04/2021 05:40:48",
      "content": "<p>No, you're right. I completely overlooked that. Thanks for sharing. =)</p>",
      "rawMarkdown": "No, you're right. I completely overlooked that. Thanks for sharing. =)",
      "votes": null
    },
    {
      "id": "1335513",
      "postDate": "06/04/2021 09:09:44",
      "content": "<p>which image size did you use?</p>",
      "rawMarkdown": "which image size did you use?",
      "votes": null
    },
    {
      "id": "1335848",
      "postDate": "06/04/2021 13:26:22",
      "content": "<p>512x512<br>\n1024x1024 didn't improve CV for me</p>",
      "rawMarkdown": "512x512\n1024x1024 didn't improve CV for me",
      "votes": null
    },
    {
      "id": "1338479",
      "postDate": "06/06/2021 13:40:41",
      "content": "<p><a href=\"https://www.kaggle.com/phalanx\" target=\"_blank\">@phalanx</a> how you're calculating the <strong>mAP</strong> for <strong>study level</strong> ? </p>",
      "rawMarkdown": "phalanx how you're calculating the **mAP** for **study level** ?",
      "votes": null
    },
    {
      "id": "1392510",
      "postDate": "07/18/2021 18:57:37",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/drtausamaru\" target=\"_blank\">@drtausamaru</a> :</p>\n<p>Any suggestions for <br>\n<a href=\"https://www.kaggle.com/c/siim-covid19-detection/discussion/253812\" target=\"_blank\">https://www.kaggle.com/c/siim-covid19-detection/discussion/253812</a></p>\n<p>Thanks!</p>",
      "rawMarkdown": "Hi @drtausamaru :\n\nAny suggestions for \nhttps://www.kaggle.com/c/siim-covid19-detection/discussion/253812\n\nThanks!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1320309,
      "author_name": "jamespopp",
      "author_url": "",
      "post_date": "05/24/2021 01:54:55",
      "content": "<p>New to kaggle and quick question- Is there any way to get your LB score without physically submitting to the leaderboard? Also, others have said they get .001 for image_level on LB, so is there a way to know the breakdown of your LB score from each of study and image level? </p>",
      "votes": null,
      "replies": [
        {
          "id": 1320330,
          "author_name": "drtausamaru",
          "author_url": "",
          "post_date": "05/24/2021 02:28:42",
          "content": "<p>What do you mean by physical submission?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1320363,
          "author_name": "yujiariyasu",
          "author_url": "",
          "post_date": "05/24/2021 03:02:15",
          "content": "<blockquote>\n  <p>Is there any way to get your LB score without physically submitting to the leaderboard?</p>\n</blockquote>\n<p>If 'physically submitting' means pressing the submit button that appears on the kaggle page, The answer is no. However, it can be estimated from the validation score.</p>\n<blockquote>\n  <p>is there a way to know the breakdown of your LB score from each of study and image level?</p>\n</blockquote>\n<p>If you send one of them with a score of 0, you will know the score of the other one.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1320407,
          "author_name": "jamespopp",
          "author_url": "",
          "post_date": "05/24/2021 04:27:29",
          "content": "<p>Gotcha, yep by physical submission I just meant pressing the submit button. Thanks so much! </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1323102,
          "author_name": "jacekpoplawski",
          "author_url": "",
          "post_date": "05/26/2021 02:10:26",
          "content": "<p>There is no way to get LB score without submitting and there is a limit of submissions you can make in one day on Kaggle. I remember there were many fake accounts on Kaggle created only to submit additional results and check the LB, you can see it often on LB (high score, unknown account) and very often people are banned because that.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1322505,
      "author_name": "hemanthhari",
      "author_url": "",
      "post_date": "05/25/2021 13:31:48",
      "content": "<p>I would suggest Stratified Shuffle Split it will shuffle as well as stratify it </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1323169,
      "author_name": "drtausamaru",
      "author_url": "",
      "post_date": "05/26/2021 04:00:46",
      "content": "<p>Updated. Bad correlation…😂</p>",
      "votes": null,
      "replies": [
        {
          "id": 1323180,
          "author_name": "yujiariyasu",
          "author_url": "",
          "post_date": "05/26/2021 04:20:23",
          "content": "<p>The expected number of atypical studies in a public test that can be calculated from the train data is less than 100, so relying on LB may be a bit risky.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1323183,
          "author_name": "drtausamaru",
          "author_url": "",
          "post_date": "05/26/2021 04:22:38",
          "content": "<p>Yeah, besides there are only 4 labels.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1332603,
      "author_name": "varundutt9213",
      "author_url": "",
      "post_date": "06/02/2021 07:55:49",
      "content": "<p>can you share your mAP function you are using?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1332608,
          "author_name": "drtausamaru",
          "author_url": "",
          "post_date": "06/02/2021 07:58:54",
          "content": "<p>Hi. I just used \"average_precision_score\" from sklearn for the classification CV.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1332647,
          "author_name": "varundutt9213",
          "author_url": "",
          "post_date": "06/02/2021 08:23:06",
          "content": "<p>okay, so you are just giving classification mAP scores?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1334223,
      "author_name": "zekunn",
      "author_url": "",
      "post_date": "06/03/2021 10:43:20",
      "content": "<p>Hello,did u use detection network?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1335124,
      "author_name": "yousof9",
      "author_url": "",
      "post_date": "06/04/2021 03:03:10",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/drtausamaru\" target=\"_blank\">@drtausamaru</a>, why CV is 2/3 * study-level mAP?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1335129,
          "author_name": "drtausamaru",
          "author_url": "",
          "post_date": "06/04/2021 03:07:01",
          "content": "<p>Because study level has 4 labels and image level has 2 labels, there are 6 labels in total.<br>\nAm I wrong? 😨</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1335252,
          "author_name": "yousof9",
          "author_url": "",
          "post_date": "06/04/2021 05:40:48",
          "content": "<p>No, you're right. I completely overlooked that. Thanks for sharing. =)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1335215,
      "author_name": "phalanx",
      "author_url": "",
      "post_date": "06/04/2021 05:00:14",
      "content": "<p>stratified kfold<br>\nefficientnet-b3, 5fold<br>\ncv 0.398, lb 0.394</p>",
      "votes": null,
      "replies": [
        {
          "id": 1335229,
          "author_name": "drtausamaru",
          "author_url": "",
          "post_date": "06/04/2021 05:25:04",
          "content": "<p>Thanks for the info!!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1335513,
          "author_name": "nguyenbadung",
          "author_url": "",
          "post_date": "06/04/2021 09:09:44",
          "content": "<p>which image size did you use?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1335848,
          "author_name": "phalanx",
          "author_url": "",
          "post_date": "06/04/2021 13:26:22",
          "content": "<p>512x512<br>\n1024x1024 didn't improve CV for me</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1338479,
          "author_name": "awsaf49",
          "author_url": "",
          "post_date": "06/06/2021 13:40:41",
          "content": "<p><a href=\"https://www.kaggle.com/phalanx\" target=\"_blank\">@phalanx</a> how you're calculating the <strong>mAP</strong> for <strong>study level</strong> ? </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1392510,
      "author_name": "rbhambri",
      "author_url": "",
      "post_date": "07/18/2021 18:57:37",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/drtausamaru\" target=\"_blank\">@drtausamaru</a> :</p>\n<p>Any suggestions for <br>\n<a href=\"https://www.kaggle.com/c/siim-covid19-detection/discussion/253812\" target=\"_blank\">https://www.kaggle.com/c/siim-covid19-detection/discussion/253812</a></p>\n<p>Thanks!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1320301": "EfficientNetB0 single fold for experiment (multilabel stratified group kfold, NFOLDS=5)\nImage Level for submission: all \"none 1 0 0 1 1\"\n\n---CV / LB---\n0.166 / 0.176 (all labels 1 0 0 1 1)\n0.176 / 0.180 (only Indeterminate predicted)\n0.190 / 0.204 (only Atypical predicted)\n0.221 / 0.237 (only Typical predicted)\n0.252 / 0.252 (only Negative predicted)\n0.327 / 0.332\n0.336 / 0.350\n0.339 / 0.345\n0.345 / 0.352\n0.346 / 0.341\n0.359 / 0.346\n0.364 / 0.363\n0.365 / 0.352\n\nCV is calculated as; study-level mAP * 2/3\nWhat is your splitting method and scores?\nI guess stratified group kfold is another choice.",
    "1320309": "New to kaggle and quick question- Is there any way to get your LB score without physically submitting to the leaderboard? Also, others have said they get .001 for image_level on LB, so is there a way to know the breakdown of your LB score from each of study and image level?",
    "1320330": "What do you mean by physical submission?",
    "1320363": "> Is there any way to get your LB score without physically submitting to the leaderboard?\n\nIf 'physically submitting' means pressing the submit button that appears on the kaggle page, The answer is no. However, it can be estimated from the validation score.\n> is there a way to know the breakdown of your LB score from each of study and image level?\n\nIf you send one of them with a score of 0, you will know the score of the other one.",
    "1320407": "Gotcha, yep by physical submission I just meant pressing the submit button. Thanks so much!",
    "1322505": "I would suggest Stratified Shuffle Split it will shuffle as well as stratify it",
    "1323102": "There is no way to get LB score without submitting and there is a limit of submissions you can make in one day on Kaggle. I remember there were many fake accounts on Kaggle created only to submit additional results and check the LB, you can see it often on LB (high score, unknown account) and very often people are banned because that.",
    "1323169": "Updated. Bad correlation...😂",
    "1323180": "The expected number of atypical studies in a public test that can be calculated from the train data is less than 100, so relying on LB may be a bit risky.",
    "1323183": "Yeah, besides there are only 4 labels.",
    "1332603": "can you share your mAP function you are using?",
    "1332608": "Hi. I just used \"average_precision_score\" from sklearn for the classification CV.",
    "1332647": "okay, so you are just giving classification mAP scores?",
    "1334223": "Hello,did u use detection network?",
    "1335124": "Hi @drtausamaru, why CV is 2/3 * study-level mAP?",
    "1335129": "Because study level has 4 labels and image level has 2 labels, there are 6 labels in total.\nAm I wrong? 😨",
    "1335215": "stratified kfold\nefficientnet-b3, 5fold\ncv 0.398, lb 0.394",
    "1335229": "Thanks for the info!!",
    "1335252": "No, you're right. I completely overlooked that. Thanks for sharing. =)",
    "1335513": "which image size did you use?",
    "1335848": "512x512\n1024x1024 didn't improve CV for me",
    "1338479": "phalanx how you're calculating the **mAP** for **study level** ?",
    "1392510": "Hi @drtausamaru :\n\nAny suggestions for \nhttps://www.kaggle.com/c/siim-covid19-detection/discussion/253812\n\nThanks!"
  },
  "source": "meta"
}