{
  "id": 117578,
  "title": "local validation score",
  "url": "/competitions/pku-autonomous-driving/discussion/117578",
  "author_name": "",
  "post_date": "2019-11-16T12:29:57.709203600Z",
  "votes": 17,
  "comment_count": 17,
  "views": 0,
  "content": "<p>Now that LB isn't working properly yet, I think it's a good idea for us to share local validation score in the meantime. </p>\n\n<p>My best MAP score is <code>0.277</code>.</p>\n\n<p>This is calculated for 20% random split hold out images using <a href=\"https://www.kaggle.com/its7171/metrics-evaluation-script\">this script</a>.</p>",
  "messages": [
    {
      "id": "674398",
      "postDate": "11/16/2019 12:29:57",
      "content": "<p>Now that LB isn't working properly yet, I think it's a good idea for us to share local validation score in the meantime. </p>\n\n<p>My best MAP score is <code>0.277</code>.</p>\n\n<p>This is calculated for 20% random split hold out images using <a href=\"https://www.kaggle.com/its7171/metrics-evaluation-script\">this script</a>.</p>",
      "rawMarkdown": "Now that LB isn't working properly yet, I think it's a good idea for us to share local validation score in the meantime. \n\nMy best MAP score is `0.277`.\n\nThis is calculated for 20% random split hold out images using [this script](https://www.kaggle.com/its7171/metrics-evaluation-script).",
      "votes": null
    },
    {
      "id": "674466",
      "postDate": "11/16/2019 14:43:12",
      "content": "<p>Thanks for the kernal! What probability threshold did you use to calc your score?</p>",
      "rawMarkdown": "Thanks for the kernal! What probability threshold did you use to calc your score?",
      "votes": null
    },
    {
      "id": "674479",
      "postDate": "11/16/2019 14:55:06",
      "content": "<p>Hi GreatGameDota,</p>\n\n<p>I used 0.1 for probability threshold.</p>",
      "rawMarkdown": "Hi GreatGameDota,\n\nI used 0.1 for probability threshold.",
      "votes": null
    },
    {
      "id": "674483",
      "postDate": "11/16/2019 14:56:54",
      "content": "<p>Thanks a lot for the kernel!</p>\n\n<p>According to the script my mAP is 0.096 for a randomly picked model (10 % holdout) which seems about right. \nI might go ahead and check all models and thresholds against your script and publish the results later.</p>\n\n<p>Edit: I checked a few more models (threshold was always at 50 %), and they are all in the range 0.070 to 0.105, so no large differences. But I was able to spot some errors in my code with the script. So thanks again!</p>",
      "rawMarkdown": "Thanks a lot for the kernel!\n\nAccording to the script my mAP is 0.096 for a randomly picked model (10 % holdout) which seems about right. \nI might go ahead and check all models and thresholds against your script and publish the results later.\n\nEdit: I checked a few more models (threshold was always at 50 %), and they are all in the range 0.070 to 0.105, so no large differences. But I was able to spot some errors in my code with the script. So thanks again!",
      "votes": null
    },
    {
      "id": "674516",
      "postDate": "11/16/2019 16:05:02",
      "content": "<p>Just ran the script with my current model. I got a score of .096 with a prob threshold of &gt;88% on 20% of the training data (800 random images). Score of .105 with threshold of &gt;50% (same images).</p>",
      "rawMarkdown": "Just ran the script with my current model. I got a score of .096 with a prob threshold of &gt;88% on 20% of the training data (800 random images). Score of .105 with threshold of &gt;50% (same images).",
      "votes": null
    },
    {
      "id": "675388",
      "postDate": "11/18/2019 02:46:09",
      "content": "<p>Thanks for your sharing.   I used your local validation script  and got the  0.0632  local score.  LB is 0.063 .  </p>",
      "rawMarkdown": "Thanks for your sharing.   I used your local validation script  and got the  0.0632  local score.  LB is 0.063 .",
      "votes": null
    },
    {
      "id": "677427",
      "postDate": "11/20/2019 06:15:36",
      "content": "<p>Did some tweaks and got the score to mAP: 0.122. Your 0.277 is impressive, tito!</p>\n\n<p>Another update: \nlocal mAP: 0.134 -&gt; LB 0.137\nlocal mAP: 0.153 -&gt; LB 0.155</p>",
      "rawMarkdown": "Did some tweaks and got the score to mAP: 0.122. Your 0.277 is impressive, tito!\n\nAnother update: \nlocal mAP: 0.134 -&gt; LB 0.137\nlocal mAP: 0.153 -&gt; LB 0.155",
      "votes": null
    },
    {
      "id": "684620",
      "postDate": "11/30/2019 04:50:32",
      "content": "<p><a href=\"/its7171\">@its7171</a> Hi, tito, Thanks for the evaluation script, your 0.277 is impressive, my CV can only reach ~0.127. May I ask if you use centernet？</p>",
      "rawMarkdown": "its7171 Hi, tito, Thanks for the evaluation script, your 0.277 is impressive, my CV can only reach ~0.127. May I ask if you use centernet？",
      "votes": null
    },
    {
      "id": "686897",
      "postDate": "12/03/2019 17:00:36",
      "content": "<p>So .277 converts to .097 on public LB, .153 -&gt; .078, and .09 -&gt; .04? Seems your metric is still different than the official one</p>",
      "rawMarkdown": "So .277 converts to .097 on public LB, .153 -&gt; .078, and .09 -&gt; .04? Seems your metric is still different than the official one",
      "votes": null
    },
    {
      "id": "689201",
      "postDate": "12/06/2019 16:16:44",
      "content": "<p>Hi Jhui,</p>\n\n<p>Yes, it's centernet.</p>",
      "rawMarkdown": "Hi Jhui,\n\nYes, it's centernet.",
      "votes": null
    },
    {
      "id": "689203",
      "postDate": "12/06/2019 16:18:03",
      "content": "<p>I think the biggest difference between my script and LB would be threshold behavior.\nThe lower threshold (the more predictoin) gives the higher mAP for my scripts, but LB dose not.\nWe need more detailed information about metrics...</p>",
      "rawMarkdown": "I think the biggest difference between my script and LB would be threshold behavior.\nThe lower threshold (the more predictoin) gives the higher mAP for my scripts, but LB dose not.\nWe need more detailed information about metrics...",
      "votes": null
    },
    {
      "id": "689571",
      "postDate": "12/07/2019 04:31:17",
      "content": "<p>OK，Thank you for sharing！</p>",
      "rawMarkdown": "OK，Thank you for sharing！",
      "votes": null
    },
    {
      "id": "696535",
      "postDate": "12/16/2019 18:23:25",
      "content": "<p><a href=\"/qinhui1999\">@qinhui1999</a> Hi! Are you intrested in teaming up? I have a 2080Ti and completed competitions previously with decent scores. I also have a pretty good LB score now. I think we can build good solutions together. Thanks!</p>",
      "rawMarkdown": "qinhui1999 Hi! Are you intrested in teaming up? I have a 2080Ti and completed competitions previously with decent scores. I also have a pretty good LB score now. I think we can build good solutions together. Thanks!",
      "votes": null
    },
    {
      "id": "699105",
      "postDate": "12/20/2019 03:50:16",
      "content": "<p>val score: 0.346, lb: 0.094</p>",
      "rawMarkdown": "val score: 0.346, lb: 0.094",
      "votes": null
    },
    {
      "id": "699121",
      "postDate": "12/20/2019 04:10:12",
      "content": "<p><a href=\"/phalanx\">@phalanx</a> amazing score！！！May I ask if you are using centernet?</p>",
      "rawMarkdown": "phalanx amazing score！！！May I ask if you are using centernet?",
      "votes": null
    },
    {
      "id": "699122",
      "postDate": "12/20/2019 04:12:23",
      "content": "<p>No.\nit didn't work for me.</p>",
      "rawMarkdown": "No.\nit didn't work for me.",
      "votes": null
    },
    {
      "id": "699123",
      "postDate": "12/20/2019 04:13:06",
      "content": "<p>Ok, Thanks</p>",
      "rawMarkdown": "Ok, Thanks",
      "votes": null
    },
    {
      "id": "699184",
      "postDate": "12/20/2019 06:10:42",
      "content": "<p>I still don't get, how you guys are achieving such high scores in the validation script. \nI am reaching 0.180 tops and have LB score of 0.099 with that. </p>",
      "rawMarkdown": "I still don't get, how you guys are achieving such high scores in the validation script. \nI am reaching 0.180 tops and have LB score of 0.099 with that.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 674466,
      "author_name": "greatgamedota",
      "author_url": "",
      "post_date": "11/16/2019 14:43:12",
      "content": "<p>Thanks for the kernal! What probability threshold did you use to calc your score?</p>",
      "votes": null,
      "replies": [
        {
          "id": 674479,
          "author_name": "its7171",
          "author_url": "",
          "post_date": "11/16/2019 14:55:06",
          "content": "<p>Hi GreatGameDota,</p>\n\n<p>I used 0.1 for probability threshold.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 674483,
      "author_name": "ilu000",
      "author_url": "",
      "post_date": "11/16/2019 14:56:54",
      "content": "<p>Thanks a lot for the kernel!</p>\n\n<p>According to the script my mAP is 0.096 for a randomly picked model (10 % holdout) which seems about right. \nI might go ahead and check all models and thresholds against your script and publish the results later.</p>\n\n<p>Edit: I checked a few more models (threshold was always at 50 %), and they are all in the range 0.070 to 0.105, so no large differences. But I was able to spot some errors in my code with the script. So thanks again!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 674516,
      "author_name": "greatgamedota",
      "author_url": "",
      "post_date": "11/16/2019 16:05:02",
      "content": "<p>Just ran the script with my current model. I got a score of .096 with a prob threshold of &gt;88% on 20% of the training data (800 random images). Score of .105 with threshold of &gt;50% (same images).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 675388,
      "author_name": "qinhui1999",
      "author_url": "",
      "post_date": "11/18/2019 02:46:09",
      "content": "<p>Thanks for your sharing.   I used your local validation script  and got the  0.0632  local score.  LB is 0.063 .  </p>",
      "votes": null,
      "replies": [
        {
          "id": 696535,
          "author_name": "tonychenxyz",
          "author_url": "",
          "post_date": "12/16/2019 18:23:25",
          "content": "<p><a href=\"/qinhui1999\">@qinhui1999</a> Hi! Are you intrested in teaming up? I have a 2080Ti and completed competitions previously with decent scores. I also have a pretty good LB score now. I think we can build good solutions together. Thanks!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 677427,
      "author_name": "ilu000",
      "author_url": "",
      "post_date": "11/20/2019 06:15:36",
      "content": "<p>Did some tweaks and got the score to mAP: 0.122. Your 0.277 is impressive, tito!</p>\n\n<p>Another update: \nlocal mAP: 0.134 -&gt; LB 0.137\nlocal mAP: 0.153 -&gt; LB 0.155</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 684620,
      "author_name": "hesene",
      "author_url": "",
      "post_date": "11/30/2019 04:50:32",
      "content": "<p><a href=\"/its7171\">@its7171</a> Hi, tito, Thanks for the evaluation script, your 0.277 is impressive, my CV can only reach ~0.127. May I ask if you use centernet？</p>",
      "votes": null,
      "replies": [
        {
          "id": 689201,
          "author_name": "its7171",
          "author_url": "",
          "post_date": "12/06/2019 16:16:44",
          "content": "<p>Hi Jhui,</p>\n\n<p>Yes, it's centernet.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 689571,
          "author_name": "hesene",
          "author_url": "",
          "post_date": "12/07/2019 04:31:17",
          "content": "<p>OK，Thank you for sharing！</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 686897,
      "author_name": "greatgamedota",
      "author_url": "",
      "post_date": "12/03/2019 17:00:36",
      "content": "<p>So .277 converts to .097 on public LB, .153 -&gt; .078, and .09 -&gt; .04? Seems your metric is still different than the official one</p>",
      "votes": null,
      "replies": [
        {
          "id": 689203,
          "author_name": "its7171",
          "author_url": "",
          "post_date": "12/06/2019 16:18:03",
          "content": "<p>I think the biggest difference between my script and LB would be threshold behavior.\nThe lower threshold (the more predictoin) gives the higher mAP for my scripts, but LB dose not.\nWe need more detailed information about metrics...</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 699105,
      "author_name": "phalanx",
      "author_url": "",
      "post_date": "12/20/2019 03:50:16",
      "content": "<p>val score: 0.346, lb: 0.094</p>",
      "votes": null,
      "replies": [
        {
          "id": 699121,
          "author_name": "hesene",
          "author_url": "",
          "post_date": "12/20/2019 04:10:12",
          "content": "<p><a href=\"/phalanx\">@phalanx</a> amazing score！！！May I ask if you are using centernet?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 699122,
          "author_name": "phalanx",
          "author_url": "",
          "post_date": "12/20/2019 04:12:23",
          "content": "<p>No.\nit didn't work for me.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 699123,
          "author_name": "hesene",
          "author_url": "",
          "post_date": "12/20/2019 04:13:06",
          "content": "<p>Ok, Thanks</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 699184,
          "author_name": "ilu000",
          "author_url": "",
          "post_date": "12/20/2019 06:10:42",
          "content": "<p>I still don't get, how you guys are achieving such high scores in the validation script. \nI am reaching 0.180 tops and have LB score of 0.099 with that. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "674398": "Now that LB isn't working properly yet, I think it's a good idea for us to share local validation score in the meantime. \n\nMy best MAP score is `0.277`.\n\nThis is calculated for 20% random split hold out images using [this script](https://www.kaggle.com/its7171/metrics-evaluation-script).",
    "674466": "Thanks for the kernal! What probability threshold did you use to calc your score?",
    "674479": "Hi GreatGameDota,\n\nI used 0.1 for probability threshold.",
    "674483": "Thanks a lot for the kernel!\n\nAccording to the script my mAP is 0.096 for a randomly picked model (10 % holdout) which seems about right. \nI might go ahead and check all models and thresholds against your script and publish the results later.\n\nEdit: I checked a few more models (threshold was always at 50 %), and they are all in the range 0.070 to 0.105, so no large differences. But I was able to spot some errors in my code with the script. So thanks again!",
    "674516": "Just ran the script with my current model. I got a score of .096 with a prob threshold of &gt;88% on 20% of the training data (800 random images). Score of .105 with threshold of &gt;50% (same images).",
    "675388": "Thanks for your sharing.   I used your local validation script  and got the  0.0632  local score.  LB is 0.063 .",
    "677427": "Did some tweaks and got the score to mAP: 0.122. Your 0.277 is impressive, tito!\n\nAnother update: \nlocal mAP: 0.134 -&gt; LB 0.137\nlocal mAP: 0.153 -&gt; LB 0.155",
    "684620": "its7171 Hi, tito, Thanks for the evaluation script, your 0.277 is impressive, my CV can only reach ~0.127. May I ask if you use centernet？",
    "686897": "So .277 converts to .097 on public LB, .153 -&gt; .078, and .09 -&gt; .04? Seems your metric is still different than the official one",
    "689201": "Hi Jhui,\n\nYes, it's centernet.",
    "689203": "I think the biggest difference between my script and LB would be threshold behavior.\nThe lower threshold (the more predictoin) gives the higher mAP for my scripts, but LB dose not.\nWe need more detailed information about metrics...",
    "689571": "OK，Thank you for sharing！",
    "696535": "qinhui1999 Hi! Are you intrested in teaming up? I have a 2080Ti and completed competitions previously with decent scores. I also have a pretty good LB score now. I think we can build good solutions together. Thanks!",
    "699105": "val score: 0.346, lb: 0.094",
    "699121": "phalanx amazing score！！！May I ask if you are using centernet?",
    "699122": "No.\nit didn't work for me.",
    "699123": "Ok, Thanks",
    "699184": "I still don't get, how you guys are achieving such high scores in the validation script. \nI am reaching 0.180 tops and have LB score of 0.099 with that."
  },
  "source": "meta"
}