{
  "id": 126771,
  "title": "how to choose logit threshold?",
  "url": "/competitions/pku-autonomous-driving/discussion/126771",
  "author_name": "",
  "post_date": "2020-01-20T05:48:56.345241Z",
  "votes": 2,
  "comment_count": 9,
  "views": 0,
  "content": "<p>I found that the local cv MAP increases consistently as I decrease the logit threshold , I am pretty new in the competition, anyone knows how to select? \nhere is what I found:\n        sig(&gt;0.1)   cv: 0.114\n        sig(&gt;0.15) cv: 0.106\n        sig(&gt;0.17)  cv: 0.098\n        sig(&gt;0.2)   cv: 0.089\n        sig(&gt;0.3)   cv: 0.066</p>",
  "messages": [
    {
      "id": "723516",
      "postDate": "01/20/2020 05:48:56",
      "content": "<p>I found that the local cv MAP increases consistently as I decrease the logit threshold , I am pretty new in the competition, anyone knows how to select? \nhere is what I found:\n        sig(&gt;0.1)   cv: 0.114\n        sig(&gt;0.15) cv: 0.106\n        sig(&gt;0.17)  cv: 0.098\n        sig(&gt;0.2)   cv: 0.089\n        sig(&gt;0.3)   cv: 0.066</p>",
      "rawMarkdown": "I found that the local cv MAP increases consistently as I decrease the logit threshold , I am pretty new in the competition, anyone knows how to select? \nhere is what I found:\n        sig(&gt;0.1)   cv: 0.114\n        sig(&gt;0.15) cv: 0.106\n        sig(&gt;0.17)  cv: 0.098\n        sig(&gt;0.2)   cv: 0.089\n        sig(&gt;0.3)   cv: 0.066",
      "votes": null
    },
    {
      "id": "723577",
      "postDate": "01/20/2020 07:12:13",
      "content": "<p>That's exactly how mAP <em>should</em> work. Apparently, in this competition they are using something else, as LB score does not move on par. It has been assumed, that ordering of confidence is not happening here. </p>\n\n<p>So it would just be:\n<code>\nprecision = n_tp / n_preds\nrecall = n_tp / n_gt\nalt_ap = precision * recall\n</code></p>",
      "rawMarkdown": "That's exactly how mAP *should* work. Apparently, in this competition they are using something else, as LB score does not move on par. It has been assumed, that ordering of confidence is not happening here. \n\nSo it would just be:\n```\nprecision = n_tp / n_preds\nrecall = n_tp / n_gt\nalt_ap = precision * recall\n```",
      "votes": null
    },
    {
      "id": "723588",
      "postDate": "01/20/2020 07:39:14",
      "content": "<p>how much data you used for calculating cv? 20%</p>",
      "rawMarkdown": "how much data you used for calculating cv? 20%",
      "votes": null
    },
    {
      "id": "723593",
      "postDate": "01/20/2020 07:45:06",
      "content": "<p>Yep，20%</p>",
      "rawMarkdown": "Yep，20%",
      "votes": null
    },
    {
      "id": "723595",
      "postDate": "01/20/2020 07:51:00",
      "content": "<p>thank you, so do you mean I should modify the evaluation script from Map into alt ap to judge my threshold?</p>",
      "rawMarkdown": "thank you, so do you mean I should modify the evaluation script from Map into alt ap to judge my threshold?",
      "votes": null
    },
    {
      "id": "723613",
      "postDate": "01/20/2020 08:14:37",
      "content": "<p><a href=\"/gungnirspledge\">@gungnirspledge</a> \ntoday i got map: 0.09216910615476379 for logits &gt; -0.84 but lb 0.059 :(\ni used 20% validation data for calculating cv :(\ndon't know how to select best model for private lb :(</p>",
      "rawMarkdown": "gungnirspledge \ntoday i got map: 0.09216910615476379 for logits &gt; -0.84 but lb 0.059 :(\ni used 20% validation data for calculating cv :(\ndon't know how to select best model for private lb :(",
      "votes": null
    },
    {
      "id": "723688",
      "postDate": "01/20/2020 10:39:19",
      "content": "<p>choose logit which keep predict car numbers in test dataset &lt; 23000 (about 11 cars per image)</p>",
      "rawMarkdown": "choose logit which keep predict car numbers in test dataset &lt; 23000 (about 11 cars per image)",
      "votes": null
    },
    {
      "id": "723850",
      "postDate": "01/20/2020 14:26:47",
      "content": "<p>May I wonder how did you come up with this number?</p>",
      "rawMarkdown": "May I wonder how did you come up with this number?",
      "votes": null
    },
    {
      "id": "723872",
      "postDate": "01/20/2020 14:57:50",
      "content": "<p>because the mean car number in train dataset is 11.6 per image</p>",
      "rawMarkdown": "because the mean car number in train dataset is 11.6 per image",
      "votes": null
    },
    {
      "id": "724239",
      "postDate": "01/21/2020 00:29:37",
      "content": "<p>That's the only thing we can do because competition metric is hidden....</p>",
      "rawMarkdown": "That's the only thing we can do because competition metric is hidden....",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 723577,
      "author_name": "ilu000",
      "author_url": "",
      "post_date": "01/20/2020 07:12:13",
      "content": "<p>That's exactly how mAP <em>should</em> work. Apparently, in this competition they are using something else, as LB score does not move on par. It has been assumed, that ordering of confidence is not happening here. </p>\n\n<p>So it would just be:\n<code>\nprecision = n_tp / n_preds\nrecall = n_tp / n_gt\nalt_ap = precision * recall\n</code></p>",
      "votes": null,
      "replies": [
        {
          "id": 723595,
          "author_name": "gungnirspledge",
          "author_url": "",
          "post_date": "01/20/2020 07:51:00",
          "content": "<p>thank you, so do you mean I should modify the evaluation script from Map into alt ap to judge my threshold?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 724239,
          "author_name": "bamps53",
          "author_url": "",
          "post_date": "01/21/2020 00:29:37",
          "content": "<p>That's the only thing we can do because competition metric is hidden....</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 723588,
      "author_name": "mobassir",
      "author_url": "",
      "post_date": "01/20/2020 07:39:14",
      "content": "<p>how much data you used for calculating cv? 20%</p>",
      "votes": null,
      "replies": [
        {
          "id": 723593,
          "author_name": "gungnirspledge",
          "author_url": "",
          "post_date": "01/20/2020 07:45:06",
          "content": "<p>Yep，20%</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 723613,
          "author_name": "mobassir",
          "author_url": "",
          "post_date": "01/20/2020 08:14:37",
          "content": "<p><a href=\"/gungnirspledge\">@gungnirspledge</a> \ntoday i got map: 0.09216910615476379 for logits &gt; -0.84 but lb 0.059 :(\ni used 20% validation data for calculating cv :(\ndon't know how to select best model for private lb :(</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 723688,
      "author_name": "welkinfeng",
      "author_url": "",
      "post_date": "01/20/2020 10:39:19",
      "content": "<p>choose logit which keep predict car numbers in test dataset &lt; 23000 (about 11 cars per image)</p>",
      "votes": null,
      "replies": [
        {
          "id": 723850,
          "author_name": "cateek",
          "author_url": "",
          "post_date": "01/20/2020 14:26:47",
          "content": "<p>May I wonder how did you come up with this number?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 723872,
          "author_name": "welkinfeng",
          "author_url": "",
          "post_date": "01/20/2020 14:57:50",
          "content": "<p>because the mean car number in train dataset is 11.6 per image</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "723516": "I found that the local cv MAP increases consistently as I decrease the logit threshold , I am pretty new in the competition, anyone knows how to select? \nhere is what I found:\n        sig(&gt;0.1)   cv: 0.114\n        sig(&gt;0.15) cv: 0.106\n        sig(&gt;0.17)  cv: 0.098\n        sig(&gt;0.2)   cv: 0.089\n        sig(&gt;0.3)   cv: 0.066",
    "723577": "That's exactly how mAP *should* work. Apparently, in this competition they are using something else, as LB score does not move on par. It has been assumed, that ordering of confidence is not happening here. \n\nSo it would just be:\n```\nprecision = n_tp / n_preds\nrecall = n_tp / n_gt\nalt_ap = precision * recall\n```",
    "723588": "how much data you used for calculating cv? 20%",
    "723593": "Yep，20%",
    "723595": "thank you, so do you mean I should modify the evaluation script from Map into alt ap to judge my threshold?",
    "723613": "gungnirspledge \ntoday i got map: 0.09216910615476379 for logits &gt; -0.84 but lb 0.059 :(\ni used 20% validation data for calculating cv :(\ndon't know how to select best model for private lb :(",
    "723688": "choose logit which keep predict car numbers in test dataset &lt; 23000 (about 11 cars per image)",
    "723850": "May I wonder how did you come up with this number?",
    "723872": "because the mean car number in train dataset is 11.6 per image",
    "724239": "That's the only thing we can do because competition metric is hidden...."
  },
  "source": "meta"
}