{
  "id": 125431,
  "title": "Negative confidence values ?",
  "url": "/competitions/pku-autonomous-driving/discussion/125431",
  "author_name": "",
  "post_date": "2020-01-10T14:58:06.186000100Z",
  "votes": 1,
  "comment_count": 5,
  "views": 0,
  "content": "<p>From the evaluation page, there is a sentence:\n<code>Confidence scores are used to sort submission objects.</code></p>\n\n<p>Seems like confidence just used for sorting the predictions.\nBut I am wondering that, are confidence values in the final submission need to be larger than zero?\nSince current LB score seems don't relate to the confidence values, so I am not able to confirm that. And some of the people are using the logits &gt; -0.5 to be the threshold. How do you process the confidence values after you filter the logits of prediction with negative threshold? Do you just fill in the confidence values with some negative values in it?\nOr I need to perform the sigmoid on the logits I filtered before I write them into confidence values?</p>\n\n<p>Again..since this is my first time to deal with detection base competition/project, there are some rules that I'm unfamiliar with. Sorry about the stupid question, thanks in advances.</p>",
  "messages": [
    {
      "id": "715454",
      "postDate": "01/10/2020 14:58:06",
      "content": "<p>From the evaluation page, there is a sentence:\n<code>Confidence scores are used to sort submission objects.</code></p>\n\n<p>Seems like confidence just used for sorting the predictions.\nBut I am wondering that, are confidence values in the final submission need to be larger than zero?\nSince current LB score seems don't relate to the confidence values, so I am not able to confirm that. And some of the people are using the logits &gt; -0.5 to be the threshold. How do you process the confidence values after you filter the logits of prediction with negative threshold? Do you just fill in the confidence values with some negative values in it?\nOr I need to perform the sigmoid on the logits I filtered before I write them into confidence values?</p>\n\n<p>Again..since this is my first time to deal with detection base competition/project, there are some rules that I'm unfamiliar with. Sorry about the stupid question, thanks in advances.</p>",
      "rawMarkdown": "From the evaluation page, there is a sentence:\n`Confidence scores are used to sort submission objects.`\n\nSeems like confidence just used for sorting the predictions.\nBut I am wondering that, are confidence values in the final submission need to be larger than zero?\nSince current LB score seems don't relate to the confidence values, so I am not able to confirm that. And some of the people are using the logits &gt; -0.5 to be the threshold. How do you process the confidence values after you filter the logits of prediction with negative threshold? Do you just fill in the confidence values with some negative values in it?\nOr I need to perform the sigmoid on the logits I filtered before I write them into confidence values?\n\nAgain..since this is my first time to deal with detection base competition/project, there are some rules that I'm unfamiliar with. Sorry about the stupid question, thanks in advances.",
      "votes": null
    },
    {
      "id": "715467",
      "postDate": "01/10/2020 15:08:29",
      "content": "<p>I used raw prediction before, but today I switched to sigmoid output. I didn't notice any difference caused by sigmoid function, but this is only my personal observations.</p>",
      "rawMarkdown": "I used raw prediction before, but today I switched to sigmoid output. I didn't notice any difference caused by sigmoid function, but this is only my personal observations.",
      "votes": null
    },
    {
      "id": "715879",
      "postDate": "01/10/2020 23:14:31",
      "content": "<p>Seems like current LB score calculating is not related to confidence values, so even you replace the confidence values with random number, the score will still be the same. So I don't know the difference between fill in confidence values with raw prediction or prediction with sigmoid function. Just wondering that is it okay to fill in the negative confidence values in final submission?</p>",
      "rawMarkdown": "Seems like current LB score calculating is not related to confidence values, so even you replace the confidence values with random number, the score will still be the same. So I don't know the difference between fill in confidence values with raw prediction or prediction with sigmoid function. Just wondering that is it okay to fill in the negative confidence values in final submission?",
      "votes": null
    },
    {
      "id": "715895",
      "postDate": "01/10/2020 23:49:16",
      "content": "<p>My guess is, if it doesn't fail on public LB, it won't fail on private either. They are using the same script for the entire set.</p>",
      "rawMarkdown": "My guess is, if it doesn't fail on public LB, it won't fail on private either. They are using the same script for the entire set.",
      "votes": null
    },
    {
      "id": "715915",
      "postDate": "01/11/2020 01:01:55",
      "content": "<p>Thanks, that's what I thought. But in this way, the confidence values seems has no point in this competition. I think maybe I will still perform the sigmoid on the filtered logits to be the confidence values in final submissions, just in case.</p>",
      "rawMarkdown": "Thanks, that's what I thought. But in this way, the confidence values seems has no point in this competition. I think maybe I will still perform the sigmoid on the filtered logits to be the confidence values in final submissions, just in case.",
      "votes": null
    },
    {
      "id": "716544",
      "postDate": "01/11/2020 20:58:50",
      "content": "<p>yes, there seem to be a bug in their scoring, so confidence is most likely not taken into account. See the other discussions regarding the metric.</p>",
      "rawMarkdown": "yes, there seem to be a bug in their scoring, so confidence is most likely not taken into account. See the other discussions regarding the metric.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 715467,
      "author_name": "richardwong1994",
      "author_url": "",
      "post_date": "01/10/2020 15:08:29",
      "content": "<p>I used raw prediction before, but today I switched to sigmoid output. I didn't notice any difference caused by sigmoid function, but this is only my personal observations.</p>",
      "votes": null,
      "replies": [
        {
          "id": 715879,
          "author_name": "xiejialun",
          "author_url": "",
          "post_date": "01/10/2020 23:14:31",
          "content": "<p>Seems like current LB score calculating is not related to confidence values, so even you replace the confidence values with random number, the score will still be the same. So I don't know the difference between fill in confidence values with raw prediction or prediction with sigmoid function. Just wondering that is it okay to fill in the negative confidence values in final submission?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 715895,
          "author_name": "ilu000",
          "author_url": "",
          "post_date": "01/10/2020 23:49:16",
          "content": "<p>My guess is, if it doesn't fail on public LB, it won't fail on private either. They are using the same script for the entire set.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 715915,
          "author_name": "xiejialun",
          "author_url": "",
          "post_date": "01/11/2020 01:01:55",
          "content": "<p>Thanks, that's what I thought. But in this way, the confidence values seems has no point in this competition. I think maybe I will still perform the sigmoid on the filtered logits to be the confidence values in final submissions, just in case.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 716544,
          "author_name": "ilu000",
          "author_url": "",
          "post_date": "01/11/2020 20:58:50",
          "content": "<p>yes, there seem to be a bug in their scoring, so confidence is most likely not taken into account. See the other discussions regarding the metric.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "715454": "From the evaluation page, there is a sentence:\n`Confidence scores are used to sort submission objects.`\n\nSeems like confidence just used for sorting the predictions.\nBut I am wondering that, are confidence values in the final submission need to be larger than zero?\nSince current LB score seems don't relate to the confidence values, so I am not able to confirm that. And some of the people are using the logits &gt; -0.5 to be the threshold. How do you process the confidence values after you filter the logits of prediction with negative threshold? Do you just fill in the confidence values with some negative values in it?\nOr I need to perform the sigmoid on the logits I filtered before I write them into confidence values?\n\nAgain..since this is my first time to deal with detection base competition/project, there are some rules that I'm unfamiliar with. Sorry about the stupid question, thanks in advances.",
    "715467": "I used raw prediction before, but today I switched to sigmoid output. I didn't notice any difference caused by sigmoid function, but this is only my personal observations.",
    "715879": "Seems like current LB score calculating is not related to confidence values, so even you replace the confidence values with random number, the score will still be the same. So I don't know the difference between fill in confidence values with raw prediction or prediction with sigmoid function. Just wondering that is it okay to fill in the negative confidence values in final submission?",
    "715895": "My guess is, if it doesn't fail on public LB, it won't fail on private either. They are using the same script for the entire set.",
    "715915": "Thanks, that's what I thought. But in this way, the confidence values seems has no point in this competition. I think maybe I will still perform the sigmoid on the filtered logits to be the confidence values in final submissions, just in case.",
    "716544": "yes, there seem to be a bug in their scoring, so confidence is most likely not taken into account. See the other discussions regarding the metric."
  },
  "source": "meta"
}