{
  "id": 263800,
  "title": "Calibrating confidence scores",
  "url": "/competitions/siim-covid19-detection/discussion/263800",
  "author_name": "",
  "post_date": "2021-08-10T09:11:16.635721600Z",
  "votes": 7,
  "comment_count": 6,
  "views": 0,
  "content": "<p>One of the key take-aways from the VinBigData competition was that calibrating confidences of the detection models can greatly improve the mAP (see <a href=\"https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/229637\" target=\"_blank\">here</a> for instance).</p>\n<p>In this competition, it seems like this didn't work consistently, any ideas why ? </p>\n<p>My guess models that predict opacity are not really better than the detection models, but i'm not entirely convinced. </p>",
  "messages": [
    {
      "id": "1463639",
      "postDate": "08/10/2021 09:11:16",
      "content": "<p>One of the key take-aways from the VinBigData competition was that calibrating confidences of the detection models can greatly improve the mAP (see <a href=\"https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/229637\" target=\"_blank\">here</a> for instance).</p>\n<p>In this competition, it seems like this didn't work consistently, any ideas why ? </p>\n<p>My guess models that predict opacity are not really better than the detection models, but i'm not entirely convinced. </p>",
      "rawMarkdown": "One of the key take-aways from the VinBigData competition was that calibrating confidences of the detection models can greatly improve the mAP (see [here](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/229637) for instance).\n\nIn this competition, it seems like this didn't work consistently, any ideas why ? \n\nMy guess models that predict opacity are not really better than the detection models, but i'm not entirely convinced.",
      "votes": null
    },
    {
      "id": "1463647",
      "postDate": "08/10/2021 09:14:46",
      "content": "<p>We used it and it worked, didn't give a huge boost it did something for sure😄. Hopefully, we'll be able to complete our write-up by this week. You're welcome to check..</p>",
      "rawMarkdown": "We used it and it worked, didn't give a huge boost it did something for sure😄. Hopefully, we'll be able to complete our write-up by this week. You're welcome to check..",
      "votes": null
    },
    {
      "id": "1463653",
      "postDate": "08/10/2021 09:16:42",
      "content": "<p>Hey. Just guessing, was it something related to geometric mean or power blending highest opacity confidence with binary predictions??</p>",
      "rawMarkdown": "Hey. Just guessing, was it something related to geometric mean or power blending highest opacity confidence with binary predictions??",
      "votes": null
    },
    {
      "id": "1463654",
      "postDate": "08/10/2021 09:17:33",
      "content": "<p>Geometric Mean</p>",
      "rawMarkdown": "Geometric Mean",
      "votes": null
    },
    {
      "id": "1463661",
      "postDate": "08/10/2021 09:19:28",
      "content": "<p>Got it, we also used something similar, which gave us a good amount of boost on both cv/lb.</p>",
      "rawMarkdown": "Got it, we also used something similar, which gave us a good amount of boost on both cv/lb.",
      "votes": null
    },
    {
      "id": "1463815",
      "postDate": "08/10/2021 10:42:50",
      "content": "<p>Interesting ! Thanks for the feedback <a href=\"https://www.kaggle.com/awsaf49\" target=\"_blank\">@awsaf49</a> <a href=\"https://www.kaggle.com/nischaydnk\" target=\"_blank\">@nischaydnk</a>, I'll definitely check that out.</p>",
      "rawMarkdown": "Interesting ! Thanks for the feedback @awsaf49 @nischaydnk, I'll definitely check that out.",
      "votes": null
    },
    {
      "id": "1467450",
      "postDate": "08/12/2021 01:55:42",
      "content": "<p>Our team image-level result with and with out post-processing calibrating confidences:</p>\n<ul>\n<li><p>With out post-processing calibrating confidences:</p>\n<ul>\n<li>Class None: PbL/PrL - 0.133/0.131</li>\n<li>Class Opacity: PbL/PrL - 0.098/0.101</li></ul></li>\n<li><p>With post-processing calibrating confidences:</p>\n<ul>\n<li>PbL/PrL score: 0.234/0.235</li></ul></li>\n</ul>",
      "rawMarkdown": "Our team image-level result with and with out post-processing calibrating confidences:\n\n- With out post-processing calibrating confidences:\n    + Class None: PbL/PrL - 0.133/0.131\n    + Class Opacity: PbL/PrL - 0.098/0.101\n\n- With post-processing calibrating confidences:\n    + PbL/PrL score: 0.234/0.235",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1463647,
      "author_name": "awsaf49",
      "author_url": "",
      "post_date": "08/10/2021 09:14:46",
      "content": "<p>We used it and it worked, didn't give a huge boost it did something for sure😄. Hopefully, we'll be able to complete our write-up by this week. You're welcome to check..</p>",
      "votes": null,
      "replies": [
        {
          "id": 1463653,
          "author_name": "nischaydnk",
          "author_url": "",
          "post_date": "08/10/2021 09:16:42",
          "content": "<p>Hey. Just guessing, was it something related to geometric mean or power blending highest opacity confidence with binary predictions??</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1463654,
          "author_name": "awsaf49",
          "author_url": "",
          "post_date": "08/10/2021 09:17:33",
          "content": "<p>Geometric Mean</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1463661,
          "author_name": "nischaydnk",
          "author_url": "",
          "post_date": "08/10/2021 09:19:28",
          "content": "<p>Got it, we also used something similar, which gave us a good amount of boost on both cv/lb.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1463815,
          "author_name": "theoviel",
          "author_url": "",
          "post_date": "08/10/2021 10:42:50",
          "content": "<p>Interesting ! Thanks for the feedback <a href=\"https://www.kaggle.com/awsaf49\" target=\"_blank\">@awsaf49</a> <a href=\"https://www.kaggle.com/nischaydnk\" target=\"_blank\">@nischaydnk</a>, I'll definitely check that out.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1467450,
      "author_name": "researchbntz",
      "author_url": "",
      "post_date": "08/12/2021 01:55:42",
      "content": "<p>Our team image-level result with and with out post-processing calibrating confidences:</p>\n<ul>\n<li><p>With out post-processing calibrating confidences:</p>\n<ul>\n<li>Class None: PbL/PrL - 0.133/0.131</li>\n<li>Class Opacity: PbL/PrL - 0.098/0.101</li></ul></li>\n<li><p>With post-processing calibrating confidences:</p>\n<ul>\n<li>PbL/PrL score: 0.234/0.235</li></ul></li>\n</ul>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1463639": "One of the key take-aways from the VinBigData competition was that calibrating confidences of the detection models can greatly improve the mAP (see [here](https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/discussion/229637) for instance).\n\nIn this competition, it seems like this didn't work consistently, any ideas why ? \n\nMy guess models that predict opacity are not really better than the detection models, but i'm not entirely convinced.",
    "1463647": "We used it and it worked, didn't give a huge boost it did something for sure😄. Hopefully, we'll be able to complete our write-up by this week. You're welcome to check..",
    "1463653": "Hey. Just guessing, was it something related to geometric mean or power blending highest opacity confidence with binary predictions??",
    "1463654": "Geometric Mean",
    "1463661": "Got it, we also used something similar, which gave us a good amount of boost on both cv/lb.",
    "1463815": "Interesting ! Thanks for the feedback @awsaf49 @nischaydnk, I'll definitely check that out.",
    "1467450": "Our team image-level result with and with out post-processing calibrating confidences:\n\n- With out post-processing calibrating confidences:\n    + Class None: PbL/PrL - 0.133/0.131\n    + Class Opacity: PbL/PrL - 0.098/0.101\n\n- With post-processing calibrating confidences:\n    + PbL/PrL score: 0.234/0.235"
  },
  "source": "meta"
}