{
  "id": 288762,
  "title": "Order of instances in metric / submission",
  "url": "/competitions/sartorius-cell-instance-segmentation/discussion/288762",
  "author_name": "",
  "post_date": "2021-11-18T10:36:16.216486900Z",
  "votes": 6,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hello fellow Kagglers, I am trying to understand this competition and specifically metric and it seems to me that order of instance rows in the submission might change the score.</p>\n<p>Unfortunately there is very little information about how actual LB score is calculated but per my experiments with <a href=\"https://www.kaggle.com/theoviel/competition-metric-map-iou\" target=\"_blank\">this notebook</a> the order of instance rows matter.</p>\n<p>I've created a notebook to demo this <a href=\"https://www.kaggle.com/vsedelnik/order-of-instances-in-metric-submission\" target=\"_blank\">here</a></p>\n<p>Any feedback is greatly appreciated!</p>",
  "messages": [
    {
      "id": "1586905",
      "postDate": "11/18/2021 10:36:16",
      "content": "<p>Hello fellow Kagglers, I am trying to understand this competition and specifically metric and it seems to me that order of instance rows in the submission might change the score.</p>\n<p>Unfortunately there is very little information about how actual LB score is calculated but per my experiments with <a href=\"https://www.kaggle.com/theoviel/competition-metric-map-iou\" target=\"_blank\">this notebook</a> the order of instance rows matter.</p>\n<p>I've created a notebook to demo this <a href=\"https://www.kaggle.com/vsedelnik/order-of-instances-in-metric-submission\" target=\"_blank\">here</a></p>\n<p>Any feedback is greatly appreciated!</p>",
      "rawMarkdown": "Hello fellow Kagglers, I am trying to understand this competition and specifically metric and it seems to me that order of instance rows in the submission might change the score.\n\nUnfortunately there is very little information about how actual LB score is calculated but per my experiments with [this notebook](https://www.kaggle.com/theoviel/competition-metric-map-iou) the order of instance rows matter.\n\nI've created a notebook to demo this [here](https://www.kaggle.com/vsedelnik/order-of-instances-in-metric-submission)\n\nAny feedback is greatly appreciated!",
      "votes": null
    },
    {
      "id": "1587350",
      "postDate": "11/18/2021 16:58:45",
      "content": "<p>This only happens because you are not handling overlaps and shuffling affects which instance gets on top. This won’t happen if you change the iou implementation not to assume there is at most one object at any pixel.</p>\n<p>[Edit:]Here is what I get using one of my validation images and the scoring logic from <a href=\"https://www.kaggle.com/slawekbiel/validation-score-example\" target=\"_blank\">here</a>:</p>\n<pre><code>for _ in range(5):\n    random.shuffle(preds)\n    random.shuffle(targs)\n    print(score(preds, targs))\n\n0.16830589731113138\n0.16830589731113138\n0.16830589731113138\n0.16830589731113138\n0.16830589731113138\n</code></pre>",
      "rawMarkdown": "This only happens because you are not handling overlaps and shuffling affects which instance gets on top. This won’t happen if you change the iou implementation not to assume there is at most one object at any pixel.\n\n[Edit:]Here is what I get using one of my validation images and the scoring logic from [here](https://www.kaggle.com/slawekbiel/validation-score-example):\n```\nfor _ in range(5):\n    random.shuffle(preds)\n    random.shuffle(targs)\n    print(score(preds, targs))\n\n0.16830589731113138\n0.16830589731113138\n0.16830589731113138\n0.16830589731113138\n0.16830589731113138\n```",
      "votes": null
    },
    {
      "id": "1587657",
      "postDate": "11/18/2021 20:32:55",
      "content": "<p>Thanks, will look into that</p>",
      "rawMarkdown": "Thanks, will look into that",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1587350,
      "author_name": "slawekbiel",
      "author_url": "",
      "post_date": "11/18/2021 16:58:45",
      "content": "<p>This only happens because you are not handling overlaps and shuffling affects which instance gets on top. This won’t happen if you change the iou implementation not to assume there is at most one object at any pixel.</p>\n<p>[Edit:]Here is what I get using one of my validation images and the scoring logic from <a href=\"https://www.kaggle.com/slawekbiel/validation-score-example\" target=\"_blank\">here</a>:</p>\n<pre><code>for _ in range(5):\n    random.shuffle(preds)\n    random.shuffle(targs)\n    print(score(preds, targs))\n\n0.16830589731113138\n0.16830589731113138\n0.16830589731113138\n0.16830589731113138\n0.16830589731113138\n</code></pre>",
      "votes": null,
      "replies": [
        {
          "id": 1587657,
          "author_name": "vsedelnik",
          "author_url": "",
          "post_date": "11/18/2021 20:32:55",
          "content": "<p>Thanks, will look into that</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1586905": "Hello fellow Kagglers, I am trying to understand this competition and specifically metric and it seems to me that order of instance rows in the submission might change the score.\n\nUnfortunately there is very little information about how actual LB score is calculated but per my experiments with [this notebook](https://www.kaggle.com/theoviel/competition-metric-map-iou) the order of instance rows matter.\n\nI've created a notebook to demo this [here](https://www.kaggle.com/vsedelnik/order-of-instances-in-metric-submission)\n\nAny feedback is greatly appreciated!",
    "1587350": "This only happens because you are not handling overlaps and shuffling affects which instance gets on top. This won’t happen if you change the iou implementation not to assume there is at most one object at any pixel.\n\n[Edit:]Here is what I get using one of my validation images and the scoring logic from [here](https://www.kaggle.com/slawekbiel/validation-score-example):\n```\nfor _ in range(5):\n    random.shuffle(preds)\n    random.shuffle(targs)\n    print(score(preds, targs))\n\n0.16830589731113138\n0.16830589731113138\n0.16830589731113138\n0.16830589731113138\n0.16830589731113138\n```",
    "1587657": "Thanks, will look into that"
  },
  "source": "meta"
}