{
  "id": 456033,
  "title": "Solving a lot of the Submission Scoring Errors",
  "url": "/competitions/blood-vessel-segmentation/discussion/456033",
  "author_name": "",
  "post_date": "2023-11-17T17:21:38.727819200Z",
  "votes": 30,
  "comment_count": 14,
  "views": 0,
  "content": "<p>So… in the past week, after the competition metric was patched to support \"1 0\", We see a lot of \"Submission Scoring Error\" where the subs that were working are now giving this error, the public leaderboard notebook of score 0.147 is giving this error, the reason is most likely the notebook runs out of memory when the metric is calculated, </p>\n<p>The current solve that i have found is by removing small pixels areas showing up in predictions</p>\n<pre><code> ():\n    \n    num_labels, labels, stats, centroids = cv2.connectedComponentsWithStats(img, connectivity=)\n\n    \n    new_img = np.zeros_like(img)\n     label  (, num_labels):\n         stats[label, cv2.CC_STAT_AREA] &gt;= min_size:\n            new_img[labels == label] = \n\n     new_img\n</code></pre>\n<p><code>mask = remove_small_objects(mask, 30)</code> is to be used, I recommend using this just before encoding the RLE, the function takes in shape (Height, width) in a numpy array with the dtype of np.uint8</p>\n<p>The <code>min_size</code> parameter used in the function should ideally be kept to as low as possible, for instance, when the value was 30, my model scored 0.458 and with the value 10, the same inference scored 0.527, I tried the value of 5 but that gave me the \"Submission Scoring Error\".</p>\n<p>Do upvote if it helped</p>",
  "messages": [
    {
      "id": "2528871",
      "postDate": "11/17/2023 17:21:38",
      "content": "<p>So… in the past week, after the competition metric was patched to support \"1 0\", We see a lot of \"Submission Scoring Error\" where the subs that were working are now giving this error, the public leaderboard notebook of score 0.147 is giving this error, the reason is most likely the notebook runs out of memory when the metric is calculated, </p>\n<p>The current solve that i have found is by removing small pixels areas showing up in predictions</p>\n<pre><code> ():\n    \n    num_labels, labels, stats, centroids = cv2.connectedComponentsWithStats(img, connectivity=)\n\n    \n    new_img = np.zeros_like(img)\n     label  (, num_labels):\n         stats[label, cv2.CC_STAT_AREA] &gt;= min_size:\n            new_img[labels == label] = \n\n     new_img\n</code></pre>\n<p><code>mask = remove_small_objects(mask, 30)</code> is to be used, I recommend using this just before encoding the RLE, the function takes in shape (Height, width) in a numpy array with the dtype of np.uint8</p>\n<p>The <code>min_size</code> parameter used in the function should ideally be kept to as low as possible, for instance, when the value was 30, my model scored 0.458 and with the value 10, the same inference scored 0.527, I tried the value of 5 but that gave me the \"Submission Scoring Error\".</p>\n<p>Do upvote if it helped</p>",
      "rawMarkdown": "So... in the past week, after the competition metric was patched to support \"1 0\", We see a lot of \"Submission Scoring Error\" where the subs that were working are now giving this error, the public leaderboard notebook of score 0.147 is giving this error, the reason is most likely the notebook runs out of memory when the metric is calculated, \n\nThe current solve that i have found is by removing small pixels areas showing up in predictions\n\n```python\ndef remove_small_objects(img, min_size):\n    # Find all connected components (labels)\n    num_labels, labels, stats, centroids = cv2.connectedComponentsWithStats(img, connectivity=8)\n\n    # Create a mask where small objects are removed\n    new_img = np.zeros_like(img)\n    for label in range(1, num_labels):\n        if stats[label, cv2.CC_STAT_AREA] >= min_size:\n            new_img[labels == label] = 255\n\n    return new_img\n```\n\n`mask = remove_small_objects(mask, 30)` is to be used, I recommend using this just before encoding the RLE, the function takes in shape (Height, width) in a numpy array with the dtype of np.uint8\n\nThe `min_size` parameter used in the function should ideally be kept to as low as possible, for instance, when the value was 30, my model scored 0.458 and with the value 10, the same inference scored 0.527, I tried the value of 5 but that gave me the \"Submission Scoring Error\".\n\nDo upvote if it helped",
      "votes": null
    },
    {
      "id": "2529173",
      "postDate": "11/18/2023 02:27:14",
      "content": "<p>Thanks for the useful information!<br>\nUnfortunately, I tried applying it to my Baseline Notebook and it did not solve the Scoring Error.<br>\nBoth min_size=10 and min_size=30 give the error.<br>\nIt seems that this countermeasure is not a panacea :(</p>\n<p>(<a href=\"https://www.kaggle.com/code/kashiwaba/sennet-hoa-inference-unet-simple-baseline?scriptVersionId=151194206\" target=\"_blank\">Notebook link</a> with apply remove_small_objects)</p>",
      "rawMarkdown": "Thanks for the useful information!\nUnfortunately, I tried applying it to my Baseline Notebook and it did not solve the Scoring Error.\nBoth min_size=10 and min_size=30 give the error.\nIt seems that this countermeasure is not a panacea :(\n\n([Notebook link](https://www.kaggle.com/code/kashiwaba/sennet-hoa-inference-unet-simple-baseline?scriptVersionId=151194206) with apply remove_small_objects)",
      "votes": null
    },
    {
      "id": "2529182",
      "postDate": "11/18/2023 02:48:27",
      "content": "<p>i used a combination of <strong>increasing the threshold</strong> and <strong>removing small pixels</strong> method but to no avail<br>\nhere the link to my code (<a href=\"https://www.kaggle.com/code/dieptam/submission-notebook\" target=\"_blank\">https://www.kaggle.com/code/dieptam/submission-notebook</a>) i modify some of <a href=\"https://www.kaggle.com/kashiwaba\" target=\"_blank\">@kashiwaba</a> code, thank you for the baseline by the way</p>",
      "rawMarkdown": "i used a combination of **increasing the threshold** and **removing small pixels** method but to no avail\nhere the link to my code (https://www.kaggle.com/code/dieptam/submission-notebook) i modify some of @kashiwaba code, thank you for the baseline by the way",
      "votes": null
    },
    {
      "id": "2529228",
      "postDate": "11/18/2023 04:10:20",
      "content": "<p>Your observation about the computation limit is spot on. It appears that the current limitations might be a significant factor in these challenges. The combined efforts and shared experiences of the community definitely highlight the need for intervention from the Kaggle team.</p>\n<p>Thank you as well for your contribution to this ongoing discussion. Let's remain hopeful that the Kaggle team will address these issues soon, making the submission process smoother for everyone.</p>",
      "rawMarkdown": "Your observation about the computation limit is spot on. It appears that the current limitations might be a significant factor in these challenges. The combined efforts and shared experiences of the community definitely highlight the need for intervention from the Kaggle team.\n\nThank you as well for your contribution to this ongoing discussion. Let's remain hopeful that the Kaggle team will address these issues soon, making the submission process smoother for everyone.",
      "votes": null
    },
    {
      "id": "2529256",
      "postDate": "11/18/2023 04:35:52",
      "content": "<p>Currently this <a href=\"https://www.kaggle.com/code/metric/surface-dice-metric/notebook\" target=\"_blank\">metric notebook</a> fails in validation due to the latest modification. I believe Kaggle team is working on this issue.</p>",
      "rawMarkdown": "Currently this [metric notebook](https://www.kaggle.com/code/metric/surface-dice-metric/notebook) fails in validation due to the latest modification. I believe Kaggle team is working on this issue.",
      "votes": null
    },
    {
      "id": "2529294",
      "postDate": "11/18/2023 05:19:52",
      "content": "<p>I tried your notebook with min_area=100 and even that fails, maybe memory usage has something more to do with than just area, one more trend we can see is that the better the score on the leaderboard, the less this problem is occurring.</p>",
      "rawMarkdown": "I tried your notebook with min_area=100 and even that fails, maybe memory usage has something more to do with than just area, one more trend we can see is that the better the score on the leaderboard, the less this problem is occurring.",
      "votes": null
    },
    {
      "id": "2529313",
      "postDate": "11/18/2023 05:44:02",
      "content": "<p>I think people at the top probably submitted right before the fixes of the scoring system ( (just after we discovered the 1 1 trick)… They are probably struggling right now too…</p>",
      "rawMarkdown": "I think people at the top probably submitted right before the fixes of the scoring system ( (just after we discovered the 1 1 trick)... They are probably struggling right now too...",
      "votes": null
    },
    {
      "id": "2529338",
      "postDate": "11/18/2023 06:20:03",
      "content": "<p>I think rank 1 jumped from his 55+ to 75+ score after the day I started experiencing the submission scoring error.</p>",
      "rawMarkdown": "I think rank 1 jumped from his 55+ to 75+ score after the day I started experiencing the submission scoring error.",
      "votes": null
    },
    {
      "id": "2529449",
      "postDate": "11/18/2023 08:53:12",
      "content": "<p>Thanks for your sharing, did you use 2D or 3D image segmentation model?</p>",
      "rawMarkdown": "Thanks for your sharing, did you use 2D or 3D image segmentation model?",
      "votes": null
    },
    {
      "id": "2529467",
      "postDate": "11/18/2023 09:16:46",
      "content": "<p>my last submission has the submission scoring error previously till that it was working perfectly</p>",
      "rawMarkdown": "my last submission has the submission scoring error previously till that it was working perfectly",
      "votes": null
    },
    {
      "id": "2529651",
      "postDate": "11/18/2023 12:42:59",
      "content": "<p>I tried this method, but the result still shows a Submission Scoring Error. Additionally, I noticed in the code you provided that there's this line \"new_img[labels == label] = 255\". Are you converting the predicted mask to 0 and 255 in the end? By the way, could you please share your RLE code? I'm curious to see it.</p>",
      "rawMarkdown": "I tried this method, but the result still shows a Submission Scoring Error. Additionally, I noticed in the code you provided that there's this line \"new_img[labels == label] = 255\". Are you converting the predicted mask to 0 and 255 in the end? By the way, could you please share your RLE code? I'm curious to see it.",
      "votes": null
    },
    {
      "id": "2529747",
      "postDate": "11/18/2023 14:04:17",
      "content": "<p>Yes, I think that the output mask of the function only has either 0 or 255 as values with dtype np.uint8 </p>\n<pre><code> ():\n    \n    pixels = img.flatten()\n    pixels = np.concatenate([[], pixels, []])\n    runs = np.where(pixels[:] != pixels[:-])[] + \n    runs[::] -= runs[::]\n    rle = .join((x)  x  runs)\n     rle==:\n        rle = \n     rle\n</code></pre>",
      "rawMarkdown": "Yes, I think that the output mask of the function only has either 0 or 255 as values with dtype np.uint8 \n\n```python\ndef rle_encode(img):\n    '''\n    img: numpy array, 1 - mask, 0 - background\n    Returns run length as string formated\n    '''\n    pixels = img.flatten()\n    pixels = np.concatenate([[0], pixels, [0]])\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n    rle = ' '.join(str(x) for x in runs)\n    if rle=='':\n        rle = '1 0'\n    return rle\n```",
      "votes": null
    },
    {
      "id": "2529748",
      "postDate": "11/18/2023 14:04:29",
      "content": "<p>I am still experimenting with 2D models</p>",
      "rawMarkdown": "I am still experimenting with 2D models",
      "votes": null
    },
    {
      "id": "2548824",
      "postDate": "12/04/2023 18:13:30",
      "content": "<p>I ran your code and got an error(</p>",
      "rawMarkdown": "I ran your code and got an error(",
      "votes": null
    },
    {
      "id": "2875103",
      "postDate": "06/16/2024 20:56:02",
      "content": "<p>hello a, anh ơi a có thể cho e xin contact liên hệ e hỏi một chút về NeoPolyp bkai đc ko a =_=</p>",
      "rawMarkdown": "hello a, anh ơi a có thể cho e xin contact liên hệ e hỏi một chút về NeoPolyp bkai đc ko a =_=",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2529173,
      "author_name": "kashiwaba",
      "author_url": "",
      "post_date": "11/18/2023 02:27:14",
      "content": "<p>Thanks for the useful information!<br>\nUnfortunately, I tried applying it to my Baseline Notebook and it did not solve the Scoring Error.<br>\nBoth min_size=10 and min_size=30 give the error.<br>\nIt seems that this countermeasure is not a panacea :(</p>\n<p>(<a href=\"https://www.kaggle.com/code/kashiwaba/sennet-hoa-inference-unet-simple-baseline?scriptVersionId=151194206\" target=\"_blank\">Notebook link</a> with apply remove_small_objects)</p>",
      "votes": null,
      "replies": [
        {
          "id": 2529182,
          "author_name": "dieptam",
          "author_url": "",
          "post_date": "11/18/2023 02:48:27",
          "content": "<p>i used a combination of <strong>increasing the threshold</strong> and <strong>removing small pixels</strong> method but to no avail<br>\nhere the link to my code (<a href=\"https://www.kaggle.com/code/dieptam/submission-notebook\" target=\"_blank\">https://www.kaggle.com/code/dieptam/submission-notebook</a>) i modify some of <a href=\"https://www.kaggle.com/kashiwaba\" target=\"_blank\">@kashiwaba</a> code, thank you for the baseline by the way</p>",
          "votes": null,
          "replies": [
            {
              "id": 2548824,
              "author_name": "ruslan1511",
              "author_url": "",
              "post_date": "12/04/2023 18:13:30",
              "content": "<p>I ran your code and got an error(</p>",
              "votes": null,
              "replies": []
            }
          ]
        },
        {
          "id": 2529294,
          "author_name": "harshitsheoran",
          "author_url": "",
          "post_date": "11/18/2023 05:19:52",
          "content": "<p>I tried your notebook with min_area=100 and even that fails, maybe memory usage has something more to do with than just area, one more trend we can see is that the better the score on the leaderboard, the less this problem is occurring.</p>",
          "votes": null,
          "replies": [
            {
              "id": 2529313,
              "author_name": "abdrah",
              "author_url": "",
              "post_date": "11/18/2023 05:44:02",
              "content": "<p>I think people at the top probably submitted right before the fixes of the scoring system ( (just after we discovered the 1 1 trick)… They are probably struggling right now too…</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2529338,
                  "author_name": "harshitsheoran",
                  "author_url": "",
                  "post_date": "11/18/2023 06:20:03",
                  "content": "<p>I think rank 1 jumped from his 55+ to 75+ score after the day I started experiencing the submission scoring error.</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 2529467,
                      "author_name": "arunodhayan",
                      "author_url": "",
                      "post_date": "11/18/2023 09:16:46",
                      "content": "<p>my last submission has the submission scoring error previously till that it was working perfectly</p>",
                      "votes": null,
                      "replies": []
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 2529228,
      "author_name": "abdrah",
      "author_url": "",
      "post_date": "11/18/2023 04:10:20",
      "content": "<p>Your observation about the computation limit is spot on. It appears that the current limitations might be a significant factor in these challenges. The combined efforts and shared experiences of the community definitely highlight the need for intervention from the Kaggle team.</p>\n<p>Thank you as well for your contribution to this ongoing discussion. Let's remain hopeful that the Kaggle team will address these issues soon, making the submission process smoother for everyone.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2529256,
          "author_name": "snorfyang",
          "author_url": "",
          "post_date": "11/18/2023 04:35:52",
          "content": "<p>Currently this <a href=\"https://www.kaggle.com/code/metric/surface-dice-metric/notebook\" target=\"_blank\">metric notebook</a> fails in validation due to the latest modification. I believe Kaggle team is working on this issue.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2529449,
      "author_name": "tungbt62hust",
      "author_url": "",
      "post_date": "11/18/2023 08:53:12",
      "content": "<p>Thanks for your sharing, did you use 2D or 3D image segmentation model?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2529748,
          "author_name": "harshitsheoran",
          "author_url": "",
          "post_date": "11/18/2023 14:04:29",
          "content": "<p>I am still experimenting with 2D models</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2875103,
          "author_name": "he163767huynq",
          "author_url": "",
          "post_date": "06/16/2024 20:56:02",
          "content": "<p>hello a, anh ơi a có thể cho e xin contact liên hệ e hỏi một chút về NeoPolyp bkai đc ko a =_=</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2529651,
      "author_name": "chrisluu",
      "author_url": "",
      "post_date": "11/18/2023 12:42:59",
      "content": "<p>I tried this method, but the result still shows a Submission Scoring Error. Additionally, I noticed in the code you provided that there's this line \"new_img[labels == label] = 255\". Are you converting the predicted mask to 0 and 255 in the end? By the way, could you please share your RLE code? I'm curious to see it.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2529747,
          "author_name": "harshitsheoran",
          "author_url": "",
          "post_date": "11/18/2023 14:04:17",
          "content": "<p>Yes, I think that the output mask of the function only has either 0 or 255 as values with dtype np.uint8 </p>\n<pre><code> ():\n    \n    pixels = img.flatten()\n    pixels = np.concatenate([[], pixels, []])\n    runs = np.where(pixels[:] != pixels[:-])[] + \n    runs[::] -= runs[::]\n    rle = .join((x)  x  runs)\n     rle==:\n        rle = \n     rle\n</code></pre>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2528871": "So... in the past week, after the competition metric was patched to support \"1 0\", We see a lot of \"Submission Scoring Error\" where the subs that were working are now giving this error, the public leaderboard notebook of score 0.147 is giving this error, the reason is most likely the notebook runs out of memory when the metric is calculated, \n\nThe current solve that i have found is by removing small pixels areas showing up in predictions\n\n```python\ndef remove_small_objects(img, min_size):\n    # Find all connected components (labels)\n    num_labels, labels, stats, centroids = cv2.connectedComponentsWithStats(img, connectivity=8)\n\n    # Create a mask where small objects are removed\n    new_img = np.zeros_like(img)\n    for label in range(1, num_labels):\n        if stats[label, cv2.CC_STAT_AREA] >= min_size:\n            new_img[labels == label] = 255\n\n    return new_img\n```\n\n`mask = remove_small_objects(mask, 30)` is to be used, I recommend using this just before encoding the RLE, the function takes in shape (Height, width) in a numpy array with the dtype of np.uint8\n\nThe `min_size` parameter used in the function should ideally be kept to as low as possible, for instance, when the value was 30, my model scored 0.458 and with the value 10, the same inference scored 0.527, I tried the value of 5 but that gave me the \"Submission Scoring Error\".\n\nDo upvote if it helped",
    "2529173": "Thanks for the useful information!\nUnfortunately, I tried applying it to my Baseline Notebook and it did not solve the Scoring Error.\nBoth min_size=10 and min_size=30 give the error.\nIt seems that this countermeasure is not a panacea :(\n\n([Notebook link](https://www.kaggle.com/code/kashiwaba/sennet-hoa-inference-unet-simple-baseline?scriptVersionId=151194206) with apply remove_small_objects)",
    "2529182": "i used a combination of **increasing the threshold** and **removing small pixels** method but to no avail\nhere the link to my code (https://www.kaggle.com/code/dieptam/submission-notebook) i modify some of @kashiwaba code, thank you for the baseline by the way",
    "2529228": "Your observation about the computation limit is spot on. It appears that the current limitations might be a significant factor in these challenges. The combined efforts and shared experiences of the community definitely highlight the need for intervention from the Kaggle team.\n\nThank you as well for your contribution to this ongoing discussion. Let's remain hopeful that the Kaggle team will address these issues soon, making the submission process smoother for everyone.",
    "2529256": "Currently this [metric notebook](https://www.kaggle.com/code/metric/surface-dice-metric/notebook) fails in validation due to the latest modification. I believe Kaggle team is working on this issue.",
    "2529294": "I tried your notebook with min_area=100 and even that fails, maybe memory usage has something more to do with than just area, one more trend we can see is that the better the score on the leaderboard, the less this problem is occurring.",
    "2529313": "I think people at the top probably submitted right before the fixes of the scoring system ( (just after we discovered the 1 1 trick)... They are probably struggling right now too...",
    "2529338": "I think rank 1 jumped from his 55+ to 75+ score after the day I started experiencing the submission scoring error.",
    "2529449": "Thanks for your sharing, did you use 2D or 3D image segmentation model?",
    "2529467": "my last submission has the submission scoring error previously till that it was working perfectly",
    "2529651": "I tried this method, but the result still shows a Submission Scoring Error. Additionally, I noticed in the code you provided that there's this line \"new_img[labels == label] = 255\". Are you converting the predicted mask to 0 and 255 in the end? By the way, could you please share your RLE code? I'm curious to see it.",
    "2529747": "Yes, I think that the output mask of the function only has either 0 or 255 as values with dtype np.uint8 \n\n```python\ndef rle_encode(img):\n    '''\n    img: numpy array, 1 - mask, 0 - background\n    Returns run length as string formated\n    '''\n    pixels = img.flatten()\n    pixels = np.concatenate([[0], pixels, [0]])\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n    rle = ' '.join(str(x) for x in runs)\n    if rle=='':\n        rle = '1 0'\n    return rle\n```",
    "2529748": "I am still experimenting with 2D models",
    "2548824": "I ran your code and got an error(",
    "2875103": "hello a, anh ơi a có thể cho e xin contact liên hệ e hỏi một chút về NeoPolyp bkai đc ko a =_="
  },
  "source": "meta"
}