{ "id": 418348, "title": "Poor submission scores", "url": "/competitions/hubmap-hacking-the-human-vasculature/discussion/418348", "author_name": "DennisSakva", "post_date": "2023-06-20T06:45:37.797000", "votes": 2, "comment_count": 19, "views": 0, "content": "

Hey guys, I'm stuck with poor submission scores and not quite sure what I'm doing wrong.
\nSo I'm treating this competition as semantic segmentation. I don't see much reason to use models designed to do real instance segmentation as our objects are well separated and thus the semantic segmentation masks can be easily converted into instance segmentation by using cv2.ConnectedComponents
\nMy code does the following:

\n
    \n
  1. Predict segmentation mask (See 1 test sample)
    \n\"\"
    2. \n
  2. Split segmentation mask into individual binarized masks using connected components
    \n\"\"
    3. \n
  3. Use the provided encode_binary_mask code to get individual mask encodings and merge them. Here is the sample submission string
    \n'0 1.0 eNrLDY8yt0u0N3Kx8TP0NTIw9EdAAygAs/0gcgZIwBCI/Q19jf3hCkBsMN/IDwRjPAEsihZJ 0 1.0 eNqLjgg0tUi3N/Ix8TMCQ0N/Q38YDYQGQOCPxkam4QCkAkJgB1ikoGYgG4XLGixOMURyLBCCPADEIRnRhgDcsC+w 0 1.0 eNqLycg1M8yzN/IzMPJLCw8wBAApugS0 0 1.0 eNoLjowys0m2N/Ix8TNChv4GUGAIZPkbwnj+hkBo4GdkYOht5mfkZuUXkR5mCAA9JhBx 0 1.0 eNqLjMsztkq1N/KDQn9Df0MYjQINoADBBglDaF8TPyMvM7+cxHhjADCrE8Y= 0 1.0 eNpLy4swM8q1N/Ex8jcAAkMQ4W/ob2DoD4YGEMrf2BcqgMYGqQdpBLFh+sEMA2RBEA1VbOhn5GcE0wxi+5l4pQakGwAAtbUgWw== 0 1.0 eNoLjs8zSTKItTfyM/My8jf2NfQ3QAeG/ghBbGx0GsbGhKbehn5hCWFGACm1GvE='
    4. \n
\n

And with this, I get 0.06-0.07. The score is above 0, so it seems to at least be working and submitting something that the kaggle server can score. My masks are not perfect, but they sure don't look like 0.06 imperfect. :) I seem to be missing something important though I cannot figure it out. Like in some segmentation competitions, you have to transpose your mask before RLE encoding it.
\nThanks!

", "messages": [ { "id": 2310991, "postDate": "2023-06-20T20:01:39.613Z", "content": "

This was my first approach as well since I wanted to try something I'm familiar with and this was my first time doing instance segmentation. This approach gave me around 0.133, and I have a few assumptions regarding that:

\n
    \n

  1. Confidence scores: Confidence scores are important so you need to try something that'll give you approximate confidence scores, I've tried using the mean of the pixels for each component's mask.

    2. \n

  2. Connected components: Maybe your model incorrectly bridges some of the components that should be separate, try separating them with morphological transformations, or maybe a loss function that emphasizes the separation between objects.

    3. \n

  3. The selection criteria are too strict: You have either set the pixel confidence threshold too high, or your model is missing components that should be present in the image. A few of the components that my models detect consistently are suspiciously missing from your example. Maybe you could get around that by choosing the pixels that have the highest scores in a locally instead of globally. A simple way to do that could be by choosing the pixels with the highest score among their neighbors and then selecting around them.

    4. \n
\n

Looking forward to seeing your results, since this approach is viable theoretically and I don't see a reason for it to give such low scores. Good luck.

", "rawMarkdown": "This was my first approach as well since I wanted to try something I'm familiar with and this was my first time doing instance segmentation. This approach gave me around 0.133, and I have a few assumptions regarding that:\n\n1. Confidence scores: Confidence scores are important so you need to try something that'll give you approximate confidence scores, I've tried using the mean of the pixels for each component's mask.\n\n2. Connected components: Maybe your model incorrectly bridges some of the components that should be separate, try separating them with morphological transformations, or maybe a loss function that emphasizes the separation between objects.\n\n3. The selection criteria are too strict: You have either set the pixel confidence threshold too high, or your model is missing components that should be present in the image. A few of the components that my models detect consistently are suspiciously missing from your example. Maybe you could get around that by choosing the pixels that have the highest scores in a locally instead of globally. A simple way to do that could be by choosing the pixels with the highest score among their neighbors and then selecting around them.\n\nLooking forward to seeing your results, since this approach is viable theoretically and I don't see a reason for it to give such low scores. Good luck.", "votes": 3 }, { "id": 2310295, "postDate": "2023-06-20T09:03:58.320Z", "content": "

Here you are using all the confident scores as 1.0.
\nSince you are converting the mask into the instances by cv2.ConnectedComponents I dont think you will be getting the confidence scores

", "rawMarkdown": "Here you are using all the confident scores as 1.0. \nSince you are converting the mask into the instances by cv2.ConnectedComponents I dont think you will be getting the confidence scores ", "votes": 1, "replies": [ { "id": 2310308, "postDate": "2023-06-20T09:10:57.997Z", "content": "

Yeah, I used mean mask value as a confidence score and got 0.12 instead of 0.06, so confidence scores are pretty important.
\nNow I need to think of a way to calibrate my confidence scores.

", "rawMarkdown": "Yeah, I used mean mask value as a confidence score and got 0.12 instead of 0.06, so confidence scores are pretty important.\nNow I need to think of a way to calibrate my confidence scores.", "votes": 2, "replies": [ { "id": 2310313, "postDate": "2023-06-20T09:16:04.763Z", "content": "

Yes the confidence scores are important !!

", "rawMarkdown": "Yes the confidence scores are important !!\n" }, { "id": 2310323, "postDate": "2023-06-20T09:28:23.313Z", "content": "

Have you seen how confidence score impacts metric calculation? I don't see it mentioned on the page which is supposed to give a detailed explanation of the metric
\nhttps://storage.googleapis.com/openimages/web/evaluation.html#instance_segmentation_eval

", "rawMarkdown": "Have you seen how confidence score impacts metric calculation? I don't see it mentioned on the page which is supposed to give a detailed explanation of the metric\nhttps://storage.googleapis.com/openimages/web/evaluation.html#instance_segmentation_eval" }, { "id": 2311000, "postDate": "2023-06-20T20:03:56.260Z", "content": "

The mAP metric, which is the one used in this competition, calculates the precision at different thresholds for confidence scores and takes the mean across all the thresholds. Confidence scores are important here.

", "rawMarkdown": "The mAP metric, which is the one used in this competition, calculates the precision at different thresholds for confidence scores and takes the mean across all the thresholds. Confidence scores are important here." }, { "id": 2311762, "postDate": "2023-06-21T12:42:33.520Z", "content": "

@sakvaua great idea, after I used same approach from 1.0 to mean of of prob mask - I got boost from 0.209 to 0.272 for Unet model

", "rawMarkdown": "@sakvaua great idea, after I used same approach from 1.0 to mean of of prob mask - I got boost from 0.209 to 0.272 for Unet model" } ] } ] }, { "id": 2310184, "postDate": "2023-06-20T07:16:25.110Z", "content": "

Mhh, yeah your masks don't look crazy.
\nOne thing I did see is that when setting confidence to 1.0 across the board my score was much worse than having values as predicted by the object detection model I used (YOLO).

\n

How do you plan on dealing with confidence scores?

\n

But I doubt this fully explains the 0.06, that's too low even for that.

", "rawMarkdown": "Mhh, yeah your masks don't look crazy.\nOne thing I did see is that when setting confidence to 1.0 across the board my score was much worse than having values as predicted by the object detection model I used (YOLO). \n\nHow do you plan on dealing with confidence scores? \n\nBut I doubt this fully explains the 0.06, that's too low even for that. ", "votes": 1, "replies": [ { "id": 2310195, "postDate": "2023-06-20T07:32:17.343Z", "content": "

Thanks, fnands. I'll play with confidence scores after I debug the main issue. Will probably use the average value of an unbinarized mask as a confidence score.

", "rawMarkdown": "Thanks, fnands. I'll play with confidence scores after I debug the main issue. Will probably use the average value of an unbinarized mask as a confidence score.", "replies": [ { "id": 2310582, "postDate": "2023-06-20T13:36:10.553Z", "content": "

Have you sorted your confidence before writing it into the submission? I'm not sure if this will affect the score

", "rawMarkdown": "Have you sorted your confidence before writing it into the submission? I'm not sure if this will affect the score", "votes": 1 }, { "id": 2310713, "postDate": "2023-06-20T15:07:23.540Z", "content": "

I haven't but it shouldn't really matter.

", "rawMarkdown": "I haven't but it shouldn't really matter." } ] } ] }, { "id": 2312064, "postDate": "2023-06-21T16:43:48.440Z", "content": "

I have found that a some amount of GT masks have a slight overlap between them. Have you found such cases?

\n

for example: b36f28986045
\n\"\"

", "rawMarkdown": "I have found that a some amount of GT masks have a slight overlap between them. Have you found such cases?\n\nfor example: b36f28986045\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14675197%2F3463ef1cad48bdad6b2b282552600f02%2FUntitled.png?generation=1687366119217014&alt=media)", "votes": 2, "replies": [ { "id": 2312086, "postDate": "2023-06-21T16:58:11.457Z", "content": "
\n

I have found that a some amount of GT masks have a slight overlap between them. Have you found such cases?

\n

for example: b36f28986045
\n \"\"

\n
\n

I've seen some overlap between different classes, but missed the same class overlaps.
\nYou've given me an idea. To add the overlap as a separate class to predict.

", "rawMarkdown": "> I have found that a some amount of GT masks have a slight overlap between them. Have you found such cases?\n> \n> for example: b36f28986045\n> ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14675197%2F3463ef1cad48bdad6b2b282552600f02%2FUntitled.png?generation=1687366119217014&alt=media)\n\nI've seen some overlap between different classes, but missed the same class overlaps.\nYou've given me an idea. To add the overlap as a separate class to predict." } ] }, { "id": 2310162, "postDate": "2023-06-20T06:45:37.797Z", "content": "

Hey guys, I'm stuck with poor submission scores and not quite sure what I'm doing wrong.
\nSo I'm treating this competition as semantic segmentation. I don't see much reason to use models designed to do real instance segmentation as our objects are well separated and thus the semantic segmentation masks can be easily converted into instance segmentation by using cv2.ConnectedComponents
\nMy code does the following:

\n
    \n
  1. Predict segmentation mask (See 1 test sample)
    \n\"\"
    2. \n
  2. Split segmentation mask into individual binarized masks using connected components
    \n\"\"
    3. \n
  3. Use the provided encode_binary_mask code to get individual mask encodings and merge them. Here is the sample submission string
    \n'0 1.0 eNrLDY8yt0u0N3Kx8TP0NTIw9EdAAygAs/0gcgZIwBCI/Q19jf3hCkBsMN/IDwRjPAEsihZJ 0 1.0 eNqLjgg0tUi3N/Ix8TMCQ0N/Q38YDYQGQOCPxkam4QCkAkJgB1ikoGYgG4XLGixOMURyLBCCPADEIRnRhgDcsC+w 0 1.0 eNqLycg1M8yzN/IzMPJLCw8wBAApugS0 0 1.0 eNoLjowys0m2N/Ix8TNChv4GUGAIZPkbwnj+hkBo4GdkYOht5mfkZuUXkR5mCAA9JhBx 0 1.0 eNqLjMsztkq1N/KDQn9Df0MYjQINoADBBglDaF8TPyMvM7+cxHhjADCrE8Y= 0 1.0 eNpLy4swM8q1N/Ex8jcAAkMQ4W/ob2DoD4YGEMrf2BcqgMYGqQdpBLFh+sEMA2RBEA1VbOhn5GcE0wxi+5l4pQakGwAAtbUgWw== 0 1.0 eNoLjs8zSTKItTfyM/My8jf2NfQ3QAeG/ghBbGx0GsbGhKbehn5hCWFGACm1GvE='
    4. \n
\n

And with this, I get 0.06-0.07. The score is above 0, so it seems to at least be working and submitting something that the kaggle server can score. My masks are not perfect, but they sure don't look like 0.06 imperfect. :) I seem to be missing something important though I cannot figure it out. Like in some segmentation competitions, you have to transpose your mask before RLE encoding it.
\nThanks!

", "rawMarkdown": "Hey guys, I'm stuck with poor submission scores and not quite sure what I'm doing wrong.\nSo I'm treating this competition as semantic segmentation. I don't see much reason to use models designed to do real instance segmentation as our objects are well separated and thus the semantic segmentation masks can be easily converted into instance segmentation by using cv2.ConnectedComponents\nMy code does the following:\n1. Predict segmentation mask (See 1 test sample)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F349155%2F31663618e54051e7c251cc69097eebfd%2FTestImg.png?generation=1687243121688714&alt=media)\n2. Split segmentation mask into individual binarized masks using connected components\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F349155%2F89792db245d18adf8fa9790a1f46af56%2FMask.png?generation=1687243159107534&alt=media)\n3. Use the provided encode_binary_mask code to get individual mask encodings and merge them. Here is the sample submission string\n'0 1.0 eNrLDY8yt0u0N3Kx8TP0NTIw9EdAAygAs/0gcgZIwBCI/Q19jf3hCkBsMN/IDwRjPAEsihZJ 0 1.0 eNqLjgg0tUi3N/Ix8TMCQ0N/Q38YDYQGQOCPxkam4QCkAkJgB1ikoGYgG4XLGixOMURyLBCCPADEIRnRhgDcsC+w 0 1.0 eNqLycg1M8yzN/IzMPJLCw8wBAApugS0 0 1.0 eNoLjowys0m2N/Ix8TNChv4GUGAIZPkbwnj+hkBo4GdkYOht5mfkZuUXkR5mCAA9JhBx 0 1.0 eNqLjMsztkq1N/KDQn9Df0MYjQINoADBBglDaF8TPyMvM7+cxHhjADCrE8Y= 0 1.0 eNpLy4swM8q1N/Ex8jcAAkMQ4W/ob2DoD4YGEMrf2BcqgMYGqQdpBLFh+sEMA2RBEA1VbOhn5GcE0wxi+5l4pQakGwAAtbUgWw== 0 1.0 eNoLjs8zSTKItTfyM/My8jf2NfQ3QAeG/ghBbGx0GsbGhKbehn5hCWFGACm1GvE='\n\nAnd with this, I get 0.06-0.07. The score is above 0, so it seems to at least be working and submitting something that the kaggle server can score. My masks are not perfect, but they sure don't look like 0.06 imperfect. :) I seem to be missing something important though I cannot figure it out. Like in some segmentation competitions, you have to transpose your mask before RLE encoding it.\nThanks!", "votes": 2 }, { "id": 2311412, "postDate": "2023-06-21T06:46:37.387Z", "content": "

Same problem here, using semantic segmentation. Still confused why confidence score affects submission score? What`s difference between 0.98,0.99 and 1.0 ? 😂

", "rawMarkdown": "Same problem here, using semantic segmentation. Still confused why confidence score affects submission score? What`s difference between 0.98,0.99 and 1.0 ? 😂", "replies": [ { "id": 2357654, "postDate": "2023-07-25T03:23:22.150Z", "content": "

I think it is important to judge whether a model is good or not, apart from whether it can correctly distinguish categories and semantics, do a good job of instance segmentation, and whether the model’s judgment on itself is certain, because the model with the same prediction accuracy, the higher the confidence in TP and TN, it means that it has learned the actual semantics and knows how to segment images and what instances it segmented.

", "rawMarkdown": "I think it is important to judge whether a model is good or not, apart from whether it can correctly distinguish categories and semantics, do a good job of instance segmentation, and whether the model’s judgment on itself is certain, because the model with the same prediction accuracy, the higher the confidence in TP and TN, it means that it has learned the actual semantics and knows how to segment images and what instances it segmented.", "isDeleted": true, "replies": [ { "id": 2357795, "postDate": "2023-07-25T05:59:01.290Z", "content": "

Finally, most of the people give up semantic segmentation on this task😂

", "rawMarkdown": "Finally, most of the people give up semantic segmentation on this task😂" }, { "id": 2357826, "postDate": "2023-07-25T06:27:34.480Z", "content": "

I think in the end there will be great semantic segmentation methods out there

", "rawMarkdown": "I think in the end there will be great semantic segmentation methods out there" }, { "id": 2357850, "postDate": "2023-07-25T06:59:15.800Z", "content": "

Most probably, the best would be object detection + instance segmentation models, like it was in Sartorius competition.

", "rawMarkdown": "Most probably, the best would be object detection + instance segmentation models, like it was in Sartorius competition." } ] } ] }, { "id": 2310857, "postDate": "2023-06-20T16:49:03.357Z", "content": "

Having same problem here with testing a segmentation model- it does seem like having different thresholds matter, but the score, like you said, are still really, really low for some reason, around 0.05-0.06 no matter what I try…

", "rawMarkdown": "Having same problem here with testing a segmentation model- it does seem like having different thresholds matter, but the score, like you said, are still really, really low for some reason, around 0.05-0.06 no matter what I try..." } ], "comments": [ { "id": 2310991, "author_name": "Abdelfattah Toulaoui", "author_url": "", "post_date": "2023-06-20T20:01:39.613000", "content": "

This was my first approach as well since I wanted to try something I'm familiar with and this was my first time doing instance segmentation. This approach gave me around 0.133, and I have a few assumptions regarding that:

\n
    \n

  1. Confidence scores: Confidence scores are important so you need to try something that'll give you approximate confidence scores, I've tried using the mean of the pixels for each component's mask.

    2. \n

  2. Connected components: Maybe your model incorrectly bridges some of the components that should be separate, try separating them with morphological transformations, or maybe a loss function that emphasizes the separation between objects.

    3. \n

  3. The selection criteria are too strict: You have either set the pixel confidence threshold too high, or your model is missing components that should be present in the image. A few of the components that my models detect consistently are suspiciously missing from your example. Maybe you could get around that by choosing the pixels that have the highest scores in a locally instead of globally. A simple way to do that could be by choosing the pixels with the highest score among their neighbors and then selecting around them.

    4. \n
\n

Looking forward to seeing your results, since this approach is viable theoretically and I don't see a reason for it to give such low scores. Good luck.

", "votes": 3, "replies": [] }, { "id": 2310295, "author_name": "Vishak K Bhat", "author_url": "", "post_date": "2023-06-20T09:03:58.320000", "content": "

Here you are using all the confident scores as 1.0.
\nSince you are converting the mask into the instances by cv2.ConnectedComponents I dont think you will be getting the confidence scores

", "votes": 1, "replies": [ { "id": 2310308, "author_name": "DennisSakva", "author_url": "", "post_date": "2023-06-20T09:10:57.997000", "content": "

Yeah, I used mean mask value as a confidence score and got 0.12 instead of 0.06, so confidence scores are pretty important.
\nNow I need to think of a way to calibrate my confidence scores.

", "votes": 2, "replies": [ { "id": 2310313, "author_name": "Vishak K Bhat", "author_url": "", "post_date": "2023-06-20T09:16:04.763000", "content": "

Yes the confidence scores are important !!

", "votes": 0, "replies": [] }, { "id": 2310323, "author_name": "DennisSakva", "author_url": "", "post_date": "2023-06-20T09:28:23.313000", "content": "

Have you seen how confidence score impacts metric calculation? I don't see it mentioned on the page which is supposed to give a detailed explanation of the metric
\nhttps://storage.googleapis.com/openimages/web/evaluation.html#instance_segmentation_eval

", "votes": 0, "replies": [] }, { "id": 2311000, "author_name": "Abdelfattah Toulaoui", "author_url": "", "post_date": "2023-06-20T20:03:56.260000", "content": "

The mAP metric, which is the one used in this competition, calculates the precision at different thresholds for confidence scores and takes the mean across all the thresholds. Confidence scores are important here.

", "votes": 0, "replies": [] }, { "id": 2311762, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2023-06-21T12:42:33.520000", "content": "

@sakvaua great idea, after I used same approach from 1.0 to mean of of prob mask - I got boost from 0.209 to 0.272 for Unet model

", "votes": 0, "replies": [] } ] } ] }, { "id": 2310184, "author_name": "fnands", "author_url": "", "post_date": "2023-06-20T07:16:25.110000", "content": "

Mhh, yeah your masks don't look crazy.
\nOne thing I did see is that when setting confidence to 1.0 across the board my score was much worse than having values as predicted by the object detection model I used (YOLO).

\n

How do you plan on dealing with confidence scores?

\n

But I doubt this fully explains the 0.06, that's too low even for that.

", "votes": 1, "replies": [ { "id": 2310195, "author_name": "DennisSakva", "author_url": "", "post_date": "2023-06-20T07:32:17.343000", "content": "

Thanks, fnands. I'll play with confidence scores after I debug the main issue. Will probably use the average value of an unbinarized mask as a confidence score.

", "votes": 0, "replies": [ { "id": 2310582, "author_name": "bent1e", "author_url": "", "post_date": "2023-06-20T13:36:10.553000", "content": "

Have you sorted your confidence before writing it into the submission? I'm not sure if this will affect the score

", "votes": 1, "replies": [] }, { "id": 2310713, "author_name": "DennisSakva", "author_url": "", "post_date": "2023-06-20T15:07:23.540000", "content": "

I haven't but it shouldn't really matter.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2312064, "author_name": "tsobolev", "author_url": "", "post_date": "2023-06-21T16:43:48.440000", "content": "

I have found that a some amount of GT masks have a slight overlap between them. Have you found such cases?

\n

for example: b36f28986045
\n\"\"

", "votes": 2, "replies": [ { "id": 2312086, "author_name": "DennisSakva", "author_url": "", "post_date": "2023-06-21T16:58:11.457000", "content": "
\n

I have found that a some amount of GT masks have a slight overlap between them. Have you found such cases?

\n

for example: b36f28986045
\n \"\"

\n
\n

I've seen some overlap between different classes, but missed the same class overlaps.
\nYou've given me an idea. To add the overlap as a separate class to predict.

", "votes": 0, "replies": [] } ] }, { "id": 2311412, "author_name": "RickyLu", "author_url": "", "post_date": "2023-06-21T06:46:37.387000", "content": "

Same problem here, using semantic segmentation. Still confused why confidence score affects submission score? What`s difference between 0.98,0.99 and 1.0 ? 😂

", "votes": 0, "replies": [ { "id": 2357654, "author_name": "", "author_url": "", "post_date": "2023-07-25T03:23:22.150000", "content": "

I think it is important to judge whether a model is good or not, apart from whether it can correctly distinguish categories and semantics, do a good job of instance segmentation, and whether the model’s judgment on itself is certain, because the model with the same prediction accuracy, the higher the confidence in TP and TN, it means that it has learned the actual semantics and knows how to segment images and what instances it segmented.

", "votes": 0, "replies": [ { "id": 2357795, "author_name": "RickyLu", "author_url": "", "post_date": "2023-07-25T05:59:01.290000", "content": "

Finally, most of the people give up semantic segmentation on this task😂

", "votes": 0, "replies": [] }, { "id": 2357826, "author_name": "bent1e", "author_url": "", "post_date": "2023-07-25T06:27:34.480000", "content": "

I think in the end there will be great semantic segmentation methods out there

", "votes": 0, "replies": [] }, { "id": 2357850, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2023-07-25T06:59:15.800000", "content": "

Most probably, the best would be object detection + instance segmentation models, like it was in Sartorius competition.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2310857, "author_name": "Kevin (Won June) Cho", "author_url": "", "post_date": "2023-06-20T16:49:03.357000", "content": "

Having same problem here with testing a segmentation model- it does seem like having different thresholds matter, but the score, like you said, are still really, really low for some reason, around 0.05-0.06 no matter what I try…

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2310991": "This was my first approach as well since I wanted to try something I'm familiar with and this was my first time doing instance segmentation. This approach gave me around 0.133, and I have a few assumptions regarding that:\n\n1. Confidence scores: Confidence scores are important so you need to try something that'll give you approximate confidence scores, I've tried using the mean of the pixels for each component's mask.\n\n2. Connected components: Maybe your model incorrectly bridges some of the components that should be separate, try separating them with morphological transformations, or maybe a loss function that emphasizes the separation between objects.\n\n3. The selection criteria are too strict: You have either set the pixel confidence threshold too high, or your model is missing components that should be present in the image. A few of the components that my models detect consistently are suspiciously missing from your example. Maybe you could get around that by choosing the pixels that have the highest scores in a locally instead of globally. A simple way to do that could be by choosing the pixels with the highest score among their neighbors and then selecting around them.\n\nLooking forward to seeing your results, since this approach is viable theoretically and I don't see a reason for it to give such low scores. Good luck.", "2310295": "Here you are using all the confident scores as 1.0. \nSince you are converting the mask into the instances by cv2.ConnectedComponents I dont think you will be getting the confidence scores ", "2310184": "Mhh, yeah your masks don't look crazy.\nOne thing I did see is that when setting confidence to 1.0 across the board my score was much worse than having values as predicted by the object detection model I used (YOLO). \n\nHow do you plan on dealing with confidence scores? \n\nBut I doubt this fully explains the 0.06, that's too low even for that. ", "2312064": "I have found that a some amount of GT masks have a slight overlap between them. Have you found such cases?\n\nfor example: b36f28986045\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14675197%2F3463ef1cad48bdad6b2b282552600f02%2FUntitled.png?generation=1687366119217014&alt=media)", "2310162": "Hey guys, I'm stuck with poor submission scores and not quite sure what I'm doing wrong.\nSo I'm treating this competition as semantic segmentation. I don't see much reason to use models designed to do real instance segmentation as our objects are well separated and thus the semantic segmentation masks can be easily converted into instance segmentation by using cv2.ConnectedComponents\nMy code does the following:\n1. Predict segmentation mask (See 1 test sample)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F349155%2F31663618e54051e7c251cc69097eebfd%2FTestImg.png?generation=1687243121688714&alt=media)\n2. Split segmentation mask into individual binarized masks using connected components\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F349155%2F89792db245d18adf8fa9790a1f46af56%2FMask.png?generation=1687243159107534&alt=media)\n3. Use the provided encode_binary_mask code to get individual mask encodings and merge them. Here is the sample submission string\n'0 1.0 eNrLDY8yt0u0N3Kx8TP0NTIw9EdAAygAs/0gcgZIwBCI/Q19jf3hCkBsMN/IDwRjPAEsihZJ 0 1.0 eNqLjgg0tUi3N/Ix8TMCQ0N/Q38YDYQGQOCPxkam4QCkAkJgB1ikoGYgG4XLGixOMURyLBCCPADEIRnRhgDcsC+w 0 1.0 eNqLycg1M8yzN/IzMPJLCw8wBAApugS0 0 1.0 eNoLjowys0m2N/Ix8TNChv4GUGAIZPkbwnj+hkBo4GdkYOht5mfkZuUXkR5mCAA9JhBx 0 1.0 eNqLjMsztkq1N/KDQn9Df0MYjQINoADBBglDaF8TPyMvM7+cxHhjADCrE8Y= 0 1.0 eNpLy4swM8q1N/Ex8jcAAkMQ4W/ob2DoD4YGEMrf2BcqgMYGqQdpBLFh+sEMA2RBEA1VbOhn5GcE0wxi+5l4pQakGwAAtbUgWw== 0 1.0 eNoLjs8zSTKItTfyM/My8jf2NfQ3QAeG/ghBbGx0GsbGhKbehn5hCWFGACm1GvE='\n\nAnd with this, I get 0.06-0.07. The score is above 0, so it seems to at least be working and submitting something that the kaggle server can score. My masks are not perfect, but they sure don't look like 0.06 imperfect. :) I seem to be missing something important though I cannot figure it out. Like in some segmentation competitions, you have to transpose your mask before RLE encoding it.\nThanks!", "2311412": "Same problem here, using semantic segmentation. Still confused why confidence score affects submission score? What`s difference between 0.98,0.99 and 1.0 ? 😂", "2310857": "Having same problem here with testing a segmentation model- it does seem like having different thresholds matter, but the score, like you said, are still really, really low for some reason, around 0.05-0.06 no matter what I try..." } }