{ "id": 291371, "title": "The clean astro mask are here!!!", "url": "/competitions/sartorius-cell-instance-segmentation/discussion/291371", "author_name": "hengck23", "post_date": "2021-11-29T01:26:53.105000", "votes": 125, "comment_count": 22, "views": 0, "content": "

Please download from the public dataset i have created:
\nhttps://www.kaggle.com/hengck23/clean-astro-mask

\n

I have filled up the holes and removed the extra wrong horizontal filling.
\nwith the clean mask, you can now train with augmentation

\n

As an example:
\nBlue: original mask
\nred: clean mask
\n(pink is the overlay of the two)

\n
from skimage import measure\n\ndef fill_hole(m):\n    filled = m.copy()\n    pad = np.pad(m, 4)\n    lb = measure.label(pad < 0.5, background=0, connectivity=1)\n    u, cc = np.unique(lb, return_counts=True)\n    if len(u) > 2:\n        #print(u, cc)\n        lb = lb[4:-4, 4:-4]\n        for uu in u[2:]:\n            filled[lb == uu] = 1\n\n    return filled\n\n\n\nhow to use?\n1. decode instance from the csv file. e.g. call this m\n2. fill the hole: m= fill_hole(m)\n3. read the red channel,e.g. r\n4. recovered clean instance mask = r & m (bitwise and operation)\n
\n

it is better to pretrain with clean mask + augmentation.
\nthen you can fine-tune with dirty mask without augmentation to learn the mask error.
\n(or you can create dirty mask version of the augmentation, e.g. make the bottom part horizontal by heuristics or via a learned model, i.e. a clean to dirty mask model)

\n

\"729cc4463650\"

", "messages": [ { "id": 1598907, "postDate": "2021-11-29T01:26:53.107Z", "content": "

Please download from the public dataset i have created:
\nhttps://www.kaggle.com/hengck23/clean-astro-mask

\n

I have filled up the holes and removed the extra wrong horizontal filling.
\nwith the clean mask, you can now train with augmentation

\n

As an example:
\nBlue: original mask
\nred: clean mask
\n(pink is the overlay of the two)

\n
from skimage import measure\n\ndef fill_hole(m):\n    filled = m.copy()\n    pad = np.pad(m, 4)\n    lb = measure.label(pad < 0.5, background=0, connectivity=1)\n    u, cc = np.unique(lb, return_counts=True)\n    if len(u) > 2:\n        #print(u, cc)\n        lb = lb[4:-4, 4:-4]\n        for uu in u[2:]:\n            filled[lb == uu] = 1\n\n    return filled\n\n\n\nhow to use?\n1. decode instance from the csv file. e.g. call this m\n2. fill the hole: m= fill_hole(m)\n3. read the red channel,e.g. r\n4. recovered clean instance mask = r & m (bitwise and operation)\n
\n

it is better to pretrain with clean mask + augmentation.
\nthen you can fine-tune with dirty mask without augmentation to learn the mask error.
\n(or you can create dirty mask version of the augmentation, e.g. make the bottom part horizontal by heuristics or via a learned model, i.e. a clean to dirty mask model)

\n

\"729cc4463650\"

", "rawMarkdown": "Please download from the public dataset i have created: \nhttps://www.kaggle.com/hengck23/clean-astro-mask\n\nI have filled up the holes and removed the extra wrong horizontal filling.\nwith the clean mask, you can now train with augmentation\n\nAs an example:\nBlue: original mask\nred: clean mask\n(pink is the overlay of the two)\n\n```\n\nfrom skimage import measure\n\ndef fill_hole(m):\n filled = m.copy()\n pad = np.pad(m, 4)\n lb = measure.label(pad < 0.5, background=0, connectivity=1)\n u, cc = np.unique(lb, return_counts=True)\n if len(u) > 2:\n #print(u, cc)\n lb = lb[4:-4, 4:-4]\n for uu in u[2:]:\n filled[lb == uu] = 1\n\n return filled\n\n\n\nhow to use?\n1. decode instance from the csv file. e.g. call this m\n2. fill the hole: m= fill_hole(m)\n3. read the red channel,e.g. r\n4. recovered clean instance mask = r & m (bitwise and operation)\n\n```\n\nit is better to pretrain with clean mask + augmentation.\nthen you can fine-tune with dirty mask without augmentation to learn the mask error.\n(or you can create dirty mask version of the augmentation, e.g. make the bottom part horizontal by heuristics or via a learned model, i.e. a clean to dirty mask model)\n\n\"729cc4463650\"", "votes": 123 }, { "id": 1599504, "postDate": "2021-11-29T13:45:38.183Z", "content": "

[Edit] Disregard this. Turns out it was some image conversion issue and I was getting pixels with different values than just 0 and 255. Changing the formula to ((r>0) & m) fixes this

\n

Are you also changing mask boundries in some way? When I intersect a filled mask with your masks there are always some pixels different that don't look like neither holes, nor artefacts.
\n\"\"

", "rawMarkdown": "**[Edit] Disregard this. Turns out it was some image conversion issue and I was getting pixels with different values than just 0 and 255. Changing the formula to ((r>0) & m) fixes this**\n\nAre you also changing mask boundries in some way? When I intersect a filled mask with your masks there are always some pixels different that don't look like neither holes, nor artefacts.\n![](https://raw.githubusercontent.com/slawekslex/random/main/masks.png)", "votes": 3, "replies": [ { "id": 1599560, "postDate": "2021-11-29T14:36:08.620Z", "content": "

These masks are generated by a learned model and manual correction.
\nIt is possible that some parts are modified.

\n

this is how I create the mask:

\n
    \n
  1. hand-corrected about 20 mask images to train a semantic segmentation model (just cell vs background)
    2. \n
  2. apply the model to all images and make blue-red-pink prediction image
    3. \n
  3. open the prediction image in GIMP. Flood-fill the wrong blue horizontal error part (or other part) using the bucket tool with black color(background).
    4. \n
  4. the error-corrected blue mask becomes the new red mask in this dataset
    5. \n
", "rawMarkdown": "These masks are generated by a learned model and manual correction.\nIt is possible that some parts are modified.\n\nthis is how I create the mask:\n\n1. hand-corrected about 20 mask images to train a semantic segmentation model (just cell vs background) \n2. apply the model to all images and make blue-red-pink prediction image\n3. open the prediction image in GIMP. Flood-fill the wrong blue horizontal error part (or other part) using the bucket tool with black color(background).\n4. the error-corrected blue mask becomes the new red mask in this dataset", "votes": 6 }, { "id": 1599569, "postDate": "2021-11-29T14:43:36.347Z", "content": "

I appreciate your hard work on this. After I've fixed my code they look generally ok.

", "rawMarkdown": "I appreciate your hard work on this. After I've fixed my code they look generally ok.", "votes": 1 } ] }, { "id": 1599932, "postDate": "2021-11-30T00:35:12.130Z", "content": "

There is a simpler way to apply the corrected mask?

\n

Original image uses original wrong mask as ground truth.

\n

Augmented images uses clean mask as ground truth.

\n

If someone can post a code to convert the clean mask into polygon and reproduce the error, it would be great. The error is due to wrong usage fill polygon function in maybe opencv ( wrong concavity ?)

", "rawMarkdown": "There is a simpler way to apply the corrected mask?\n\nOriginal image uses original wrong mask as ground truth.\n\nAugmented images uses clean mask as ground truth.\n\nIf someone can post a code to convert the clean mask into polygon and reproduce the error, it would be great. The error is due to wrong usage fill polygon function in maybe opencv ( wrong concavity ?)", "votes": 4, "replies": [ { "id": 1599985, "postDate": "2021-11-30T02:43:50.420Z", "content": "

https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/291311

\n

Doesn't cv2.findContours do this more or less exactly? I assume it is also likely what the hosts used..

", "rawMarkdown": "https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/291311\n\nDoesn't `cv2.findContours` do this more or less exactly? I assume it is also likely what the hosts used.." }, { "id": 1600081, "postDate": "2021-11-30T05:25:18Z", "content": "

we need a way to make error mask so that we can learn the error (the private test is assumed to have the same error)

", "rawMarkdown": "we need a way to make error mask so that we can learn the error (the private test is assumed to have the same error)", "votes": 1 }, { "id": 1600357, "postDate": "2021-11-30T10:27:50.103Z", "content": "

I've tried that, see here: https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/291639

", "rawMarkdown": "I've tried that, see here: https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/291639", "votes": 2 } ] }, { "id": 1599352, "postDate": "2021-11-29T11:14:49.167Z", "content": "

@hengck23 , thanks a lot for your efforts! One question: By taking r & m, won't you loose the holes you filled?

", "rawMarkdown": "@hengck23 , thanks a lot for your efforts! One question: By taking `r & m`, won't you loose the holes you filled?", "votes": 1, "replies": [ { "id": 1599363, "postDate": "2021-11-29T11:22:28.237Z", "content": "

you are correct. i make the correction. you need to fill the hole first before \"and operation\"

", "rawMarkdown": "you are correct. i make the correction. you need to fill the hole first before \"and operation\"", "votes": 3 }, { "id": 1611868, "postDate": "2021-12-08T10:43:28.607Z", "content": "

@hengck23
\nwhen i try to fill in the broken part of first instance
\nplt.imshow( cv2.bitwise_and(fill_hole(masks), clean[:,:,0].astype('uint8')))

\n

there is no change ,m i making a mistake any where ?
\nWhile in one of mask instance where there was big Triangle filled blue was lost after bitwise And.
\nI mean what should be the difference between red portion in full mask,blue portion in full mask and that corresponding portion in mask instance before and after And operation,

", "rawMarkdown": "@hengck23 \nwhen i try to fill in the broken part of first instance\nplt.imshow( cv2.bitwise_and(fill_hole(masks), clean[:,:,0].astype('uint8')))\n\nthere is no change ,m i making a mistake any where ?\nWhile in one of mask instance where there was big Triangle filled blue was lost after bitwise And.\nI mean what should be the difference between red portion in full mask,blue portion in full mask and that corresponding portion in mask instance before and after And operation," } ] }, { "id": 1598976, "postDate": "2021-11-29T03:39:18.377Z", "content": "
\n

it is better to pretrain with clean mask + augmentation.

\n
\n

Could you share how much improvement it is

", "rawMarkdown": "> it is better to pretrain with clean mask + augmentation.\n\nCould you share how much improvement it is", "votes": 1 }, { "id": 1619472, "postDate": "2021-12-16T01:36:13.127Z", "content": "

Hi, I have a question, Is any channel OK? Won't get different clean_masks?
\nread the red channel,e.g. r
\nrecovered clean instance mask = r & m (bitwise and operation)

", "rawMarkdown": "Hi, I have a question, Is any channel OK? Won't get different clean_masks?\n`read the red channel,e.g. r`\n`recovered clean instance mask = r & m (bitwise and operation)`", "votes": -1, "replies": [ { "id": 1620014, "postDate": "2021-12-16T10:37:31.077Z", "content": "

I've Tried your \"filled_hole\", but what I got are not clean_masks.
\nOr cv2 dosen't work for this job?

", "rawMarkdown": "I've Tried your \"filled_hole\", but what I got are not clean_masks.\nOr cv2 dosen't work for this job? " } ] }, { "id": 1621181, "postDate": "2021-12-17T14:06:24.640Z", "content": "

Thank you for your sharing! This is very useful for freshman in the competition.

", "rawMarkdown": "Thank you for your sharing! This is very useful for freshman in the competition." }, { "id": 1600677, "postDate": "2021-11-30T16:10:37.740Z", "content": "

Awesome! I've never thought to approach data cleaning in this way

", "rawMarkdown": "Awesome! I've never thought to approach data cleaning in this way" }, { "id": 1600132, "postDate": "2021-11-30T06:28:41.547Z", "content": "

this is awesome!

", "rawMarkdown": "this is awesome!" }, { "id": 1599361, "postDate": "2021-11-29T11:21:33.247Z", "content": "

Thank you for sharing your work. May I ask how many such masks you filled ? I used a public kernel and found ~507 bad masks in the training set.

", "rawMarkdown": "Thank you for sharing your work. May I ask how many such masks you filled ? I used a public kernel and found ~507 bad masks in the training set. " }, { "id": 1600208, "postDate": "2021-11-30T07:49:21.787Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1615131, "postDate": "2021-12-11T21:37:48.317Z", "content": "

thanks a lot for your efforts!

", "rawMarkdown": " thanks a lot for your efforts! " }, { "id": 1606767, "postDate": "2021-12-05T08:41:43.523Z", "content": "

Thanks for sharing this one

", "rawMarkdown": "Thanks for sharing this one" }, { "id": 1604973, "postDate": "2021-12-03T22:26:52.843Z", "content": "

Thanks a lot!

", "rawMarkdown": "Thanks a lot!" }, { "id": 1598963, "postDate": "2021-11-29T03:15:56.853Z", "content": "

Amazing!!! Thank you:)

", "rawMarkdown": "Amazing!!! Thank you:)" } ], "comments": [ { "id": 1599504, "author_name": "Slawek Biel", "author_url": "", "post_date": "2021-11-29T13:45:38.183000", "content": "

[Edit] Disregard this. Turns out it was some image conversion issue and I was getting pixels with different values than just 0 and 255. Changing the formula to ((r>0) & m) fixes this

\n

Are you also changing mask boundries in some way? When I intersect a filled mask with your masks there are always some pixels different that don't look like neither holes, nor artefacts.
\n\"\"

", "votes": 3, "replies": [ { "id": 1599560, "author_name": "hengck23", "author_url": "", "post_date": "2021-11-29T14:36:08.620000", "content": "

These masks are generated by a learned model and manual correction.
\nIt is possible that some parts are modified.

\n

this is how I create the mask:

\n
    \n
  1. hand-corrected about 20 mask images to train a semantic segmentation model (just cell vs background)
    2. \n
  2. apply the model to all images and make blue-red-pink prediction image
    3. \n
  3. open the prediction image in GIMP. Flood-fill the wrong blue horizontal error part (or other part) using the bucket tool with black color(background).
    4. \n
  4. the error-corrected blue mask becomes the new red mask in this dataset
    5. \n
", "votes": 6, "replies": [] }, { "id": 1599569, "author_name": "Slawek Biel", "author_url": "", "post_date": "2021-11-29T14:43:36.347000", "content": "

I appreciate your hard work on this. After I've fixed my code they look generally ok.

", "votes": 1, "replies": [] } ] }, { "id": 1599932, "author_name": "hengck23", "author_url": "", "post_date": "2021-11-30T00:35:12.130000", "content": "

There is a simpler way to apply the corrected mask?

\n

Original image uses original wrong mask as ground truth.

\n

Augmented images uses clean mask as ground truth.

\n

If someone can post a code to convert the clean mask into polygon and reproduce the error, it would be great. The error is due to wrong usage fill polygon function in maybe opencv ( wrong concavity ?)

", "votes": 4, "replies": [ { "id": 1599985, "author_name": "عثمان", "author_url": "", "post_date": "2021-11-30T02:43:50.420000", "content": "

https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/291311

\n

Doesn't cv2.findContours do this more or less exactly? I assume it is also likely what the hosts used..

", "votes": 0, "replies": [] }, { "id": 1600081, "author_name": "hengck23", "author_url": "", "post_date": "2021-11-30T05:25:18", "content": "

we need a way to make error mask so that we can learn the error (the private test is assumed to have the same error)

", "votes": 1, "replies": [] }, { "id": 1600357, "author_name": "Slawek Biel", "author_url": "", "post_date": "2021-11-30T10:27:50.103000", "content": "

I've tried that, see here: https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/291639

", "votes": 2, "replies": [] } ] }, { "id": 1599352, "author_name": "omallo", "author_url": "", "post_date": "2021-11-29T11:14:49.167000", "content": "

@hengck23 , thanks a lot for your efforts! One question: By taking r & m, won't you loose the holes you filled?

", "votes": 1, "replies": [ { "id": 1599363, "author_name": "hengck23", "author_url": "", "post_date": "2021-11-29T11:22:28.237000", "content": "

you are correct. i make the correction. you need to fill the hole first before \"and operation\"

", "votes": 3, "replies": [] }, { "id": 1611868, "author_name": "Jaideep", "author_url": "", "post_date": "2021-12-08T10:43:28.607000", "content": "

@hengck23
\nwhen i try to fill in the broken part of first instance
\nplt.imshow( cv2.bitwise_and(fill_hole(masks), clean[:,:,0].astype('uint8')))

\n

there is no change ,m i making a mistake any where ?
\nWhile in one of mask instance where there was big Triangle filled blue was lost after bitwise And.
\nI mean what should be the difference between red portion in full mask,blue portion in full mask and that corresponding portion in mask instance before and after And operation,

", "votes": 0, "replies": [] } ] }, { "id": 1598976, "author_name": "Phat Tran", "author_url": "", "post_date": "2021-11-29T03:39:18.377000", "content": "
\n

it is better to pretrain with clean mask + augmentation.

\n
\n

Could you share how much improvement it is

", "votes": 1, "replies": [] }, { "id": 1619472, "author_name": "Forte Lean", "author_url": "", "post_date": "2021-12-16T01:36:13.127000", "content": "

Hi, I have a question, Is any channel OK? Won't get different clean_masks?
\nread the red channel,e.g. r
\nrecovered clean instance mask = r & m (bitwise and operation)

", "votes": -1, "replies": [ { "id": 1620014, "author_name": "Forte Lean", "author_url": "", "post_date": "2021-12-16T10:37:31.077000", "content": "

I've Tried your \"filled_hole\", but what I got are not clean_masks.
\nOr cv2 dosen't work for this job?

", "votes": 0, "replies": [] } ] }, { "id": 1621181, "author_name": "mac xiao", "author_url": "", "post_date": "2021-12-17T14:06:24.640000", "content": "

Thank you for your sharing! This is very useful for freshman in the competition.

", "votes": 0, "replies": [] }, { "id": 1600677, "author_name": "Linas Kondrackis", "author_url": "", "post_date": "2021-11-30T16:10:37.740000", "content": "

Awesome! I've never thought to approach data cleaning in this way

", "votes": 0, "replies": [] }, { "id": 1600132, "author_name": "Weikoi", "author_url": "", "post_date": "2021-11-30T06:28:41.547000", "content": "

this is awesome!

", "votes": 0, "replies": [] }, { "id": 1599361, "author_name": "RB", "author_url": "", "post_date": "2021-11-29T11:21:33.247000", "content": "

Thank you for sharing your work. May I ask how many such masks you filled ? I used a public kernel and found ~507 bad masks in the training set.

", "votes": 0, "replies": [] }, { "id": 1600208, "author_name": "", "author_url": "", "post_date": "2021-11-30T07:49:21.787000", "content": "", "votes": 0, "replies": [] }, { "id": 1615131, "author_name": "Vipin Kumar", "author_url": "", "post_date": "2021-12-11T21:37:48.317000", "content": "

thanks a lot for your efforts!

", "votes": 0, "replies": [] }, { "id": 1606767, "author_name": "KAZI SHAMIM SHAHAREAR ISLAM", "author_url": "", "post_date": "2021-12-05T08:41:43.523000", "content": "

Thanks for sharing this one

", "votes": 0, "replies": [] }, { "id": 1604973, "author_name": "Michael005", "author_url": "", "post_date": "2021-12-03T22:26:52.843000", "content": "

Thanks a lot!

", "votes": 0, "replies": [] }, { "id": 1598963, "author_name": "Mintwater", "author_url": "", "post_date": "2021-11-29T03:15:56.853000", "content": "

Amazing!!! Thank you:)

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1598907": "Please download from the public dataset i have created: \nhttps://www.kaggle.com/hengck23/clean-astro-mask\n\nI have filled up the holes and removed the extra wrong horizontal filling.\nwith the clean mask, you can now train with augmentation\n\nAs an example:\nBlue: original mask\nred: clean mask\n(pink is the overlay of the two)\n\n```\n\nfrom skimage import measure\n\ndef fill_hole(m):\n filled = m.copy()\n pad = np.pad(m, 4)\n lb = measure.label(pad < 0.5, background=0, connectivity=1)\n u, cc = np.unique(lb, return_counts=True)\n if len(u) > 2:\n #print(u, cc)\n lb = lb[4:-4, 4:-4]\n for uu in u[2:]:\n filled[lb == uu] = 1\n\n return filled\n\n\n\nhow to use?\n1. decode instance from the csv file. e.g. call this m\n2. fill the hole: m= fill_hole(m)\n3. read the red channel,e.g. r\n4. recovered clean instance mask = r & m (bitwise and operation)\n\n```\n\nit is better to pretrain with clean mask + augmentation.\nthen you can fine-tune with dirty mask without augmentation to learn the mask error.\n(or you can create dirty mask version of the augmentation, e.g. make the bottom part horizontal by heuristics or via a learned model, i.e. a clean to dirty mask model)\n\n\"729cc4463650\"", "1599504": "**[Edit] Disregard this. Turns out it was some image conversion issue and I was getting pixels with different values than just 0 and 255. Changing the formula to ((r>0) & m) fixes this**\n\nAre you also changing mask boundries in some way? When I intersect a filled mask with your masks there are always some pixels different that don't look like neither holes, nor artefacts.\n![](https://raw.githubusercontent.com/slawekslex/random/main/masks.png)", "1599932": "There is a simpler way to apply the corrected mask?\n\nOriginal image uses original wrong mask as ground truth.\n\nAugmented images uses clean mask as ground truth.\n\nIf someone can post a code to convert the clean mask into polygon and reproduce the error, it would be great. The error is due to wrong usage fill polygon function in maybe opencv ( wrong concavity ?)", "1599352": "@hengck23 , thanks a lot for your efforts! One question: By taking `r & m`, won't you loose the holes you filled?", "1598976": "> it is better to pretrain with clean mask + augmentation.\n\nCould you share how much improvement it is", "1619472": "Hi, I have a question, Is any channel OK? Won't get different clean_masks?\n`read the red channel,e.g. r`\n`recovered clean instance mask = r & m (bitwise and operation)`", "1621181": "Thank you for your sharing! This is very useful for freshman in the competition.", "1600677": "Awesome! I've never thought to approach data cleaning in this way", "1600132": "this is awesome!", "1599361": "Thank you for sharing your work. May I ask how many such masks you filled ? I used a public kernel and found ~507 bad masks in the training set. ", "1600208": "", "1615131": " thanks a lot for your efforts! ", "1606767": "Thanks for sharing this one", "1604973": "Thanks a lot!", "1598963": "Amazing!!! Thank you:)" } }