\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F82351c00fc0cd7979ced374ae5190108%2FSelection_999(2454).png?generation=1688300870548277&alt=media\")
this method is quite promising. hence i am starting a new thread on it
\n
you can check a notebbook code on it
\npart1:
\nhttps://www.kaggle.com/code/hengck23/unet-instance-segmentation-onmipose-part1
\npart2:
\nhttps://www.kaggle.com/code/hengck23/unet-instance-segmentation-onmipose-part2
\npart3:
\nto be updated (learn a neutral network to do the post processing)
this method is quite promising. hence i am starting a new thread on it
\n
you can check a notebbook code on it
\npart1:
\nhttps://www.kaggle.com/code/hengck23/unet-instance-segmentation-onmipose-part1
\npart2:
\nhttps://www.kaggle.com/code/hengck23/unet-instance-segmentation-onmipose-part2
\npart3:
\nto be updated (learn a neutral network to do the post processing)
not sure about yolo (but i think so be similar), but here in unet, the predicted mask is mostly smaller than the truth mask.
\nthis explains why dilation helps?
this is results of inconsistent labelling, i think
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fe4dbc166bcbd78eb84604137addf0a12%2FSelection_999(2486).png?generation=1688339250820661&alt=media\")
Interesting, then this means we are supposed to optimize performance of bbox prediction? i.e. locate where the instance is then use a better segmentation algorithm.
", "rawMarkdown": "Interesting, then this means we are supposed to optimize performance of bbox prediction? i.e. locate where the instance is then use a better segmentation algorithm." } ] }, { "id": 2327325, "postDate": "2023-07-02T21:03:04.107Z", "content": "detect intstances is not the problem on postprocessing, but because of the metric used in this competition define proper confidence is a challenge, I was able reach only 0.325 with unet on LB
", "rawMarkdown": "detect intstances is not the problem on postprocessing, but because of the metric used in this competition define proper confidence is a challenge, I was able reach only 0.325 with unet on LB", "votes": 1, "replies": [ { "id": 2327366, "postDate": "2023-07-02T22:45:51.627Z", "content": "quite a lot of object masks are touching each other, so you definitely need to seprate them as instances.
\nyou also need a head in unet to predict iou or some way to rank or output confidence score.
\nfor example.
let ground truthbe\n\ntruth = a WxH image with =0 means pixel is background, =1 means pixel is object 1, =2 \npredict = . same format .\n\nU,C = np.unique (truth, =)\n\nduring training, you need some dynamic matching\niou_truth = zeros()\n i range(1, predict.max()+1):\n u,c = np.unique ( truth[==i], =)\n .choose max count k.\n iou = count k/ count K\n iou_truth[==i] = iou\n\n\n\niou_predict = conv_layer (feature)\nloss = F.binary_cross_entropy_with_logits(iou_predict,iou_truth)\n\n\n\na another solution is input the results of unet to some single instance segmentation network like mask-rcnn or train a SAM-like mask decoder head with e.g. instance distance transform as prompt
", "rawMarkdown": "a another solution is input the results of unet to some single instance segmentation network like mask-rcnn or train a SAM-like mask decoder head with e.g. instance distance transform as prompt" }, { "id": 2327998, "postDate": "2023-07-03T09:39:04.913Z", "content": "as an example for iou predictor
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fdc8a55261891d8056d72f30368a8e4ac%2FSelection_999(2504).png?generation=1688377027269280&alt=media\")
https://arxiv.org/pdf/2107.06278.pdf
\n\"Per-Pixel Classification is Not All You Need for Semantic Segmentation\" - Bowen Cheng
![\"https://i.ibb.co/QpGsRN2/Selection-999-2452.png\"](\"https://i.ibb.co/QpGsRN2/Selection-999-2452.png\")
Background Reading
\nFor information on onmipose, check these
code: https://github.com/kevinjohncutler/omnipose
\ndoc: https://omnipose.readthedocs.io/
\npaper: https://www.nature.com/articles/s41592-022-01639-4
\n\"Omnipose: a high-precision morphology-independent solution for bacterial cell segmentation\" - Kevin J. Cutler
early results
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F5579a7bead0ea9a3855eaf483ad8c292%2FSelection_999(2485).png?generation=1688323533434447&alt=media\")
it can detect large, small, cluttered, elongated, complex/non-straight objects.
\nBut the seresnext101 encoder seems to be not strong enough (not enough context).
i will try a strong VIT encoder tmr.
", "rawMarkdown": "early results\n\n.png?generation=1688323533434447&alt=media)\n\nit can detect large, small, cluttered, elongated, complex/non-straight objects.\nBut the seresnext101 encoder seems to be not strong enough (not enough context).\n\ni will try a strong VIT encoder tmr." } ] }, { "id": 2335893, "postDate": "2023-07-08T23:34:17.677Z", "content": "i used previous kaggle datasciencebowl 2018 winner solution to seprarte touching instance in segmentation + public note unet-plus-plus
\nhttps://www.kaggle.com/code/hidngnguyna/baseline-unet-semantic-as-instance-segmentation/notebook
\nhttps://www.kaggle.com/competitions/data-science-bowl-2018/discussion/54741
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F7d3867ab74e4357323f11f0b21aca0cb%2FSelection_999(2594).png?generation=1688859241725503&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F2b0ed74ffb465fa720ca4d69d32c2c37%2FSelection_999(2593).png?generation=1688859255439643&alt=media\")
according to datasciencebowl 2018 winner solution, you can use watershed to postprocess (to convert
\nsemantic to instance mask)
to compute score, you can use
\nconf = (semantic_prob*instance_mask).() / instance_mask.() \n\n \n\nconf = - (unsure_prob*instance_mask).() / instance_mask.() \n\n\nself.semantic = sigmoid of : background, vessel\nself.part = ... softmax of : background, seed, border\nself.unsure = ... softmax of : background, unsure, sure\ni suddenly recalled an old related kaggle competition:
\nhttps://github.com/ternaus/TernausNetV2
\nafter they separate the instances from semantic mask, they crop the features and predict iou:
\ncrop = roi(feature_map) \niou = iou_head(crop_instance, crop_feature_map)\n\nthere are sets of (roi, iou_head) for `
", "rawMarkdown": "i suddenly recalled an old related kaggle competition:\n\nhttps://github.com/ternaus/TernausNetV2\n\nafter they separate the instances from semantic mask, they crop the features and predict iou:\n\n```\ncrop = roi(instance, feature_map) #modify from torchvision rpn roi align\niou = iou_head(crop_instance, crop_feature_map)\n\nthere are sets of (roi, iou_head) for differently-sized instances\n\n````" }, { "id": 2328804, "postDate": "2023-07-03T22:57:10.533Z", "content": "results for convnext base encoder
\n(i choose convext becuase it can do self-supervised conv-MAE learning, see convnext-v2 paper)
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F0bf589cc649dc32c67da0865c4f0040b%2FSelection_999(2535).png?generation=1688424663534110&alt=media\")
i am surprise that the model is not able to floodfill waht seems to be \"simple region\" (constant intensity)
\nit seems that the issue is not context?
\nyou can see from above that i almost detect all objects (even very small ones) at the validation, but detected objects are much smaller in size.
some interesting fact about using convnext encoder:
\ninstead of heuristics for post processing, you can learn a mask decoding head to convert flow to instance mask.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F7940430ef9d10e6060ee048c4b32269b%2FSelection_999(2510).png?generation=1688381905786325&alt=media\")
i just realise \"dynamic conv kernel\" of box free instance segmentation head (e.g. solov2, YOLACT) is related to transformer and attnetion.
\nsuch segmentation head predict the \"weight of convolution kernel\" from test image (not memorised from train image) and use it extract feature for instance mask prediction.
\nisn't it:
\nquery = dynamic weight
\nkey = image feature
\ndot-product attnetion = convolution + gate
Great work！
", "rawMarkdown": "Great work！", "isDeleted": true } ], "comments": [ { "id": 2327377, "author_name": "hengck23", "author_url": "", "post_date": "2023-07-02T23:08:05.740000", "content": "not sure about yolo (but i think so be similar), but here in unet, the predicted mask is mostly smaller than the truth mask.
\nthis explains why dilation helps?
this is results of inconsistent labelling, i think
\n
Interesting, then this means we are supposed to optimize performance of bbox prediction? i.e. locate where the instance is then use a better segmentation algorithm.
", "votes": 0, "replies": [] } ] }, { "id": 2327325, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2023-07-02T21:03:04.107000", "content": "detect intstances is not the problem on postprocessing, but because of the metric used in this competition define proper confidence is a challenge, I was able reach only 0.325 with unet on LB
", "votes": 1, "replies": [ { "id": 2327366, "author_name": "hengck23", "author_url": "", "post_date": "2023-07-02T22:45:51.627000", "content": "quite a lot of object masks are touching each other, so you definitely need to seprate them as instances.
\nyou also need a head in unet to predict iou or some way to rank or output confidence score.
\nfor example.
let ground truthbe\n\ntruth = a WxH image with =0 means pixel is background, =1 means pixel is object 1, =2 \npredict = . same format .\n\nU,C = np.unique (truth, =)\n\nduring training, you need some dynamic matching\niou_truth = zeros()\n i range(1, predict.max()+1):\n u,c = np.unique ( truth[==i], =)\n .choose max count k.\n iou = count k/ count K\n iou_truth[==i] = iou\n\n\n\niou_predict = conv_layer (feature)\nloss = F.binary_cross_entropy_with_logits(iou_predict,iou_truth)\n\n\n\na another solution is input the results of unet to some single instance segmentation network like mask-rcnn or train a SAM-like mask decoder head with e.g. instance distance transform as prompt
", "votes": 0, "replies": [] }, { "id": 2327998, "author_name": "hengck23", "author_url": "", "post_date": "2023-07-03T09:39:04.913000", "content": "as an example for iou predictor
\n
https://arxiv.org/pdf/2107.06278.pdf
\n\"Per-Pixel Classification is Not All You Need for Semantic Segmentation\" - Bowen Cheng
Background Reading
\nFor information on onmipose, check these
code: https://github.com/kevinjohncutler/omnipose
\ndoc: https://omnipose.readthedocs.io/
\npaper: https://www.nature.com/articles/s41592-022-01639-4
\n\"Omnipose: a high-precision morphology-independent solution for bacterial cell segmentation\" - Kevin J. Cutler
early results
\n
it can detect large, small, cluttered, elongated, complex/non-straight objects.
\nBut the seresnext101 encoder seems to be not strong enough (not enough context).
i will try a strong VIT encoder tmr.
", "votes": 0, "replies": [] } ] }, { "id": 2335893, "author_name": "hengck23", "author_url": "", "post_date": "2023-07-08T23:34:17.677000", "content": "i used previous kaggle datasciencebowl 2018 winner solution to seprarte touching instance in segmentation + public note unet-plus-plus
\nhttps://www.kaggle.com/code/hidngnguyna/baseline-unet-semantic-as-instance-segmentation/notebook
\nhttps://www.kaggle.com/competitions/data-science-bowl-2018/discussion/54741
according to datasciencebowl 2018 winner solution, you can use watershed to postprocess (to convert
\nsemantic to instance mask)
to compute score, you can use
\nconf = (semantic_prob*instance_mask).() / instance_mask.() \n\n \n\nconf = - (unsure_prob*instance_mask).() / instance_mask.() \n\n\nself.semantic = sigmoid of : background, vessel\nself.part = ... softmax of : background, seed, border\nself.unsure = ... softmax of : background, unsure, sure\ni suddenly recalled an old related kaggle competition:
\nhttps://github.com/ternaus/TernausNetV2
\nafter they separate the instances from semantic mask, they crop the features and predict iou:
\ncrop = roi(feature_map) \niou = iou_head(crop_instance, crop_feature_map)\n\nthere are sets of (roi, iou_head) for `
", "votes": 0, "replies": [] }, { "id": 2328804, "author_name": "hengck23", "author_url": "", "post_date": "2023-07-03T22:57:10.533000", "content": "results for convnext base encoder
\n(i choose convext becuase it can do self-supervised conv-MAE learning, see convnext-v2 paper)
i am surprise that the model is not able to floodfill waht seems to be \"simple region\" (constant intensity)
\nit seems that the issue is not context?
\nyou can see from above that i almost detect all objects (even very small ones) at the validation, but detected objects are much smaller in size.
some interesting fact about using convnext encoder:
\ninstead of heuristics for post processing, you can learn a mask decoding head to convert flow to instance mask.
\n
i just realise \"dynamic conv kernel\" of box free instance segmentation head (e.g. solov2, YOLACT) is related to transformer and attnetion.
\nsuch segmentation head predict the \"weight of convolution kernel\" from test image (not memorised from train image) and use it extract feature for instance mask prediction.
\nisn't it:
\nquery = dynamic weight
\nkey = image feature
\ndot-product attnetion = convolution + gate
Great work！
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2326806": "this method is quite promising. hence i am starting a new thread on it\n .png?generation=1688300870548277&alt=media)\n\nyou can check a notebbook code on it\npart1:\nhttps://www.kaggle.com/code/hengck23/unet-instance-segmentation-onmipose-part1\npart2:\nhttps://www.kaggle.com/code/hengck23/unet-instance-segmentation-onmipose-part2\npart3:\nto be updated (learn a neutral network to do the post processing)", "2327377": "not sure about yolo (but i think so be similar), but here in unet, the predicted mask is mostly smaller than the truth mask.\nthis explains why dilation helps?\n\nthis is results of inconsistent labelling, i think\n\n.png?generation=1688339250820661&alt=media)", "2327325": "detect intstances is not the problem on postprocessing, but because of the metric used in this competition define proper confidence is a challenge, I was able reach only 0.325 with unet on LB", "2326916": "\n\nBackground Reading\nFor information on onmipose, check these\n\ncode: https://github.com/kevinjohncutler/omnipose\ndoc: https://omnipose.readthedocs.io/\npaper: https://www.nature.com/articles/s41592-022-01639-4\n\"Omnipose: a high-precision morphology-independent solution for bacterial cell segmentation\" - Kevin J. Cutler", "2335893": "i used previous kaggle datasciencebowl 2018 winner solution to seprarte touching instance in segmentation + public note unet-plus-plus\n\nhttps://www.kaggle.com/code/hidngnguyna/baseline-unet-semantic-as-instance-segmentation/notebook\nhttps://www.kaggle.com/competitions/data-science-bowl-2018/discussion/54741\n\n.png?generation=1688859241725503&alt=media)\n\n.png?generation=1688859255439643&alt=media)\n\naccording to datasciencebowl 2018 winner solution, you can use watershed to postprocess (to convert \nsemantic to instance mask)\n\n\n---\nto compute score, you can use\n\n```\nconf = (semantic_prob*instance_mask).sum() / instance_mask.sum() \n\nor \n\nconf = 1- (unsure_prob*instance_mask).sum() / instance_mask.sum() \n\n#assume you have the following segmentation head\nself.semantic = sigmoid of 1 class: background, vessel\nself.part = ... softmax of 3 class: background, seed, border\nself.unsure = ... softmax of 3 class: background, unsure, sure\n\n```", "2328817": "i suddenly recalled an old related kaggle competition:\n\nhttps://github.com/ternaus/TernausNetV2\n\nafter they separate the instances from semantic mask, they crop the features and predict iou:\n\n```\ncrop = roi(instance, feature_map) #modify from torchvision rpn roi align\niou = iou_head(crop_instance, crop_feature_map)\n\nthere are sets of (roi, iou_head) for differently-sized instances\n\n````", "2328804": "results for convnext base encoder\n(i choose convext becuase it can do self-supervised conv-MAE learning, see convnext-v2 paper)\n\n.png?generation=1688424663534110&alt=media)\n\ni am surprise that the model is not able to floodfill waht seems to be \"simple region\" (constant intensity)\n\nit seems that the issue is not context?\nyou can see from above that i almost detect all objects (even very small ones) at the validation, but detected objects are much smaller in size.\n\nsome interesting fact about using convnext encoder:\n- you can actually freeze it entirely and just train the decoder (blazing fast). then finetune end-to-end at last the end with small learning rate\n- typically i need 400 to 500 epoches (even for seresnext101)\n- results are pretty good and would even be better if i can solve \"unknown missing floodfill issue\"", "2328094": "instead of heuristics for post processing, you can learn a mask decoding head to convert flow to instance mask.\n\n.png?generation=1688381905786325&alt=media)", "2357686": "Great work！" } }