![\"score\"](\"https://pbs.twimg.com/media/F2avc0eaEAEmdvL.jpg\")
\nDespite achieving 9th place in the public leaderboard, I experienced a frightening shakeup, ending up at 295th place in the private leaderboard. However, many of the techniques I used proved effective in the private leaderboard, enabling me to achieve a score of 0.575 (equivalent to 4th place) by making just one modification to the 295th place code. Below, I will share the methods used and my reflections.
\nI primarily utilized \"mmdet\" and employed ensemble learning with numerous models. I had hoped to use YOLO v8 and v7 as well, but the host did not disclose their availability, so I had to abandon the idea.
\nFor the ensemble, I made modifications to the \"WBF\" (Weighted Box Fusion) to use it with masks. Additionally, the major innovation was not limited to instance segmentation models but also involved using object detection models. I performed WBF based on bounding boxes and made mask predictions solely using the instance segmentation models. I used the object detection models only to calculate confidence scores. As you can see from the predictions, the models made far more predictions than needed. Given the nature of the evaluation metric, mAP, it was evident that properly aligning confidence scores could significantly improve the overall score.
\nThe following is a list of the models used:
\nInstance Segmentation Models:
\nObject Detection Models:
\nI combined the outputs of the mask models using addition during post-processing. This resulted in improvements in ds1 (average IoU) and the public cross-validation (CV) score. However, as many of you may have noticed, when applying post-processing, the score on the private dataset degrades significantly. This indicates that the ds1 and private datasets have entirely different annotation tendencies. The annotations provided for ds1 were quite rough, and the dilation technique seemed effective due to this. One of the hosts mentioned that annotations were done in the same manner for both public and private datasets, but this information turned out to be inaccurate.
\nAnticipating that the annotations for the private dataset might be more substantial, I submitted two versions: one with post-processing and another without. However, as mentioned earlier, during the ensemble mask generation, I was merely adding the masks instead of using averaging or voting. This method worked well for ds1 and the public dataset. On the other hand, I should have easily foreseen that if the private dataset had accurate and precise annotations, this approach could significantly reduce the score (which I overlooked).
\nTherefore, instead of adding the masks, I changed to using voting. This simple change dramatically improved the score from 0.397 to 0.575.
\nOne of the points I regret is being content with just having two submissions: one with post-processing and one without. Since I could have diversified the risks, I should have prepared submissions that could adapt to both small and accurate annotation masks.
", "messages": [ { "id": "2368561", "postDate": "08/01/2023 08:34:16", "content": "Despite achieving 9th place in the public leaderboard, I experienced a frightening shakeup, ending up at 295th place in the private leaderboard. However, many of the techniques I used proved effective in the private leaderboard, enabling me to achieve a score of 0.575 (equivalent to 4th place) by making just one modification to the 295th place code. Below, I will share the methods used and my reflections.
\nI primarily utilized \"mmdet\" and employed ensemble learning with numerous models. I had hoped to use YOLO v8 and v7 as well, but the host did not disclose their availability, so I had to abandon the idea.
\nFor the ensemble, I made modifications to the \"WBF\" (Weighted Box Fusion) to use it with masks. Additionally, the major innovation was not limited to instance segmentation models but also involved using object detection models. I performed WBF based on bounding boxes and made mask predictions solely using the instance segmentation models. I used the object detection models only to calculate confidence scores. As you can see from the predictions, the models made far more predictions than needed. Given the nature of the evaluation metric, mAP, it was evident that properly aligning confidence scores could significantly improve the overall score.
\nThe following is a list of the models used:
\nInstance Segmentation Models:
\nObject Detection Models:
\nI combined the outputs of the mask models using addition during post-processing. This resulted in improvements in ds1 (average IoU) and the public cross-validation (CV) score. However, as many of you may have noticed, when applying post-processing, the score on the private dataset degrades significantly. This indicates that the ds1 and private datasets have entirely different annotation tendencies. The annotations provided for ds1 were quite rough, and the dilation technique seemed effective due to this. One of the hosts mentioned that annotations were done in the same manner for both public and private datasets, but this information turned out to be inaccurate.
\nAnticipating that the annotations for the private dataset might be more substantial, I submitted two versions: one with post-processing and another without. However, as mentioned earlier, during the ensemble mask generation, I was merely adding the masks instead of using averaging or voting. This method worked well for ds1 and the public dataset. On the other hand, I should have easily foreseen that if the private dataset had accurate and precise annotations, this approach could significantly reduce the score (which I overlooked).
\nTherefore, instead of adding the masks, I changed to using voting. This simple change dramatically improved the score from 0.397 to 0.575.
\nOne of the points I regret is being content with just having two submissions: one with post-processing and one without. Since I could have diversified the risks, I should have prepared submissions that could adapt to both small and accurate annotation masks.
", "rawMarkdown": "Despite achieving 9th place in the public leaderboard, I experienced a frightening shakeup, ending up at 295th place in the private leaderboard. However, many of the techniques I used proved effective in the private leaderboard, enabling me to achieve a score of 0.575 (equivalent to 4th place) by making just one modification to the 295th place code. Below, I will share the methods used and my reflections.\n\n### Method Overview:\nI primarily utilized \"mmdet\" and employed ensemble learning with numerous models. I had hoped to use YOLO v8 and v7 as well, but the host did not disclose their availability, so I had to abandon the idea.\n\nFor the ensemble, I made modifications to the \"WBF\" (Weighted Box Fusion) to use it with masks. Additionally, the major innovation was not limited to instance segmentation models but also involved using object detection models. I performed WBF based on bounding boxes and made mask predictions solely using the instance segmentation models. I used the object detection models only to calculate confidence scores. As you can see from the predictions, the models made far more predictions than needed. Given the nature of the evaluation metric, mAP, it was evident that properly aligning confidence scores could significantly improve the overall score.\n\n### Models\nThe following is a list of the models used:\n\nInstance Segmentation Models:\n* Mask2Former\n* MaskRCNN + various ConvNext v1 and v2\n* Cascade RCNN + various ConvNext v1 and v2\n* MaskRCNN + Swin Transformer\nOther: MaskDino doesn’t work well.\n\nObject Detection Models:\n* DINO\n* Mask2Former (used only for object detection part)\n* Diffusion Det\n* ViT Det\nOther: yolox doesn’t work well\n\n### Major Flaw:\nI combined the outputs of the mask models using addition during post-processing. This resulted in improvements in ds1 (average IoU) and the public cross-validation (CV) score. However, as many of you may have noticed, when applying post-processing, the score on the private dataset degrades significantly. This indicates that the ds1 and private datasets have entirely different annotation tendencies. The annotations provided for ds1 were quite rough, and the dilation technique seemed effective due to this. One of the hosts mentioned that annotations were done in the same manner for both public and private datasets, but this information turned out to be inaccurate.\n\n### Single modification to achieve private 0.575:\nAnticipating that the annotations for the private dataset might be more substantial, I submitted two versions: one with post-processing and another without. However, as mentioned earlier, during the ensemble mask generation, I was merely adding the masks instead of using averaging or voting. This method worked well for ds1 and the public dataset. On the other hand, I should have easily foreseen that if the private dataset had accurate and precise annotations, this approach could significantly reduce the score (which I overlooked).\nTherefore, instead of adding the masks, I changed to using voting. This simple change dramatically improved the score from 0.397 to 0.575.\n\n\nthanks for sharing!!
\nI could not get a good local cv with mask2former. can you tell me what config you used to learn mask2former?
Of course it's OK!
\nHowever, this was written by me when I was a mmdetection beginner, so there are many mistakes. For example, the size of the inference is improperly 1333x800 because it is taken from the base, it should be 1024x1024.
\nThe scheduler is also unchanged.
However, the basic setting is this one, so I think it will give some good results!
\nhttps://www.kaggle.com/code/yosukeyama/hubmap23-mmdet-train-mask2former-1024/notebook
\nEDIT: I have checked past scores. Using this training note, even a single-fold and improperly size inference can earn a silver medal.
", "rawMarkdown": "Of course it's OK!\n\nHowever, this was written by me when I was a mmdetection beginner, so there are many mistakes. For example, the size of the inference is improperly 1333x800 because it is taken from the base, it should be 1024x1024.\nThe scheduler is also unchanged.\n\nHowever, the basic setting is this one, so I think it will give some good results!\n\nhttps://www.kaggle.com/code/yosukeyama/hubmap23-mmdet-train-mask2former-1024/notebook\n\nEDIT: I have checked past scores. Using this training note, even a single-fold and improperly size inference can earn a silver medal.", "votes": null }, { "id": "2368855", "postDate": "08/01/2023 12:21:42", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": null }, { "id": "2370433", "postDate": "08/02/2023 12:04:59", "content": "Thank you, @yosukeyama for the report! This is quite an elegant solution with a very strong private result after switching to voting. Btw what voting method did you use in the corrected ensembling of masks?
\nThank you for your question!
\nThe \"voting\" here is that once the mask is obtained in binary and the majority of the model predicts the pixel to be a binary mask. The process is called.
\nSo the former of the codes is my method.
@yosukeyama It was unfortunate to miss out 4th place.
\n\n\nFor the ensemble, I made modifications to the \"WBF\" (Weighted Box Fusion) to use it with masks.
\n
Could you share the detail of WDF adaptation for mask prediction and CV strategy?
", "rawMarkdown": "yosukeyama It was unfortunate to miss out 4th place.\n\n> For the ensemble, I made modifications to the \"WBF\" (Weighted Box Fusion) to use it with masks.\n\nCould you share the detail of WDF adaptation for mask prediction and CV strategy?", "votes": null }, { "id": "2377771", "postDate": "08/07/2023 10:17:57", "content": "Thank you for your question!
\nThe base is utilizing the WBF library.
\nWhen clustering boxes, the boxes and scores are processed together, but in the original code, masks could not be used as inputs.
\nTherefore, when acquiring the combinations, I've made it so that masks are clustered along with them.
\nI thought it might be good to calculate the IoU using masks, but like WBF, I calculated the IoU using boxes.
\nUsually, a single model will output multiple overlapping boxes. Generally, after a model's output, techniques like NMS are used to leave only the high-scoring ones before performing WBF.
\nAt the time, I didn't think to apply NMS, so I decided to use the same number of top-scoring boxes as the number of models.
\nI believe this is essentially similar, but I think this way might result in lower confidence scores when less accurate masks are outputted in some cases. I have not been able to examine the superiority compared to using NMS. I would like to try it when I have time, but it seems like it will be difficult for a while.
CV strategy. This is my biggest failure.
\nBasically, I divided the fold into six (although I intended to divide the folds into five, due to typo, it was divided into six…), and essentially calculated only with fold0.
\nI mixed up dataset1 and dataset2 in the CV, and I now think this was a significant mistake.
\nFurthermore, I also conducted studies with a small number of ds1 only in fold0.
\nThis was because I had much work privately, and I had to choose whether to experiment with fold0-5 or reduce the number of experiments.
\nIn retrospect, I should have prioritized robustness even if reducing experiments, carried out training with all folds in all experiments, and calculated the CV.
\nThat would have prevented this situation…
Thank you for additional explanation. Below is my understanding, please correct them if they are wrong.
\n(By the way, it reminds me max pooling vs mean pooling on CV tasks.)
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fe6faf525d2ba296a54a327f93c2632a0%2FScreenshot%202023-08-07%20at%2021.25.14.png?generation=1691411164747665&alt=media\")
StratifiedKFold might mitigate the issue.About WBF.
\nThis is the same as my understanding.
\nYes, it is similar to max and mean pooling lol.
\nHowever, mAP can be improved by reordering the forecasts correctly. Therefore, I think it is quite nonsense to use max to aggregate predictions. This is because the implication is the same as taking max when performing an ensemble of many models in image classification. In such cases, taking mean or median is the most basic and robust method, isn't it?
\nIn object detection, as in image classification, it seems to be a good idea to calculate the confidence score using mean or weighted mean.
You are right about the second.
\nFurthermore, we were informed in advance that only ds1 was used in the test data, and as ds1 and ds2 are annotated differently, it was more efficient to calculate CVs according to ds1 only. So I should have spent more time considering how to fit ds1, as in the top solutions…
\n\nIn such cases, taking mean or median is the most basic and robust method, isn't it?
\n
Yeah, I agree with that for the ensemble method of different models' predictions.
\n\nSorry, I was misunderstood. I misunderstood NMS is applied before calculating loss. As NMS is applied as a post process, it would surely tend to generate over-confident boxes. So in the situation where accurate confidence value is required, NMS could be bad choice.
", "rawMarkdown": "> In such cases, taking mean or median is the most basic and robust method, isn't it?\n\nYeah, I agree with that for the ensemble method of different models' predictions.\n\n~~However, I am not sure if WBF is always effective as a suppression method for duplicated boxes from a single model. As global max pooling is successful on CV tasks, I believe NMS also could learn correct confidence values from the statistics of the dataset. I think try and error is required per dataset/tasks which aggregation strategy is optimal.~~\n\nSorry, I was misunderstood. I misunderstood NMS is applied before calculating loss. As NMS is applied as a post process, it would surely tend to generate over-confident boxes. So in the situation where accurate confidence value is required, NMS could be bad choice.", "votes": null }, { "id": "2378939", "postDate": "08/07/2023 21:44:28", "content": "After looking at the revised comments, I understood what you meant. Indeed, if we assume the process before loss calculation, it becomes almost the same argument as max-pooling, and I don't know which one is more suitable.
\nThat's right, in cases where mAP is used as an evaluation metric (requring confidence scores), one can say that WBF is a superior process to NMS. If we aggregate scores from multiple models like stacking and use gradient boosting, we might be able to predict even better scores, but the process becomes too complicated. So calculating mean values is reasonable.
", "rawMarkdown": "After looking at the revised comments, I understood what you meant. Indeed, if we assume the process before loss calculation, it becomes almost the same argument as max-pooling, and I don't know which one is more suitable.\n\nThat's right, in cases where mAP is used as an evaluation metric (requring confidence scores), one can say that WBF is a superior process to NMS. If we aggregate scores from multiple models like stacking and use gradient boosting, we might be able to predict even better scores, but the process becomes too complicated. So calculating mean values is reasonable.", "votes": null } ], "comments": [ { "id": 2368652, "author_name": "abebe9849", "author_url": "", "post_date": "08/01/2023 09:48:56", "content": "thanks for sharing!!
\nI could not get a good local cv with mask2former. can you tell me what config you used to learn mask2former?
Of course it's OK!
\nHowever, this was written by me when I was a mmdetection beginner, so there are many mistakes. For example, the size of the inference is improperly 1333x800 because it is taken from the base, it should be 1024x1024.
\nThe scheduler is also unchanged.
However, the basic setting is this one, so I think it will give some good results!
\nhttps://www.kaggle.com/code/yosukeyama/hubmap23-mmdet-train-mask2former-1024/notebook
\nEDIT: I have checked past scores. Using this training note, even a single-fold and improperly size inference can earn a silver medal.
", "votes": null, "replies": [] } ] }, { "id": 2368855, "author_name": "shigengtian", "author_url": "", "post_date": "08/01/2023 12:21:42", "content": "Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 2370433, "author_name": "vslaykovsky", "author_url": "", "post_date": "08/02/2023 12:04:59", "content": "Thank you, @yosukeyama for the report! This is quite an elegant solution with a very strong private result after switching to voting. Btw what voting method did you use in the corrected ensembling of masks?
\nThank you for your question!
\nThe \"voting\" here is that once the mask is obtained in binary and the majority of the model predicts the pixel to be a binary mask. The process is called.
\nSo the former of the codes is my method.
@yosukeyama It was unfortunate to miss out 4th place.
\n\n\nFor the ensemble, I made modifications to the \"WBF\" (Weighted Box Fusion) to use it with masks.
\n
Could you share the detail of WDF adaptation for mask prediction and CV strategy?
", "votes": null, "replies": [ { "id": 2377771, "author_name": "yosukeyama", "author_url": "", "post_date": "08/07/2023 10:17:57", "content": "Thank you for your question!
\nThe base is utilizing the WBF library.
\nWhen clustering boxes, the boxes and scores are processed together, but in the original code, masks could not be used as inputs.
\nTherefore, when acquiring the combinations, I've made it so that masks are clustered along with them.
\nI thought it might be good to calculate the IoU using masks, but like WBF, I calculated the IoU using boxes.
\nUsually, a single model will output multiple overlapping boxes. Generally, after a model's output, techniques like NMS are used to leave only the high-scoring ones before performing WBF.
\nAt the time, I didn't think to apply NMS, so I decided to use the same number of top-scoring boxes as the number of models.
\nI believe this is essentially similar, but I think this way might result in lower confidence scores when less accurate masks are outputted in some cases. I have not been able to examine the superiority compared to using NMS. I would like to try it when I have time, but it seems like it will be difficult for a while.
CV strategy. This is my biggest failure.
\nBasically, I divided the fold into six (although I intended to divide the folds into five, due to typo, it was divided into six…), and essentially calculated only with fold0.
\nI mixed up dataset1 and dataset2 in the CV, and I now think this was a significant mistake.
\nFurthermore, I also conducted studies with a small number of ds1 only in fold0.
\nThis was because I had much work privately, and I had to choose whether to experiment with fold0-5 or reduce the number of experiments.
\nIn retrospect, I should have prioritized robustness even if reducing experiments, carried out training with all folds in all experiments, and calculated the CV.
\nThat would have prevented this situation…
Thank you for additional explanation. Below is my understanding, please correct them if they are wrong.
\n(By the way, it reminds me max pooling vs mean pooling on CV tasks.)
\n
StratifiedKFold might mitigate the issue.About WBF.
\nThis is the same as my understanding.
\nYes, it is similar to max and mean pooling lol.
\nHowever, mAP can be improved by reordering the forecasts correctly. Therefore, I think it is quite nonsense to use max to aggregate predictions. This is because the implication is the same as taking max when performing an ensemble of many models in image classification. In such cases, taking mean or median is the most basic and robust method, isn't it?
\nIn object detection, as in image classification, it seems to be a good idea to calculate the confidence score using mean or weighted mean.
You are right about the second.
\nFurthermore, we were informed in advance that only ds1 was used in the test data, and as ds1 and ds2 are annotated differently, it was more efficient to calculate CVs according to ds1 only. So I should have spent more time considering how to fit ds1, as in the top solutions…
\n\nIn such cases, taking mean or median is the most basic and robust method, isn't it?
\n
Yeah, I agree with that for the ensemble method of different models' predictions.
\n\nSorry, I was misunderstood. I misunderstood NMS is applied before calculating loss. As NMS is applied as a post process, it would surely tend to generate over-confident boxes. So in the situation where accurate confidence value is required, NMS could be bad choice.
", "votes": null, "replies": [ { "id": 2378939, "author_name": "yosukeyama", "author_url": "", "post_date": "08/07/2023 21:44:28", "content": "After looking at the revised comments, I understood what you meant. Indeed, if we assume the process before loss calculation, it becomes almost the same argument as max-pooling, and I don't know which one is more suitable.
\nThat's right, in cases where mAP is used as an evaluation metric (requring confidence scores), one can say that WBF is a superior process to NMS. If we aggregate scores from multiple models like stacking and use gradient boosting, we might be able to predict even better scores, but the process becomes too complicated. So calculating mean values is reasonable.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2368561": "Despite achieving 9th place in the public leaderboard, I experienced a frightening shakeup, ending up at 295th place in the private leaderboard. However, many of the techniques I used proved effective in the private leaderboard, enabling me to achieve a score of 0.575 (equivalent to 4th place) by making just one modification to the 295th place code. Below, I will share the methods used and my reflections.\n\n### Method Overview:\nI primarily utilized \"mmdet\" and employed ensemble learning with numerous models. I had hoped to use YOLO v8 and v7 as well, but the host did not disclose their availability, so I had to abandon the idea.\n\nFor the ensemble, I made modifications to the \"WBF\" (Weighted Box Fusion) to use it with masks. Additionally, the major innovation was not limited to instance segmentation models but also involved using object detection models. I performed WBF based on bounding boxes and made mask predictions solely using the instance segmentation models. I used the object detection models only to calculate confidence scores. As you can see from the predictions, the models made far more predictions than needed. Given the nature of the evaluation metric, mAP, it was evident that properly aligning confidence scores could significantly improve the overall score.\n\n### Models\nThe following is a list of the models used:\n\nInstance Segmentation Models:\n* Mask2Former\n* MaskRCNN + various ConvNext v1 and v2\n* Cascade RCNN + various ConvNext v1 and v2\n* MaskRCNN + Swin Transformer\nOther: MaskDino doesn’t work well.\n\nObject Detection Models:\n* DINO\n* Mask2Former (used only for object detection part)\n* Diffusion Det\n* ViT Det\nOther: yolox doesn’t work well\n\n### Major Flaw:\nI combined the outputs of the mask models using addition during post-processing. This resulted in improvements in ds1 (average IoU) and the public cross-validation (CV) score. However, as many of you may have noticed, when applying post-processing, the score on the private dataset degrades significantly. This indicates that the ds1 and private datasets have entirely different annotation tendencies. The annotations provided for ds1 were quite rough, and the dilation technique seemed effective due to this. One of the hosts mentioned that annotations were done in the same manner for both public and private datasets, but this information turned out to be inaccurate.\n\n### Single modification to achieve private 0.575:\nAnticipating that the annotations for the private dataset might be more substantial, I submitted two versions: one with post-processing and another without. However, as mentioned earlier, during the ensemble mask generation, I was merely adding the masks instead of using averaging or voting. This method worked well for ds1 and the public dataset. On the other hand, I should have easily foreseen that if the private dataset had accurate and precise annotations, this approach could significantly reduce the score (which I overlooked).\nTherefore, instead of adding the masks, I changed to using voting. This simple change dramatically improved the score from 0.397 to 0.575.\n\n\n