\n
![\"TTA](\"https://raw.githubusercontent.com/amirassov/kaggle-imaterialist/master/figures/tta.png\")
\nbut the metrics in local CV only got worse (maybe because of hard NMS, perhaps I should try WBF?)
Hi everyone. I was experimenting with different tta/ensembling schemes on the weekends. However, nothing works reliably for me. My general TTA/Ensembling scheme looks like this:
\nHere are the methods I've tried:
\nI'm wondering if anyone has managed to reliably get improvements from ensembling and/or TTA?
", "messages": [ { "id": "1583062", "postDate": "11/15/2021 14:05:16", "content": "Hi everyone. I was experimenting with different tta/ensembling schemes on the weekends. However, nothing works reliably for me. My general TTA/Ensembling scheme looks like this:
\nHere are the methods I've tried:
\nI'm wondering if anyone has managed to reliably get improvements from ensembling and/or TTA?
", "rawMarkdown": "Hi everyone. I was experimenting with different tta/ensembling schemes on the weekends. However, nothing works reliably for me. My general TTA/Ensembling scheme looks like this:\n\n1. Inference, get T*N sets of predictions from T augmentations and N models (currently T=4 and N=1, just have a simple baseline Mask-RCNN model right now lol).\n2. Filter out small masks from those predictions.\n3. Combine predictions (using methods below).\n4. Filter out overlapping masks, giving priority to predictions with higher score.\n\nHere are the methods I've tried:\n\n1. Take unions of masks with large overlap and discard masks with small overlap (as suggested by winners of DS Bowl 2018, e.g. [here](https://www.kaggle.com/c/data-science-bowl-2018/discussion/56326). I tried different thresholds for that. No visible difference in local CV and on LB.\n2. I tried the following scheme as well (from [1st place in iMaterialist competition](https://www.kaggle.com/c/imaterialist-fashion-2019-FGVC6/discussion/95247)):\n \n but the metrics in local CV only got worse (maybe because of hard NMS, perhaps I should try WBF?)\n\nI'm wondering if anyone has managed to reliably get improvements from ensembling and/or TTA?", "votes": null }, { "id": "1583204", "postDate": "11/15/2021 16:12:36", "content": "I have faced the same problem. The method I've used is the union of the masks (with horizontal flip as you mentioned) and have not seen any improvement in my scores.
", "rawMarkdown": "I have faced the same problem. The method I've used is the union of the masks (with horizontal flip as you mentioned) and have not seen any improvement in my scores.", "votes": null }, { "id": "1583985", "postDate": "11/16/2021 07:28:04", "content": "Can I know, What Data-Augmentation transformations have you performed during test time.
", "rawMarkdown": "Can I know, What Data-Augmentation transformations have you performed during test time.", "votes": null }, { "id": "1584010", "postDate": "11/16/2021 07:55:21", "content": "As mentioned in https://www.kaggle.com/c/data-science-bowl-2018/discussion/56326:
\nCombined predictions on actual image and horizontally flipped image: took unions of masks with maximum overlap and removed false positive masks with small overlap.
Excuse me, what is TTA?
", "rawMarkdown": "Excuse me, what is TTA?", "votes": null }, { "id": "1584144", "postDate": "11/16/2021 10:05:09", "content": "I learn what TTA is.
\nTTA is test time augmentation.
\nI found the good source for TTA.
\nhttps://www.kaggle.com/andrewkh/test-time-augmentation-tta-worth-it
According to this article, TTA for mask rcnn has good effect on the accuracy.
\n", "rawMarkdown": "I learn what TTA is.\nTTA is test time augmentation.\nI found the good source for TTA.\nhttps://www.kaggle.com/andrewkh/test-time-augmentation-tta-worth-it\n\nAccording to this article, TTA for mask rcnn has good effect on the accuracy.\n\nhttps://www.researchgate.net/publication/340031950_Test-time_augmentation_for_deep_learning-based_cell_segmentation_on_microscopy_images", "votes": null }, { "id": "1584250", "postDate": "11/16/2021 11:43:01", "content": "I could improve it from 0.299 to 0.300 with ensemble. I will explore more.
", "rawMarkdown": "I could improve it from 0.299 to 0.300 with ensemble. I will explore more.", "votes": null }, { "id": "1584260", "postDate": "11/16/2021 11:52:46", "content": "Detectron's implementation is averaging the augmented mask.
\ndef _reduce_pred_masks(self, outputs, tfms):\n # Should apply inverse transforms on masks.\n # We assume only resize & flip are used. pred_masks is a scale-invariant\n # representation, so we handle flip specially\n for output, tfm in zip(outputs, tfms):\n if any(isinstance(t, HFlipTransform) for t in tfm.transforms):\n output.pred_masks = output.pred_masks.flip(dims=[3])\n all_pred_masks = torch.stack([o.pred_masks for o in outputs], dim=0)\n avg_pred_masks = torch.mean(all_pred_masks, dim=0)\n return avg_pred_masks\nOh, yeah, that's the implementation I used :)
", "rawMarkdown": "Oh, yeah, that's the implementation I used :)", "votes": null }, { "id": "1585802", "postDate": "11/17/2021 14:56:00", "content": "Alright guys, we had fun and all.. buttt it's over. @onodera started ensembling.
\nThe competition is over! we can all go home now!
Seriously, I just watched a video of you on youtube and you spoke about some past winning solution..
\nMan.. you pull out some insaaaaaaane stacking pipelines.
\nI couldn't figure how you manage to never overfit with such a huge ensemble. I usually overfit already with 10% of the models.
Respect!
", "rawMarkdown": "Alright guys, we had fun and all.. buttt it's over. @onodera started ensembling. \nThe competition is over! we can all go home now! \n\nSeriously, I just watched a video of you on youtube and you spoke about some past winning solution.. \nMan.. you pull out some insaaaaaaane stacking pipelines. \nI couldn't figure how you manage to never overfit with such a huge ensemble. I usually overfit already with 10% of the models. \n\nRespect!", "votes": null }, { "id": "1586225", "postDate": "11/17/2021 22:51:36", "content": "I also actually found some thresholds/parameters to make my ensemble improve by 0.001-0.002, but it's not stable (3x tta is fine, but 4x or more will produce worse results). However, it seems a bit meaningless since the metric itself is quite noisy, so I doubt such small difference may propagate to private LB. I might be terribly wrong though.
", "rawMarkdown": "I also actually found some thresholds/parameters to make my ensemble improve by 0.001-0.002, but it's not stable (3x tta is fine, but 4x or more will produce worse results). However, it seems a bit meaningless since the metric itself is quite noisy, so I doubt such small difference may propagate to private LB. I might be terribly wrong though.", "votes": null }, { "id": "1589523", "postDate": "11/20/2021 11:10:13", "content": "I also tried two simple ensembles, both of which improved my score from 0.305 to 0.306.
\nI'm thinking of trying TTA as well. When I tried to use only vertical flip for prediction, the score became much worse.
\nI think this has something to do with the fact that only horizontal flip is used for training. However, that was surprising since I thought there was no vertical difference in optical micrographs. There was almost no deterioration in the score with horizontal flip.
Hi, How to add it?
", "rawMarkdown": "Hi, How to add it?", "votes": null }, { "id": "1591382", "postDate": "11/22/2021 09:14:59", "content": "@chankhavu I have tried NMS at stage 2, WBF at stage 2, NMS at each stage like mentioned above, and also average the masks, however none of them gave a significant gain in CV and LB unfortunately.
", "rawMarkdown": "chankhavu I have tried NMS at stage 2, WBF at stage 2, NMS at each stage like mentioned above, and also average the masks, however none of them gave a significant gain in CV and LB unfortunately.", "votes": null }, { "id": "1594444", "postDate": "11/24/2021 20:42:41", "content": "I also experimented with TTA and ended up writing a Weighted Segments Fusion algorithm - works well on a mediocre performing model. Haven’t succeeded in training a good performing model yet, so can’t say if it also improves score on that…
", "rawMarkdown": "I also experimented with TTA and ended up writing a [Weighted Segments Fusion](https://www.kaggle.com/mistag/sartorius-tta-with-weighted-segments-fusion/notebook) algorithm - works well on a mediocre performing model. Haven’t succeeded in training a good performing model yet, so can’t say if it also improves score on that…", "votes": null }, { "id": "1613570", "postDate": "12/10/2021 05:29:45", "content": "I found the paper for applying TTA to cell instance segmentation. It should have good effect.
\nhttps://www.nature.com/articles/s41598-020-61808-3
I have faced the same problem. The method I've used is the union of the masks (with horizontal flip as you mentioned) and have not seen any improvement in my scores.
", "votes": null, "replies": [] }, { "id": 1583985, "author_name": "gowrishankarp", "author_url": "", "post_date": "11/16/2021 07:28:04", "content": "Can I know, What Data-Augmentation transformations have you performed during test time.
", "votes": null, "replies": [ { "id": 1584010, "author_name": "hsadeghian", "author_url": "", "post_date": "11/16/2021 07:55:21", "content": "As mentioned in https://www.kaggle.com/c/data-science-bowl-2018/discussion/56326:
\nCombined predictions on actual image and horizontally flipped image: took unions of masks with maximum overlap and removed false positive masks with small overlap.
Excuse me, what is TTA?
", "votes": null, "replies": [ { "id": 1584144, "author_name": "osamurai", "author_url": "", "post_date": "11/16/2021 10:05:09", "content": "I learn what TTA is.
\nTTA is test time augmentation.
\nI found the good source for TTA.
\nhttps://www.kaggle.com/andrewkh/test-time-augmentation-tta-worth-it
According to this article, TTA for mask rcnn has good effect on the accuracy.
\n", "votes": null, "replies": [] } ] }, { "id": 1584250, "author_name": "onodera", "author_url": "", "post_date": "11/16/2021 11:43:01", "content": "I could improve it from 0.299 to 0.300 with ensemble. I will explore more.
", "votes": null, "replies": [ { "id": 1585802, "author_name": "", "author_url": "", "post_date": "11/17/2021 14:56:00", "content": "Alright guys, we had fun and all.. buttt it's over. @onodera started ensembling.
\nThe competition is over! we can all go home now!
Seriously, I just watched a video of you on youtube and you spoke about some past winning solution..
\nMan.. you pull out some insaaaaaaane stacking pipelines.
\nI couldn't figure how you manage to never overfit with such a huge ensemble. I usually overfit already with 10% of the models.
Respect!
", "votes": null, "replies": [] }, { "id": 1586225, "author_name": "chankhavu", "author_url": "", "post_date": "11/17/2021 22:51:36", "content": "I also actually found some thresholds/parameters to make my ensemble improve by 0.001-0.002, but it's not stable (3x tta is fine, but 4x or more will produce worse results). However, it seems a bit meaningless since the metric itself is quite noisy, so I doubt such small difference may propagate to private LB. I might be terribly wrong though.
", "votes": null, "replies": [] }, { "id": 1589523, "author_name": "yosukeyama", "author_url": "", "post_date": "11/20/2021 11:10:13", "content": "I also tried two simple ensembles, both of which improved my score from 0.305 to 0.306.
\nI'm thinking of trying TTA as well. When I tried to use only vertical flip for prediction, the score became much worse.
\nI think this has something to do with the fact that only horizontal flip is used for training. However, that was surprising since I thought there was no vertical difference in optical micrographs. There was almost no deterioration in the score with horizontal flip.
Detectron's implementation is averaging the augmented mask.
\ndef _reduce_pred_masks(self, outputs, tfms):\n # Should apply inverse transforms on masks.\n # We assume only resize & flip are used. pred_masks is a scale-invariant\n # representation, so we handle flip specially\n for output, tfm in zip(outputs, tfms):\n if any(isinstance(t, HFlipTransform) for t in tfm.transforms):\n output.pred_masks = output.pred_masks.flip(dims=[3])\n all_pred_masks = torch.stack([o.pred_masks for o in outputs], dim=0)\n avg_pred_masks = torch.mean(all_pred_masks, dim=0)\n return avg_pred_masks\nOh, yeah, that's the implementation I used :)
", "votes": null, "replies": [] }, { "id": 1589632, "author_name": "zekunn", "author_url": "", "post_date": "11/20/2021 12:56:47", "content": "Hi, How to add it?
", "votes": null, "replies": [] } ] }, { "id": 1591382, "author_name": "namgalielei", "author_url": "", "post_date": "11/22/2021 09:14:59", "content": "@chankhavu I have tried NMS at stage 2, WBF at stage 2, NMS at each stage like mentioned above, and also average the masks, however none of them gave a significant gain in CV and LB unfortunately.
", "votes": null, "replies": [] }, { "id": 1594444, "author_name": "mistag", "author_url": "", "post_date": "11/24/2021 20:42:41", "content": "I also experimented with TTA and ended up writing a Weighted Segments Fusion algorithm - works well on a mediocre performing model. Haven’t succeeded in training a good performing model yet, so can’t say if it also improves score on that…
", "votes": null, "replies": [] }, { "id": 1613570, "author_name": "osamurai", "author_url": "", "post_date": "12/10/2021 05:29:45", "content": "I found the paper for applying TTA to cell instance segmentation. It should have good effect.
\nhttps://www.nature.com/articles/s41598-020-61808-3