![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F940556%2Faec538c6a536cc9dd070697b4e416870%2F2019-11-19%2010-58-58.png?generation=1574150366439784&alt=media\")
I have adopted my pipeline from Severstal Defect Detection and was able to get silver medal in last two days with just 6 submissions, here is a short description of 55th place solution.
\n\n2 step pileline\n 1) Multi-task network (classification + segmentation) as classifier to remove empty masks\n 2) Binary segmentation for each class
\n\nI have trained 5-fold FPN(resnet34) + aux classfication output on 480x640 images using Flip, RandomBrightness as augmentations. Model trained just 6-7 epochs and than starts to overfit, I do nothing with that, just save top 5 checkpoints according to metric.
Loss (segmentation head): bce+dice\nLoss (classification head): bce\nOptimizer: AdamW\nPostprocessing: remove masks less than 10000 pixels\nThresholds: [0.6, 0.6, 0.6, 0.6]\n
For each class trained 2 x Unet(se_resnext50_34x4d) only on images with masks of that class!\nwith same optimizer, image size and augmentations.
Loss: bce+dice\nThresholds: [0.4, 0.4, 0.4, 0.4]
\n\nFor all models made checkpoints weights! averaging (+0.005-0.01 on validation).\nModels over each stage have been just averaged with Flip TTA.
\n\nAnd congratulations to winners!
", "messages": [ { "id": "676474", "postDate": "11/19/2019 08:06:08", "content": "I have adopted my pipeline from Severstal Defect Detection and was able to get silver medal in last two days with just 6 submissions, here is a short description of 55th place solution.
\n\n2 step pileline\n 1) Multi-task network (classification + segmentation) as classifier to remove empty masks\n 2) Binary segmentation for each class
\n\nI have trained 5-fold FPN(resnet34) + aux classfication output on 480x640 images using Flip, RandomBrightness as augmentations. Model trained just 6-7 epochs and than starts to overfit, I do nothing with that, just save top 5 checkpoints according to metric.
Loss (segmentation head): bce+dice\nLoss (classification head): bce\nOptimizer: AdamW\nPostprocessing: remove masks less than 10000 pixels\nThresholds: [0.6, 0.6, 0.6, 0.6]\n
For each class trained 2 x Unet(se_resnext50_34x4d) only on images with masks of that class!\nwith same optimizer, image size and augmentations.
Loss: bce+dice\nThresholds: [0.4, 0.4, 0.4, 0.4]
\n\nFor all models made checkpoints weights! averaging (+0.005-0.01 on validation).\nModels over each stage have been just averaged with Flip TTA.
\n\nAnd congratulations to winners!
", "rawMarkdown": "## Easy silver in last days\nI have adopted my pipeline from Severstal Defect Detection and was able to get silver medal in last two days with just 6 submissions, here is a short description of 55th place solution.\n\n2 step pileline\n 1) Multi-task network (classification + segmentation) as classifier to remove empty masks\n 2) Binary segmentation for each class\n \n### 1st step.\nI have trained 5-fold `FPN(resnet34) + aux classfication output` on 480x640 images using `Flip`, `RandomBrightness` as augmentations. Model trained just 6-7 epochs and than starts to overfit, I do nothing with that, just save top 5 checkpoints according to metric.\n\nLoss (segmentation head): bce+dice\nLoss (classification head): bce\nOptimizer: AdamW\nPostprocessing: remove masks less than 10000 pixels\nThresholds: [0.6, 0.6, 0.6, 0.6]\n\n\n### 2nd step.\nFor each class trained `2 x Unet(se_resnext50_34x4d)` only on images with masks of that class!\nwith same optimizer, image size and augmentations.\n\nLoss: bce+dice\nThresholds: [0.4, 0.4, 0.4, 0.4]\n\n### Ensemble\nFor all models made checkpoints weights! averaging (+0.005-0.01 on validation).\nModels over each stage have been just averaged with Flip TTA.\n\n### Useful links\n - Segmentation Models: https://github.com/qubvel/segmentation_models.pytorch\n - Test Time Augmentation for PyTorch: https://github.com/qubvel/ttach\n\n**And congratulations to winners!**", "votes": null }, { "id": "676481", "postDate": "11/19/2019 08:13:57", "content": "Sweet and Simple!! Thank you for sharing and also thank you for your segmentation_models.pytorch library. Extremely helpful and convenient
most of the time i use your library,thanks a lot and congratulations
", "rawMarkdown": "most of the time i use your library,thanks a lot and congratulations", "votes": null }, { "id": "676523", "postDate": "11/19/2019 09:03:50", "content": "Congratulations\nThank you for Sharing your Insights & Approach! @pavel92
", "rawMarkdown": "Congratulations\nThank you for Sharing your Insights & Approach! @pavel92", "votes": null }, { "id": "676551", "postDate": "11/19/2019 09:44:30", "content": "@pavel92 thanks for your solution writeup. This was my first computer vision competition and I used only your library 'segmentation models' to implement my solution. I must say this library is really helpful for beginners like me. After reading the comments of top solutions, I came to know about the term 'weights averaging from different checkpoints'. Will you please share some resources to understand this concept and may be a sample boiler-plate implementation of this idea?
", "rawMarkdown": "pavel92 thanks for your solution writeup. This was my first computer vision competition and I used only your library 'segmentation models' to implement my solution. I must say this library is really helpful for beginners like me. After reading the comments of top solutions, I came to know about the term 'weights averaging from different checkpoints'. Will you please share some resources to understand this concept and may be a sample boiler-plate implementation of this idea?", "votes": null }, { "id": "676573", "postDate": "11/19/2019 10:12:22", "content": "```\nimport torch\nfrom collections import OrderedDict\nfrom typing import List
\n\ncheckpoints_weights_paths: List[str] = ... # sorted in descending order by score\nmodel: torch.nn.Module = ...
\n\ndef average_weights(state_dicts: List[dict]):\n everage_dict = OrderedDict()\n for k in state_dicts[0].keys():\n everage_dict[k] = sum([state_dict[k] for state_dict in state_dicts]) / len(state_dicts)\n return everage_dict
\n\nall_weights = [torch.load(path) for path in checkpoints_weights_paths]
\n\nbest_score = 0\nbest_weights = []
\n\nfor w in all_weights:\n current_weights = best_weights + [w]\n average_dict = average_weights(current_weights)\n model.load_state_dict(average_dict)\n score = evaluate_model(model, ...)\n if score > best_score:\n best_score = score\n best_weights.append(w)\n```
\n\nhttps://gist.github.com/qubvel/70c3d5e4cddcde731408f478e12ef87b
", "rawMarkdown": "```\nimport torch\nfrom collections import OrderedDict\nfrom typing import List\n\ncheckpoints_weights_paths: List[str] = ... # sorted in descending order by score\nmodel: torch.nn.Module = ...\n\n\ndef average_weights(state_dicts: List[dict]):\n everage_dict = OrderedDict()\n for k in state_dicts[0].keys():\n everage_dict[k] = sum([state_dict[k] for state_dict in state_dicts]) / len(state_dicts)\n return everage_dict\n\n\nall_weights = [torch.load(path) for path in checkpoints_weights_paths]\n\nbest_score = 0\nbest_weights = []\n\nfor w in all_weights:\n current_weights = best_weights + [w]\n average_dict = average_weights(current_weights)\n model.load_state_dict(average_dict)\n score = evaluate_model(model, ...)\n if score > best_score:\n best_score = score\n best_weights.append(w)\n```\n\nhttps://gist.github.com/qubvel/70c3d5e4cddcde731408f478e12ef87b", "votes": null }, { "id": "676590", "postDate": "11/19/2019 10:30:49", "content": "Very impressive!!! Congratulations 🎉 and thanks for sharing ❤️
", "rawMarkdown": "Very impressive!!! Congratulations 🎉 and thanks for sharing ❤️", "votes": null }, { "id": "676608", "postDate": "11/19/2019 10:50:36", "content": "Thanks a lot :)
", "rawMarkdown": "Thanks a lot :)", "votes": null }, { "id": "676609", "postDate": "11/19/2019 10:51:51", "content": "Pavel, can you please leave here some info (maybe links to read) about such multi-task network and how to implement it?
", "rawMarkdown": "Pavel, can you please leave here some info (maybe links to read) about such multi-task network and how to implement it?", "votes": null }, { "id": "676632", "postDate": "11/19/2019 11:27:40", "content": "I have prepared new feature for SMP library that add aux output for models (look at https://github.com/qubvel/segmentation_models.pytorch/tree/models-refactoring)
\n\nAccording to this implementation my multi-task network defined as follows:\n```\nclass GatedFPN(smp.FPN):
\n\ndef forward(self, x):\n mask, label = super().forward(x)\n return dict(\n mask=mask*label.reshape(*label.size(), 1, 1),\n label=label,\n )\naux_params = dict(classes=4, activation='sigmoid', dropout=0.5, pooling='avg')\nmodel = GatedFPN('resnet34', encoder_weights='imagenet', classes=4, activation='sigmoid', aux_params=aux_params)\n```
", "rawMarkdown": "I have prepared new feature for SMP library that add aux output for models (look at https://github.com/qubvel/segmentation_models.pytorch/tree/models-refactoring)\n\nAccording to this implementation my multi-task network defined as follows:\n```\nclass GatedFPN(smp.FPN):\n\n def forward(self, x):\n mask, label = super().forward(x)\n return dict(\n mask=mask*label.reshape(*label.size(), 1, 1),\n label=label,\n )\n\naux_params = dict(classes=4, activation='sigmoid', dropout=0.5, pooling='avg')\nmodel = GatedFPN('resnet34', encoder_weights='imagenet', classes=4, activation='sigmoid', aux_params=aux_params)\n```", "votes": null }, { "id": "676661", "postDate": "11/19/2019 11:56:26", "content": "strong man, congratulationns qubvel
", "rawMarkdown": "strong man, congratulationns qubvel", "votes": null }, { "id": "676663", "postDate": "11/19/2019 12:04:59", "content": "Congratulations @pavel92 ,
\n\nIt's amazing how some people can get such great results with just a few days and a pipeline simples as that.
\n\nI was planning to use AdamW as well, but did not have the time, if you used an implementation from a Git repository do you mind sharing?
\n\nAlso thanks for your amazing Git repositories.
", "rawMarkdown": "Congratulations @pavel92 , \n\nIt's amazing how some people can get such great results with just a few days and a pipeline simples as that.\n\nI was planning to use AdamW as well, but did not have the time, if you used an implementation from a Git repository do you mind sharing?\n\nAlso thanks for your amazing Git repositories.", "votes": null }, { "id": "676670", "postDate": "11/19/2019 12:08:17", "content": "If you use pytorch, then AdamW is already implemented here
", "rawMarkdown": "If you use pytorch, then AdamW is already implemented [here](https://pytorch.org/docs/stable/optim.html#torch.optim.AdamW)", "votes": null }, { "id": "676882", "postDate": "11/19/2019 15:11:56", "content": "Thanks @bibek777 , but I mainly use Keras 😄
", "rawMarkdown": "Thanks @bibek777 , but I mainly use Keras 😄", "votes": null }, { "id": "676885", "postDate": "11/19/2019 15:15:42", "content": "Very! useful. Specialty TTAch ...
", "rawMarkdown": "Very! useful. Specialty TTAch ...", "votes": null }, { "id": "676997", "postDate": "11/19/2019 17:35:35", "content": "Congrats Qubvel. Simple and effective.
\n\nIn step 2, do you use multi-task networks (two headed models) again? In steps 1 and 2 does threshold refer to \"label threshold\" as in if label < threshold: mask = np.zeros() or \"pixel threshold\" as in mask = (mask > threshold).astype(int). If label threshold, what is your pixel threshold?
I also added a second head to your GitHub Unet model. I added it after the segmentation output, I should try to add it after the encoder and see if that works better. A picture of my design is here
", "rawMarkdown": "Congrats Qubvel. Simple and effective. \n \nIn step 2, do you use multi-task networks (two headed models) again? In steps 1 and 2 does threshold refer to \"label threshold\" as in `if label < threshold: mask = np.zeros()` or \"pixel threshold\" as in `mask = (mask > threshold).astype(int)`. If label threshold, what is your pixel threshold?\n\nI also added a second head to your GitHub Unet model. I added it after the segmentation output, I should try to add it after the encoder and see if that works better. A picture of my design is [here][1]\n\n[1]: https://www.kaggle.com/c/understanding_cloud_organization/discussion/118086", "votes": null }, { "id": "677012", "postDate": "11/19/2019 17:54:59", "content": "That's cool!\nFast track to medal zone is using segmentation_models lib
", "rawMarkdown": "That's cool!\nFast track to medal zone is using [segmentation_models](https://github.com/qubvel/segmentation_models.pytorch/https://github.com/qubvel/segmentation_models.pytorch/) lib", "votes": null }, { "id": "677413", "postDate": "11/20/2019 05:41:34", "content": "Thanks!\nTo select images for the second stage I use max pixel value from mask output on first stage. Actually, at first, I was think that first stage will be enough and create a submission with only these models, but get not satisfied result. After that I decided to train stage 2 models and replace masks from that submission with new ones (so, may be using label output can give better score, but I did not have time and submissions to test it).\nIn step two I used just simple unet without classification head.
Congrats and thanks for sharing!! Great approach!!
", "rawMarkdown": "Congrats and thanks for sharing!! Great approach!!", "votes": null }, { "id": "677467", "postDate": "11/20/2019 07:52:54", "content": "I use pytorch optimizers
", "rawMarkdown": "I use pytorch optimizers", "votes": null }, { "id": "678097", "postDate": "11/21/2019 01:42:13", "content": "@dimitreoliveira AdamW saved me in this competition. Few breakthrough I got in this competition are:\nMy network - 2 X FPN EfficientNetB1 + 2 X FPN EfficientNetB2 + 4 Classifiers(2 each of EfficientNetB2 and EfficientNetB3)\n1. Changed optimizer from Adam to AdamW and did not change anything - jumped from top 48% to top 15%.\n2. Increased min pixel area from 2500 to 10000 - jumped from top 15% to top 8%.\n3. Increased min pixel area from 10000 to 15000 - jumped from top 8% to top 4%.
\n\nAfter that I tried many other methods but did not work.
", "rawMarkdown": "dimitreoliveira AdamW saved me in this competition. Few breakthrough I got in this competition are:\nMy network - 2 X FPN EfficientNetB1 + 2 X FPN EfficientNetB2 + 4 Classifiers(2 each of EfficientNetB2 and EfficientNetB3)\n1. Changed optimizer from Adam to AdamW and did not change anything - jumped from top 48% to top 15%.\n2. Increased min pixel area from 2500 to 10000 - jumped from top 15% to top 8%.\n3. Increased min pixel area from 10000 to 15000 - jumped from top 8% to top 4%.\n\nAfter that I tried many other methods but did not work.", "votes": null }, { "id": "678433", "postDate": "11/21/2019 11:46:52", "content": "Thanks @raghaw , your results were very impressive, I was planning to experiment with AdamW on the last 2 weeks, but with all the ensembling I ended not having the time.
\n\nIf you don't mind what LR Scheduler were you using? and if was Keras/Tensorflow, do you mind sharing your source/implementation of AdamW?
", "rawMarkdown": "Thanks @raghaw , your results were very impressive, I was planning to experiment with AdamW on the last 2 weeks, but with all the ensembling I ended not having the time.\n\nIf you don't mind what LR Scheduler were you using? and if was Keras/Tensorflow, do you mind sharing your source/implementation of AdamW?", "votes": null }, { "id": "678550", "postDate": "11/21/2019 14:31:07", "content": "@dimitreoliveira, I used ExponentialLR scheduler of pytorch with gamma=0.80 and initial learning rate 5e-4. AdamW is already in pytorch as mentioned by @bibek777 so I did not face any problem. I found AdamW for keras version on github hope it will be helpfull to you.\nhttps://github.com/GLambard/AdamW_Keras
", "rawMarkdown": "dimitreoliveira, I used ExponentialLR scheduler of pytorch with gamma=0.80 and initial learning rate 5e-4. AdamW is already in pytorch as mentioned by @bibek777 so I did not face any problem. I found AdamW for keras version on github hope it will be helpfull to you.\nhttps://github.com/GLambard/AdamW_Keras", "votes": null }, { "id": "678573", "postDate": "11/21/2019 15:10:07", "content": "Thanks again @raghaw , about the repository it seems that it has some issues, but the topic author linked a more reliable source.
", "rawMarkdown": "Thanks again @raghaw , about the repository it seems that it has [some issues](https://github.com/GLambard/AdamW_Keras/issues/2), but the topic author linked a more reliable source.", "votes": null }, { "id": "678626", "postDate": "11/21/2019 16:53:21", "content": "@dimitreoliveira I was reading this blog:\nhttps://www.iprally.com/news/recent-improvements-to-the-adam-optimizer\nIn AdamW section they have given hyperlink of its tensorflow implementation. It is already added in tensorflow. check this link.\nhttps://www.tensorflow.org/versions/r1.15/api_docs/python/tf/contrib/opt/AdamWOptimizer
", "rawMarkdown": "dimitreoliveira I was reading this blog:\nhttps://www.iprally.com/news/recent-improvements-to-the-adam-optimizer\nIn AdamW section they have given hyperlink of its tensorflow implementation. It is already added in tensorflow. check this link.\nhttps://www.tensorflow.org/versions/r1.15/api_docs/python/tf/contrib/opt/AdamWOptimizer", "votes": null }, { "id": "678661", "postDate": "11/21/2019 17:53:45", "content": "Nice, thanks a lot!
", "rawMarkdown": "Nice, thanks a lot!", "votes": null }, { "id": "678707", "postDate": "11/21/2019 18:33:44", "content": "Interesting. So you don't use the classification output from Stage 1. Then is the only purpose of the classification head to back propagate label error into the encoder? Or does it server some other purpose that I'm missing?
", "rawMarkdown": "Interesting. So you don't use the classification output from Stage 1. Then is the only purpose of the classification head to back propagate label error into the encoder? Or does it server some other purpose that I'm missing?", "votes": null }, { "id": "679407", "postDate": "11/22/2019 18:45:20", "content": "You are right, but also decoder output is multiplied by 'label' output to generate mask (look at the pic.).
", "rawMarkdown": "You are right, but also decoder output is multiplied by 'label' output to generate mask (look at the pic.).", "votes": null }, { "id": "679699", "postDate": "11/23/2019 07:34:30", "content": "Thank you , very useful
", "rawMarkdown": "Thank you , very useful", "votes": null }, { "id": "679712", "postDate": "11/23/2019 08:36:07", "content": "p.s. merged to master
", "rawMarkdown": "p.s. merged to master", "votes": null }, { "id": "680925", "postDate": "11/25/2019 12:21:20", "content": "Congrats！one question, In step 2 , for every class, you trained a binary-seg model or train a seg model for all class
", "rawMarkdown": "Congrats！one question, In step 2 , for every class, you trained a binary-seg model or train a seg model for all class", "votes": null }, { "id": "681940", "postDate": "11/26/2019 18:02:55", "content": "step 2 - separate binary segmentation model for each class
", "rawMarkdown": "step 2 - separate binary segmentation model for each class", "votes": null } ], "comments": [ { "id": 676481, "author_name": "bibek777", "author_url": "", "post_date": "11/19/2019 08:13:57", "content": "Sweet and Simple!! Thank you for sharing and also thank you for your segmentation_models.pytorch library. Extremely helpful and convenient
most of the time i use your library,thanks a lot and congratulations
", "votes": null, "replies": [] }, { "id": 676523, "author_name": "veeralakrishna", "author_url": "", "post_date": "11/19/2019 09:03:50", "content": "Congratulations\nThank you for Sharing your Insights & Approach! @pavel92
", "votes": null, "replies": [] }, { "id": 676551, "author_name": "adish333", "author_url": "", "post_date": "11/19/2019 09:44:30", "content": "@pavel92 thanks for your solution writeup. This was my first computer vision competition and I used only your library 'segmentation models' to implement my solution. I must say this library is really helpful for beginners like me. After reading the comments of top solutions, I came to know about the term 'weights averaging from different checkpoints'. Will you please share some resources to understand this concept and may be a sample boiler-plate implementation of this idea?
", "votes": null, "replies": [ { "id": 676573, "author_name": "pavel92", "author_url": "", "post_date": "11/19/2019 10:12:22", "content": "```\nimport torch\nfrom collections import OrderedDict\nfrom typing import List
\n\ncheckpoints_weights_paths: List[str] = ... # sorted in descending order by score\nmodel: torch.nn.Module = ...
\n\ndef average_weights(state_dicts: List[dict]):\n everage_dict = OrderedDict()\n for k in state_dicts[0].keys():\n everage_dict[k] = sum([state_dict[k] for state_dict in state_dicts]) / len(state_dicts)\n return everage_dict
\n\nall_weights = [torch.load(path) for path in checkpoints_weights_paths]
\n\nbest_score = 0\nbest_weights = []
\n\nfor w in all_weights:\n current_weights = best_weights + [w]\n average_dict = average_weights(current_weights)\n model.load_state_dict(average_dict)\n score = evaluate_model(model, ...)\n if score > best_score:\n best_score = score\n best_weights.append(w)\n```
\n\nhttps://gist.github.com/qubvel/70c3d5e4cddcde731408f478e12ef87b
", "votes": null, "replies": [] }, { "id": 676608, "author_name": "adish333", "author_url": "", "post_date": "11/19/2019 10:50:36", "content": "Thanks a lot :)
", "votes": null, "replies": [] } ] }, { "id": 676590, "author_name": "phunghieu", "author_url": "", "post_date": "11/19/2019 10:30:49", "content": "Very impressive!!! Congratulations 🎉 and thanks for sharing ❤️
", "votes": null, "replies": [] }, { "id": 676609, "author_name": "detkov", "author_url": "", "post_date": "11/19/2019 10:51:51", "content": "Pavel, can you please leave here some info (maybe links to read) about such multi-task network and how to implement it?
", "votes": null, "replies": [ { "id": 676632, "author_name": "pavel92", "author_url": "", "post_date": "11/19/2019 11:27:40", "content": "I have prepared new feature for SMP library that add aux output for models (look at https://github.com/qubvel/segmentation_models.pytorch/tree/models-refactoring)
\n\nAccording to this implementation my multi-task network defined as follows:\n```\nclass GatedFPN(smp.FPN):
\n\ndef forward(self, x):\n mask, label = super().forward(x)\n return dict(\n mask=mask*label.reshape(*label.size(), 1, 1),\n label=label,\n )\naux_params = dict(classes=4, activation='sigmoid', dropout=0.5, pooling='avg')\nmodel = GatedFPN('resnet34', encoder_weights='imagenet', classes=4, activation='sigmoid', aux_params=aux_params)\n```
", "votes": null, "replies": [] }, { "id": 679712, "author_name": "pavel92", "author_url": "", "post_date": "11/23/2019 08:36:07", "content": "p.s. merged to master
", "votes": null, "replies": [] } ] }, { "id": 676661, "author_name": "jt120lz", "author_url": "", "post_date": "11/19/2019 11:56:26", "content": "strong man, congratulationns qubvel
", "votes": null, "replies": [] }, { "id": 676663, "author_name": "dimitreoliveira", "author_url": "", "post_date": "11/19/2019 12:04:59", "content": "Congratulations @pavel92 ,
\n\nIt's amazing how some people can get such great results with just a few days and a pipeline simples as that.
\n\nI was planning to use AdamW as well, but did not have the time, if you used an implementation from a Git repository do you mind sharing?
\n\nAlso thanks for your amazing Git repositories.
", "votes": null, "replies": [ { "id": 676670, "author_name": "bibek777", "author_url": "", "post_date": "11/19/2019 12:08:17", "content": "If you use pytorch, then AdamW is already implemented here
", "votes": null, "replies": [] }, { "id": 676882, "author_name": "dimitreoliveira", "author_url": "", "post_date": "11/19/2019 15:11:56", "content": "Thanks @bibek777 , but I mainly use Keras 😄
", "votes": null, "replies": [] }, { "id": 677467, "author_name": "pavel92", "author_url": "", "post_date": "11/20/2019 07:52:54", "content": "I use pytorch optimizers
", "votes": null, "replies": [] }, { "id": 678097, "author_name": "raghaw", "author_url": "", "post_date": "11/21/2019 01:42:13", "content": "@dimitreoliveira AdamW saved me in this competition. Few breakthrough I got in this competition are:\nMy network - 2 X FPN EfficientNetB1 + 2 X FPN EfficientNetB2 + 4 Classifiers(2 each of EfficientNetB2 and EfficientNetB3)\n1. Changed optimizer from Adam to AdamW and did not change anything - jumped from top 48% to top 15%.\n2. Increased min pixel area from 2500 to 10000 - jumped from top 15% to top 8%.\n3. Increased min pixel area from 10000 to 15000 - jumped from top 8% to top 4%.
\n\nAfter that I tried many other methods but did not work.
", "votes": null, "replies": [] }, { "id": 678433, "author_name": "dimitreoliveira", "author_url": "", "post_date": "11/21/2019 11:46:52", "content": "Thanks @raghaw , your results were very impressive, I was planning to experiment with AdamW on the last 2 weeks, but with all the ensembling I ended not having the time.
\n\nIf you don't mind what LR Scheduler were you using? and if was Keras/Tensorflow, do you mind sharing your source/implementation of AdamW?
", "votes": null, "replies": [] }, { "id": 678550, "author_name": "raghaw", "author_url": "", "post_date": "11/21/2019 14:31:07", "content": "@dimitreoliveira, I used ExponentialLR scheduler of pytorch with gamma=0.80 and initial learning rate 5e-4. AdamW is already in pytorch as mentioned by @bibek777 so I did not face any problem. I found AdamW for keras version on github hope it will be helpfull to you.\nhttps://github.com/GLambard/AdamW_Keras
", "votes": null, "replies": [] }, { "id": 678573, "author_name": "dimitreoliveira", "author_url": "", "post_date": "11/21/2019 15:10:07", "content": "Thanks again @raghaw , about the repository it seems that it has some issues, but the topic author linked a more reliable source.
", "votes": null, "replies": [] }, { "id": 678626, "author_name": "raghaw", "author_url": "", "post_date": "11/21/2019 16:53:21", "content": "@dimitreoliveira I was reading this blog:\nhttps://www.iprally.com/news/recent-improvements-to-the-adam-optimizer\nIn AdamW section they have given hyperlink of its tensorflow implementation. It is already added in tensorflow. check this link.\nhttps://www.tensorflow.org/versions/r1.15/api_docs/python/tf/contrib/opt/AdamWOptimizer
", "votes": null, "replies": [] }, { "id": 678661, "author_name": "dimitreoliveira", "author_url": "", "post_date": "11/21/2019 17:53:45", "content": "Nice, thanks a lot!
", "votes": null, "replies": [] } ] }, { "id": 676885, "author_name": "vingou", "author_url": "", "post_date": "11/19/2019 15:15:42", "content": "Very! useful. Specialty TTAch ...
", "votes": null, "replies": [] }, { "id": 676997, "author_name": "cdeotte", "author_url": "", "post_date": "11/19/2019 17:35:35", "content": "Congrats Qubvel. Simple and effective.
\n\nIn step 2, do you use multi-task networks (two headed models) again? In steps 1 and 2 does threshold refer to \"label threshold\" as in if label < threshold: mask = np.zeros() or \"pixel threshold\" as in mask = (mask > threshold).astype(int). If label threshold, what is your pixel threshold?
I also added a second head to your GitHub Unet model. I added it after the segmentation output, I should try to add it after the encoder and see if that works better. A picture of my design is here
", "votes": null, "replies": [ { "id": 677413, "author_name": "pavel92", "author_url": "", "post_date": "11/20/2019 05:41:34", "content": "Thanks!\nTo select images for the second stage I use max pixel value from mask output on first stage. Actually, at first, I was think that first stage will be enough and create a submission with only these models, but get not satisfied result. After that I decided to train stage 2 models and replace masks from that submission with new ones (so, may be using label output can give better score, but I did not have time and submissions to test it).\nIn step two I used just simple unet without classification head.
Interesting. So you don't use the classification output from Stage 1. Then is the only purpose of the classification head to back propagate label error into the encoder? Or does it server some other purpose that I'm missing?
", "votes": null, "replies": [] }, { "id": 679407, "author_name": "pavel92", "author_url": "", "post_date": "11/22/2019 18:45:20", "content": "You are right, but also decoder output is multiplied by 'label' output to generate mask (look at the pic.).
", "votes": null, "replies": [] }, { "id": 680925, "author_name": "hustkevin1037", "author_url": "", "post_date": "11/25/2019 12:21:20", "content": "Congrats！one question, In step 2 , for every class, you trained a binary-seg model or train a seg model for all class
", "votes": null, "replies": [] }, { "id": 681940, "author_name": "pavel92", "author_url": "", "post_date": "11/26/2019 18:02:55", "content": "step 2 - separate binary segmentation model for each class
", "votes": null, "replies": [] } ] }, { "id": 677012, "author_name": "mystery", "author_url": "", "post_date": "11/19/2019 17:54:59", "content": "That's cool!\nFast track to medal zone is using segmentation_models lib
", "votes": null, "replies": [] }, { "id": 677428, "author_name": "sleepysleeping", "author_url": "", "post_date": "11/20/2019 06:18:04", "content": "Congrats and thanks for sharing!! Great approach!!
", "votes": null, "replies": [] }, { "id": 679699, "author_name": "deividasmataciunas", "author_url": "", "post_date": "11/23/2019 07:34:30", "content": "Thank you , very useful
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "676474": "## Easy silver in last days\nI have adopted my pipeline from Severstal Defect Detection and was able to get silver medal in last two days with just 6 submissions, here is a short description of 55th place solution.\n\n2 step pileline\n 1) Multi-task network (classification + segmentation) as classifier to remove empty masks\n 2) Binary segmentation for each class\n \n### 1st step.\nI have trained 5-fold `FPN(resnet34) + aux classfication output` on 480x640 images using `Flip`, `RandomBrightness` as augmentations. Model trained just 6-7 epochs and than starts to overfit, I do nothing with that, just save top 5 checkpoints according to metric.\n\nLoss (segmentation head): bce+dice\nLoss (classification head): bce\nOptimizer: AdamW\nPostprocessing: remove masks less than 10000 pixels\nThresholds: [0.6, 0.6, 0.6, 0.6]\n\n\n### 2nd step.\nFor each class trained `2 x Unet(se_resnext50_34x4d)` only on images with masks of that class!\nwith same optimizer, image size and augmentations.\n\nLoss: bce+dice\nThresholds: [0.4, 0.4, 0.4, 0.4]\n\n### Ensemble\nFor all models made checkpoints weights! averaging (+0.005-0.01 on validation).\nModels over each stage have been just averaged with Flip TTA.\n\n### Useful links\n - Segmentation Models: https://github.com/qubvel/segmentation_models.pytorch\n - Test Time Augmentation for PyTorch: https://github.com/qubvel/ttach\n\n**And congratulations to winners!**", "676481": "Sweet and Simple!! Thank you for sharing and also thank you for your `segmentation_models.pytorch` library. Extremely helpful and convenient", "676482": "most of the time i use your library,thanks a lot and congratulations", "676523": "Congratulations\nThank you for Sharing your Insights & Approach! @pavel92", "676551": "pavel92 thanks for your solution writeup. This was my first computer vision competition and I used only your library 'segmentation models' to implement my solution. I must say this library is really helpful for beginners like me. After reading the comments of top solutions, I came to know about the term 'weights averaging from different checkpoints'. Will you please share some resources to understand this concept and may be a sample boiler-plate implementation of this idea?", "676573": "```\nimport torch\nfrom collections import OrderedDict\nfrom typing import List\n\ncheckpoints_weights_paths: List[str] = ... # sorted in descending order by score\nmodel: torch.nn.Module = ...\n\n\ndef average_weights(state_dicts: List[dict]):\n everage_dict = OrderedDict()\n for k in state_dicts[0].keys():\n everage_dict[k] = sum([state_dict[k] for state_dict in state_dicts]) / len(state_dicts)\n return everage_dict\n\n\nall_weights = [torch.load(path) for path in checkpoints_weights_paths]\n\nbest_score = 0\nbest_weights = []\n\nfor w in all_weights:\n current_weights = best_weights + [w]\n average_dict = average_weights(current_weights)\n model.load_state_dict(average_dict)\n score = evaluate_model(model, ...)\n if score > best_score:\n best_score = score\n best_weights.append(w)\n```\n\nhttps://gist.github.com/qubvel/70c3d5e4cddcde731408f478e12ef87b", "676590": "Very impressive!!! Congratulations 🎉 and thanks for sharing ❤️", "676608": "Thanks a lot :)", "676609": "Pavel, can you please leave here some info (maybe links to read) about such multi-task network and how to implement it?", "676632": "I have prepared new feature for SMP library that add aux output for models (look at https://github.com/qubvel/segmentation_models.pytorch/tree/models-refactoring)\n\nAccording to this implementation my multi-task network defined as follows:\n```\nclass GatedFPN(smp.FPN):\n\n def forward(self, x):\n mask, label = super().forward(x)\n return dict(\n mask=mask*label.reshape(*label.size(), 1, 1),\n label=label,\n )\n\naux_params = dict(classes=4, activation='sigmoid', dropout=0.5, pooling='avg')\nmodel = GatedFPN('resnet34', encoder_weights='imagenet', classes=4, activation='sigmoid', aux_params=aux_params)\n```", "676661": "strong man, congratulationns qubvel", "676663": "Congratulations @pavel92 , \n\nIt's amazing how some people can get such great results with just a few days and a pipeline simples as that.\n\nI was planning to use AdamW as well, but did not have the time, if you used an implementation from a Git repository do you mind sharing?\n\nAlso thanks for your amazing Git repositories.", "676670": "If you use pytorch, then AdamW is already implemented [here](https://pytorch.org/docs/stable/optim.html#torch.optim.AdamW)", "676882": "Thanks @bibek777 , but I mainly use Keras 😄", "676885": "Very! useful. Specialty TTAch ...", "676997": "Congrats Qubvel. Simple and effective. \n \nIn step 2, do you use multi-task networks (two headed models) again? In steps 1 and 2 does threshold refer to \"label threshold\" as in `if label < threshold: mask = np.zeros()` or \"pixel threshold\" as in `mask = (mask > threshold).astype(int)`. If label threshold, what is your pixel threshold?\n\nI also added a second head to your GitHub Unet model. I added it after the segmentation output, I should try to add it after the encoder and see if that works better. A picture of my design is [here][1]\n\n[1]: https://www.kaggle.com/c/understanding_cloud_organization/discussion/118086", "677012": "That's cool!\nFast track to medal zone is using [segmentation_models](https://github.com/qubvel/segmentation_models.pytorch/https://github.com/qubvel/segmentation_models.pytorch/) lib", "677413": "Thanks!\nTo select images for the second stage I use max pixel value from `mask` output on first stage. Actually, at first, I was think that first stage will be enough and create a submission with only these models, but get not satisfied result. After that I decided to train stage 2 models and replace masks from that submission with new ones (so, may be using label output can give better score, but I did not have time and submissions to test it).\nIn step two I used just simple unet without classification head.", "677428": "Congrats and thanks for sharing!! Great approach!!", "677467": "I use pytorch optimizers", "678097": "dimitreoliveira AdamW saved me in this competition. Few breakthrough I got in this competition are:\nMy network - 2 X FPN EfficientNetB1 + 2 X FPN EfficientNetB2 + 4 Classifiers(2 each of EfficientNetB2 and EfficientNetB3)\n1. Changed optimizer from Adam to AdamW and did not change anything - jumped from top 48% to top 15%.\n2. Increased min pixel area from 2500 to 10000 - jumped from top 15% to top 8%.\n3. Increased min pixel area from 10000 to 15000 - jumped from top 8% to top 4%.\n\nAfter that I tried many other methods but did not work.", "678433": "Thanks @raghaw , your results were very impressive, I was planning to experiment with AdamW on the last 2 weeks, but with all the ensembling I ended not having the time.\n\nIf you don't mind what LR Scheduler were you using? and if was Keras/Tensorflow, do you mind sharing your source/implementation of AdamW?", "678550": "dimitreoliveira, I used ExponentialLR scheduler of pytorch with gamma=0.80 and initial learning rate 5e-4. AdamW is already in pytorch as mentioned by @bibek777 so I did not face any problem. I found AdamW for keras version on github hope it will be helpfull to you.\nhttps://github.com/GLambard/AdamW_Keras", "678573": "Thanks again @raghaw , about the repository it seems that it has [some issues](https://github.com/GLambard/AdamW_Keras/issues/2), but the topic author linked a more reliable source.", "678626": "dimitreoliveira I was reading this blog:\nhttps://www.iprally.com/news/recent-improvements-to-the-adam-optimizer\nIn AdamW section they have given hyperlink of its tensorflow implementation. It is already added in tensorflow. check this link.\nhttps://www.tensorflow.org/versions/r1.15/api_docs/python/tf/contrib/opt/AdamWOptimizer", "678661": "Nice, thanks a lot!", "678707": "Interesting. So you don't use the classification output from Stage 1. Then is the only purpose of the classification head to back propagate label error into the encoder? Or does it server some other purpose that I'm missing?", "679407": "You are right, but also decoder output is multiplied by 'label' output to generate mask (look at the pic.).", "679699": "Thank you , very useful", "679712": "p.s. merged to master", "680925": "Congrats！one question, In step 2 , for every class, you trained a binary-seg model or train a seg model for all class", "681940": "step 2 - separate binary segmentation model for each class" }, "source": "meta" }