![\"\"](\"https://i.imgur.com/gH2ONQh.png\")
Thanks Kaggle for this exciting competition and our team ( @dathudeptrai @mcggood @akensert @theoviel @ratthachat ) congratulate all winners -- we have learned a lot from this competition and all winners’ solutions !!
\nFrom the winner solutions, it turns out there are mainly 4 ways to break 0.95x
\nWe actually tried the first three but unfortunately could not really make them work effectively.
\nHere, as an alternative solution, we would love to share our own solution which is able to reach 0.943 Private.
Our baseline models score slightly above 0.8. We adopt an audio tagging approach using a densenet121 backbone and the BCE loss.
\nOur training pipeline includes the following tricks :
\nBecause of the small amount of data, models overfit quickly. To solve this problem,two options were using external data and cleverly pretraining our models. Unlike a lot of competitors, we focused on self-pretraining techniques : @dathudeptrai tried auto-encoders, GANs, SimCLR, Cola, and Supervised Contrastive Learning which ultimately was the only thing to work.
\n
Our sampling strategy consists of randomly selecting a crop containing the label. Most of the time, crops are bigger than labels which introduces false positives. One idea to make full use of our windows was to adapt cutmix to concatenate samples such that labels are entirely kept (when possible).
\nDomain shift always exists in deep learning, in both practice and kaggle challenges, especially for small data. Therefore, domain generalization techniques should help with robustness. We applied a simple yet effective technique called Mixstyle.
\nDuration of species’ call varies quite a lot. For example, for class 3 it is around 0.7 seconds while for class 23 is around 8 seconds. To use this prior information, we use multiple window sizes (instead of using a single one). For each class, we choose the one that yields the best CV. In case we have multiple window sizes reaching the maximum, we take the largest window. Although our CV setup which consists of crops centered around the labels did not correlate really well with LB, the 2% CV improvement reflected on LB quite well.
\nWe used the following assumption to improve the training of our models :
\nFor a given recording, if a species has a FP and no TP, then it is not in the call. Our BCE was then updated to make sure the model predicts 0 for such species.
Our best single model densenet121 scores around 0.92 public and 0.93 private. Averaging some models with different backbones, we were able to reach 0.937. We tried many different ensembling, scale fixing and post-processing ideas, and were able to improve our score a bit, but unfortunately we could not find the real magic.
\nIn the end, we empirically analyzed the most uncertain class predictions from our best models, and averaged predictions with other (weaker) models. We relied on diversity to make our submission more robust. Our final ensemble scored public 0.942 and private 0.943.
\nThanks Kaggle for this exciting competition and our team ( @dathudeptrai @mcggood @akensert @theoviel @ratthachat ) congratulate all winners -- we have learned a lot from this competition and all winners’ solutions !!
\nFrom the winner solutions, it turns out there are mainly 4 ways to break 0.95x
\nWe actually tried the first three but unfortunately could not really make them work effectively.
\nHere, as an alternative solution, we would love to share our own solution which is able to reach 0.943 Private.
Our baseline models score slightly above 0.8. We adopt an audio tagging approach using a densenet121 backbone and the BCE loss.
\nOur training pipeline includes the following tricks :
\nBecause of the small amount of data, models overfit quickly. To solve this problem,two options were using external data and cleverly pretraining our models. Unlike a lot of competitors, we focused on self-pretraining techniques : @dathudeptrai tried auto-encoders, GANs, SimCLR, Cola, and Supervised Contrastive Learning which ultimately was the only thing to work.
\n
Our sampling strategy consists of randomly selecting a crop containing the label. Most of the time, crops are bigger than labels which introduces false positives. One idea to make full use of our windows was to adapt cutmix to concatenate samples such that labels are entirely kept (when possible).
\nDomain shift always exists in deep learning, in both practice and kaggle challenges, especially for small data. Therefore, domain generalization techniques should help with robustness. We applied a simple yet effective technique called Mixstyle.
\nDuration of species’ call varies quite a lot. For example, for class 3 it is around 0.7 seconds while for class 23 is around 8 seconds. To use this prior information, we use multiple window sizes (instead of using a single one). For each class, we choose the one that yields the best CV. In case we have multiple window sizes reaching the maximum, we take the largest window. Although our CV setup which consists of crops centered around the labels did not correlate really well with LB, the 2% CV improvement reflected on LB quite well.
\nWe used the following assumption to improve the training of our models :
\nFor a given recording, if a species has a FP and no TP, then it is not in the call. Our BCE was then updated to make sure the model predicts 0 for such species.
Our best single model densenet121 scores around 0.92 public and 0.93 private. Averaging some models with different backbones, we were able to reach 0.937. We tried many different ensembling, scale fixing and post-processing ideas, and were able to improve our score a bit, but unfortunately we could not find the real magic.
\nIn the end, we empirically analyzed the most uncertain class predictions from our best models, and averaged predictions with other (weaker) models. We relied on diversity to make our submission more robust. Our final ensemble scored public 0.942 and private 0.943.
\nWow interesting solution! Good work and thanks for sharing @dathudeptrai
", "rawMarkdown": "Wow interesting solution! Good work and thanks for sharing @dathudeptrai", "votes": null }, { "id": "1211561", "postDate": "02/20/2021 10:29:45", "content": "Thanks @duykhanh99 !
", "rawMarkdown": "Thanks @duykhanh99 !", "votes": null }, { "id": "1211576", "postDate": "02/20/2021 10:37:05", "content": "Thanks @dathudeptrai @mcggood @akensert @ratthachat for the nice competition, it was a pleasure !
", "rawMarkdown": "Thanks @dathudeptrai @mcggood @akensert @ratthachat for the nice competition, it was a pleasure !", "votes": null }, { "id": "1219715", "postDate": "02/27/2021 07:00:25", "content": "Thanks for sharing the code, I have starred!
", "rawMarkdown": "Thanks for sharing the code, I have starred!", "votes": null } ], "comments": [ { "id": 1211556, "author_name": "duykhanh99", "author_url": "", "post_date": "02/20/2021 10:26:29", "content": "Wow interesting solution! Good work and thanks for sharing @dathudeptrai
", "votes": null, "replies": [ { "id": 1211561, "author_name": "theoviel", "author_url": "", "post_date": "02/20/2021 10:29:45", "content": "Thanks @duykhanh99 !
", "votes": null, "replies": [] } ] }, { "id": 1211576, "author_name": "theoviel", "author_url": "", "post_date": "02/20/2021 10:37:05", "content": "Thanks @dathudeptrai @mcggood @akensert @ratthachat for the nice competition, it was a pleasure !
", "votes": null, "replies": [] }, { "id": 1219715, "author_name": "wubinbai", "author_url": "", "post_date": "02/27/2021 07:00:25", "content": "Thanks for sharing the code, I have starred!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1211554": "### Introduction\n\nThanks Kaggle for this exciting competition and our team ( @dathudeptrai @mcggood @akensert @theoviel @ratthachat ) congratulate all winners -- we have learned a lot from this competition and all winners’ solutions !!\n\nFrom the winner solutions, it turns out there are mainly 4 ways to break 0.95x\n\n1. Masked loss\n2. Post-processing\n3. Pseudo-labelling\n4. Extra labeling\n\n\nWe actually tried the first three but unfortunately could not really make them work effectively.\nHere, as an alternative solution, we would love to share our own solution which is able to reach 0.943 Private. \n\n### Training pipeline (improve from 0.80x baseline to 0.92x)\n\n#### Baseline\n\nOur baseline models score slightly above 0.8. We adopt an audio tagging approach using a densenet121 backbone and the BCE loss. \n\nOur training pipeline includes the following tricks :\n \n- Class-balanced sampling, using 8 classes per batch. Batch sizes were usually 64, and 32 for bigger models.\n- Cyclical learning rate with min_lr is 0.0001 and max_lr is 0.001, step_size is 3 epochs. We train models for 100 epochs with early stopping. \n- LookAhead with Adam optimizer (sync_period is 10 and slow_step_size is 0.5)\n\n#### Pretraining with Supervised contrastive Learning (SCL) [0.81x -> 0.85x]\n\nBecause of the small amount of data, models overfit quickly. To solve this problem,two options were using external data and cleverly pretraining our models. Unlike a lot of competitors, we focused on self-pretraining techniques : @dathudeptrai tried auto-encoders, GANs, SimCLR, Cola, and [Supervised Contrastive Learning](https://arxiv.org/abs/2004.11362) which ultimately was the only thing to work. \n\n\n#### Non-overlap time Cutmix [0.85x -> 0.88x]\n\n\n\nOur sampling strategy consists of randomly selecting a crop containing the label. Most of the time, crops are bigger than labels which introduces false positives. One idea to make full use of our windows was to adapt cutmix to concatenate samples such that labels are entirely kept (when possible). \n\n#### Domain Generalization with MixStyle [0.88x -> 0.89x]\n\nDomain shift always exists in deep learning, in both practice and kaggle challenges, especially for small data. Therefore, domain generalization techniques should help with robustness. We applied a simple yet effective technique called [Mixstyle](https://openreview.net/pdf?id=6xHJ37MVxxp).\n\n\n#### Multi Scale inference (MSI) [0.89x -> 0.91x]\n\nDuration of species’ call varies quite a lot. For example, for class 3 it is around 0.7 seconds while for class 23 is around 8 seconds. To use this prior information, we use multiple window sizes (instead of using a single one). For each class, we choose the one that yields the best CV. In case we have multiple window sizes reaching the maximum, we take the largest window. Although our CV setup which consists of crops centered around the labels did not correlate really well with LB, the 2% CV improvement reflected on LB quite well.\n\n\n#### Positive learning and Negative learning [0.91x -> 0.92x]\n\nWe used the following assumption to improve the training of our models : \nFor a given recording, if a species has a FP and no TP, then it is not in the call. Our BCE was then updated to make sure the model predicts 0 for such species. \n\n### Ensembling\n\nOur best single model densenet121 scores around 0.92 public and 0.93 private. Averaging some models with different backbones, we were able to reach 0.937. We tried many different ensembling, scale fixing and post-processing ideas, and were able to improve our score a bit, but unfortunately we could not find the real magic.\n\nIn the end, we empirically analyzed the most uncertain class predictions from our best models, and averaged predictions with other (weaker) models. We relied on diversity to make our submission more robust. Our final ensemble scored public 0.942 and private 0.943.\n\n##### Thanks for reading ! \n\n[TensorFlow Code Here](https://github.com/dathudeptrai/rfcx-kaggle)", "1211556": "Wow interesting solution! Good work and thanks for sharing @dathudeptrai", "1211561": "Thanks @duykhanh99 !", "1211576": "Thanks @dathudeptrai @mcggood @akensert @ratthachat for the nice competition, it was a pleasure !", "1219715": "Thanks for sharing the code, I have starred!" }, "source": "meta" }