TL;DR:\n- Siamese architecture\n- Metric learning featuring brand-new CVPR 2019 method (will be published soon)\n- Classification on features\n- Large blend for new whale/not new whale binary classification
\n\nFirst of all, I would like to thank the competition hosts for such an amazing competition. Also, special thanks to all of my teammates for the hard work during the competition.
\n\nOur solution features three main approaches.
\n\nThe first one is a Siamese Net, heavily based on Martin’s Piotte kernel in the beginning. However, as it was noted in the Kernels, Martin’s solution out of the box could’ve got you about 0.822 LB.
\n\nWe added lots of augmentations and tried a lot of different branches (mainly ResNet-18, ResNet-34, SE-ResNeXt-50, ResNet-50). We also pursued with a smart flipping strategy, which makes the model differentiate between the left and the right part of the fluke. Specifically, during training, for each pair X, Y of the same whale, we get one more training pair by flipping both of them. On the other hand, if X and Y are different whales, then we can flip any of two images, which therefore gives us 3 more training pairs. Also, we added hard positive mining (basically, LAP solving for the positive pairs as well). LAP strategy was also modified to be able to run on multiple threads. The NN was trained progressively 299->384->512.
\n\nThe best-performing model could get us 0.929 LB, and the ensemble of all gave 0.940.
\n\nAnother solution will be explained later in detail by @asanakoy. In two words, it is metric learning with multiple branches and margin loss, trained on multiple resolution crops using bboxes, grayscale and RGB input images. He also used his brand-new method from CVPR which allowed for 1-2% score boost.
\n\nNote that new whales were removed from training in our approach.
\n\nOur third approach is classification on features. We concatenated all of the features generated by our branch models and trained classification model on top of them. The head of classification was two dense layers with a little dropout. This model allowed us to achieve 0.924 MAP@5.
\n\nFinally, we decided to make gradient boosting to decide, whether or not the whale is new whale or not. To do this, we took our top-performing models and ensembles and took their TOP-4 predictions for each whale. Then, for all of our models, we took their predictions on these set of classes. We used a blend of LogReg, SVM, several KNN models, and LightGBM to solve a binary classification problem. @ppleskov did this very well, which allowed us to discover hard cases of new whales and further boost the score.
\n\nNow, a couple of words on duplicates. As it was noted previously in the discussions, there are lots of (over 46 pairs) of duplicate whale ids. However, my team has managed to find over 106 pairs of duplicates, which affected more than a thousand images.\nNow, I present three strategies that could’ve been used to tackle this problem:
\n\nStrategy 1: the first strategy is to compare the ids amongst the same group and to always put the whale id with the larger count in front.
\n\nStrategy 2 would be to place in front whatever your network predicts to be the first, and then just put the rest afterwards.
\n\nStrategy 3: just let the net decide and don’t modify the submission files.
\n\nNote that in both strategies, we always put the whale ids that belong to the same group one after another, which is completely logical.\nFor us, strategy 2 worked the best and gave 0.0002 LB improvement. Also, when we obtained a new set of duplicate images, we modified our training labels not to confuse the network.
\n\nFinally, I want to congratulate all people who achieved what they wanted and thank the Kaggle and ODS.ai community for one more amazing experience!
", "messages": [ { "id": "481355", "postDate": "03/01/2019 09:53:54", "content": "TL;DR:\n- Siamese architecture\n- Metric learning featuring brand-new CVPR 2019 method (will be published soon)\n- Classification on features\n- Large blend for new whale/not new whale binary classification
\n\nFirst of all, I would like to thank the competition hosts for such an amazing competition. Also, special thanks to all of my teammates for the hard work during the competition.
\n\nOur solution features three main approaches.
\n\nThe first one is a Siamese Net, heavily based on Martin’s Piotte kernel in the beginning. However, as it was noted in the Kernels, Martin’s solution out of the box could’ve got you about 0.822 LB.
\n\nWe added lots of augmentations and tried a lot of different branches (mainly ResNet-18, ResNet-34, SE-ResNeXt-50, ResNet-50). We also pursued with a smart flipping strategy, which makes the model differentiate between the left and the right part of the fluke. Specifically, during training, for each pair X, Y of the same whale, we get one more training pair by flipping both of them. On the other hand, if X and Y are different whales, then we can flip any of two images, which therefore gives us 3 more training pairs. Also, we added hard positive mining (basically, LAP solving for the positive pairs as well). LAP strategy was also modified to be able to run on multiple threads. The NN was trained progressively 299->384->512.
\n\nThe best-performing model could get us 0.929 LB, and the ensemble of all gave 0.940.
\n\nAnother solution will be explained later in detail by @asanakoy. In two words, it is metric learning with multiple branches and margin loss, trained on multiple resolution crops using bboxes, grayscale and RGB input images. He also used his brand-new method from CVPR which allowed for 1-2% score boost.
\n\nNote that new whales were removed from training in our approach.
\n\nOur third approach is classification on features. We concatenated all of the features generated by our branch models and trained classification model on top of them. The head of classification was two dense layers with a little dropout. This model allowed us to achieve 0.924 MAP@5.
\n\nFinally, we decided to make gradient boosting to decide, whether or not the whale is new whale or not. To do this, we took our top-performing models and ensembles and took their TOP-4 predictions for each whale. Then, for all of our models, we took their predictions on these set of classes. We used a blend of LogReg, SVM, several KNN models, and LightGBM to solve a binary classification problem. @ppleskov did this very well, which allowed us to discover hard cases of new whales and further boost the score.
\n\nNow, a couple of words on duplicates. As it was noted previously in the discussions, there are lots of (over 46 pairs) of duplicate whale ids. However, my team has managed to find over 106 pairs of duplicates, which affected more than a thousand images.\nNow, I present three strategies that could’ve been used to tackle this problem:
\n\nStrategy 1: the first strategy is to compare the ids amongst the same group and to always put the whale id with the larger count in front.
\n\nStrategy 2 would be to place in front whatever your network predicts to be the first, and then just put the rest afterwards.
\n\nStrategy 3: just let the net decide and don’t modify the submission files.
\n\nNote that in both strategies, we always put the whale ids that belong to the same group one after another, which is completely logical.\nFor us, strategy 2 worked the best and gave 0.0002 LB improvement. Also, when we obtained a new set of duplicate images, we modified our training labels not to confuse the network.
\n\nFinally, I want to congratulate all people who achieved what they wanted and thank the Kaggle and ODS.ai community for one more amazing experience!
", "rawMarkdown": "TL;DR:\n- Siamese architecture\n- Metric learning featuring brand-new CVPR 2019 method (will be published soon)\n- Classification on features\n- Large blend for new whale/not new whale binary classification\n\nFirst of all, I would like to thank the competition hosts for such an amazing competition. Also, special thanks to all of my teammates for the hard work during the competition.\n\nOur solution features three main approaches.\n\nThe first one is a Siamese Net, heavily based on Martin’s Piotte kernel in the beginning. However, as it was noted in the Kernels, Martin’s solution out of the box could’ve got you about 0.822 LB.\n\nWe added lots of augmentations and tried a lot of different branches (mainly ResNet-18, ResNet-34, SE-ResNeXt-50, ResNet-50). We also pursued with a smart flipping strategy, which makes the model differentiate between the left and the right part of the fluke. Specifically, during training, for each pair X, Y of the same whale, we get one more training pair by flipping **both** of them. On the other hand, if X and Y are different whales, then we can flip any of two images, which therefore gives us 3 more training pairs. Also, we added hard positive mining (basically, LAP solving for the positive pairs as well). LAP strategy was also modified to be able to run on multiple threads. The NN was trained progressively 299->384->512.\n\n\nThe best-performing model could get us 0.929 LB, and the ensemble of all gave 0.940.\n\nAnother solution will be explained later in detail by @asanakoy. In two words, it is metric learning with multiple branches and margin loss, trained on multiple resolution crops using bboxes, grayscale and RGB input images. He also used his brand-new method from CVPR which allowed for 1-2% score boost.\n\nNote that new whales were removed from training in our approach.\n\nOur third approach is classification on features. We concatenated all of the features generated by our branch models and trained **classification** model on top of them. The head of classification was two dense layers with a little dropout. This model allowed us to achieve 0.924 MAP@5.\n\nFinally, we decided to make gradient boosting to decide, whether or not the whale is new whale or not. To do this, we took our top-performing models and ensembles and took their TOP-4 predictions for each whale. Then, for all of our models, we took their predictions on these set of classes. We used a blend of LogReg, SVM, several KNN models, and LightGBM to solve a binary classification problem. @ppleskov did this very well, which allowed us to discover hard cases of new whales and further boost the score. \n\nNow, a couple of words on duplicates. As it was noted previously in the discussions, there are lots of (over 46 pairs) of duplicate whale ids. However, my team has managed to find over 106 pairs of duplicates, which affected **more than a thousand images**.\nNow, I present three strategies that could’ve been used to tackle this problem:\n\n\n**Strategy 1:** the first strategy is to compare the ids amongst the same group and to always put the whale id with the larger count in front.\n\n**Strategy 2** would be to place in front whatever your network predicts to be the first, and then just put the rest afterwards.\n\n**Strategy 3:** just let the net decide and don’t modify the submission files.\n\nNote that in both strategies, we always put the whale ids that belong to the same group one after another, which is completely logical.\nFor us, strategy 2 worked the best and gave 0.0002 LB improvement. Also, when we obtained a new set of duplicate images, we modified our training labels not to confuse the network.\n\n\nFinally, I want to congratulate all people who achieved what they wanted and thank the Kaggle and ODS.ai community for one more amazing experience!", "votes": null }, { "id": "481460", "postDate": "03/01/2019 12:31:53", "content": "Congrats @vshakhray and team. Thanks for sharing.
", "rawMarkdown": "Congrats @vshakhray and team. Thanks for sharing.", "votes": null }, { "id": "481834", "postDate": "03/01/2019 22:38:22", "content": "Siamese again, congrats and thanks for sharing!
", "rawMarkdown": "Siamese again, congrats and thanks for sharing!", "votes": null }, { "id": "482036", "postDate": "03/02/2019 08:17:35", "content": "Wow, 106 pairs of duplicates is a very impressive finding. could you publish the list?
", "rawMarkdown": "Wow, 106 pairs of duplicates is a very impressive finding. could you publish the list?", "votes": null }, { "id": "482086", "postDate": "03/02/2019 10:01:57", "content": "Great idea! I've shared them in another thread: https://www.kaggle.com/c/humpback-whale-identification/discussion/82557. Also, I'm attaching it to the original post.
", "rawMarkdown": "Great idea! I've shared them in another thread: https://www.kaggle.com/c/humpback-whale-identification/discussion/82557. Also, I'm attaching it to the original post.", "votes": null }, { "id": "482102", "postDate": "03/02/2019 10:36:59", "content": "Thank you, very well done!
", "rawMarkdown": "Thank you, very well done!", "votes": null }, { "id": "571636", "postDate": "07/09/2019 21:44:51", "content": "Our CVPR 2019 paper which we used in the solution + code: \nDivide and Conquer the Embedding Space for Metric Learning\nAuthors: Artsiom Sanakoyeu, Vadim Tschernezki, Uta Büchler, Björn Ommer
\n\nhttps://arxiv.org/abs/1906.05990\nhttps://github.com/CompVis/metric-learning-divide-and-conquer
\n\nSlides of the solution: https://slides.com/asanakoy/metric-learning-kaggle-whales
", "rawMarkdown": "Our **CVPR 2019 paper** which we used in the solution + **code**: \n**Divide and Conquer the Embedding Space for Metric Learning**\nAuthors: Artsiom Sanakoyeu, Vadim Tschernezki, Uta Büchler, Björn Ommer \n\nhttps://arxiv.org/abs/1906.05990\nhttps://github.com/CompVis/metric-learning-divide-and-conquer\n\nSlides of the solution: https://slides.com/asanakoy/metric-learning-kaggle-whales", "votes": null } ], "comments": [ { "id": 481460, "author_name": "sheriytm", "author_url": "", "post_date": "03/01/2019 12:31:53", "content": "Congrats @vshakhray and team. Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 481834, "author_name": "titericz", "author_url": "", "post_date": "03/01/2019 22:38:22", "content": "Siamese again, congrats and thanks for sharing!
", "votes": null, "replies": [] }, { "id": 482036, "author_name": "tivfrvqhs5", "author_url": "", "post_date": "03/02/2019 08:17:35", "content": "Wow, 106 pairs of duplicates is a very impressive finding. could you publish the list?
", "votes": null, "replies": [ { "id": 482086, "author_name": "vshakhray", "author_url": "", "post_date": "03/02/2019 10:01:57", "content": "Great idea! I've shared them in another thread: https://www.kaggle.com/c/humpback-whale-identification/discussion/82557. Also, I'm attaching it to the original post.
", "votes": null, "replies": [] }, { "id": 482102, "author_name": "tivfrvqhs5", "author_url": "", "post_date": "03/02/2019 10:36:59", "content": "Thank you, very well done!
", "votes": null, "replies": [] } ] }, { "id": 571636, "author_name": "asanakoev", "author_url": "", "post_date": "07/09/2019 21:44:51", "content": "Our CVPR 2019 paper which we used in the solution + code: \nDivide and Conquer the Embedding Space for Metric Learning\nAuthors: Artsiom Sanakoyeu, Vadim Tschernezki, Uta Büchler, Björn Ommer
\n\nhttps://arxiv.org/abs/1906.05990\nhttps://github.com/CompVis/metric-learning-divide-and-conquer
\n\nSlides of the solution: https://slides.com/asanakoy/metric-learning-kaggle-whales
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "481355": "TL;DR:\n- Siamese architecture\n- Metric learning featuring brand-new CVPR 2019 method (will be published soon)\n- Classification on features\n- Large blend for new whale/not new whale binary classification\n\nFirst of all, I would like to thank the competition hosts for such an amazing competition. Also, special thanks to all of my teammates for the hard work during the competition.\n\nOur solution features three main approaches.\n\nThe first one is a Siamese Net, heavily based on Martin’s Piotte kernel in the beginning. However, as it was noted in the Kernels, Martin’s solution out of the box could’ve got you about 0.822 LB.\n\nWe added lots of augmentations and tried a lot of different branches (mainly ResNet-18, ResNet-34, SE-ResNeXt-50, ResNet-50). We also pursued with a smart flipping strategy, which makes the model differentiate between the left and the right part of the fluke. Specifically, during training, for each pair X, Y of the same whale, we get one more training pair by flipping **both** of them. On the other hand, if X and Y are different whales, then we can flip any of two images, which therefore gives us 3 more training pairs. Also, we added hard positive mining (basically, LAP solving for the positive pairs as well). LAP strategy was also modified to be able to run on multiple threads. The NN was trained progressively 299->384->512.\n\n\nThe best-performing model could get us 0.929 LB, and the ensemble of all gave 0.940.\n\nAnother solution will be explained later in detail by @asanakoy. In two words, it is metric learning with multiple branches and margin loss, trained on multiple resolution crops using bboxes, grayscale and RGB input images. He also used his brand-new method from CVPR which allowed for 1-2% score boost.\n\nNote that new whales were removed from training in our approach.\n\nOur third approach is classification on features. We concatenated all of the features generated by our branch models and trained **classification** model on top of them. The head of classification was two dense layers with a little dropout. This model allowed us to achieve 0.924 MAP@5.\n\nFinally, we decided to make gradient boosting to decide, whether or not the whale is new whale or not. To do this, we took our top-performing models and ensembles and took their TOP-4 predictions for each whale. Then, for all of our models, we took their predictions on these set of classes. We used a blend of LogReg, SVM, several KNN models, and LightGBM to solve a binary classification problem. @ppleskov did this very well, which allowed us to discover hard cases of new whales and further boost the score. \n\nNow, a couple of words on duplicates. As it was noted previously in the discussions, there are lots of (over 46 pairs) of duplicate whale ids. However, my team has managed to find over 106 pairs of duplicates, which affected **more than a thousand images**.\nNow, I present three strategies that could’ve been used to tackle this problem:\n\n\n**Strategy 1:** the first strategy is to compare the ids amongst the same group and to always put the whale id with the larger count in front.\n\n**Strategy 2** would be to place in front whatever your network predicts to be the first, and then just put the rest afterwards.\n\n**Strategy 3:** just let the net decide and don’t modify the submission files.\n\nNote that in both strategies, we always put the whale ids that belong to the same group one after another, which is completely logical.\nFor us, strategy 2 worked the best and gave 0.0002 LB improvement. Also, when we obtained a new set of duplicate images, we modified our training labels not to confuse the network.\n\n\nFinally, I want to congratulate all people who achieved what they wanted and thank the Kaggle and ODS.ai community for one more amazing experience!", "481460": "Congrats @vshakhray and team. Thanks for sharing.", "481834": "Siamese again, congrats and thanks for sharing!", "482036": "Wow, 106 pairs of duplicates is a very impressive finding. could you publish the list?", "482086": "Great idea! I've shared them in another thread: https://www.kaggle.com/c/humpback-whale-identification/discussion/82557. Also, I'm attaching it to the original post.", "482102": "Thank you, very well done!", "571636": "Our **CVPR 2019 paper** which we used in the solution + **code**: \n**Divide and Conquer the Embedding Space for Metric Learning**\nAuthors: Artsiom Sanakoyeu, Vadim Tschernezki, Uta Büchler, Björn Ommer \n\nhttps://arxiv.org/abs/1906.05990\nhttps://github.com/CompVis/metric-learning-divide-and-conquer\n\nSlides of the solution: https://slides.com/asanakoy/metric-learning-kaggle-whales" }, "source": "meta" }