During this competition, we have seen several approaches to preprocess the competition's dataset. It includes full body cropping, back fin cropping and its variant with the removed background.
\nFull body and back fin cropping approaches are very valuable independently of each other. We have made that conclusion when we have found that the model trained on the back fin dataset was better when trained on the bigger image sizes with optimal image size around 768x768. So there is no way to train models on full body images while keeping backfin size close to 768x768, and it is also unreasonable to get the final predictions using only back fin images because there are body pigmentations and other body features that can be helpful for identification. That’s why you 100% need to train at least two separate models on full body crops and back fin crops with consequent ensemble mechanisms.
So, we have trained several classifiers on different datasets and ensembled their embeddings using Multi Layer Perceptron (MLP). MLP was chosen because it has a better potential to get insights from embeddings of the models trained on different dataset sources. Specifically, we concatenated features from different models before feeding them to the MLP. It was trained using ArcFace. The model is very shallow and has dropouts at the beginning (0.15) and at the end (0.50). You can find more details in the attached GitHub repository. In general, MLP gave us roughly +5% to the best performing CNN classifier.
\nIn total, we had 8 CNN classifiers that were used in MLP training.
\n| Model | \nTrain set images | \nTest set images | \nLB score | \n
|---|---|---|---|
| efficientnetv1-b7 | \nmix-dataset | \nfull body | \n81.6% | \n
| efficientnetv1-b6 | \nmix-dataset | \nfull body | \n79.8% | \n
| eca_nfnet_l2 | \nmix-dataset | \nfull body | \n79.3% | \n
| eca_nfnet_l2 | \nmix-dataset | \nfull body | \n81.2% | \n
| efficientnetv1-b6 | \nback fin | \nbackfin | \n77.8% | \n
| efficientnetv1-b6 | \nmix-dataset | \nbackfin | \n80.0% | \n
| efficientnetv1-b7 | \noriginal images | \noriginal images | \n66.0% | \n
| efficientnetv1-b7 | \nfull body | \nfull body | \n76.8% | \n
A big improvement came from the thing we called mix-dataset. During the mix-dataset training procedure our data loader was sampling images from different data sources (back fin and full body images), but testing images were always from a single dataset source. Efficientnetv1-b6 trained and tested on only backfin images had 77.8% LB score, but the same model with the same hyperparameters had 80.0% LB score when trained on mix-dataset and tested on backfin images.
\nIn my understanding, such an approach introduced more training diversity and prevented models from overfitting. An overfitting apparently took place in our training procedure because most of the time we saw >99% training accuracy with a tiny training loss.
We had two tricks to improve predictions generated by the MLP model. In the prediction .csv file we had 5 potential match candidates sorted by confidence, from the most confident to the least confident.
\ngithub repo: https://github.com/achilleess/happywhale-2022
\nThis code does not include all of the above-mentioned models. You can find here training procedures for MLP, back fin models, and some of the full body models. The rest of the code may be released later.
Teammates:
\n@neomaoro
\n@chihantsai
\n@atom1231
\n@kzvdar42
During this competition, we have seen several approaches to preprocess the competition's dataset. It includes full body cropping, back fin cropping and its variant with the removed background.
\nFull body and back fin cropping approaches are very valuable independently of each other. We have made that conclusion when we have found that the model trained on the back fin dataset was better when trained on the bigger image sizes with optimal image size around 768x768. So there is no way to train models on full body images while keeping backfin size close to 768x768, and it is also unreasonable to get the final predictions using only back fin images because there are body pigmentations and other body features that can be helpful for identification. That’s why you 100% need to train at least two separate models on full body crops and back fin crops with consequent ensemble mechanisms.
So, we have trained several classifiers on different datasets and ensembled their embeddings using Multi Layer Perceptron (MLP). MLP was chosen because it has a better potential to get insights from embeddings of the models trained on different dataset sources. Specifically, we concatenated features from different models before feeding them to the MLP. It was trained using ArcFace. The model is very shallow and has dropouts at the beginning (0.15) and at the end (0.50). You can find more details in the attached GitHub repository. In general, MLP gave us roughly +5% to the best performing CNN classifier.
\nIn total, we had 8 CNN classifiers that were used in MLP training.
\n| Model | \nTrain set images | \nTest set images | \nLB score | \n
|---|---|---|---|
| efficientnetv1-b7 | \nmix-dataset | \nfull body | \n81.6% | \n
| efficientnetv1-b6 | \nmix-dataset | \nfull body | \n79.8% | \n
| eca_nfnet_l2 | \nmix-dataset | \nfull body | \n79.3% | \n
| eca_nfnet_l2 | \nmix-dataset | \nfull body | \n81.2% | \n
| efficientnetv1-b6 | \nback fin | \nbackfin | \n77.8% | \n
| efficientnetv1-b6 | \nmix-dataset | \nbackfin | \n80.0% | \n
| efficientnetv1-b7 | \noriginal images | \noriginal images | \n66.0% | \n
| efficientnetv1-b7 | \nfull body | \nfull body | \n76.8% | \n
A big improvement came from the thing we called mix-dataset. During the mix-dataset training procedure our data loader was sampling images from different data sources (back fin and full body images), but testing images were always from a single dataset source. Efficientnetv1-b6 trained and tested on only backfin images had 77.8% LB score, but the same model with the same hyperparameters had 80.0% LB score when trained on mix-dataset and tested on backfin images.
\nIn my understanding, such an approach introduced more training diversity and prevented models from overfitting. An overfitting apparently took place in our training procedure because most of the time we saw >99% training accuracy with a tiny training loss.
We had two tricks to improve predictions generated by the MLP model. In the prediction .csv file we had 5 potential match candidates sorted by confidence, from the most confident to the least confident.
\ngithub repo: https://github.com/achilleess/happywhale-2022
\nThis code does not include all of the above-mentioned models. You can find here training procedures for MLP, back fin models, and some of the full body models. The rest of the code may be released later.
Teammates:
\n@neomaoro
\n@chihantsai
\n@atom1231
\n@kzvdar42
Congrats on your first kaggle (silver ?) medal and thanks for your wonderful work !!!
", "rawMarkdown": "Congrats on your first kaggle (silver ?) medal and thanks for your wonderful work !!!", "votes": null }, { "id": "1760146", "postDate": "04/19/2022 05:05:59", "content": "Congrats! Appreciate great effort you have made to this comp!!
", "rawMarkdown": "Congrats! Appreciate great effort you have made to this comp!!", "votes": null }, { "id": "1760919", "postDate": "04/19/2022 15:35:11", "content": "Great job! I will learn your sharing, especially MLP
", "rawMarkdown": "Great job! I will learn your sharing, especially MLP", "votes": null }, { "id": "1761675", "postDate": "04/20/2022 04:45:59", "content": "Great job @achilles38
", "rawMarkdown": "Great job @achilles38", "votes": null }, { "id": "1761800", "postDate": "04/20/2022 07:24:59", "content": "Congrats! I train efficientnetv1-b5 with image size 512 on backfin dataset, also get nearly 80.0% score on public LB. It's amazing that MLP ensembling has such a potential !
", "rawMarkdown": "Congrats! I train efficientnetv1-b5 with image size 512 on backfin dataset, also get nearly 80.0% score on public LB. It's amazing that MLP ensembling has such a potential !", "votes": null } ], "comments": [ { "id": 1760113, "author_name": "atom1231", "author_url": "", "post_date": "04/19/2022 04:31:59", "content": "Congrats on your first kaggle (silver ?) medal and thanks for your wonderful work !!!
", "votes": null, "replies": [] }, { "id": 1760146, "author_name": "neomaoro", "author_url": "", "post_date": "04/19/2022 05:05:59", "content": "Congrats! Appreciate great effort you have made to this comp!!
", "votes": null, "replies": [] }, { "id": 1760919, "author_name": "autoatom", "author_url": "", "post_date": "04/19/2022 15:35:11", "content": "Great job! I will learn your sharing, especially MLP
", "votes": null, "replies": [] }, { "id": 1761675, "author_name": "ishanmehta115", "author_url": "", "post_date": "04/20/2022 04:45:59", "content": "Great job @achilles38
", "votes": null, "replies": [] }, { "id": 1761800, "author_name": "jisongxie", "author_url": "", "post_date": "04/20/2022 07:24:59", "content": "Congrats! I train efficientnetv1-b5 with image size 512 on backfin dataset, also get nearly 80.0% score on public LB. It's amazing that MLP ensembling has such a potential !
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1760097": "## Intro\nDuring this competition, we have seen several approaches to preprocess the competition's dataset. It includes full body cropping, back fin cropping and its variant with the removed background.\nFull body and back fin cropping approaches are very valuable independently of each other. We have made that conclusion when we have found that the model trained on the back fin dataset was better when trained on the bigger image sizes with optimal image size around 768x768. So there is no way to train models on full body images while keeping backfin size close to 768x768, and it is also unreasonable to get the final predictions using only back fin images because there are body pigmentations and other body features that can be helpful for identification. That’s why you 100% need to train at least two separate models on full body crops and back fin crops with consequent ensemble mechanisms.\n\n## MLP\nSo, we have trained several classifiers on different datasets and ensembled their embeddings using Multi Layer Perceptron (MLP). MLP was chosen because it has a better potential to get insights from embeddings of the models trained on different dataset sources. Specifically, we concatenated features from different models before feeding them to the MLP. It was trained using ArcFace. The model is very shallow and has dropouts at the beginning (0.15) and at the end (0.50). You can find more details in the attached GitHub repository. In general, MLP gave us roughly +5% to the best performing CNN classifier.\n\n## CNNs\nIn total, we had 8 CNN classifiers that were used in MLP training.\n\n| Model | Train set images | Test set images | LB score |\n| --- | --- | --- | --- |\n| efficientnetv1-b7 | mix-dataset | full body | 81.6% |\n| efficientnetv1-b6 | mix-dataset | full body | 79.8% |\n| eca_nfnet_l2 | mix-dataset | full body | 79.3% |\n| eca_nfnet_l2 | mix-dataset| full body | 81.2% |\n| efficientnetv1-b6 | back fin | backfin | 77.8% |\n| efficientnetv1-b6 | mix-dataset | backfin | 80.0% |\n| efficientnetv1-b7 | original images | original images | 66.0% |\n| efficientnetv1-b7 | full body | full body | 76.8% |\n\n## Mix-dataset\nA big improvement came from the thing we called mix-dataset. During the mix-dataset training procedure our data loader was sampling images from different data sources (back fin and full body images), but testing images were always from a single dataset source. Efficientnetv1-b6 trained and tested on only backfin images had 77.8% LB score, but the same model with the same hyperparameters had 80.0% LB score when trained on mix-dataset and tested on backfin images. \nIn my understanding, such an approach introduced more training diversity and prevented models from overfitting. An overfitting apparently took place in our training procedure because most of the time we saw >99% training accuracy with a tiny training loss. \n\n## Class balancing\nWe had two tricks to improve predictions generated by the MLP model. In the prediction .csv file we had 5 potential match candidates sorted by confidence, from the most confident to the least confident.\n1. In each row, If we see the first element which was also the first element in more than 10 other rows, and the second element which was never the first in other rows, then we swap them.\n [ind_1, ind_2, ind_3, …] -> [ind_2, ind_1, ind_3, …]\n2. If the first element in the row is new_individual and the second element in the row is the element which was never the first in other rows, then we swap them.\n[new_ind, ind_1, ind_2, …] -> [ind_1, new_ind, ind_2, …]\nMLP without class balancing had 0.865 LB score and MLP with class balancing had 0.868 LB score.\n\ngithub repo: [https://github.com/achilleess/happywhale-2022](https://github.com/achilleess/happywhale-2022)\nThis code does not include all of the above-mentioned models. You can find here training procedures for MLP, back fin models, and some of the full body models. The rest of the code may be released later.\n\nTeammates:\n@neomaoro \n@chihantsai\n@atom1231\n@kzvdar42", "1760113": "Congrats on your first kaggle (silver ?) medal and thanks for your wonderful work !!!", "1760146": "Congrats! Appreciate great effort you have made to this comp!!", "1760919": "Great job! I will learn your sharing, especially MLP", "1761675": "Great job @achilles38", "1761800": "Congrats! I train efficientnetv1-b5 with image size 512 on backfin dataset, also get nearly 80.0% score on public LB. It's amazing that MLP ensembling has such a potential !" }, "source": "meta" }