Congratulations to all the winners. Thanks Happywhale and Kaggle for hosting this competition.
\nMany thanks for the sharing of Lex Toumbourou and Jan bre, I started from their significant work of [Happywhale - Effnet B6 fork with Detic crop] and the dataset of backfintfrecords.
\nMany experements in the competition had been tried and finally only three significant improvements in model.
1.Dynamic margins: publicLB +0.02 (Thanks Landmark team to share the solution, paper is here: https://arxiv.org/abs/2010.05350):
\n Tested and found the best margins range is 0.05-0.6. but training became difficult( loss is easy to Nan) and need to reduce LR.
\n2.Batch normalization: publicLB +0.02 (Thanks Heng, discussion is here: https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/315129 )
\n add Batch normalization layer before dense layer, add L2 normalization for dense:
\n3.FreezeBN: publicLB +0.01(Thanks Balaji, discussion is here: https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/309582 )
\n it works for EfficientB6 and B5, but not works for B7(loss always to nan)
here is a training notebook included above 3 implementations
\nhttps://www.kaggle.com/code/liuzhangzhen/happywhale-bodydataset-training
finally, two ensembles help to improve public LB to 0.858
\n Concatenate 7 model embeddings: improve from LB0.815 to 0.835(body dataset)
\n Combine the two datasets(body and fin) test neighbor and take the max confidence: improve from LB0.835 to 0.858
Here is my final solution:
\n· Dataset: fin and body, use yolov5x6 to train 10folds, test by b0rev256 then pickup best 5 folds, choose the best 1 for train and all others for TTA.(Result: huge effort but slight improvement, maybe should be more carefully to check the dataset and remove noise)
· Split: random choose 500 2 pictures and 55 pictures as new whales(about 11% just same percentage as the LB) as validation dataset and use all others for train. (Result: validation dataset is proved to co-related with LB and useful)
\n· Pseudo: take 4000+ with confidence > 0.95 and all other targets confidence < 0.65
\n· Augmentation: base notebook + 50% rotation, shear, shift, zoom.(Result: slight improvement)
\n· Model: dynamic margins, batch normalization layer before dense layer, add L2 normalization for dense, FreezeBN(Result: significant improvements)
\n · Model candidate: (final 7-8 models for fin and body)
\n · Effv1: b7 768 Adam(use TPUv3-8), b7 608 Adam(Colab), b6 640 Adam(Colab), b5 640 Adam(Colab), b6 480 Adam(Colab), + some SGD model (ex: b7 640)
· Ensemble
\n · TTA:5 fold embeddings and take the mean (small improvement)
\n · Embedding concatenate for body and fin dataset(Large improvement)
\n · Merge fin and body by max conf, leverage by the LB score(use confidence * (LB score**2))(Large improvement)
Congratulations to all the winners. Thanks Happywhale and Kaggle for hosting this competition.
\nMany thanks for the sharing of Lex Toumbourou and Jan bre, I started from their significant work of [Happywhale - Effnet B6 fork with Detic crop] and the dataset of backfintfrecords.
\nMany experements in the competition had been tried and finally only three significant improvements in model.
1.Dynamic margins: publicLB +0.02 (Thanks Landmark team to share the solution, paper is here: https://arxiv.org/abs/2010.05350):
\n Tested and found the best margins range is 0.05-0.6. but training became difficult( loss is easy to Nan) and need to reduce LR.
\n2.Batch normalization: publicLB +0.02 (Thanks Heng, discussion is here: https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/315129 )
\n add Batch normalization layer before dense layer, add L2 normalization for dense:
\n3.FreezeBN: publicLB +0.01(Thanks Balaji, discussion is here: https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/309582 )
\n it works for EfficientB6 and B5, but not works for B7(loss always to nan)
here is a training notebook included above 3 implementations
\nhttps://www.kaggle.com/code/liuzhangzhen/happywhale-bodydataset-training
finally, two ensembles help to improve public LB to 0.858
\n Concatenate 7 model embeddings: improve from LB0.815 to 0.835(body dataset)
\n Combine the two datasets(body and fin) test neighbor and take the max confidence: improve from LB0.835 to 0.858
Here is my final solution:
\n· Dataset: fin and body, use yolov5x6 to train 10folds, test by b0rev256 then pickup best 5 folds, choose the best 1 for train and all others for TTA.(Result: huge effort but slight improvement, maybe should be more carefully to check the dataset and remove noise)
· Split: random choose 500 2 pictures and 55 pictures as new whales(about 11% just same percentage as the LB) as validation dataset and use all others for train. (Result: validation dataset is proved to co-related with LB and useful)
\n· Pseudo: take 4000+ with confidence > 0.95 and all other targets confidence < 0.65
\n· Augmentation: base notebook + 50% rotation, shear, shift, zoom.(Result: slight improvement)
\n· Model: dynamic margins, batch normalization layer before dense layer, add L2 normalization for dense, FreezeBN(Result: significant improvements)
\n · Model candidate: (final 7-8 models for fin and body)
\n · Effv1: b7 768 Adam(use TPUv3-8), b7 608 Adam(Colab), b6 640 Adam(Colab), b5 640 Adam(Colab), b6 480 Adam(Colab), + some SGD model (ex: b7 640)
· Ensemble
\n · TTA:5 fold embeddings and take the mean (small improvement)
\n · Embedding concatenate for body and fin dataset(Large improvement)
\n · Merge fin and body by max conf, leverage by the LB score(use confidence * (LB score**2))(Large improvement)
That's too bad I did 3 methods with no luck :(
", "rawMarkdown": "That's too bad I did 3 methods with no luck :(", "votes": null }, { "id": "1760077", "postDate": "04/19/2022 03:54:24", "content": "Thanks for sharing and thanks for the shout out!
", "rawMarkdown": "Thanks for sharing and thanks for the shout out!", "votes": null }, { "id": "1760203", "postDate": "04/19/2022 06:05:48", "content": "here is a training notebook included 3 implementations, you can check the differences with yours.
\nhttps://www.kaggle.com/code/liuzhangzhen/happywhale-bodydataset-training
Thanks @liuzhangzhen
\nGreat work
thank you Balaji, hope we can team up in the future.
", "rawMarkdown": "thank you Balaji, hope we can team up in the future.", "votes": null } ], "comments": [ { "id": 1760020, "author_name": "atom1231", "author_url": "", "post_date": "04/19/2022 03:22:36", "content": "That's too bad I did 3 methods with no luck :(
", "votes": null, "replies": [ { "id": 1760203, "author_name": "liuzhangzhen", "author_url": "", "post_date": "04/19/2022 06:05:48", "content": "here is a training notebook included 3 implementations, you can check the differences with yours.
\nhttps://www.kaggle.com/code/liuzhangzhen/happywhale-bodydataset-training
Thanks for sharing and thanks for the shout out!
", "votes": null, "replies": [] }, { "id": 1760224, "author_name": "dhakshiin1601", "author_url": "", "post_date": "04/19/2022 06:18:50", "content": "Thanks @liuzhangzhen
\nGreat work
thank you Balaji, hope we can team up in the future.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1760003": "Congratulations to all the winners. Thanks Happywhale and Kaggle for hosting this competition.\n\nMany thanks for the sharing of Lex Toumbourou and Jan bre, I started from their significant work of [Happywhale - Effnet B6 fork with Detic crop] and the dataset of backfintfrecords.\nMany experements in the competition had been tried and finally only three significant improvements in model.\n\n1.**Dynamic margins**: publicLB +0.02 (Thanks Landmark team to share the solution, paper is here: https://arxiv.org/abs/2010.05350): \n Tested and found the best margins range is 0.05-0.6. but training became difficult( loss is easy to Nan) and need to reduce LR.\n2.**Batch normalization**: publicLB +0.02 (Thanks Heng, discussion is here: https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/315129 )\n add Batch normalization layer before dense layer, add L2 normalization for dense:\n3.**FreezeBN**: publicLB +0.01(Thanks Balaji, discussion is here: https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/309582 )\n it works for EfficientB6 and B5, but not works for B7(loss always to nan)\n\nhere is a training notebook included above 3 implementations\nhttps://www.kaggle.com/code/liuzhangzhen/happywhale-bodydataset-training\n\nfinally, two ensembles help to improve public LB to 0.858\n\tConcatenate 7 model embeddings: improve from LB0.815 to 0.835(body dataset)\n\tCombine the two datasets(body and fin) test neighbor and take the max confidence: improve from LB0.835 to 0.858 \n\t\nHere is my final solution:\n· Dataset: fin and body, use yolov5x6 to train 10folds, test by b0rev256 then pickup best 5 folds, choose the best 1 for train and all others for TTA.(Result: huge effort but slight improvement, maybe should be more carefully to check the dataset and remove noise)\n\t\n· Split: random choose 500 2 pictures and 55 pictures as new whales(about 11% just same percentage as the LB) as validation dataset and use all others for train. (Result: validation dataset is proved to co-related with LB and useful)\n\n· Pseudo: take 4000+ with confidence > 0.95 and all other targets confidence < 0.65\n\n· Augmentation: base notebook + 50% rotation, shear, shift, zoom.(Result: slight improvement)\n\n· Model: dynamic margins, batch normalization layer before dense layer, add L2 normalization for dense, FreezeBN(Result: significant improvements)\n\t· Model candidate: (final 7-8 models for fin and body)\n\t\t· Effv1: b7 768 Adam(use TPUv3-8), b7 608 Adam(Colab), b6 640 Adam(Colab), b5 640 Adam(Colab), b6 480 Adam(Colab), + some SGD model (ex: b7 640) \n\n· Ensemble\n\t\t· TTA:5 fold embeddings and take the mean (small improvement)\n\t\t· Embedding concatenate for body and fin dataset(Large improvement)\n\t\t· Merge fin and body by max conf, leverage by the LB score(use confidence * (LB score**2))(Large improvement)", "1760020": "That's too bad I did 3 methods with no luck :(", "1760077": "Thanks for sharing and thanks for the shout out!", "1760203": "here is a training notebook included 3 implementations, you can check the differences with yours.\nhttps://www.kaggle.com/code/liuzhangzhen/happywhale-bodydataset-training", "1760224": "Thanks @liuzhangzhen \nGreat work", "1760228": "thank you Balaji, hope we can team up in the future." }, "source": "meta" }