{ "id": 268231, "title": "Question aboue a metric learning(ArcFace)", "url": "/competitions/landmark-recognition-2021/discussion/268231", "author_name": "", "post_date": "2021-08-26T14:24:23.874968500Z", "votes": 2, "comment_count": 6, "views": 0, "content": "

This is my first really big size image and metric learning competition.

\n

I have been modifying my model from just softmax classification to arcface metric learning classification for building my baseline model.

\n

Since there are two many classes in this competition, no margined model's score is two low( Validation accuracy : 0.889 but a leaderboard score is just 0.18x)

\n

So, I changed my model from softmax to arcface by referring to a kaggle code in Shopee competition which was held three months ago.

\n

When I added a margin(0.3), the model's accuracy score is similar to the previous model of softmax but the validation score dropped a lot(0.88 -> 0.22)

\n

Is it natural? If the arcface version's score is lower than the softmax version that I mentioned above, can get a better score in submission with the arcface version?
\n(Since training takes a lot of time, I can't afford to do many experiments with my devices.

\n

=========================================================
\nMy Model:
\nEffcientNet3, GeM Polling(p=3), ArcFace(margin=0.5, scaler = 50)

\n

Optimizer:
\nCosineRAdam or SGD

\n

Batch size: 16 * 8 = 128

\n

Image size: 448
\nAugmentation : HorizontalFlip(p=0.5), RandomResizedCrop(scale(0.8,1.0))

", "messages": [ { "id": "1491652", "postDate": "08/26/2021 14:24:23", "content": "

This is my first really big size image and metric learning competition.

\n

I have been modifying my model from just softmax classification to arcface metric learning classification for building my baseline model.

\n

Since there are two many classes in this competition, no margined model's score is two low( Validation accuracy : 0.889 but a leaderboard score is just 0.18x)

\n

So, I changed my model from softmax to arcface by referring to a kaggle code in Shopee competition which was held three months ago.

\n

When I added a margin(0.3), the model's accuracy score is similar to the previous model of softmax but the validation score dropped a lot(0.88 -> 0.22)

\n

Is it natural? If the arcface version's score is lower than the softmax version that I mentioned above, can get a better score in submission with the arcface version?
\n(Since training takes a lot of time, I can't afford to do many experiments with my devices.

\n

=========================================================
\nMy Model:
\nEffcientNet3, GeM Polling(p=3), ArcFace(margin=0.5, scaler = 50)

\n

Optimizer:
\nCosineRAdam or SGD

\n

Batch size: 16 * 8 = 128

\n

Image size: 448
\nAugmentation : HorizontalFlip(p=0.5), RandomResizedCrop(scale(0.8,1.0))

", "rawMarkdown": "This is my first really big size image and metric learning competition.\n\nI have been modifying my model from just softmax classification to arcface metric learning classification for building my baseline model.\n\nSince there are two many classes in this competition, no margined model's score is two low( Validation accuracy : 0.889 but a leaderboard score is just 0.18x)\n\nSo, I changed my model from softmax to arcface by referring to a kaggle code in Shopee competition which was held three months ago.\n\nWhen I added a margin(0.3), the model's accuracy score is similar to the previous model of softmax but the validation score dropped a lot(0.88 -> 0.22)\n\nIs it natural? If the arcface version's score is lower than the softmax version that I mentioned above, can get a better score in submission with the arcface version?\n(Since training takes a lot of time, I can't afford to do many experiments with my devices.\n\n\n=========================================================\nMy Model:\nEffcientNet3, GeM Polling(p=3), ArcFace(margin=0.5, scaler = 50)\n\nOptimizer:\nCosineRAdam or SGD\n\nBatch size: 16 * 8 = 128\n\nImage size: 448\nAugmentation : HorizontalFlip(p=0.5), RandomResizedCrop(scale(0.8,1.0))", "votes": null }, { "id": "1493183", "postDate": "08/27/2021 17:15:29", "content": "

From my experience participating in shopee, I think it would be easy to leak due to the way validation is created.
\nAlso, for large models, the overconfidence problem occurs even with data that is not included in the training.

\n

Overconfidence paper -> https://arxiv.org/abs/1706.04599

\n

In the case of Arcface, it calculates cos, which I felt was a reasonable value in terms of LB for statking.

\n

Good luck👍

", "rawMarkdown": "From my experience participating in shopee, I think it would be easy to leak due to the way validation is created.\nAlso, for large models, the overconfidence problem occurs even with data that is not included in the training.\n\nOverconfidence paper -> https://arxiv.org/abs/1706.04599\n\nIn the case of Arcface, it calculates cos, which I felt was a reasonable value in terms of LB for statking.\n\nGood luck👍", "votes": null }, { "id": "1493790", "postDate": "08/28/2021 06:52:25", "content": "

Thanks a lot. i did many experiments. Setting hyperparameters scale = 50 and m = 0.3 seems like a good choice for me
\nI hope your good luck👍👍

", "rawMarkdown": "Thanks a lot. i did many experiments. Setting hyperparameters scale = 50 and m = 0.3 seems like a good choice for me\nI hope your good luck👍👍", "votes": null }, { "id": "1497345", "postDate": "08/31/2021 06:38:52", "content": "

yeah its better to use arc face for multiclass classification with millions of classes. Its normal that ur validation score goes below ur softmax score

", "rawMarkdown": "yeah its better to use arc face for multiclass classification with millions of classes. Its normal that ur validation score goes below ur softmax score", "votes": null }, { "id": "1503379", "postDate": "09/05/2021 10:03:01", "content": "

Thanks a lot!

", "rawMarkdown": "Thanks a lot!", "votes": null }, { "id": "1514721", "postDate": "09/16/2021 11:38:34", "content": "

Hi @deepkim what's your arcface validation accuracy now? I couldn't get it over 58% while training EfficienetB7 progressively from 384 to 736 image size.

", "rawMarkdown": "Hi @deepkim what's your arcface validation accuracy now? I couldn't get it over 58% while training EfficienetB7 progressively from 384 to 736 image size.", "votes": null }, { "id": "1516681", "postDate": "09/18/2021 15:48:06", "content": "

around 80%

", "rawMarkdown": "around 80%", "votes": null } ], "comments": [ { "id": 1493183, "author_name": "dolphin15", "author_url": "", "post_date": "08/27/2021 17:15:29", "content": "

From my experience participating in shopee, I think it would be easy to leak due to the way validation is created.
\nAlso, for large models, the overconfidence problem occurs even with data that is not included in the training.

\n

Overconfidence paper -> https://arxiv.org/abs/1706.04599

\n

In the case of Arcface, it calculates cos, which I felt was a reasonable value in terms of LB for statking.

\n

Good luck👍

", "votes": null, "replies": [ { "id": 1493790, "author_name": "deepkim", "author_url": "", "post_date": "08/28/2021 06:52:25", "content": "

Thanks a lot. i did many experiments. Setting hyperparameters scale = 50 and m = 0.3 seems like a good choice for me
\nI hope your good luck👍👍

", "votes": null, "replies": [] } ] }, { "id": 1497345, "author_name": "nishanthaddagatla", "author_url": "", "post_date": "08/31/2021 06:38:52", "content": "

yeah its better to use arc face for multiclass classification with millions of classes. Its normal that ur validation score goes below ur softmax score

", "votes": null, "replies": [ { "id": 1503379, "author_name": "deepkim", "author_url": "", "post_date": "09/05/2021 10:03:01", "content": "

Thanks a lot!

", "votes": null, "replies": [] } ] }, { "id": 1514721, "author_name": "tuanthanht", "author_url": "", "post_date": "09/16/2021 11:38:34", "content": "

Hi @deepkim what's your arcface validation accuracy now? I couldn't get it over 58% while training EfficienetB7 progressively from 384 to 736 image size.

", "votes": null, "replies": [ { "id": 1516681, "author_name": "deepkim", "author_url": "", "post_date": "09/18/2021 15:48:06", "content": "

around 80%

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1491652": "This is my first really big size image and metric learning competition.\n\nI have been modifying my model from just softmax classification to arcface metric learning classification for building my baseline model.\n\nSince there are two many classes in this competition, no margined model's score is two low( Validation accuracy : 0.889 but a leaderboard score is just 0.18x)\n\nSo, I changed my model from softmax to arcface by referring to a kaggle code in Shopee competition which was held three months ago.\n\nWhen I added a margin(0.3), the model's accuracy score is similar to the previous model of softmax but the validation score dropped a lot(0.88 -> 0.22)\n\nIs it natural? If the arcface version's score is lower than the softmax version that I mentioned above, can get a better score in submission with the arcface version?\n(Since training takes a lot of time, I can't afford to do many experiments with my devices.\n\n\n=========================================================\nMy Model:\nEffcientNet3, GeM Polling(p=3), ArcFace(margin=0.5, scaler = 50)\n\nOptimizer:\nCosineRAdam or SGD\n\nBatch size: 16 * 8 = 128\n\nImage size: 448\nAugmentation : HorizontalFlip(p=0.5), RandomResizedCrop(scale(0.8,1.0))", "1493183": "From my experience participating in shopee, I think it would be easy to leak due to the way validation is created.\nAlso, for large models, the overconfidence problem occurs even with data that is not included in the training.\n\nOverconfidence paper -> https://arxiv.org/abs/1706.04599\n\nIn the case of Arcface, it calculates cos, which I felt was a reasonable value in terms of LB for statking.\n\nGood luck👍", "1493790": "Thanks a lot. i did many experiments. Setting hyperparameters scale = 50 and m = 0.3 seems like a good choice for me\nI hope your good luck👍👍", "1497345": "yeah its better to use arc face for multiclass classification with millions of classes. Its normal that ur validation score goes below ur softmax score", "1503379": "Thanks a lot!", "1514721": "Hi @deepkim what's your arcface validation accuracy now? I couldn't get it over 58% while training EfficienetB7 progressively from 384 to 736 image size.", "1516681": "around 80%" }, "source": "meta" }