Except believe cv and how to select compete, what your guys learned? some tricks or some new magic method? welcome sharing
\nfor me follows boost cv:
\n1 bit loss
\n2 self distilling
\n3 ema
Except believe cv and how to select compete, what your guys learned? some tricks or some new magic method? welcome sharing
\nfor me follows boost cv:
\n1 bit loss
\n2 self distilling
\n3 ema
1) Validations strategy
\n2) Knowledge distillation
\n3) TTA
\n4) Pseudo-labeling
\n5) Ensembling
\nAnd not taking part in the competitions with the noisy test data
In the end I gave up on competition because I just did not have time, because I was learning web dev LOL. But in the start of competition I experimentd with metric learning a lot and my best model used
\n1) Multi Layer Fusion
\n2) Additive Margin Softmax
My CV for my best model was 0.884 and Private LB was 0.888
\nI read tons of paper on distillation but didnt have much time. But now I am confident that metric learning works. I wish I had experimented more. But thats fine, we here to learn, wining medals is just the part of journey
", "rawMarkdown": "In the end I gave up on competition because I just did not have time, because I was learning web dev LOL. But in the start of competition I experimentd with metric learning a lot and my best model used \n1) Multi Layer Fusion \n2) Additive Margin Softmax \n\nMy CV for my best model was 0.884 and Private LB was 0.888\n\nI read tons of paper on distillation but didnt have much time. But now I am confident that metric learning works. I wish I had experimented more. But thats fine, we here to learn, wining medals is just the part of journey", "votes": null }, { "id": "1210017", "postDate": "02/19/2021 06:50:35", "content": "Thank you. What's the meaning of muklti layer fusion? additive marhin softmax means metric learning like arcface? Do you mind explained it detail?
", "rawMarkdown": "Thank you. What's the meaning of muklti layer fusion? additive marhin softmax means metric learning like arcface? Do you mind explained it detail?", "votes": null }, { "id": "1210035", "postDate": "02/19/2021 07:02:23", "content": "1) Multi layer fusion means I extracted the outputs from layers before final layer as well. I use pytorch, so lets say you are using resnet, the model will usually have 4 blocks i.e layer 1, 2,3,4. I extracted the output from layers 1,2,3,4 average pooled them and passed them down to final layer.
\n2) yes just like arcface Additive margin softmax is a loss function used for face recognition. The general idea of metric learning is that in an embedding space you pull similar images closer and different images apart.
I also experiments with cosface, arcface, subcenter arcface, Adaptive CosFace.
\n3) Also I used mean teacher as explained in \" Mean Teachers are better role models\" paper. (You can refer to paper for more details)
\nMy end goal for this comp was to just learn more, I read papers of FGVC as well which is pretty amazing. Overall nice experience for me, because I learnt a lot.
", "rawMarkdown": "1) Multi layer fusion means I extracted the outputs from layers before final layer as well. I use pytorch, so lets say you are using resnet, the model will usually have 4 blocks i.e layer 1, 2,3,4. I extracted the output from layers 1,2,3,4 average pooled them and passed them down to final layer. \n2) yes just like arcface Additive margin softmax is a loss function used for face recognition. The general idea of metric learning is that in an embedding space you pull similar images closer and different images apart. \n\nI also experiments with cosface, arcface, subcenter arcface, Adaptive CosFace. \n\n3) Also I used mean teacher as explained in \" Mean Teachers are better role models\" paper. (You can refer to paper for more details)\n\nMy end goal for this comp was to just learn more, I read papers of FGVC as well which is pretty amazing. Overall nice experience for me, because I learnt a lot.", "votes": null }, { "id": "1210056", "postDate": "02/19/2021 07:13:46", "content": "Actually I learned that sometimes luck is more important than skill.
", "rawMarkdown": "Actually I learned that sometimes luck is more important than skill.", "votes": null }, { "id": "1210127", "postDate": "02/19/2021 08:09:31", "content": "Yes, but lucky won't come every time😂
", "rawMarkdown": "Yes, but lucky won't come every time😂", "votes": null }, { "id": "1210136", "postDate": "02/19/2021 08:15:23", "content": "Intuitively, i think Multi layer fusion will hard to train, because it will confuse the model which layer respect to which level feature
", "rawMarkdown": "Intuitively, i think Multi layer fusion will hard to train, because it will confuse the model which layer respect to which level feature", "votes": null }, { "id": "1210156", "postDate": "02/19/2021 08:29:37", "content": "I thought so too. But if you see the images it is of utmost important to locate the object properly and multi layer fusion kinda helps. Yes you maybe right, maybe further experiments will shows the incongruence of the method
", "rawMarkdown": "I thought so too. But if you see the images it is of utmost important to locate the object properly and multi layer fusion kinda helps. Yes you maybe right, maybe further experiments will shows the incongruence of the method", "votes": null }, { "id": "1211428", "postDate": "02/20/2021 08:05:41", "content": "100% agree, I dropped massively in this competition, my learning is trusting own cross validation is much more important than leaderboard position.
", "rawMarkdown": "100% agree, I dropped massively in this competition, my learning is trusting own cross validation is much more important than leaderboard position.", "votes": null }, { "id": "1213759", "postDate": "02/22/2021 10:21:11", "content": "I am just understanding 16th private solution and I realized EMA is very useful!
", "rawMarkdown": "I am just understanding 16th private solution and I realized EMA is very useful!", "votes": null } ], "comments": [ { "id": 1210005, "author_name": "vadimtimakin", "author_url": "", "post_date": "02/19/2021 06:44:50", "content": "1) Validations strategy
\n2) Knowledge distillation
\n3) TTA
\n4) Pseudo-labeling
\n5) Ensembling
\nAnd not taking part in the competitions with the noisy test data
In the end I gave up on competition because I just did not have time, because I was learning web dev LOL. But in the start of competition I experimentd with metric learning a lot and my best model used
\n1) Multi Layer Fusion
\n2) Additive Margin Softmax
My CV for my best model was 0.884 and Private LB was 0.888
\nI read tons of paper on distillation but didnt have much time. But now I am confident that metric learning works. I wish I had experimented more. But thats fine, we here to learn, wining medals is just the part of journey
", "votes": null, "replies": [ { "id": 1210017, "author_name": "cswwp347724", "author_url": "", "post_date": "02/19/2021 06:50:35", "content": "Thank you. What's the meaning of muklti layer fusion? additive marhin softmax means metric learning like arcface? Do you mind explained it detail?
", "votes": null, "replies": [] }, { "id": 1210035, "author_name": "atharvap329", "author_url": "", "post_date": "02/19/2021 07:02:23", "content": "1) Multi layer fusion means I extracted the outputs from layers before final layer as well. I use pytorch, so lets say you are using resnet, the model will usually have 4 blocks i.e layer 1, 2,3,4. I extracted the output from layers 1,2,3,4 average pooled them and passed them down to final layer.
\n2) yes just like arcface Additive margin softmax is a loss function used for face recognition. The general idea of metric learning is that in an embedding space you pull similar images closer and different images apart.
I also experiments with cosface, arcface, subcenter arcface, Adaptive CosFace.
\n3) Also I used mean teacher as explained in \" Mean Teachers are better role models\" paper. (You can refer to paper for more details)
\nMy end goal for this comp was to just learn more, I read papers of FGVC as well which is pretty amazing. Overall nice experience for me, because I learnt a lot.
", "votes": null, "replies": [] }, { "id": 1210148, "author_name": "cswwp347724", "author_url": "", "post_date": "02/19/2021 08:20:47", "content": "Intuitively, i think Multi layer fusion will hard to train, because it will confuse the model which layer respect to which level feature
", "votes": null, "replies": [] }, { "id": 1210156, "author_name": "atharvap329", "author_url": "", "post_date": "02/19/2021 08:29:37", "content": "I thought so too. But if you see the images it is of utmost important to locate the object properly and multi layer fusion kinda helps. Yes you maybe right, maybe further experiments will shows the incongruence of the method
", "votes": null, "replies": [] } ] }, { "id": 1210056, "author_name": "underwearfitting", "author_url": "", "post_date": "02/19/2021 07:13:46", "content": "Actually I learned that sometimes luck is more important than skill.
", "votes": null, "replies": [ { "id": 1210127, "author_name": "cswwp347724", "author_url": "", "post_date": "02/19/2021 08:09:31", "content": "Yes, but lucky won't come every time😂
", "votes": null, "replies": [] }, { "id": 1211428, "author_name": "saurabhbagchi", "author_url": "", "post_date": "02/20/2021 08:05:41", "content": "100% agree, I dropped massively in this competition, my learning is trusting own cross validation is much more important than leaderboard position.
", "votes": null, "replies": [] } ] }, { "id": 1210136, "author_name": "dunklerwald", "author_url": "", "post_date": "02/19/2021 08:15:23", "content": "I am just understanding 16th private solution and I realized EMA is very useful!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1209932": "Except believe cv and how to select compete, what your guys learned? some tricks or some new magic method? welcome sharing\nfor me follows boost cv:\n1 bit loss\n2 self distilling\n3 ema", "1210005": "1) Validations strategy\n2) Knowledge distillation\n3) TTA \n4) Pseudo-labeling\n5) Ensembling\nAnd not taking part in the competitions with the noisy test data", "1210007": "In the end I gave up on competition because I just did not have time, because I was learning web dev LOL. But in the start of competition I experimentd with metric learning a lot and my best model used \n1) Multi Layer Fusion \n2) Additive Margin Softmax \n\nMy CV for my best model was 0.884 and Private LB was 0.888\n\nI read tons of paper on distillation but didnt have much time. But now I am confident that metric learning works. I wish I had experimented more. But thats fine, we here to learn, wining medals is just the part of journey", "1210017": "Thank you. What's the meaning of muklti layer fusion? additive marhin softmax means metric learning like arcface? Do you mind explained it detail?", "1210035": "1) Multi layer fusion means I extracted the outputs from layers before final layer as well. I use pytorch, so lets say you are using resnet, the model will usually have 4 blocks i.e layer 1, 2,3,4. I extracted the output from layers 1,2,3,4 average pooled them and passed them down to final layer. \n2) yes just like arcface Additive margin softmax is a loss function used for face recognition. The general idea of metric learning is that in an embedding space you pull similar images closer and different images apart. \n\nI also experiments with cosface, arcface, subcenter arcface, Adaptive CosFace. \n\n3) Also I used mean teacher as explained in \" Mean Teachers are better role models\" paper. (You can refer to paper for more details)\n\nMy end goal for this comp was to just learn more, I read papers of FGVC as well which is pretty amazing. Overall nice experience for me, because I learnt a lot.", "1210056": "Actually I learned that sometimes luck is more important than skill.", "1210127": "Yes, but lucky won't come every time😂", "1210136": "1. Discriminative learning rate \n2. Freezing of a model (its feature learning part) after a predefined number of epochs", "1210148": "Intuitively, i think Multi layer fusion will hard to train, because it will confuse the model which layer respect to which level feature", "1210156": "I thought so too. But if you see the images it is of utmost important to locate the object properly and multi layer fusion kinda helps. Yes you maybe right, maybe further experiments will shows the incongruence of the method", "1211428": "100% agree, I dropped massively in this competition, my learning is trusting own cross validation is much more important than leaderboard position.", "1213759": "I am just understanding 16th private solution and I realized EMA is very useful!" }, "source": "meta" }