{ "id": 99000, "title": "Unofficial Metric Learning Thread", "url": "/competitions/recursion-cellular-image-classification/discussion/99000", "author_name": "", "post_date": "2019-07-08T06:55:52.320422500Z", "votes": 27, "comment_count": 15, "views": 0, "content": "

Let's discuss different ideas on how to incorporate metric learning in this competition.\nSo far I tried different {Name}Face losses like ArcFace, CosFace, SphereFace etc. but couldn't get anywhere comparable performance to regular cross entropy. I tried using it both as a single loss and combining with cross entropy.\nWhat's your experience?

", "messages": [ { "id": "570344", "postDate": "07/08/2019 06:55:52", "content": "

Let's discuss different ideas on how to incorporate metric learning in this competition.\nSo far I tried different {Name}Face losses like ArcFace, CosFace, SphereFace etc. but couldn't get anywhere comparable performance to regular cross entropy. I tried using it both as a single loss and combining with cross entropy.\nWhat's your experience?

", "rawMarkdown": "Let's discuss different ideas on how to incorporate metric learning in this competition.\nSo far I tried different {Name}Face losses like ArcFace, CosFace, SphereFace etc. but couldn't get anywhere comparable performance to regular cross entropy. I tried using it both as a single loss and combining with cross entropy.\nWhat's your experience?", "votes": null }, { "id": "570365", "postDate": "07/08/2019 07:36:12", "content": "

Could you point to some resources? I never tried metric learning.

", "rawMarkdown": "Could you point to some resources? I never tried metric learning.", "votes": null }, { "id": "570378", "postDate": "07/08/2019 07:47:36", "content": "

I also never tried it before the competition, so far I found useful info in solutions for https://www.kaggle.com/c/humpback-whale-identification/discussion/ and https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/78109

", "rawMarkdown": "I also never tried it before the competition, so far I found useful info in solutions for https://www.kaggle.com/c/humpback-whale-identification/discussion/ and https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/78109", "votes": null }, { "id": "570380", "postDate": "07/08/2019 07:50:08", "content": "

I tried two different variations of ArcFace and also didn't manage to achieve reasonable convergence. Never tried metric learning before so I must be doing something wrong. Trying another metric learning approach now, also plan to try triplet loss.

", "rawMarkdown": "I tried two different variations of ArcFace and also didn't manage to achieve reasonable convergence. Never tried metric learning before so I must be doing something wrong. Trying another metric learning approach now, also plan to try triplet loss.", "votes": null }, { "id": "570531", "postDate": "07/08/2019 12:17:18", "content": "

At lest metric learning gives the strangest learning curves (and this is training losses with a constant learning rate!)

\n\n

\"\"

", "rawMarkdown": "At lest metric learning gives the strangest learning curves (and this is training losses with a constant learning rate!)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F19390%2F5bce7c237cef86e1fc5392d18b94fea0%2FScreenshot%202019-07-08%20at%2015.15.11.png?generation=1562588208876248&alt=media)", "votes": null }, { "id": "570546", "postDate": "07/08/2019 12:42:12", "content": "

I also observed unstable learning curve at the first few epochs. It ended up stable but the score was not as good as regular classification.

", "rawMarkdown": "I also observed unstable learning curve at the first few epochs. It ended up stable but the score was not as good as regular classification.", "votes": null }, { "id": "570704", "postDate": "07/08/2019 17:12:46", "content": "

Thank all for the helpful discussion.\nI know the competition just got started, but I am so looking forward to finally find out about all those magics that will be used in this competition, metric learning, domain adaptation, meta-learning, etc. \nP/s: @sawseen and @vlad0922 must be so lonely at the far top 0.70+

", "rawMarkdown": "Thank all for the helpful discussion.\nI know the competition just got started, but I am so looking forward to finally find out about all those magics that will be used in this competition, metric learning, domain adaptation, meta-learning, etc. \nP/s: @sawseen and @vlad0922 must be so lonely at the far top 0.70+", "votes": null }, { "id": "570747", "postDate": "07/08/2019 18:24:50", "content": "

That's interesting, I have huge gap between train/val loss using *Face losses, but it decreases monotonically anyway. Working on triplet loss now.

", "rawMarkdown": "That's interesting, I have huge gap between train/val loss using *Face losses, but it decreases monotonically anyway. Working on triplet loss now.", "votes": null }, { "id": "570769", "postDate": "07/08/2019 18:42:13", "content": "

Don't worry. We are doing just fine :)

", "rawMarkdown": "Don't worry. We are doing just fine :)", "votes": null }, { "id": "571087", "postDate": "07/09/2019 06:35:17", "content": "

thank you for sharing please provide some resources!

", "rawMarkdown": "thank you for sharing please provide some resources!", "votes": null }, { "id": "571144", "postDate": "07/09/2019 08:14:27", "content": "

You can start with resources provided by @lopuhin + this thread https://www.kaggle.com/c/humpback-whale-identification/discussion/82352.\nFurther you can inspect top solutions in https://www.kaggle.com/c/humpback-whale-identification and https://www.kaggle.com/c/humpback-whale-identification for some examples on how to use metric learning

", "rawMarkdown": "You can start with resources provided by @lopuhin + this thread https://www.kaggle.com/c/humpback-whale-identification/discussion/82352.\nFurther you can inspect top solutions in https://www.kaggle.com/c/humpback-whale-identification and https://www.kaggle.com/c/humpback-whale-identification for some examples on how to use metric learning", "votes": null }, { "id": "572617", "postDate": "07/11/2019 07:19:38", "content": "

If pure metric learning is unstable, you can always combine the two, like a two-branched model: https://www.kaggle.com/c/humpback-whale-identification/discussion/82366#latest-523382

", "rawMarkdown": "If pure metric learning is unstable, you can always combine the two, like a two-branched model: https://www.kaggle.com/c/humpback-whale-identification/discussion/82366#latest-523382", "votes": null }, { "id": "572620", "postDate": "07/11/2019 07:22:35", "content": "

thank you!!

", "rawMarkdown": "thank you!!", "votes": null }, { "id": "573384", "postDate": "07/12/2019 07:39:07", "content": "

Lots of people saying they have not done metric learning before (neither have I). This makes me think that there is something about the competition that really benefits from it. Excuse my ignorance, literally just starting this competition so going to spend the weekend getting an understanding. \nBut can anyone give me the one-sentence 'why metric learning is really powerful here (more-so than elsewhere)'?

", "rawMarkdown": "Lots of people saying they have not done metric learning before (neither have I). This makes me think that there is something about the competition that really benefits from it. Excuse my ignorance, literally just starting this competition so going to spend the weekend getting an understanding. \nBut can anyone give me the one-sentence 'why metric learning is really powerful here (more-so than elsewhere)'?", "votes": null }, { "id": "575118", "postDate": "07/15/2019 04:43:42", "content": "

My 2 cents:\nThe goals of the learned , but the goal of the embedding is different:\n- Classification: Embedding from different classes need to be easily separable.\n- Metric learning: Embedding from the same class need to be close together, and embedding from different classes need to be far from each others.

\n\n

If my understanding is correct, the purposes from classification and metric learning is complimentary, and with solid model and training techniques should always be at least as good as doing only 1 of the above.

\n\n

From the reasoning above, metrics learning will be helpful where there are a lot of intra-class variation (sample from the same class are very different) because they are explicitly asked to be close to each other in the loss function, such as in Face Recognition (and maybe this competition).

", "rawMarkdown": "My 2 cents:\nThe goals of the learned , but the goal of the embedding is different:\n- Classification: Embedding from different classes need to be easily separable.\n- Metric learning: Embedding from the same class need to be close together, and embedding from different classes need to be far from each others.\n\nIf my understanding is correct, the purposes from classification and metric learning is complimentary, and with solid model and training techniques should always be at least as good as doing only 1 of the above.\n\nFrom the reasoning above, metrics learning will be helpful where there are a lot of intra-class variation (sample from the same class are very different) because they are explicitly asked to be close to each other in the loss function, such as in Face Recognition (and maybe this competition).", "votes": null }, { "id": "671282", "postDate": "11/12/2019 13:29:05", "content": "

Hi, newbie here!\n I would like to like to ask how a lot of intra-class variation can benefit from metric-learning when embedding from the same class need to be close together. Aren't the two opposite to each other?

", "rawMarkdown": "Hi, newbie here!\n I would like to like to ask how a lot of intra-class variation can benefit from metric-learning when embedding from the same class need to be close together. Aren't the two opposite to each other?", "votes": null } ], "comments": [ { "id": 570365, "author_name": "christofhenkel", "author_url": "", "post_date": "07/08/2019 07:36:12", "content": "

Could you point to some resources? I never tried metric learning.

", "votes": null, "replies": [ { "id": 570378, "author_name": "lopuhin", "author_url": "", "post_date": "07/08/2019 07:47:36", "content": "

I also never tried it before the competition, so far I found useful info in solutions for https://www.kaggle.com/c/humpback-whale-identification/discussion/ and https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/78109

", "votes": null, "replies": [] } ] }, { "id": 570380, "author_name": "lopuhin", "author_url": "", "post_date": "07/08/2019 07:50:08", "content": "

I tried two different variations of ArcFace and also didn't manage to achieve reasonable convergence. Never tried metric learning before so I must be doing something wrong. Trying another metric learning approach now, also plan to try triplet loss.

", "votes": null, "replies": [] }, { "id": 570531, "author_name": "lopuhin", "author_url": "", "post_date": "07/08/2019 12:17:18", "content": "

At lest metric learning gives the strangest learning curves (and this is training losses with a constant learning rate!)

\n\n

\"\"

", "votes": null, "replies": [ { "id": 570546, "author_name": "tascj0", "author_url": "", "post_date": "07/08/2019 12:42:12", "content": "

I also observed unstable learning curve at the first few epochs. It ended up stable but the score was not as good as regular classification.

", "votes": null, "replies": [] }, { "id": 570747, "author_name": "vshmyhlo", "author_url": "", "post_date": "07/08/2019 18:24:50", "content": "

That's interesting, I have huge gap between train/val loss using *Face losses, but it decreases monotonically anyway. Working on triplet loss now.

", "votes": null, "replies": [] } ] }, { "id": 570704, "author_name": "lkhphuc", "author_url": "", "post_date": "07/08/2019 17:12:46", "content": "

Thank all for the helpful discussion.\nI know the competition just got started, but I am so looking forward to finally find out about all those magics that will be used in this competition, metric learning, domain adaptation, meta-learning, etc. \nP/s: @sawseen and @vlad0922 must be so lonely at the far top 0.70+

", "votes": null, "replies": [ { "id": 570769, "author_name": "sawseen", "author_url": "", "post_date": "07/08/2019 18:42:13", "content": "

Don't worry. We are doing just fine :)

", "votes": null, "replies": [] } ] }, { "id": 571087, "author_name": "krishnakatyal", "author_url": "", "post_date": "07/09/2019 06:35:17", "content": "

thank you for sharing please provide some resources!

", "votes": null, "replies": [ { "id": 571144, "author_name": "vshmyhlo", "author_url": "", "post_date": "07/09/2019 08:14:27", "content": "

You can start with resources provided by @lopuhin + this thread https://www.kaggle.com/c/humpback-whale-identification/discussion/82352.\nFurther you can inspect top solutions in https://www.kaggle.com/c/humpback-whale-identification and https://www.kaggle.com/c/humpback-whale-identification for some examples on how to use metric learning

", "votes": null, "replies": [] }, { "id": 572620, "author_name": "krishnakatyal", "author_url": "", "post_date": "07/11/2019 07:22:35", "content": "

thank you!!

", "votes": null, "replies": [] } ] }, { "id": 572617, "author_name": "alexanderliao", "author_url": "", "post_date": "07/11/2019 07:19:38", "content": "

If pure metric learning is unstable, you can always combine the two, like a two-branched model: https://www.kaggle.com/c/humpback-whale-identification/discussion/82366#latest-523382

", "votes": null, "replies": [] }, { "id": 573384, "author_name": "cherring", "author_url": "", "post_date": "07/12/2019 07:39:07", "content": "

Lots of people saying they have not done metric learning before (neither have I). This makes me think that there is something about the competition that really benefits from it. Excuse my ignorance, literally just starting this competition so going to spend the weekend getting an understanding. \nBut can anyone give me the one-sentence 'why metric learning is really powerful here (more-so than elsewhere)'?

", "votes": null, "replies": [ { "id": 575118, "author_name": "lkhphuc", "author_url": "", "post_date": "07/15/2019 04:43:42", "content": "

My 2 cents:\nThe goals of the learned , but the goal of the embedding is different:\n- Classification: Embedding from different classes need to be easily separable.\n- Metric learning: Embedding from the same class need to be close together, and embedding from different classes need to be far from each others.

\n\n

If my understanding is correct, the purposes from classification and metric learning is complimentary, and with solid model and training techniques should always be at least as good as doing only 1 of the above.

\n\n

From the reasoning above, metrics learning will be helpful where there are a lot of intra-class variation (sample from the same class are very different) because they are explicitly asked to be close to each other in the loss function, such as in Face Recognition (and maybe this competition).

", "votes": null, "replies": [] }, { "id": 671282, "author_name": "imenhaddad", "author_url": "", "post_date": "11/12/2019 13:29:05", "content": "

Hi, newbie here!\n I would like to like to ask how a lot of intra-class variation can benefit from metric-learning when embedding from the same class need to be close together. Aren't the two opposite to each other?

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "570344": "Let's discuss different ideas on how to incorporate metric learning in this competition.\nSo far I tried different {Name}Face losses like ArcFace, CosFace, SphereFace etc. but couldn't get anywhere comparable performance to regular cross entropy. I tried using it both as a single loss and combining with cross entropy.\nWhat's your experience?", "570365": "Could you point to some resources? I never tried metric learning.", "570378": "I also never tried it before the competition, so far I found useful info in solutions for https://www.kaggle.com/c/humpback-whale-identification/discussion/ and https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/78109", "570380": "I tried two different variations of ArcFace and also didn't manage to achieve reasonable convergence. Never tried metric learning before so I must be doing something wrong. Trying another metric learning approach now, also plan to try triplet loss.", "570531": "At lest metric learning gives the strangest learning curves (and this is training losses with a constant learning rate!)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F19390%2F5bce7c237cef86e1fc5392d18b94fea0%2FScreenshot%202019-07-08%20at%2015.15.11.png?generation=1562588208876248&alt=media)", "570546": "I also observed unstable learning curve at the first few epochs. It ended up stable but the score was not as good as regular classification.", "570704": "Thank all for the helpful discussion.\nI know the competition just got started, but I am so looking forward to finally find out about all those magics that will be used in this competition, metric learning, domain adaptation, meta-learning, etc. \nP/s: @sawseen and @vlad0922 must be so lonely at the far top 0.70+", "570747": "That's interesting, I have huge gap between train/val loss using *Face losses, but it decreases monotonically anyway. Working on triplet loss now.", "570769": "Don't worry. We are doing just fine :)", "571087": "thank you for sharing please provide some resources!", "571144": "You can start with resources provided by @lopuhin + this thread https://www.kaggle.com/c/humpback-whale-identification/discussion/82352.\nFurther you can inspect top solutions in https://www.kaggle.com/c/humpback-whale-identification and https://www.kaggle.com/c/humpback-whale-identification for some examples on how to use metric learning", "572617": "If pure metric learning is unstable, you can always combine the two, like a two-branched model: https://www.kaggle.com/c/humpback-whale-identification/discussion/82366#latest-523382", "572620": "thank you!!", "573384": "Lots of people saying they have not done metric learning before (neither have I). This makes me think that there is something about the competition that really benefits from it. Excuse my ignorance, literally just starting this competition so going to spend the weekend getting an understanding. \nBut can anyone give me the one-sentence 'why metric learning is really powerful here (more-so than elsewhere)'?", "575118": "My 2 cents:\nThe goals of the learned , but the goal of the embedding is different:\n- Classification: Embedding from different classes need to be easily separable.\n- Metric learning: Embedding from the same class need to be close together, and embedding from different classes need to be far from each others.\n\nIf my understanding is correct, the purposes from classification and metric learning is complimentary, and with solid model and training techniques should always be at least as good as doing only 1 of the above.\n\nFrom the reasoning above, metrics learning will be helpful where there are a lot of intra-class variation (sample from the same class are very different) because they are explicitly asked to be close to each other in the loss function, such as in Face Recognition (and maybe this competition).", "671282": "Hi, newbie here!\n I would like to like to ask how a lot of intra-class variation can benefit from metric-learning when embedding from the same class need to be close together. Aren't the two opposite to each other?" }, "source": "meta" }