I am using the arcFace approach to competition. And I am building the model with 5-fold cross-validation to create the embedded vectors. Then I don't know how to handle the 5 embedding vectors made by the models when scoring.
\nNow, for the test data, I am taking the arithmetic mean of the five embedding vectors and creating a single vector to calculate the distance.
\nIf you have any useful information, please let me know.
I am using the arcFace approach to competition. And I am building the model with 5-fold cross-validation to create the embedded vectors. Then I don't know how to handle the 5 embedding vectors made by the models when scoring.
\nNow, for the test data, I am taking the arithmetic mean of the five embedding vectors and creating a single vector to calculate the distance.
\nIf you have any useful information, please let me know.
The easiest good solution - concat, i.e you have 5 models with embedding size 512, so, you will have one embedding with 2560 size, usually it is much better then mean. Also, averaging in some conditions could decrease metric
", "rawMarkdown": "The easiest good solution - concat, i.e you have 5 models with embedding size 512, so, you will have one embedding with 2560 size, usually it is much better then mean. Also, averaging in some conditions could decrease metric", "votes": 7, "replies": [ { "id": 1710602, "postDate": "2022-03-03T06:59:54.147Z", "content": "Do you normaize the embeddings before concatening? or concat then normalize or use something else
", "rawMarkdown": "Do you normaize the embeddings before concatening? or concat then normalize or use something else", "votes": 1 }, { "id": 1710688, "postDate": "2022-03-03T08:39:59.350Z", "content": "Normalize after concat, also, normalize before concat
", "rawMarkdown": "Normalize after concat, also, normalize before concat", "votes": 2 }, { "id": 1724127, "postDate": "2022-03-16T03:15:50.230Z", "content": "I have seen that concatening embeddings between different models actually decreases the score.
\nOne of my model results are 0.5(CV) , 0.512(LB)
\nthe other one's results are 0.61(CV) , 0.535(LB)
Concatening the embeddings gives a score of 0.532, it decreases!
\nis it happening to you too? how are you handling it
\n@kwentar
Nope, I have boost, problem may be in different thresholds for two models, e.g. one has thresh 0.5, second has 0.7, so, concat will be worse
", "rawMarkdown": "Nope, I have boost, problem may be in different thresholds for two models, e.g. one has thresh 0.5, second has 0.7, so, concat will be worse" }, { "id": 1725615, "postDate": "2022-03-17T08:57:26.610Z", "content": "So after concatening there we will set only one threshold right, so how can we deal with this then? @kwentar
", "rawMarkdown": "So after concatening there we will set only one threshold right, so how can we deal with this then? @kwentar " }, { "id": 1730433, "postDate": "2022-03-21T08:44:23.507Z", "content": "Thank you very much.
\nI have now submitted the results using the tf_efficientnet_b0 and image size of 224 and simply concatenating the 5-fold model. And I have confirmed that the public score has increased.
\nThe threshold for whether it is a new individual or not is the same as the single model.
\n1 of 5-fold model PB is 0.187 -> 5-fold model PB is 0.193
\nWe plan to try normalizing the embedding vectors before and after concatenation, etc.
@mrinath yeap, we have some problem with thresholds, but it doesn't look critical
", "rawMarkdown": "@mrinath yeap, we have some problem with thresholds, but it doesn't look critical" }, { "id": 1730526, "postDate": "2022-03-21T11:44:03.923Z", "content": "@kwentar you write:
\n\n\nwe have some problem with thresholds, but it doesn't look critical
\n
what do you mean by we ? Do you mean you and your colleague from NTechLab Ilya Dobrynin who you are on the same score with (currently position 4 and 5 on the leaderboard)? But both of you pretend to go solo so how can you know what your colleague has a problem with?
\n@addisonhoward isn't it private sharing?
", "rawMarkdown": "@kwentar you write:\n\n> we have some problem with thresholds, but it doesn't look critical \n\nwhat do you mean by **we** ? Do you mean you and **your colleague** from NTechLab Ilya Dobrynin who you are **on the same score** with (currently position 4 and 5 on the leaderboard)? But both of you pretend to go solo so how can you know what your colleague has a problem with?\n\n@addisonhoward isn't it private sharing?", "votes": 1 }, { "id": 1730594, "postDate": "2022-03-21T12:44:13.267Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1730681, "postDate": "2022-03-21T14:18:29.193Z", "content": "@kwentar , @ilyadobrynin
\nO.K. boys, you teamed up in several minutes after reading (and downvoting) my comment. It only proves the fact that you have already been working together for some time.
@addisonhoward
\nWhat do the rules say? Does subsequent merger simply clear a case of private sharing?
And yet another merger! Looks like NTechLab avalanche!😎
", "rawMarkdown": "And yet another merger! Looks like NTechLab avalanche!😎 " }, { "id": 1730768, "postDate": "2022-03-21T15:40:05.813Z", "content": "First of all, let me emphasize that \"private sharing outside the team\" is a serious accusation in kaggle competitions, and if you have no proofs - it is not a fact, its just an opinion.
\nSecond. Our merge means only that we have decided to participate in this competition together to find the best solution. It is absolutely not an indicator of \"private sharing outside the team\".
\nPreviously you were confused that we go solo, now you are confused that we have merged. Is there any case where you are not confused by other participants activity?
My personal opinion: you are already have very good solution and I believe that it is not the end. So instead of being jealous, try your best to win this contest (solo or in team). I wish you good luck and the ocean of peace.
\n
P.S. @kwentar is very generous to share his findings in discussions here, so I believe that his thoughts will be useful to you.
", "rawMarkdown": "First of all, let me emphasize that \"private sharing **outside the team**\" is a serious accusation in kaggle competitions, and if you have no proofs - it is not a fact, its just an opinion. \n\nSecond. Our merge means only that we have decided to participate in this competition together to find the best solution. It is absolutely not an indicator of \"private sharing outside the team\".\nPreviously you were confused that we go solo, now you are confused that we have merged. Is there any case where you are not confused by other participants activity? \n\nMy personal opinion: you are already have very good solution and I believe that it is not the end. So instead of being jealous, try your best to win this contest (solo or in team). I wish you good luck and the ocean of peace.\n\n
P.S. @kwentar is very generous to share his findings in discussions here, so I believe that his thoughts will be useful to you.", "votes": -2 }, { "id": 1730841, "postDate": "2022-03-21T16:53:04.583Z", "content": "@ilyadobrynin
\nI am not surprised that just you are using such a strategy - instead of explaining or apologizing you start an unscrupulous personal attack.
\nInstead of speaking only about the main topic (which is your team) you start accusing and deriding me.
My personal opinion: You know very well that I am not confused and that I precisely pointed at very strange coincidences. So you are trying your best to depreciate my comments.
\nI am far to be jealous of anybody, I know, that with being eco-friendly even with my HW I cannot win. I only wish the host Happywhale to get really the best solution for helping the magnificent creatures. But it doesn't mean that I should give up the effort for fairness.
", "rawMarkdown": "@ilyadobrynin\nI am not surprised that just you are using such a strategy - instead of explaining or apologizing you start an unscrupulous personal attack.\nInstead of speaking only about the main topic (which is your team) you start accusing and deriding me.\n\nMy personal opinion: You know very well that I am not confused and that I precisely pointed at very strange coincidences. So you are trying your best to depreciate my comments.\n\nI am far to be jealous of anybody, I know, that with being eco-friendly even with my HW I cannot win. I only wish the host Happywhale to get really the best solution for helping the magnificent creatures. But it doesn't mean that I should give up the effort for fairness.\n" }, { "id": 1730847, "postDate": "2022-03-21T17:10:32.127Z", "content": "I believe that the person who start personal attacks is you, with your baseless accusations. My strategy is to focus the attention that this attacks are inappropriate and I want you to stop it.
\nI know nothing about you and why you start this. My personal opinion is that you are jealous and try to devalue other participants effort. I'll be very glad to make a mistake at this point.
\nSo I ask you to stop this pointless thread (so do I) and to focus on the competition.
@ilyadobrynin
\nYet another vain effort from you to overturn everything. Dark will never be light and vice versa only because you wish it.
\nMy objective description of facts was neither an accusation nor a personal attack.
\nNow citing you:
\n\n… I want you to stop it.
\nSo I ask you to stop ….
\n
Who do you think that you are to give me orders? Do you feel like my master?
\nYour only strategy is to ASAP and ruthlessly make me silent with your attacks as it is in your highest interest to let the dust quickly set on all the case.
\nI would also strongly prefer to concentrate on something else than your mocking and arrogant comments are. So I am willing to stop the thread (not my effort for fairness) now.
\nP.S.
\nIt's really funny if you repeatedly speak about jealousy. Look at the competitions' rankings and you may get it.
@blankaf @ilyadobrynin Can somebody bring up pre-merge lb-score? If two top-5 independent solutions merges i would expect somewhat of big lb jump. For now merged team stays on 4-th place, as @blankaf claimed, it is the same place as pre-merge.
", "rawMarkdown": "@blankaf @ilyadobrynin Can somebody bring up pre-merge lb-score? If two top-5 independent solutions merges i would expect somewhat of big lb jump. For now merged team stays on 4-th place, as @blankaf claimed, it is the same place as pre-merge.", "votes": 1 }, { "id": 1731400, "postDate": "2022-03-22T10:28:32.953Z", "content": "Good note @bakeryproducts,
\nthe pre-merge lb-score both on the 4-th and 5-th place was the same 0.841 (I have a screenshot).
\nSo now, we can look forward to something as 0.888, Happywhale will really appreciate it 🙂.
@bakeryproducts thank you for your question.
\nAs you can see from LB submissions last date, we have not submit solutions after our team merge (pre-merge scores were 0.841, 0.841 and 0.811).
\nI may only hope that when we do so, the score will be significantly higher. So stay tuned.
By the way, same score on the LB speaks nothing about submissions similarity - as you can see from the LB (including current top solutions), plenty of teams have same score (I suppose the difference is in 4th+ decimals).
", "rawMarkdown": "@bakeryproducts thank you for your question.\nAs you can see from LB submissions last date, we have not submit solutions after our team merge (pre-merge scores were 0.841, 0.841 and 0.811).\nI may only hope that when we do so, the score will be significantly higher. So stay tuned.\n\nBy the way, same score on the LB speaks nothing about submissions similarity - as you can see from the LB (including current top solutions), plenty of teams have same score (I suppose the difference is in 4th+ decimals)." } ] }, { "id": 1712229, "postDate": "2022-03-04T17:32:13.493Z", "content": "So far, I have used mean, max and a custom voting system. Between them, making a custom voting system based on max score(score = 1- distance) give the best results. Now I am exploring the concatenation part.
", "rawMarkdown": "So far, I have used mean, max and a custom voting system. Between them, making a custom voting system based on max score(score = 1- distance) give the best results. Now I am exploring the concatenation part. ", "votes": 6 } ], "comments": [ { "id": 1710272, "author_name": "Aleksey Alekseev", "author_url": "", "post_date": "2022-03-02T22:03:47.720000", "content": "The easiest good solution - concat, i.e you have 5 models with embedding size 512, so, you will have one embedding with 2560 size, usually it is much better then mean. Also, averaging in some conditions could decrease metric
", "votes": 7, "replies": [ { "id": 1710602, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2022-03-03T06:59:54.147000", "content": "Do you normaize the embeddings before concatening? or concat then normalize or use something else
", "votes": 1, "replies": [] }, { "id": 1710688, "author_name": "Aleksey Alekseev", "author_url": "", "post_date": "2022-03-03T08:39:59.350000", "content": "Normalize after concat, also, normalize before concat
", "votes": 2, "replies": [] }, { "id": 1724127, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2022-03-16T03:15:50.230000", "content": "I have seen that concatening embeddings between different models actually decreases the score.
\nOne of my model results are 0.5(CV) , 0.512(LB)
\nthe other one's results are 0.61(CV) , 0.535(LB)
Concatening the embeddings gives a score of 0.532, it decreases!
\nis it happening to you too? how are you handling it
\n@kwentar
Nope, I have boost, problem may be in different thresholds for two models, e.g. one has thresh 0.5, second has 0.7, so, concat will be worse
", "votes": 0, "replies": [] }, { "id": 1725615, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2022-03-17T08:57:26.610000", "content": "So after concatening there we will set only one threshold right, so how can we deal with this then? @kwentar
", "votes": 0, "replies": [] }, { "id": 1730433, "author_name": "Adana ina", "author_url": "", "post_date": "2022-03-21T08:44:23.507000", "content": "Thank you very much.
\nI have now submitted the results using the tf_efficientnet_b0 and image size of 224 and simply concatenating the 5-fold model. And I have confirmed that the public score has increased.
\nThe threshold for whether it is a new individual or not is the same as the single model.
\n1 of 5-fold model PB is 0.187 -> 5-fold model PB is 0.193
\nWe plan to try normalizing the embedding vectors before and after concatenation, etc.
@mrinath yeap, we have some problem with thresholds, but it doesn't look critical
", "votes": 0, "replies": [] }, { "id": 1730526, "author_name": "Allie K.", "author_url": "", "post_date": "2022-03-21T11:44:03.923000", "content": "@kwentar you write:
\n\n\nwe have some problem with thresholds, but it doesn't look critical
\n
what do you mean by we ? Do you mean you and your colleague from NTechLab Ilya Dobrynin who you are on the same score with (currently position 4 and 5 on the leaderboard)? But both of you pretend to go solo so how can you know what your colleague has a problem with?
\n@addisonhoward isn't it private sharing?
", "votes": 1, "replies": [] }, { "id": 1730594, "author_name": "", "author_url": "", "post_date": "2022-03-21T12:44:13.267000", "content": "", "votes": 0, "replies": [] }, { "id": 1730681, "author_name": "Allie K.", "author_url": "", "post_date": "2022-03-21T14:18:29.193000", "content": "@kwentar , @ilyadobrynin
\nO.K. boys, you teamed up in several minutes after reading (and downvoting) my comment. It only proves the fact that you have already been working together for some time.
@addisonhoward
\nWhat do the rules say? Does subsequent merger simply clear a case of private sharing?
And yet another merger! Looks like NTechLab avalanche!😎
", "votes": 0, "replies": [] }, { "id": 1730768, "author_name": "Ilya Dobrynin", "author_url": "", "post_date": "2022-03-21T15:40:05.813000", "content": "First of all, let me emphasize that \"private sharing outside the team\" is a serious accusation in kaggle competitions, and if you have no proofs - it is not a fact, its just an opinion.
\nSecond. Our merge means only that we have decided to participate in this competition together to find the best solution. It is absolutely not an indicator of \"private sharing outside the team\".
\nPreviously you were confused that we go solo, now you are confused that we have merged. Is there any case where you are not confused by other participants activity?
My personal opinion: you are already have very good solution and I believe that it is not the end. So instead of being jealous, try your best to win this contest (solo or in team). I wish you good luck and the ocean of peace.
\n
P.S. @kwentar is very generous to share his findings in discussions here, so I believe that his thoughts will be useful to you.
", "votes": -2, "replies": [] }, { "id": 1730841, "author_name": "Allie K.", "author_url": "", "post_date": "2022-03-21T16:53:04.583000", "content": "@ilyadobrynin
\nI am not surprised that just you are using such a strategy - instead of explaining or apologizing you start an unscrupulous personal attack.
\nInstead of speaking only about the main topic (which is your team) you start accusing and deriding me.
My personal opinion: You know very well that I am not confused and that I precisely pointed at very strange coincidences. So you are trying your best to depreciate my comments.
\nI am far to be jealous of anybody, I know, that with being eco-friendly even with my HW I cannot win. I only wish the host Happywhale to get really the best solution for helping the magnificent creatures. But it doesn't mean that I should give up the effort for fairness.
", "votes": 0, "replies": [] }, { "id": 1730847, "author_name": "Ilya Dobrynin", "author_url": "", "post_date": "2022-03-21T17:10:32.127000", "content": "I believe that the person who start personal attacks is you, with your baseless accusations. My strategy is to focus the attention that this attacks are inappropriate and I want you to stop it.
\nI know nothing about you and why you start this. My personal opinion is that you are jealous and try to devalue other participants effort. I'll be very glad to make a mistake at this point.
\nSo I ask you to stop this pointless thread (so do I) and to focus on the competition.
@ilyadobrynin
\nYet another vain effort from you to overturn everything. Dark will never be light and vice versa only because you wish it.
\nMy objective description of facts was neither an accusation nor a personal attack.
\nNow citing you:
\n\n… I want you to stop it.
\nSo I ask you to stop ….
\n
Who do you think that you are to give me orders? Do you feel like my master?
\nYour only strategy is to ASAP and ruthlessly make me silent with your attacks as it is in your highest interest to let the dust quickly set on all the case.
\nI would also strongly prefer to concentrate on something else than your mocking and arrogant comments are. So I am willing to stop the thread (not my effort for fairness) now.
\nP.S.
\nIt's really funny if you repeatedly speak about jealousy. Look at the competitions' rankings and you may get it.
@blankaf @ilyadobrynin Can somebody bring up pre-merge lb-score? If two top-5 independent solutions merges i would expect somewhat of big lb jump. For now merged team stays on 4-th place, as @blankaf claimed, it is the same place as pre-merge.
", "votes": 1, "replies": [] }, { "id": 1731400, "author_name": "Allie K.", "author_url": "", "post_date": "2022-03-22T10:28:32.953000", "content": "Good note @bakeryproducts,
\nthe pre-merge lb-score both on the 4-th and 5-th place was the same 0.841 (I have a screenshot).
\nSo now, we can look forward to something as 0.888, Happywhale will really appreciate it 🙂.
@bakeryproducts thank you for your question.
\nAs you can see from LB submissions last date, we have not submit solutions after our team merge (pre-merge scores were 0.841, 0.841 and 0.811).
\nI may only hope that when we do so, the score will be significantly higher. So stay tuned.
By the way, same score on the LB speaks nothing about submissions similarity - as you can see from the LB (including current top solutions), plenty of teams have same score (I suppose the difference is in 4th+ decimals).
", "votes": 0, "replies": [] } ] }, { "id": 1712229, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2022-03-04T17:32:13.493000", "content": "So far, I have used mean, max and a custom voting system. Between them, making a custom voting system based on max score(score = 1- distance) give the best results. Now I am exploring the concatenation part.
", "votes": 6, "replies": [] } ], "raw_markdown_by_id": { "1706587": "I am using the arcFace approach to competition. And I am building the model with 5-fold cross-validation to create the embedded vectors. Then I don't know how to handle the 5 embedding vectors made by the models when scoring.\nNow, for the test data, I am taking the arithmetic mean of the five embedding vectors and creating a single vector to calculate the distance.\nIf you have any useful information, please let me know.\n", "1710272": "The easiest good solution - concat, i.e you have 5 models with embedding size 512, so, you will have one embedding with 2560 size, usually it is much better then mean. Also, averaging in some conditions could decrease metric", "1712229": "So far, I have used mean, max and a custom voting system. Between them, making a custom voting system based on max score(score = 1- distance) give the best results. Now I am exploring the concatenation part. " } }