Ensembling techniques can bring a significative boost to leaderboard score, especially in competition where we have access to the whole test data and no time restriction for the code to run.
\nSo far in my tests performed in this competition, I can separate the ensembling methods that I have tried in 3 type of categories
1. By ensembling technique
\na) Mean ensembling
\nFor each image I have selected the best 100 predictions confidence from each model.
\nThen average confidences across all models and choose the ones with the highest mean of confidences
b) Max ensembling
\nFor each image I have selected the best 100 predictions confidence from each model.
\nThen choose the top 5 predictions by the max confidence across all models.
c) Concatenate ensembling
\nConcatenate all the embeddings from all the models chosen to ensemble and use it to predict the top 5 predictions
2. By method of choosing \"new_individual\"
\na) If the predicted categories are [a,b,c,d,e] and the confidences are [0.9, 0.8, 0.7, 0.6, 0.5] we start comparing the first element with the threshold and then move along the vector (the vector is always sorted descended, so it starts with the most confident predictions, than follows the second most confident prediction and so on).
\nSo, let's say the threshold is 0.75. The first element is above the threshold and keeps his value, the second one the same, but the third one being below the threshold gets replaces with \"new_individual\" and everything makes one more to the right. Result will be [a,b,\"new_individual\",c,d].
b) Using just the the best confidence prediction to make the comparation. If that value is bigger than the threhold insert \"new_individual\" in the second position, otherwise insert \"new_individual\" in the first position.
\n3. By type of models to be ensembled
\na) Using k folds of the same model
\nb) Using k different type of distinct models
It seems that most people had good results with the 1c + 2b methodology and probably a mix of 3(a+b). So far my best results were with 1b + 2a + 3a (did not tried yes using different models, training runs pretty slow on my machine).
\nTheoretically, using different models will bring a bigger plus comparative with folds of the same model, so I have some hopes boosting the ensembling score.
\nWhat was your experience with ensembling technique so far ?
p.s: forgot to mention the public \"pseudo-random\" weights ensemble directly on csv's 😁 . I guess there are a type of ensembling also
", "messages": [ { "id": "1739073", "postDate": "03/29/2022 18:25:00", "content": "Ensembling techniques can bring a significative boost to leaderboard score, especially in competition where we have access to the whole test data and no time restriction for the code to run.
\nSo far in my tests performed in this competition, I can separate the ensembling methods that I have tried in 3 type of categories
1. By ensembling technique
\na) Mean ensembling
\nFor each image I have selected the best 100 predictions confidence from each model.
\nThen average confidences across all models and choose the ones with the highest mean of confidences
b) Max ensembling
\nFor each image I have selected the best 100 predictions confidence from each model.
\nThen choose the top 5 predictions by the max confidence across all models.
c) Concatenate ensembling
\nConcatenate all the embeddings from all the models chosen to ensemble and use it to predict the top 5 predictions
2. By method of choosing \"new_individual\"
\na) If the predicted categories are [a,b,c,d,e] and the confidences are [0.9, 0.8, 0.7, 0.6, 0.5] we start comparing the first element with the threshold and then move along the vector (the vector is always sorted descended, so it starts with the most confident predictions, than follows the second most confident prediction and so on).
\nSo, let's say the threshold is 0.75. The first element is above the threshold and keeps his value, the second one the same, but the third one being below the threshold gets replaces with \"new_individual\" and everything makes one more to the right. Result will be [a,b,\"new_individual\",c,d].
b) Using just the the best confidence prediction to make the comparation. If that value is bigger than the threhold insert \"new_individual\" in the second position, otherwise insert \"new_individual\" in the first position.
\n3. By type of models to be ensembled
\na) Using k folds of the same model
\nb) Using k different type of distinct models
It seems that most people had good results with the 1c + 2b methodology and probably a mix of 3(a+b). So far my best results were with 1b + 2a + 3a (did not tried yes using different models, training runs pretty slow on my machine).
\nTheoretically, using different models will bring a bigger plus comparative with folds of the same model, so I have some hopes boosting the ensembling score.
\nWhat was your experience with ensembling technique so far ?
p.s: forgot to mention the public \"pseudo-random\" weights ensemble directly on csv's 😁 . I guess there are a type of ensembling also
", "rawMarkdown": "Ensembling techniques can bring a significative boost to leaderboard score, especially in competition where we have access to the whole test data and no time restriction for the code to run. \nSo far in my tests performed in this competition, I can separate the ensembling methods that I have tried in 3 type of categories\n\n**1. By ensembling technique**\na) Mean ensembling \nFor each image I have selected the best 100 predictions confidence from each model. \nThen average confidences across all models and choose the ones with the highest mean of confidences\n\nb) Max ensembling \nFor each image I have selected the best 100 predictions confidence from each model. \nThen choose the top 5 predictions by the max confidence across all models.\n\nc) Concatenate ensembling\nConcatenate all the embeddings from all the models chosen to ensemble and use it to predict the top 5 predictions\n\n**2. By method of choosing \"new_individual\"**\n\na) If the predicted categories are [a,b,c,d,e] and the confidences are [0.9, 0.8, 0.7, 0.6, 0.5] we start comparing the first element with the threshold and then move along the vector (the vector is always sorted descended, so it starts with the most confident predictions, than follows the second most confident prediction and so on).\nSo, let's say the threshold is 0.75. The first element is above the threshold and keeps his value, the second one the same, but the third one being below the threshold gets replaces with \"new_individual\" and everything makes one more to the right. Result will be [a,b,\"new_individual\",c,d].\n\nb) Using just the the best confidence prediction to make the comparation. If that value is bigger than the threhold insert \"new_individual\" in the second position, otherwise insert \"new_individual\" in the first position.\n\n**3. By type of models to be ensembled**\n\na) Using k folds of the same model\nb) Using k different type of distinct models\n\nIt seems that most people had good results with the 1c + 2b methodology and probably a mix of 3(a+b). So far my best results were with 1b + 2a + 3a (did not tried yes using different models, training runs pretty slow on my machine). \nTheoretically, using different models will bring a bigger plus comparative with folds of the same model, so I have some hopes boosting the ensembling score.\nWhat was your experience with ensembling technique so far ? \n\np.s: forgot to mention the public \"pseudo-random\" weights ensemble directly on csv's 😁 . I guess there are a type of ensembling also", "votes": null }, { "id": "1739321", "postDate": "03/30/2022 00:46:09", "content": "Public \"pseudo-random\" weights ensemble seems like hard ensamble, it will loss useful information. I think use pure embedding to ensemble is a better way.
", "rawMarkdown": "Public \"pseudo-random\" weights ensemble seems like hard ensamble, it will loss useful information. I think use pure embedding to ensemble is a better way.", "votes": null }, { "id": "1739557", "postDate": "03/30/2022 06:32:14", "content": "I completely agree, plus the fact that public \"pseudo-random\" weights ensemble are severely overfiting the public leaderboard (modify weights until the ensemble gets a higher public score)
", "rawMarkdown": "I completely agree, plus the fact that public \"pseudo-random\" weights ensemble are severely overfiting the public leaderboard (modify weights until the ensemble gets a higher public score)", "votes": null }, { "id": "1740662", "postDate": "03/31/2022 07:03:54", "content": "thanks for sharing.
\nmy best result is 1c + 2a +3b
thanks for sharing.
", "rawMarkdown": "thanks for sharing.", "votes": null }, { "id": "1742219", "postDate": "04/01/2022 15:28:50", "content": "Did you also tried 1a or 1b ? How did that worked for you ?
", "rawMarkdown": "Did you also tried 1a or 1b ? How did that worked for you ?", "votes": null }, { "id": "1742220", "postDate": "04/01/2022 15:29:04", "content": "Glad it helped
", "rawMarkdown": "Glad it helped", "votes": null }, { "id": "1745149", "postDate": "04/04/2022 16:35:53", "content": "Hi, I don't understand 1a) and 1b). For each test image, which dim to do mean or max operation? There exists the same category samples in the best 100 predictions(the top 100 nearest samples), after group_by operation by target, there is no more 100. Really confused, can you give some help to understand this, Thank you!
", "rawMarkdown": "Hi, I don't understand 1a) and 1b). For each test image, which dim to do mean or max operation? There exists the same category samples in the best 100 predictions(the top 100 nearest samples), after group_by operation by target, there is no more 100. Really confused, can you give some help to understand this, Thank you!", "votes": null }, { "id": "1749805", "postDate": "04/09/2022 02:15:56", "content": "@vladvdv ,
\nHi, sorry for disturbing you.
\nI am facing an issue of ensemble models.
\nI tried the concatenation of embedding outputs but there may be something I missed.
\nAfter doing that, the prediction result was much worse than before.
I leant a lot of things from your notebook and discussion content.
\nCould you please help me to check my embedding logic is right or not? Thank you so much.
\nThe topic link is https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/317609
Using Nearest Neighbors method I am inferring 100 closest matching labels with their confidence (1- distance). Using a second model, I do the same methodology and then average (or get the max) from the same prediction.
\nFor example (use first 3 prediction instead of 100) model 1 predicts:
\na with confidence 0.9
\nb with cofidence 0.88
\nc with confidence 0.73
and model 2 predicts
\nb with confidence 0.9
\na with confidence 0.83
\nd with confidence 0.4
Mean represents for a: (0.9+0.83)/2, for b (0.88+0.9)/2, for c (0.73+0)/2 , for d (0.4+0)/2
\nMax represents for a 0.9, for b 0.9, for c 0.73 and for d 0.4
Can u explain for me what 1c + 2a + 3b means?
", "rawMarkdown": "Can u explain for me what 1c + 2a + 3b means?", "votes": null }, { "id": "1752740", "postDate": "04/12/2022 05:02:03", "content": "Thanks for reply!
", "rawMarkdown": "Thanks for reply!", "votes": null } ], "comments": [ { "id": 1739321, "author_name": "librauee", "author_url": "", "post_date": "03/30/2022 00:46:09", "content": "Public \"pseudo-random\" weights ensemble seems like hard ensamble, it will loss useful information. I think use pure embedding to ensemble is a better way.
", "votes": null, "replies": [ { "id": 1739557, "author_name": "vladvdv", "author_url": "", "post_date": "03/30/2022 06:32:14", "content": "I completely agree, plus the fact that public \"pseudo-random\" weights ensemble are severely overfiting the public leaderboard (modify weights until the ensemble gets a higher public score)
", "votes": null, "replies": [] } ] }, { "id": 1740662, "author_name": "liuzhangzhen", "author_url": "", "post_date": "03/31/2022 07:03:54", "content": "thanks for sharing.
\nmy best result is 1c + 2a +3b
Did you also tried 1a or 1b ? How did that worked for you ?
", "votes": null, "replies": [] }, { "id": 1750999, "author_name": "dqhdqmcttdqx", "author_url": "", "post_date": "04/10/2022 09:26:20", "content": "Can u explain for me what 1c + 2a + 3b means?
", "votes": null, "replies": [] } ] }, { "id": 1741303, "author_name": "nikitadilman", "author_url": "", "post_date": "03/31/2022 17:42:25", "content": "thanks for sharing.
", "votes": null, "replies": [ { "id": 1742220, "author_name": "vladvdv", "author_url": "", "post_date": "04/01/2022 15:29:04", "content": "Glad it helped
", "votes": null, "replies": [] } ] }, { "id": 1745149, "author_name": "rainfalllove", "author_url": "", "post_date": "04/04/2022 16:35:53", "content": "Hi, I don't understand 1a) and 1b). For each test image, which dim to do mean or max operation? There exists the same category samples in the best 100 predictions(the top 100 nearest samples), after group_by operation by target, there is no more 100. Really confused, can you give some help to understand this, Thank you!
", "votes": null, "replies": [ { "id": 1749938, "author_name": "vladvdv", "author_url": "", "post_date": "04/09/2022 06:42:37", "content": "Using Nearest Neighbors method I am inferring 100 closest matching labels with their confidence (1- distance). Using a second model, I do the same methodology and then average (or get the max) from the same prediction.
\nFor example (use first 3 prediction instead of 100) model 1 predicts:
\na with confidence 0.9
\nb with cofidence 0.88
\nc with confidence 0.73
and model 2 predicts
\nb with confidence 0.9
\na with confidence 0.83
\nd with confidence 0.4
Mean represents for a: (0.9+0.83)/2, for b (0.88+0.9)/2, for c (0.73+0)/2 , for d (0.4+0)/2
\nMax represents for a 0.9, for b 0.9, for c 0.73 and for d 0.4
Thanks for reply!
", "votes": null, "replies": [] } ] }, { "id": 1749805, "author_name": "yue300c", "author_url": "", "post_date": "04/09/2022 02:15:56", "content": "@vladvdv ,
\nHi, sorry for disturbing you.
\nI am facing an issue of ensemble models.
\nI tried the concatenation of embedding outputs but there may be something I missed.
\nAfter doing that, the prediction result was much worse than before.
I leant a lot of things from your notebook and discussion content.
\nCould you please help me to check my embedding logic is right or not? Thank you so much.
\nThe topic link is https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/317609