I've seen a lot less about blending/ensembling/stacking methods for label-weighted label-ranking average precision than for other competition metrics. Have I overlooked some obvious things that get used a lot or that should obviously be helpful?
\nSince the competition metric is rank based, I assume a lot of the approaches often used for other rank based metrics like AUC may make sense here. Those would be things like
\nscipy.optimize). In this context, presumably one should see whether it is a good idea to have seperate weight schemes for the different species?As always stacking may also be interesting, but when one fits a second level model, it's also not immediately obvious than some particular approach would be especially obviously good. E.g. is using a classification objective with binary cross-entropy a good training objective for the second stage model (given that we cannot easily use the non-differentiable competition metric as a loss function)?
\nIn the most similar competition with a similar metric (Freesound Audio Tagging 2019) that I could find, I saw the following in top solutions:
\nI've seen a lot less about blending/ensembling/stacking methods for label-weighted label-ranking average precision than for other competition metrics. Have I overlooked some obvious things that get used a lot or that should obviously be helpful?
\nSince the competition metric is rank based, I assume a lot of the approaches often used for other rank based metrics like AUC may make sense here. Those would be things like
\nscipy.optimize). In this context, presumably one should see whether it is a good idea to have seperate weight schemes for the different species?As always stacking may also be interesting, but when one fits a second level model, it's also not immediately obvious than some particular approach would be especially obviously good. E.g. is using a classification objective with binary cross-entropy a good training objective for the second stage model (given that we cannot easily use the non-differentiable competition metric as a loss function)?
\nIn the most similar competition with a similar metric (Freesound Audio Tagging 2019) that I could find, I saw the following in top solutions:
\nSimple average >>
\nA lot of times, the simpler the better.
", "rawMarkdown": "Simple average >>\n\nA lot of times, the simpler the better.", "votes": null }, { "id": "1172756", "postDate": "01/27/2021 14:38:25", "content": "So far I just average predictions.
", "rawMarkdown": "So far I just average predictions.", "votes": null }, { "id": "1172917", "postDate": "01/27/2021 15:43:42", "content": "you can also combine them. It's risky but hey, you never know.
\nOnce I won a competition with the arithmetic mean for all categories but one where I used geometric. It boosted my CV and LB by 0.00002 enough to jump to the 1 place :)
", "rawMarkdown": "you can also combine them. It's risky but hey, you never know.\n\nOnce I won a competition with the arithmetic mean for all categories but one where I used geometric. It boosted my CV and LB by 0.00002 enough to jump to the 1 place :)", "votes": null }, { "id": "1173641", "postDate": "01/28/2021 03:05:47", "content": "I tried Stacking. But it didn't work. Maybe there is too little training data(about 1200 annotations) to use stacking.
\nSo I used Simple average. It is better performance and simple.
I just used Simple Averaging.
", "rawMarkdown": "I just used Simple Averaging.", "votes": null }, { "id": "1189363", "postDate": "02/06/2021 23:48:20", "content": "I am using simple average as well. In the cornell birdcall competition 2nd place solution was using square, average, and taking the square root again according to solution repo on github here
\nIt didnt help me but might be worth to try
", "rawMarkdown": "I am using simple average as well. In the cornell birdcall competition 2nd place solution was using square, average, and taking the square root again according to solution repo on github [here](https://github.com/vlomme/Birdcall-Identification-competition)\n\nIt didnt help me but might be worth to try", "votes": null }, { "id": "1199672", "postDate": "02/14/2021 03:25:07", "content": "Understand the metric. Check the score (and data) distribution. Check your error rate.
\nIt is a ranking metric so, \"move your prediction rank position\" carefully.
\nOne can definitely do better than simple average. … maybe much better
I agree with that, i.e. cheating with a single specie can bring LB0.01 boost.
", "rawMarkdown": "I agree with that, i.e. cheating with a single specie can bring LB0.01 boost.", "votes": null }, { "id": "1200429", "postDate": "02/14/2021 16:47:08", "content": "0.03 for me
", "rawMarkdown": "0.03 for me", "votes": null }, { "id": "1200434", "postDate": "02/14/2021 16:51:02", "content": "this is called \"manual data de-biasing\".
", "rawMarkdown": "this is called \"manual data de-biasing\".", "votes": null } ], "comments": [ { "id": 1172439, "author_name": "theoviel", "author_url": "", "post_date": "01/27/2021 11:40:03", "content": "Simple average >>
\nA lot of times, the simpler the better.
", "votes": null, "replies": [] }, { "id": 1172756, "author_name": "cpmpml", "author_url": "", "post_date": "01/27/2021 14:38:25", "content": "So far I just average predictions.
", "votes": null, "replies": [] }, { "id": 1172917, "author_name": "valanm", "author_url": "", "post_date": "01/27/2021 15:43:42", "content": "you can also combine them. It's risky but hey, you never know.
\nOnce I won a competition with the arithmetic mean for all categories but one where I used geometric. It boosted my CV and LB by 0.00002 enough to jump to the 1 place :)
", "votes": null, "replies": [] }, { "id": 1173641, "author_name": "shinmurashinmura", "author_url": "", "post_date": "01/28/2021 03:05:47", "content": "I tried Stacking. But it didn't work. Maybe there is too little training data(about 1200 annotations) to use stacking.
\nSo I used Simple average. It is better performance and simple.
I just used Simple Averaging.
", "votes": null, "replies": [] }, { "id": 1189363, "author_name": "snnclsr", "author_url": "", "post_date": "02/06/2021 23:48:20", "content": "I am using simple average as well. In the cornell birdcall competition 2nd place solution was using square, average, and taking the square root again according to solution repo on github here
\nIt didnt help me but might be worth to try
", "votes": null, "replies": [] }, { "id": 1199672, "author_name": "hengck23", "author_url": "", "post_date": "02/14/2021 03:25:07", "content": "Understand the metric. Check the score (and data) distribution. Check your error rate.
\nIt is a ranking metric so, \"move your prediction rank position\" carefully.
\nOne can definitely do better than simple average. … maybe much better
I agree with that, i.e. cheating with a single specie can bring LB0.01 boost.
", "votes": null, "replies": [] }, { "id": 1200429, "author_name": "valanm", "author_url": "", "post_date": "02/14/2021 16:47:08", "content": "0.03 for me
", "votes": null, "replies": [] }, { "id": 1200434, "author_name": "hengck23", "author_url": "", "post_date": "02/14/2021 16:51:02", "content": "this is called \"manual data de-biasing\".
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1172312": "I've seen a lot less about blending/ensembling/stacking methods for label-weighted label-ranking average precision than for other competition metrics. Have I overlooked some obvious things that get used a lot or that should obviously be helpful?\n\nSince the competition metric is rank based, I assume a lot of the approaches often used for other rank based metrics like [AUC](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/211221) may make sense here. Those would be things like\n* [Rank averaging](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205564) as also mentioned in the [Kaggle ensembling guide](https://mlwave.com/kaggle-ensembling-guide/)\n* [Power averaging](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/165653) (including e.g. [square root averaging](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/211194))\n* The above can of course be weighted averages with the weights optimized using out-of-fold predictions. Finding weights that optimize out-of-fold predictions in terms of the competition metric sounds quite doable using `scipy.optimize`). In this context, presumably one should see whether it is a good idea to have seperate weight schemes for the different species?\n\nAs always stacking may also be interesting, but when one fits a second level model, it's also not immediately obvious than some particular approach would be especially obviously good. E.g. is using a classification objective with binary cross-entropy a good training objective for the second stage model (given that we cannot easily use the non-differentiable competition metric as a loss function)?\n\nIn the most similar competition with a similar metric ([Freesound Audio Tagging 2019](https://www.kaggle.com/c/freesound-audio-tagging-2019)) that I could find, I saw the following in top solutions:\n* [Stacking with neural net using binary_crossentropy](https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/97815)\n* [Geometric mean blending](https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/95924)\n* [Averaging weights based on OOF predictions](https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/96440)\n* [Simple average](https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/97812)", "1172439": "Simple average >>\n\nA lot of times, the simpler the better.", "1172756": "So far I just average predictions.", "1172917": "you can also combine them. It's risky but hey, you never know.\n\nOnce I won a competition with the arithmetic mean for all categories but one where I used geometric. It boosted my CV and LB by 0.00002 enough to jump to the 1 place :)", "1173641": "I tried Stacking. But it didn't work. Maybe there is too little training data(about 1200 annotations) to use stacking.\nSo I used **Simple average**. It is better performance and simple.", "1173953": "I just used Simple Averaging.", "1189363": "I am using simple average as well. In the cornell birdcall competition 2nd place solution was using square, average, and taking the square root again according to solution repo on github [here](https://github.com/vlomme/Birdcall-Identification-competition)\n\nIt didnt help me but might be worth to try", "1199672": "Understand the metric. Check the score (and data) distribution. Check your error rate.\nIt is a ranking metric so, \"move your prediction rank position\" carefully.\nOne can definitely do better than simple average. ... maybe much better", "1200313": "I agree with that, i.e. cheating with a single specie can bring LB0.01 boost.", "1200429": "0.03 for me", "1200434": "this is called \"manual data de-biasing\"." }, "source": "meta" }