In Chris's excellent post CNN Input Size Explained, he suggests ensembling models of different input sizes. For blending the predictions of all these models, see Power Averaging technique below, which was used successfully in older competitions to obtain Medals π
\n\nVarious Prediction Averaging/Blending Techniques:
\n\nSimple Averaging: Most participants are using just a simple mean of predictions generated by different models
Rank Averaging: Use the \"rank\" of an input image instead it's prediction value. See public notebook Improve blending using Rankdata
Weighted Averaging: Specify weights, say 0.5 each in case of two models\nWeightedAverage(p) = (wt1 x Pred1 + wt2 x Pred2 + β¦ + wtn x Predn)\nwhere, n is the number of models, and sum of weights wt1+wt2+β¦+wtn = 1
Stretch Averaging: Stretch predictions using min and max values first, before averaging\nPred = (Pred - min(Pred)) / (max(Pred) - min(Pred))
Power Averaging: Choose a power p = 2, 4, 8, 16\nPowerAverage(p) = (Pred1^p + Pred2^p + β¦ + Predn^p) / n\nNote: Power Averaging to be used only when all the models are highly correlated, otherwise your score may become worse.
Power Averaging with weights:\nPowerAverageWithWeights(p) = (wt1 x Pred1^p + wt2 x Pred2^p + β¦ + wtn x Predn^p)
Note: Ensemble your best models for final submission, typically about 2 weeks before the competition deadline.
\n\nPS: Read this post by a Kaggle Master Power/geometric ensembling when targeting the AUC metric for a more detailed explanation with a lot of calculations.
", "messages": [ { "id": 922999, "postDate": "2020-07-10T13:23:47.603Z", "content": "In Chris's excellent post CNN Input Size Explained, he suggests ensembling models of different input sizes. For blending the predictions of all these models, see Power Averaging technique below, which was used successfully in older competitions to obtain Medals π
\n\nVarious Prediction Averaging/Blending Techniques:
\n\nSimple Averaging: Most participants are using just a simple mean of predictions generated by different models
Rank Averaging: Use the \"rank\" of an input image instead it's prediction value. See public notebook Improve blending using Rankdata
Weighted Averaging: Specify weights, say 0.5 each in case of two models\nWeightedAverage(p) = (wt1 x Pred1 + wt2 x Pred2 + β¦ + wtn x Predn)\nwhere, n is the number of models, and sum of weights wt1+wt2+β¦+wtn = 1
Stretch Averaging: Stretch predictions using min and max values first, before averaging\nPred = (Pred - min(Pred)) / (max(Pred) - min(Pred))
Power Averaging: Choose a power p = 2, 4, 8, 16\nPowerAverage(p) = (Pred1^p + Pred2^p + β¦ + Predn^p) / n\nNote: Power Averaging to be used only when all the models are highly correlated, otherwise your score may become worse.
Power Averaging with weights:\nPowerAverageWithWeights(p) = (wt1 x Pred1^p + wt2 x Pred2^p + β¦ + wtn x Predn^p)
Note: Ensemble your best models for final submission, typically about 2 weeks before the competition deadline.
\n\nPS: Read this post by a Kaggle Master Power/geometric ensembling when targeting the AUC metric for a more detailed explanation with a lot of calculations.
", "rawMarkdown": "In Chris's excellent post [CNN Input Size Explained](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147), he suggests ensembling models of different input sizes. For blending the predictions of all these models, see **Power Averaging** technique below, which was used successfully in older competitions to **obtain Medals** π \n\nVarious Prediction Averaging/Blending Techniques:\n\n1. Simple Averaging: Most participants are using just a simple mean of predictions generated by different models\n\n2. Rank Averaging: Use the \"rank\" of an input image instead it's prediction value. See public notebook [Improve blending using Rankdata](https://www.kaggle.com/niteshx2/improve-blending-using-rankdata)\n\n3. Weighted Averaging: Specify weights, say 0.5 each in case of two models\n WeightedAverage(p) = (wt1 x Pred1 + wt2 x Pred2 + β¦ + wtn x Predn)\n where, n is the number of models, and sum of weights wt1+wt2+β¦+wtn = 1\n\n4. Stretch Averaging: Stretch predictions using min and max values first, before averaging\n Pred = (Pred - min(Pred)) / (max(Pred) - min(Pred))\n\n5. Power Averaging: Choose a power p = 2, 4, 8, 16\n PowerAverage(p) = (Pred1^p + Pred2^p + β¦ + Predn^p) / n\n Note: Power Averaging to be used only when all the models are highly correlated, otherwise your score may become worse.\n\n6. Power Averaging with weights:\n PowerAverageWithWeights(p) = (wt1 x Pred1^p + wt2 x Pred2^p + β¦ + wtn x Predn^p)\n\nNote: Ensemble your **best models** for final submission, typically about 2 weeks before the competition deadline.\n\nPS: Read this post by a Kaggle Master [Power/geometric ensembling when targeting the AUC metric](https://medium.com/data-design/reaching-the-depths-of-power-geometric-ensembling-when-targeting-the-auc-metric-2f356ea3250e) for a more detailed explanation with a lot of calculations.", "votes": 113 }, { "id": 961196, "postDate": "2020-08-07T01:54:15.857Z", "content": "wow.. amazing technique..
\nActually I was looking for something like this for a long time.
", "rawMarkdown": "wow.. amazing technique..\n\nActually I was looking for something like this for a long time. \n", "votes": 1 }, { "id": 954676, "postDate": "2020-08-02T01:53:49.433Z", "content": "Thank you for sharing this, I have to take notes and probably make a notebook on all this.
", "rawMarkdown": "Thank you for sharing this, I have to take notes and probably make a notebook on all this.", "votes": 1 }, { "id": 925894, "postDate": "2020-07-12T11:20:45.713Z", "content": "Thanks for sharing this @sirishks.
\n\nI guess the theoretical validation of using rank averaging is a bit stronger than simple averaging since we are choosing the best epoch on AUC where probabilities by themselves are disregarded (only the order of probabilities is considered). However, theory hardly agrees with practice.
\n\nHow would you apply advanced concepts like weighted averaging or power averaging to rank averaging? Using power averaging on rank averaging by itself would potentially lead to disastrous effects since it would scale images with values closer to 1 much lesser than those close to N, whereas the metric confers equal weight to each image whether the output is on one end of the spectrum or the other. However, to think about it, even in the case of power averaging on the probabilities, the probabilities closer to 1.00 are suppressed much lesser than those closer to 0.00. But, probably that's a desirable quality. And since the higher p's provide better results, probably power averaging on rank averaging doesn't sound that bad an idea after all.
What do you think?
", "rawMarkdown": "Thanks for sharing this @sirishks.\n\nI guess the theoretical validation of using rank averaging is a bit stronger than simple averaging since we are choosing the best epoch on AUC where probabilities by themselves are disregarded (only the order of probabilities is considered). However, theory hardly agrees with practice.\n\nHow would you apply advanced concepts like weighted averaging or power averaging to rank averaging? Using power averaging on rank averaging by itself would potentially lead to disastrous effects since it would scale images with values closer to `1` much lesser than those close to `N`, whereas the metric confers equal weight to each image whether the output is on one end of the spectrum or the other. However, to think about it, even in the case of power averaging on the probabilities, the probabilities closer to `1.00` are suppressed much lesser than those closer to `0.00`. But, probably that's a desirable quality. And since the higher p's provide better results, probably power averaging on rank averaging doesn't sound that bad an idea after all.\n\nWhat do you think?", "votes": 1 }, { "id": 923129, "postDate": "2020-07-10T14:52:04.553Z", "content": "Thanks a lot for this @sirishks, very helpful!
\n\nBut we need to be extremely careful about choosing the models for power averaging, one bad model and everything might go for a toss.
\n\nAlso, in point 3 you mention:
\n\n\n\n\nWeighted Averaging: Specify weights, say 0.5 each in case of two models\n WeightedAverage(p) = (wt1 x Pred1 + wt2 x Pred2 + β¦ + wtn x Predn) / n\n where, n is the number of models, and sum of weights wt1+wt2+β¦+wtn = 1
\n
Shouldn't we have wt1+...+wtn = n, since we are dividing the equation by n OR just remove n from the denominator and have wt1+...+wtn = 1? Correct me if I am wrong.
@abhishekgbhat yes, it was a typo. You are correctπ
\n\nI have edited the answer and removed \"/n\" when using weights in the range [0,1] such that sum of weights = 1.
", "rawMarkdown": "@abhishekgbhat yes, it was a typo. You are correctπ \n\nI have edited the answer and removed \"/n\" when using weights in the range [0,1] such that sum of weights = 1.", "votes": 1 }, { "id": 923295, "postDate": "2020-07-10T17:42:22.710Z", "content": "@abhishekgbhat AUC is an interesting metric -- all it cares about is the order of your predictions. Dividing the predictions by any common number (like n) is not going to change the values of the metric, so, from practical standpoint, it doesn't make a difference.
", "rawMarkdown": "@abhishekgbhat AUC is an interesting metric -- all it cares about is the order of your predictions. Dividing the predictions by any common number (like n) is not going to change the values of the metric, so, from practical standpoint, it doesn't make a difference.", "votes": 1 }, { "id": 923314, "postDate": "2020-07-10T18:04:44.183Z", "content": "@graf10a I totally agree with what you are saying about the AUC.. I was making a general comment about the weighted average definition. But as you mentioned, when the metric is AUC all that matters is the order.
", "rawMarkdown": "@graf10a I totally agree with what you are saying about the AUC.. I was making a general comment about the weighted average definition. But as you mentioned, when the metric is AUC all that matters is the order.", "votes": 1 } ] }, { "id": 3361837, "postDate": "2025-12-03T21:29:27.237Z", "content": "Well Written!
\nHave a look at my notebook https://www.kaggle.com/code/shivanisharma1297/ai-text-detection-myensemble
", "rawMarkdown": "Well Written!\n\n Have a look at my notebook https://www.kaggle.com/code/shivanisharma1297/ai-text-detection-myensemble" }, { "id": 969206, "postDate": "2020-08-13T14:46:43.310Z", "content": "Nice Explanation
", "rawMarkdown": "Nice Explanation " }, { "id": 951748, "postDate": "2020-07-30T11:28:35.280Z", "content": "@sirishks thanks for the post. In the mentioned blog, there is a term weight_initializerin part 5 of blog which I could not understand. Can you please clarify what is weight_initializerand how one can calculate it ?
Thanks a lot for sharing the article\nHow do you define highly correlation between models .Is it CV or Model architecture or something else?
", "rawMarkdown": "Thanks a lot for sharing the article\nHow do you define highly correlation between models .Is it CV or Model architecture or something else?" }, { "id": 928929, "postDate": "2020-07-14T10:30:17.527Z", "content": "Your 5th method is wrong. Don't you think it should be ((pred1^p + pred2^p + .... predn^p)/n)^(1/n)\nIf it is not then you are basically changing the scale.
", "rawMarkdown": "Your 5th method is wrong. Don't you think it should be ((pred1^p + pred2^p + .... predn^p)/n)^(1/n)\nIf it is not then you are basically changing the scale.", "replies": [ { "id": 929171, "postDate": "2020-07-14T13:58:24.263Z", "content": "Power averaging ensures that the top values are pushed further away from the rest, thereby separating Malignant images from Benign, resulting in a higher AUC/LB score.
\n\nHowever, that does not mean that you cannot apply a square-root. In Data Science, it is all about the data i.e. depending on your models and predicted probabilities, you may get better results with a square-root vs someone else whose models work better when directly used with power values.
", "rawMarkdown": "Power averaging ensures that the top values are pushed further away from the rest, thereby **separating** Malignant images from Benign, resulting in a higher AUC/LB score.\n\nHowever, that does not mean that you cannot apply a square-root. In Data Science, it is all about the data i.e. depending on your models and predicted probabilities, you may get better results with a square-root vs someone else whose models work better when directly used with power values.", "votes": 2 }, { "id": 929832, "postDate": "2020-07-15T02:20:07.337Z", "content": "For a metric like AUC where only the rank matters, applying functions that don't change the rank but only the scale like power or multiplications or divisions have absolutely no impact on the results. However, care must be taken while ensembling outputs from multiple models - a model with higher variation in output is most likely to supersede the other outputs and hence the outputs must be normalised before ensembling.
", "rawMarkdown": "For a metric like AUC where only the rank matters, applying functions that don't change the rank but only the scale like power or multiplications or divisions have absolutely no impact on the results. However, care must be taken while ensembling outputs from multiple models - a model with higher variation in output is most likely to supersede the other outputs and hence the outputs must be normalised before ensembling." }, { "id": 961628, "postDate": "2020-08-07T10:56:50.290Z", "rawMarkdown": "", "votes": -2, "isDeleted": true }, { "id": 961860, "postDate": "2020-08-07T15:07:40.583Z", "content": "@krisho007 Can you show a small reproducible example illustrating your statement?
", "rawMarkdown": "@krisho007 Can you show a small reproducible example illustrating your statement?" } ] }, { "id": 926101, "postDate": "2020-07-12T13:52:39.277Z", "content": "Nice insight
", "rawMarkdown": "Nice insight" }, { "id": 923270, "postDate": "2020-07-10T17:06:40.747Z", "content": "Cool, thanks for sharing! π Let me try this power averaging method!
", "rawMarkdown": "Cool, thanks for sharing! π Let me try this power averaging method! " }, { "id": 961622, "postDate": "2020-08-07T10:49:59.050Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 930093, "postDate": "2020-07-15T07:35:07.173Z", "content": "thanks for your contibution
", "rawMarkdown": "thanks for your contibution" } ], "comments": [ { "id": 961196, "author_name": "Redwan Sony", "author_url": "", "post_date": "2020-08-07T01:54:15.857000", "content": "wow.. amazing technique..
\nActually I was looking for something like this for a long time.
", "votes": 1, "replies": [] }, { "id": 954676, "author_name": "Alin Cijov", "author_url": "", "post_date": "2020-08-02T01:53:49.433000", "content": "Thank you for sharing this, I have to take notes and probably make a notebook on all this.
", "votes": 1, "replies": [] }, { "id": 925894, "author_name": "Rohit Agarwal", "author_url": "", "post_date": "2020-07-12T11:20:45.713000", "content": "Thanks for sharing this @sirishks.
\n\nI guess the theoretical validation of using rank averaging is a bit stronger than simple averaging since we are choosing the best epoch on AUC where probabilities by themselves are disregarded (only the order of probabilities is considered). However, theory hardly agrees with practice.
\n\nHow would you apply advanced concepts like weighted averaging or power averaging to rank averaging? Using power averaging on rank averaging by itself would potentially lead to disastrous effects since it would scale images with values closer to 1 much lesser than those close to N, whereas the metric confers equal weight to each image whether the output is on one end of the spectrum or the other. However, to think about it, even in the case of power averaging on the probabilities, the probabilities closer to 1.00 are suppressed much lesser than those closer to 0.00. But, probably that's a desirable quality. And since the higher p's provide better results, probably power averaging on rank averaging doesn't sound that bad an idea after all.
What do you think?
", "votes": 1, "replies": [] }, { "id": 923129, "author_name": "Abhishek Bhat", "author_url": "", "post_date": "2020-07-10T14:52:04.553000", "content": "Thanks a lot for this @sirishks, very helpful!
\n\nBut we need to be extremely careful about choosing the models for power averaging, one bad model and everything might go for a toss.
\n\nAlso, in point 3 you mention:
\n\n\n\n\nWeighted Averaging: Specify weights, say 0.5 each in case of two models\n WeightedAverage(p) = (wt1 x Pred1 + wt2 x Pred2 + β¦ + wtn x Predn) / n\n where, n is the number of models, and sum of weights wt1+wt2+β¦+wtn = 1
\n
Shouldn't we have wt1+...+wtn = n, since we are dividing the equation by n OR just remove n from the denominator and have wt1+...+wtn = 1? Correct me if I am wrong.
@abhishekgbhat yes, it was a typo. You are correctπ
\n\nI have edited the answer and removed \"/n\" when using weights in the range [0,1] such that sum of weights = 1.
", "votes": 1, "replies": [] }, { "id": 923295, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2020-07-10T17:42:22.710000", "content": "@abhishekgbhat AUC is an interesting metric -- all it cares about is the order of your predictions. Dividing the predictions by any common number (like n) is not going to change the values of the metric, so, from practical standpoint, it doesn't make a difference.
", "votes": 1, "replies": [] }, { "id": 923314, "author_name": "Abhishek Bhat", "author_url": "", "post_date": "2020-07-10T18:04:44.183000", "content": "@graf10a I totally agree with what you are saying about the AUC.. I was making a general comment about the weighted average definition. But as you mentioned, when the metric is AUC all that matters is the order.
", "votes": 1, "replies": [] } ] }, { "id": 3361837, "author_name": "Shivani Sharma", "author_url": "", "post_date": "2025-12-03T21:29:27.237000", "content": "Well Written!
\nHave a look at my notebook https://www.kaggle.com/code/shivanisharma1297/ai-text-detection-myensemble
", "votes": 0, "replies": [] }, { "id": 969206, "author_name": "Haider Ali Shuvo", "author_url": "", "post_date": "2020-08-13T14:46:43.310000", "content": "Nice Explanation
", "votes": 0, "replies": [] }, { "id": 951748, "author_name": "Abdur Rehman", "author_url": "", "post_date": "2020-07-30T11:28:35.280000", "content": "@sirishks thanks for the post. In the mentioned blog, there is a term weight_initializerin part 5 of blog which I could not understand. Can you please clarify what is weight_initializerand how one can calculate it ?
Thanks a lot for sharing the article\nHow do you define highly correlation between models .Is it CV or Model architecture or something else?
", "votes": 0, "replies": [] }, { "id": 928929, "author_name": "", "author_url": "", "post_date": "2020-07-14T10:30:17.527000", "content": "Your 5th method is wrong. Don't you think it should be ((pred1^p + pred2^p + .... predn^p)/n)^(1/n)\nIf it is not then you are basically changing the scale.
", "votes": 0, "replies": [ { "id": 929171, "author_name": "Sirish Somanchi", "author_url": "", "post_date": "2020-07-14T13:58:24.263000", "content": "Power averaging ensures that the top values are pushed further away from the rest, thereby separating Malignant images from Benign, resulting in a higher AUC/LB score.
\n\nHowever, that does not mean that you cannot apply a square-root. In Data Science, it is all about the data i.e. depending on your models and predicted probabilities, you may get better results with a square-root vs someone else whose models work better when directly used with power values.
", "votes": 2, "replies": [] }, { "id": 929832, "author_name": "Rohit Agarwal", "author_url": "", "post_date": "2020-07-15T02:20:07.337000", "content": "For a metric like AUC where only the rank matters, applying functions that don't change the rank but only the scale like power or multiplications or divisions have absolutely no impact on the results. However, care must be taken while ensembling outputs from multiple models - a model with higher variation in output is most likely to supersede the other outputs and hence the outputs must be normalised before ensembling.
", "votes": 0, "replies": [] }, { "id": 961628, "author_name": "", "author_url": "", "post_date": "2020-08-07T10:56:50.290000", "content": "", "votes": -2, "replies": [] }, { "id": 961860, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2020-08-07T15:07:40.583000", "content": "@krisho007 Can you show a small reproducible example illustrating your statement?
", "votes": 0, "replies": [] } ] }, { "id": 926101, "author_name": "virendra yadav0207", "author_url": "", "post_date": "2020-07-12T13:52:39.277000", "content": "Nice insight
", "votes": 0, "replies": [] }, { "id": 923270, "author_name": "Helen", "author_url": "", "post_date": "2020-07-10T17:06:40.747000", "content": "Cool, thanks for sharing! π Let me try this power averaging method!
", "votes": 0, "replies": [] }, { "id": 961622, "author_name": "", "author_url": "", "post_date": "2020-08-07T10:49:59.050000", "content": "", "votes": -1, "replies": [] }, { "id": 930093, "author_name": "SURJEET KUMAR", "author_url": "", "post_date": "2020-07-15T07:35:07.173000", "content": "thanks for your contibution
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "922999": "In Chris's excellent post [CNN Input Size Explained](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147), he suggests ensembling models of different input sizes. For blending the predictions of all these models, see **Power Averaging** technique below, which was used successfully in older competitions to **obtain Medals** π \n\nVarious Prediction Averaging/Blending Techniques:\n\n1. Simple Averaging: Most participants are using just a simple mean of predictions generated by different models\n\n2. Rank Averaging: Use the \"rank\" of an input image instead it's prediction value. See public notebook [Improve blending using Rankdata](https://www.kaggle.com/niteshx2/improve-blending-using-rankdata)\n\n3. Weighted Averaging: Specify weights, say 0.5 each in case of two models\n WeightedAverage(p) = (wt1 x Pred1 + wt2 x Pred2 + β¦ + wtn x Predn)\n where, n is the number of models, and sum of weights wt1+wt2+β¦+wtn = 1\n\n4. Stretch Averaging: Stretch predictions using min and max values first, before averaging\n Pred = (Pred - min(Pred)) / (max(Pred) - min(Pred))\n\n5. Power Averaging: Choose a power p = 2, 4, 8, 16\n PowerAverage(p) = (Pred1^p + Pred2^p + β¦ + Predn^p) / n\n Note: Power Averaging to be used only when all the models are highly correlated, otherwise your score may become worse.\n\n6. Power Averaging with weights:\n PowerAverageWithWeights(p) = (wt1 x Pred1^p + wt2 x Pred2^p + β¦ + wtn x Predn^p)\n\nNote: Ensemble your **best models** for final submission, typically about 2 weeks before the competition deadline.\n\nPS: Read this post by a Kaggle Master [Power/geometric ensembling when targeting the AUC metric](https://medium.com/data-design/reaching-the-depths-of-power-geometric-ensembling-when-targeting-the-auc-metric-2f356ea3250e) for a more detailed explanation with a lot of calculations.", "961196": "wow.. amazing technique..\n\nActually I was looking for something like this for a long time. \n", "954676": "Thank you for sharing this, I have to take notes and probably make a notebook on all this.", "925894": "Thanks for sharing this @sirishks.\n\nI guess the theoretical validation of using rank averaging is a bit stronger than simple averaging since we are choosing the best epoch on AUC where probabilities by themselves are disregarded (only the order of probabilities is considered). However, theory hardly agrees with practice.\n\nHow would you apply advanced concepts like weighted averaging or power averaging to rank averaging? Using power averaging on rank averaging by itself would potentially lead to disastrous effects since it would scale images with values closer to `1` much lesser than those close to `N`, whereas the metric confers equal weight to each image whether the output is on one end of the spectrum or the other. However, to think about it, even in the case of power averaging on the probabilities, the probabilities closer to `1.00` are suppressed much lesser than those closer to `0.00`. But, probably that's a desirable quality. And since the higher p's provide better results, probably power averaging on rank averaging doesn't sound that bad an idea after all.\n\nWhat do you think?", "923129": "Thanks a lot for this @sirishks, very helpful!\n\nBut we need to be **extremely** careful about choosing the models for power averaging, one bad model and everything might go for a toss.\n\nAlso, in point 3 you mention:\n>Weighted Averaging: Specify weights, say 0.5 each in case of two models\nWeightedAverage(p) = (wt1 x Pred1 + wt2 x Pred2 + β¦ + wtn x Predn) / n\nwhere, n is the number of models, and sum of weights wt1+wt2+β¦+wtn = 1\n\nShouldn't we have `wt1+...+wtn = n`, since we are dividing the equation by `n` OR just remove `n` from the denominator and have `wt1+...+wtn = 1`? Correct me if I am wrong.", "3361837": "Well Written!\n\n Have a look at my notebook https://www.kaggle.com/code/shivanisharma1297/ai-text-detection-myensemble", "969206": "Nice Explanation ", "951748": "@sirishks thanks for the post. In the mentioned blog, there is a term `weight_initializer `in part 5 of blog which I could not understand. Can you please clarify what is `weight_initializer `and how one can calculate it ?", "949974": "Thanks a lot for sharing the article\nHow do you define highly correlation between models .Is it CV or Model architecture or something else?", "928929": "Your 5th method is wrong. Don't you think it should be ((pred1^p + pred2^p + .... predn^p)/n)^(1/n)\nIf it is not then you are basically changing the scale.", "926101": "Nice insight", "923270": "Cool, thanks for sharing! π Let me try this power averaging method! ", "961622": "", "930093": "thanks for your contibution" } }