![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2510328%2F9d32e2a8e63ef772c2dac0196752488b%2Ffindbestalpha.png?generation=1581597400311903&alt=media\")
How to decide the weights?
", "messages": [ { "id": "744876", "postDate": "02/13/2020 08:39:07", "content": "How to decide the weights?
", "rawMarkdown": "How to decide the weights?", "votes": null }, { "id": "745052", "postDate": "02/13/2020 12:36:53", "content": "Let me explain one way you might do this:
\n\nSay we've two models with outputs A and B.
\n\nA very simple ensemble would be:\n(A + B) / 2 = A / 2 + B / 2 = 0.5 * A + 0.5 * B = 0.5 * A + (1 - 0.5) * B.
\n\nSo the value 0.5 means that we're giving both outputs the same weight.
\n\nSo for the ensemble ALPHA * A + (1 - ALPHA) * B, what you want to do is decide on the best ALPHA.
\n\nThe simplest way do this is to try all possible values for ALPHA on a validation set and pick up the one that yields the highest score.
\n\nYou can view this kernel in which @wrrosa does exactly that.
\n\n
Hope this helps.
", "rawMarkdown": "Let me explain one way you might do this:\n\nSay we've two models with outputs A and B. \n\nA very simple ensemble would be:\n(A + B) / 2 = A / 2 + B / 2 = 0.5 * A + 0.5 * B = 0.5 * A + (1 - 0.5) * B.\n\nSo the value 0.5 means that we're giving both outputs the same weight.\n\nSo for the ensemble ALPHA * A + (1 - ALPHA) * B, what you want to do is decide on the best ALPHA.\n\nThe simplest way do this is to try all possible values for ALPHA on a validation set and pick up the one that yields the highest score.\n\nYou can view this [kernel](https://www.kaggle.com/wrrosa/tpu-enet-b7-densenet) in which @wrrosa does exactly that.\n\n\n\nHope this helps.", "votes": null }, { "id": "745118", "postDate": "02/13/2020 14:11:07", "content": "Thank you.
", "rawMarkdown": "Thank you.", "votes": null }, { "id": "745146", "postDate": "02/13/2020 14:52:55", "content": "Or you can do it like this with 3 models:
\n\n\n\nNote that it is important to validate the combination, because the peak can shift for different data constellations.
", "rawMarkdown": "Or you can do it like this with 3 models:\n\n[Combining 3 Models](https://www.kaggle.com/frankmollard/bag3models)\n\n*Note that it is important to validate the combination, because the peak can shift for different data constellations.*", "votes": null }, { "id": "745156", "postDate": "02/13/2020 15:04:43", "content": "Anytime. Don't forget to upvote answers you find helpful ;)
", "rawMarkdown": "Anytime. Don't forget to upvote answers you find helpful ;)", "votes": null }, { "id": "745352", "postDate": "02/13/2020 18:28:25", "content": "Open question here: isn't it better to ensemble models before applying softmax rather than after ?\nThe main idea of ensembling is that if one model has a lukewarm opinion about a class but the other model identifies it strongly, then the answer is the strong opinion.
\n\nBut after softmax, there are only strong opinions left. That's the whole point of softmax. Doesn't this run contrary to what ensembling is supposed to achieve ?
", "rawMarkdown": "Open question here: isn't it better to ensemble models **before** applying softmax rather than after ?\nThe main idea of ensembling is that if one model has a lukewarm opinion about a class but the other model identifies it strongly, then the answer is the strong opinion.\n\nBut after softmax, there are only strong opinions left. That's the whole point of softmax. Doesn't this run contrary to what ensembling is supposed to achieve ?", "votes": null }, { "id": "745915", "postDate": "02/14/2020 11:22:47", "content": "So you mean if we have 2 models then we add the outputs of those models first then we apply softmax. But when we add the 2 models before softmax we will also need some weights.
\n\nIf I am wrong somewhere in understanding please correct me! 😅
", "rawMarkdown": "So you mean if we have 2 models then we add the outputs of those models first then we apply softmax. But when we add the 2 models before softmax we will also need some weights. \n\nIf I am wrong somewhere in understanding please correct me! 😅", "votes": null }, { "id": "746229", "postDate": "02/14/2020 19:09:24", "content": "I was not talking about the weights, assuming 0.5/0.5
", "rawMarkdown": "I was not talking about the weights, assuming 0.5/0.5", "votes": null } ], "comments": [ { "id": 745052, "author_name": "msheriey", "author_url": "", "post_date": "02/13/2020 12:36:53", "content": "Let me explain one way you might do this:
\n\nSay we've two models with outputs A and B.
\n\nA very simple ensemble would be:\n(A + B) / 2 = A / 2 + B / 2 = 0.5 * A + 0.5 * B = 0.5 * A + (1 - 0.5) * B.
\n\nSo the value 0.5 means that we're giving both outputs the same weight.
\n\nSo for the ensemble ALPHA * A + (1 - ALPHA) * B, what you want to do is decide on the best ALPHA.
\n\nThe simplest way do this is to try all possible values for ALPHA on a validation set and pick up the one that yields the highest score.
\n\nYou can view this kernel in which @wrrosa does exactly that.
\n\n
Hope this helps.
", "votes": null, "replies": [ { "id": 745118, "author_name": "imvivek14", "author_url": "", "post_date": "02/13/2020 14:11:07", "content": "Thank you.
", "votes": null, "replies": [] }, { "id": 745156, "author_name": "msheriey", "author_url": "", "post_date": "02/13/2020 15:04:43", "content": "Anytime. Don't forget to upvote answers you find helpful ;)
", "votes": null, "replies": [] } ] }, { "id": 745146, "author_name": "frankmollard", "author_url": "", "post_date": "02/13/2020 14:52:55", "content": "Or you can do it like this with 3 models:
\n\n\n\nNote that it is important to validate the combination, because the peak can shift for different data constellations.
", "votes": null, "replies": [] }, { "id": 745352, "author_name": "mgorner", "author_url": "", "post_date": "02/13/2020 18:28:25", "content": "Open question here: isn't it better to ensemble models before applying softmax rather than after ?\nThe main idea of ensembling is that if one model has a lukewarm opinion about a class but the other model identifies it strongly, then the answer is the strong opinion.
\n\nBut after softmax, there are only strong opinions left. That's the whole point of softmax. Doesn't this run contrary to what ensembling is supposed to achieve ?
", "votes": null, "replies": [ { "id": 745915, "author_name": "imvivek14", "author_url": "", "post_date": "02/14/2020 11:22:47", "content": "So you mean if we have 2 models then we add the outputs of those models first then we apply softmax. But when we add the 2 models before softmax we will also need some weights.
\n\nIf I am wrong somewhere in understanding please correct me! 😅
", "votes": null, "replies": [] }, { "id": 746229, "author_name": "mgorner", "author_url": "", "post_date": "02/14/2020 19:09:24", "content": "I was not talking about the weights, assuming 0.5/0.5
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "744876": "How to decide the weights?", "745052": "Let me explain one way you might do this:\n\nSay we've two models with outputs A and B. \n\nA very simple ensemble would be:\n(A + B) / 2 = A / 2 + B / 2 = 0.5 * A + 0.5 * B = 0.5 * A + (1 - 0.5) * B.\n\nSo the value 0.5 means that we're giving both outputs the same weight.\n\nSo for the ensemble ALPHA * A + (1 - ALPHA) * B, what you want to do is decide on the best ALPHA.\n\nThe simplest way do this is to try all possible values for ALPHA on a validation set and pick up the one that yields the highest score.\n\nYou can view this [kernel](https://www.kaggle.com/wrrosa/tpu-enet-b7-densenet) in which @wrrosa does exactly that.\n\n\n\nHope this helps.", "745118": "Thank you.", "745146": "Or you can do it like this with 3 models:\n\n[Combining 3 Models](https://www.kaggle.com/frankmollard/bag3models)\n\n*Note that it is important to validate the combination, because the peak can shift for different data constellations.*", "745156": "Anytime. Don't forget to upvote answers you find helpful ;)", "745352": "Open question here: isn't it better to ensemble models **before** applying softmax rather than after ?\nThe main idea of ensembling is that if one model has a lukewarm opinion about a class but the other model identifies it strongly, then the answer is the strong opinion.\n\nBut after softmax, there are only strong opinions left. That's the whole point of softmax. Doesn't this run contrary to what ensembling is supposed to achieve ?", "745915": "So you mean if we have 2 models then we add the outputs of those models first then we apply softmax. But when we add the 2 models before softmax we will also need some weights. \n\nIf I am wrong somewhere in understanding please correct me! 😅", "746229": "I was not talking about the weights, assuming 0.5/0.5" }, "source": "meta" }