{
  "id": 56120,
  "title": "How much gain do you get from ensemble model？",
  "url": "/competitions/talkingdata-adtracking-fraud-detection/discussion/56120",
  "author_name": "",
  "post_date": "2018-05-06T11:51:19.971878400Z",
  "votes": 8,
  "comment_count": 24,
  "views": 0,
  "content": "<p>Since it is very time-consuming to do a k-folds to get oof.  </p>\n\n<p>I just skip stacking.\nSo I want to know how much gain do you get from ensemble model. </p>\n\n<p>Especially different scale model. I find the predicted value of NN is very small. Maybe the staking is best way to handle this.  If I have more powerful work station or time..</p>",
  "messages": [
    {
      "id": "323842",
      "postDate": "05/06/2018 11:51:19",
      "content": "<p>Since it is very time-consuming to do a k-folds to get oof.  </p>\n\n<p>I just skip stacking.\nSo I want to know how much gain do you get from ensemble model. </p>\n\n<p>Especially different scale model. I find the predicted value of NN is very small. Maybe the staking is best way to handle this.  If I have more powerful work station or time..</p>",
      "rawMarkdown": "Since it is very time-consuming to do a k-folds to get oof.  \n\nI just skip stacking.\nSo I want to know how much gain do you get from ensemble model. \n\nEspecially different scale model. I find the predicted value of NN is very small. Maybe the staking is best way to handle this.  If I have more powerful work station or time..",
      "votes": null
    },
    {
      "id": "323861",
      "postDate": "05/06/2018 12:37:00",
      "content": "<p>We've tried weighted rank average, but it seems the results were not stable. In most cases, we got a 0.0002 lift.</p>",
      "rawMarkdown": "We've tried weighted rank average, but it seems the results were not stable. In most cases, we got a 0.0002 lift.",
      "votes": null
    },
    {
      "id": "323876",
      "postDate": "05/06/2018 13:37:02",
      "content": "<p>stacking-nn, you can get some idea from this link:<a href=\"https://github.com/daxiongshu/stack-nn-tensorflow\">https://github.com/daxiongshu/stack-nn-tensorflow</a></p>",
      "rawMarkdown": "stacking-nn, you can get some idea from this link:https://github.com/daxiongshu/stack-nn-tensorflow",
      "votes": null
    },
    {
      "id": "324210",
      "postDate": "05/07/2018 11:37:12",
      "content": "<p>We get only around 0.0002/3 increase from our best model. I am not sure if it is worth it.</p>",
      "rawMarkdown": "We get only around 0.0002/3 increase from our best model. I am not sure if it is worth it.",
      "votes": null
    },
    {
      "id": "324235",
      "postDate": "05/07/2018 13:11:27",
      "content": "<p>9806 &amp; 9807 =&gt; 9811\n9807 &amp; 9811 =&gt; 9814\ntwo 9812 results =&gt; 9816</p>",
      "rawMarkdown": "9806 &amp; 9807 =&gt; 9811\n9807 &amp; 9811 =&gt; 9814\ntwo 9812 results =&gt; 9816",
      "votes": null
    },
    {
      "id": "324238",
      "postDate": "05/07/2018 13:16:35",
      "content": "<p>wow, is blending with blended results really work? What strategy did you use? (mean/harmonic/log) Are all the models different? (I mean, trained on xgb, lgb, etc. Btw, blending xgb with lgb only gives me 0.0001 boost</p>",
      "rawMarkdown": "wow, is blending with blended results really work? What strategy did you use? (mean/harmonic/log) Are all the models different? (I mean, trained on xgb, lgb, etc. Btw, blending xgb with lgb only gives me 0.0001 boost",
      "votes": null
    },
    {
      "id": "324240",
      "postDate": "05/07/2018 13:19:27",
      "content": "<p>just normal weighted average : )</p>",
      "rawMarkdown": "just normal weighted average : )",
      "votes": null
    },
    {
      "id": "324242",
      "postDate": "05/07/2018 13:21:44",
      "content": "<p>BTW, the models I mentioned were blended by two players' results, thus they should have more diversities than your XGB &amp; LGB</p>",
      "rawMarkdown": "BTW, the models I mentioned were blended by two players' results, thus they should have more diversities than your XGB &amp; LGB",
      "votes": null
    },
    {
      "id": "324244",
      "postDate": "05/07/2018 13:21:46",
      "content": "<p>Thanks, that's impressive. I'd like to have a try</p>",
      "rawMarkdown": "Thanks, that's impressive. I'd like to have a try",
      "votes": null
    },
    {
      "id": "324286",
      "postDate": "05/07/2018 14:25:16",
      "content": "<p>0.9824 &amp; 0.9821 -&gt; 0.9826</p>",
      "rawMarkdown": "0.9824 &amp; 0.9821 -&gt; 0.9826",
      "votes": null
    },
    {
      "id": "324300",
      "postDate": "05/07/2018 14:47:46",
      "content": "<p>0.9811 &amp; 0.9809 &amp; 0.9772 &amp; 0.9796 =&gt; 0.9815</p>",
      "rawMarkdown": "0.9811 &amp; 0.9809 &amp; 0.9772 &amp; 0.9796 =&gt; 0.9815",
      "votes": null
    },
    {
      "id": "324321",
      "postDate": "05/07/2018 15:21:18",
      "content": "<p>0.0 for now :(</p>\n\n<p>Last try: 0.9824 &amp; 0.9824 = 0.9823 ?!?</p>",
      "rawMarkdown": "0.0 for now :(\n\nLast try: 0.9824 &amp; 0.9824 = 0.9823 ?!?",
      "votes": null
    },
    {
      "id": "324324",
      "postDate": "05/07/2018 15:27:01",
      "content": "<p>wow .. your single model is toooooo strong</p>",
      "rawMarkdown": "wow .. your single model is toooooo strong",
      "votes": null
    },
    {
      "id": "324327",
      "postDate": "05/07/2018 15:29:25",
      "content": "<p>Yeah, I focused on improving a single model given all my attempts to ensemble lead to at most 0.0001 improvement.  I must be missing something...  Hope to know why when reading  top teams write ups!</p>",
      "rawMarkdown": "Yeah, I focused on improving a single model given all my attempts to ensemble lead to at most 0.0001 improvement.  I must be missing something...  Hope to know why when reading  top teams write ups!",
      "votes": null
    },
    {
      "id": "324339",
      "postDate": "05/07/2018 15:47:50",
      "content": "<p>Final update before the DDL: \n9817 &amp; 9813 &amp; 9813 =&gt; 9819  (wish to keep the place in private board)</p>",
      "rawMarkdown": "Final update before the DDL: \n9817 &amp; 9813 &amp; 9813 =&gt; 9819  (wish to keep the place in private board)",
      "votes": null
    },
    {
      "id": "324343",
      "postDate": "05/07/2018 15:53:15",
      "content": "<p>Our last submission is our best for now: </p>\n\n<p>public 0.9764 NN - thanks to @<strong>HuyenNguyen</strong></p>\n\n<p><a href=\"https://www.kaggle.com/huyenvyvy/deep-learning-kernel-runnable-0-9760/output\">Deep Learning kernel</a>\nOur: 0.9796 Light GBM \nOur: ???? Light GBM with much better CV score than our second model, both were splitted into 5 sets, then lightGBM was 5 times scoring test set. Finally we add results and divide them by 5 - beacuse of the lack of memory.</p>\n\n<p>public + our1 + our2 = 0.9811</p>",
      "rawMarkdown": "Our last submission is our best for now: \n\npublic 0.9764 NN - thanks to @**HuyenNguyen**\n\n[Deep Learning kernel][1]\n  [1]: https://www.kaggle.com/huyenvyvy/deep-learning-kernel-runnable-0-9760/output\n\nOur: 0.9796 Light GBM \nOur: ???? Light GBM with much better CV score than our second model, both were splitted into 5 sets, then lightGBM was 5 times scoring test set. Finally we add results and divide them by 5 - beacuse of the lack of memory.\n\npublic + our1 + our2 = 0.9811",
      "votes": null
    },
    {
      "id": "324357",
      "postDate": "05/07/2018 16:12:51",
      "content": "<p>For my understanding, ensemble is always about single model performance and diversity which is from three perspectives:  models diversity, feature sets diversity and training size diversity.</p>",
      "rawMarkdown": "For my understanding, ensemble is always about single model performance and diversity which is from three perspectives:  models diversity, feature sets diversity and training size diversity.",
      "votes": null
    },
    {
      "id": "324371",
      "postDate": "05/07/2018 16:26:43",
      "content": "<p>@KALE, I agree, and I used ensembling and stacking successfully in other competitions.  Here, a simple average can degrade performance significantly.</p>",
      "rawMarkdown": "KALE, I agree, and I used ensembling and stacking successfully in other competitions.  Here, a simple average can degrade performance significantly.",
      "votes": null
    },
    {
      "id": "324575",
      "postDate": "05/07/2018 20:09:42",
      "content": "<p>0.9814, 0.9814, 0.9813, 0.9812, 0.9810 -&gt; 0.9819/0.9818 for weighted mean/ median accordingly.\nThe measurement are from different times so there is a bit of diversity in them</p>",
      "rawMarkdown": "0.9814, 0.9814, 0.9813, 0.9812, 0.9810 -&gt; 0.9819/0.9818 for weighted mean/ median accordingly.\nThe measurement are from different times so there is a bit of diversity in them",
      "votes": null
    },
    {
      "id": "324633",
      "postDate": "05/07/2018 21:40:33",
      "content": "<p>My experience is if the learning rate is small in lightgbm, the gain from ensemble would be small. I was able to get 9820 with 9818+9815, but only able to get 9822 with multiple 9821...</p>",
      "rawMarkdown": "My experience is if the learning rate is small in lightgbm, the gain from ensemble would be small. I was able to get 9820 with 9818+9815, but only able to get 9822 with multiple 9821...",
      "votes": null
    },
    {
      "id": "324855",
      "postDate": "05/08/2018 00:43:30",
      "content": "<p>We ended about .0007 above our best single model through a combination of stacking and blending (of stacks), so ensembling made all the difference for us.</p>",
      "rawMarkdown": "We ended about .0007 above our best single model through a combination of stacking and blending (of stacks), so ensembling made all the difference for us.",
      "votes": null
    },
    {
      "id": "324862",
      "postDate": "05/08/2018 00:47:57",
      "content": "<p>I checked my private LB and noticed ALL my ensembles did not improve my best single model at all... Maybe we can team up in next competition =)</p>",
      "rawMarkdown": "I checked my private LB and noticed ALL my ensembles did not improve my best single model at all... Maybe we can team up in next competition =)",
      "votes": null
    },
    {
      "id": "324946",
      "postDate": "05/08/2018 02:04:21",
      "content": "<p>For this competition, because only rank is important - blending worked for me on the level of ranks, helping me lift public leader board score by about 0.003. I think that different LGBM models may not be too good to average directly if you play too much with the parameters - e.g. scale_pos_weight - as it may lead to different scale of target variable.</p>",
      "rawMarkdown": "For this competition, because only rank is important - blending worked for me on the level of ranks, helping me lift public leader board score by about 0.003. I think that different LGBM models may not be too good to average directly if you play too much with the parameters - e.g. scale_pos_weight - as it may lead to different scale of target variable.",
      "votes": null
    },
    {
      "id": "325050",
      "postDate": "05/08/2018 04:09:03",
      "content": "<p>The better single model you have, the less you will gain from ensemble~</p>",
      "rawMarkdown": "The better single model you have, the less you will gain from ensemble~",
      "votes": null
    },
    {
      "id": "325442",
      "postDate": "05/08/2018 12:22:55",
      "content": "<p>@Cheng, I got at most 0.0001 uplift with blending, you're not alone.  I think the reason is what @KALE says.</p>",
      "rawMarkdown": "Cheng, I got at most 0.0001 uplift with blending, you're not alone.  I think the reason is what @KALE says.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 323861,
      "author_name": "laevatein",
      "author_url": "",
      "post_date": "05/06/2018 12:37:00",
      "content": "<p>We've tried weighted rank average, but it seems the results were not stable. In most cases, we got a 0.0002 lift.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 323876,
      "author_name": "mapodoufu",
      "author_url": "",
      "post_date": "05/06/2018 13:37:02",
      "content": "<p>stacking-nn, you can get some idea from this link:<a href=\"https://github.com/daxiongshu/stack-nn-tensorflow\">https://github.com/daxiongshu/stack-nn-tensorflow</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 324210,
      "author_name": "antmarakis",
      "author_url": "",
      "post_date": "05/07/2018 11:37:12",
      "content": "<p>We get only around 0.0002/3 increase from our best model. I am not sure if it is worth it.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 324235,
      "author_name": "yiheng",
      "author_url": "",
      "post_date": "05/07/2018 13:11:27",
      "content": "<p>9806 &amp; 9807 =&gt; 9811\n9807 &amp; 9811 =&gt; 9814\ntwo 9812 results =&gt; 9816</p>",
      "votes": null,
      "replies": [
        {
          "id": 324238,
          "author_name": "atm584",
          "author_url": "",
          "post_date": "05/07/2018 13:16:35",
          "content": "<p>wow, is blending with blended results really work? What strategy did you use? (mean/harmonic/log) Are all the models different? (I mean, trained on xgb, lgb, etc. Btw, blending xgb with lgb only gives me 0.0001 boost</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 324240,
          "author_name": "yiheng",
          "author_url": "",
          "post_date": "05/07/2018 13:19:27",
          "content": "<p>just normal weighted average : )</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 324242,
          "author_name": "yiheng",
          "author_url": "",
          "post_date": "05/07/2018 13:21:44",
          "content": "<p>BTW, the models I mentioned were blended by two players' results, thus they should have more diversities than your XGB &amp; LGB</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 324244,
          "author_name": "atm584",
          "author_url": "",
          "post_date": "05/07/2018 13:21:46",
          "content": "<p>Thanks, that's impressive. I'd like to have a try</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 324339,
          "author_name": "yiheng",
          "author_url": "",
          "post_date": "05/07/2018 15:47:50",
          "content": "<p>Final update before the DDL: \n9817 &amp; 9813 &amp; 9813 =&gt; 9819  (wish to keep the place in private board)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 324286,
      "author_name": "senkin13",
      "author_url": "",
      "post_date": "05/07/2018 14:25:16",
      "content": "<p>0.9824 &amp; 0.9821 -&gt; 0.9826</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 324300,
      "author_name": "deepdreamer",
      "author_url": "",
      "post_date": "05/07/2018 14:47:46",
      "content": "<p>0.9811 &amp; 0.9809 &amp; 0.9772 &amp; 0.9796 =&gt; 0.9815</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 324321,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "05/07/2018 15:21:18",
      "content": "<p>0.0 for now :(</p>\n\n<p>Last try: 0.9824 &amp; 0.9824 = 0.9823 ?!?</p>",
      "votes": null,
      "replies": [
        {
          "id": 324324,
          "author_name": "baomengjiao",
          "author_url": "",
          "post_date": "05/07/2018 15:27:01",
          "content": "<p>wow .. your single model is toooooo strong</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 324327,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "05/07/2018 15:29:25",
          "content": "<p>Yeah, I focused on improving a single model given all my attempts to ensemble lead to at most 0.0001 improvement.  I must be missing something...  Hope to know why when reading  top teams write ups!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 324357,
          "author_name": "cczaixian",
          "author_url": "",
          "post_date": "05/07/2018 16:12:51",
          "content": "<p>For my understanding, ensemble is always about single model performance and diversity which is from three perspectives:  models diversity, feature sets diversity and training size diversity.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 324371,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "05/07/2018 16:26:43",
          "content": "<p>@KALE, I agree, and I used ensembling and stacking successfully in other competitions.  Here, a simple average can degrade performance significantly.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 324575,
          "author_name": "mrbeer",
          "author_url": "",
          "post_date": "05/07/2018 20:09:42",
          "content": "<p>0.9814, 0.9814, 0.9813, 0.9812, 0.9810 -&gt; 0.9819/0.9818 for weighted mean/ median accordingly.\nThe measurement are from different times so there is a bit of diversity in them</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 324633,
          "author_name": "chengju",
          "author_url": "",
          "post_date": "05/07/2018 21:40:33",
          "content": "<p>My experience is if the learning rate is small in lightgbm, the gain from ensemble would be small. I was able to get 9820 with 9818+9815, but only able to get 9822 with multiple 9821...</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 324343,
      "author_name": "meykds",
      "author_url": "",
      "post_date": "05/07/2018 15:53:15",
      "content": "<p>Our last submission is our best for now: </p>\n\n<p>public 0.9764 NN - thanks to @<strong>HuyenNguyen</strong></p>\n\n<p><a href=\"https://www.kaggle.com/huyenvyvy/deep-learning-kernel-runnable-0-9760/output\">Deep Learning kernel</a>\nOur: 0.9796 Light GBM \nOur: ???? Light GBM with much better CV score than our second model, both were splitted into 5 sets, then lightGBM was 5 times scoring test set. Finally we add results and divide them by 5 - beacuse of the lack of memory.</p>\n\n<p>public + our1 + our2 = 0.9811</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 324855,
      "author_name": "aquatic",
      "author_url": "",
      "post_date": "05/08/2018 00:43:30",
      "content": "<p>We ended about .0007 above our best single model through a combination of stacking and blending (of stacks), so ensembling made all the difference for us.</p>",
      "votes": null,
      "replies": [
        {
          "id": 324862,
          "author_name": "chengju",
          "author_url": "",
          "post_date": "05/08/2018 00:47:57",
          "content": "<p>I checked my private LB and noticed ALL my ensembles did not improve my best single model at all... Maybe we can team up in next competition =)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 325050,
          "author_name": "cczaixian",
          "author_url": "",
          "post_date": "05/08/2018 04:09:03",
          "content": "<p>The better single model you have, the less you will gain from ensemble~</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 325442,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "05/08/2018 12:22:55",
          "content": "<p>@Cheng, I got at most 0.0001 uplift with blending, you're not alone.  I think the reason is what @KALE says.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 324946,
      "author_name": "pgyrya",
      "author_url": "",
      "post_date": "05/08/2018 02:04:21",
      "content": "<p>For this competition, because only rank is important - blending worked for me on the level of ranks, helping me lift public leader board score by about 0.003. I think that different LGBM models may not be too good to average directly if you play too much with the parameters - e.g. scale_pos_weight - as it may lead to different scale of target variable.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "323842": "Since it is very time-consuming to do a k-folds to get oof.  \n\nI just skip stacking.\nSo I want to know how much gain do you get from ensemble model. \n\nEspecially different scale model. I find the predicted value of NN is very small. Maybe the staking is best way to handle this.  If I have more powerful work station or time..",
    "323861": "We've tried weighted rank average, but it seems the results were not stable. In most cases, we got a 0.0002 lift.",
    "323876": "stacking-nn, you can get some idea from this link:https://github.com/daxiongshu/stack-nn-tensorflow",
    "324210": "We get only around 0.0002/3 increase from our best model. I am not sure if it is worth it.",
    "324235": "9806 &amp; 9807 =&gt; 9811\n9807 &amp; 9811 =&gt; 9814\ntwo 9812 results =&gt; 9816",
    "324238": "wow, is blending with blended results really work? What strategy did you use? (mean/harmonic/log) Are all the models different? (I mean, trained on xgb, lgb, etc. Btw, blending xgb with lgb only gives me 0.0001 boost",
    "324240": "just normal weighted average : )",
    "324242": "BTW, the models I mentioned were blended by two players' results, thus they should have more diversities than your XGB &amp; LGB",
    "324244": "Thanks, that's impressive. I'd like to have a try",
    "324286": "0.9824 &amp; 0.9821 -&gt; 0.9826",
    "324300": "0.9811 &amp; 0.9809 &amp; 0.9772 &amp; 0.9796 =&gt; 0.9815",
    "324321": "0.0 for now :(\n\nLast try: 0.9824 &amp; 0.9824 = 0.9823 ?!?",
    "324324": "wow .. your single model is toooooo strong",
    "324327": "Yeah, I focused on improving a single model given all my attempts to ensemble lead to at most 0.0001 improvement.  I must be missing something...  Hope to know why when reading  top teams write ups!",
    "324339": "Final update before the DDL: \n9817 &amp; 9813 &amp; 9813 =&gt; 9819  (wish to keep the place in private board)",
    "324343": "Our last submission is our best for now: \n\npublic 0.9764 NN - thanks to @**HuyenNguyen**\n\n[Deep Learning kernel][1]\n  [1]: https://www.kaggle.com/huyenvyvy/deep-learning-kernel-runnable-0-9760/output\n\nOur: 0.9796 Light GBM \nOur: ???? Light GBM with much better CV score than our second model, both were splitted into 5 sets, then lightGBM was 5 times scoring test set. Finally we add results and divide them by 5 - beacuse of the lack of memory.\n\npublic + our1 + our2 = 0.9811",
    "324357": "For my understanding, ensemble is always about single model performance and diversity which is from three perspectives:  models diversity, feature sets diversity and training size diversity.",
    "324371": "KALE, I agree, and I used ensembling and stacking successfully in other competitions.  Here, a simple average can degrade performance significantly.",
    "324575": "0.9814, 0.9814, 0.9813, 0.9812, 0.9810 -&gt; 0.9819/0.9818 for weighted mean/ median accordingly.\nThe measurement are from different times so there is a bit of diversity in them",
    "324633": "My experience is if the learning rate is small in lightgbm, the gain from ensemble would be small. I was able to get 9820 with 9818+9815, but only able to get 9822 with multiple 9821...",
    "324855": "We ended about .0007 above our best single model through a combination of stacking and blending (of stacks), so ensembling made all the difference for us.",
    "324862": "I checked my private LB and noticed ALL my ensembles did not improve my best single model at all... Maybe we can team up in next competition =)",
    "324946": "For this competition, because only rank is important - blending worked for me on the level of ranks, helping me lift public leader board score by about 0.003. I think that different LGBM models may not be too good to average directly if you play too much with the parameters - e.g. scale_pos_weight - as it may lead to different scale of target variable.",
    "325050": "The better single model you have, the less you will gain from ensemble~",
    "325442": "Cheng, I got at most 0.0001 uplift with blending, you're not alone.  I think the reason is what @KALE says."
  },
  "source": "meta"
}