{
  "id": 93431,
  "title": "Confusion of the ensemble model",
  "url": "/competitions/LANL-Earthquake-Prediction/discussion/93431",
  "author_name": "",
  "post_date": "2019-05-27T04:31:36.490598400Z",
  "votes": -2,
  "comment_count": 12,
  "views": 0,
  "content": "<p>I believe that all of you guys find that it is easy to get a better score by simply fuse two model. \nFor example, i have a lgbm model with 1.414 score and have another nn model with 1.412.And then, ensemble them by 0.7 * nn  prediction result+ 0.3 * lgbm prediction result.\n```python</p>\n\n<h1>lgbmpred have 1.414 score</h1>\n\n<h1>nnpred have 1.412 score</h1>\n\n<p>fusionpred['time to failure'] = lgbmpred['time to failure']  * 0.7 + lgbmpred['time to failure']  * 0.3\n```\nFinally, I got a amazing score 1.404 and get a bronze medal place.</p>\n\n<p>My confusion is whether such ensemble model will collapse in private leaderboard?Because I think this way will lead to overfitting. \nIf it will be, how can I solve it?</p>\n\n<p>Thanks for bluetrain to point out my term mistake!</p>",
  "messages": [
    {
      "id": "537470",
      "postDate": "05/27/2019 04:31:36",
      "content": "<p>I believe that all of you guys find that it is easy to get a better score by simply fuse two model. \nFor example, i have a lgbm model with 1.414 score and have another nn model with 1.412.And then, ensemble them by 0.7 * nn  prediction result+ 0.3 * lgbm prediction result.\n```python</p>\n\n<h1>lgbmpred have 1.414 score</h1>\n\n<h1>nnpred have 1.412 score</h1>\n\n<p>fusionpred['time to failure'] = lgbmpred['time to failure']  * 0.7 + lgbmpred['time to failure']  * 0.3\n```\nFinally, I got a amazing score 1.404 and get a bronze medal place.</p>\n\n<p>My confusion is whether such ensemble model will collapse in private leaderboard?Because I think this way will lead to overfitting. \nIf it will be, how can I solve it?</p>\n\n<p>Thanks for bluetrain to point out my term mistake!</p>",
      "rawMarkdown": "I believe that all of you guys find that it is easy to get a better score by simply fuse two model. \nFor example, i have a lgbm model with 1.414 score and have another nn model with 1.412.And then, ensemble them by 0.7 * nn  prediction result+ 0.3 * lgbm prediction result.\n```python\n# lgbmpred have 1.414 score\n# nnpred have 1.412 score\nfusionpred['time to failure'] = lgbmpred['time to failure']  * 0.7 + lgbmpred['time to failure']  * 0.3\n```\nFinally, I got a amazing score 1.404 and get a bronze medal place.\n\nMy confusion is whether such ensemble model will collapse in private leaderboard?Because I think this way will lead to overfitting. \nIf it will be, how can I solve it?\n\nThanks for bluetrain to point out my term mistake!",
      "votes": null
    },
    {
      "id": "537497",
      "postDate": "05/27/2019 06:03:43",
      "content": "<p>I would say the contrary: a \"fusion\" (I prefer \"ensemble\") of different models reduces the variance and has a higher chance to generalize better to private LB. \nBtw: don't look at the public LB only ;)</p>",
      "rawMarkdown": "I would say the contrary: a \"fusion\" (I prefer \"ensemble\") of different models reduces the variance and has a higher chance to generalize better to private LB. \nBtw: don't look at the public LB only ;)",
      "votes": null
    },
    {
      "id": "537498",
      "postDate": "05/27/2019 06:06:59",
      "content": "<p>how did you tune the 0.3/0.7 parameters ?\ndid you test them with  oof of your models  or with lb probing ?\nThe second one is the main street to overfitting </p>",
      "rawMarkdown": "how did you tune the 0.3/0.7 parameters ?\ndid you test them with  oof of your models  or with lb probing ?\nThe second one is the main street to overfitting",
      "votes": null
    },
    {
      "id": "537525",
      "postDate": "05/27/2019 07:10:22",
      "content": "<p>&gt; different models reduces the variance and has a higher chance to generalize</p>\n\n<p>I see it the same way. But in this competition like discussed <a href=\"https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/90281#latest-537346\">here</a> or <a href=\"https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/93148#latest-537413\">here</a> it's reasonable not to trust the public LB.</p>",
      "rawMarkdown": "&gt; different models reduces the variance and has a higher chance to generalize\n\nI see it the same way. But in this competition like discussed [here](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/90281#latest-537346) or [here](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/93148#latest-537413) it's reasonable not to trust the public LB.",
      "votes": null
    },
    {
      "id": "537546",
      "postDate": "05/27/2019 08:07:49",
      "content": "<p>test it several times, and mainly by my own experience  about model building. \nBTW, a little trick, maybe you can give higher-score prediction more weight, but no too more.:)</p>",
      "rawMarkdown": "test it several times, and mainly by my own experience  about model building. \nBTW, a little trick, maybe you can give higher-score prediction more weight, but no too more.:)",
      "votes": null
    },
    {
      "id": "537548",
      "postDate": "05/27/2019 08:10:28",
      "content": "<p>Thank you! You give me confidence from a surprise perspective.\nAnd I want to ask one more question? Is it the more varieties of model I  use, such as linear, tree and NN and so on. The more variance I can reduce?</p>",
      "rawMarkdown": "Thank you! You give me confidence from a surprise perspective.\nAnd I want to ask one more question? Is it the more varieties of model I  use, such as linear, tree and NN and so on. The more variance I can reduce?",
      "votes": null
    },
    {
      "id": "537551",
      "postDate": "05/27/2019 08:18:21",
      "content": "<blockquote>\n  <p>linear, tree and NN</p>\n</blockquote>\n\n<p>If the models have a similar CV the combination should reduce the variance, IMHO.</p>",
      "rawMarkdown": "&gt; linear, tree and NN\n\nIf the models have a similar CV the combination should reduce the variance, IMHO.",
      "votes": null
    },
    {
      "id": "537666",
      "postDate": "05/27/2019 12:36:25",
      "content": "<p>I don't understand. You said you have a NN model with 1.412 and then you got the same score after ensemble. How is that good for you?\nIf that's really the case, you will get a better score with the original model, since you will lose some points for overfitting.</p>",
      "rawMarkdown": "I don't understand. You said you have a NN model with 1.412 and then you got the same score after ensemble. How is that good for you?\nIf that's really the case, you will get a better score with the original model, since you will lose some points for overfitting.",
      "votes": null
    },
    {
      "id": "537708",
      "postDate": "05/27/2019 13:44:58",
      "content": "<p>I am sorry. It's my fault and I've solved it. The score is 1.404</p>",
      "rawMarkdown": "I am sorry. It's my fault and I've solved it. The score is 1.404",
      "votes": null
    },
    {
      "id": "539151",
      "postDate": "05/29/2019 15:43:12",
      "content": "<p>This might help you with understanding how ensembles work: <a href=\"https://mlwave.com/kaggle-ensembling-guide/\">https://mlwave.com/kaggle-ensembling-guide/</a></p>",
      "rawMarkdown": "This might help you with understanding how ensembles work: https://mlwave.com/kaggle-ensembling-guide/",
      "votes": null
    },
    {
      "id": "539671",
      "postDate": "05/30/2019 11:29:34",
      "content": "<p>Maybe stacking could lead in this competition to overfitting, but blending should help to generalize better.</p>",
      "rawMarkdown": "Maybe stacking could lead in this competition to overfitting, but blending should help to generalize better.",
      "votes": null
    },
    {
      "id": "540134",
      "postDate": "05/31/2019 04:44:21",
      "content": "<p>As a general rule, the more variety there is in the models (actually their predictions) that you ensemble the better the results, as long as each model is sufficiently accurate, otherwise it won't contribute usefully.  To measure the difference between two sets of predictions for this contest I am using this R program under Linux OS, which reads a stream of ASCII records with 2 white-space-separated numeric fields (the predictions of each model) on each line, and calculates the mean absolute difference between them:</p>\n\n<p>#!/usr/bin/env Rscript\n  Data &lt;- scan( file = \"stdin\", what = list( double( 0), double( 0)))\n  cat( mean( abs( Data[[ 1]] - Data[[ 2]])), \"\\n\")</p>\n\n<p>Hope this helps.</p>",
      "rawMarkdown": "As a general rule, the more variety there is in the models (actually their predictions) that you ensemble the better the results, as long as each model is sufficiently accurate, otherwise it won't contribute usefully.  To measure the difference between two sets of predictions for this contest I am using this R program under Linux OS, which reads a stream of ASCII records with 2 white-space-separated numeric fields (the predictions of each model) on each line, and calculates the mean absolute difference between them:\n\n  #!/usr/bin/env Rscript\n  Data &lt;- scan( file = \"stdin\", what = list( double( 0), double( 0)))\n  cat( mean( abs( Data[[ 1]] - Data[[ 2]])), \"\\n\")\n\nHope this helps.",
      "votes": null
    },
    {
      "id": "540162",
      "postDate": "05/31/2019 05:43:10",
      "content": "<p>haha, and then identify it youself?</p>",
      "rawMarkdown": "haha, and then identify it youself?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 537497,
      "author_name": "stecasasso",
      "author_url": "",
      "post_date": "05/27/2019 06:03:43",
      "content": "<p>I would say the contrary: a \"fusion\" (I prefer \"ensemble\") of different models reduces the variance and has a higher chance to generalize better to private LB. \nBtw: don't look at the public LB only ;)</p>",
      "votes": null,
      "replies": [
        {
          "id": 537525,
          "author_name": "tocha4",
          "author_url": "",
          "post_date": "05/27/2019 07:10:22",
          "content": "<p>&gt; different models reduces the variance and has a higher chance to generalize</p>\n\n<p>I see it the same way. But in this competition like discussed <a href=\"https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/90281#latest-537346\">here</a> or <a href=\"https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/93148#latest-537413\">here</a> it's reasonable not to trust the public LB.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 537548,
          "author_name": "dyyalex",
          "author_url": "",
          "post_date": "05/27/2019 08:10:28",
          "content": "<p>Thank you! You give me confidence from a surprise perspective.\nAnd I want to ask one more question? Is it the more varieties of model I  use, such as linear, tree and NN and so on. The more variance I can reduce?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 537551,
          "author_name": "tocha4",
          "author_url": "",
          "post_date": "05/27/2019 08:18:21",
          "content": "<blockquote>\n  <p>linear, tree and NN</p>\n</blockquote>\n\n<p>If the models have a similar CV the combination should reduce the variance, IMHO.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 540134,
          "author_name": "dslate",
          "author_url": "",
          "post_date": "05/31/2019 04:44:21",
          "content": "<p>As a general rule, the more variety there is in the models (actually their predictions) that you ensemble the better the results, as long as each model is sufficiently accurate, otherwise it won't contribute usefully.  To measure the difference between two sets of predictions for this contest I am using this R program under Linux OS, which reads a stream of ASCII records with 2 white-space-separated numeric fields (the predictions of each model) on each line, and calculates the mean absolute difference between them:</p>\n\n<p>#!/usr/bin/env Rscript\n  Data &lt;- scan( file = \"stdin\", what = list( double( 0), double( 0)))\n  cat( mean( abs( Data[[ 1]] - Data[[ 2]])), \"\\n\")</p>\n\n<p>Hope this helps.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 540162,
          "author_name": "dyyalex",
          "author_url": "",
          "post_date": "05/31/2019 05:43:10",
          "content": "<p>haha, and then identify it youself?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 537498,
      "author_name": "steubk",
      "author_url": "",
      "post_date": "05/27/2019 06:06:59",
      "content": "<p>how did you tune the 0.3/0.7 parameters ?\ndid you test them with  oof of your models  or with lb probing ?\nThe second one is the main street to overfitting </p>",
      "votes": null,
      "replies": [
        {
          "id": 537546,
          "author_name": "dyyalex",
          "author_url": "",
          "post_date": "05/27/2019 08:07:49",
          "content": "<p>test it several times, and mainly by my own experience  about model building. \nBTW, a little trick, maybe you can give higher-score prediction more weight, but no too more.:)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 537666,
      "author_name": "",
      "author_url": "",
      "post_date": "05/27/2019 12:36:25",
      "content": "<p>I don't understand. You said you have a NN model with 1.412 and then you got the same score after ensemble. How is that good for you?\nIf that's really the case, you will get a better score with the original model, since you will lose some points for overfitting.</p>",
      "votes": null,
      "replies": [
        {
          "id": 537708,
          "author_name": "dyyalex",
          "author_url": "",
          "post_date": "05/27/2019 13:44:58",
          "content": "<p>I am sorry. It's my fault and I've solved it. The score is 1.404</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 539151,
      "author_name": "antaresnyc",
      "author_url": "",
      "post_date": "05/29/2019 15:43:12",
      "content": "<p>This might help you with understanding how ensembles work: <a href=\"https://mlwave.com/kaggle-ensembling-guide/\">https://mlwave.com/kaggle-ensembling-guide/</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 539671,
      "author_name": "carlospk",
      "author_url": "",
      "post_date": "05/30/2019 11:29:34",
      "content": "<p>Maybe stacking could lead in this competition to overfitting, but blending should help to generalize better.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "537470": "I believe that all of you guys find that it is easy to get a better score by simply fuse two model. \nFor example, i have a lgbm model with 1.414 score and have another nn model with 1.412.And then, ensemble them by 0.7 * nn  prediction result+ 0.3 * lgbm prediction result.\n```python\n# lgbmpred have 1.414 score\n# nnpred have 1.412 score\nfusionpred['time to failure'] = lgbmpred['time to failure']  * 0.7 + lgbmpred['time to failure']  * 0.3\n```\nFinally, I got a amazing score 1.404 and get a bronze medal place.\n\nMy confusion is whether such ensemble model will collapse in private leaderboard?Because I think this way will lead to overfitting. \nIf it will be, how can I solve it?\n\nThanks for bluetrain to point out my term mistake!",
    "537497": "I would say the contrary: a \"fusion\" (I prefer \"ensemble\") of different models reduces the variance and has a higher chance to generalize better to private LB. \nBtw: don't look at the public LB only ;)",
    "537498": "how did you tune the 0.3/0.7 parameters ?\ndid you test them with  oof of your models  or with lb probing ?\nThe second one is the main street to overfitting",
    "537525": "&gt; different models reduces the variance and has a higher chance to generalize\n\nI see it the same way. But in this competition like discussed [here](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/90281#latest-537346) or [here](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/93148#latest-537413) it's reasonable not to trust the public LB.",
    "537546": "test it several times, and mainly by my own experience  about model building. \nBTW, a little trick, maybe you can give higher-score prediction more weight, but no too more.:)",
    "537548": "Thank you! You give me confidence from a surprise perspective.\nAnd I want to ask one more question? Is it the more varieties of model I  use, such as linear, tree and NN and so on. The more variance I can reduce?",
    "537551": "&gt; linear, tree and NN\n\nIf the models have a similar CV the combination should reduce the variance, IMHO.",
    "537666": "I don't understand. You said you have a NN model with 1.412 and then you got the same score after ensemble. How is that good for you?\nIf that's really the case, you will get a better score with the original model, since you will lose some points for overfitting.",
    "537708": "I am sorry. It's my fault and I've solved it. The score is 1.404",
    "539151": "This might help you with understanding how ensembles work: https://mlwave.com/kaggle-ensembling-guide/",
    "539671": "Maybe stacking could lead in this competition to overfitting, but blending should help to generalize better.",
    "540134": "As a general rule, the more variety there is in the models (actually their predictions) that you ensemble the better the results, as long as each model is sufficiently accurate, otherwise it won't contribute usefully.  To measure the difference between two sets of predictions for this contest I am using this R program under Linux OS, which reads a stream of ASCII records with 2 white-space-separated numeric fields (the predictions of each model) on each line, and calculates the mean absolute difference between them:\n\n  #!/usr/bin/env Rscript\n  Data &lt;- scan( file = \"stdin\", what = list( double( 0), double( 0)))\n  cat( mean( abs( Data[[ 1]] - Data[[ 2]])), \"\\n\")\n\nHope this helps.",
    "540162": "haha, and then identify it youself?"
  },
  "source": "meta"
}