{
  "id": 347460,
  "title": "Engineering of additional features or management of missing data does not produce improvements to the score 0.799",
  "url": "/competitions/amex-default-prediction/discussion/347460",
  "author_name": "",
  "post_date": "2022-08-24T08:11:03.158430800Z",
  "votes": null,
  "comment_count": 8,
  "views": 0,
  "content": "<p>I have been trying for days with all possible algorithms but the score is fixed (0.799). It occurs to me that getting 0.80 is just luck.</p>",
  "messages": [
    {
      "id": "1911653",
      "postDate": "08/24/2022 08:11:03",
      "content": "<p>I have been trying for days with all possible algorithms but the score is fixed (0.799). It occurs to me that getting 0.80 is just luck.</p>",
      "rawMarkdown": "I have been trying for days with all possible algorithms but the score is fixed (0.799). It occurs to me that getting 0.80 is just luck.",
      "votes": null
    },
    {
      "id": "1911668",
      "postDate": "08/24/2022 08:22:45",
      "content": "<p>Try ensemble your models</p>\n<p>Think about the competition metric more and how you can optimize your score</p>\n<p>There is a difference between customer statements in train and test, can you do something with this?</p>\n<p>These should help break 0.799 </p>",
      "rawMarkdown": "Try ensemble your models\n\nThink about the competition metric more and how you can optimize your score\n\nThere is a difference between customer statements in train and test, can you do something with this?\n\nThese should help break 0.799",
      "votes": null
    },
    {
      "id": "1911682",
      "postDate": "08/24/2022 08:39:13",
      "content": "<p>do you mean the average of predictions obtained from different models?</p>",
      "rawMarkdown": "do you mean the average of predictions obtained from different models?",
      "votes": null
    },
    {
      "id": "1911758",
      "postDate": "08/24/2022 09:40:48",
      "content": "<p>Since you have tried different algorithms, assuming you created all the models yourself, get the OOF predictions of each model into a new dataframe, so your new columns are like:</p>\n<blockquote>\n  <p>customer_ID, prediction_lightgbm1, prediction_lightgbm2, prediction_xgb, prediction_cnn, prediction_mlp, target</p>\n</blockquote>\n<p>Then train a logistic regression model with your 'prediction_' as features against your target, then use that logistic regression model on your test dataset to get your new predictions.</p>\n<p>This is usually a better ensemble than just average of all.</p>",
      "rawMarkdown": "Since you have tried different algorithms, assuming you created all the models yourself, get the OOF predictions of each model into a new dataframe, so your new columns are like:\n\n> customer_ID, prediction_lightgbm1, prediction_lightgbm2, prediction_xgb, prediction_cnn, prediction_mlp, target\n\nThen train a logistic regression model with your 'prediction_' as features against your target, then use that logistic regression model on your test dataset to get your new predictions.\n\nThis is usually a better ensemble than just average of all.",
      "votes": null
    },
    {
      "id": "1911959",
      "postDate": "08/24/2022 12:11:49",
      "content": "<p>probabilistic predictions or in binary form?</p>",
      "rawMarkdown": "probabilistic predictions or in binary form?",
      "votes": null
    },
    {
      "id": "1911961",
      "postDate": "08/24/2022 12:12:32",
      "content": "<p>predict_proba </p>",
      "rawMarkdown": "predict_proba",
      "votes": null
    },
    {
      "id": "1912076",
      "postDate": "08/24/2022 13:44:04",
      "content": "<p>is the method you just pointed it out called stacking? am I correct? my question is how do we decide this new output is good or not? In other words, how do we make sure it is not overfitting?<br>\nOne way I used is to produce oof prediction for the last model (which is the logistic regression in your case) and calculate overall cv score. It turns out the cv score of this oof is not better than weighted average. So I have no idea to select which one is better. Can you provide some advices? Thank you so much. </p>",
      "rawMarkdown": "is the method you just pointed it out called stacking? am I correct? my question is how do we decide this new output is good or not? In other words, how do we make sure it is not overfitting?\nOne way I used is to produce oof prediction for the last model (which is the logistic regression in your case) and calculate overall cv score. It turns out the cv score of this oof is not better than weighted average. So I have no idea to select which one is better. Can you provide some advices? Thank you so much.",
      "votes": null
    },
    {
      "id": "1912132",
      "postDate": "08/24/2022 14:28:10",
      "content": "<blockquote>\n  <p>is the method you just pointed it out called stacking?</p>\n</blockquote>\n<p>Yes it is called stacking</p>\n<blockquote>\n  <p>how do we make sure it is not overfitting?</p>\n</blockquote>\n<p>You can also create the predictions within a CV, the overfitting reduction should be done prior to this in your individual models IMO.</p>\n<blockquote>\n  <p>It turns out the cv score of this oof is not better than weighted average.</p>\n</blockquote>\n<p>Your models are too correlated with each other then probably, stacking usually works best with varying model predictions</p>",
      "rawMarkdown": ">is the method you just pointed it out called stacking?\n\nYes it is called stacking\n\n>how do we make sure it is not overfitting?\n\nYou can also create the predictions within a CV, the overfitting reduction should be done prior to this in your individual models IMO.\n\n> It turns out the cv score of this oof is not better than weighted average.\n\nYour models are too correlated with each other then probably, stacking usually works best with varying model predictions",
      "votes": null
    },
    {
      "id": "1912159",
      "postDate": "08/24/2022 14:46:35",
      "content": "<p>Thanks for the helpful advice!</p>",
      "rawMarkdown": "Thanks for the helpful advice!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1911668,
      "author_name": "julianmukaj",
      "author_url": "",
      "post_date": "08/24/2022 08:22:45",
      "content": "<p>Try ensemble your models</p>\n<p>Think about the competition metric more and how you can optimize your score</p>\n<p>There is a difference between customer statements in train and test, can you do something with this?</p>\n<p>These should help break 0.799 </p>",
      "votes": null,
      "replies": [
        {
          "id": 1911682,
          "author_name": "raimondomelis",
          "author_url": "",
          "post_date": "08/24/2022 08:39:13",
          "content": "<p>do you mean the average of predictions obtained from different models?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1911758,
          "author_name": "julianmukaj",
          "author_url": "",
          "post_date": "08/24/2022 09:40:48",
          "content": "<p>Since you have tried different algorithms, assuming you created all the models yourself, get the OOF predictions of each model into a new dataframe, so your new columns are like:</p>\n<blockquote>\n  <p>customer_ID, prediction_lightgbm1, prediction_lightgbm2, prediction_xgb, prediction_cnn, prediction_mlp, target</p>\n</blockquote>\n<p>Then train a logistic regression model with your 'prediction_' as features against your target, then use that logistic regression model on your test dataset to get your new predictions.</p>\n<p>This is usually a better ensemble than just average of all.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1911959,
          "author_name": "raimondomelis",
          "author_url": "",
          "post_date": "08/24/2022 12:11:49",
          "content": "<p>probabilistic predictions or in binary form?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1911961,
          "author_name": "julianmukaj",
          "author_url": "",
          "post_date": "08/24/2022 12:12:32",
          "content": "<p>predict_proba </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1912076,
          "author_name": "dzbsun",
          "author_url": "",
          "post_date": "08/24/2022 13:44:04",
          "content": "<p>is the method you just pointed it out called stacking? am I correct? my question is how do we decide this new output is good or not? In other words, how do we make sure it is not overfitting?<br>\nOne way I used is to produce oof prediction for the last model (which is the logistic regression in your case) and calculate overall cv score. It turns out the cv score of this oof is not better than weighted average. So I have no idea to select which one is better. Can you provide some advices? Thank you so much. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1912132,
          "author_name": "julianmukaj",
          "author_url": "",
          "post_date": "08/24/2022 14:28:10",
          "content": "<blockquote>\n  <p>is the method you just pointed it out called stacking?</p>\n</blockquote>\n<p>Yes it is called stacking</p>\n<blockquote>\n  <p>how do we make sure it is not overfitting?</p>\n</blockquote>\n<p>You can also create the predictions within a CV, the overfitting reduction should be done prior to this in your individual models IMO.</p>\n<blockquote>\n  <p>It turns out the cv score of this oof is not better than weighted average.</p>\n</blockquote>\n<p>Your models are too correlated with each other then probably, stacking usually works best with varying model predictions</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1912159,
          "author_name": "dzbsun",
          "author_url": "",
          "post_date": "08/24/2022 14:46:35",
          "content": "<p>Thanks for the helpful advice!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1911653": "I have been trying for days with all possible algorithms but the score is fixed (0.799). It occurs to me that getting 0.80 is just luck.",
    "1911668": "Try ensemble your models\n\nThink about the competition metric more and how you can optimize your score\n\nThere is a difference between customer statements in train and test, can you do something with this?\n\nThese should help break 0.799",
    "1911682": "do you mean the average of predictions obtained from different models?",
    "1911758": "Since you have tried different algorithms, assuming you created all the models yourself, get the OOF predictions of each model into a new dataframe, so your new columns are like:\n\n> customer_ID, prediction_lightgbm1, prediction_lightgbm2, prediction_xgb, prediction_cnn, prediction_mlp, target\n\nThen train a logistic regression model with your 'prediction_' as features against your target, then use that logistic regression model on your test dataset to get your new predictions.\n\nThis is usually a better ensemble than just average of all.",
    "1911959": "probabilistic predictions or in binary form?",
    "1911961": "predict_proba",
    "1912076": "is the method you just pointed it out called stacking? am I correct? my question is how do we decide this new output is good or not? In other words, how do we make sure it is not overfitting?\nOne way I used is to produce oof prediction for the last model (which is the logistic regression in your case) and calculate overall cv score. It turns out the cv score of this oof is not better than weighted average. So I have no idea to select which one is better. Can you provide some advices? Thank you so much.",
    "1912132": ">is the method you just pointed it out called stacking?\n\nYes it is called stacking\n\n>how do we make sure it is not overfitting?\n\nYou can also create the predictions within a CV, the overfitting reduction should be done prior to this in your individual models IMO.\n\n> It turns out the cv score of this oof is not better than weighted average.\n\nYour models are too correlated with each other then probably, stacking usually works best with varying model predictions",
    "1912159": "Thanks for the helpful advice!"
  },
  "source": "meta"
}