{
  "id": 347863,
  "title": "72nd place solution(ensemble of LightGBM and Sequential NN)",
  "url": "/competitions/amex-default-prediction/writeups/taichicchi-72nd-place-solution-ensemble-of-lightgb",
  "author_name": "",
  "post_date": "2023-01-15T09:31:52.460Z",
  "votes": 20,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Dear Kagglers.</p>\n<p>Thank you to the competition organizers hosting this interesting competition.<br>\nThank you to everyone involved in this competition. We learned a lot from public notebooks and discussions.</p>\n<h1>Our Final Result</h1>\n<ul>\n<li>Our submission<ul>\n<li>Local CV：0.80011</li>\n<li>Public: 0.80040</li>\n<li>Private: 0.80780</li></ul></li>\n<li>Result<ul>\n<li>Public: 61st → Private 72nd</li></ul></li>\n</ul>\n<h1>Summary of Our Solution</h1>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1452109%2F8a13e3e0ee80dcd0e41dbb726d7461b2%2Famex-solution.drawio.png?generation=1661445323488649&amp;alt=media\" alt=\"\"></p>\n<h1>Single Models</h1>\n<h2>LightGBM(DART)</h2>\n<p>Based on the public notebook <a href=\"https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977\" target=\"_blank\">here</a>, we trained lightgbm(dart) models with several feature patterns and achieved a public score of <strong>0.799</strong>.</p>\n<ul>\n<li>feature engineering patterns<ul>\n<li>basic aggregation per customer (mean, std, max, min, first, last, count, nunique)</li>\n<li>combinations of aggregated features (diff as last - mean, fraction as last / mean, etc.)</li>\n<li>last difference features(aggregation with diff(1).iloc[-1], diff(2).iloc[-1], …)</li></ul></li>\n</ul>\n<p>we made about 1,000~3,000 features for each model.</p>\n<h2>GRU or Transformer Encoder Model</h2>\n<p>Based on the public notebook <a href=\"https://www.kaggle.com/code/cdeotte/tensorflow-gru-starter-0-790\" target=\"_blank\">here</a>, we trained NN models and achieved a public score of <strong>0.792</strong>.</p>\n<ul>\n<li>Improvements from the public notebook<ul>\n<li>one hot encoding of each category features</li>\n<li>adding NA indication columns of features that have NA</li>\n<li>filling NA by linear interpolation</li>\n<li>multiple layers of GRU or TransformerEncoder(4~8 layers)</li></ul></li>\n</ul>\n<h1>Ensemble</h1>\n<p>NN models by themselves had only a low score on the Public Leaderboard, but we noticed that the public score improved from 0.799 to 0.800 by ensembling the LightGBM and NN models.<br>\nWe then experimented with ensemble patterns of multiple models and found that Stacking by LogisticRegression or MLP yielded particularly high scores.</p>\n<h2>finally</h2>\n<p>Advice is always welcome!<br>\nThank you for your attention.</p>",
  "messages": [
    {
      "id": "1914032",
      "postDate": "08/25/2022 17:08:00",
      "content": "<p>Dear Kagglers.</p>\n<p>Thank you to the competition organizers hosting this interesting competition.<br>\nThank you to everyone involved in this competition. We learned a lot from public notebooks and discussions.</p>\n<h1>Our Final Result</h1>\n<ul>\n<li>Our submission<ul>\n<li>Local CV：0.80011</li>\n<li>Public: 0.80040</li>\n<li>Private: 0.80780</li></ul></li>\n<li>Result<ul>\n<li>Public: 61st → Private 72nd</li></ul></li>\n</ul>\n<h1>Summary of Our Solution</h1>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1452109%2F8a13e3e0ee80dcd0e41dbb726d7461b2%2Famex-solution.drawio.png?generation=1661445323488649&amp;alt=media\" alt=\"\"></p>\n<h1>Single Models</h1>\n<h2>LightGBM(DART)</h2>\n<p>Based on the public notebook <a href=\"https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977\" target=\"_blank\">here</a>, we trained lightgbm(dart) models with several feature patterns and achieved a public score of <strong>0.799</strong>.</p>\n<ul>\n<li>feature engineering patterns<ul>\n<li>basic aggregation per customer (mean, std, max, min, first, last, count, nunique)</li>\n<li>combinations of aggregated features (diff as last - mean, fraction as last / mean, etc.)</li>\n<li>last difference features(aggregation with diff(1).iloc[-1], diff(2).iloc[-1], …)</li></ul></li>\n</ul>\n<p>we made about 1,000~3,000 features for each model.</p>\n<h2>GRU or Transformer Encoder Model</h2>\n<p>Based on the public notebook <a href=\"https://www.kaggle.com/code/cdeotte/tensorflow-gru-starter-0-790\" target=\"_blank\">here</a>, we trained NN models and achieved a public score of <strong>0.792</strong>.</p>\n<ul>\n<li>Improvements from the public notebook<ul>\n<li>one hot encoding of each category features</li>\n<li>adding NA indication columns of features that have NA</li>\n<li>filling NA by linear interpolation</li>\n<li>multiple layers of GRU or TransformerEncoder(4~8 layers)</li></ul></li>\n</ul>\n<h1>Ensemble</h1>\n<p>NN models by themselves had only a low score on the Public Leaderboard, but we noticed that the public score improved from 0.799 to 0.800 by ensembling the LightGBM and NN models.<br>\nWe then experimented with ensemble patterns of multiple models and found that Stacking by LogisticRegression or MLP yielded particularly high scores.</p>\n<h2>finally</h2>\n<p>Advice is always welcome!<br>\nThank you for your attention.</p>",
      "rawMarkdown": "Dear Kagglers.\n\nThank you to the competition organizers hosting this interesting competition.\nThank you to everyone involved in this competition. We learned a lot from public notebooks and discussions.\n\n# Our Final Result\n- Our submission\n  - Local CV：0.80011\n  - Public: 0.80040\n  - Private: 0.80780\n- Result\n  - Public: 61st → Private 72nd\n\n# Summary of Our Solution\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1452109%2F8a13e3e0ee80dcd0e41dbb726d7461b2%2Famex-solution.drawio.png?generation=1661445323488649&alt=media)\n\n# Single Models\n## LightGBM(DART)\nBased on the public notebook [here](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977), we trained lightgbm(dart) models with several feature patterns and achieved a public score of __0.799__.\n\n- feature engineering patterns\n  - basic aggregation per customer (mean, std, max, min, first, last, count, nunique)\n  - combinations of aggregated features (diff as last - mean, fraction as last / mean, etc.)\n  - last difference features(aggregation with diff(1).iloc[-1], diff(2).iloc[-1], ...)\n\nwe made about 1,000~3,000 features for each model.\n\n\n\n## GRU or Transformer Encoder Model\nBased on the public notebook [here](https://www.kaggle.com/code/cdeotte/tensorflow-gru-starter-0-790), we trained NN models and achieved a public score of __0.792__.\n\n- Improvements from the public notebook\n  - one hot encoding of each category features\n  - adding NA indication columns of features that have NA\n  - filling NA by linear interpolation\n  - multiple layers of GRU or TransformerEncoder(4~8 layers)\n\n\n# Ensemble\nNN models by themselves had only a low score on the Public Leaderboard, but we noticed that the public score improved from 0.799 to 0.800 by ensembling the LightGBM and NN models.\nWe then experimented with ensemble patterns of multiple models and found that Stacking by LogisticRegression or MLP yielded particularly high scores.\n\n## finally\nAdvice is always welcome!\nThank you for your attention.",
      "votes": null
    },
    {
      "id": "1914051",
      "postDate": "08/25/2022 17:22:15",
      "content": "<p>hi <a href=\"https://www.kaggle.com/hutch1221\" target=\"_blank\">@hutch1221</a>  congratulations, did you try bfill and ffill instead of interpolation? In my case that seemed to improve the score of GRUs.</p>",
      "rawMarkdown": "hi @hutch1221  congratulations, did you try bfill and ffill instead of interpolation? In my case that seemed to improve the score of GRUs.",
      "votes": null
    },
    {
      "id": "1914073",
      "postDate": "08/25/2022 17:47:39",
      "content": "<p>hi <a href=\"https://www.kaggle.com/chaudharypriyanshu\" target=\"_blank\">@chaudharypriyanshu</a> congratulations for your great result too, thank you! <br>\nI didn't try them… I tried only linear interpolation or fillna(0) as masking values.</p>",
      "rawMarkdown": "hi @chaudharypriyanshu congratulations for your great result too, thank you! \nI didn't try them... I tried only linear interpolation or fillna(0) as masking values.",
      "votes": null
    },
    {
      "id": "1922175",
      "postDate": "09/01/2022 10:14:27",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/hutch1221\" target=\"_blank\">@hutch1221</a> May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: <a href=\"https://cityuhk.questionpro.com/survey-of-kaggle-contestants\" target=\"_blank\">https://cityuhk.questionpro.com/survey-of-kaggle-contestants</a></p>",
      "rawMarkdown": "Hi @hutch1221 May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1914051,
      "author_name": "chaudharypriyanshu",
      "author_url": "",
      "post_date": "08/25/2022 17:22:15",
      "content": "<p>hi <a href=\"https://www.kaggle.com/hutch1221\" target=\"_blank\">@hutch1221</a>  congratulations, did you try bfill and ffill instead of interpolation? In my case that seemed to improve the score of GRUs.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1914073,
          "author_name": "hutch1221",
          "author_url": "",
          "post_date": "08/25/2022 17:47:39",
          "content": "<p>hi <a href=\"https://www.kaggle.com/chaudharypriyanshu\" target=\"_blank\">@chaudharypriyanshu</a> congratulations for your great result too, thank you! <br>\nI didn't try them… I tried only linear interpolation or fillna(0) as masking values.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1922175,
      "author_name": "lystriving",
      "author_url": "",
      "post_date": "09/01/2022 10:14:27",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/hutch1221\" target=\"_blank\">@hutch1221</a> May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: <a href=\"https://cityuhk.questionpro.com/survey-of-kaggle-contestants\" target=\"_blank\">https://cityuhk.questionpro.com/survey-of-kaggle-contestants</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1914032": "Dear Kagglers.\n\nThank you to the competition organizers hosting this interesting competition.\nThank you to everyone involved in this competition. We learned a lot from public notebooks and discussions.\n\n# Our Final Result\n- Our submission\n  - Local CV：0.80011\n  - Public: 0.80040\n  - Private: 0.80780\n- Result\n  - Public: 61st → Private 72nd\n\n# Summary of Our Solution\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1452109%2F8a13e3e0ee80dcd0e41dbb726d7461b2%2Famex-solution.drawio.png?generation=1661445323488649&alt=media)\n\n# Single Models\n## LightGBM(DART)\nBased on the public notebook [here](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977), we trained lightgbm(dart) models with several feature patterns and achieved a public score of __0.799__.\n\n- feature engineering patterns\n  - basic aggregation per customer (mean, std, max, min, first, last, count, nunique)\n  - combinations of aggregated features (diff as last - mean, fraction as last / mean, etc.)\n  - last difference features(aggregation with diff(1).iloc[-1], diff(2).iloc[-1], ...)\n\nwe made about 1,000~3,000 features for each model.\n\n\n\n## GRU or Transformer Encoder Model\nBased on the public notebook [here](https://www.kaggle.com/code/cdeotte/tensorflow-gru-starter-0-790), we trained NN models and achieved a public score of __0.792__.\n\n- Improvements from the public notebook\n  - one hot encoding of each category features\n  - adding NA indication columns of features that have NA\n  - filling NA by linear interpolation\n  - multiple layers of GRU or TransformerEncoder(4~8 layers)\n\n\n# Ensemble\nNN models by themselves had only a low score on the Public Leaderboard, but we noticed that the public score improved from 0.799 to 0.800 by ensembling the LightGBM and NN models.\nWe then experimented with ensemble patterns of multiple models and found that Stacking by LogisticRegression or MLP yielded particularly high scores.\n\n## finally\nAdvice is always welcome!\nThank you for your attention.",
    "1914051": "hi @hutch1221  congratulations, did you try bfill and ffill instead of interpolation? In my case that seemed to improve the score of GRUs.",
    "1914073": "hi @chaudharypriyanshu congratulations for your great result too, thank you! \nI didn't try them... I tried only linear interpolation or fillna(0) as masking values.",
    "1922175": "Hi @hutch1221 May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants"
  },
  "source": "meta"
}