{
  "id": 497163,
  "title": "LGBM Model Training Eats Up RAM",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/497163",
  "author_name": "",
  "post_date": "2024-04-23T21:48:15.142189300Z",
  "votes": 3,
  "comment_count": 3,
  "views": 0,
  "content": "<p>So I have a 4.5GB dataset and I have changed data types and everything to optimize the size of it as much as possible.</p>\n<p>It's a pretty small size, but when I try to fit a lgbm model (metric=\"auc\" only change from default) on 80% of the data it uses up all the RAM, that's <strong>more than 25GB</strong>. I really don't want to be super limited on my model options, but I have no idea what I can do. </p>\n<p>Does anyone know how to fix this?</p>",
  "messages": [
    {
      "id": "2770511",
      "postDate": "04/23/2024 21:48:15",
      "content": "<p>So I have a 4.5GB dataset and I have changed data types and everything to optimize the size of it as much as possible.</p>\n<p>It's a pretty small size, but when I try to fit a lgbm model (metric=\"auc\" only change from default) on 80% of the data it uses up all the RAM, that's <strong>more than 25GB</strong>. I really don't want to be super limited on my model options, but I have no idea what I can do. </p>\n<p>Does anyone know how to fix this?</p>",
      "rawMarkdown": "So I have a 4.5GB dataset and I have changed data types and everything to optimize the size of it as much as possible.\n\nIt's a pretty small size, but when I try to fit a lgbm model (metric=\"auc\" only change from default) on 80% of the data it uses up all the RAM, that's **more than 25GB**. I really don't want to be super limited on my model options, but I have no idea what I can do. \n\nDoes anyone know how to fix this?",
      "votes": null
    },
    {
      "id": "2775047",
      "postDate": "04/25/2024 13:03:11",
      "content": "<p>I have the same question with lgb and xgb</p>",
      "rawMarkdown": "I have the same question with lgb and xgb",
      "votes": null
    },
    {
      "id": "2775188",
      "postDate": "04/25/2024 14:24:48",
      "content": "<p>Take a look at this: <a href=\"https://github.com/microsoft/LightGBM/issues/562\" target=\"_blank\">https://github.com/microsoft/LightGBM/issues/562</a></p>\n<p>In general, lgbm parameters that highly impact memory are: max_bin, num_leaves, histogram_pool_size, bin_construct_sample_cnt. Usually, when running lgbm there's a memory peak at the beginning when creating the feature histograms, and then the memory drops. Try reducing max_bin, bin_construct_sample_cnt and histogram_pool_size</p>",
      "rawMarkdown": "Take a look at this: https://github.com/microsoft/LightGBM/issues/562\n\nIn general, lgbm parameters that highly impact memory are: max_bin, num_leaves, histogram_pool_size, bin_construct_sample_cnt. Usually, when running lgbm there's a memory peak at the beginning when creating the feature histograms, and then the memory drops. Try reducing max_bin, bin_construct_sample_cnt and histogram_pool_size",
      "votes": null
    },
    {
      "id": "2782945",
      "postDate": "04/29/2024 14:08:59",
      "content": "<p>Try to encode the categoricals and feed the model with float32 only.</p>",
      "rawMarkdown": "Try to encode the categoricals and feed the model with float32 only.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2775047,
      "author_name": "huangshibao",
      "author_url": "",
      "post_date": "04/25/2024 13:03:11",
      "content": "<p>I have the same question with lgb and xgb</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2775188,
      "author_name": "mavillan",
      "author_url": "",
      "post_date": "04/25/2024 14:24:48",
      "content": "<p>Take a look at this: <a href=\"https://github.com/microsoft/LightGBM/issues/562\" target=\"_blank\">https://github.com/microsoft/LightGBM/issues/562</a></p>\n<p>In general, lgbm parameters that highly impact memory are: max_bin, num_leaves, histogram_pool_size, bin_construct_sample_cnt. Usually, when running lgbm there's a memory peak at the beginning when creating the feature histograms, and then the memory drops. Try reducing max_bin, bin_construct_sample_cnt and histogram_pool_size</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2782945,
      "author_name": "eu1234",
      "author_url": "",
      "post_date": "04/29/2024 14:08:59",
      "content": "<p>Try to encode the categoricals and feed the model with float32 only.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2770511": "So I have a 4.5GB dataset and I have changed data types and everything to optimize the size of it as much as possible.\n\nIt's a pretty small size, but when I try to fit a lgbm model (metric=\"auc\" only change from default) on 80% of the data it uses up all the RAM, that's **more than 25GB**. I really don't want to be super limited on my model options, but I have no idea what I can do. \n\nDoes anyone know how to fix this?",
    "2775047": "I have the same question with lgb and xgb",
    "2775188": "Take a look at this: https://github.com/microsoft/LightGBM/issues/562\n\nIn general, lgbm parameters that highly impact memory are: max_bin, num_leaves, histogram_pool_size, bin_construct_sample_cnt. Usually, when running lgbm there's a memory peak at the beginning when creating the feature histograms, and then the memory drops. Try reducing max_bin, bin_construct_sample_cnt and histogram_pool_size",
    "2782945": "Try to encode the categoricals and feed the model with float32 only."
  },
  "source": "meta"
}