{ "id": 502182, "title": "What is the maximal number of features that you are able to use without running into the Out of Memory error for LGBM?", "url": "/competitions/home-credit-credit-risk-model-stability/discussion/502182", "author_name": "", "post_date": "2024-05-12T12:43:59.968764300Z", "votes": null, "comment_count": 9, "views": 0, "content": "

For me it is around 520 without much optimization, but would like to know some upper bounds from other people.

", "messages": [ { "id": "2808895", "postDate": "05/12/2024 12:43:59", "content": "

For me it is around 520 without much optimization, but would like to know some upper bounds from other people.

", "rawMarkdown": "For me it is around 520 without much optimization, but would like to know some upper bounds from other people.", "votes": null }, { "id": "2808933", "postDate": "05/12/2024 13:08:39", "content": "

I am uncertain whether it's the limit, but I can run without OOM errors a notebook with:

\n
\n

The shape of the train data is (1526659, 610)
\n Memory usage of train data is 2386.31 MB

\n
\n

Note that I train and infer LGB seperately.

", "rawMarkdown": "I am uncertain whether it's the limit, but I can run without OOM errors a notebook with:\n\n> The shape of the train data is (1526659, 610)\n> Memory usage of train data is 2386.31 MB\n\nNote that I train and infer LGB seperately.", "votes": null }, { "id": "2809013", "postDate": "05/12/2024 13:42:32", "content": "

about 700, https://www.kaggle.com/code/yunsuxiaozi/home-credit-lgbm-with-677-features-training

", "rawMarkdown": "about 700, https://www.kaggle.com/code/yunsuxiaozi/home-credit-lgbm-with-677-features-training", "votes": null }, { "id": "2809016", "postDate": "05/12/2024 13:43:28", "content": "

Hey guys, As a general guideline, if you're experiencing memory issues, you can try reducing the number of features by removing irrelevant or redundant features through feature selection techniques. Additionally, you can consider using LGBM's parameters such as max_bin and max_depth to control the memory usage during training. Technically, there isn't a hard maximum number of features that LightGBM can handle.

", "rawMarkdown": "Hey guys, As a general guideline, if you're experiencing memory issues, you can try reducing the number of features by removing irrelevant or redundant features through feature selection techniques. Additionally, you can consider using LGBM's parameters such as max_bin and max_depth to control the memory usage during training. Technically, there isn't a hard maximum number of features that LightGBM can handle.", "votes": null }, { "id": "2809048", "postDate": "05/12/2024 13:53:03", "content": "

More than 2000

", "rawMarkdown": "More than 2000", "votes": null }, { "id": "2809140", "postDate": "05/12/2024 14:49:48", "content": "

Thanks, will try calling gc.collect() proactively and see if I can include more features.

", "rawMarkdown": "Thanks, will try calling gc.collect() proactively and see if I can include more features.", "votes": null }, { "id": "2809153", "postDate": "05/12/2024 14:55:43", "content": "

Thanks. I also did the training and inference separately, but during inference also included the training dataset because the columns used during inference have to be same as the ones used during training for the model.

\n

But I realized that one can just save the columns to a .csv file and load that file directly during inference, to avoid loading the training dataset. Will try that and see how it goes.

", "rawMarkdown": "Thanks. I also did the training and inference separately, but during inference also included the training dataset because the columns used during inference have to be same as the ones used during training for the model. \n\nBut I realized that one can just save the columns to a .csv file and load that file directly during inference, to avoid loading the training dataset. Will try that and see how it goes.", "votes": null }, { "id": "2809644", "postDate": "05/12/2024 21:43:01", "content": "

Or save the list of the columns, or even better - process in a pipeline and then apply the pipeline to the test

", "rawMarkdown": "Or save the list of the columns, or even better - process in a pipeline and then apply the pipeline to the test", "votes": null }, { "id": "2810048", "postDate": "05/13/2024 05:46:03", "content": "

Additionally, the next approach can be used:

\n
columns = model.feature_name_\npred = model.predict_proba(df[columns])[:, ]\n
", "rawMarkdown": "Additionally, the next approach can be used:\n```python\ncolumns = model.feature_name_\npred = model.predict_proba(df[columns])[:, 1]\n```", "votes": null }, { "id": "2817311", "postDate": "05/16/2024 20:13:36", "content": "

@skrrydg, that's impressive. May I know how did you attempt to do the memory management during the inference stage in order to use this many features?

\n

I tried the below:

\n
    \n
  1. Actively call del var; gc.collect() after being done with using that var
    2. \n
  2. Predict in batches
    3. \n
  3. Separate the training and inference into two notebooks
    4. \n
  4. Since the purpose of including the train dataset during the inference is just to select the features used during the training stage, instead of loading the whole train dataset, I just save the train feature columns to a .csv file, and load that .csv file during inference
    5. \n
\n

With the above approaches the model can predict with 770+ features in the private dataset without any problems, but when the number of features increases to 1100+, the notebook encounters OOM error during the submission.

\n

Do you have any other suggestions to reduce the memory usage? Thanks a lot!

", "rawMarkdown": "skrrydg, that's impressive. May I know how did you attempt to do the memory management during the inference stage in order to use this many features?\n\nI tried the below: \n1. Actively call `del var; gc.collect()` after being done with using that var\n2. Predict in batches \n3. Separate the training and inference into two notebooks\n4. Since the purpose of including the train dataset during the inference is just to select the features used during the training stage, instead of loading the whole train dataset, I just save the train feature columns to a .csv file, and load that .csv file during inference\n\nWith the above approaches the model can predict with 770+ features in the private dataset without any problems, but when the number of features increases to 1100+, the notebook encounters OOM error during the submission. \n\nDo you have any other suggestions to reduce the memory usage? Thanks a lot!", "votes": null } ], "comments": [ { "id": 2808933, "author_name": "andreasbis", "author_url": "", "post_date": "05/12/2024 13:08:39", "content": "

I am uncertain whether it's the limit, but I can run without OOM errors a notebook with:

\n
\n

The shape of the train data is (1526659, 610)
\n Memory usage of train data is 2386.31 MB

\n
\n

Note that I train and infer LGB seperately.

", "votes": null, "replies": [ { "id": 2809153, "author_name": "faithk7u", "author_url": "", "post_date": "05/12/2024 14:55:43", "content": "

Thanks. I also did the training and inference separately, but during inference also included the training dataset because the columns used during inference have to be same as the ones used during training for the model.

\n

But I realized that one can just save the columns to a .csv file and load that file directly during inference, to avoid loading the training dataset. Will try that and see how it goes.

", "votes": null, "replies": [ { "id": 2809644, "author_name": "eu1234", "author_url": "", "post_date": "05/12/2024 21:43:01", "content": "

Or save the list of the columns, or even better - process in a pipeline and then apply the pipeline to the test

", "votes": null, "replies": [] }, { "id": 2810048, "author_name": "andreynesterov", "author_url": "", "post_date": "05/13/2024 05:46:03", "content": "

Additionally, the next approach can be used:

\n
columns = model.feature_name_\npred = model.predict_proba(df[columns])[:, ]\n
", "votes": null, "replies": [] } ] } ] }, { "id": 2809013, "author_name": "yunsuxiaozi", "author_url": "", "post_date": "05/12/2024 13:42:32", "content": "

about 700, https://www.kaggle.com/code/yunsuxiaozi/home-credit-lgbm-with-677-features-training

", "votes": null, "replies": [ { "id": 2809140, "author_name": "faithk7u", "author_url": "", "post_date": "05/12/2024 14:49:48", "content": "

Thanks, will try calling gc.collect() proactively and see if I can include more features.

", "votes": null, "replies": [] } ] }, { "id": 2809016, "author_name": "masroormohajerani", "author_url": "", "post_date": "05/12/2024 13:43:28", "content": "

Hey guys, As a general guideline, if you're experiencing memory issues, you can try reducing the number of features by removing irrelevant or redundant features through feature selection techniques. Additionally, you can consider using LGBM's parameters such as max_bin and max_depth to control the memory usage during training. Technically, there isn't a hard maximum number of features that LightGBM can handle.

", "votes": null, "replies": [] }, { "id": 2809048, "author_name": "skrrydg", "author_url": "", "post_date": "05/12/2024 13:53:03", "content": "

More than 2000

", "votes": null, "replies": [ { "id": 2817311, "author_name": "faithk7u", "author_url": "", "post_date": "05/16/2024 20:13:36", "content": "

@skrrydg, that's impressive. May I know how did you attempt to do the memory management during the inference stage in order to use this many features?

\n

I tried the below:

\n
    \n
  1. Actively call del var; gc.collect() after being done with using that var
    2. \n
  2. Predict in batches
    3. \n
  3. Separate the training and inference into two notebooks
    4. \n
  4. Since the purpose of including the train dataset during the inference is just to select the features used during the training stage, instead of loading the whole train dataset, I just save the train feature columns to a .csv file, and load that .csv file during inference
    5. \n
\n

With the above approaches the model can predict with 770+ features in the private dataset without any problems, but when the number of features increases to 1100+, the notebook encounters OOM error during the submission.

\n

Do you have any other suggestions to reduce the memory usage? Thanks a lot!

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2808895": "For me it is around 520 without much optimization, but would like to know some upper bounds from other people.", "2808933": "I am uncertain whether it's the limit, but I can run without OOM errors a notebook with:\n\n> The shape of the train data is (1526659, 610)\n> Memory usage of train data is 2386.31 MB\n\nNote that I train and infer LGB seperately.", "2809013": "about 700, https://www.kaggle.com/code/yunsuxiaozi/home-credit-lgbm-with-677-features-training", "2809016": "Hey guys, As a general guideline, if you're experiencing memory issues, you can try reducing the number of features by removing irrelevant or redundant features through feature selection techniques. Additionally, you can consider using LGBM's parameters such as max_bin and max_depth to control the memory usage during training. Technically, there isn't a hard maximum number of features that LightGBM can handle.", "2809048": "More than 2000", "2809140": "Thanks, will try calling gc.collect() proactively and see if I can include more features.", "2809153": "Thanks. I also did the training and inference separately, but during inference also included the training dataset because the columns used during inference have to be same as the ones used during training for the model. \n\nBut I realized that one can just save the columns to a .csv file and load that file directly during inference, to avoid loading the training dataset. Will try that and see how it goes.", "2809644": "Or save the list of the columns, or even better - process in a pipeline and then apply the pipeline to the test", "2810048": "Additionally, the next approach can be used:\n```python\ncolumns = model.feature_name_\npred = model.predict_proba(df[columns])[:, 1]\n```", "2817311": "skrrydg, that's impressive. May I know how did you attempt to do the memory management during the inference stage in order to use this many features?\n\nI tried the below: \n1. Actively call `del var; gc.collect()` after being done with using that var\n2. Predict in batches \n3. Separate the training and inference into two notebooks\n4. Since the purpose of including the train dataset during the inference is just to select the features used during the training stage, instead of loading the whole train dataset, I just save the train feature columns to a .csv file, and load that .csv file during inference\n\nWith the above approaches the model can predict with 770+ features in the private dataset without any problems, but when the number of features increases to 1100+, the notebook encounters OOM error during the submission. \n\nDo you have any other suggestions to reduce the memory usage? Thanks a lot!" }, "source": "meta" }