Hi guys,
\nI tried fitting both LightGBM and XGBoost to a processed dataset containing samples starting from date_id = 1190 and 84 features. The models were trained on Kaggle. While training LightGBM was successful, XGBoost was failed due to memory issues. I believe this is because LightGBM uses less RAM compared to XGBoost during training (please correct me if I’m wrong).
I’d like to ask: what is the shape of your dataset when training XGBoost? Were you able to train the model successfully on Kaggle, or did you use your PC? Given that the Kaggle notebook provides 32GB of RAM, which is significantly more than my PC’s 8GB, I’m curious about your experience. If someone has trained XGBoost on a larger dataset using a Kaggle notebook, I suspect the issue might be related to the efficiency of my code.
\nIf you’re not training on Kaggle, would you mind to share the specs of your machine?
\nCheers!
", "messages": [ { "id": "3086068", "postDate": "01/02/2025 00:34:27", "content": "Hi guys,
\nI tried fitting both LightGBM and XGBoost to a processed dataset containing samples starting from date_id = 1190 and 84 features. The models were trained on Kaggle. While training LightGBM was successful, XGBoost was failed due to memory issues. I believe this is because LightGBM uses less RAM compared to XGBoost during training (please correct me if I’m wrong).
I’d like to ask: what is the shape of your dataset when training XGBoost? Were you able to train the model successfully on Kaggle, or did you use your PC? Given that the Kaggle notebook provides 32GB of RAM, which is significantly more than my PC’s 8GB, I’m curious about your experience. If someone has trained XGBoost on a larger dataset using a Kaggle notebook, I suspect the issue might be related to the efficiency of my code.
\nIf you’re not training on Kaggle, would you mind to share the specs of your machine?
\nCheers!
", "rawMarkdown": "Hi guys,\n\nI tried fitting both LightGBM and XGBoost to a processed dataset containing samples starting from `date_id = 1190` and 84 features. The models were trained on Kaggle. While training LightGBM was successful, XGBoost was failed due to memory issues. I believe this is because LightGBM uses less RAM compared to XGBoost during training (please correct me if I’m wrong).\n\nI’d like to ask: what is the shape of your dataset when training XGBoost? Were you able to train the model successfully on Kaggle, or did you use your PC? Given that the Kaggle notebook provides 32GB of RAM, which is significantly more than my PC’s 8GB, I’m curious about your experience. If someone has trained XGBoost on a larger dataset using a Kaggle notebook, I suspect the issue might be related to the efficiency of my code.\n\nIf you’re not training on Kaggle, would you mind to share the specs of your machine?\n\nCheers!", "votes": null }, { "id": "3086075", "postDate": "01/02/2025 00:49:45", "content": "You are correct. LightGBM indeed consumes less RAM during the preprocessing stage. However, based on my personal experience, if you conduct training over a period of approximately 500 days, it should be feasible to execute on Kaggle. This can be accomplished when you make use of the.fit() approach and avoid using xgb.DMatrix() for the preprocessing phase, as the latter is truly a process that consumes a significant amount of RAM.
\nYou can perform an experiment by referring to the following notebook:
\nhttps://www.kaggle.com/code/yuanzhezhou/jane-street-baseline-lgb-xgb-and-catboost
I think it's better to use Colab if you can, you'll have way more RAM and you'll get to run your experiments more efficiently.
", "rawMarkdown": "I think it's better to use Colab if you can, you'll have way more RAM and you'll get to run your experiments more efficiently.", "votes": null }, { "id": "3086138", "postDate": "01/02/2025 03:39:53", "content": "Thank you for your suggestion! I’ll try this approach later. It’s interesting to learn that the XGBoost native API consumes more RAM compared to the scikit-learn API, as I thought the native API was designed to be more memory-efficient.
", "rawMarkdown": "Thank you for your suggestion! I’ll try this approach later. It’s interesting to learn that the XGBoost native API consumes more RAM compared to the scikit-learn API, as I thought the native API was designed to be more memory-efficient.", "votes": null }, { "id": "3089824", "postDate": "01/06/2025 15:16:56", "content": "you can trried this ?
\nTo answer this question effectively, you should address the following points:
\nMemory Usage Differences Between LightGBM and XGBoost:
\nExplain why LightGBM typically uses less memory than XGBoost during training.
\nHighlight the differences in their algorithms (e.g., histogram-based approach in LightGBM vs. pre-sorted algorithm in XGBoost).
Dataset Size and Shape:
\nShare your experience with the dataset size and shape when training XGBoost.
\nMention if you’ve encountered similar memory issues and how you resolved them.
Tips for Reducing Memory Usage in XGBoost:
\nProvide practical advice for optimizing memory usage when training XGBoost, such as:
\nUsing sparse matrices.
\nAdjusting parameters like max_bin, max_depth, and subsample.
\nEnabling external memory (if applicable).
Kaggle vs. Local Machine:
\nShare your experience training XGBoost on Kaggle vs. a local machine.
\nIf you’ve trained on Kaggle, confirm whether 32GB of RAM was sufficient for your dataset.
Code Efficiency:
\nSuggest checking for inefficiencies in the code, such as redundant data copies or improper data types.
\nHere’s an example response:
\nHi there,
\nYou’re correct that LightGBM generally uses less memory than XGBoost during training. This is because LightGBM employs a histogram-based algorithm, which is more memory-efficient compared to XGBoost’s pre-sorted algorithm. XGBoost tends to require more memory, especially for large datasets or high-dimensional data.
\nDataset Size and Shape
\nI’ve trained XGBoost on datasets with millions of rows and dozens of features on Kaggle, and while it worked, I had to optimize memory usage carefully. For reference, my dataset had around 1 million rows and 100 features, and I was able to train it successfully on Kaggle’s 32GB RAM environment.
\nTips for Reducing Memory Usage in XGBoost
\nHere are some suggestions to help with memory issues:
Use Sparse Matrices: If your dataset contains many zeros, convert it to a sparse matrix format before feeding it into XGBoost. This can significantly reduce memory usage.
\nfrom scipy.sparse import csr_matrix
\ndtrain = xgb.DMatrix(csr_matrix(X_train), label=y_train)
Adjust Parameters:
\nmax_depth: Reduce the maximum depth of trees to limit memory usage.
\nmax_bin: Lower the number of bins for feature discretization.
\nsubsample and colsample_bytree: Use subsampling to train on a fraction of the data and features.
params = {
\n 'max_depth': 6,
\n 'max_bin': 256,
\n 'subsample': 0.8,
\n 'colsample_bytree': 0.8,
\n 'tree_method': 'hist' # Use histogram-based algorithm for better memory efficiency
\n}
Enable External Memory: If your dataset is too large, you can use XGBoost’s external memory feature to train in chunks. This is particularly useful for very large datasets.
\ndtrain = xgb.DMatrix('train.svm#dtrain.cache')
Check Data Types: Ensure your dataset uses the most memory-efficient data types. For example, use float32 instead of float64 for numerical features.
\nKaggle vs. Local Machine
\nI’ve found Kaggle’s 32GB RAM environment sufficient for most datasets, but if your dataset is very large, you may still run into issues. On my local machine with 16GB of RAM, I’ve had to downsample the data or use aggressive parameter tuning to make it work.
\nCode Efficiency
\nLastly, double-check your code for inefficiencies. For example:
Avoid creating unnecessary copies of the dataset.
\nEnsure you’re not loading the entire dataset into memory multiple times.
If you’re still running into issues, feel free to share more details about your dataset size and the parameters you’re using. I’d be happy to help troubleshoot further!
\nCheers!
I've solved the problem, but still thanks for your suggestions. You're right—I did generate a redundant copy of the data when I used dtrain = xgb.DMatrix(X_train, y_train), which I hadn’t realised before posting. I resolved this by simply replacing X_train, and the problem is now fixed. As I add a bit more features to the dataset, I’ve found that around 500 days of data are the maximum amount Kaggle notebooks can handle (of course, this might just be because my code isn't efficient enough).
", "rawMarkdown": "I've solved the problem, but still thanks for your suggestions. You're right—I did generate a redundant copy of the data when I used dtrain = xgb.DMatrix(X_train, y_train), which I hadn’t realised before posting. I resolved this by simply replacing X_train, and the problem is now fixed. As I add a bit more features to the dataset, I’ve found that around 500 days of data are the maximum amount Kaggle notebooks can handle (of course, this might just be because my code isn't efficient enough).", "votes": null } ], "comments": [ { "id": 3086075, "author_name": "carrotwait", "author_url": "", "post_date": "01/02/2025 00:49:45", "content": "You are correct. LightGBM indeed consumes less RAM during the preprocessing stage. However, based on my personal experience, if you conduct training over a period of approximately 500 days, it should be feasible to execute on Kaggle. This can be accomplished when you make use of the.fit() approach and avoid using xgb.DMatrix() for the preprocessing phase, as the latter is truly a process that consumes a significant amount of RAM.
\nYou can perform an experiment by referring to the following notebook:
\nhttps://www.kaggle.com/code/yuanzhezhou/jane-street-baseline-lgb-xgb-and-catboost
Thank you for your suggestion! I’ll try this approach later. It’s interesting to learn that the XGBoost native API consumes more RAM compared to the scikit-learn API, as I thought the native API was designed to be more memory-efficient.
", "votes": null, "replies": [] } ] }, { "id": 3086078, "author_name": "sritichaimae", "author_url": "", "post_date": "01/02/2025 00:53:05", "content": "I think it's better to use Colab if you can, you'll have way more RAM and you'll get to run your experiments more efficiently.
", "votes": null, "replies": [] }, { "id": 3089824, "author_name": "angadye", "author_url": "", "post_date": "01/06/2025 15:16:56", "content": "you can trried this ?
\nTo answer this question effectively, you should address the following points:
\nMemory Usage Differences Between LightGBM and XGBoost:
\nExplain why LightGBM typically uses less memory than XGBoost during training.
\nHighlight the differences in their algorithms (e.g., histogram-based approach in LightGBM vs. pre-sorted algorithm in XGBoost).
Dataset Size and Shape:
\nShare your experience with the dataset size and shape when training XGBoost.
\nMention if you’ve encountered similar memory issues and how you resolved them.
Tips for Reducing Memory Usage in XGBoost:
\nProvide practical advice for optimizing memory usage when training XGBoost, such as:
\nUsing sparse matrices.
\nAdjusting parameters like max_bin, max_depth, and subsample.
\nEnabling external memory (if applicable).
Kaggle vs. Local Machine:
\nShare your experience training XGBoost on Kaggle vs. a local machine.
\nIf you’ve trained on Kaggle, confirm whether 32GB of RAM was sufficient for your dataset.
Code Efficiency:
\nSuggest checking for inefficiencies in the code, such as redundant data copies or improper data types.
\nHere’s an example response:
\nHi there,
\nYou’re correct that LightGBM generally uses less memory than XGBoost during training. This is because LightGBM employs a histogram-based algorithm, which is more memory-efficient compared to XGBoost’s pre-sorted algorithm. XGBoost tends to require more memory, especially for large datasets or high-dimensional data.
\nDataset Size and Shape
\nI’ve trained XGBoost on datasets with millions of rows and dozens of features on Kaggle, and while it worked, I had to optimize memory usage carefully. For reference, my dataset had around 1 million rows and 100 features, and I was able to train it successfully on Kaggle’s 32GB RAM environment.
\nTips for Reducing Memory Usage in XGBoost
\nHere are some suggestions to help with memory issues:
Use Sparse Matrices: If your dataset contains many zeros, convert it to a sparse matrix format before feeding it into XGBoost. This can significantly reduce memory usage.
\nfrom scipy.sparse import csr_matrix
\ndtrain = xgb.DMatrix(csr_matrix(X_train), label=y_train)
Adjust Parameters:
\nmax_depth: Reduce the maximum depth of trees to limit memory usage.
\nmax_bin: Lower the number of bins for feature discretization.
\nsubsample and colsample_bytree: Use subsampling to train on a fraction of the data and features.
params = {
\n 'max_depth': 6,
\n 'max_bin': 256,
\n 'subsample': 0.8,
\n 'colsample_bytree': 0.8,
\n 'tree_method': 'hist' # Use histogram-based algorithm for better memory efficiency
\n}
Enable External Memory: If your dataset is too large, you can use XGBoost’s external memory feature to train in chunks. This is particularly useful for very large datasets.
\ndtrain = xgb.DMatrix('train.svm#dtrain.cache')
Check Data Types: Ensure your dataset uses the most memory-efficient data types. For example, use float32 instead of float64 for numerical features.
\nKaggle vs. Local Machine
\nI’ve found Kaggle’s 32GB RAM environment sufficient for most datasets, but if your dataset is very large, you may still run into issues. On my local machine with 16GB of RAM, I’ve had to downsample the data or use aggressive parameter tuning to make it work.
\nCode Efficiency
\nLastly, double-check your code for inefficiencies. For example:
Avoid creating unnecessary copies of the dataset.
\nEnsure you’re not loading the entire dataset into memory multiple times.
If you’re still running into issues, feel free to share more details about your dataset size and the parameters you’re using. I’d be happy to help troubleshoot further!
\nCheers!
I've solved the problem, but still thanks for your suggestions. You're right—I did generate a redundant copy of the data when I used dtrain = xgb.DMatrix(X_train, y_train), which I hadn’t realised before posting. I resolved this by simply replacing X_train, and the problem is now fixed. As I add a bit more features to the dataset, I’ve found that around 500 days of data are the maximum amount Kaggle notebooks can handle (of course, this might just be because my code isn't efficient enough).
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3086068": "Hi guys,\n\nI tried fitting both LightGBM and XGBoost to a processed dataset containing samples starting from `date_id = 1190` and 84 features. The models were trained on Kaggle. While training LightGBM was successful, XGBoost was failed due to memory issues. I believe this is because LightGBM uses less RAM compared to XGBoost during training (please correct me if I’m wrong).\n\nI’d like to ask: what is the shape of your dataset when training XGBoost? Were you able to train the model successfully on Kaggle, or did you use your PC? Given that the Kaggle notebook provides 32GB of RAM, which is significantly more than my PC’s 8GB, I’m curious about your experience. If someone has trained XGBoost on a larger dataset using a Kaggle notebook, I suspect the issue might be related to the efficiency of my code.\n\nIf you’re not training on Kaggle, would you mind to share the specs of your machine?\n\nCheers!", "3086075": "You are correct. LightGBM indeed consumes less RAM during the preprocessing stage. However, based on my personal experience, if you conduct training over a period of approximately 500 days, it should be feasible to execute on Kaggle. This can be accomplished when you make use of the.fit() approach and avoid using xgb.DMatrix() for the preprocessing phase, as the latter is truly a process that consumes a significant amount of RAM. \nYou can perform an experiment by referring to the following notebook: \nhttps://www.kaggle.com/code/yuanzhezhou/jane-street-baseline-lgb-xgb-and-catboost", "3086078": "I think it's better to use Colab if you can, you'll have way more RAM and you'll get to run your experiments more efficiently.", "3086138": "Thank you for your suggestion! I’ll try this approach later. It’s interesting to learn that the XGBoost native API consumes more RAM compared to the scikit-learn API, as I thought the native API was designed to be more memory-efficient.", "3089824": "you can trried this ?\n\nTo answer this question effectively, you should address the following points:\n\n\nMemory Usage Differences Between LightGBM and XGBoost:\n\nExplain why LightGBM typically uses less memory than XGBoost during training.\nHighlight the differences in their algorithms (e.g., histogram-based approach in LightGBM vs. pre-sorted algorithm in XGBoost).\n\n\n\nDataset Size and Shape:\n\nShare your experience with the dataset size and shape when training XGBoost.\nMention if you’ve encountered similar memory issues and how you resolved them.\n\n\n\nTips for Reducing Memory Usage in XGBoost:\n\nProvide practical advice for optimizing memory usage when training XGBoost, such as:\n\nUsing sparse matrices.\nAdjusting parameters like max_bin, max_depth, and subsample.\nEnabling external memory (if applicable).\n\n\n\n\n\nKaggle vs. Local Machine:\n\nShare your experience training XGBoost on Kaggle vs. a local machine.\nIf you’ve trained on Kaggle, confirm whether 32GB of RAM was sufficient for your dataset.\n\n\n\nCode Efficiency:\n\nSuggest checking for inefficiencies in the code, such as redundant data copies or improper data types.\n\n\n\nHere’s an example response:\n\nHi there,\nYou’re correct that LightGBM generally uses less memory than XGBoost during training. This is because LightGBM employs a histogram-based algorithm, which is more memory-efficient compared to XGBoost’s pre-sorted algorithm. XGBoost tends to require more memory, especially for large datasets or high-dimensional data.\nDataset Size and Shape\nI’ve trained XGBoost on datasets with millions of rows and dozens of features on Kaggle, and while it worked, I had to optimize memory usage carefully. For reference, my dataset had around 1 million rows and 100 features, and I was able to train it successfully on Kaggle’s 32GB RAM environment.\nTips for Reducing Memory Usage in XGBoost\nHere are some suggestions to help with memory issues:\n\n\nUse Sparse Matrices: If your dataset contains many zeros, convert it to a sparse matrix format before feeding it into XGBoost. This can significantly reduce memory usage.\nfrom scipy.sparse import csr_matrix\ndtrain = xgb.DMatrix(csr_matrix(X_train), label=y_train)\n\n\n\nAdjust Parameters:\n\nmax_depth: Reduce the maximum depth of trees to limit memory usage.\nmax_bin: Lower the number of bins for feature discretization.\nsubsample and colsample_bytree: Use subsampling to train on a fraction of the data and features.\n\nparams = {\n 'max_depth': 6,\n 'max_bin': 256,\n 'subsample': 0.8,\n 'colsample_bytree': 0.8,\n 'tree_method': 'hist' # Use histogram-based algorithm for better memory efficiency\n}\n\n\n\nEnable External Memory: If your dataset is too large, you can use XGBoost’s external memory feature to train in chunks. This is particularly useful for very large datasets.\ndtrain = xgb.DMatrix('train.svm#dtrain.cache')\n\n\n\nCheck Data Types: Ensure your dataset uses the most memory-efficient data types. For example, use float32 instead of float64 for numerical features.\n\n\nKaggle vs. Local Machine\nI’ve found Kaggle’s 32GB RAM environment sufficient for most datasets, but if your dataset is very large, you may still run into issues. On my local machine with 16GB of RAM, I’ve had to downsample the data or use aggressive parameter tuning to make it work.\nCode Efficiency\nLastly, double-check your code for inefficiencies. For example:\n\nAvoid creating unnecessary copies of the dataset.\nEnsure you’re not loading the entire dataset into memory multiple times.\n\nIf you’re still running into issues, feel free to share more details about your dataset size and the parameters you’re using. I’d be happy to help troubleshoot further!\nCheers!", "3089891": "I've solved the problem, but still thanks for your suggestions. You're right—I did generate a redundant copy of the data when I used dtrain = xgb.DMatrix(X_train, y_train), which I hadn’t realised before posting. I resolved this by simply replacing X_train, and the problem is now fixed. As I add a bit more features to the dataset, I’ve found that around 500 days of data are the maximum amount Kaggle notebooks can handle (of course, this might just be because my code isn't efficient enough)." }, "source": "meta" }