Hi just started with the competitions and was looking at some advice on improving performance of training and experiments on collab . So far I have observed my training and inference suprisingly is faster on Kaggle compared to collab . I guess it might be due to custom steps we need to install these libraries on the collab container .
\nSo far this is what I have done :-
\nInstalling Cudf and rapids utils like below but still the training is slow compared to in Kaggle . Cuda version in kaggle though is 21.10.01 compared to on collab 21.12.02.On top of this the below installation is very very slow need to see if Nvidia folks here have a better alternative.
\n!git clone https://github.com/rapidsai/rapidsai-csp-utils.git\n!python rapidsai-csp-utils/colab/env-check.py\n\n!bash rapidsai-csp-utils/colab/update_gcc.sh\nimport os\nos._exit(00)\n\nimport condacolab\ncondacolab.install()\n\nimport condacolab\ncondacolab.check()\n\n!python rapidsai-csp-utils/colab/install_rapids.py stable\nimport os\nos.environ['NUMBAPRO_NVVM'] = '/usr/local/cuda/nvvm/lib64/libnvvm.so'\nos.environ['NUMBAPRO_LIBDEVICE'] = '/usr/local/cuda/nvvm/libdevice/'\nos.environ['CONDA_PREFIX'] = '/usr/local'\nDid below steps to recompile lgbm for GPU but seems as I think observed in kaggle somehow LGBM doesn't use GPU that efficiently like XGboost.
\n! git clone --recursive https://github.com/Microsoft/LightGBM\n! cd LightGBM && rm -rf build && mkdir build && cd build && cmake -DUSE_GPU=1 ../../LightGBM && make -j4 && cd ../python-package && python3 setup.py install --precompile --gpu;\nNot related to this but I have seen CATBOOST working pretty well and fast on GPUs on collab and making most of the available GPUS .
\nIf anyone has any tips please do let me know, I am definitely missing something 😊 Thanks .
", "messages": [ { "id": "1834454", "postDate": "06/27/2022 00:31:59", "content": "Hi just started with the competitions and was looking at some advice on improving performance of training and experiments on collab . So far I have observed my training and inference suprisingly is faster on Kaggle compared to collab . I guess it might be due to custom steps we need to install these libraries on the collab container .
\nSo far this is what I have done :-
\nInstalling Cudf and rapids utils like below but still the training is slow compared to in Kaggle . Cuda version in kaggle though is 21.10.01 compared to on collab 21.12.02.On top of this the below installation is very very slow need to see if Nvidia folks here have a better alternative.
\n!git clone https://github.com/rapidsai/rapidsai-csp-utils.git\n!python rapidsai-csp-utils/colab/env-check.py\n\n!bash rapidsai-csp-utils/colab/update_gcc.sh\nimport os\nos._exit(00)\n\nimport condacolab\ncondacolab.install()\n\nimport condacolab\ncondacolab.check()\n\n!python rapidsai-csp-utils/colab/install_rapids.py stable\nimport os\nos.environ['NUMBAPRO_NVVM'] = '/usr/local/cuda/nvvm/lib64/libnvvm.so'\nos.environ['NUMBAPRO_LIBDEVICE'] = '/usr/local/cuda/nvvm/libdevice/'\nos.environ['CONDA_PREFIX'] = '/usr/local'\nDid below steps to recompile lgbm for GPU but seems as I think observed in kaggle somehow LGBM doesn't use GPU that efficiently like XGboost.
\n! git clone --recursive https://github.com/Microsoft/LightGBM\n! cd LightGBM && rm -rf build && mkdir build && cd build && cmake -DUSE_GPU=1 ../../LightGBM && make -j4 && cd ../python-package && python3 setup.py install --precompile --gpu;\nNot related to this but I have seen CATBOOST working pretty well and fast on GPUs on collab and making most of the available GPUS .
\nIf anyone has any tips please do let me know, I am definitely missing something 😊 Thanks .
", "rawMarkdown": "Hi just started with the competitions and was looking at some advice on improving performance of training and experiments on collab . So far I have observed my training and inference suprisingly is faster on Kaggle compared to collab . I guess it might be due to custom steps we need to install these libraries on the collab container .\n\nSo far this is what I have done :-\n\n- **XGB with cudf cupy**\n\nInstalling Cudf and rapids utils like below but still the training is slow compared to in Kaggle . Cuda version in kaggle though is **21.10.01** compared to on collab **21.12.02**.On top of this the below installation is very very slow need to see if Nvidia folks here have a better alternative.\n\n```python\n!git clone https://github.com/rapidsai/rapidsai-csp-utils.git\n!python rapidsai-csp-utils/colab/env-check.py\n\n!bash rapidsai-csp-utils/colab/update_gcc.sh\nimport os\nos._exit(00)\n\nimport condacolab\ncondacolab.install()\n\nimport condacolab\ncondacolab.check()\n\n!python rapidsai-csp-utils/colab/install_rapids.py stable\nimport os\nos.environ['NUMBAPRO_NVVM'] = '/usr/local/cuda/nvvm/lib64/libnvvm.so'\nos.environ['NUMBAPRO_LIBDEVICE'] = '/usr/local/cuda/nvvm/libdevice/'\nos.environ['CONDA_PREFIX'] = '/usr/local'\n```\n- **LGBM**\n\nDid below steps to recompile lgbm for GPU but seems as I think observed in kaggle somehow LGBM doesn't use GPU that efficiently like XGboost.\n\n```python\n! git clone --recursive https://github.com/Microsoft/LightGBM\n! cd LightGBM && rm -rf build && mkdir build && cd build && cmake -DUSE_GPU=1 ../../LightGBM && make -j4 && cd ../python-package && python3 setup.py install --precompile --gpu;\n```\n- **CATBOOST**\n\nNot related to this but I have seen CATBOOST working pretty well and fast on GPUs on collab and making most of the available GPUS .\n\nIf anyone has any tips please do let me know, I am definitely missing something 😊 Thanks .", "votes": null }, { "id": "1835094", "postDate": "06/27/2022 13:11:34", "content": "Hi, @gauravbrills, thanks for sharing, it really takes some time to find out how to enable GPU tools on Colab
\nCouple of my personal hints there:
\n1) For LGBM with GPU you can use simple pip install right AFTER rapids.ai installation script (!python rapidsai-csp-utils/colab/install_rapids.py stable):
import os\n%pip install -U lightgbm --install-option=--gpu\nos._exit(00)\nThat will install LGBM with GPU support
\n2) To speed up RAPIDS.ai installation script you can pass only component to install, like cudf:
\n!python rapidsai-csp-utils/colab/install_rapids.py stable cudf
As by default RAPIDS.ai install all the components
", "rawMarkdown": "Hi, @gauravbrills, thanks for sharing, it really takes some time to find out how to enable GPU tools on Colab\n\nCouple of my personal hints there:\n1) For LGBM with GPU you can use simple pip install right AFTER rapids.ai installation script `(!python rapidsai-csp-utils/colab/install_rapids.py stable)`:\n```\nimport os\n%pip install -U lightgbm --install-option=--gpu\nos._exit(00)\n```\nThat will install LGBM **with GPU support**\n\n2) To speed up RAPIDS.ai installation script you can pass only component to install, like cudf:\n` !python rapidsai-csp-utils/colab/install_rapids.py stable cudf`\n\nAs by default RAPIDS.ai install all the components", "votes": null }, { "id": "1835136", "postDate": "06/27/2022 13:59:27", "content": "Thanks @romaupgini these make sense will try and revert . But from prior experience with LGBM have observed it doesnt use GPU that efficiently and still need to check how it rolls with cudf
", "rawMarkdown": "Thanks @romaupgini these make sense will try and revert . But from prior experience with LGBM have observed it doesnt use GPU that efficiently and still need to check how it rolls with cudf", "votes": null }, { "id": "1836016", "postDate": "06/28/2022 09:31:05", "content": "As far a I know LGBM doesnt support cudf directly (in contrast to XGB), and after several trials I found out that the most memory efficient way to pass cudf to LGBM Dataset would be following:
\ndtrain = lgb.Dataset(train.loc[train_idx, FEATURES].fillna(NAN_VALUE).as_gpu_matrix(),\n train.loc[train_idx, 'target'].values.get())
\nAs as_gpu_matrix() doesn't create copy of dataset. BTW you'll get warning on that and proposal to use to_cupy() what creates A COPY. Seems like a lot of inefficiency under the hood in cudf/cupy.
yup I am doing the same right now but not much performance gains so sticking to CPU .
", "rawMarkdown": "yup I am doing the same right now but not much performance gains so sticking to CPU .", "votes": null }, { "id": "1836593", "postDate": "06/28/2022 20:27:33", "content": "Hi, @gauravbrills This approach somehow stopped working on Colab for me.
\nStarting with missing module 'pynvml' and ending in install_rapids.py completely failing due to Conda failed with initial frozen solve.
Did you encounter any problems or can you share some insights on your current Colab setup?
", "rawMarkdown": "Hi, @gauravbrills This approach somehow stopped working on Colab for me. \nStarting with missing module 'pynvml' and ending in install_rapids.py completely failing due to Conda failed with initial frozen solve.\n\nDid you encounter any problems or can you share some insights on your current Colab setup?", "votes": null }, { "id": "1836743", "postDate": "06/29/2022 03:02:22", "content": "Which setup specifically . I was able to run XGB fine with the above code , You might need to run the above snippet in separate cells as they do restart in between the COllab container .
", "rawMarkdown": "Which setup specifically . I was able to run XGB fine with the above code , You might need to run the above snippet in separate cells as they do restart in between the COllab container .", "votes": null }, { "id": "1836915", "postDate": "06/29/2022 08:02:35", "content": "@matthiasanderer hope this helps https://colab.research.google.com/drive/1kS61kCHnbZOw5dO82Um0W2FO7UWOvfpJ?usp=sharing
", "rawMarkdown": "matthiasanderer hope this helps https://colab.research.google.com/drive/1kS61kCHnbZOw5dO82Um0W2FO7UWOvfpJ?usp=sharing", "votes": null } ], "comments": [ { "id": 1835094, "author_name": "romaupgini", "author_url": "", "post_date": "06/27/2022 13:11:34", "content": "Hi, @gauravbrills, thanks for sharing, it really takes some time to find out how to enable GPU tools on Colab
\nCouple of my personal hints there:
\n1) For LGBM with GPU you can use simple pip install right AFTER rapids.ai installation script (!python rapidsai-csp-utils/colab/install_rapids.py stable):
import os\n%pip install -U lightgbm --install-option=--gpu\nos._exit(00)\nThat will install LGBM with GPU support
\n2) To speed up RAPIDS.ai installation script you can pass only component to install, like cudf:
\n!python rapidsai-csp-utils/colab/install_rapids.py stable cudf
As by default RAPIDS.ai install all the components
", "votes": null, "replies": [ { "id": 1835136, "author_name": "gauravbrills", "author_url": "", "post_date": "06/27/2022 13:59:27", "content": "Thanks @romaupgini these make sense will try and revert . But from prior experience with LGBM have observed it doesnt use GPU that efficiently and still need to check how it rolls with cudf
", "votes": null, "replies": [] }, { "id": 1836016, "author_name": "romaupgini", "author_url": "", "post_date": "06/28/2022 09:31:05", "content": "As far a I know LGBM doesnt support cudf directly (in contrast to XGB), and after several trials I found out that the most memory efficient way to pass cudf to LGBM Dataset would be following:
\ndtrain = lgb.Dataset(train.loc[train_idx, FEATURES].fillna(NAN_VALUE).as_gpu_matrix(),\n train.loc[train_idx, 'target'].values.get())
\nAs as_gpu_matrix() doesn't create copy of dataset. BTW you'll get warning on that and proposal to use to_cupy() what creates A COPY. Seems like a lot of inefficiency under the hood in cudf/cupy.
yup I am doing the same right now but not much performance gains so sticking to CPU .
", "votes": null, "replies": [] } ] }, { "id": 1836593, "author_name": "matthiasanderer", "author_url": "", "post_date": "06/28/2022 20:27:33", "content": "Hi, @gauravbrills This approach somehow stopped working on Colab for me.
\nStarting with missing module 'pynvml' and ending in install_rapids.py completely failing due to Conda failed with initial frozen solve.
Did you encounter any problems or can you share some insights on your current Colab setup?
", "votes": null, "replies": [ { "id": 1836743, "author_name": "gauravbrills", "author_url": "", "post_date": "06/29/2022 03:02:22", "content": "Which setup specifically . I was able to run XGB fine with the above code , You might need to run the above snippet in separate cells as they do restart in between the COllab container .
", "votes": null, "replies": [] }, { "id": 1836915, "author_name": "romaupgini", "author_url": "", "post_date": "06/29/2022 08:02:35", "content": "@matthiasanderer hope this helps https://colab.research.google.com/drive/1kS61kCHnbZOw5dO82Um0W2FO7UWOvfpJ?usp=sharing
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1834454": "Hi just started with the competitions and was looking at some advice on improving performance of training and experiments on collab . So far I have observed my training and inference suprisingly is faster on Kaggle compared to collab . I guess it might be due to custom steps we need to install these libraries on the collab container .\n\nSo far this is what I have done :-\n\n- **XGB with cudf cupy**\n\nInstalling Cudf and rapids utils like below but still the training is slow compared to in Kaggle . Cuda version in kaggle though is **21.10.01** compared to on collab **21.12.02**.On top of this the below installation is very very slow need to see if Nvidia folks here have a better alternative.\n\n```python\n!git clone https://github.com/rapidsai/rapidsai-csp-utils.git\n!python rapidsai-csp-utils/colab/env-check.py\n\n!bash rapidsai-csp-utils/colab/update_gcc.sh\nimport os\nos._exit(00)\n\nimport condacolab\ncondacolab.install()\n\nimport condacolab\ncondacolab.check()\n\n!python rapidsai-csp-utils/colab/install_rapids.py stable\nimport os\nos.environ['NUMBAPRO_NVVM'] = '/usr/local/cuda/nvvm/lib64/libnvvm.so'\nos.environ['NUMBAPRO_LIBDEVICE'] = '/usr/local/cuda/nvvm/libdevice/'\nos.environ['CONDA_PREFIX'] = '/usr/local'\n```\n- **LGBM**\n\nDid below steps to recompile lgbm for GPU but seems as I think observed in kaggle somehow LGBM doesn't use GPU that efficiently like XGboost.\n\n```python\n! git clone --recursive https://github.com/Microsoft/LightGBM\n! cd LightGBM && rm -rf build && mkdir build && cd build && cmake -DUSE_GPU=1 ../../LightGBM && make -j4 && cd ../python-package && python3 setup.py install --precompile --gpu;\n```\n- **CATBOOST**\n\nNot related to this but I have seen CATBOOST working pretty well and fast on GPUs on collab and making most of the available GPUS .\n\nIf anyone has any tips please do let me know, I am definitely missing something 😊 Thanks .", "1835094": "Hi, @gauravbrills, thanks for sharing, it really takes some time to find out how to enable GPU tools on Colab\n\nCouple of my personal hints there:\n1) For LGBM with GPU you can use simple pip install right AFTER rapids.ai installation script `(!python rapidsai-csp-utils/colab/install_rapids.py stable)`:\n```\nimport os\n%pip install -U lightgbm --install-option=--gpu\nos._exit(00)\n```\nThat will install LGBM **with GPU support**\n\n2) To speed up RAPIDS.ai installation script you can pass only component to install, like cudf:\n` !python rapidsai-csp-utils/colab/install_rapids.py stable cudf`\n\nAs by default RAPIDS.ai install all the components", "1835136": "Thanks @romaupgini these make sense will try and revert . But from prior experience with LGBM have observed it doesnt use GPU that efficiently and still need to check how it rolls with cudf", "1836016": "As far a I know LGBM doesnt support cudf directly (in contrast to XGB), and after several trials I found out that the most memory efficient way to pass cudf to LGBM Dataset would be following:\n`dtrain = lgb.Dataset(train.loc[train_idx, FEATURES].fillna(NAN_VALUE).as_gpu_matrix(),\n train.loc[train_idx, 'target'].values.get())`\nAs as_gpu_matrix() doesn't create copy of dataset. BTW you'll get warning on that and proposal to use to_cupy() what creates A COPY. Seems like a lot of inefficiency under the hood in cudf/cupy.", "1836323": "yup I am doing the same right now but not much performance gains so sticking to CPU .", "1836593": "Hi, @gauravbrills This approach somehow stopped working on Colab for me. \nStarting with missing module 'pynvml' and ending in install_rapids.py completely failing due to Conda failed with initial frozen solve.\n\nDid you encounter any problems or can you share some insights on your current Colab setup?", "1836743": "Which setup specifically . I was able to run XGB fine with the above code , You might need to run the above snippet in separate cells as they do restart in between the COllab container .", "1836915": "matthiasanderer hope this helps https://colab.research.google.com/drive/1kS61kCHnbZOw5dO82Um0W2FO7UWOvfpJ?usp=sharing" }, "source": "meta" }