This competition provides large train and test data.
\nIn this situation, pandas library is very slow to process because of large data.
\nCompared to pandas, cudf library uses GPUs and incredibly fast. So, I want to use cudf library when I create new features.
However, I'm using Google Colab in this competition and I have trouble installing cudf in Google Colab.
\nI searched for the way on the website and tried some approaches. But none of these can work.
\nAre there anyone who has some tips installing cudf library in Google Colab.
\nI'm glad if someone shares the method.
Thank you.
", "messages": [ { "id": "1870186", "postDate": "07/25/2022 11:01:36", "content": "This competition provides large train and test data.
\nIn this situation, pandas library is very slow to process because of large data.
\nCompared to pandas, cudf library uses GPUs and incredibly fast. So, I want to use cudf library when I create new features.
However, I'm using Google Colab in this competition and I have trouble installing cudf in Google Colab.
\nI searched for the way on the website and tried some approaches. But none of these can work.
\nAre there anyone who has some tips installing cudf library in Google Colab.
\nI'm glad if someone shares the method.
Thank you.
", "rawMarkdown": "This competition provides large train and test data.\n\nIn this situation, pandas library is very slow to process because of large data.\nCompared to pandas, cudf library uses GPUs and incredibly fast. So, I want to use cudf library when I create new features.\n\nHowever, I'm using Google Colab in this competition and I have trouble installing cudf in Google Colab.\n\nI searched for the way on the website and tried some approaches. But none of these can work.\n\nAre there anyone who has some tips installing cudf library in Google Colab.\nI'm glad if someone shares the method.\n\nThank you.", "votes": null }, { "id": "1870306", "postDate": "07/25/2022 13:11:55", "content": "Did you try this one:
\n\nI did try this one(https://colab.research.google.com/github/ritchieng/deep-learning-wizard/blob/master/docs/machine_learning/gpu/rapids_cudf.ipynb). The challenge I faced was getting a T4 Machine. So eventually dropped GPU based approach for this competition.
", "rawMarkdown": "Did you try this one:\n\nhttps://colab.research.google.com/drive/1rY7Ln6rEE1pOlfSHCYOVaqt8OvDO35J0#forceEdit=true&offline=true&sandboxMode=true\n\nI did try this one(https://colab.research.google.com/github/ritchieng/deep-learning-wizard/blob/master/docs/machine_learning/gpu/rapids_cudf.ipynb). The challenge I faced was getting a T4 Machine. So eventually dropped GPU based approach for this competition.", "votes": null }, { "id": "1874038", "postDate": "07/28/2022 03:16:12", "content": "Thank you for reply.
\nI tried the link you sent me. However, It took too long times to install RAPIDS in google colab.
\nI think it is better to create features by using cudf in kaggle notebook and train various model in colab using these features.
", "rawMarkdown": "Thank you for reply.\n\nI tried the link you sent me. However, It took too long times to install RAPIDS in google colab.\n\nI think it is better to create features by using cudf in kaggle notebook and train various model in colab using these features.", "votes": null }, { "id": "1874945", "postDate": "07/28/2022 16:04:52", "content": "I tried to go down this route. An issue with Colab is it's hard to set up a 'persistent' environment. You can do things like mount your Google drive and install your packages and set up environments, but with RAPIDS it's a bit trickier to get that all sorted plus, as you saw, it takes a fricking long time to install. So I bailed on that approach. But if somebody has encountered a better way to do, it I'd be interested. Colab sure is nice when you need GPU/TPU, but things like this limit its utility.
", "rawMarkdown": "I tried to go down this route. An issue with Colab is it's hard to set up a 'persistent' environment. You can do things like mount your Google drive and install your packages and set up environments, but with RAPIDS it's a bit trickier to get that all sorted plus, as you saw, it takes a fricking long time to install. So I bailed on that approach. But if somebody has encountered a better way to do, it I'd be interested. Colab sure is nice when you need GPU/TPU, but things like this limit its utility.", "votes": null }, { "id": "1884535", "postDate": "08/04/2022 13:40:27", "content": "Hi guys, not sure if you were able to make this work, but doing what @manojprabhaakr suggested, i managed to make it work.
\n\nIt takes around 15 minutes to set up, after you do it once, you can reset the runtime as many times as you want, the env changes will persist (as long as you dont go for the \"disconnect and delete runtime\" reset)
\nFor me this is way less painful than setting up a local env, thank you so much @manojprabhaakr !
", "rawMarkdown": "Hi guys, not sure if you were able to make this work, but doing what @manojprabhaakr suggested, i managed to make it work.\n\nhttps://colab.research.google.com/drive/1rY7Ln6rEE1pOlfSHCYOVaqt8OvDO35J0#forceEdit=true&offline=true&sandboxMode=true\n\nIt takes around 15 minutes to set up, after you do it once, you can reset the runtime as many times as you want, the env changes will persist *(as long as you dont go for the \"disconnect and delete runtime\" reset)* \n\nFor me this is way less painful than setting up a local env, thank you so much @manojprabhaakr !", "votes": null } ], "comments": [ { "id": 1870306, "author_name": "manojprabhaakr", "author_url": "", "post_date": "07/25/2022 13:11:55", "content": "Did you try this one:
\n\nI did try this one(https://colab.research.google.com/github/ritchieng/deep-learning-wizard/blob/master/docs/machine_learning/gpu/rapids_cudf.ipynb). The challenge I faced was getting a T4 Machine. So eventually dropped GPU based approach for this competition.
", "votes": null, "replies": [] }, { "id": 1874038, "author_name": "yoshoo", "author_url": "", "post_date": "07/28/2022 03:16:12", "content": "Thank you for reply.
\nI tried the link you sent me. However, It took too long times to install RAPIDS in google colab.
\nI think it is better to create features by using cudf in kaggle notebook and train various model in colab using these features.
", "votes": null, "replies": [] }, { "id": 1874945, "author_name": "scharlesworth", "author_url": "", "post_date": "07/28/2022 16:04:52", "content": "I tried to go down this route. An issue with Colab is it's hard to set up a 'persistent' environment. You can do things like mount your Google drive and install your packages and set up environments, but with RAPIDS it's a bit trickier to get that all sorted plus, as you saw, it takes a fricking long time to install. So I bailed on that approach. But if somebody has encountered a better way to do, it I'd be interested. Colab sure is nice when you need GPU/TPU, but things like this limit its utility.
", "votes": null, "replies": [] }, { "id": 1884535, "author_name": "victorasso", "author_url": "", "post_date": "08/04/2022 13:40:27", "content": "Hi guys, not sure if you were able to make this work, but doing what @manojprabhaakr suggested, i managed to make it work.
\n\nIt takes around 15 minutes to set up, after you do it once, you can reset the runtime as many times as you want, the env changes will persist (as long as you dont go for the \"disconnect and delete runtime\" reset)
\nFor me this is way less painful than setting up a local env, thank you so much @manojprabhaakr !
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1870186": "This competition provides large train and test data.\n\nIn this situation, pandas library is very slow to process because of large data.\nCompared to pandas, cudf library uses GPUs and incredibly fast. So, I want to use cudf library when I create new features.\n\nHowever, I'm using Google Colab in this competition and I have trouble installing cudf in Google Colab.\n\nI searched for the way on the website and tried some approaches. But none of these can work.\n\nAre there anyone who has some tips installing cudf library in Google Colab.\nI'm glad if someone shares the method.\n\nThank you.", "1870306": "Did you try this one:\n\nhttps://colab.research.google.com/drive/1rY7Ln6rEE1pOlfSHCYOVaqt8OvDO35J0#forceEdit=true&offline=true&sandboxMode=true\n\nI did try this one(https://colab.research.google.com/github/ritchieng/deep-learning-wizard/blob/master/docs/machine_learning/gpu/rapids_cudf.ipynb). The challenge I faced was getting a T4 Machine. So eventually dropped GPU based approach for this competition.", "1874038": "Thank you for reply.\n\nI tried the link you sent me. However, It took too long times to install RAPIDS in google colab.\n\nI think it is better to create features by using cudf in kaggle notebook and train various model in colab using these features.", "1874945": "I tried to go down this route. An issue with Colab is it's hard to set up a 'persistent' environment. You can do things like mount your Google drive and install your packages and set up environments, but with RAPIDS it's a bit trickier to get that all sorted plus, as you saw, it takes a fricking long time to install. So I bailed on that approach. But if somebody has encountered a better way to do, it I'd be interested. Colab sure is nice when you need GPU/TPU, but things like this limit its utility.", "1884535": "Hi guys, not sure if you were able to make this work, but doing what @manojprabhaakr suggested, i managed to make it work.\n\nhttps://colab.research.google.com/drive/1rY7Ln6rEE1pOlfSHCYOVaqt8OvDO35J0#forceEdit=true&offline=true&sandboxMode=true\n\nIt takes around 15 minutes to set up, after you do it once, you can reset the runtime as many times as you want, the env changes will persist *(as long as you dont go for the \"disconnect and delete runtime\" reset)* \n\nFor me this is way less painful than setting up a local env, thank you so much @manojprabhaakr !" }, "source": "meta" }