Hello all,
\nI presume we are all collectively using polars for feature engineering and data loads herewith. I am sure we are also facing a few problems with data loads as the data is quite huge here.
\nRecently, polars integrated with NVidia RAPIDS framework to improve the speed of data wrangling using GPUs without explicit code changes as well. References for this are as below-
\nOne simply needs to install polars with GPU as below-
\n!pip install --extra-index-url=https://pypi.nvidia.com polars[gpu]==1.9.0 -q -d /kaggle/working/polars
While collecting from lazy-frame to eager-frames, one needs to specify the engine as GPU as below-
\ndf.collect(engine = \"gpu\")
This has the potential to speed up your features tremendously! Are you going to try this in your pipelines?
\nUpdated wheel file is present here-
\nhttps://www.kaggle.com/code/ravi20076/janestreet2024-imports-v1
Usage of FE with GPU is here-
\nhttps://www.kaggle.com/code/ravi20076/janestreet2024-baseline-train-v1
Best regards!
", "messages": [ { "id": 3018595, "postDate": "2024-10-15T21:50:56.103Z", "content": "Hello all,
\nI presume we are all collectively using polars for feature engineering and data loads herewith. I am sure we are also facing a few problems with data loads as the data is quite huge here.
\nRecently, polars integrated with NVidia RAPIDS framework to improve the speed of data wrangling using GPUs without explicit code changes as well. References for this are as below-
\nOne simply needs to install polars with GPU as below-
\n!pip install --extra-index-url=https://pypi.nvidia.com polars[gpu]==1.9.0 -q -d /kaggle/working/polars
While collecting from lazy-frame to eager-frames, one needs to specify the engine as GPU as below-
\ndf.collect(engine = \"gpu\")
This has the potential to speed up your features tremendously! Are you going to try this in your pipelines?
\nUpdated wheel file is present here-
\nhttps://www.kaggle.com/code/ravi20076/janestreet2024-imports-v1
Usage of FE with GPU is here-
\nhttps://www.kaggle.com/code/ravi20076/janestreet2024-baseline-train-v1
Best regards!
", "rawMarkdown": "Hello all,\n\nI presume we are all collectively using polars for feature engineering and data loads herewith. I am sure we are also facing a few problems with data loads as the data is quite huge here. \n\nRecently, polars integrated with NVidia RAPIDS framework to improve the speed of data wrangling using GPUs without explicit code changes as well. References for this are as below-\n- https://pola.rs/posts/polars-on-gpu/\n- https://pola.rs/posts/gpu-engine-release/\n- https://developer.nvidia.com/blog/polars-gpu-engine-powered-by-rapids-cudf-now-available-in-open-beta/\n- https://www.datacamp.com/blog/polars-gpu-engine\n- https://www.youtube.com/watch?v=SFdKFhOjVLQ\n- https://rapids.ai/polars-gpu-engine/\n\nOne simply needs to install polars with GPU as below- \n`!pip install --extra-index-url=https://pypi.nvidia.com polars[gpu]==1.9.0 -q -d /kaggle/working/polars`\n\nWhile collecting from lazy-frame to eager-frames, one needs to specify the engine as GPU as below-\n`df.collect(engine = \"gpu\")`\n\nThis has the potential to speed up your features tremendously! Are you going to try this in your pipelines?\nUpdated wheel file is present here-\nhttps://www.kaggle.com/code/ravi20076/janestreet2024-imports-v1\n\nUsage of FE with GPU is here-\nhttps://www.kaggle.com/code/ravi20076/janestreet2024-baseline-train-v1\n\nBest regards!", "votes": 5 }, { "id": 3020014, "postDate": "2024-10-17T05:24:20.240Z", "content": "Is this comp viable to those without GPUs? It seems like volume of data is quite a constraint. On an M2 Macbook here.
", "rawMarkdown": "Is this comp viable to those without GPUs? It seems like volume of data is quite a constraint. On an M2 Macbook here.", "votes": 1, "replies": [ { "id": 3020293, "postDate": "2024-10-17T11:18:32.010Z", "content": "@jcc310 you can avoid GPU usage as well, but you will have to be creative with your code!
", "rawMarkdown": "@jcc310 you can avoid GPU usage as well, but you will have to be creative with your code!" } ] }, { "id": 3018626, "postDate": "2024-10-15T23:19:52.827Z", "content": "Hey Ravi, I tried to use polars for \"used car prices regression competition\" as well. But i faced numerous problems to achieve same data engineering using pandas and numpy. Do you have any other resource focusing on polars and pandas syntax difference. (apart from documentation - Which is very clear and provides good comparison to pandas).
", "rawMarkdown": "Hey Ravi, I tried to use polars for \"used car prices regression competition\" as well. But i faced numerous problems to achieve same data engineering using pandas and numpy. Do you have any other resource focusing on polars and pandas syntax difference. (apart from documentation - Which is very clear and provides good comparison to pandas).", "votes": 2, "replies": [ { "id": 3018989, "postDate": "2024-10-16T07:31:00.127Z", "rawMarkdown": "", "votes": -1, "isDeleted": true, "replies": [ { "id": 3019381, "postDate": "2024-10-16T14:26:32.063Z", "content": "Polars has very good user documentation
\nhttps://docs.pola.rs/
I would advise you to forget about thinking in a \"pandas\" way, and just treat it as a completely different way to structure data. I think it's too conceptually different to be able to translate directly, and it may be better to just think of it in abstract terms of what you want to achieve with it. That's my personal approach, at least.
", "rawMarkdown": "I would advise you to forget about thinking in a \"pandas\" way, and just treat it as a completely different way to structure data. I think it's too conceptually different to be able to translate directly, and it may be better to just think of it in abstract terms of what you want to achieve with it. That's my personal approach, at least.", "votes": 1 } ] } ], "comments": [ { "id": 3020014, "author_name": "JamesC", "author_url": "", "post_date": "2024-10-17T05:24:20.240000", "content": "Is this comp viable to those without GPUs? It seems like volume of data is quite a constraint. On an M2 Macbook here.
", "votes": 1, "replies": [ { "id": 3020293, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2024-10-17T11:18:32.010000", "content": "@jcc310 you can avoid GPU usage as well, but you will have to be creative with your code!
", "votes": 0, "replies": [] } ] }, { "id": 3018626, "author_name": "Jay Shrivastava", "author_url": "", "post_date": "2024-10-15T23:19:52.827000", "content": "Hey Ravi, I tried to use polars for \"used car prices regression competition\" as well. But i faced numerous problems to achieve same data engineering using pandas and numpy. Do you have any other resource focusing on polars and pandas syntax difference. (apart from documentation - Which is very clear and provides good comparison to pandas).
", "votes": 2, "replies": [ { "id": 3018989, "author_name": "", "author_url": "", "post_date": "2024-10-16T07:31:00.127000", "content": "", "votes": -1, "replies": [ { "id": 3019381, "author_name": "Simon Veitner", "author_url": "", "post_date": "2024-10-16T14:26:32.063000", "content": "Polars has very good user documentation
\nhttps://docs.pola.rs/
I would advise you to forget about thinking in a \"pandas\" way, and just treat it as a completely different way to structure data. I think it's too conceptually different to be able to translate directly, and it may be better to just think of it in abstract terms of what you want to achieve with it. That's my personal approach, at least.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "3018595": "Hello all,\n\nI presume we are all collectively using polars for feature engineering and data loads herewith. I am sure we are also facing a few problems with data loads as the data is quite huge here. \n\nRecently, polars integrated with NVidia RAPIDS framework to improve the speed of data wrangling using GPUs without explicit code changes as well. References for this are as below-\n- https://pola.rs/posts/polars-on-gpu/\n- https://pola.rs/posts/gpu-engine-release/\n- https://developer.nvidia.com/blog/polars-gpu-engine-powered-by-rapids-cudf-now-available-in-open-beta/\n- https://www.datacamp.com/blog/polars-gpu-engine\n- https://www.youtube.com/watch?v=SFdKFhOjVLQ\n- https://rapids.ai/polars-gpu-engine/\n\nOne simply needs to install polars with GPU as below- \n`!pip install --extra-index-url=https://pypi.nvidia.com polars[gpu]==1.9.0 -q -d /kaggle/working/polars`\n\nWhile collecting from lazy-frame to eager-frames, one needs to specify the engine as GPU as below-\n`df.collect(engine = \"gpu\")`\n\nThis has the potential to speed up your features tremendously! Are you going to try this in your pipelines?\nUpdated wheel file is present here-\nhttps://www.kaggle.com/code/ravi20076/janestreet2024-imports-v1\n\nUsage of FE with GPU is here-\nhttps://www.kaggle.com/code/ravi20076/janestreet2024-baseline-train-v1\n\nBest regards!", "3020014": "Is this comp viable to those without GPUs? It seems like volume of data is quite a constraint. On an M2 Macbook here.", "3018626": "Hey Ravi, I tried to use polars for \"used car prices regression competition\" as well. But i faced numerous problems to achieve same data engineering using pandas and numpy. Do you have any other resource focusing on polars and pandas syntax difference. (apart from documentation - Which is very clear and provides good comparison to pandas)." } }