I had never heard of the polars library prior to this competition, so I got curious and found these benchmarks. Question is, why is pandas so widespread if it's abysmally slower than other libraries? Are we just too accustomed to the library to make the swap? Does pandas have any inherent capabilities that are superior to polars?
", "messages": [ { "id": "3031813", "postDate": "10/30/2024 06:42:56", "content": "I had never heard of the polars library prior to this competition, so I got curious and found these benchmarks. Question is, why is pandas so widespread if it's abysmally slower than other libraries? Are we just too accustomed to the library to make the swap? Does pandas have any inherent capabilities that are superior to polars?
", "rawMarkdown": "I had never heard of the polars library prior to this competition, so I got curious and found [these benchmarks](https://pola.rs/posts/benchmarks/). Question is, why is pandas so widespread if it's abysmally slower than other libraries? Are we just too accustomed to the library to make the swap? Does pandas have any inherent capabilities that are superior to polars?", "votes": null }, { "id": "3031866", "postDate": "10/30/2024 08:13:02", "content": "Benchmark gaming aside: pit single-threaded (pandas) vs multi-threaded (polars) on a 22-core machine.
\nThe sweet spot for both libraries is medium data. Wes McKinney: Medium Data (n): Not too big for a single machine, but too big to be dumb about.
\nPandas has been the standard in this space for a long time. I don't like lots about it, but It's pretty versatile, robust, reliable, and has the power of incumbency. Call it Stockholm syndrome, but I will not be switching over lots of existing code - which means maintaining a level of expertise in pandas.
\nBut for new projects? That gets interesting. Polars has an appeal in the \"medium space\", but I'm interested in scaling seamlessly from small to big data.
\nI'm leaning more towards Ibis. Ibis provides a consistent API across different backends (including DuckDB, spark and Polars). We're not locked into a single-machine execution engine and can scale from local to production.
", "rawMarkdown": "Benchmark gaming aside: pit single-threaded (pandas) vs multi-threaded (polars) on a 22-core machine. \n\nThe sweet spot for both libraries is medium data. **Wes McKinney**: Medium Data (n): *Not too big for a single machine, but too big to be dumb about.*\n\nPandas has been the standard in this space for a long time. I don't like lots about it, but It's pretty versatile, robust, reliable, and has the power of incumbency. Call it Stockholm syndrome, but I will not be switching over lots of existing code - which means maintaining a level of expertise in pandas.\n\nBut for new projects? That gets interesting. Polars has an appeal in the \"medium space\", but I'm interested in scaling seamlessly from small to big data. \n\nI'm leaning more towards [Ibis](https://ibis-project.org/why). Ibis provides a consistent API across different backends (including DuckDB, spark and Polars). We're not locked into a single-machine execution engine and can scale from local to production.", "votes": null }, { "id": "3031885", "postDate": "10/30/2024 08:47:09", "content": "Pandas has been in the game for so long that any ML library is compatible with its data structures (DataFrame, Series), so it's very comfortable to use it with other libraries. While today most of the libraries also work well with polars, there are still some that aren't. There are some preprocessing libraries and methods that only work with the pandas DataFrame. Another reason is legacy code - for older projects it may be hard and not worth it to refactor and change everything into polars. But I think the primary reason is the people are just afraid to learn new library. The syntax is a bit different to pandas and as you said, people are too accustomed to it to make the swap.
\nFor me, switching to polars was great. I find the syntax to be very comfortable (reminds me of SQL, which I really like), so I rarely use pandas these days. I use pandas only for older projects and when I need to be able to use libraries that can't work with polars.
", "rawMarkdown": "Pandas has been in the game for so long that any ML library is compatible with its data structures (DataFrame, Series), so it's very comfortable to use it with other libraries. While today most of the libraries also work well with polars, there are still some that aren't. There are some preprocessing libraries and methods that only work with the pandas DataFrame. Another reason is legacy code - for older projects it may be hard and not worth it to refactor and change everything into polars. But I think the primary reason is the people are just afraid to learn new library. The syntax is a bit different to pandas and as you said, people are too accustomed to it to make the swap.\n\nFor me, switching to polars was great. I find the syntax to be very comfortable (reminds me of SQL, which I really like), so I rarely use pandas these days. I use pandas only for older projects and when I need to be able to use libraries that can't work with polars.", "votes": null }, { "id": "3031901", "postDate": "10/30/2024 09:16:48", "content": "Polars is relatively new… it lacks some capabilities related to pandas and its support is not widespread in existing libraries. But its getting traction.
", "rawMarkdown": "Polars is relatively new... it lacks some capabilities related to pandas and its support is not widespread in existing libraries. But its getting traction.", "votes": null }, { "id": "3032619", "postDate": "10/31/2024 06:43:53", "content": "this is my first time using polars too, i will name a few things that makes me not switch to polars:
\nDear @paddykb
\nMany thanks for the heads-up regarding Ibis.
\nI did notice whilst perusing the documentation that
\n\n\n…matplotlib has not implemented the dataframe interchange protocol so it is recommended to call to_pandas() on the Ibis table before plotting. [Ref]
\n
so it could be a little clunky when it comes to performing extensive EDA.
\nAll the best,
\ncarl
Inplace assignments like this df.loc[validation_mask, 'prediction'] = df.loc[validation_mask, 'prediction'].clip(lower=-5, upper=5) are slower in polars and they return a copy of the original dataframe when you use df.with_columns so the memory usage spikes when execution reaches that point. I'm using polars for faster joins and feature engineering but then I cast it to pandas dataframe before my training loop.
Benchmark gaming aside: pit single-threaded (pandas) vs multi-threaded (polars) on a 22-core machine.
\nThe sweet spot for both libraries is medium data. Wes McKinney: Medium Data (n): Not too big for a single machine, but too big to be dumb about.
\nPandas has been the standard in this space for a long time. I don't like lots about it, but It's pretty versatile, robust, reliable, and has the power of incumbency. Call it Stockholm syndrome, but I will not be switching over lots of existing code - which means maintaining a level of expertise in pandas.
\nBut for new projects? That gets interesting. Polars has an appeal in the \"medium space\", but I'm interested in scaling seamlessly from small to big data.
\nI'm leaning more towards Ibis. Ibis provides a consistent API across different backends (including DuckDB, spark and Polars). We're not locked into a single-machine execution engine and can scale from local to production.
", "votes": null, "replies": [ { "id": 3033124, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "10/31/2024 19:07:12", "content": "Dear @paddykb
\nMany thanks for the heads-up regarding Ibis.
\nI did notice whilst perusing the documentation that
\n\n\n…matplotlib has not implemented the dataframe interchange protocol so it is recommended to call to_pandas() on the Ibis table before plotting. [Ref]
\n
so it could be a little clunky when it comes to performing extensive EDA.
\nAll the best,
\ncarl
Pandas has been in the game for so long that any ML library is compatible with its data structures (DataFrame, Series), so it's very comfortable to use it with other libraries. While today most of the libraries also work well with polars, there are still some that aren't. There are some preprocessing libraries and methods that only work with the pandas DataFrame. Another reason is legacy code - for older projects it may be hard and not worth it to refactor and change everything into polars. But I think the primary reason is the people are just afraid to learn new library. The syntax is a bit different to pandas and as you said, people are too accustomed to it to make the swap.
\nFor me, switching to polars was great. I find the syntax to be very comfortable (reminds me of SQL, which I really like), so I rarely use pandas these days. I use pandas only for older projects and when I need to be able to use libraries that can't work with polars.
", "votes": null, "replies": [] }, { "id": 3031901, "author_name": "lucasmorin", "author_url": "", "post_date": "10/30/2024 09:16:48", "content": "Polars is relatively new… it lacks some capabilities related to pandas and its support is not widespread in existing libraries. But its getting traction.
", "votes": null, "replies": [] }, { "id": 3032619, "author_name": "younesbenalia", "author_url": "", "post_date": "10/31/2024 06:43:53", "content": "this is my first time using polars too, i will name a few things that makes me not switch to polars:
\nInplace assignments like this df.loc[validation_mask, 'prediction'] = df.loc[validation_mask, 'prediction'].clip(lower=-5, upper=5) are slower in polars and they return a copy of the original dataframe when you use df.with_columns so the memory usage spikes when execution reaches that point. I'm using polars for faster joins and feature engineering but then I cast it to pandas dataframe before my training loop.