![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2099265%2Fe314f1bee56e388ecb7ec1b090aec82d%2Fpandas_polars_datatable.png?generation=1707791559484413&alt=media\")
For this competition we've got quite a lot of data: dozens of files with the overall size of ~26 GB. Loading data is going to be an intrinsic part of every kernel, so it is obviously important to minimize reading times as much as possible.
\nAccording to the official dataset description, all files can be found in both CSV and Parquet formats. So what is the fastest way to read them? In home-credit-faster-data-loading notebook I did some benchmarks of a few popular packages: pandas, polars and datatable. In this post I will briefly summarize the main findings.
\n
TL;DR: use polars to read Parquet or datatable to load memory-mapped files almost instantly.
The table below presents time taken by packages to load all the competition's data from the CSV files.
\n| Package | \nTime to read all CSV files [s] | \n
|---|---|
| pandas | \n1100.03 | \n
| polars | \n142.73 | \n
| datatable | \n118.37 | \n
We can also look at this benchmark file-wise for the first few files in the training dataset.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2099265%2Fb546572054224b09182d6e8be084afd5%2Fload_csv.png?generation=1707382589451428&alt=media\")
As we see,pandas is pretty slow when it comes to load larger files. datatable is doing a little bit better than polars, though both these packages are quite fast demonstrating similar performance file-wise.
Loading data from Parquet is much faster, because that's a binary format. We only compare pandas and polars, because datatable doesn't have a dedicated function to load .parquet relying on the arrow library.
| Package | \nTime to read all Parquet files [s] | \n
|---|---|
| pandas | \n129.83 | \n
| polars | \n55.22 | \n
Similarly, we can also plot file-wise comparison.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2099265%2Fdd96fa932bf9aeba7d17c60f7fc6de90%2Fload_parquet.png?generation=1707382782914363&alt=media\")
As expected, both packages process Parquet much faster than CSV with polars demonstrating better performance. However, processing times are still non-negligible when it comes to loading all the data.
Once data is loaded into a dataframe, we can actually export it in a binary format appropriate for memory mapping: ipc for polars and jay for datatable. Reading data back should literally take no time, because files are mapped into the memory address space very fast. To confirm that, I benchmarked .ipc and .jay loading for five first files from the competition's data.
| Package | \nTime to memory map five files [s] | \n
|---|---|
| polars | \n1.25 | \n
| datatable | \n0.04 | \n
The measured timings are much smaller than even those for Parquet with datatable memory mapping being almost immediate. When comparing file-wise, one can see that datatable's performance it almost file independent, while polars timings change quite a lot.
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2099265%2F78f6b9f2c118f52878c4e01b83394008%2Fload_mmap.png?generation=1707382990401744&alt=media\")
To sum up, CSV is the slowest way to get the data, reading Parquet is about twice faster and memory mapping could be almost instant. The final benchmarking results are also summarized in the table below. Note, we are not using actual numbers in this table, because those could vary on different systems from run to run.
\n| \n | pandas | \npolars | \ndatatable | \n
|---|---|---|---|
| CSV | \nSlow | \nFast | \nFast | \n
| Parquet | \nOK | \nFast | \nN/A | \n
| Memory mapping | \nN/A | \nOK | \nFast | \n
For more details, please refer to home-credit-faster-data-loading notebook. Hope it helps and good luck with this awesome competition!
", "messages": [ { "id": "2642588", "postDate": "02/08/2024 09:10:58", "content": "For this competition we've got quite a lot of data: dozens of files with the overall size of ~26 GB. Loading data is going to be an intrinsic part of every kernel, so it is obviously important to minimize reading times as much as possible.
\nAccording to the official dataset description, all files can be found in both CSV and Parquet formats. So what is the fastest way to read them? In home-credit-faster-data-loading notebook I did some benchmarks of a few popular packages: pandas, polars and datatable. In this post I will briefly summarize the main findings.
\n
TL;DR: use polars to read Parquet or datatable to load memory-mapped files almost instantly.
The table below presents time taken by packages to load all the competition's data from the CSV files.
\n| Package | \nTime to read all CSV files [s] | \n
|---|---|
| pandas | \n1100.03 | \n
| polars | \n142.73 | \n
| datatable | \n118.37 | \n
We can also look at this benchmark file-wise for the first few files in the training dataset.
\n
As we see,pandas is pretty slow when it comes to load larger files. datatable is doing a little bit better than polars, though both these packages are quite fast demonstrating similar performance file-wise.
Loading data from Parquet is much faster, because that's a binary format. We only compare pandas and polars, because datatable doesn't have a dedicated function to load .parquet relying on the arrow library.
| Package | \nTime to read all Parquet files [s] | \n
|---|---|
| pandas | \n129.83 | \n
| polars | \n55.22 | \n
Similarly, we can also plot file-wise comparison.
\n
As expected, both packages process Parquet much faster than CSV with polars demonstrating better performance. However, processing times are still non-negligible when it comes to loading all the data.
Once data is loaded into a dataframe, we can actually export it in a binary format appropriate for memory mapping: ipc for polars and jay for datatable. Reading data back should literally take no time, because files are mapped into the memory address space very fast. To confirm that, I benchmarked .ipc and .jay loading for five first files from the competition's data.
| Package | \nTime to memory map five files [s] | \n
|---|---|
| polars | \n1.25 | \n
| datatable | \n0.04 | \n
The measured timings are much smaller than even those for Parquet with datatable memory mapping being almost immediate. When comparing file-wise, one can see that datatable's performance it almost file independent, while polars timings change quite a lot.
To sum up, CSV is the slowest way to get the data, reading Parquet is about twice faster and memory mapping could be almost instant. The final benchmarking results are also summarized in the table below. Note, we are not using actual numbers in this table, because those could vary on different systems from run to run.
\n| \n | pandas | \npolars | \ndatatable | \n
|---|---|---|---|
| CSV | \nSlow | \nFast | \nFast | \n
| Parquet | \nOK | \nFast | \nN/A | \n
| Memory mapping | \nN/A | \nOK | \nFast | \n
For more details, please refer to home-credit-faster-data-loading notebook. Hope it helps and good luck with this awesome competition!
", "rawMarkdown": "For this competition we've got quite a lot of data: dozens of files with the overall size of ~26 GB. Loading data is going to be an intrinsic part of every kernel, so it is obviously important to minimize reading times as much as possible. \n\nAccording to the official dataset description, all files can be found in both CSV and Parquet formats. So what is the fastest way to read them? In [home-credit-faster-data-loading](https://www.kaggle.com/code/kononenko/home-credit-faster-data-loading/) notebook I did some benchmarks of a few popular packages: [pandas](https://pandas.pydata.org/docs/), [polars](https://pola.rs/) and [datatable](https://datatable.readthedocs.io/). In this post I will briefly summarize the main findings.\n\n\n\n**TL;DR:** use `polars` to read Parquet or `datatable` to load memory-mapped files almost instantly.\n\n# 1. Loading data from CSV files\n\nThe table below presents time taken by packages to load all the competition's data from the CSV files.\n\n| Package | Time to read all CSV files [s] |\n|----------|----------|\n| pandas | 1100.03 |\n| polars |142.73 |\n| datatable | 118.37 |\n\nImpressive, thanks for the effort and comparison.
", "rawMarkdown": "Impressive, thanks for the effort and comparison.", "votes": null }, { "id": "2646893", "postDate": "02/11/2024 09:11:58", "content": "Hi, do you think loading data faster is crucial in this competion?
", "rawMarkdown": "Hi, do you think loading data faster is crucial in this competion?", "votes": null }, { "id": "2646926", "postDate": "02/11/2024 09:42:03", "content": "I think so, because if one makes an incorrect choice by sticking to pandas to read all the .csv files, it will take 1000+ seconds just to load the data, and this is something that needs to be done as often as we start a new session in our competition notebook…
But pandas and parquet would be ok in your opinion?
", "rawMarkdown": "But pandas and parquet would be ok in your opinion?", "votes": null }, { "id": "2646946", "postDate": "02/11/2024 09:51:58", "content": "It is not that bad, see benchmarks here, about 2 minutes to load all the data, polars would do it in 1 minute. So it is up to you to decide, if you need 50% speed-up or not.
Thank You dear Oleksiy @kononenko for Your comparison and getting me familiar with datatable! I will try to use them both!
", "rawMarkdown": "Thank You dear Oleksiy @kononenko for Your comparison and getting me familiar with datatable! I will try to use them both!", "votes": null }, { "id": "2652002", "postDate": "02/14/2024 13:59:30", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing", "votes": null }, { "id": "2671195", "postDate": "02/27/2024 11:30:43", "content": "Thanks for the comparison, very interesting!
\nFrom a kaggle newbie: Do you plan to then manipulate the data with the loading library, or do you load it with the fastest library and then convert it to your favorite one? (or the most capable one?). For example, I feel comfortable with pandas, and learning to do all the same things in polars is not very appealing.
", "rawMarkdown": "Thanks for the comparison, very interesting! \n\nFrom a kaggle newbie: Do you plan to then manipulate the data with the loading library, or do you load it with the fastest library and then convert it to your favorite one? (or the most capable one?). For example, I feel comfortable with pandas, and learning to do all the same things in polars is not very appealing.", "votes": null }, { "id": "2671268", "postDate": "02/27/2024 12:21:32", "content": "You can go with any of the options you mention, just additional benchmarks are needed as to how fast/memory hungry are conversions, join and other munging capabilities.
One thing to keep in mind is that at the very end one needs to pass the resulting frames to modeling packages like LightGBM and those may not support all the possible frame types, so conversion to pandas or arrow may be necessary, that could take additional time.
thank you share,very good !
", "rawMarkdown": "thank you share,very good !", "votes": null } ], "comments": [ { "id": 2642745, "author_name": "jetakow", "author_url": "", "post_date": "02/08/2024 11:31:23", "content": "Impressive, thanks for the effort and comparison.
", "votes": null, "replies": [] }, { "id": 2646893, "author_name": "lucamtb", "author_url": "", "post_date": "02/11/2024 09:11:58", "content": "Hi, do you think loading data faster is crucial in this competion?
", "votes": null, "replies": [ { "id": 2646926, "author_name": "kononenko", "author_url": "", "post_date": "02/11/2024 09:42:03", "content": "I think so, because if one makes an incorrect choice by sticking to pandas to read all the .csv files, it will take 1000+ seconds just to load the data, and this is something that needs to be done as often as we start a new session in our competition notebook…
But pandas and parquet would be ok in your opinion?
", "votes": null, "replies": [ { "id": 2646946, "author_name": "kononenko", "author_url": "", "post_date": "02/11/2024 09:51:58", "content": "It is not that bad, see benchmarks here, about 2 minutes to load all the data, polars would do it in 1 minute. So it is up to you to decide, if you need 50% speed-up or not.
Thank You dear Oleksiy @kononenko for Your comparison and getting me familiar with datatable! I will try to use them both!
", "votes": null, "replies": [] }, { "id": 2652002, "author_name": "", "author_url": "", "post_date": "02/14/2024 13:59:30", "content": "Thanks for sharing
", "votes": null, "replies": [] }, { "id": 2671195, "author_name": "rpicatoste", "author_url": "", "post_date": "02/27/2024 11:30:43", "content": "Thanks for the comparison, very interesting!
\nFrom a kaggle newbie: Do you plan to then manipulate the data with the loading library, or do you load it with the fastest library and then convert it to your favorite one? (or the most capable one?). For example, I feel comfortable with pandas, and learning to do all the same things in polars is not very appealing.
", "votes": null, "replies": [ { "id": 2671268, "author_name": "kononenko", "author_url": "", "post_date": "02/27/2024 12:21:32", "content": "You can go with any of the options you mention, just additional benchmarks are needed as to how fast/memory hungry are conversions, join and other munging capabilities.
One thing to keep in mind is that at the very end one needs to pass the resulting frames to modeling packages like LightGBM and those may not support all the possible frame types, so conversion to pandas or arrow may be necessary, that could take additional time.
thank you share,very good !
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2642588": "For this competition we've got quite a lot of data: dozens of files with the overall size of ~26 GB. Loading data is going to be an intrinsic part of every kernel, so it is obviously important to minimize reading times as much as possible. \n\nAccording to the official dataset description, all files can be found in both CSV and Parquet formats. So what is the fastest way to read them? In [home-credit-faster-data-loading](https://www.kaggle.com/code/kononenko/home-credit-faster-data-loading/) notebook I did some benchmarks of a few popular packages: [pandas](https://pandas.pydata.org/docs/), [polars](https://pola.rs/) and [datatable](https://datatable.readthedocs.io/). In this post I will briefly summarize the main findings.\n\n\n\n**TL;DR:** use `polars` to read Parquet or `datatable` to load memory-mapped files almost instantly.\n\n# 1. Loading data from CSV files\n\nThe table below presents time taken by packages to load all the competition's data from the CSV files.\n\n| Package | Time to read all CSV files [s] |\n|----------|----------|\n| pandas | 1100.03 |\n| polars |142.73 |\n| datatable | 118.37 |\n\n