{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Pandas vs Polars\n🐼 🆚 🐻‍❄️\n\nPandas and Polars packages both provide almost similar functionalities for data manipulation and analysis. In this notebook we will compare the performance of these bears in a fair arena.\n\nLet's start with an introduction:\n\n* **Pandas**: More features, Popular, Large community of supporter.\n* **Polars**: Lightweight, Minimal, Fast.","metadata":{}},{"cell_type":"code","source":"!pip install polars","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-01-25T04:52:19.891672Z","iopub.execute_input":"2023-01-25T04:52:19.892088Z","iopub.status.idle":"2023-01-25T04:52:33.018278Z","shell.execute_reply.started":"2023-01-25T04:52:19.892062Z","shell.execute_reply":"2023-01-25T04:52:33.016618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Pandas\nimport pandas as pd\n\n# Polars\nimport polars as pl\n\n# Scoreboard\nimport matplotlib.pyplot as plt","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-01-25T04:52:33.021100Z","iopub.execute_input":"2023-01-25T04:52:33.021575Z","iopub.status.idle":"2023-01-25T04:52:33.085581Z","shell.execute_reply.started":"2023-01-25T04:52:33.021539Z","shell.execute_reply":"2023-01-25T04:52:33.082781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# dataset files\ncsv_file = '/kaggle/input/playground-series-s3e2/train.csv'\nsqlite_file = '/kaggle/input/basketball/basketball.sqlite'\nparquet_file = '/kaggle/input/icecube-neutrinos-in-deep-ice/train/batch_1.parquet'\n\n# load dataframes\npd_df = pd.read_csv(csv_file)\npl_df = pl.scan_csv(csv_file)\n\n# By default polars reads (scans) csv in Lazy Mode and suggest using it.\n# But some functionalities are not available in Lazy Mode.\npl_df_not_lazy = pl.from_pandas(pd_df)\n\n# Plotting function to compare the results\ndef plot(pd_time, pl_time, title, unit = 'ms'):\n    Name = ['Pandas', 'Polars']\n    Values = [pd_time, pl_time]\n    plt.subplots(figsize=(6,3))\n    plt.grid()\n    plt.rc('axes', axisbelow=True)\n    plt.barh(Name, Values, color=['#999','#fff'], edgecolor='#666', height=.7)\n    plt.title(title)\n    plt.xlabel(f\"Average Time ({unit})\")\n    plt.show()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-01-25T04:52:33.088264Z","iopub.execute_input":"2023-01-25T04:52:33.089244Z","iopub.status.idle":"2023-01-25T04:52:33.099984Z","shell.execute_reply.started":"2023-01-25T04:52:33.089164Z","shell.execute_reply":"2023-01-25T04:52:33.097946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Read CSV File","metadata":{}},{"cell_type":"code","source":"%timeit pd.read_csv(csv_file)","metadata":{"execution":{"iopub.status.busy":"2023-01-25T04:52:33.118881Z","iopub.execute_input":"2023-01-25T04:52:33.119572Z","iopub.status.idle":"2023-01-25T04:52:49.200036Z","shell.execute_reply.started":"2023-01-25T04:52:33.119518Z","shell.execute_reply":"2023-01-25T04:52:49.197518Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%timeit pl.scan_csv(csv_file)","metadata":{"execution":{"iopub.status.busy":"2023-01-25T04:52:49.203537Z","iopub.execute_input":"2023-01-25T04:52:49.205078Z","iopub.status.idle":"2023-01-25T04:52:55.714965Z","shell.execute_reply.started":"2023-01-25T04:52:49.205006Z","shell.execute_reply":"2023-01-25T04:52:55.713644Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot(18.8, .747, 'Load CSV File')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-01-25T04:52:55.716870Z","iopub.execute_input":"2023-01-25T04:52:55.717236Z","iopub.status.idle":"2023-01-25T04:52:55.919940Z","shell.execute_reply.started":"2023-01-25T04:52:55.717203Z","shell.execute_reply":"2023-01-25T04:52:55.918410Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Read Parquet File","metadata":{}},{"cell_type":"code","source":"%timeit pd.read_parquet(parquet_file)","metadata":{"execution":{"iopub.status.busy":"2023-01-25T04:52:55.921454Z","iopub.execute_input":"2023-01-25T04:52:55.921807Z","iopub.status.idle":"2023-01-25T04:53:03.779791Z","shell.execute_reply.started":"2023-01-25T04:52:55.921768Z","shell.execute_reply":"2023-01-25T04:53:03.778875Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%timeit pl.scan_parquet(parquet_file)","metadata":{"execution":{"iopub.status.busy":"2023-01-25T04:53:03.783924Z","iopub.execute_input":"2023-01-25T04:53:03.786063Z","iopub.status.idle":"2023-01-25T04:53:08.694266Z","shell.execute_reply.started":"2023-01-25T04:53:03.786017Z","shell.execute_reply":"2023-01-25T04:53:08.693188Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot(1090, .621, 'Load Parquet File')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-01-25T04:53:08.698478Z","iopub.execute_input":"2023-01-25T04:53:08.699049Z","iopub.status.idle":"2023-01-25T04:53:08.797166Z","shell.execute_reply.started":"2023-01-25T04:53:08.699014Z","shell.execute_reply":"2023-01-25T04:53:08.796250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Select Columns","metadata":{}},{"cell_type":"code","source":"%timeit pd_df[[\"id\", \"age\", \"stroke\"]]","metadata":{"execution":{"iopub.status.busy":"2023-01-25T04:53:08.824891Z","iopub.execute_input":"2023-01-25T04:53:08.826075Z","iopub.status.idle":"2023-01-25T04:53:12.274636Z","shell.execute_reply.started":"2023-01-25T04:53:08.826022Z","shell.execute_reply":"2023-01-25T04:53:12.273621Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%timeit pl_df.select((\"id\", \"age\", \"stroke\"))","metadata":{"execution":{"iopub.status.busy":"2023-01-25T04:53:12.277614Z","iopub.execute_input":"2023-01-25T04:53:12.278829Z","iopub.status.idle":"2023-01-25T04:53:20.242915Z","shell.execute_reply.started":"2023-01-25T04:53:12.278777Z","shell.execute_reply":"2023-01-25T04:53:20.241858Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot(536, 11, 'Select Columns')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-01-25T04:53:20.244249Z","iopub.execute_input":"2023-01-25T04:53:20.244584Z","iopub.status.idle":"2023-01-25T04:53:20.349904Z","shell.execute_reply.started":"2023-01-25T04:53:20.244551Z","shell.execute_reply":"2023-01-25T04:53:20.349043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Filtering","metadata":{}},{"cell_type":"code","source":"%timeit pd_df[(pd_df[\"age\"] > 40) & (pd_df[\"stroke\"] == 1)]","metadata":{"execution":{"iopub.status.busy":"2023-01-25T04:53:20.351543Z","iopub.execute_input":"2023-01-25T04:53:20.352187Z","iopub.status.idle":"2023-01-25T04:53:25.361794Z","shell.execute_reply.started":"2023-01-25T04:53:20.352153Z","shell.execute_reply":"2023-01-25T04:53:25.359533Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%timeit pl_df.filter((pl.col(\"age\") > 40) & (pl.col(\"stroke\") == 0))","metadata":{"execution":{"iopub.status.busy":"2023-01-25T04:53:25.364806Z","iopub.execute_input":"2023-01-25T04:53:25.365831Z","iopub.status.idle":"2023-01-25T04:53:38.453220Z","shell.execute_reply.started":"2023-01-25T04:53:25.365739Z","shell.execute_reply":"2023-01-25T04:53:38.451453Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot(796, 18.2, 'Filtering')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-01-25T04:53:38.455138Z","iopub.execute_input":"2023-01-25T04:53:38.455891Z","iopub.status.idle":"2023-01-25T04:53:38.573968Z","shell.execute_reply.started":"2023-01-25T04:53:38.455853Z","shell.execute_reply":"2023-01-25T04:53:38.572956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Grouping","metadata":{}},{"cell_type":"code","source":"%timeit pd_df.groupby(\"age\").mean()","metadata":{"execution":{"iopub.status.busy":"2023-01-25T04:53:38.577042Z","iopub.execute_input":"2023-01-25T04:53:38.577400Z","iopub.status.idle":"2023-01-25T04:53:40.799355Z","shell.execute_reply.started":"2023-01-25T04:53:38.577371Z","shell.execute_reply":"2023-01-25T04:53:40.797527Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%timeit pl_df.groupby(\"age\").agg(pl.all().mean())","metadata":{"execution":{"iopub.status.busy":"2023-01-25T04:53:40.801110Z","iopub.execute_input":"2023-01-25T04:53:40.801517Z","iopub.status.idle":"2023-01-25T04:53:55.345163Z","shell.execute_reply.started":"2023-01-25T04:53:40.801490Z","shell.execute_reply":"2023-01-25T04:53:55.343926Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot(3.1, .02, 'Grouping')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-01-25T04:53:55.346952Z","iopub.execute_input":"2023-01-25T04:53:55.347379Z","iopub.status.idle":"2023-01-25T04:53:55.615279Z","shell.execute_reply.started":"2023-01-25T04:53:55.347319Z","shell.execute_reply":"2023-01-25T04:53:55.614376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Sorting","metadata":{}},{"cell_type":"code","source":"%timeit pd_df.sort_values([\"age\", \"bmi\"])","metadata":{"execution":{"iopub.status.busy":"2023-01-25T05:00:35.573561Z","iopub.execute_input":"2023-01-25T05:00:35.573856Z","iopub.status.idle":"2023-01-25T05:00:39.812185Z","shell.execute_reply.started":"2023-01-25T05:00:35.573831Z","shell.execute_reply":"2023-01-25T05:00:39.810894Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%timeit pl_df.sort([\"age\", \"bmi\"])","metadata":{"execution":{"iopub.status.busy":"2023-01-25T05:00:29.460396Z","iopub.execute_input":"2023-01-25T05:00:29.460808Z","iopub.status.idle":"2023-01-25T05:00:35.571577Z","shell.execute_reply.started":"2023-01-25T05:00:29.460783Z","shell.execute_reply":"2023-01-25T05:00:35.569921Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot(5.2, .008, 'Sorting')","metadata":{"execution":{"iopub.status.busy":"2023-01-25T05:00:55.537284Z","iopub.execute_input":"2023-01-25T05:00:55.537731Z","iopub.status.idle":"2023-01-25T05:00:55.651095Z","shell.execute_reply.started":"2023-01-25T05:00:55.537704Z","shell.execute_reply":"2023-01-25T05:00:55.650226Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Merging","metadata":{}},{"cell_type":"code","source":"%timeit pd_df.merge(pd_df, on=\"age\")","metadata":{"execution":{"iopub.status.busy":"2023-01-25T05:03:25.916679Z","iopub.execute_input":"2023-01-25T05:03:25.917107Z","iopub.status.idle":"2023-01-25T05:03:45.576296Z","shell.execute_reply.started":"2023-01-25T05:03:25.917072Z","shell.execute_reply":"2023-01-25T05:03:45.574714Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%timeit pl_df.join(pl_df, on=\"age\")","metadata":{"execution":{"iopub.status.busy":"2023-01-25T05:03:45.578826Z","iopub.execute_input":"2023-01-25T05:03:45.579222Z","iopub.status.idle":"2023-01-25T05:03:53.045378Z","shell.execute_reply.started":"2023-01-25T05:03:45.579196Z","shell.execute_reply":"2023-01-25T05:03:53.043870Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot(2450, .01, 'Merging')","metadata":{"execution":{"iopub.status.busy":"2023-01-25T05:04:02.840664Z","iopub.execute_input":"2023-01-25T05:04:02.841117Z","iopub.status.idle":"2023-01-25T05:04:02.955740Z","shell.execute_reply.started":"2023-01-25T05:04:02.841080Z","shell.execute_reply":"2023-01-25T05:04:02.954732Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That's a huge difference!","metadata":{}},{"cell_type":"markdown","source":"# Pivoting","metadata":{}},{"cell_type":"code","source":"%timeit pd_df.pivot_table(values='age', index='bmi', columns='gender')","metadata":{"execution":{"iopub.status.busy":"2023-01-25T05:07:43.890989Z","iopub.execute_input":"2023-01-25T05:07:43.891523Z","iopub.status.idle":"2023-01-25T05:07:51.417922Z","shell.execute_reply.started":"2023-01-25T05:07:43.891481Z","shell.execute_reply":"2023-01-25T05:07:51.416218Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%timeit pl_df_not_lazy.pivot(values='age', index='bmi', columns='gender')","metadata":{"execution":{"iopub.status.busy":"2023-01-25T05:16:14.007620Z","iopub.execute_input":"2023-01-25T05:16:14.008050Z","iopub.status.idle":"2023-01-25T05:16:25.509855Z","shell.execute_reply.started":"2023-01-25T05:16:14.008017Z","shell.execute_reply":"2023-01-25T05:16:25.507267Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot(9.26, 1.4, 'Pivoting')","metadata":{"execution":{"iopub.status.busy":"2023-01-25T05:16:28.547662Z","iopub.execute_input":"2023-01-25T05:16:28.548170Z","iopub.status.idle":"2023-01-25T05:16:28.655786Z","shell.execute_reply.started":"2023-01-25T05:16:28.548135Z","shell.execute_reply":"2023-01-25T05:16:28.654900Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We know who is the winner.\n\nPolars is still a young library with decent functionalities and amazingly fast performance. I'm going to start learning more about polars to become more familiar with it, but for now I'll continue using pandas.\nI encourage you all to have a look at the polars' API documentation. https://pola-rs.github.io/polars/py-polars/html/reference/index.html","metadata":{}}]}