{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<h1 style=\"text-align: center;\"> Polars memory usage optimization (Jo Wilder competition example)</h1>\n\n<h2><center> <img src=\"https://upload.wikimedia.org/wikipedia/commons/2/2f/Corsair_CM2X1024-6400C5DHX_20080221.jpg\" style=\"max-width: 40%;\" alt=\"RAMMemory\"></center></h2>","metadata":{}},{"cell_type":"code","source":"import polars as pl\nimport numpy as np\nimport gc","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-27T10:06:10.750470Z","iopub.execute_input":"2023-03-27T10:06:10.751636Z","iopub.status.idle":"2023-03-27T10:06:10.966726Z","shell.execute_reply.started":"2023-03-27T10:06:10.751591Z","shell.execute_reply":"2023-03-27T10:06:10.965542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Polars uses less memory than Pandas, but sometimes it's not enough. This notebook researches specific case when data load fails with memory errors. For example, in Jo Wilder competition after the [data update](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202) train set became > 4 GB. As a result, it does not fit the kaggle machines' memory.\n\nIn `polars` there are three ways to handle the issue.\n\n### 1. Use low_memory load setting\n\nIf the data is smaller than the available memory, it can be loaded via the `low_memory` setting:\n\n```python\ntrain = pl.read_csv(\"/kaggle/input/predict-student-performance-from-game-play/train.csv\", low_memory=True)\n```\n\n\nNevertheless, we might get memory errors in further operations, so this is not a viable solution in most cases.","metadata":{}},{"cell_type":"markdown","source":"### 2. Assign data types during csv read","metadata":{}},{"cell_type":"code","source":"dtypes = {\"session_id\": pl.Int64,\n          \"elapsed_time\": pl.Int64,\n          \"event_name\": pl.Categorical,\n          \"name\": pl.Categorical,\n          \"level\": pl.Int8,\n          \"page\": pl.Float32,\n          \"room_coor_x\": pl.Float32,\n          \"room_coor_y\": pl.Float32,\n          \"screen_coor_x\": pl.Float32,\n          \"screen_coor_y\": pl.Float32,\n          \"hover_duration\": pl.Float32,\n          \"text\": pl.Categorical,\n          \"fqid\": pl.Categorical,\n          \"room_fqid\": pl.Categorical,\n          \"text_fqid\": pl.Categorical,\n          \"fullscreen\": pl.Int8,\n          \"hq\": pl.Int8,\n          \"music\": pl.Int8,\n          \"level_group\": pl.Categorical\n          }\ntrain = pl.read_csv(\"/kaggle/input/predict-student-performance-from-game-play/train.csv\", dtypes=dtypes)\nprint(f\"Memory usage of dataframe is {round(train.estimated_size('mb'), 2)} MB\")","metadata":{"execution":{"iopub.status.busy":"2023-03-27T10:06:10.968516Z","iopub.execute_input":"2023-03-27T10:06:10.968935Z","iopub.status.idle":"2023-03-27T10:06:54.688069Z","shell.execute_reply.started":"2023-03-27T10:06:10.968902Z","shell.execute_reply":"2023-03-27T10:06:54.685328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train\ngc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-03-27T10:06:54.689628Z","iopub.execute_input":"2023-03-27T10:06:54.689965Z","iopub.status.idle":"2023-03-27T10:06:54.969485Z","shell.execute_reply.started":"2023-03-27T10:06:54.689932Z","shell.execute_reply":"2023-03-27T10:06:54.968510Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3. Use lazy load and fetch data subset\n\nPolars allows you to lazily read from a CSV file, and `select` a subset of columns (and/or `filter` a subset of rows) before fetching full data into memory.\n\nThis is a good option if you're OK working with a subset of data.","metadata":{}},{"cell_type":"code","source":"# for example, let's load limited number of columns for level_group 0-4\ncols = [\"session_id\", \"elapsed_time\", \"event_name\", \"name\", \"level\", \"page\", \"room_coor_x\", \"room_coor_y\", \"screen_coor_x\",\n        \"screen_coor_y\", \"hover_duration\", \"text\", \"fqid\", \"room_fqid\", \"text_fqid\", \"level_group\"]\ntrain_subset = pl.scan_csv(\"/kaggle/input/predict-student-performance-from-game-play/train.csv\", low_memory=True) \\\n    .select(cols).filter(pl.col(\"level_group\") == \"0-4\").collect()","metadata":{"execution":{"iopub.status.busy":"2023-03-27T10:06:54.971971Z","iopub.execute_input":"2023-03-27T10:06:54.972632Z","iopub.status.idle":"2023-03-27T10:07:15.012644Z","shell.execute_reply.started":"2023-03-27T10:06:54.972595Z","shell.execute_reply":"2023-03-27T10:07:15.011349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Memory usage of the already loaded dataset can be further optimized by changing data types:","metadata":{}},{"cell_type":"code","source":"def reduce_memory_usage_pl(df, name):\n    \"\"\" Reduce memory usage by polars dataframe {df} with name {name} by changing its data types.\n        Original pandas version of this function: https://www.kaggle.com/code/arjanso/reducing-dataframe-memory-size-by-65 \"\"\"\n    print(f\"Memory usage of dataframe {name} is {round(df.estimated_size('mb'), 2)} MB\")\n    Numeric_Int_types = [pl.Int8,pl.Int16,pl.Int32,pl.Int64]\n    Numeric_Float_types = [pl.Float32,pl.Float64]    \n    for col in df.columns:\n        col_type = df[col].dtype\n        c_min = df[col].min()\n        c_max = df[col].max()\n        if col_type in Numeric_Int_types:\n            if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                df = df.with_columns(df[col].cast(pl.Int8))\n            elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                df = df.with_columns(df[col].cast(pl.Int16))\n            elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                df = df.with_columns(df[col].cast(pl.Int32))\n            elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                df = df.with_columns(df[col].cast(pl.Int64))\n        elif col_type in Numeric_Float_types:\n            if c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                df = df.with_columns(df[col].cast(pl.Float32))\n            else:\n                pass\n        elif col_type == pl.Utf8:\n            df = df.with_columns(df[col].cast(pl.Categorical))\n        else:\n            pass\n    print(f\"Memory usage of dataframe {name} became {round(df.estimated_size('mb'), 2)} MB\")\n    return df\n\n\n# usage example\ntrain_subset = reduce_memory_usage_pl(train_subset, \"train_subset\")","metadata":{"execution":{"iopub.status.busy":"2023-03-27T10:07:15.013949Z","iopub.execute_input":"2023-03-27T10:07:15.014521Z","iopub.status.idle":"2023-03-27T10:07:16.969304Z","shell.execute_reply.started":"2023-03-27T10:07:15.014482Z","shell.execute_reply":"2023-03-27T10:07:16.968228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### References\n\n- [Polars data types](https://pola-rs.github.io/polars/py-polars/html/reference/datatypes.html) (official documentation)\n- [Polars LazyFrame](https://pola-rs.github.io/polars/py-polars/html/reference/lazyframe/index.html) (official documentation)","metadata":{}}]}