{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#

Lets optimize

\n\n> The objective of this NB is to showcase some **optimization (Processing & Memory optimization)** while working with Pandas \n\n> The NB includes optimization techniques involving the use of **Pandas** and **Numpy** mainly\n\n> I am also planning to add **Python optimization** techniques & maybe something on **ML & NLP models optimization** depending on the feedback this NB receives\n\n-----------------\n\n
\n

\n

  • I am no expert in optimizations, the whole point of the NB is to optmize the code especially building data pipelines
    \n \n
  • Please feel free to share any feedback or share any corrections

    • \n \n
  • For a few usecases I have used TPS March 2022 and Ubiquant market prediction data & others Just to make sure these optimizations work for real data as well
    • \n

    \n
    \n\n\n
    \nTricks with emojis must not be missed\n
    ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\nimport numpy as np\nfrom matplotlib import pyplot as plt\nimport os","metadata":{"execution":{"iopub.status.busy":"2022-03-08T05:50:21.516519Z","iopub.execute_input":"2022-03-08T05:50:21.516914Z","iopub.status.idle":"2022-03-08T05:50:21.545906Z","shell.execute_reply.started":"2022-03-08T05:50:21.51681Z","shell.execute_reply":"2022-03-08T05:50:21.544916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

    📌Datatypes

    \n\n\nWe generally dont give much importance to datatypes in datasets, but the fact is using correct datatypes can save us a lot of memory and time which working/loading the datasets\n\n\nFollowing tricks can be used to optimize using correct datatypes\n\n- Float & Int datatype\n- datetime vs string\n- object vs category\n--------------\n\nWill use `ubiquant-market-prediction` dataset for this section\nhttps://www.kaggle.com/c/ubiquant-market-prediction","metadata":{}},{"cell_type":"markdown","source":"### Float & Int datatype\n\n\n- Downcasting `float` and `int` datatypes can give huge memory boost\n- Based on the range of data and the usecase, we can choose to downcast the datatypes\n\n-------------------\nFor **Int** , following are the ranges for each int (precision) can store\n\n```\nint8 can store integers from -128 to 127\nint16 can store integers from -32768 to 32767.\nint64 can store integers from -9223372036854775808 to 9223372036854775807.\n```\n\n-------------------\n**Float** also has float8,float16, etc precision values\n\nref : https://numpy.org/doc/stable/user/basics.types.html\n\n\n\n
    \n

    \nLoading data with downcasted dtypes can save a lot of space and time \n

    \n
    \n\n\n- As can be seen, using **lower precision** in floats and ints can easily give us **2X- 4X** boost\n> Size of **64bit (df64)** precision data is **4x** to that of **16bit (df16)** precision","metadata":{}},{"cell_type":"code","source":"data_sample = pd.read_csv(\"../input/ubiquant-market-prediction/train.csv\", nrows = 10)\nfloat_col_list = data_sample.filter(like = \"f_\").columns.tolist()\n\n# 32 bit conversion mapping\nfp32_dtypes = {x:y for x,y in zip(float_col_list, [\"float32\"]*len(float_col_list))}\nfp32_dtypes.update({\"time_id\" : \"int32\", \"investment_id\" : \"int32\", \"target\" : \"float32\"})\n\n# 16 bit conversion mapping\nfp16_dtypes = {x:y for x,y in zip(float_col_list, [\"float16\"]*len(float_col_list))}\nfp16_dtypes.update({\"time_id\" : \"int16\", \"investment_id\" : \"int16\", \"target\" : \"float16\"})","metadata":{"execution":{"iopub.status.busy":"2022-03-08T05:50:25.226473Z","iopub.execute_input":"2022-03-08T05:50:25.226767Z","iopub.status.idle":"2022-03-08T05:50:25.272177Z","shell.execute_reply.started":"2022-03-08T05:50:25.226735Z","shell.execute_reply":"2022-03-08T05:50:25.271288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df64 = pd.read_csv(\"../input/ubiquant-market-prediction/train.csv\", nrows = 10000)\nprint(\"df size with int64/float64 in MB:\",df64.memory_usage(deep = True).sum()/(1024**2))","metadata":{"execution":{"iopub.status.busy":"2022-03-08T05:50:28.699133Z","iopub.execute_input":"2022-03-08T05:50:28.699423Z","iopub.status.idle":"2022-03-08T05:50:30.113914Z","shell.execute_reply.started":"2022-03-08T05:50:28.699391Z","shell.execute_reply":"2022-03-08T05:50:30.112968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df32 = pd.read_csv(\"../input/ubiquant-market-prediction/train.csv\", dtype = fp32_dtypes ,nrows = 10000)\nprint(\"df size with int32/float32 in MB:\",df32.memory_usage(deep = True).sum()/(1024**2))","metadata":{"execution":{"iopub.status.busy":"2022-03-08T05:50:30.115489Z","iopub.execute_input":"2022-03-08T05:50:30.115741Z","iopub.status.idle":"2022-03-08T05:50:31.001279Z","shell.execute_reply.started":"2022-03-08T05:50:30.115712Z","shell.execute_reply":"2022-03-08T05:50:31.000052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df16 = pd.read_csv(\"../input/ubiquant-market-prediction/train.csv\", dtype = fp16_dtypes ,nrows = 10000)\nprint(\"df size with int16/float16 in MB:\",df16.memory_usage(deep = True).sum()/(1024**2))","metadata":{"execution":{"iopub.status.busy":"2022-03-08T05:50:31.003457Z","iopub.execute_input":"2022-03-08T05:50:31.003797Z","iopub.status.idle":"2022-03-08T05:50:32.030763Z","shell.execute_reply.started":"2022-03-08T05:50:31.003752Z","shell.execute_reply":"2022-03-08T05:50:32.029635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Datetime datatype\n\n- Generally when we load csv data, **datetime** data gets loaded as **object** dtype\n- Loading date-time data in datetime format using `pd.to_datetime` can save a sizeable memory\n- similar to shown incase of int/float downcasting , we can specify datetime dtype at the time of loading as well\n\n----------------------\n\nUsing https://www.kaggle.com/c/tabular-playground-series-mar-2022/data `TPS march 2022` data\n\n\n**Memory reduction : 61 MB to 6.5 MB (~90% reduction)**","metadata":{}},{"cell_type":"code","source":"# tabular playground - mar 2022 data\ntps_march = pd.read_csv(\"../input/tabular-playground-series-mar-2022/train.csv\")\ntps_march.head(2)","metadata":{"execution":{"iopub.status.busy":"2022-03-08T05:55:34.714567Z","iopub.execute_input":"2022-03-08T05:55:34.714872Z","iopub.status.idle":"2022-03-08T05:55:35.593169Z","shell.execute_reply.started":"2022-03-08T05:55:34.714842Z","shell.execute_reply":"2022-03-08T05:55:35.592379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"time column dtype :\", tps_march.time.dtype)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:21:05.503696Z","iopub.execute_input":"2022-03-05T13:21:05.503955Z","iopub.status.idle":"2022-03-05T13:21:05.513029Z","shell.execute_reply.started":"2022-03-05T13:21:05.503929Z","shell.execute_reply":"2022-03-05T13:21:05.512149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"size of time column in MB (when loaded as object dtype) : \",tps_march.time.memory_usage(deep = True)/(1024**2))","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:21:08.043958Z","iopub.execute_input":"2022-03-05T13:21:08.044211Z","iopub.status.idle":"2022-03-05T13:21:08.208102Z","shell.execute_reply.started":"2022-03-05T13:21:08.044184Z","shell.execute_reply":"2022-03-05T13:21:08.207146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# converting time column to datetime from object\ntps_march[\"time\"] = pd.to_datetime(tps_march.time)\nprint(\"size of time column in MB (when loaded as datetime dtype) : \",tps_march.time.memory_usage(deep = True)/(1024**2))","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:21:26.10606Z","iopub.execute_input":"2022-03-05T13:21:26.106331Z","iopub.status.idle":"2022-03-05T13:21:26.293328Z","shell.execute_reply.started":"2022-03-05T13:21:26.106303Z","shell.execute_reply":"2022-03-05T13:21:26.292377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Category datatype\n\n- Category dtype is a must to use datatype whenever you have **less # unique categories** in string data\n- Category dtype does not provide much gains for `high # unique categories`\n\n\n
    \n

    \nIt's not a magic how the size is getting reduced, it has got to do with how the data is stored in memory \n

    \n
    \n\nref : https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html\n\n\n----------------------\n\n**Memory reduction : 47.7MB to 0.81MB (~98% reduction)**","metadata":{}},{"cell_type":"code","source":"# tabular playground - mar 2022 data\ntps_march = pd.read_csv(\"../input/tabular-playground-series-mar-2022/train.csv\")\ntps_march.head(2)","metadata":{"execution":{"iopub.status.busy":"2022-03-08T04:32:41.34983Z","iopub.execute_input":"2022-03-08T04:32:41.350055Z","iopub.status.idle":"2022-03-08T04:32:42.293756Z","shell.execute_reply.started":"2022-03-08T04:32:41.350033Z","shell.execute_reply":"2022-03-08T04:32:42.293095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"direction column dtype :\", tps_march.time.dtype)","metadata":{"execution":{"iopub.status.busy":"2022-03-08T04:32:42.332792Z","iopub.execute_input":"2022-03-08T04:32:42.334018Z","iopub.status.idle":"2022-03-08T04:32:42.340356Z","shell.execute_reply.started":"2022-03-08T04:32:42.333964Z","shell.execute_reply":"2022-03-08T04:32:42.339096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"size of direction column in MB (when loaded as object dtype) :\",tps_march.direction.memory_usage(deep = True)/(1024**2))","metadata":{"execution":{"iopub.status.busy":"2022-03-08T04:32:43.155459Z","iopub.execute_input":"2022-03-08T04:32:43.156611Z","iopub.status.idle":"2022-03-08T04:32:43.224633Z","shell.execute_reply.started":"2022-03-08T04:32:43.15651Z","shell.execute_reply":"2022-03-08T04:32:43.223824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# converting direction column to category from object\ntps_march[\"direction\"] = tps_march.direction.astype(\"category\")\n\nprint(\"size of direction column in MB (when loaded as category dtype) :\",tps_march.direction.memory_usage(deep = True)/(1024**2))","metadata":{"execution":{"iopub.status.busy":"2022-03-08T04:32:44.021017Z","iopub.execute_input":"2022-03-08T04:32:44.021752Z","iopub.status.idle":"2022-03-08T04:32:44.143231Z","shell.execute_reply.started":"2022-03-08T04:32:44.021702Z","shell.execute_reply":"2022-03-08T04:32:44.142222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"```\n> Dummy usecase to test above rule\n\nv1 = np.random.choice([\"1\",\"2\",\"3\",\"4\",\"5\"], size = 100000)\nv2 = np.random.choice([\"XXL\",\"XL\",\"L\",\"M\",\"S\"], size = 100000)\n\ndf_obj = pd.DataFrame(zip(v1, v2), columns = [\"col1\",'col2'])\n\nprint(\"df size in MB (with object dtypes) :\", df_obj.memory_usage(deep = True).sum()/(1024**2))\n\ndf_cat = df_obj.apply(lambda x:x.astype(\"category\"))\n\nprint(\"df size in MB (with category dtypes) :\",df_cat.memory_usage(deep = True).sum()/(1024**2))\n```","metadata":{}},{"cell_type":"markdown","source":"#

    Merge operation

    \n\n- **Merge operation** on **dataframe's index** is faster than on a non-index columns\n- A smart way of merging dataframes is by using the **joining key as index**\n--------------\n\n- we are able to save **1X - 5X** time by merging df on **index**","metadata":{"execution":{"iopub.status.busy":"2022-02-16T18:07:13.003643Z","iopub.execute_input":"2022-02-16T18:07:13.00413Z","iopub.status.idle":"2022-02-16T18:07:13.007364Z","shell.execute_reply.started":"2022-02-16T18:07:13.00409Z","shell.execute_reply":"2022-02-16T18:07:13.006774Z"}}},{"cell_type":"code","source":"np.random.seed(123)\n\nnrows = 100000\nncols = 70\n\nkey = np.array([\"id_\" + str(x) for x in range(nrows)]).reshape(nrows,1)\nvalues1 = np.random.rand(nrows,ncols)\nvalues2 = np.random.rand(nrows,ncols)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:26:33.051963Z","iopub.execute_input":"2022-03-05T13:26:33.052657Z","iopub.status.idle":"2022-03-05T13:26:33.261299Z","shell.execute_reply.started":"2022-03-05T13:26:33.052585Z","shell.execute_reply":"2022-03-05T13:26:33.260672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df1 = pd.DataFrame(np.concatenate([key, values1],axis =1), columns = [\"id\"] + [\"col_1\" + str(x) for x in range(ncols)])\ndf2 = pd.DataFrame(np.concatenate([key, values2],axis =1), columns = [\"id\"] + [\"col_2\" + str(x) for x in range(ncols)])","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:26:34.898731Z","iopub.execute_input":"2022-03-05T13:26:34.899448Z","iopub.status.idle":"2022-03-05T13:26:51.620185Z","shell.execute_reply.started":"2022-03-05T13:26:34.899406Z","shell.execute_reply":"2022-03-05T13:26:51.619343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"df1\")\ndisplay(df1.head(2))\n\nprint()\n\nprint(\"df2\")\ndisplay(df2.head(2))","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:27:36.787294Z","iopub.execute_input":"2022-03-05T13:27:36.787555Z","iopub.status.idle":"2022-03-05T13:27:36.829945Z","shell.execute_reply.started":"2022-03-05T13:27:36.787528Z","shell.execute_reply":"2022-03-05T13:27:36.829152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Merging on column = \"id\"","metadata":{"execution":{"iopub.status.busy":"2022-02-21T18:03:08.980553Z","iopub.execute_input":"2022-02-21T18:03:08.981086Z","iopub.status.idle":"2022-02-21T18:03:08.987288Z","shell.execute_reply.started":"2022-02-21T18:03:08.981043Z","shell.execute_reply":"2022-02-21T18:03:08.986021Z"}}},{"cell_type":"code","source":"%%time\nout = df1.merge(df2, on = \"id\")","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:04:10.900622Z","iopub.execute_input":"2022-03-04T19:04:10.900972Z","iopub.status.idle":"2022-03-04T19:04:16.351815Z","shell.execute_reply.started":"2022-03-04T19:04:10.900938Z","shell.execute_reply":"2022-03-04T19:04:16.350806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Merging on index = \"id\"","metadata":{}},{"cell_type":"code","source":"# setting column = id as index\ndf1_ = df1.set_index(\"id\")\ndf2_ = df2.set_index(\"id\")","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:29:00.736925Z","iopub.execute_input":"2022-03-05T13:29:00.737864Z","iopub.status.idle":"2022-03-05T13:29:03.576816Z","shell.execute_reply.started":"2022-03-05T13:29:00.737821Z","shell.execute_reply":"2022-03-05T13:29:03.575936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"df1_\")\ndisplay(df1_.head(2))\n\nprint()\n\nprint(\"df2_\")\ndisplay(df2_.head(2))","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:29:04.038989Z","iopub.execute_input":"2022-03-05T13:29:04.03928Z","iopub.status.idle":"2022-03-05T13:29:04.090256Z","shell.execute_reply.started":"2022-03-05T13:29:04.039243Z","shell.execute_reply":"2022-03-05T13:29:04.089463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nout = df1_.merge(df2_, left_index = True, right_index = True)","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:04:31.289012Z","iopub.execute_input":"2022-03-04T19:04:31.289389Z","iopub.status.idle":"2022-03-04T19:04:32.846813Z","shell.execute_reply.started":"2022-03-04T19:04:31.289352Z","shell.execute_reply":"2022-03-04T19:04:32.845159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

    Chaining operations

    \n\n\n- When chaining multiple operations, the order is important. For example, it's faster to **filter first and then merge**\n- This technique can give great performance boost if we have duplicate keys in dataframe we are filtering\n\n-------------\n\n```\n* df1 : 50k records\n* df2 : 100k records\n\n> df1 & df2 has 50k common records\n```\n\n**We can save ~30% time by chaining operation**","metadata":{}},{"cell_type":"code","source":"np.random.seed(123)\n\nnrows = 100000\nncols = 50\n\nkey1 = np.array([\"id_\" + str(x) for x in range(nrows//2)]).reshape(nrows//2,1)\nvalues1 = np.random.rand(nrows//2,ncols)\n\nkey2 = np.array([\"id_\" + str(x) for x in range(nrows)]).reshape(nrows,1)\nvalues2 = np.random.rand(nrows,ncols)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:31:59.438917Z","iopub.execute_input":"2022-03-05T13:31:59.439347Z","iopub.status.idle":"2022-03-05T13:31:59.608795Z","shell.execute_reply.started":"2022-03-05T13:31:59.439317Z","shell.execute_reply":"2022-03-05T13:31:59.607866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df1 = pd.DataFrame(np.concatenate([key1, values1],axis =1), columns = [\"id\"] + [\"col_1\" + str(x) for x in range(ncols)])\ndf2 = pd.DataFrame(np.concatenate([key2, values2],axis =1), columns = [\"id\"] + [\"col_2\" + str(x) for x in range(ncols)])\n\n\nprint(f'df1 shape : {df1.shape} | df2 shape : {df2.shape}')","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:32:03.509162Z","iopub.execute_input":"2022-03-05T13:32:03.509442Z","iopub.status.idle":"2022-03-05T13:32:12.360884Z","shell.execute_reply.started":"2022-03-05T13:32:03.509409Z","shell.execute_reply":"2022-03-05T13:32:12.35996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Merging df1 & df2 directly","metadata":{}},{"cell_type":"code","source":"%%timeit\nout = df1.merge(df2, on = \"id\")","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:05:34.204225Z","iopub.execute_input":"2022-03-04T19:05:34.204562Z","iopub.status.idle":"2022-03-04T19:05:49.84301Z","shell.execute_reply.started":"2022-03-04T19:05:34.204527Z","shell.execute_reply":"2022-03-04T19:05:49.841992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Merging df1 & df2 by filtering df2 first","metadata":{"execution":{"iopub.status.busy":"2022-02-21T18:10:53.681252Z","iopub.execute_input":"2022-02-21T18:10:53.681499Z","iopub.status.idle":"2022-02-21T18:10:53.688916Z","shell.execute_reply.started":"2022-02-21T18:10:53.681475Z","shell.execute_reply":"2022-02-21T18:10:53.687281Z"}}},{"cell_type":"code","source":"# common ids - df1 & df2\nreq_id = set(df2.id) & set(df1.id)","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:05:49.844834Z","iopub.execute_input":"2022-03-04T19:05:49.84518Z","iopub.status.idle":"2022-03-04T19:05:49.898335Z","shell.execute_reply.started":"2022-03-04T19:05:49.845137Z","shell.execute_reply":"2022-03-04T19:05:49.897444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\nout = df1.merge(df2[df2.id.isin(req_id)], on = \"id\")","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:05:49.900179Z","iopub.execute_input":"2022-03-04T19:05:49.900426Z","iopub.status.idle":"2022-03-04T19:06:00.942408Z","shell.execute_reply.started":"2022-03-04T19:05:49.900397Z","shell.execute_reply":"2022-03-04T19:06:00.941701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

    Apply

    \n\n- Below mentioned are the good practices while applying functions using **apply** method\n\n- `Some of these might not always give you better performance but will help you in many cases`\n\n\n-------------\n> 1. Calling functions directly without lambda\n\n> 2. Using Raw = True for row/columnwise operations","metadata":{}},{"cell_type":"code","source":"# generating data\ndf = pd.DataFrame({\"A\" : np.random.choice([\"a\",\"abc\",\"prdde\",\"eass\",\"rffd\",\n \"ashcs\",\"rprf\",\"a\",\"\",\"sca\",\"cas\"],1000000)})\n\ndf.head(4)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:44:22.433488Z","iopub.execute_input":"2022-03-05T13:44:22.433808Z","iopub.status.idle":"2022-03-05T13:44:22.610336Z","shell.execute_reply.started":"2022-03-05T13:44:22.433779Z","shell.execute_reply":"2022-03-05T13:44:22.609537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Calling functions directly without lambda\n- Calling functions directly without using lambda gives you some additional performance gains","metadata":{}},{"cell_type":"markdown","source":"- **case1**\n\n> using builtin `len` function","metadata":{}},{"cell_type":"code","source":"%%timeit -r 10 -n 10\ndf.applymap(lambda x:len(x))","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:47:02.32517Z","iopub.execute_input":"2022-03-05T13:47:02.325517Z","iopub.status.idle":"2022-03-05T13:48:01.870472Z","shell.execute_reply.started":"2022-03-05T13:47:02.32549Z","shell.execute_reply":"2022-03-05T13:48:01.869566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit -r 10 -n 10\ndf.applymap(len)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:48:01.872201Z","iopub.execute_input":"2022-03-05T13:48:01.872417Z","iopub.status.idle":"2022-03-05T13:48:53.020301Z","shell.execute_reply.started":"2022-03-05T13:48:01.872392Z","shell.execute_reply":"2022-03-05T13:48:53.019366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- **case2**\n\n> using builtin `str.upper` function","metadata":{}},{"cell_type":"code","source":"%%timeit -r 10 -n 10\ndf.applymap(lambda x:str.upper(x))","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:50:26.649012Z","iopub.execute_input":"2022-03-05T13:50:26.64969Z","iopub.status.idle":"2022-03-05T13:51:06.970519Z","shell.execute_reply.started":"2022-03-05T13:50:26.649653Z","shell.execute_reply":"2022-03-05T13:51:06.969673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit -r 10 -n 10\ndf.applymap(str.upper)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:51:06.972077Z","iopub.execute_input":"2022-03-05T13:51:06.972303Z","iopub.status.idle":"2022-03-05T13:51:35.785269Z","shell.execute_reply.started":"2022-03-05T13:51:06.972275Z","shell.execute_reply":"2022-03-05T13:51:35.784232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- **case3**\n\n> using custom function - `sigmoid`\n\n> Performance gains might look small now but think about 1000 activations with sigmoid!!","metadata":{}},{"cell_type":"code","source":"def sigmoid(x):\n sig = 1/(1 + np.exp(-x))\n return sig","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:52:20.956666Z","iopub.execute_input":"2022-03-05T13:52:20.957131Z","iopub.status.idle":"2022-03-05T13:52:20.961198Z","shell.execute_reply.started":"2022-03-05T13:52:20.957086Z","shell.execute_reply":"2022-03-05T13:52:20.960349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# generating data\nnp.random.seed(123)\ndf1 = pd.DataFrame({\"B\" : np.random.rand(1000000)})\n\ndf1.head(4)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:52:57.811055Z","iopub.execute_input":"2022-03-05T13:52:57.811571Z","iopub.status.idle":"2022-03-05T13:52:57.832649Z","shell.execute_reply.started":"2022-03-05T13:52:57.81152Z","shell.execute_reply":"2022-03-05T13:52:57.832052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit -r 10 -n 10\ndf1.applymap(lambda x:sigmoid(x))","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:53:25.753892Z","iopub.execute_input":"2022-03-05T13:53:25.754343Z","iopub.status.idle":"2022-03-05T13:58:04.367453Z","shell.execute_reply.started":"2022-03-05T13:53:25.754305Z","shell.execute_reply":"2022-03-05T13:58:04.366645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit -r 10 -n 10\ndf1.applymap(sigmoid)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T13:58:04.368987Z","iopub.execute_input":"2022-03-05T13:58:04.369204Z","iopub.status.idle":"2022-03-05T14:02:21.874047Z","shell.execute_reply.started":"2022-03-05T13:58:04.369176Z","shell.execute_reply":"2022-03-05T14:02:21.873145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Using Raw = True for transformations across row/columns\n\nhttps://towardsdatascience.com/how-to-make-your-pandas-operation-100x-faster-81ebcd09265c\n\n-------\n\n**Raw = True in apply** \n\n- Determines if row or column is passed as a `Series or ndarray object`\n- if True, `bypasses` the `overhead` associated with the `Pandas series object` and use simple map objects instead\n- using Raw = True can give us a good boost in performance","metadata":{}},{"cell_type":"code","source":"# generating data\nnp.random.seed(123)\ndf = pd.DataFrame(np.random.rand(10000,100))\n\ndf.head(4) ","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:03:12.855524Z","iopub.execute_input":"2022-03-05T14:03:12.855842Z","iopub.status.idle":"2022-03-05T14:03:12.889787Z","shell.execute_reply.started":"2022-03-05T14:03:12.855809Z","shell.execute_reply":"2022-03-05T14:03:12.8892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- **case 1**\n\n> calc. max across rows","metadata":{}},{"cell_type":"code","source":"%%timeit\ndf.apply(lambda x: max(x) , axis = 1)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:03:30.042318Z","iopub.execute_input":"2022-03-05T14:03:30.042801Z","iopub.status.idle":"2022-03-05T14:03:31.994924Z","shell.execute_reply.started":"2022-03-05T14:03:30.042747Z","shell.execute_reply":"2022-03-05T14:03:31.99399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\ndf.apply(lambda x: max(x) , axis = 1, raw = True)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:03:32.925163Z","iopub.execute_input":"2022-03-05T14:03:32.925436Z","iopub.status.idle":"2022-03-05T14:03:34.532958Z","shell.execute_reply.started":"2022-03-05T14:03:32.925408Z","shell.execute_reply":"2022-03-05T14:03:34.532035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- **case 2**\n\n> performing calc across rows","metadata":{}},{"cell_type":"code","source":"%%timeit\ndf.apply(lambda x: (x[0] + x[2] + x[4])**2/x[6] , axis = 1)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:04:02.528801Z","iopub.execute_input":"2022-03-05T14:04:02.529073Z","iopub.status.idle":"2022-03-05T14:04:04.599653Z","shell.execute_reply.started":"2022-03-05T14:04:02.529044Z","shell.execute_reply":"2022-03-05T14:04:04.59903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\ndf.apply(lambda x: (x[0] + x[2] + x[4])**2/x[6] , axis = 1, raw = True)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:04:04.601074Z","iopub.execute_input":"2022-03-05T14:04:04.601395Z","iopub.status.idle":"2022-03-05T14:04:09.532547Z","shell.execute_reply.started":"2022-03-05T14:04:04.601362Z","shell.execute_reply":"2022-03-05T14:04:09.53167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

    🚀Looping

    \n\n- We love looping, there are better alternatives that can get us **100,000X** performance gains \n- Did you say 100k X gains!đŸ˜„","metadata":{"execution":{"iopub.status.busy":"2022-02-16T18:04:30.686408Z","iopub.execute_input":"2022-02-16T18:04:30.686893Z","iopub.status.idle":"2022-02-16T18:04:30.691313Z","shell.execute_reply.started":"2022-02-16T18:04:30.686842Z","shell.execute_reply":"2022-02-16T18:04:30.690401Z"}}},{"cell_type":"code","source":"def celsius_to_faren(temp_c):\n return (temp_c * 9/5) + 32","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:04:44.507636Z","iopub.execute_input":"2022-03-05T14:04:44.50813Z","iopub.status.idle":"2022-03-05T14:04:44.511321Z","shell.execute_reply.started":"2022-03-05T14:04:44.508085Z","shell.execute_reply":"2022-03-05T14:04:44.510766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Iterrows \n\n- It generally works slightly better than **df.iloc[i][j] within a for loop** \n- Takes **~30** sec to apply `celsius_to_faren` to **100k data**","metadata":{}},{"cell_type":"code","source":"np.random.seed(123)\ndf = pd.DataFrame(np.random.randint(1,100,100000), columns = [\"temp_celsius\"])\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:06:28.877885Z","iopub.execute_input":"2022-03-05T14:06:28.878319Z","iopub.status.idle":"2022-03-05T14:06:28.890274Z","shell.execute_reply.started":"2022-03-05T14:06:28.878281Z","shell.execute_reply":"2022-03-05T14:06:28.88954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nout = []\nfor index, row in df.iterrows():\n temp_f = celsius_to_faren(row)\n out.append(temp_f.values[0])","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:05:41.21814Z","iopub.execute_input":"2022-03-05T14:05:41.218398Z","iopub.status.idle":"2022-03-05T14:06:11.45214Z","shell.execute_reply.started":"2022-03-05T14:05:41.218371Z","shell.execute_reply":"2022-03-05T14:06:11.451355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Itertuples\n\n- Itertuple is **300-400X faster** than iterrows\n\n----------------\n\n Why Itertuple is faster \n> Itertuples make a comparatively less number of function calls than iterrows() and carry much lesser overhead\n\nref1 : https://towardsdatascience.com/heres-the-most-efficient-way-to-iterate-through-your-pandas-dataframe-4dad88ac92ee\n\nref2 : https://medium.com/swlh/why-pandas-itertuples-is-faster-than-iterrows-and-how-to-make-it-even-faster-bc50c0edd30d","metadata":{"execution":{"iopub.status.busy":"2022-02-17T17:14:02.445519Z","iopub.execute_input":"2022-02-17T17:14:02.445795Z","iopub.status.idle":"2022-02-17T17:14:02.452365Z","shell.execute_reply.started":"2022-02-17T17:14:02.445763Z","shell.execute_reply":"2022-02-17T17:14:02.451012Z"}}},{"cell_type":"code","source":"np.random.seed(123)\ndf = pd.DataFrame(np.random.randint(1,100,100000), columns = [\"temp_celsius\"])","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:06:32.125377Z","iopub.execute_input":"2022-03-05T14:06:32.126261Z","iopub.status.idle":"2022-03-05T14:06:32.132228Z","shell.execute_reply.started":"2022-03-05T14:06:32.126219Z","shell.execute_reply":"2022-03-05T14:06:32.131429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\nout = []\nfor row in df.itertuples():\n row_ = row.temp_celsius \n temp_f = celsius_to_faren(row_)\n out.append(temp_f)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:06:40.124129Z","iopub.execute_input":"2022-03-05T14:06:40.124444Z","iopub.status.idle":"2022-03-05T14:06:50.447234Z","shell.execute_reply.started":"2022-03-05T14:06:40.124402Z","shell.execute_reply":"2022-03-05T14:06:50.446443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### apply\n\n> Apply is relatively faster than the above methods because it uses **Cython** backend. However, it can also use **Python** backend depending on the nature of **lambda function** in apply\n> - Apply is **600-800X faster** than iterrows\n\n\n\n---------------\n- ref1 : https://towardsdatascience.com/how-to-make-your-pandas-loop-71-803-times-faster-805030df4f06\n- ref2 : https://realpython.com/fast-flexible-pandas/\n\nCython : https://cython.org/","metadata":{}},{"cell_type":"code","source":"np.random.seed(123)\ndf = pd.DataFrame(np.random.randint(1,100,100000), columns = [\"temp_celsius\"])","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:12:29.627079Z","iopub.execute_input":"2022-03-05T14:12:29.627358Z","iopub.status.idle":"2022-03-05T14:12:29.633445Z","shell.execute_reply.started":"2022-03-05T14:12:29.627331Z","shell.execute_reply":"2022-03-05T14:12:29.632544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\nout = df.temp_celsius.apply(lambda x:celsius_to_faren(x))","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:12:30.794535Z","iopub.execute_input":"2022-03-05T14:12:30.79516Z","iopub.status.idle":"2022-03-05T14:12:35.217282Z","shell.execute_reply.started":"2022-03-05T14:12:30.795121Z","shell.execute_reply":"2022-03-05T14:12:35.216633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Eval\n\n> - Eval is **15K times faster** than iterrows\n\n- Eval is much more than iterator, it can used for doing any transformations on dataframe\n- Eval relies on **Numexpr** package for faster processing\n\n> The point of using eval() for expression evaluation rather than plain Python is two-fold: \n > 1. large DataFrame objects are evaluated more efficiently \n > 2. large arithmetic and boolean expressions are evaluated all at once by the underlying engine (by default numexpr is used for evaluation\n\n\n`You should not use eval() for simple expressions or for expressions involving small DataFrames`\n\n---------------\n\nAwesome blog on **eval**\nhttps://pandas.pydata.org/docs/user_guide/enhancingperf.html#expression-evaluation-via-eval","metadata":{}},{"cell_type":"code","source":"!pip install numexpr -q","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:13:07.624361Z","iopub.execute_input":"2022-03-05T14:13:07.625117Z","iopub.status.idle":"2022-03-05T14:13:18.220974Z","shell.execute_reply.started":"2022-03-05T14:13:07.625077Z","shell.execute_reply":"2022-03-05T14:13:18.219939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.random.seed(123)\ndf = pd.DataFrame(np.random.randint(1,100,100000), columns = [\"temp_celsius\"])","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:13:18.223077Z","iopub.execute_input":"2022-03-05T14:13:18.223329Z","iopub.status.idle":"2022-03-05T14:13:18.230483Z","shell.execute_reply.started":"2022-03-05T14:13:18.223296Z","shell.execute_reply":"2022-03-05T14:13:18.229667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Eval on dataframe ","metadata":{"execution":{"iopub.status.busy":"2022-02-26T13:17:32.918915Z","iopub.execute_input":"2022-02-26T13:17:32.919419Z","iopub.status.idle":"2022-02-26T13:17:32.928594Z","shell.execute_reply.started":"2022-02-26T13:17:32.919382Z","shell.execute_reply":"2022-02-26T13:17:32.926856Z"}}},{"cell_type":"code","source":"%%timeit -r 10 -n 200\nout = df.eval(\"(temp_celsius*9/5) + 32\")","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:13:46.276841Z","iopub.execute_input":"2022-03-05T14:13:46.277149Z","iopub.status.idle":"2022-03-05T14:13:51.400377Z","shell.execute_reply.started":"2022-03-05T14:13:46.277113Z","shell.execute_reply":"2022-03-05T14:13:51.399496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Eval on numpy array ","metadata":{"execution":{"iopub.status.busy":"2022-02-26T13:17:19.746918Z","iopub.execute_input":"2022-02-26T13:17:19.747284Z","iopub.status.idle":"2022-02-26T13:17:19.754156Z","shell.execute_reply.started":"2022-02-26T13:17:19.747247Z","shell.execute_reply":"2022-02-26T13:17:19.752656Z"}}},{"cell_type":"code","source":"vals = df.temp_celsius","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:14:24.422394Z","iopub.execute_input":"2022-03-05T14:14:24.422885Z","iopub.status.idle":"2022-03-05T14:14:24.427662Z","shell.execute_reply.started":"2022-03-05T14:14:24.422831Z","shell.execute_reply":"2022-03-05T14:14:24.4268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit -r 10 -n 200\nout = pd.eval(\"(vals*9/5) + 32\")","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:14:25.327798Z","iopub.execute_input":"2022-03-05T14:14:25.328076Z","iopub.status.idle":"2022-03-05T14:14:29.325041Z","shell.execute_reply.started":"2022-03-05T14:14:25.328048Z","shell.execute_reply":"2022-03-05T14:14:29.324105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Numexpr\n\n- NumExpr is **50 - 80K times faster** than iterrows\n\n- NumExpr is a fast numerical expression evaluator for NumPy\n\n- NumExpr achieves better performance than NumPy because it avoids allocating memory for intermediate results. \n\n- This results in better `cache utilization` and `reduces memory access` in general. Due to this, NumExpr works best with large arrays\n--------------\n\n- eval uses numexpr engine which makes it really fast. if you dont want to try numexpr, just use eval\n\nref: https://github.com/pydata/numexpr#:~:text=NumExpr%20is%20a%20fast%20numerical,the%20same%20calculation%20in%20Python.","metadata":{}},{"cell_type":"code","source":"np.random.seed(123)\ndf = pd.DataFrame(np.random.randint(1,100,100000), columns = [\"temp_celsius\"])","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:14:42.793661Z","iopub.execute_input":"2022-03-05T14:14:42.793948Z","iopub.status.idle":"2022-03-05T14:14:42.799892Z","shell.execute_reply.started":"2022-03-05T14:14:42.793917Z","shell.execute_reply":"2022-03-05T14:14:42.799006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numexpr as ne","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:14:43.827827Z","iopub.execute_input":"2022-03-05T14:14:43.828124Z","iopub.status.idle":"2022-03-05T14:14:43.832257Z","shell.execute_reply.started":"2022-03-05T14:14:43.828091Z","shell.execute_reply":"2022-03-05T14:14:43.831497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vals = df.temp_celsius","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:14:44.56159Z","iopub.execute_input":"2022-03-05T14:14:44.562296Z","iopub.status.idle":"2022-03-05T14:14:44.566163Z","shell.execute_reply.started":"2022-03-05T14:14:44.56226Z","shell.execute_reply":"2022-03-05T14:14:44.565576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\nout = ne.evaluate(\"(vals*9/5) + 32\")","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:14:45.76404Z","iopub.execute_input":"2022-03-05T14:14:45.764356Z","iopub.status.idle":"2022-03-05T14:14:48.6831Z","shell.execute_reply.started":"2022-03-05T14:14:45.764323Z","shell.execute_reply":"2022-03-05T14:14:48.682203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Vectorization on pandas series\n\n- If possible always try to vectorize your code. \n- Vectorization will always give you better results than native pandas functions (like apply,etc)\n------------------\n\nNumexpr beats vectorization but still vectorization is still way faster than eval & other native pandas functions","metadata":{}},{"cell_type":"code","source":"np.random.seed(123)\ndf = pd.DataFrame(np.random.randint(1,100,100000), columns = [\"temp_celsius\"])","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:15:05.763788Z","iopub.execute_input":"2022-03-05T14:15:05.764346Z","iopub.status.idle":"2022-03-05T14:15:05.770147Z","shell.execute_reply.started":"2022-03-05T14:15:05.764303Z","shell.execute_reply":"2022-03-05T14:15:05.769143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\nout = celsius_to_faren(df.temp_celsius)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:15:06.967502Z","iopub.execute_input":"2022-03-05T14:15:06.96821Z","iopub.status.idle":"2022-03-05T14:15:12.873849Z","shell.execute_reply.started":"2022-03-05T14:15:06.968166Z","shell.execute_reply":"2022-03-05T14:15:12.8728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Vectorization on numpy arrays\n\n- Finally we can see that vectorization performed on np arrays is **as good as numexpr**\n- Always try vectorizing (with numpy arrays) to get maximum performance gains","metadata":{"execution":{"iopub.status.busy":"2022-02-17T17:24:55.206483Z","iopub.execute_input":"2022-02-17T17:24:55.206738Z","iopub.status.idle":"2022-02-17T17:24:55.212781Z","shell.execute_reply.started":"2022-02-17T17:24:55.206709Z","shell.execute_reply":"2022-02-17T17:24:55.21171Z"}}},{"cell_type":"code","source":"np.random.seed(123)\ndf = pd.DataFrame(np.random.randint(1,100,100000), columns = [\"temp_celsius\"])","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:16:03.438873Z","iopub.execute_input":"2022-03-05T14:16:03.439155Z","iopub.status.idle":"2022-03-05T14:16:03.445172Z","shell.execute_reply.started":"2022-03-05T14:16:03.439126Z","shell.execute_reply":"2022-03-05T14:16:03.444569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\nout = celsius_to_faren(df.temp_celsius.values)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:16:04.328545Z","iopub.execute_input":"2022-03-05T14:16:04.329369Z","iopub.status.idle":"2022-03-05T14:16:07.209169Z","shell.execute_reply.started":"2022-03-05T14:16:04.32933Z","shell.execute_reply":"2022-03-05T14:16:07.20824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### numba\n\n- Numba is a **just-in-time (JIT) compiler** for Python specifically focused on code that runs in loops over NumPy arrays\n- With numba, Vectorized operations tends to become even faster \n- numba is **100k - 200k** times faster than iterrows\n\n> Numba allows you to write a pure Python function which can be **JIT compiled** to native machine instructions, similar in performance to C, C++ and Fortran, by decorating your function with @jit.\n\n\n---------------\n> The @jit compilation will add overhead to the runtime of the function, so performance benefits may not be realized especially when using small data sets\n\n- https://pandas.pydata.org/docs/user_guide/enhancingperf.html#numba-jit-compilation","metadata":{}},{"cell_type":"code","source":"from numba import njit\n\n@njit\ndef celsius_to_faren_numba(temp_c):\n return (temp_c * 9/5) + 32","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:17:50.793229Z","iopub.execute_input":"2022-03-05T14:17:50.793547Z","iopub.status.idle":"2022-03-05T14:17:51.673194Z","shell.execute_reply.started":"2022-03-05T14:17:50.793513Z","shell.execute_reply":"2022-03-05T14:17:51.67228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.random.seed(123)\ndf = pd.DataFrame(np.random.randint(1,100,100000), columns = [\"temp_celsius\"])","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:17:52.830086Z","iopub.execute_input":"2022-03-05T14:17:52.830373Z","iopub.status.idle":"2022-03-05T14:17:52.836424Z","shell.execute_reply.started":"2022-03-05T14:17:52.830339Z","shell.execute_reply":"2022-03-05T14:17:52.835781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\nout = celsius_to_faren_numba(df.temp_celsius.values)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:17:54.634042Z","iopub.execute_input":"2022-03-05T14:17:54.634592Z","iopub.status.idle":"2022-03-05T14:17:55.41355Z","shell.execute_reply.started":"2022-03-05T14:17:54.634558Z","shell.execute_reply":"2022-03-05T14:17:55.412775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

    đŸ’„Appending

    \n\n- Appending to a list is a faster operation than appending directly to a dataframe because\n\n> 1. Initializing dataframe is a slow process\n\n> 2. Appending to a df involves a lot of `overhead` and `copy/paste operation` in memory\n\n\n-------------\n\n- We are easily getting close to **1000X** performance gain\n\nref: \nhttps://stackoverflow.com/questions/27929472/improve-row-append-performance-on-pandas-dataframes#:~:text=append%20will%20be%20faster%20if,but%20it%20doesn't%20scale.&text=When%20we%20run%20this%20with,see%20much%20more%20dramatic%20results.&text=So%20we%20can%20see%20an,insert%20with%20a%20numpy%20array.","metadata":{}},{"cell_type":"markdown","source":"- Appending directly to a dataframe","metadata":{}},{"cell_type":"code","source":"%%timeit\ndf= pd.DataFrame()\n\nfor x,y in zip(range(1000),range(1000,2000)):\n \n df = df.append({\"A\" : x , \"B\" : y} \n , ignore_index = True)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:18:26.584647Z","iopub.execute_input":"2022-03-05T14:18:26.585096Z","iopub.status.idle":"2022-03-05T14:18:36.464868Z","shell.execute_reply.started":"2022-03-05T14:18:26.585037Z","shell.execute_reply":"2022-03-05T14:18:36.464019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Appending to a list & then creating a df\n- **~1000X gains**","metadata":{}},{"cell_type":"code","source":"%%timeit\nli = []\nfor x,y in zip(range(1000),range(1000,2000)):\n li.append(([x,y]))\n\ndf = pd.DataFrame(li, columns = [\"A\",\"B\"])","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:18:58.56546Z","iopub.execute_input":"2022-03-05T14:18:58.565762Z","iopub.status.idle":"2022-03-05T14:19:09.64251Z","shell.execute_reply.started":"2022-03-05T14:18:58.565729Z","shell.execute_reply":"2022-03-05T14:19:09.641706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

    Initialization

    \n\nWe can also squeeze some time in how we initialize variables (esp list and dictionary)\n\n- li = list() vs li = [] \n- d = dict() vs d = {} \n\n\n-----------------\nFor building pipelines, if we are initializing multiple DS, then these difference can make a big impact \n\n**initlizing without using functions is always faster.eg li = [] is faster than li = list()**","metadata":{}},{"cell_type":"markdown","source":"- **li = list() vs li = []**","metadata":{}},{"cell_type":"code","source":"%%timeit\nli = list()","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:29:05.147862Z","iopub.execute_input":"2022-03-05T14:29:05.148146Z","iopub.status.idle":"2022-03-05T14:29:12.615679Z","shell.execute_reply.started":"2022-03-05T14:29:05.148117Z","shell.execute_reply":"2022-03-05T14:29:12.614814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\nli = []","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:29:12.617699Z","iopub.execute_input":"2022-03-05T14:29:12.618198Z","iopub.status.idle":"2022-03-05T14:29:14.679744Z","shell.execute_reply.started":"2022-03-05T14:29:12.61815Z","shell.execute_reply":"2022-03-05T14:29:14.678885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- **d = dict() vs d = {}**","metadata":{}},{"cell_type":"code","source":"%%timeit\nd = dict()","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:29:14.681224Z","iopub.execute_input":"2022-03-05T14:29:14.681709Z","iopub.status.idle":"2022-03-05T14:29:24.604429Z","shell.execute_reply.started":"2022-03-05T14:29:14.681666Z","shell.execute_reply":"2022-03-05T14:29:24.603578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\nd = {}","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:29:24.606253Z","iopub.execute_input":"2022-03-05T14:29:24.606495Z","iopub.status.idle":"2022-03-05T14:29:27.790893Z","shell.execute_reply.started":"2022-03-05T14:29:24.606467Z","shell.execute_reply":"2022-03-05T14:29:27.790074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

    Misc Tricks

    \n\n- replace vs df.loc[] == val\n- appending to list vs dictionary (incase of k,v type)","metadata":{}},{"cell_type":"markdown","source":"### Replace vs loc\n\n> Replace is generally a better choice over df.loc for imputing/replacing constants in dataframes","metadata":{"execution":{"iopub.status.busy":"2022-02-20T10:43:27.662624Z","iopub.execute_input":"2022-02-20T10:43:27.662904Z","iopub.status.idle":"2022-02-20T10:43:27.66686Z","shell.execute_reply.started":"2022-02-20T10:43:27.662875Z","shell.execute_reply":"2022-02-20T10:43:27.665834Z"}}},{"cell_type":"markdown","source":"- **using df.loc for imputation**","metadata":{}},{"cell_type":"code","source":"np.random.seed(123)\ndf = pd.DataFrame(np.random.choice(list(\"abcdeffghijk\"), (1000000,2)) , columns = [\"A\",\"B\"])","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:24:50.407433Z","iopub.execute_input":"2022-03-04T19:24:50.408377Z","iopub.status.idle":"2022-03-04T19:24:50.476557Z","shell.execute_reply.started":"2022-03-04T19:24:50.408329Z","shell.execute_reply":"2022-03-04T19:24:50.475407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\ndf.loc[df.A == \"c\", \"A\"] = \"replace_val\"","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:24:52.235743Z","iopub.execute_input":"2022-03-04T19:24:52.236222Z","iopub.status.idle":"2022-03-04T19:24:57.676974Z","shell.execute_reply.started":"2022-03-04T19:24:52.236117Z","shell.execute_reply":"2022-03-04T19:24:57.675949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- **using replace for imputation**","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:25:29.09888Z","iopub.execute_input":"2022-03-04T19:25:29.099161Z","iopub.status.idle":"2022-03-04T19:25:29.104957Z","shell.execute_reply.started":"2022-03-04T19:25:29.099132Z","shell.execute_reply":"2022-03-04T19:25:29.104013Z"}}},{"cell_type":"code","source":"np.random.seed(123)\ndf = pd.DataFrame(np.random.choice(list(\"abcdeffghijk\"), (1000000,2)) , columns = [\"A\",\"B\"])","metadata":{"execution":{"iopub.status.busy":"2022-03-05T16:10:30.301384Z","iopub.execute_input":"2022-03-05T16:10:30.30219Z","iopub.status.idle":"2022-03-05T16:10:30.398706Z","shell.execute_reply.started":"2022-03-05T16:10:30.302135Z","shell.execute_reply":"2022-03-05T16:10:30.397758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\ndf.A.replace(\"c\" ,\"replace_val\", inplace = True)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T16:10:42.151299Z","iopub.execute_input":"2022-03-05T16:10:42.151568Z","iopub.status.idle":"2022-03-05T16:10:43.901622Z","shell.execute_reply.started":"2022-03-05T16:10:42.151526Z","shell.execute_reply":"2022-03-05T16:10:43.900497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### df.at vs df.loc\n\n- **df.at** is generally a faster method for indexing pandas dataframe \n- df.at is limited only to indexing value for a specific index. we cant use for indexing a range\n\nhttps://medium.com/bigdatarepublic/advanced-pandas-optimize-speed-and-memory-a654b53be6c2#:~:text=Pandas%20has%20optimized%20operations%20based,merging%20on%20indices%20is%20faster","metadata":{"execution":{"iopub.status.busy":"2022-02-18T13:59:45.672553Z","iopub.execute_input":"2022-02-18T13:59:45.672809Z","iopub.status.idle":"2022-02-18T13:59:45.679901Z","shell.execute_reply.started":"2022-02-18T13:59:45.67278Z","shell.execute_reply":"2022-02-18T13:59:45.678825Z"}}},{"cell_type":"code","source":"%%timeit\ndf.loc[1001,\"A\"]","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:25:50.223876Z","iopub.execute_input":"2022-03-04T19:25:50.224226Z","iopub.status.idle":"2022-03-04T19:26:01.382844Z","shell.execute_reply.started":"2022-03-04T19:25:50.224188Z","shell.execute_reply":"2022-03-04T19:26:01.381628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\ndf.at[1001,\"A\"]","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:26:01.384573Z","iopub.execute_input":"2022-03-04T19:26:01.384813Z","iopub.status.idle":"2022-03-04T19:26:08.793197Z","shell.execute_reply.started":"2022-03-04T19:26:01.384787Z","shell.execute_reply":"2022-03-04T19:26:08.792139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

    Pandas-Block

    \n\n-------------------\n**How does Pandas store a DataFrame under the hood?**\n\n- Pandas groups columns of the `same type` into what is called a **block** \n- A DataFrame is actually stored as `one or more blocks` \n- Using metadata, these blocks are composed into a DataFrame by a **BlockManager**. Thus, only a `block is contiguous in memory`\n----------------\n\n We have 3 Datablocks \n\n> NumericBlock : 2 columns (dtype : int32)\n\n> NumericBlock : 301 columns (dtype : float32)\n\n> ObjectBlock : 1 columns (dtype : object)\n\n\n
    \n\n

    \n

  • Block are important when dealing with mixed dtypes,
    • \n
  • Eg : slicing df.loc[: 1000,[\"string_col\",\"int_col\"]] might be slower than df.loc[: 1000,[\"int_col1\",\"int_col2\"]] because of operations on contiguous data is always faster
    • \n

    \n
    ","metadata":{"execution":{"iopub.status.busy":"2022-02-17T17:32:24.806656Z","iopub.execute_input":"2022-02-17T17:32:24.807435Z","iopub.status.idle":"2022-02-17T17:32:24.817152Z","shell.execute_reply.started":"2022-02-17T17:32:24.807388Z","shell.execute_reply":"2022-02-17T17:32:24.816089Z"}}},{"cell_type":"code","source":"block = df32._data\n\nprint(\"3 datablocks - grouped basis the datatype\")\nblock.blocks","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:26:08.795424Z","iopub.execute_input":"2022-03-04T19:26:08.795761Z","iopub.status.idle":"2022-03-04T19:26:08.803662Z","shell.execute_reply.started":"2022-03-04T19:26:08.795716Z","shell.execute_reply":"2022-03-04T19:26:08.802719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"columns - block mapping\")\nblock.blknos","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:26:08.805084Z","iopub.execute_input":"2022-03-04T19:26:08.806582Z","iopub.status.idle":"2022-03-04T19:26:08.825276Z","shell.execute_reply.started":"2022-03-04T19:26:08.806517Z","shell.execute_reply":"2022-03-04T19:26:08.824231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

    đŸ”„Saving & Loading data

    \n\n- **csv**\n> Comma seperated files\n\n- **Parquet**\n> Apache Parquet is a data storage format designed for efficiency. The reason behind this is the column storage architecture, as it allows you to skip data that isn’t relevant quickly. This way, both queries and aggregations are faster, resulting in hardware savings.\n\n- **feather**\n> Feather is a data format for storing data frames. It’s designed around a simple premise — to push data frames in and out of memory as efficiently as possible. It was initially designed for fast communication between Python and R, but you’re not limited to this use case.\n\nref : https://towardsdatascience.com/stop-using-csvs-for-storage-here-are-the-top-5-alternatives-e3a7c9018de0","metadata":{}},{"cell_type":"markdown","source":"### csv\n\n- **50-70 sec** to save 100K rows (from ubiquant data)\n- **6 - 8 sec** to load 100K rows (from ubiquant data)\n- **500+ MB** size on disk","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"../input/ubiquant-market-prediction/train.csv\",nrows = 100000)\ndf.head(2)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:43:14.098963Z","iopub.execute_input":"2022-03-05T14:43:14.100283Z","iopub.status.idle":"2022-03-05T14:43:22.377375Z","shell.execute_reply.started":"2022-03-05T14:43:14.100221Z","shell.execute_reply":"2022-03-05T14:43:22.376388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- saving","metadata":{}},{"cell_type":"code","source":"%%time\ndf.to_csv(\"df.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:43:33.029312Z","iopub.execute_input":"2022-03-05T14:43:33.029644Z","iopub.status.idle":"2022-03-05T14:44:33.107327Z","shell.execute_reply.started":"2022-03-05T14:43:33.02958Z","shell.execute_reply":"2022-03-05T14:44:33.106301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- loading","metadata":{}},{"cell_type":"code","source":"%%time\ndf_load = pd.read_csv(\"df.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:44:33.109436Z","iopub.execute_input":"2022-03-05T14:44:33.109858Z","iopub.status.idle":"2022-03-05T14:44:40.736028Z","shell.execute_reply.started":"2022-03-05T14:44:33.109814Z","shell.execute_reply":"2022-03-05T14:44:40.735206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- File size on Disk","metadata":{}},{"cell_type":"code","source":"print(\"file size in MB:\" ,os.path.getsize(\"df.csv\")/(1024**2))","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:44:40.737396Z","iopub.execute_input":"2022-03-05T14:44:40.738253Z","iopub.status.idle":"2022-03-05T14:44:40.743628Z","shell.execute_reply.started":"2022-03-05T14:44:40.73821Z","shell.execute_reply":"2022-03-05T14:44:40.742771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Parquet\n\n**~ 2.5 - 4 sec** to **save** 100K rows from ubiquant data **(20X time improvement over csv)**\n\n**~ 0.5 sec** to **load** 100K rows from ubiquant data **(15X time improvement over csv)**\n\n**~ 216 MB** file size **(2.5X improvement over csv)**","metadata":{}},{"cell_type":"markdown","source":"- Saving\n> 2.5 sec saving time","metadata":{}},{"cell_type":"code","source":"%%timeit\n\n# saving as parquet & not using any compression (for apple to apple comparison)\ndf.to_parquet(\"df.parquet\", compression=None)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T14:41:18.752886Z","iopub.execute_input":"2022-03-05T14:41:18.753164Z","iopub.status.idle":"2022-03-05T14:41:38.007225Z","shell.execute_reply.started":"2022-03-05T14:41:18.753136Z","shell.execute_reply":"2022-03-05T14:41:38.006068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Loading\n> 0.5 sec saving time","metadata":{"execution":{"iopub.status.busy":"2022-02-24T18:01:45.512401Z","iopub.execute_input":"2022-02-24T18:01:45.512726Z","iopub.status.idle":"2022-02-24T18:01:45.539826Z","shell.execute_reply.started":"2022-02-24T18:01:45.512691Z","shell.execute_reply":"2022-02-24T18:01:45.538457Z"}}},{"cell_type":"code","source":"pip install fastparquet -q","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:29:10.915156Z","iopub.execute_input":"2022-03-04T19:29:10.915919Z","iopub.status.idle":"2022-03-04T19:29:23.23656Z","shell.execute_reply.started":"2022-03-04T19:29:10.915829Z","shell.execute_reply":"2022-03-04T19:29:23.235095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\ndf = pd.read_parquet(\"df.parquet\", engine=\"fastparquet\")","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:29:55.377787Z","iopub.execute_input":"2022-03-04T19:29:55.378805Z","iopub.status.idle":"2022-03-04T19:30:01.387998Z","shell.execute_reply.started":"2022-03-04T19:29:55.378757Z","shell.execute_reply":"2022-03-04T19:30:01.386599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\ndf = pd.read_parquet(\"df.parquet\", engine=\"pyarrow\")","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:30:01.389809Z","iopub.execute_input":"2022-03-04T19:30:01.391585Z","iopub.status.idle":"2022-03-04T19:30:05.098161Z","shell.execute_reply.started":"2022-03-04T19:30:01.391527Z","shell.execute_reply":"2022-03-04T19:30:05.096401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- File size on Disk","metadata":{}},{"cell_type":"code","source":"print(\"file size in MB:\" ,os.path.getsize(\"df.parquet\")/(1024**2))","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:30:29.908669Z","iopub.execute_input":"2022-03-04T19:30:29.90902Z","iopub.status.idle":"2022-03-04T19:30:29.916378Z","shell.execute_reply.started":"2022-03-04T19:30:29.908984Z","shell.execute_reply":"2022-03-04T19:30:29.915223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### Feather\n\n**App. ~1.3 sec** to **save** 100K rows from ubiquant data **(2X time improvement over Parquet)**\n\n**App. ~0.3 sec** to **load** 100K rows from ubiquant data **(33% time improvement over Parquet)**\n\n**App. ~ 142 MB** file size **(3.5X improvement over csv)**\n\n---------------------\n- https://github.com/wesm/feather\n- https://pythontic.com/pandas/serialization/feather","metadata":{}},{"cell_type":"code","source":"pip install feather-format -q","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:30:35.716074Z","iopub.execute_input":"2022-03-04T19:30:35.716429Z","iopub.status.idle":"2022-03-04T19:30:45.054567Z","shell.execute_reply.started":"2022-03-04T19:30:35.716387Z","shell.execute_reply":"2022-03-04T19:30:45.053577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Saving\n> 1.28 sec saving time","metadata":{}},{"cell_type":"code","source":"%%timeit\ndf.to_feather(\"df.ftr\")","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:30:53.505648Z","iopub.execute_input":"2022-03-04T19:30:53.506253Z","iopub.status.idle":"2022-03-04T19:31:02.241364Z","shell.execute_reply.started":"2022-03-04T19:30:53.506216Z","shell.execute_reply":"2022-03-04T19:31:02.239703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Loading\n> 0.3 sec loading time","metadata":{}},{"cell_type":"code","source":"%%timeit\ndf_load = pd.read_feather('df.ftr')","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:31:02.243802Z","iopub.execute_input":"2022-03-04T19:31:02.244533Z","iopub.status.idle":"2022-03-04T19:31:04.456888Z","shell.execute_reply.started":"2022-03-04T19:31:02.244476Z","shell.execute_reply":"2022-03-04T19:31:04.455654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- File size on Disk","metadata":{}},{"cell_type":"code","source":"print(\"file size in MB:\" ,os.path.getsize(\"df.ftr\")/(1024**2))","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:31:04.459019Z","iopub.execute_input":"2022-03-04T19:31:04.459304Z","iopub.status.idle":"2022-03-04T19:31:04.465758Z","shell.execute_reply.started":"2022-03-04T19:31:04.459252Z","shell.execute_reply":"2022-03-04T19:31:04.464721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

    Chunking

    \n\n\n> Sometimes your data file is so large you can’t load it into memory at all, even with compression. So how do you process it quickly?\n\n> By loading and then processing the data in chunks, you can load only part of the file into memory at any given time. And that means you can process files that don’t fit in memory\n\n``\nChunking works well when the operation you’re performing requires zero or minimal coordination between chunks. For more complicated workflows, you’re better off using another library.\n``\n\nhttps://www.geeksforgeeks.org/monitoring-memory-usage-of-a-running-python-program/\n\n----------------------\n\n\n Value counts implementation using chunking \n> **Peak memory usage : 90% performance gain**","metadata":{}},{"cell_type":"code","source":"import tracemalloc","metadata":{"execution":{"iopub.status.busy":"2022-03-05T15:01:36.369346Z","iopub.execute_input":"2022-03-05T15:01:36.369696Z","iopub.status.idle":"2022-03-05T15:01:36.374029Z","shell.execute_reply.started":"2022-03-05T15:01:36.369659Z","shell.execute_reply":"2022-03-05T15:01:36.373202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# starting the monitoring\ntracemalloc.start()\n\ntps_march = pd.read_csv(\"../input/tabular-playground-series-mar-2022/train.csv\")\nresult = tps_march.direction.value_counts()\n\n# displaying the memory\nprint(\"peak memory usage: \",tracemalloc.get_traced_memory()[1])\n \n# stopping the library\ntracemalloc.stop()","metadata":{"execution":{"iopub.status.busy":"2022-03-05T15:02:15.812392Z","iopub.execute_input":"2022-03-05T15:02:15.812727Z","iopub.status.idle":"2022-03-05T15:02:16.42073Z","shell.execute_reply.started":"2022-03-05T15:02:15.812695Z","shell.execute_reply":"2022-03-05T15:02:16.419659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# starting the monitoring\ntracemalloc.start()\n \n# function call\nresult = None\nfor chunk in pd.read_csv(\"../input/tabular-playground-series-mar-2022/train.csv\" ,chunksize=1000 ):\n df_chunk = chunk[\"direction\"]\n \n value_counter = df_chunk.value_counts()\n if result is None:\n result = value_counter\n else:\n result = result.add(value_counter, fill_value=0)\n\nresult.sort_values(ascending=False, inplace=True)\n \n# displaying the memory\nprint(\"peak memory usage:\",tracemalloc.get_traced_memory()[1])\n \n# stopping the library\ntracemalloc.stop()","metadata":{"execution":{"iopub.status.busy":"2022-03-05T15:00:50.688878Z","iopub.execute_input":"2022-03-05T15:00:50.689216Z","iopub.status.idle":"2022-03-05T15:00:54.281364Z","shell.execute_reply.started":"2022-03-05T15:00:50.689182Z","shell.execute_reply":"2022-03-05T15:00:54.280284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

    List comprehension vs loops

    \n\n- Always prefer using **list comprehensions** over **looping** directly","metadata":{}},{"cell_type":"markdown","source":"- Normal loop","metadata":{}},{"cell_type":"code","source":"%%timeit\nli = []\nfor x in range(100000):\n li.append(x)","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:34:20.716807Z","iopub.execute_input":"2022-03-04T19:34:20.717213Z","iopub.status.idle":"2022-03-04T19:34:26.480464Z","shell.execute_reply.started":"2022-03-04T19:34:20.717173Z","shell.execute_reply":"2022-03-04T19:34:26.479536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\nli = [x for x in range(100000)]","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:34:26.482342Z","iopub.execute_input":"2022-03-04T19:34:26.482707Z","iopub.status.idle":"2022-03-04T19:34:29.776274Z","shell.execute_reply.started":"2022-03-04T19:34:26.482671Z","shell.execute_reply":"2022-03-04T19:34:29.775024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Nested loop","metadata":{}},{"cell_type":"code","source":"%%timeit\nli = []\nfor x in range(100):\n for y in range(100):\n li.append((x,y))","metadata":{"execution":{"iopub.status.busy":"2022-03-04T19:34:48.322834Z","iopub.execute_input":"2022-03-04T19:34:48.323125Z","iopub.status.idle":"2022-03-04T19:34:57.11655Z","shell.execute_reply.started":"2022-03-04T19:34:48.323096Z","shell.execute_reply":"2022-03-04T19:34:57.11538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%timeit\nli = [(x,y) for x in range(100) for y in range(100)]","metadata":{"execution":{"iopub.status.busy":"2022-03-05T15:18:14.894924Z","iopub.execute_input":"2022-03-05T15:18:14.895286Z","iopub.status.idle":"2022-03-05T15:18:22.175147Z","shell.execute_reply.started":"2022-03-05T15:18:14.895251Z","shell.execute_reply":"2022-03-05T15:18:22.174195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

    Thanks

    \n\n\n
    \n\n

  • Please upvote 👍, it will motivate me to create more such content


    • \n

  • Also checkout my one of my most upvoted NB Pandas tips & tricks for great Pandas tricks

    • \n\n
    \n","metadata":{"execution":{"iopub.status.busy":"2022-02-26T18:17:08.90525Z","iopub.execute_input":"2022-02-26T18:17:08.905569Z","iopub.status.idle":"2022-02-26T18:17:08.909934Z","shell.execute_reply.started":"2022-02-26T18:17:08.905537Z","shell.execute_reply":"2022-02-26T18:17:08.908801Z"}}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}