{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"The goal of this notebook is to offer a set of tips and tricks to load the tabular dataset and manipulate it easily and quickly.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport datetime","metadata":{"execution":{"iopub.status.busy":"2022-02-18T21:42:05.734663Z","iopub.execute_input":"2022-02-18T21:42:05.735557Z","iopub.status.idle":"2022-02-18T21:42:05.758862Z","shell.execute_reply.started":"2022-02-18T21:42:05.735433Z","shell.execute_reply":"2022-02-18T21:42:05.758236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading The Data Files Efficiently :\n## Using Parquet Format\n\nUsing another file format than CSV can help you load data files faster than the classic read_csv. there are several formats to handle big datasets :\n- HDF\n- Feather\n- Parquet\n- Pickle (used to store any serilizable python object)\n\nIn this case we will use the parquet format, first let's load the transactions csv file which is over 3GB in size : ","metadata":{}},{"cell_type":"code","source":"%%time\ntransactions = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv')","metadata":{"execution":{"iopub.status.busy":"2022-02-18T21:42:54.218934Z","iopub.execute_input":"2022-02-18T21:42:54.219767Z","iopub.status.idle":"2022-02-18T21:43:57.481670Z","shell.execute_reply.started":"2022-02-18T21:42:54.219722Z","shell.execute_reply":"2022-02-18T21:43:57.480401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This operation takes around 39 seconds on Kaggle Kernel and 3 Gigs of RAM.","metadata":{}},{"cell_type":"code","source":"# Let's convert back to parquet and load it in parquet format\ntransactions.to_parquet('transactions.parquet')","metadata":{"execution":{"iopub.status.busy":"2022-02-18T21:45:08.219985Z","iopub.execute_input":"2022-02-18T21:45:08.220994Z","iopub.status.idle":"2022-02-18T21:45:24.211956Z","shell.execute_reply.started":"2022-02-18T21:45:08.220934Z","shell.execute_reply":"2022-02-18T21:45:24.211102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time \ntransactions_parquet = pd.read_parquet('./transactions.parquet')\n","metadata":{"execution":{"iopub.status.busy":"2022-02-18T21:46:22.018340Z","iopub.execute_input":"2022-02-18T21:46:22.018648Z","iopub.status.idle":"2022-02-18T21:46:33.093112Z","shell.execute_reply.started":"2022-02-18T21:46:22.018615Z","shell.execute_reply":"2022-02-18T21:46:33.092231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This time the file takes only 9 secondes to load and the file size is only 786 MB rather than 3.4 GB, this is an improvement of over 200% ! Amazing !","metadata":{}},{"cell_type":"code","source":"del transactions","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Using another library : \n\nYou can use other libraries such as cuDF which uses GPU to load dataframes faster (but I think it's an unnecessary flex in this case and just a waste of computing ressources because the data file is still not that huge (not over 20 GB))\n\nYou can also use a distributed library such as Dask or PySpark if you can set up a cluster on Amazon EMR or GCP Dataproc.","metadata":{}},{"cell_type":"markdown","source":"# Converting dates to datetime in a blink ! \n\nHere is a neat technique that help you convert the `t_dat` column in transactions data to python datetimes quickly.\n\nSure `pd.to_datetime` is clean and easy to use but it can be a bit slow sometimes depending on your machine and with 31 million rows of transactions that's not our best bet.","metadata":{}},{"cell_type":"code","source":"def convert_to_date(s):\n    \"\"\"\n    Memoization technique - very fast conversion to pure python dates\n    \"\"\"\n    dates = {date:datetime.datetime.strptime(date,'%Y-%m-%d') for date in s.unique()}\n    return s.map(dates)","metadata":{"execution":{"iopub.status.busy":"2022-02-18T21:53:52.610951Z","iopub.execute_input":"2022-02-18T21:53:52.611802Z","iopub.status.idle":"2022-02-18T21:53:52.617952Z","shell.execute_reply.started":"2022-02-18T21:53:52.611763Z","shell.execute_reply":"2022-02-18T21:53:52.616726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nconvert_to_date(transactions_parquet.t_dat)","metadata":{"execution":{"iopub.status.busy":"2022-02-18T21:54:28.568089Z","iopub.execute_input":"2022-02-18T21:54:28.568364Z","iopub.status.idle":"2022-02-18T21:54:32.255515Z","shell.execute_reply.started":"2022-02-18T21:54:28.568336Z","shell.execute_reply":"2022-02-18T21:54:32.254647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Use Pandas Vectorization :\n\nInstead of looping through the dataframe with `iterrows()` which will take an eternity, you should use vectorization of Pandas whenever possible. \n\nLet's say you've built a function to compute a mathematical function on a column, just to show how fast vectorization is, we will take a useless function $f(x)$ to prove this point and apply it on the price column.\n\n$$ f(x) = cos(x^2) $$","metadata":{}},{"cell_type":"code","source":"def f(x):\n    return np.cos(x**2)","metadata":{"execution":{"iopub.status.busy":"2022-02-18T22:00:11.879941Z","iopub.execute_input":"2022-02-18T22:00:11.880253Z","iopub.status.idle":"2022-02-18T22:00:11.884579Z","shell.execute_reply.started":"2022-02-18T22:00:11.880212Z","shell.execute_reply":"2022-02-18T22:00:11.883790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntransactions_parquet.price.apply(lambda x: f(x))\n","metadata":{"execution":{"iopub.status.busy":"2022-02-18T22:02:57.491857Z","iopub.execute_input":"2022-02-18T22:02:57.492419Z","iopub.status.idle":"2022-02-18T22:03:40.307312Z","shell.execute_reply.started":"2022-02-18T22:02:57.492371Z","shell.execute_reply":"2022-02-18T22:03:40.306222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This took 41s :o ,Now let's with pandas vectorization super power","metadata":{}},{"cell_type":"code","source":"%%time\nf(transactions_parquet.price)","metadata":{"execution":{"iopub.status.busy":"2022-02-18T22:02:44.752830Z","iopub.execute_input":"2022-02-18T22:02:44.753108Z","iopub.status.idle":"2022-02-18T22:02:45.440624Z","shell.execute_reply.started":"2022-02-18T22:02:44.753076Z","shell.execute_reply":"2022-02-18T22:02:45.439738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is amazing, we did the exact same thing **13x times faster** ! \n\nYou can even get faster results with numpy vectorization because pandas dataframe have some overhead.","metadata":{}},{"cell_type":"code","source":"%%time\nf(transactions_parquet.price.values)","metadata":{"execution":{"iopub.status.busy":"2022-02-18T22:05:28.326320Z","iopub.execute_input":"2022-02-18T22:05:28.327242Z","iopub.status.idle":"2022-02-18T22:05:28.817401Z","shell.execute_reply.started":"2022-02-18T22:05:28.327172Z","shell.execute_reply":"2022-02-18T22:05:28.816523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Did you see that ! It takes less than half a second to do the same thing with numpy vectorization and 39 seconds with apply ! **that's an improvement of 102%**","metadata":{}},{"cell_type":"markdown","source":"# Final Tip : Del and gc.collect()","metadata":{}},{"cell_type":"markdown","source":"This tip is a bit obvious but it's very easy to forget : always deleted objects that you're no longer using with the`del` keyword.\n\nAlso try to look at how the garbage collector works in Python and try `gc.collect()` ","metadata":{}},{"cell_type":"markdown","source":"# Conclusion\n\nIn this notebook we saw that by using some simple techniques available in pandas, numpy and python we can really improve loading and opration speed and memory to a great extent !\n\nThere are several other techniques, if you have some other techniques please leave a comment and I will try to add it with a credit in this notebook.\n\nFinally if this notebook was helpful please give it an upvote ! ","metadata":{}}]}