{"cells":[{"metadata":{"_uuid":"8e211ac22052d6adaa75d872f15740749b4c5acf"},"cell_type":"markdown","source":"Hi people !\n\nThe aim of this kernel is to show a very simple method **to reduce our dataframe size by 55%**. The principle is to convert numeric data from float64 ou int64 to more memory efficient types.\n\nI'll use a sample of 59,633,310 rows as an example (every clicks from 2017-11-07 00:00:00 to 2017-11-07 23:59:59)."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true,"collapsed":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport gc","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"ce8af37aba6ca60ec837923b72ce83f88c5bc1ad"},"cell_type":"markdown","source":"**Importation of a entire day data :**"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# Rows importation\ndf = pd.read_csv('../input/train.csv', skiprows = 9308568, nrows = 59633310)\n\n# Header importation\nheader = pd.read_csv('../input/train.csv', nrows = 0) \ndf.columns = header.columns\ndf\n\n# Cleaning\ndel header\ngc.collect()\n\n# And check his size        \nprint(\"The created dataframe contains\", df.shape[0], \"rows.\")   ","execution_count":6,"outputs":[]},{"metadata":{"_uuid":"5b53ed2214d81d224eeeb8ce162aa2bc5c6d7bc1"},"cell_type":"markdown","source":"**The magical function :**"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"c3872b618cd07279a0d368a4ceef0f899e8a98eb"},"cell_type":"code","source":"total_before_opti = sum(df.memory_usage())\n\n# Type's conversions\ndef conversion (var):\n    if df[var].dtype != object:\n        maxi = df[var].max()\n        if maxi < 255:\n            df[var] = df[var].astype(np.uint8)\n            print(var,\"converted to uint8\")\n        elif maxi < 65535:\n            df[var] = df[var].astype(np.uint16)\n            print(var,\"converted to uint16\")\n        elif maxi < 4294967295:\n            df[var] = df[var].astype(np.uint32)\n            print(var,\"converted to uint32\")\n        else:\n            df[var] = df[var].astype(np.uint64)\n            print(var,\"converted to uint64\")","execution_count":7,"outputs":[]},{"metadata":{"_uuid":"75cc5c38da8db471d227f272b70214d44e9e34a4"},"cell_type":"markdown","source":"**Function's launch :**"},{"metadata":{"trusted":true,"_uuid":"5bef120395cb8dd0de99fbce9a31538b983d6f20"},"cell_type":"code","source":"for v in ['ip', 'app', 'device','os', 'channel', 'is_attributed'] :\n    conversion(v)","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"19bcea35093f9df070192445f592de64597880a3"},"cell_type":"markdown","source":"**We can now print the results :**"},{"metadata":{"trusted":true,"_uuid":"5beae2e09c0759b758c99ab332df4bcc859b9e62"},"cell_type":"code","source":"print(\"Memory usage before optimization :\", str(round(total_before_opti/1000000000,2))+'GB')\nprint(\"Memory usage after optimization :\", str(round(sum(df.memory_usage())/1000000000,2))+'GB')\nprint(\"We reduced the dataframe size by\",str(round(((total_before_opti - sum(df.memory_usage())) /total_before_opti)*100,2))+'%')","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"2ba9a8b029ccb25fdf4592332911ee2bce17c386"},"cell_type":"markdown","source":"**You can find my EDA here **(which really need so visibility ! :p ) : https://www.kaggle.com/valentinw/eda-of-the-whole-dataset\n\nThanks to this kernel :  https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65/notebook\n\nThanks for your time !"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}