{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport os\nprint(os.listdir(\"../input\"))\n\nfrom sklearn import preprocessing\nimport random","execution_count":134,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true},"cell_type":"markdown","source":"The Avito challenge uses a number of files that, when we are adding features or doing train-test-splits, can consume more memory than some people have available.\n\nIn this notebook, we will be looking at a few things we can do to save memory and increase the overall workflow speed from loading data to creating numerical features."},{"metadata":{"_uuid":"3b8f74310767f7c0e9ecd0480eaa9d63dd200c8b"},"cell_type":"markdown","source":"# Loading files and reducing  file size\nTo test how long certain operations take, we can use the `%%time` magic command at the top of the cell.\n\nLoading a csv file into memory can take some time since pandas needs to infer datatypes for every row."},{"metadata":{"trusted":true,"_uuid":"0c6578c7d4e952121c2ba72e600c34517a8a9c9e"},"cell_type":"code","source":"%%time\ntrain = pd.read_csv(\"../input/train.csv\")","execution_count":135,"outputs":[]},{"metadata":{"_uuid":"5dfe6c0e3982afc28472b62d7054d9d93798c556"},"cell_type":"markdown","source":"Our goal is to reduce the loading time as well as the memory usage while the object is loaded.\n\nWith `.info()` we can check datatypes and the total memory usage. For this, we need to set `memory_usage=True`, which will take a bit longer but return a more accurate size."},{"metadata":{"trusted":true,"_uuid":"e11519756bbe01a8bd7ec8cc7e1695abf044c100"},"cell_type":"code","source":"train.info(memory_usage=\"deep\")","execution_count":136,"outputs":[]},{"metadata":{"_uuid":"28a4b5493b5a41a54266a28d374290cc50d7805d"},"cell_type":"markdown","source":"Each DataFrame and Series also has a `.memory_usage` attribute which shows the memory usage in bytes."},{"metadata":{"trusted":true,"_uuid":"9ef86df6f196d2a9b43ee2708b9f4a4676091a9f"},"cell_type":"code","source":"train.memory_usage(deep=True)","execution_count":137,"outputs":[]},{"metadata":{"_uuid":"3b95fb846a0084edda5ea8dcc2e630828315cfd6"},"cell_type":"markdown","source":"To make this easier to read, we convert all values to megabytes"},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"096cd23b657b65068b88261980664d52cbb133ee"},"cell_type":"code","source":"train.memory_usage(deep=True) * 1e-6","execution_count":138,"outputs":[]},{"metadata":{"_uuid":"1da89e04de39a21dad36a254c71507348ba7bf00"},"cell_type":"markdown","source":"The total memory usage can be seen like this:"},{"metadata":{"trusted":true,"_uuid":"b39eb0638b48817b01ca26afbb090a3966487f0d"},"cell_type":"code","source":"train.memory_usage(deep=True).sum() * 1e-6","execution_count":139,"outputs":[]},{"metadata":{"_uuid":"350691d14d72db3f85351bc3f283d9f03b96a830"},"cell_type":"markdown","source":"We can see that the title and description collumns take up the most space but even region, city, param_1-3 and the activation date use quite a bit of memory.\n\nWhile 2500 MB are not too large, this can easily grow to 10+GB with some column-combinations, text transformations and new objects like those created during train-test-splits, etc."},{"metadata":{"_uuid":"98fbf960b194273d9024b0c08b4b3c8dd5c3ccec"},"cell_type":"markdown","source":"## File/object size reduction with correct datatypes"},{"metadata":{"_uuid":"9c5623968e22f90da6b4a4de88bf92da5667ff3b"},"cell_type":"markdown","source":"One of the easiest ways to reduce sizes is by converting columns to the right datatype. Currently, almost every column uses the `object` type, which is basically strings that are very memory-inefficent.\n\nPandas uses Numpy's datatypes along with a few own additions.. The most important ones are integers, floats, datetime, boolean, string and categorical types. \nMore information on them can be found [here](https://docs.scipy.org/doc/numpy-1.13.0/user/basics.types.html)."},{"metadata":{"_uuid":"efeae571463efe26f3df697cc285a0e00f5dda1b"},"cell_type":"markdown","source":"### datetime\nThe easiest and most well-known is converting dates to a datetime-type."},{"metadata":{"trusted":true,"_uuid":"1a7b8af148d3b99deebd31d98a5e7b832a4e5f8a"},"cell_type":"code","source":"print(\"size before:\", train[\"activation_date\"].memory_usage(deep=True) * 1e-6)\ntrain[\"activation_date\"] = pd.to_datetime(train[\"activation_date\"])\nprint(\"size after: \", train[\"activation_date\"].memory_usage(deep=True) * 1e-6)","execution_count":140,"outputs":[]},{"metadata":{"_uuid":"588d54dc29cf7cdf9d98903a4fa908484a1cf6e1"},"cell_type":"markdown","source":"### categorical\nText columns that have a lot of repeating values, for example `region` and `city` should be converted to categorical columns. This special datatype basically saves all unique values in a dictionary, then places memory-efficient integers in each column and displays the corresponding text-values when using the DataFrame. This is similar to overwriting the column with label-encoded values except for keeping the readability.\n\nSome tools like XGBoost and LightGBM can also use categorical columns directly without converting them to integer labels. Some other libraries have trouble using them, though, and require string or integer values instead."},{"metadata":{"trusted":false,"_uuid":"fbe8da34fe58f18f64786b0064abca6304d50bb3"},"cell_type":"code","source":"print(\"size before:\", train[\"region\"].memory_usage(deep=True) * 1e-6)\ntrain[\"region\"] = train[\"region\"].astype(\"category\")\nprint(\"size after :\", train[\"region\"].memory_usage(deep=True) * 1e-6)","execution_count":34,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"708a50063ce75291fc89cb9d8bc1e9b0a6189fc0"},"cell_type":"code","source":"print(\"size before:\", train[\"city\"].memory_usage(deep=True) * 1e-6)\ntrain[\"city\"] = train[\"city\"].astype(\"category\")\nprint(\"size after :\", train[\"city\"].memory_usage(deep=True) * 1e-6)","execution_count":141,"outputs":[]},{"metadata":{"_uuid":"1280d19d565e50274df0df84567447dd31ec43e5"},"cell_type":"markdown","source":"This only works well for columns with lower cardinality, meaning a lower number of unique values. For columns like `title` which has more than 90% unique values, this does not help.\n\nTo make this easier, we can define a function to help with the categorical conversions and print the before/after object sizes:"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"7f74d9c8270c4b4a4742a06153756c34677bca6a"},"cell_type":"code","source":"def convert_columns_to_catg(df, column_list):\n    for col in column_list:\n        print(\"converting\", col.ljust(30), \"size: \", round(df[col].memory_usage(deep=True)*1e-6,2), end=\"\\t\")\n        df[col] = df[col].astype(\"category\")\n        print(\"->\\t\", round(df[col].memory_usage(deep=True)*1e-6,2))","execution_count":142,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6f9e7e398763eaf48b2e2561039aca95a739f988"},"cell_type":"code","source":"convert_columns_to_catg(train, column_list=[\"param_1\", \"param_2\", \"param_3\", \"parent_category_name\", \"category_name\", \"user_type\"])","execution_count":143,"outputs":[]},{"metadata":{"_uuid":"1d5c7c66583432e2d1b9325a3107e9ecea3b5169"},"cell_type":"markdown","source":"The total memory size has decreased a lot (2500 MB to 1300 MB) now with title and description taking up the majority of the size:"},{"metadata":{"trusted":true,"_uuid":"680a45a11c0e5254f8ae5a8870e21ccaf6c3151c"},"cell_type":"code","source":"print(train.memory_usage(deep=True)*1e-6)\nprint(\"total:\", train.memory_usage(deep=True).sum()*1e-6)","execution_count":144,"outputs":[]},{"metadata":{"_uuid":"14423b47c01873b7595603dfaaa5c19a3a0b1a8d"},"cell_type":"markdown","source":"# Saving objects as pickle-files for faster loading\n\nAfter these basic improvements, we can save the DataFrame as a so called pickle-file, which is the entire Python object saved to your hard drive complete with all datatypes intact. Compared to csv which just stores raw string values, this will be a lot smaller. The resulting file will not have the same size reduction but still be smaller.\n\nUnfortunately we can't save larger files in Kaggle kernels so we will just compare the 1300 MB to the original csv-file:"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"919bbe935a7b87cafc1eb3ce056061172e097d41"},"cell_type":"code","source":"train.to_pickle(\"train.pkl\")","execution_count":145,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"85f8e5860cf893aab9abe3fa444cb51106888303"},"cell_type":"code","source":"# size is shown in bytes again and needs to be converted to megabytes\nprint(\"train.csv:\", os.stat('../input/train.csv').st_size * 1e-6)\nprint(\"train.pkl:\", os.stat('train.pkl').st_size * 1e-6)","execution_count":146,"outputs":[]},{"metadata":{"_uuid":"8adc172fb725ef90ab18cba6b0fe887ba9c6263a"},"cell_type":"markdown","source":"Loading from a .pkl file will also be a lot faster (remember the 20-25s load time for the csv). If you do the datatype improvements once, save everything as a pickle file, then only load from those, you can save time each time you have to load your data again."},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"016db1a705d2205e5bdd0d260f88f8791d7b4500"},"cell_type":"code","source":"del train","execution_count":147,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"f74718b0fd99b04bfb0a4ad0836d2fd208dfb128"},"cell_type":"code","source":"%%time\ntrain = pd.read_pickle(\"train.pkl\")","execution_count":148,"outputs":[]},{"metadata":{"_uuid":"98e092b4adf7f3f64a03b7669e043180aaa1cdf9"},"cell_type":"markdown","source":"When working with a new dataset, I usually create a first notebook to load all relevant files, convert datatypes, save the DataFrame as a pickle file and then only load this in the main feature-enginering notebook. This saves time and memory when I actuall start working with the data ."},{"metadata":{"trusted":true,"_uuid":"541e28d95fcac49d8e5817c8afa0b6bbdd71936e"},"cell_type":"code","source":"# We will remove the file from the Kernels virtual environment.\nos.remove(\"train.pkl\")","execution_count":149,"outputs":[]},{"metadata":{"_uuid":"7fc2600a46d70a6c96b579c0d70b81506cd11123"},"cell_type":"markdown","source":"# Garbage Collector\nPython has a library for controlling it's garbace collector, a system to manage objects in memory and specifically removing unneeded objects.\n\nAfter doing larger transformations,  object creations/deletions or generally anything else that runs for more than a few seconds,, it can help to free up memory by calling the garbage collector directly.\n\nOn your own computer you can use htop, the Windows taks manager and similar tools to monitor the RAM, for demonstration in this notebook we can use `psutil` to show the used RAM."},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"a7854236ff17f8a5588a974d7673177207096285"},"cell_type":"code","source":"import gc\nimport psutil","execution_count":150,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"47f8ef97388241fdb33a5947c7421244dfc6fd7c"},"cell_type":"code","source":"print(\"available RAM:\", psutil.virtual_memory())\n\ngc.collect()\n\nprint(\"available RAM:\", psutil.virtual_memory())","execution_count":151,"outputs":[]},{"metadata":{"_uuid":"2d2644399d70cf1529551453a5b32d7481ba41fd"},"cell_type":"markdown","source":"# Numerical data, label-encoding and high cardinality features"},{"metadata":{"_uuid":"0abd70a0f508562f406c478e473811bf8271885f"},"cell_type":"markdown","source":"While text columns take up the most memory, having a lot of numerical columns can also add up over time if they don't use optimal datatypes.\n\nWe will look at some newly created label columns and explore a way to handle high-cardinality features at the same time."},{"metadata":{"_uuid":"65b9a3915e797b34b447c363dde555ecfbad3ca0"},"cell_type":"markdown","source":"For tree-based models, we usually apply a label-encoding to categorical columns.\n\nFor columns like `region` this works well as even for the region with the least occurences, we still have enough values in the whole dataset to allow the tree to find patterns."},{"metadata":{"trusted":true,"_uuid":"5419890d45dbbfbb833e39566e7fcf54bc54fad3"},"cell_type":"code","source":"train[\"region\"].value_counts()","execution_count":152,"outputs":[]},{"metadata":{"_uuid":"6ef8ec45e9a9607200445b023015418b36dd1e5c"},"cell_type":"markdown","source":"For other columns like the `user_id` there are only a few dozen users with more than hundreds of rows of data and a lot of users with less than a handful of rows, making labels for each unique user too complex for a tree based model."},{"metadata":{"trusted":true,"_uuid":"64e64de4928cde73cacb96335feab88236337cf3"},"cell_type":"code","source":"train[\"user_id\"].value_counts().head(5)","execution_count":153,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1879750ecffde085e9707c6aaa87019e0c87119f"},"cell_type":"code","source":"train[\"user_id\"].value_counts().tail(5)","execution_count":154,"outputs":[]},{"metadata":{"_uuid":"b74d8067638e4ed25c6781ea892e2d22679eec8b"},"cell_type":"markdown","source":"## Single column encoding"},{"metadata":{"_uuid":"de854c6c62218a01d46565ae41a741396eb73c8d"},"cell_type":"markdown","source":"When applying label-encoding to high-cardinality columns like this, you should only apply these to categories with at least 20/50/100 rows of data, depending on your preferences.\n\nTo make this easier, let's define another function to label-encode with a count-threshold:"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"bcf3b605f0ed7736471923d1c25d5db9c112408f"},"cell_type":"code","source":"def create_label_encoding_with_min_count(df, column, min_count=50):\n    column_counts = df.groupby([column])[column].transform(\"count\").astype(int)\n    column_values = np.where(column_counts >= min_count, df[column], \"\")\n    train[column+\"_label\"] = preprocessing.LabelEncoder().fit_transform(column_values)\n    \n    return df[column+\"_label\"]","execution_count":155,"outputs":[]},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"52ecd154bc6c77570bcb16751a780c2f2f3a5a6d"},"cell_type":"code","source":"train[\"user_id_label\"] = create_label_encoding_with_min_count(train, \"user_id\", min_count=50)","execution_count":156,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"22809664a5bb34a6d9c6d6b0fceb14484dc3ec61"},"cell_type":"code","source":"print(\"number of unique users      :\", len(train[\"user_id\"].unique()))\nprint(\"number of unique user labels:\", len(train[\"user_id_label\"].unique()))","execution_count":157,"outputs":[]},{"metadata":{"_uuid":"3f11e7cfb1a05bcd3fccac582bb80ee897904bd7"},"cell_type":"markdown","source":"These 562 values are much easier to use than 700k values, if you would have used them at all."},{"metadata":{"_uuid":"d1daea5b9e48e869c500dbd1e3266e840b880127"},"cell_type":"markdown","source":"## Multi column encoding"},{"metadata":{"_uuid":"c153b674e137566145783ae96623a11911f5c610"},"cell_type":"markdown","source":"In some cases it makes sense to concatenate columns first, then apply a label encoding on the resulting column\n* to help the tree-model to find structure \n* or to avoid labelling categories with the same number although they belong to different parent categories\n\nIn the Avito dataset, this occurs with the region-city hierarchies. Many kernels apply label encodings to both columns individually which causes information loss. There are a number of cities with the same name that belong to different regions:"},{"metadata":{"trusted":true,"_uuid":"9cf5fc324a7e2b37c383aa3428363caea6d8fdb5"},"cell_type":"code","source":"train.loc[train[\"city\"]==\"Светлый\", \"region\"].value_counts().head()","execution_count":158,"outputs":[]},{"metadata":{"_uuid":"d41fe10ec78e82c786a0c91e2275a695699ca2f5"},"cell_type":"markdown","source":"For this, we should concatenate region and city to assign better labels to the individual cities.\n\nWe can't add the values of categorical columns together so we need to use `.apply()` with a join-function which is rather slow and creates a new column with high memory usage."},{"metadata":{"trusted":true,"_uuid":"b8455f5f95f362a9f4d7accd2dc1277ae0a24e17"},"cell_type":"code","source":"%%time\ntrain[\"region_city\"] = train.loc[:, [\"region\", \"city\"]].apply(lambda s: \" \".join(s), axis=1)","execution_count":159,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"451a1002d671f16573ff8d248ee6a4b2f03ecf14"},"cell_type":"code","source":"print(\"unique:\", len(train[\"region_city\"].unique()))\nprint(\"size:  \", train[\"region_city\"].memory_usage(deep=True)*1e-6)","execution_count":160,"outputs":[]},{"metadata":{"_uuid":"dea507ca7d45e60a99ccfcfc02e79741e3e2ee8e"},"cell_type":"markdown","source":"If you want to use multiple column-combinations, the creation alone can take a few minutes.\n\nTo speed this up, we can use the groupby-function and apply unique values for each column-combination. Using the `.transform()` method we usually return an aggregated value back to each row. In this case, we will just return one unique random number for each grouped combination."},{"metadata":{"trusted":true,"_uuid":"a990f387817cb198370b5b152260a23294a06cb0"},"cell_type":"code","source":"%%time\ntrain[\"region_city_2\"] = train.groupby([\"region\", \"city\"])[\"region\"].transform(lambda x: random.random())","execution_count":161,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"862648d4a17b28e678e0db60dfeb9cf9040335b1"},"cell_type":"code","source":"print(\"unique:\", len(train[\"region_city_2\"].unique()))\nprint(\"size:  \", train[\"region_city_2\"].memory_usage(deep=True)*1e-6)","execution_count":162,"outputs":[]},{"metadata":{"_uuid":"e7b80e0c6e4423fa1c81eb25c8a61bd188f9f7d4"},"cell_type":"markdown","source":"This is not only 10 times faster but also creates a much smaller new column, which can then be used in a LabelEncoder()."},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"ade6da24f525836f035d0711b5960b75098e01df"},"cell_type":"code","source":"train[\"region_city_2_label\"] = create_label_encoding_with_min_count(train, \"region_city_2\", min_count=50)","execution_count":163,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e9b8011f9be96e3c1822e6c927c5b59f06caccda"},"cell_type":"code","source":"gc.collect()","execution_count":164,"outputs":[]},{"metadata":{"_uuid":"a55c3cd3c6ff0c970fa1a43cb35a2dafe61505a6"},"cell_type":"markdown","source":"## Numerical data size reduction"},{"metadata":{"_uuid":"8afd0d0bca6da01eaa6255fc6f409f609f27bddc"},"cell_type":"markdown","source":"Let's add a few more numerical columns to see what we can do with their datatypes and sizes."},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"9d084a243973916163e79f100a068e940e25f862"},"cell_type":"code","source":"train[\"description_len\"] = train[\"description\"].fillna(\"\").apply(len)\ntrain[\"description_count_words\"] = train[\"description\"].fillna(\"\").apply(lambda s: len(s.split(\" \")))","execution_count":165,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"833bfff6c876157b98d9008764832d65761452a2"},"cell_type":"code","source":"train.loc[:, [\"user_id_label\", \"region_city_2_label\", \"description_len\", \"description_count_words\"]\n         ].info()","execution_count":166,"outputs":[]},{"metadata":{"_uuid":"21e52822b7fc687757a2bd422287aa40a9142e87"},"cell_type":"markdown","source":"We can see that pandas created all new columns as [int64](https://docs.scipy.org/doc/numpy-1.13.0/user/basics.types.html), meaning they can store values between -9223372036854775808 to 9223372036854775807.\n\nLooking at the min and max values of our new columns, it becomes obvious we can use a smaller datatype as we use smaller min-max-ranges:"},{"metadata":{"trusted":true,"_uuid":"c00407abfb3f76c7366a862c78c83bb1d1c1aee0"},"cell_type":"code","source":"for col in [\"user_id_label\", \"region_city_2_label\", \"description_len\", \"description_count_words\"]:\n    print(col.ljust(30), \"min:\", train[col].min(), \"  max:\", train[col].max())","execution_count":167,"outputs":[]},{"metadata":{"_uuid":"c05119932fff97975dbb7b99c8a2257d7703ea8b"},"cell_type":"markdown","source":"The memory usages of these columns are comparitevly small for now. If you use the train and test set togther, and create a few dozen of integer columns, this can still quickly add up to hundreds of megabytes of extra RAM usage."},{"metadata":{"trusted":true,"_uuid":"ecdc4ae6f7193dd85670394506a8de82c71c1a48"},"cell_type":"code","source":"train.loc[:, [\"user_id_label\", \"region_city_2_label\", \"description_len\", \"description_count_words\"]\n         ].memory_usage(deep=True)*1e-6","execution_count":168,"outputs":[]},{"metadata":{"_uuid":"f49ff114a9a0a0cf0909b36a64ca17e40c8f679b"},"cell_type":"markdown","source":"Pandas offers the `.to_numeric` method to convert columns to numeric values and at the same time downcast them to the most efficient datatype for the given value range."},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"e24da8f190b91dfd620c11e48c622bc7093df894"},"cell_type":"code","source":"train[\"user_id_label\"] = pd.to_numeric(train[\"user_id_label\"], downcast=\"integer\")","execution_count":169,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a72c7726c4667c77217de49dc52bd37e13ae18ac"},"cell_type":"code","source":"train.loc[:, [\"user_id_label\", \"region_city_2_label\", \"description_len\", \"description_count_words\"]\n         ].info()\n# note the int16 here","execution_count":170,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2998199b9464e341952c67a9b7729131918faf48"},"cell_type":"code","source":"train.loc[:, [\"user_id_label\", \"region_city_2_label\", \"description_len\", \"description_count_words\"]\n         ].memory_usage(deep=True)*1e-6","execution_count":171,"outputs":[]},{"metadata":{"_uuid":"9faafba3504f8004e769cc80752d9cb085b29931"},"cell_type":"markdown","source":"For dozens or hundreds of integer columns, a 50-75% size reduction can help a lot in the end.\n\nTo downcast all available integer columns in our DataFrame, we can also use this function:"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"53a045d7f81d4e39f98ecaed9703d211c66c4108"},"cell_type":"code","source":"def downcast_df_int_columns(df):\n    list_of_columns = list(df.select_dtypes(include=[\"int32\", \"int64\"]).columns)\n        \n    if len(list_of_columns)>=1:\n        max_string_length = max([len(col) for col in list_of_columns]) # finds max string length for better status printing\n        print(\"downcasting integers for:\", list_of_columns, \"\\n\")\n        \n        for col in list_of_columns:\n            print(\"reduced memory usage for:  \", col.ljust(max_string_length+2)[:max_string_length+2],\n                  \"from\", str(round(df[col].memory_usage(deep=True)*1e-6,2)).rjust(8), \"to\", end=\" \")\n            df[col] = pd.to_numeric(df[col], downcast=\"integer\")\n            print(str(round(df[col].memory_usage(deep=True)*1e-6,2)).rjust(8))\n    else:\n        print(\"no columns to downcast\")\n    \n    gc.collect()\n    \n    print(\"done\")","execution_count":174,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e75e8e98457fe86bf64ee17a5b46df746a4c310c"},"cell_type":"code","source":"downcast_df_int_columns(train)","execution_count":173,"outputs":[]},{"metadata":{"collapsed":true,"_uuid":"29117b9c373dc2d1910784368b42ceb7a05f720d"},"cell_type":"markdown","source":"You can also downcast float values from float64 to float32, but this can cause data loss when working with a lot of decimal places.\n\nEven if your dataset starts out smaller and you don't have any memory issues, it can come in handy if you convert texts to categories, downcast integer values and in general try to use approaches that avoid handling a lot of string-values."},{"metadata":{"_uuid":"2323351abb498a184e9c9c5af7e7b1c34f8f2ca1"},"cell_type":"markdown","source":"**Do you have other tips and tricks to work with larger datasets?**"},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"9badc186a1027ca05c2542b6709a478faaa6786d"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}