{"nbformat_minor": 1, "metadata": {"kernelspec": {"language": "python", "display_name": "Python 3", "name": "python3"}, "language_info": {"nbconvert_exporter": "python", "mimetype": "text/x-python", "name": "python", "codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "pygments_lexer": "ipython3", "version": "3.6.1"}}, "cells": [{"source": ["# Introduction\n", "This Notebook will try to show my approach no memory reduction, by taking into account [Jeru666's notebook](http://https://www.kaggle.com/jeru666/memory-reduction-and-some-data-insights), where he did a nice job on reducing memory. <br>\n", "Improvements where made on the fact that the types for each column in the dataset, are directly set to the read_csv method.<br>\n", "Please comment on anythong you feel like!"], "metadata": {"_uuid": "936f66e04a45271616ca596be918ebc297069305", "_cell_guid": "bf6b2325-6a65-4b0c-9c7d-26ce686a4598"}, "cell_type": "markdown"}, {"source": ["# First Steps"], "metadata": {"_uuid": "b6200f84f571cd8211daf37d84ca441178449d1d", "_cell_guid": "ec35a83b-8a80-403d-a146-7481999d1762"}, "cell_type": "markdown"}, {"source": ["## Library Imports"], "metadata": {"_uuid": "09fdda5864792ac4fc6b510c78ca694c52729c7e", "_cell_guid": "835d1bb9-d364-4f39-be38-cd2f10d7aa9b"}, "cell_type": "markdown"}, {"source": ["import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "from sklearn import svm # Support vector Machines\n", "from sklearn import neighbors # K Nearest Neighbors\n", "from sklearn.model_selection import cross_val_score # Cross validation\n", "\n", "# Any results you write to the current directory are saved as output."], "metadata": {"_uuid": "603f1d090ab20c092971425972ed383b4a6adc4b", "_cell_guid": "7d678174-a771-4b14-b180-78e306ef7d20", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code"}, {"source": ["## Notebook Parameters\n", "Here we will define some parameters and constants that we will use through the rest of the notebook."], "metadata": {"_uuid": "eef5334adb1e72e92912374caaf143a8a727b335", "_cell_guid": "34857ba3-3007-423d-93f7-68aa47df9c52"}, "cell_type": "markdown"}, {"source": ["# Stores the input path to where the csv files are located\n", "INPUT_PATH = '../input/'"], "metadata": {"_uuid": "acb365957c94d55900bbe94ff341acf776ac9e63", "_cell_guid": "62a014a8-1819-40b3-895e-66fc8412e6c3", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code"}, {"source": ["## Read Dataframes\n", "Main dataframes will be stores on the **dict_dfs** dictionary. <br>\n", "**user_logs** dataset seems to be too big for the kernels so we will skip loading it for now (but not I'm giving up on finding a solution)."], "metadata": {"_uuid": "088c3e38963c6846167db058241f62b5467e11c7", "_cell_guid": "4d63151a-fa82-4851-8e98-05e1b3e6d30c"}, "cell_type": "markdown"}, {"source": ["# Initialize the dataframes dictonary\n", "dict_dfs = {}\n", "\n", "# Read the csvs into the dictonary\n", "dict_dfs['members'] = pd.read_csv(INPUT_PATH + 'members.csv', parse_dates=['registration_init_time','expiration_date'], dtype={'city': np.int8, 'bd': np.int16, 'registered_via': np.int8})\n", "dict_dfs['train'] = pd.read_csv(INPUT_PATH + 'train.csv', dtype={'is_churn' : np.int8})\n", "dict_dfs['predict'] = pd.read_csv(INPUT_PATH + 'sample_submission_zero.csv', dtype={'is_churn' : np.int8})\n", "dict_dfs['transactions'] = pd.read_csv(INPUT_PATH + 'transactions.csv', parse_dates=['transaction_date','membership_expire_date'], dtype={'payment_method_id': np.int8, 'payment_plan_days': np.int16, 'plan_list_price': np.int16, 'actual_amount_paid': np.int16, 'is_auto_renew': np.int8, 'is_cancel': np.int8})\n", "#dict_dfs['user_logs'] = pd.read_csv(INPUT_PATH + 'user_logs.csv') # TOO MUCH MEMORY\n"], "metadata": {"_uuid": "050ab2667af228f2918352c5a2f9dee31a0ad01e", "_cell_guid": "132f36f0-8f4c-48a7-a80a-4aa1e129b2b2", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code"}, {"source": ["## Memory usage reduction\n", "With some help from [Jeru666's notebook](http://https://www.kaggle.com/jeru666/memory-reduction-and-some-data-insights) we will reduce the memory used on the dataframes, because they are too big for the kernels."], "metadata": {"_uuid": "ae4a0b027e33120d1b0ca106191116f0765886d6", "_cell_guid": "23199667-3625-4af6-a41b-d148f34cf2a2"}, "cell_type": "markdown"}, {"source": ["def get_memory_usage_datafame():\n", "    \"Returns a dataframe with the memory usage of each dataframe.\"\n", "    \n", "    # Dataframe to store the memory usage\n", "    df_memory_usage = pd.DataFrame(columns=['DataFrame','Memory MB','Records'])\n", "\n", "    # For each dataframe\n", "    for key, value in dict_dfs.items():\n", "    \n", "        # Get the memory usage of the dataframe\n", "        mem_usage = value.memory_usage(index=True).sum()\n", "        mem_usage = mem_usage / 1024**2\n", "    \n", "        # Append the memory usage to the result dataframe\n", "        df_memory_usage = df_memory_usage.append({'DataFrame': key, 'Memory MB': mem_usage,'Records': len(value)}, ignore_index=True)\n", "    \n", "    # return the dataframe\n", "    return df_memory_usage"], "metadata": {"_uuid": "82a98d033112fbda56b0fc9d0c50a8fc87d2657f", "_cell_guid": "746ccb7a-ac3b-43ab-8536-4669af49df72", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code"}, {"source": ["Let's see the initial memory usage"], "metadata": {"_uuid": "4b53ac1b2a3126050bf017acaf6ec16be7ea3460", "_cell_guid": "a390c350-b221-40be-803e-787a518f4e98"}, "cell_type": "markdown"}, {"source": ["get_memory_usage_datafame()"], "metadata": {"_uuid": "2a47c90b3a8c37ff25355b27d3403e19fe502b0f", "_cell_guid": "07c1dafc-9829-4f22-95c9-9538320ff13c"}, "outputs": [], "execution_count": null, "cell_type": "code"}, {"source": ["### members.csv\n", "Release memory from the members dataframe"], "metadata": {"_uuid": "f9dcc498e74c35b4a23ef721e7611b25db6f113c", "_cell_guid": "ce3e75ed-4ccf-448c-a68e-976be0be2d7f"}, "cell_type": "markdown"}, {"source": ["# In case we run the cell twice\n", "if 'registration_init_time' in dict_dfs['members'].columns:\n", "    \n", "    # Split registration init date into 3 columns\n", "    dict_dfs['members']['registration_init_year'] = dict_dfs['members'].registration_init_time.dt.year.astype(np.int16)\n", "    dict_dfs['members']['registration_init_month'] = dict_dfs['members'].registration_init_time.dt.month.astype(np.int8)\n", "    dict_dfs['members']['registration_init_date'] = dict_dfs['members'].registration_init_time.dt.day.astype(np.int8)\n", "    \n", "    # Drop the registration init date \n", "    dict_dfs['members'] = dict_dfs['members'].drop('registration_init_time', axis=1)\n", "\n", "# In case we run the cell twice\n", "if 'expiration_date' in dict_dfs['members'].columns:\n", "    \n", "    # Split the expiration date into 3 columns\n", "    dict_dfs['members']['expiration_year'] = dict_dfs['members'].expiration_date.dt.year.astype(np.int16)\n", "    dict_dfs['members']['expiration_month'] = dict_dfs['members'].expiration_date.dt.month.astype(np.int8)\n", "    dict_dfs['members']['expiration_date'] = dict_dfs['members'].expiration_date.dt.day.astype(np.int8)\n", "    \n", "    # Drop the expiration date \n", "    dict_dfs['members'] = dict_dfs['members'].drop('expiration_date', axis=1)"], "metadata": {"_uuid": "015a0d2a036cb5f74203b0a9808e27065bf94bda", "_cell_guid": "68723fcc-8e0d-4f75-b8e9-e14f1a83b998", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code"}, {"source": ["Let's see the change"], "metadata": {"_uuid": "92785e4da760684a0f3fa8bde2b764a5bf829cae", "_cell_guid": "64e19990-f871-4919-9480-711b2a4da674"}, "cell_type": "markdown"}, {"source": ["get_memory_usage_datafame()"], "metadata": {"_uuid": "6d105dc3c2efe0dc393fec951b766138fa3a88c8", "_cell_guid": "921dbdc8-620b-4e04-8ec4-a75cf6e14020"}, "outputs": [], "execution_count": null, "cell_type": "code"}, {"source": ["### train.csv and sample_submission_zero.csv\n", "By setting the dtype on the read_csv method, we took care of the columns that could be improved in memory."], "metadata": {"_uuid": "aa9f6386f830b4c75a282facea350d566e28fdf5", "_cell_guid": "4c32bb6b-462d-41b9-8d48-7d1c3f3641be"}, "cell_type": "markdown"}, {"source": ["### transactions.csv\n", "Release memory from the transactions dataframe, by separating datetime columns into int columns and removing the initial date column"], "metadata": {"_uuid": "7aa4928d56040df2f01fac973cacffc7ec749f80", "_cell_guid": "bfb9ab09-5005-4cf2-8d49-9e04298fbdf1"}, "cell_type": "markdown"}, {"source": ["# In case we run the cell more than once\n", "if 'membership_expire_date' in dict_dfs['transactions']:\n", "    # Split membership_expire_date into 3 columns\n", "    dict_dfs['transactions']['membership_expire_year'] = dict_dfs['transactions'].membership_expire_date.dt.year.astype(np.int16)\n", "    dict_dfs['transactions']['membership_expire_month'] = dict_dfs['transactions'].membership_expire_date.dt.month.astype(np.int8)\n", "    dict_dfs['transactions']['membership_expire_date'] = dict_dfs['transactions'].membership_expire_date.dt.day.astype(np.int8)\n", "    \n", "    # Drop the registration init date \n", "    dict_dfs['transactions'] = dict_dfs['transactions'].drop('membership_expire_date', axis=1)\n", "    \n", "# In case we run the cell more than once\n", "if 'transaction_date' in dict_dfs['transactions']:\n", "    # Split membership_expire_date into 3 columns\n", "    dict_dfs['transactions']['transaction_year'] = dict_dfs['transactions'].transaction_date.dt.year.astype(np.int16)\n", "    dict_dfs['transactions']['transaction_month'] = dict_dfs['transactions'].transaction_date.dt.month.astype(np.int8)\n", "    dict_dfs['transactions']['transaction_date'] = dict_dfs['transactions'].transaction_date.dt.day.astype(np.int8)\n", "    \n", "    # Drop the registration init date \n", "    dict_dfs['transactions'] = dict_dfs['transactions'].drop('transaction_date', axis=1)\n", "    "], "metadata": {"_uuid": "9ee5e4cbbcdc6323aba3e03a329eca3e1768f1f1", "_cell_guid": "17eddfb4-305c-436a-a6dc-2a59682654d6", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code"}, {"source": ["get_memory_usage_datafame()"], "metadata": {"_uuid": "72ea5e219664dad4d27d7c5f77f25266d72e375e", "_cell_guid": "885ca686-ae2f-4131-9a01-a7fde1aef069"}, "outputs": [], "execution_count": null, "cell_type": "code"}, {"source": ["## Joining Datasets\n", "Looking at [the1owl's notebook](https://www.kaggle.com/the1owl/regressing-during-insomnia-0-21496/notebook) we grab the idea of merging dataframes"], "metadata": {"_uuid": "be3f8290ada0702844505c3b22c5f40102378d5a", "_cell_guid": "b16d4a98-323b-494f-abdb-9cd293bd346f"}, "cell_type": "markdown"}, {"source": ["# Merge members to the train and test dataframes\n", "dict_dfs['train'] = pd.merge(dict_dfs['train'], dict_dfs['members'], on='msno')\n", "dict_dfs['predict'] = pd.merge(dict_dfs['predict'], dict_dfs['members'], on='msno')"], "metadata": {"_uuid": "b76b6456efcc79414c035fbd82ccd234228f3a66", "_cell_guid": "cc36dc09-2ac6-4a41-9135-2b9e0ea04299", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code"}, {"source": ["# Data Munging\n", "Let's do stuff with the data. Clean Null values, prepare categorical features and some other magic stuff."], "metadata": {"_uuid": "96ee54600b8e5f403721a75624038b5ef64c1a6b", "_cell_guid": "bf70791a-5251-4650-b643-2f295e9d9b06"}, "cell_type": "markdown"}, {"source": ["## Gender to categorical\n", "The gender column is a string of male, female an NaN values. Let's convert them to 1 and 0."], "metadata": {"_uuid": "76e54ad9b7968c03792c1c3266a694072dc678a0", "_cell_guid": "a33ddccd-19dd-4748-9fe3-eaab4c3b448d"}, "cell_type": "markdown"}, {"source": ["# Set the gender values\n", "gender = {'male':1, 'female':2}\n", "# Map the int values to the gender columns of test and predict dataframes\n", "dict_dfs['train'].gender = dict_dfs['train'].gender.map(gender)\n", "dict_dfs['predict'].gender = dict_dfs['predict'].gender.map(gender)\n", "\n", "# Set the NaN to 0 and convert the type to int8\n", "dict_dfs['train'].gender = dict_dfs['train'].gender.fillna(0).astype(np.int8)\n", "dict_dfs['predict'].gender = dict_dfs['predict'].gender.fillna(0).astype(np.int8)"], "metadata": {"_uuid": "b9ae0dc7999c101117df2ef913dbc9d7394b45f1", "_cell_guid": "fbb83ca4-61ed-41c2-b170-2a332e540157", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code"}, {"source": ["get_memory_usage_datafame()"], "metadata": {"_uuid": "95f5cc8cfe1ac4f309190297f0a5077467a9f0a6", "_cell_guid": "d1cafd05-9ae9-4749-9e2a-f5f7c42be260"}, "outputs": [], "execution_count": null, "cell_type": "code"}, {"source": ["# Predictive Model\n", "We are going to create the predictive model in order to have some start point and adjust our model from the main error."], "metadata": {"_uuid": "9f144e67527c7602010e7a202f7f6d492e13b8bd", "_cell_guid": "5d106015-d3cf-4fbe-a012-126cc8efacff"}, "cell_type": "markdown"}, {"source": ["## Separate X from Y\n", "Next we will separate the features (X) from the labels (Y)"], "metadata": {"_uuid": "69b94f7bdce292549634a9fb66c3367a10dd7371", "_cell_guid": "da0e80eb-e466-4954-b599-60b259c90165"}, "cell_type": "markdown"}, {"source": ["# Get the Y labels from the is_churn column\n", "Y = dict_dfs['train']['is_churn']\n", "# Drop the is_churn column from the train dataframe and store it on the X dataframe\n", "X = dict_dfs['train'].drop(['is_churn','msno'], axis=1)"], "metadata": {"_uuid": "c71f02cbbcc6fecddeff971a555bb339095538b5", "_cell_guid": "11ed935f-a680-42eb-9aab-f864fb6b08e7", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code"}, {"source": ["# Next steps...\n", "That's all for now. I hope this helps on your first steps on this challenge. Next steps will be to implement a predictive algorithm and after selecting the right one, modelling your data to improve the score. Feel free to ask or comment and fork on this notebook!"], "metadata": {"_uuid": "e7461211f0278d84fe50ce27a6e7c1208c78775f", "_cell_guid": "f9f6ccb5-fdd8-461c-9803-f74564518776"}, "cell_type": "markdown"}, {"source": [], "metadata": {"_uuid": "43d371290ceaba8d547d16dd52edff16102195a6", "_cell_guid": "d362befc-784a-4db8-a6e3-1a4b6598036c", "collapsed": true}, "outputs": [], "execution_count": null, "cell_type": "code"}], "nbformat": 4}