{"nbformat_minor": 1, "nbformat": 4, "cells": [{"cell_type": "markdown", "metadata": {"_cell_guid": "d339e976-f564-4f3c-bdf7-c9c425870b91", "_uuid": "a1dffd87f49cc2b45fa923fee4aa1277439c2955"}, "source": ["<img src=\"https://i1.creativecow.net/u/301161/ezgif.com-resize5.gif\"/>"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "15bec6b9-750a-4fcd-931f-ec01f0775c1b", "_uuid": "7ea0ab58a7f6ec667931160006e344fb63749557"}, "source": ["# About This Kernel\n", "****\n", "\n", "This Kernel will be updated daily. I'll be updating you guys with more statistical and graphical analysis. Feel free to leave any important findings or questions during your exploration! Let's d\n", "\n", "This notebook will always be a work in progress. Please leave any comments about further improvements to the notebook! Any feedback or constructive criticism is greatly appreciated!. Thank you guys!"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "be813eca-cb38-46d6-9f54-572fcae2be4d", "_uuid": "c8075db96a9c553cc4581e6b23757a793851f1bf"}, "source": ["# Part 1: Obtaining the Data \n", "***"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "bd367025-a046-4f7e-a9d7-e6dbf5285e6d", "_uuid": "2b5206ccead7f946e48082c6c06b0d5265f7c901"}, "source": ["# Import the neccessary modules for data manipulation and visual representation\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import matplotlib as matplot\n", "import seaborn as sns\n", "%matplotlib inline"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "e96d6fa4-425c-41b0-bc66-651b79766073", "_uuid": "697e349dbe1b9f6df80d07ed38cd997f74f82944"}, "source": ["train = pd.read_csv('../input/train.csv')\n", "members = pd.read_csv('../input/members.csv')\n", "transactions = pd.read_csv('../input/transactions.csv')\n", "#sample_submission_zero= pd.read_csv('../input/sample_submission_zero.csv')\n", "#user_logs = pd.read_csv('../input/user_logs.csv',nrows = 2e7)"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "85370efe-254c-468f-9d26-3769f38975e1", "_uuid": "5585507e42211a9b729eacec983056ad67edda0b"}, "source": ["# Part 2: Scrubbing the Data \n", "***"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "b9f98bfc-051f-492f-b99d-2315bb090a84", "_uuid": "bc104bd12bb149bfa0a1e3bc61c0f4b6eb2eb1c3"}, "source": ["### Overview of Train DataFrame\n", "***\n", "\n", "**The dataset has:**\n", " - **Observations:** 992,931\n", " - **Features:** 2\n", " - **Churn Rate:** 6.4%"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "3cb616a6-3275-422e-aed9-574c0cbb0b74", "_uuid": "a544be0819ac55ba97899040f505c5d9683c1163"}, "source": ["**Feature Description:**\n", "- **msno:** This feature represents the customer's user **ID**, which is labeled as long character strings.\n", "- **is_churn:** This feature is our target variable. **0's** represent no churn. **1's** represent churn."]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "4d49492e-e169-4d82-8d14-a5df807d6da5", "_uuid": "5b9b803ed7d070ae37935a3be473c789d9e28632"}, "source": ["**Questions & Concerns:**\n", "- \"The provided training data set is derived from transaction log. We picked the users who have their expiration dates fall in Feb, 2017 and check whether those people renew their subscription with 30 days after expiration to generate training label. Our method is not the only way to generate the training data. The training data set can be generate using different logic. Say, you can check each user's transaction log and calculate the interval between two consecutive entries. In this case, you will generate a training data set much bigger than what we provided in the data section.\" - Arden Chiu \n", "\n", "\n", "- \"One reminder, we did make a filter on the expiration date associated with each transaction. We removed the entries that have expiration date > 2017-03-31.\" - Arden Chiu\n", "\n", "\n", "- **Qustion:** \"Can we use the expiration date in the members.csv? Is it future information?\" - yangyang\n", "    - **Response**: \"The expiration date in the members.csv is a snapshot of our member table. Hence it is possible to contain future information, but it may not give you much useful information. Say a user made a two-year term subscription on 2017-03-15. We will have the membership expiration in member.csv set to 2019-05-15, however, this does not mean the user will not have other transaction between those two dates (2017-03-15- 2019-03-15).\" - Arden Chiu\n", "\n", "Source: https://www.kaggle.com/c/kkbox-churn-prediction-challenge/discussion/39756"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "8879adae-79a0-428b-a067-0e6d5b2f0f4c", "_uuid": "b5a8f83ad2458bd11c50c1b1af672ae78af015dc"}, "source": ["train.head()"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "ebecc513-117a-47ed-a211-a220e96f6b46", "_uuid": "bf8296cabff444100ffcc8e544ca3105ba1f23f2"}, "source": ["# The dataset contains 2 columns and 992931 observations\n", "train.shape"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "87bcc231-4efe-4e64-9f13-117d39a0efa8", "_uuid": "745d9d27174a7db813deb1870d8af9246b193dc4", "scrolled": true}, "source": ["# Check to see if the train set has any missing values. No missing values!\n", "train.isnull().any()"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "3b514036-de60-48c4-a5ee-6493b93edbf8", "_uuid": "0fd82e4159b0632c618da797d995d81e536ae513"}, "source": ["# Looks like about 93.6% of customers stayed and 6.4% of customers left. \n", "# NOTE: When performing cross validation, its important to maintain this turnover ratio\n", "churn_rate = train.is_churn.value_counts() / len(train)\n", "churn_rate"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "8c7cc074-fb08-4e4e-94b8-6cd457a90fcc", "_uuid": "8e98df7f6cd994833343b2919cfde16e17453bfa"}, "source": ["### Overview of Members DataFrame\n", "***\n", "\n", "**The dataset has:**\n", " - **Observations:** 5,116,194\n", " - **Features:** 7\n", " - **Missing Value(s):** gender"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "5811f2ad-3770-4a2d-ac63-b8abbb44b0f1", "_uuid": "7d9b86efbd4e54419a96c39ba7f8138a31a50a8a"}, "source": ["**Feature Description:**\n", "- **msno:** This feature represents the customer's user **ID**, which is labeled as long character strings.\n", "- **city:** This feature contains **21** different unique cities, ranging from 1-22 (**excluding** the number **2**)\n", "- **bd:** This feature contains a lot of **outliers**. It represents the **age** of the user. Probably not a useful variable to use.\n", "- **gender:** This feature represents the customer's gender. The distrubution of this feature contains **A LOT OF MISSING VALUES**. About **17%** are males, **17%** are females, and **66%** are NaN's. Probably not a useful variable to use.\n", "- **registered_via:** This feature represents the registration method of the user. There are **7** unique labels. \n", "- **registration_init_time:** This feature is just the date of registration of the user\n", "- **expiration_date:** This feature represents the expiration date of the user's subscription\n"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "d6af6d32-2f84-4464-a14d-689927eebb35", "_uuid": "28c2e7915759aab7cdc3e734a6369817531249b2"}, "source": ["members.tail()"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "c777254c-57c9-4728-9216-7123c2c150d0", "_uuid": "f498e228310cefcf09adb57b08afb8c9d42e3be4"}, "source": ["# The dataset contains 2 columns and 992931 observations\n", "members.shape"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "968f2f8e-30bd-4a03-9822-79dc03c25f6d", "_uuid": "d0aa2af071f1abbf4577224d9c2e8f46372d1542"}, "source": ["# Check to see if the train set has any missing values.\n", "members.isnull().any()"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "792e4e95-6541-423d-93d6-20e78a8073c3", "_uuid": "854f36650f569b22033ab08b0092e6e1050099b0"}, "source": ["# Quick Overview of the members dataframe\n", "members.describe()"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "c56e1ca6-ab2f-4c44-8db0-3917656f7c2d", "_uuid": "3a57684456547cc19e456d43b526934ef27e2cbb"}, "source": ["members.city.describe()"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "7159a0c3-51ad-4ee3-83d0-6bb1d052994d", "_uuid": "ee5128442cbec0bc433aa1b5c2b567e49a507ce0"}, "source": ["# Display the unique values in the city variable\n", "# It has 21 unique city values and the #2 is missing\n", "members.city.unique()"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "56cf4df4-b31d-4eff-a428-d643f982728f", "_uuid": "27765a68c8cca789446c5bf56645d73ddf4e71d1"}, "source": ["# Display the unique values in the bd variable\n", "# It contains many outliers and random numbers. Maybe this variable shouldn't be used\n", "members.bd.unique()"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "30431496-703e-44c7-a812-02e66a6e02cf", "_uuid": "9c6b6621c6471a3847b044b748d668a90e4530b7"}, "source": ["# Display the distrubtion of gender variable\n", "members.gender.value_counts() / len(members)"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "af5273ab-e433-4365-a28b-2c9e6a71a862", "_uuid": "ad3d17ed6cda7379973f1e3986e449cf1f0d6122"}, "source": ["members.registered_via.unique()"]}, {"cell_type": "markdown", "metadata": {"collapsed": true, "_cell_guid": "624eb2c2-df6f-4632-bb9a-efdf51b0f02a", "_uuid": "f06a8c0694b7e818890a5a91236b353b2764634a"}, "source": ["### Overview of Transaction DataFrame\n", "***\n", "\n", "**The dataset has:**\n", " - **Observations:** 5,116,194\n", " - **Features:** 7\n", " - **Missing Value(s):** gender"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "d6f5eb70-7f30-4d68-81cb-cfedebccf110", "_uuid": "0db212e6ea6c09ebf7e2fa2419c859141634d868"}, "source": ["transactions.head()"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "2e69d4ad-9d8f-448f-b756-41487bb103c3", "_uuid": "5d38ec7ab1e9432c2bdcce0cb8d4b9170438fbd7"}, "source": ["# This data frame \n", "transactions.shape"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "5b552817-f3ea-46e9-bc0c-b0637892d82f", "_uuid": "3477cbe79e78ebb39ab017d7bfcad03d88fd0cdd"}, "source": ["# Check to see if the transaction set has any missing values.\n", "transactions.isnull().any()"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "98caff55-9deb-4f09-b506-cb200c2e85ea", "_uuid": "8d38f869c98f4bf2b1255493650bed96b2a324a2"}, "source": ["# Reformating Features in Train/Memebers Dataset\n", "***\n", "\n", "### Create dummy variables for the 'department' and 'salary' features, since they are categorical \n"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "a6c40e81-bdb7-4e1c-8fc0-84d06b8796ba", "_uuid": "8b21bb27a49d1f0ce8c9d5c03fbcc11b6598c884"}, "source": ["# Convert is_churn into a categorical variable\n", "train[\"is_churn\"] = train[\"is_churn\"].astype('category')\n", "\n", "# Convert these features from members dataset into categorical variables\n", "members[\"city\"] = members[\"city\"].astype('category')\n", "members[\"gender\"] = members[\"gender\"].astype('category')\n", "members[\"registered_via\"] = members[\"registered_via\"].astype('category')\n", "members[\"registration_init_time\"] = members[\"registration_init_time\"].astype('category')\n", "members[\"expiration_date\"] = members[\"expiration_date\"].astype('category')"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "4f74aefb-2564-4b1d-86d6-f5014ce276d0", "_uuid": "f860ed22955c66cd8e71d764f6f913cee3eb22ab"}, "source": ["# Merge Train & Members Dataset\n", "***"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "5cd7a5a3-f3ae-4c4c-9684-ba9a255ce10c", "_uuid": "1b778ae58fea6ab3e957156b87bc53766611f055"}, "source": ["training = pd.merge(left = train,right = members,how = 'left',on=['msno'])\n", "training.head()"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "f19ef933-e904-41d6-8ead-5d663f347d4e", "_uuid": "0c6fb549beeeb5a91a1440da11315312fd3d3d4f"}, "source": ["training.dtypes"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "55ae7355-6a4a-4a80-b8f1-ec1ab40aba1c", "_uuid": "ff3c8c4183d996847d6dd471b6457c4cc491b9e4"}, "source": ["training['city'].fillna(method='ffill', inplace=True)\n", "training['bd'].fillna(method='ffill', inplace=True)\n", "\n", "training['gender'].fillna(method='ffill', inplace=True)\n", "\n", "training['registered_via'].fillna(method='ffill', inplace=True)\n", "training.isnull().any()"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "3fc2807d-ac72-4fee-a14b-6a38be852131", "_uuid": "88959f233767864d41f0f588eff92b252c8df3a4"}, "source": ["# Exploring the Data\n", "***"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "41a25426-edfc-432d-889e-a9a03f80d8d3", "_uuid": "eba3031c3151b4136b99f224d9b5b4f9f2513af3"}, "source": ["## Members Exploration\n", "### City / Gender / Churn Distributions\n", "***"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "bc4b467d-aacb-4a33-aa1d-ed34f44d0d62", "_uuid": "5996b2bf3ba9fe7c61cc3b4cc96cfb99ac503d5a"}, "source": ["# Set up the matplotlib figure\n", "f, axes = plt.subplots(ncols=3, figsize=(20, 6))\n", "\n", "# Graph User City Distribution\n", "# sns.distplot(training.city, kde=False, color=\"g\",  ax=axes[0]).set_title('User City Distribution')\n", "data = training.groupby('city').aggregate({'msno':'count'}).reset_index()\n", "sns.barplot(x='city', y='msno', data=data, ax=axes[0]).set_title('User City Distribution')\n", "\n", "# Graph User Gender Distrubtion\n", "##sns.barplot(x=\"gender\", data=training, ax=axes[1]).set_title('User Register_Via Distribution')\n", "sns.countplot(y=\"gender\", data=training, color=\"c\",  ax=axes[1]).set_title('User Gender Distribution')\n", "\n", "# Graph User Churn Distribution\n", "sns.distplot(training.is_churn, kde=False, color=\"b\", bins = 3,  ax=axes[2]).set_title('User Churn Distribution')"]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "47ea0a7b-a036-4980-9c0b-b370bf694398", "_uuid": "7f6eda7d52f32562046c6f929ad6b397cb8ae7b6"}, "source": ["## Member's Registration Type Distribution\n", "***"]}, {"execution_count": null, "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_cell_guid": "7cf8f16e-0845-4911-86c2-03bd51fc67f3", "_uuid": "f2d77d75455a7fa16603430e457bac0aeb8f1175"}, "source": ["sns.countplot(y=\"registered_via\", data=training, color=\"c\").set_title('Registration Type Distribution')"]}], "metadata": {"language_info": {"mimetype": "text/x-python", "version": "3.6.1", "name": "python", "pygments_lexer": "ipython3", "file_extension": ".py", "nbconvert_exporter": "python", "codemirror_mode": {"version": 3, "name": "ipython"}}, "kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}}}