{
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0,
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "5b825984-72f9-589b-f4a1-d3fbad1d9598",
        "_active": false
      },
      "source": "# **An Interactive Data Science Tutorial**\n\n\n*[Based on the Titanic competition on Kaggle](https://www.kaggle.com/c/titanic)*\n\n*by Helge Bjorland & Stian Eide*\n\n*January 2017*\n\n---\n\n## Content\n\n\n 1. Business Understanding (5 min)\n     * Objective\n     * Description\n 2. Data Understanding (15 min)\n    * Import Libraries\n    * Load data\n    * Statistical summaries and visualisations\n    * Excersises\n 3. Data Preparation (5 min)\n    * Missing values imputation\n    * Feature Engineering\n 4. Modeling (5 min)\n     * Build the model\n 5. Evaluation (25 min)\n     * Model performance\n     * Feature importance\n     * Who gets the best performing model?\n 6. Deployment  (5 min)\n     * Submit result to Kaggle leaderboard     \n\n[*Adopted from Cross Industry Standard Process for Data Mining (CRISP-DM)*](http://www.sv-europe.com/crisp-dm-methodology/)\n\n![CripsDM](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b9/CRISP-DM_Process_Diagram.png/220px-CRISP-DM_Process_Diagram.png \"Process diagram showing the relationship between the different phases of CRISP-DM\")",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "6f9380bf-1835-9f4f-b728-48bdb84e7cab",
        "_active": false
      },
      "source": "# 1. Business Understanding\n\n## 1.1 Objective\nPredict survival on the Titanic\n\n## 1.2 Description\nThe sinking of the RMS Titanic is one of the most infamous shipwrecks in history.  On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 passengers and crew. This sensational tragedy shocked the international community and led to better safety regulations for ships.\n\nOne of the reasons that the shipwreck led to such loss of life was that there were not enough lifeboats for the passengers and crew. Although there was some element of luck involved in surviving the sinking, some groups of people were more likely to survive than others, such as women, children, and the upper-class.\n\nIn this challenge, we ask you to complete the analysis of what sorts of people were likely to survive. In particular, we ask you to apply the tools of machine learning to predict which passengers survived the tragedy.\n\n**Before going further, what do you think is the most important reasons passangers survived the Titanic sinking?**\n\n[Description from Kaggle](https://www.kaggle.com/c/titanic)",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "af225757-e074-9b64-ba8f-137252ea90e0",
        "_active": false
      },
      "source": "# 2. Data Understanding\n\n## 2.1 Import Libraries\nFirst of some preparation. We need to import python libraries containing the necessary functionality we will need. \n\n*Simply run the cell below by selecting it and pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d3cb1c42-90ba-9674-0e72-0b4ee496fa42",
        "_active": false
      },
      "outputs": [],
      "source": "# Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Handle table-like data and matrices\nimport numpy as np\nimport pandas as pd\n\n# Modelling Algorithms\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn.svm import SVC, LinearSVC\nfrom sklearn.ensemble import RandomForestClassifier , GradientBoostingClassifier\n\n# Modelling Helpers\nfrom sklearn.preprocessing import Imputer , Normalizer , scale\nfrom sklearn.cross_validation import train_test_split , StratifiedKFold\nfrom sklearn.feature_selection import RFECV\n\n# Visualisation\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nimport matplotlib.pylab as pylab\nimport seaborn as sns\n\n# Configure visualisations\n%matplotlib inline\nmpl.style.use( 'ggplot' )\nsns.set_style( 'white' )\npylab.rcParams[ 'figure.figsize' ] = 8 , 6",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "cb44cb03-a5be-653d-34bf-f1e7eba133a4",
        "_active": false
      },
      "source": "## 2.2 Setup helper Functions\nThere is no need to understand this code. Just run it to simplify the code later in the tutorial.\n\n*Simply run the cell below by selecting it and pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0bb8cf49-d080-46a7-2c66-fa967ad4db97",
        "_active": false
      },
      "outputs": [],
      "source": "def plot_histograms( df , variables , n_rows , n_cols ):\n    fig = plt.figure( figsize = ( 16 , 12 ) )\n    for i, var_name in enumerate( variables ):\n        ax=fig.add_subplot( n_rows , n_cols , i+1 )\n        df[ var_name ].hist( bins=10 , ax=ax )\n        ax.set_title( 'Skew: ' + str( round( float( df[ var_name ].skew() ) , ) ) ) # + ' ' + var_name ) #var_name+\" Distribution\")\n        ax.set_xticklabels( [] , visible=False )\n        ax.set_yticklabels( [] , visible=False )\n    fig.tight_layout()  # Improves appearance a bit.\n    plt.show()\n\ndef plot_distribution( df , var , target , **kwargs ):\n    row = kwargs.get( 'row' , None )\n    col = kwargs.get( 'col' , None )\n    facet = sns.FacetGrid( df , hue=target , aspect=4 , row = row , col = col )\n    facet.map( sns.kdeplot , var , shade= True )\n    facet.set( xlim=( 0 , df[ var ].max() ) )\n    facet.add_legend()\n\ndef plot_categories( df , cat , target , **kwargs ):\n    row = kwargs.get( 'row' , None )\n    col = kwargs.get( 'col' , None )\n    facet = sns.FacetGrid( df , row = row , col = col )\n    facet.map( sns.barplot , cat , target )\n    facet.add_legend()\n\ndef plot_correlation_map( df ):\n    corr = titanic.corr()\n    _ , ax = plt.subplots( figsize =( 12 , 10 ) )\n    cmap = sns.diverging_palette( 220 , 10 , as_cmap = True )\n    _ = sns.heatmap(\n        corr, \n        cmap = cmap,\n        square=True, \n        cbar_kws={ 'shrink' : .9 }, \n        ax=ax, \n        annot = True, \n        annot_kws = { 'fontsize' : 12 }\n    )\n\ndef describe_more( df ):\n    var = [] ; l = [] ; t = []\n    for x in df:\n        var.append( x )\n        l.append( len( pd.value_counts( df[ x ] ) ) )\n        t.append( df[ x ].dtypes )\n    levels = pd.DataFrame( { 'Variable' : var , 'Levels' : l , 'Datatype' : t } )\n    levels.sort_values( by = 'Levels' , inplace = True )\n    return levels\n\ndef plot_variable_importance( X , y ):\n    tree = DecisionTreeClassifier( random_state = 99 )\n    tree.fit( X , y )\n    plot_model_var_imp( tree , X , y )\n    \ndef plot_model_var_imp( model , X , y ):\n    imp = pd.DataFrame( \n        model.feature_importances_  , \n        columns = [ 'Importance' ] , \n        index = X.columns \n    )\n    imp = imp.sort_values( [ 'Importance' ] , ascending = True )\n    imp[ : 10 ].plot( kind = 'barh' )\n    print (model.score( X , y ))\n    ",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ea8b0e99-e512-f1f5-ed3d-e7df876b9bed",
        "_active": false
      },
      "source": "## 2.3 Load data\nNow that our packages are loaded, let's read in and take a peek at the data.\n\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ee2677e2-b78b-250b-a908-816109e3ab91",
        "_active": false
      },
      "outputs": [],
      "source": "# get titanic & test csv files as a DataFrame\ntrain = pd.read_csv(\"../input/train.csv\")\ntest    = pd.read_csv(\"../input/test.csv\")\n\nfull = train.append( test , ignore_index = True )\ntitanic = full[ :891 ]\n\ndel train , test\n\nprint ('Datasets:' , 'full:' , full.shape , 'titanic:' , titanic.shape)",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "76852c59-23bf-55b6-5b6d-f672c97114ae",
        "_active": false
      },
      "source": "## 2.4 Statistical summaries and visualisations\n\nTo understand the data we are now going to consider some key facts about various variables including their relationship with the target variable, i.e. survival.\n\nWe start by looking at a few lines of the data\n\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "60d5efad-7649-c1e8-3cae-59bf562e8457",
        "_active": false
      },
      "outputs": [],
      "source": "# Run the code to see the variables, then read the variable description below to understand them.\ntitanic.head()",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1f3708e1-b5ca-7540-39fc-da7453d0fb80",
        "_active": false
      },
      "source": "**VARIABLE DESCRIPTIONS:**\n\nWe've got a sense of our variables, their class type, and the first few observations of each. We know we're working with 1309 observations of 12 variables. To make things a bit more explicit since a couple of the variable names aren't 100% illuminating, here's what we've got to deal with:\n\n\n**Variable Description**\n\n - Survived: Survived (1) or died (0)\n - Pclass: Passenger's class\n - Name: Passenger's name\n - Sex: Passenger's sex\n - Age: Passenger's age\n - SibSp: Number of siblings/spouses aboard\n - Parch: Number of parents/children aboard\n - Ticket: Ticket number\n - Fare: Fare\n - Cabin: Cabin\n - Embarked: Port of embarkation\n\n[More information on the Kaggle site](https://www.kaggle.com/c/titanic/data)",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "bcc371f8-8fbb-a582-5944-8537c152c6c9",
        "_active": false
      },
      "source": "### 2.4.1 Next have a look at some key information about the variables\nAn numeric variable is one with values of integers or real numbers while a categorical variable is a variable that can take on one of a limited, and usually fixed, number of possible values, such as blood type.\n\nNotice especially what type of variable each is, how many observations there are and some of the variable values.\n\nAn interesting observation could for example be the minimum age 0.42, do you know why this is?\n\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "88c8b958-0973-d27e-d463-58fada41900e",
        "_active": false
      },
      "outputs": [],
      "source": "titanic.describe()",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c006f42b-cba3-7109-92fb-79676f726afb",
        "_active": false
      },
      "source": "### 2.4.2 A heat map of correlation may give us a understanding of which variables are important\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "84418f3e-32ab-7a6c-a60e-6d45760ee666",
        "_active": false
      },
      "outputs": [],
      "source": "plot_correlation_map( titanic )",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "952846c2-63a9-1923-2501-d3036056855b",
        "_active": false
      },
      "source": "### 2.4.3 Let's further explore the relationship between the features and survival of passengers \nWe start by looking at the relationship between age and survival.\n\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "617703c7-4ab2-186c-f4b1-40d5ffa37850",
        "_active": false
      },
      "outputs": [],
      "source": "# Plot distributions of Age of passangers who survived or did not survive\nplot_distribution( titanic , var = 'Age' , target = 'Survived' , row = 'Sex' )",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "834c62d5-23e7-7e0c-d2d4-bc960d09a409",
        "_active": false
      },
      "source": "Consider the graphs above. Differences between survival for different values is what will be used to separate the target variable (survival in this case) in the model. If the two lines had been about the same, then it would not have been a good variable for our predictive model. \n\nConsider some key questions such as; what age does males/females have a higher or lower probability of survival? ",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f8d077bf-e70c-29df-253e-5bd22ff3f06e",
        "_active": false
      },
      "source": "### 2.4.3 Excersise 1: Investigating numeric variables\nIt's time to get your hands dirty and do some coding! Try to plot the distributions of Fare of passangers who survived or did not survive. Then consider if this could be a good predictive variable.\n\n*Hint: use the code from the previous cell as a starting point.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6fcddc9f-19be-c474-a639-79c6b2d2a41a",
        "_active": false,
        "collapsed": false
      },
      "outputs": [],
      "source": "# Excersise 1\n# Plot distributions of Fare of passangers who survived or did not survive\nplot_distribution(titanic, var = 'Fare', target = 'Survived', row = 'Sex')",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e9dffe91-ee29-9cca-2860-29a32a44e2af",
        "_active": false
      },
      "source": "### 2.4.4 Embarked\nWe can also look at categorical variables like Embarked and their relationship with survival.\n\n- C = Cherbourg  \n- Q = Queenstown\n- S = Southampton",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b5beadda-fe40-d017-9bf8-29d372401c10",
        "_active": false
      },
      "outputs": [],
      "source": "# Plot survival rate by Embarked\nplot_categories( titanic , cat = 'Embarked' , target = 'Survived' )",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "33e7d091-af47-9a77-c907-81e710200c5f",
        "_active": false
      },
      "source": "### 2.4.4 Excersise 2 - 5: Investigating categorical variables\nEven more coding practice! Try to plot the survival rate of Sex, Pclass, SibSp and Parch below. \n\n*Hint: use the code from the previous cell as a starting point.*\n\nAfter considering these graphs, which variables do you expect to be good predictors of survival? ",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "25022b4c-a631-45fa-ec2b-e7bd7ed11987",
        "_active": false,
        "collapsed": false
      },
      "outputs": [],
      "source": "# Excersise 2\n# Plot survival rate by Sex\nplot_categories(titanic, cat = 'Sex', target = 'Survived')",
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8c7af046-677c-07a7-d4bf-b28d10092f73",
        "_active": false,
        "collapsed": false
      },
      "outputs": [],
      "source": "# Excersise 3\n# Plot survival rate by Pclass\nplot_categories(titanic, cat = 'Pclass', target = 'Survived')",
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f5b4252e-4800-d674-f86f-01eba244230d",
        "_active": false,
        "collapsed": false
      },
      "outputs": [],
      "source": "# Excersise 4\n# Plot survival rate by SibSp\nplot_categories(titanic, cat = 'SibSp', target = 'Survived')",
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4dc79695-082e-84c0-0f0d-6fca5ad414dd",
        "_active": false,
        "collapsed": false
      },
      "outputs": [],
      "source": "# Excersise 5\n# Plot survival rate by Parch\nplot_categories(titanic, cat = 'Parch', target = 'Survived')",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c9888e98-091a-d98c-6ccf-b733a4f0d941",
        "_active": false
      },
      "source": "# 3. Data Preparation",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e280f761-4b30-9776-09e3-fcaecf932a01",
        "_active": false
      },
      "source": "## 3.1 Categorical variables need to be transformed to numeric variables\nThe variables *Embarked*, *Pclass* and *Sex* are treated as categorical variables. Some of our model algorithms can only handle numeric values and so we need to create a new variable (dummy variable) for every unique value of the categorical variables.\n\nThis variable will have a value 1 if the row has a particular value and a value 0 if not. *Sex* is a dichotomy (old school gender theory) and will be encoded as one binary variable (0 or 1).\n\n*Select the cells below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "75023b46-ded0-94ab-3ea1-7da1eec945fe",
        "_active": false
      },
      "outputs": [],
      "source": "# Transform Sex into binary values 0 and 1\nsex = pd.Series( np.where( full.Sex == 'male' , 1 , 0 ) , name = 'Sex' )",
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "cf816542-c40f-abe0-b662-a91211882bff",
        "_active": false
      },
      "outputs": [],
      "source": "# Create a new variable for every unique value of Embarked\nembarked = pd.get_dummies( full.Embarked , prefix='Embarked' )\nembarked.head()",
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "01758920-20cc-e200-90aa-e4404c41a8d7",
        "_active": false
      },
      "outputs": [],
      "source": "# Create a new variable for every unique value of Embarked\npclass = pd.get_dummies( full.Pclass , prefix='Pclass' )\npclass.head()",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d1269afc-b929-d519-9646-146f8a91b472",
        "_active": false
      },
      "source": "## 3.2 Fill missing values in variables\nMost machine learning alghorims require all variables to have values in order to use it for training the model. The simplest method is to fill missing values with the average of the variable across all observations in the training set.\n\n*Select the cells below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "04044a9c-603d-1963-34f4-a85efb8c9166",
        "_active": false
      },
      "outputs": [],
      "source": "# Create dataset\nimputed = pd.DataFrame()\n\n# Fill missing values of Age with the average of Age (mean)\nimputed[ 'Age' ] = full.Age.fillna( full.Age.mean() )\n\n# Fill missing values of Fare with the average of Fare (mean)\nimputed[ 'Fare' ] = full.Fare.fillna( full.Fare.mean() )\n\nimputed.head()",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "4f2ce8f7-8bf6-69f8-e2ae-5593b5cf98ac",
        "_active": false
      },
      "source": "## 3.3 Feature Engineering – Creating new variables\nCredit: http://ahmedbesbes.com/how-to-score-08134-in-titanic-kaggle-challenge.html",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "bd8efd0f-4af4-d709-cfeb-1907f885e341",
        "_active": false
      },
      "source": "### 3.3.1 Extract titles from passenger names\nTitles reflect social status and may predict survival probability\n\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "32772d9e-8c05-4eaf-5cd3-8425ae5837cd",
        "_active": false
      },
      "outputs": [],
      "source": "title = pd.DataFrame()\n# we extract the title from each name\ntitle[ 'Title' ] = full[ 'Name' ].map( lambda name: name.split( ',' )[1].split( '.' )[0].strip() )\n\n# a map of more aggregated titles\nTitle_Dictionary = {\n                    \"Capt\":       \"Officer\",\n                    \"Col\":        \"Officer\",\n                    \"Major\":      \"Officer\",\n                    \"Jonkheer\":   \"Royalty\",\n                    \"Don\":        \"Royalty\",\n                    \"Sir\" :       \"Royalty\",\n                    \"Dr\":         \"Officer\",\n                    \"Rev\":        \"Officer\",\n                    \"the Countess\":\"Royalty\",\n                    \"Dona\":       \"Royalty\",\n                    \"Mme\":        \"Mrs\",\n                    \"Mlle\":       \"Miss\",\n                    \"Ms\":         \"Mrs\",\n                    \"Mr\" :        \"Mr\",\n                    \"Mrs\" :       \"Mrs\",\n                    \"Miss\" :      \"Miss\",\n                    \"Master\" :    \"Master\",\n                    \"Lady\" :      \"Royalty\"\n\n                    }\n\n# we map each title\ntitle[ 'Title' ] = title.Title.map( Title_Dictionary )\ntitle = pd.get_dummies( title.Title )\n#title = pd.concat( [ title , titles_dummies ] , axis = 1 )\n\ntitle.head()",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a56838a1-5d47-194f-7ac6-06c397b60482",
        "_active": false
      },
      "source": "### 3.3.2 Extract Cabin category information from the Cabin number\n\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "87ed124f-d659-bf03-7dbf-accee0ddfe75",
        "_active": false
      },
      "outputs": [],
      "source": "cabin = pd.DataFrame()\n\n# replacing missing cabins with U (for Uknown)\ncabin[ 'Cabin' ] = full.Cabin.fillna( 'U' )\n\n# mapping each Cabin value with the cabin letter\ncabin[ 'Cabin' ] = cabin[ 'Cabin' ].map( lambda c : c[0] )\n\n# dummy encoding ...\ncabin = pd.get_dummies( cabin['Cabin'] , prefix = 'Cabin' )\n\ncabin.head()",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d2dba1d1-e7cd-b9c8-fcc7-fae22cca38da",
        "_active": false
      },
      "source": "### 3.3.3 Extract ticket class from ticket number\n\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5319db41-01dd-3a48-83d8-56572272d966",
        "_active": false
      },
      "outputs": [],
      "source": "# a function that extracts each prefix of the ticket, returns 'XXX' if no prefix (i.e the ticket is a digit)\ndef cleanTicket( ticket ):\n    ticket = ticket.replace( '.' , '' )\n    ticket = ticket.replace( '/' , '' )\n    ticket = ticket.split()\n    ticket = map( lambda t : t.strip() , ticket )\n    ticket = list(filter( lambda t : not t.isdigit() , ticket ))\n    if len( ticket ) > 0:\n        return ticket[0]\n    else: \n        return 'XXX'\n\nticket = pd.DataFrame()\n\n# Extracting dummy variables from tickets:\nticket[ 'Ticket' ] = full[ 'Ticket' ].map( cleanTicket )\nticket = pd.get_dummies( ticket[ 'Ticket' ] , prefix = 'Ticket' )\n\nticket.shape\nticket.head()",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7e812a29-3885-30a9-3d51-beda531016d9",
        "_active": false
      },
      "source": "### 3.3.4 Create family size and category for family size\nThe two variables *Parch* and *SibSp* are used to create the famiy size variable\n\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d885986a-77ea-b35c-5ddc-c16f54357232",
        "_active": false
      },
      "outputs": [],
      "source": "family = pd.DataFrame()\n\n# introducing a new feature : the size of families (including the passenger)\nfamily[ 'FamilySize' ] = full[ 'Parch' ] + full[ 'SibSp' ] + 1\n\n# introducing other features based on the family size\nfamily[ 'Family_Single' ] = family[ 'FamilySize' ].map( lambda s : 1 if s == 1 else 0 )\nfamily[ 'Family_Small' ]  = family[ 'FamilySize' ].map( lambda s : 1 if 2 <= s <= 4 else 0 )\nfamily[ 'Family_Large' ]  = family[ 'FamilySize' ].map( lambda s : 1 if 5 <= s else 0 )\n\nfamily.head()",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c20b06e8-7126-a80e-762a-8a383ffba9e1",
        "_active": false
      },
      "source": "## 3.4 Assemble final datasets for modelling\n\nSplit dataset by rows into test and train in order to have a holdout set to do model evaluation on. The dataset is also split by columns in a matrix (X) containing the input data and a vector (y) containing the target (or labels).",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "29663deb-5bbf-b621-abe5-37cfa5e3d29f",
        "_active": false
      },
      "source": "### 3.4.1 Variable selection\nSelect which features/variables to inculde in the dataset from the list below:\n\n - imputed \n - embarked\n - pclass\n - sex\n - family\n - cabin\n - ticket\n\n*Include the variables you would like to use in the function below seperated by comma, then run the cell*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4b3459bd-752c-0a96-22dc-8672900a0bb9",
        "_active": false
      },
      "outputs": [],
      "source": "# Select which features/variables to include in the dataset from the list below:\n# imputed , embarked , pclass , sex , family , cabin , ticket\n\nfull_X = pd.concat( [ imputed , embarked , cabin , sex ] , axis=1 )\nfull_X.head()",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "cb10186c-81b3-f9cd-250a-fab46b0001f5",
        "_active": false
      },
      "source": "### 3.4.2 Create datasets\nBelow we will seperate the data into training and test datasets.\n\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7ca114b3-f22c-b8bf-4e61-9c9bcd82fa63",
        "_active": false
      },
      "outputs": [],
      "source": "# Create all datasets that are necessary to train, validate and test models\ntrain_valid_X = full_X[ 0:891 ]\ntrain_valid_y = titanic.Survived\ntest_X = full_X[ 891: ]\ntrain_X , valid_X , train_y , valid_y = train_test_split( train_valid_X , train_valid_y , train_size = .7 )\n\nprint (full_X.shape , train_X.shape , valid_X.shape , train_y.shape , valid_y.shape , test_X.shape)",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f269d9b9-73fc-6d34-b46a-61fd6f1357ae",
        "_active": false
      },
      "source": "### 3.4.3 Feature importance\nSelecting the optimal features in the model is important. \nWe will now try to evaluate what the most important variables are for the model to make the prediction.\n\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "17c2f033-c43d-93f8-a697-bf32eec3b550",
        "_active": false
      },
      "outputs": [],
      "source": "plot_variable_importance(train_X, train_y)",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "84f76826-572a-e65d-7dd7-cd3791bf5237",
        "_active": false
      },
      "source": "# 4. Modeling\nWe will now select a model we would like to try then use the training dataset to train this model and thereby check the performance of the model using the test set. \n\n## 4.1 Model Selection\nThen there are several options to choose from when it comes to models. A good starting point is logisic regression. \n\n**Select ONLY the model you would like to try below and run the corresponding cell by pressing the play button.**",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "bd56cc59-9abd-fb1f-e5e1-211c48776863",
        "_active": false
      },
      "source": "### 4.1.1 Random Forests Model\nTry a random forest model by running the cell below. ",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "29893a22-aa92-b35d-881e-7ce907f3b3b2",
        "_active": false
      },
      "outputs": [],
      "source": "model = RandomForestClassifier(n_estimators=100)",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "777e1893-585a-c545-250a-f6ea12c10fb9",
        "_active": false
      },
      "source": "### 4.1.2 Support Vector Machines\nTry a Support Vector Machines model by running the cell below. ",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c34e2a74-23b9-2916-683d-f0d6956ad5e6",
        "_active": false
      },
      "outputs": [],
      "source": "model = SVC()",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "bc024820-b3db-4b2f-01aa-f9b35e413be1",
        "_active": false
      },
      "source": "### 4.1.3 Gradient Boosting Classifier\nTry a Gradient Boosting Classifier model by running the cell below. ",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "120f00c8-b568-f3af-97fd-df9762d4aefb",
        "_active": false
      },
      "outputs": [],
      "source": "model = GradientBoostingClassifier()",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "452de3e7-c672-0aeb-b49e-5bb6c400de75",
        "_active": false
      },
      "source": "### 4.1.4 K-nearest neighbors\nTry a k-nearest neighbors model by running the cell below. ",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "862cf425-ab7c-6abb-3f86-fe6a63f4a790",
        "_active": false
      },
      "outputs": [],
      "source": "model = KNeighborsClassifier(n_neighbors = 3)",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "34eb3eed-6d89-5c9e-31e7-40eb7e01639e",
        "_active": false
      },
      "source": "### 4.1.5 Gaussian Naive Bayes\nTry a Gaussian Naive Bayes model by running the cell below. ",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e6b97cb8-56c1-3b01-ca52-c2e9685b68d0",
        "_active": false
      },
      "outputs": [],
      "source": "model = GaussianNB()",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c3c89f76-4d2e-ae80-8b61-13feb6f33831",
        "_active": false
      },
      "source": "### 4.1.6 Logistic Regression\nTry a Logistic Regression model by running the cell below. ",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f3573fce-2045-aa1d-e010-dc28139b5a16",
        "_active": false
      },
      "outputs": [],
      "source": "model = LogisticRegression()",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7e8d2a8d-a9f6-d416-74e7-4f5b711dcd98",
        "_active": false
      },
      "source": "## 4.2 Train the selected model\nWhen you have selected a dataset with the features you want and a model you would like to try it is now time to train the model. After all our preparation model training is simply done with the one line below.\n\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "02a54d08-b02a-a4df-6d74-540be7d243d8",
        "_active": false
      },
      "outputs": [],
      "source": "model.fit( train_X , train_y )",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3868397c-30ed-a870-9c61-30b5d5ebdce7",
        "_active": false
      },
      "source": "# 5. Evaluation\nNow we are going to evaluate model performance and the feature importance.\n\n## 5.1 Model performance\nWe can evaluate the accuracy of the model by using the validation set where we know the actual outcome. This data set have not been used for training the model, so it's completely new to the model. \n\nWe then compare this accuracy score with the accuracy when using the model on the training data. If the difference between these are significant this is an indication of overfitting. We try to avoid this because it means the model will not generalize well to new data and is expected to perform poorly.\n\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c4e9a0a3-f637-5568-e92c-07c0548cd89d",
        "_active": false
      },
      "outputs": [],
      "source": "# Score the model\nprint (model.score( train_X , train_y ) , model.score( valid_X , valid_y ))",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "cd75f2ca-dcb1-e2af-ebf8-e1dea0cf44a5",
        "_active": false
      },
      "source": "## 5.2 Feature importance - selecting the optimal features in the model\nWe will now try to evaluate what the most important variables are for the model to make the prediction. The function below will only work for decision trees, so if that's the model you chose you can uncomment the code below (remove # in the beginning)  and see the feature importance.\n\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f9412dd9-11ce-74ba-7fd6-148b2ae3bce3",
        "_active": false
      },
      "outputs": [],
      "source": "#plot_model_var_imp(model, train_X, train_y)",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e7d3ad7a-92eb-f0e0-4028-3eec53c65581",
        "_active": false
      },
      "source": "### 5.2.1 Automagic\nIt's also possible to automatically select the optimal number of features and visualize this. This is uncommented and can be tried in the competition part of the tutorial.\n\n*Select the cell below and run it by pressing the play button.*",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2e290c27-d9d2-4ebf-32bc-8ff56c68f763",
        "_active": false
      },
      "outputs": [],
      "source": "rfecv = RFECV( estimator = model , step = 1 , cv = StratifiedKFold( train_y , 2 ) , scoring = 'accuracy' )\nrfecv.fit( train_X , train_y )\n\n#print (rfecv.score( train_X , train_y ) , rfecv.score( valid_X , valid_y ))\n#print( \"Optimal number of features : %d\" % rfecv.n_features_ )\n\n# Plot number of features VS. cross-validation scores\n#plt.figure()\n#plt.xlabel( \"Number of features selected\" )\n#plt.ylabel( \"Cross validation score (nb of correct classifications)\" )\n#plt.plot( range( 1 , len( rfecv.grid_scores_ ) + 1 ) , rfecv.grid_scores_ )\n#plt.show()",
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "61945225-1dcd-870d-bc49-851d5c91d1d5",
        "_active": false
      },
      "source": "## 5.3 Competition time!\nIt's now time for you to get your hands even dirtier and go at it all by yourself in a `challenge`! \n\n1. Try to the other models in step 4.1 and compare their result\n    * Do this by uncommenting the code and running the cell you want to try\n2. Try adding new features in step 3.4.1\n    * Do this by adding them in to the function in the feature section.\n\n\n**The winner is the one to get the highest scoring model for the validation set**",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "598cfdec-2804-0312-e69e-2ea779f045bc",
        "_active": false
      },
      "source": "# 6. Deployment\n\nDeployment in this context means publishing the resulting prediction from the model to the Kaggle leaderboard. To do this do the following:\n\n 1. select the cell below and run it by pressing the play button.\n 2. Press the `Publish` button in top right corner.\n 3. Select `Output` on the notebook menubar\n 4. Select the result dataset and press `Submit to Competition` button",
      "execution_count": null,
      "outputs": [],
      "execution_state": "idle"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "25cb37ca-225b-2917-00a8-5c08708ccf1d",
        "_active": false
      },
      "outputs": [],
      "source": "test_Y = model.predict( test_X )\npassenger_id = full[891:].PassengerId\ntest = pd.DataFrame( { 'PassengerId': passenger_id , 'Survived': test_Y } )\ntest.shape\ntest.head()\ntest.to_csv( 'titanic_pred.csv' , index = False )",
      "execution_state": "idle"
    }
  ]
}