{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "d4920287-8aeb-3069-c9bc-121862e91123" }, "source": [ "# Predicting Survival on the Titanic\n", "This notebook is my first attempt at classification and machine learning in Python. For this reason, I welcome all comments and recommendations. \n", "\n", "## Competition Description \n", "Kaggle describes the competition as follows: \n", "\n", "> \"The sinking of the RMS Titanic is one of the most infamous shipwrecks in history. On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 passengers and crew. . .One of the reasons that the shipwreck led to such loss of life was that there were not enough lifeboats for the passengers and crew. . .In this challenge, we ask you to complete the analysis of what sorts of people were likely to survive. In particular, we ask you to apply the tools of machine learning to predict which passengers survived the tragedy.\"\n", "\n", "## Goal \n", "Predict if a passenger survived the sinking of the Titanic or not. For each PassengerId in the test dataset , you must predict a 0 (did not survive) or 1 (survived) value for the Survived variable.\n", "\n", "### Metric \n", "*Accuracy* - the percentage of passengers you correctly predict.\n", "\n", "###Submission File Format\n", "\n", "- A .csv file with exactly *418* entries plus a header row.\n", "- The file should have exactly 2 columns:\n", " - PassengerId (sorted in any order)\n", " - Survived (binary predictions: 0 for did not survive, 1 for survived)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "c041ca21-0710-37ab-9fab-c1ceb14bc414" }, "source": [ "#Getting Started\n", "\n", "**Initializing Libraries**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "74ede981-7e97-87d3-4449-567129df2348" }, "outputs": [], "source": [ "# Data Analysis and Wrangling\n", "import pandas as pd\n", "import numpy as np\n", "import random as rnd\n", "\n", "# Visualization\n", "import matplotlib as mpl\n", "import matplotlib.pyplot as plt\n", "import matplotlib.pylab as pylab\n", "import seaborn as sns\n", "\n", "# Machine learning\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.svm import SVC, LinearSVC\n", "from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier\n", "from sklearn.neighbors import KNeighborsClassifier\n", "from sklearn.naive_bayes import GaussianNB\n", "from sklearn.linear_model import Perceptron\n", "from sklearn.linear_model import SGDClassifier\n", "from sklearn.tree import DecisionTreeClassifier\n", "from sklearn.metrics import make_scorer, accuracy_score\n", "from sklearn.model_selection import GridSearchCV\n", "from sklearn.feature_selection import RFECV\n", "from sklearn.cross_validation import train_test_split , StratifiedKFold\n", "import xgboost as xgb\n", "\n", "\n", "# Configure Visualizations\n", "%matplotlib inline\n", "mpl.style.use('ggplot')\n", "sns.set_style('white')\n", "pylab.rcParams['figure.figsize'] = 8 , 6" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "f3a6ab91-e0b8-8b91-b2bb-362b4bc8c09f" }, "source": [ "**Loading the data into python**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "8fd07e2d-da19-0a75-ea62-565739ce3dd5" }, "outputs": [], "source": [ "train_df = pd.read_csv('../input/train.csv')\n", "test_df = pd.read_csv('../input/test.csv')\n", "combine = [train_df, test_df]" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "1bd9db12-6e54-cd85-a3f8-b27f7eb1bf43" }, "source": [ "**The variables in the training dataset are:**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1615cafe-8357-3745-ff3a-b24fe1977066" }, "outputs": [], "source": [ "print(train_df.columns.values)\n", "train_df.head()\n", "# train_df.tail()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "1f62ce06-0ad9-84bc-8a0e-ad26ca4825f2" }, "source": [ "**Variable Types:**\n", " \n", "- Categorical - Survived, Pclass, Sex, Embarked, Name, PassengerId, \n", "- Quantitative - Age, Fare, SibSp, Parch\n", "- Mixed - Cabin, Ticket\n", "\n", "**Variable Description:**\n", "\n", "- Survived: survival of passenger (0 = did not survive; 1 = survived)\n", "- Pclass: Passenger's class\n", "- Name: Passenger's name\n", "- Sex: Passenger's sex\n", "- Age: Passenger's age\n", "- SibSp: Number of siblings/spouses the passenger had aboard\n", "- Parch: Number of parents/children the passenger had aboard\n", "- Ticket: Ticket number\n", "- Fare: Ticket fare\n", "- Cabin: Cabin number\n", "- Embarked: Port of embarkation\n", "\n", "**Structure of train and test datasets**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "fdd2d813-84fe-bf0d-d4b6-4e8c8c6d7777" }, "outputs": [], "source": [ "train_df.info()\n", "print('_'*40)\n", "test_df.info()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "523450c2-10ee-ac14-d453-380b4c69c733" }, "source": [ "**Variables with missing data**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "17dbcd27-e1ce-145d-b503-2550a92242a5" }, "outputs": [], "source": [ "print('In the train dataset:')\n", "print('Age has ' + str(891 - 714) + ' missing values') \n", "print('Cabin has ' + str(891 - 204) + ' missing values') \n", "print('Embarked has ' + str(891 - 889) + ' missing values') \n", "print('_'*40)\n", "print('In the test dataset:')\n", "print('Age has ' + str(418 - 332) + ' missing values') \n", "print('Fare has ' + str(418 - 417) + ' missing values')\n", "print('Cabin has ' + str(418 - 91) + ' missing values')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "e89027f3-1da9-cf6f-4bdc-13b6e1e9cfc6" }, "source": [ "##Defining Functions to be Used in Exploratory Data Analysis" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "f491b3bd-a9e8-db92-ad2d-acaf546cf93b" }, "outputs": [], "source": [ "def plotHist(df, x, col, row = None, bins = 20): \n", " grid = sns.FacetGrid(df, col = col, row = row)\n", " grid.map(plt.hist, x, bins = bins)\n", " \n", "def plotDistribution(df, var, target, **kwargs):\n", " rowVar = kwargs.get('rowVar', None)\n", " colVar = kwargs.get('colVar' , None)\n", " grid = sns.FacetGrid(df, hue = target, aspect = 4, row = rowVar, col = colVar)\n", " grid.map(sns.kdeplot, var, shade = True)\n", " grid.set(xlim = (0, df[var].max()))\n", " grid.add_legend()\n", "\n", "def plotCategorical(df, cat, target, **kwargs):\n", " rowVar = kwargs.get('rowVar', None)\n", " colVar = kwargs.get('colVar', None)\n", " grid = sns.FacetGrid(df, row = rowVar, col = colVar)\n", " grid.map(sns.barplot, cat, target)\n", " grid.add_legend()\n", "\n", "def plotCorrelation(df):\n", " corr = df.corr()\n", " heat, ax = plt.subplots(figsize = (12, 10))\n", " cmap = sns.diverging_palette(220, 10, as_cmap = True)\n", " heat = sns.heatmap(\n", " corr, \n", " cmap = cmap,\n", " square = True, \n", " cbar_kws = {'shrink': .9}, \n", " ax = ax, \n", " annot = True, \n", " annot_kws = {'fontsize': 12})\n", "\n", "def describeMore(df):\n", " var = [] ; l = [] ; t = []\n", " for x in df:\n", " var.append(x)\n", " l.append(len(pd.value_counts(df[x])))\n", " t.append(df[x].dtypes)\n", " levels = pd.DataFrame({'Variable': var, 'Levels': l, 'Datatype': t})\n", " levels.sort_values(by = 'Levels', inplace = True)\n", " return levels\n", "\n", "def analyzeByPivot(df, grouping_var, sorting_var):\n", " place = df[[grouping_var, sorting_var]].groupby([grouping_var], \n", " as_index = False).mean().sort_values(by = sorting_var, ascending = False)\n", " return place" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "ac3d00e9-8d44-c8fd-df21-70c1df7a76ee" }, "source": [ "#Exploratory Data Analysis (EDA)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "2c652078-ff2c-6808-2810-4ec320255c2b" }, "source": [ "**General info on the variables**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "39ea643d-01d4-40e9-07f8-b33041913db8" }, "outputs": [], "source": [ "describeMore(train_df)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a22f3003-68ee-9682-9db2-23e3fd7fc790" }, "source": [ "**Summary statistics of the variables**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "f6edfbe2-5dc5-29ab-7076-0d94fa9fcc05" }, "outputs": [], "source": [ "train_df.describe(percentiles = [.01, .05, .10, .25, .50, .75, .90, .95, .99])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "bb49e16f-b3d2-d670-7f56-596f01202c2c" }, "outputs": [], "source": [ "train_df.describe(include=['O'])" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "215405b4-3eed-ac03-e31c-9c03da1ffc3e" }, "source": [ "**Quantitative Variables:**\n", "\n", "- There were passengers of 88 different ages in the training dataset. The average age was about 29.7, 90% of passengers were 50 years old or younger, 99% were less than 66 years old and at least 25% of passengers were less than 21 years old. \n", "- At least 25% of passengers in the training data had a sibling or spouse on board \n", "- At least 75% of passengers did not have parents or children on board. \n", "- The distribution of ticket fares is heavily skewed to the right as evidenced by a mean of about 32, but a median of about 14.5. About 75% of people paid a fare of 31 or less, while others paid up to 512. \n", "\n", "**Categorical Variables:**\n", "\n", "- All passengers in the training data have a unique PassengerId\n", "- Approximately 38 % of passengers in the training dataset survived in comparison to the true population survival rate of about 32.5% (specified in the description). \n", "- There are 3 levels of Passenger classes and at least 50% of passenger had a Pclass of 3. \n", "- 577 of the 891 passengers in the training set are males (about 65%). " ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "ba7a7bd0-520e-74bc-d7dd-9c08e79fdc7b" }, "source": [ "## Correlation of Variables" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d04217e1-119b-904f-2551-42a61b3e32e6" }, "outputs": [], "source": [ "plotCorrelation(train_df)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "656cf566-e19c-332d-808d-6d4361dad5b2" }, "source": [ "The variables that are more strongly correlated (positively or negatively) are represented by darker colors in the heat map. \n", "\n", "**Analyze by pivoting variables with classes/levels against each other**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2c723ddb-7e8b-2e5f-dd0e-7d491739038a" }, "outputs": [], "source": [ "analyzeByPivot(train_df, 'Pclass', 'Survived')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "6b0b973c-a06c-ecbe-6311-f2649819d75e" }, "source": [ "It appears passengers in Pclass 1 survived at much higher rates in the training data than the other classes. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "eae64343-55d6-426c-d5f1-deb6a3dc21ea" }, "outputs": [], "source": [ "analyzeByPivot(train_df, 'Sex', 'Survived')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b640242d-5f79-5107-0618-d8eb2e9c348e" }, "source": [ "It appears females survived at much higher rates than males in the training data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1c5aff9f-06ad-cb78-531a-da96593d16b6" }, "outputs": [], "source": [ "analyzeByPivot(train_df, 'SibSp', 'Survived')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0ef5bcb8-f2ec-e6db-a240-582bd173432b" }, "source": [ "Passengers traveling with 0-2 SibSp had the highest survival rates in the training data. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2516ccb2-a423-13e8-44d7-395e15384f5c" }, "outputs": [], "source": [ "analyzeByPivot(train_df, 'Parch', 'Survived')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3424d0a5-0df4-a20e-687b-1cc79bfa3de2" }, "source": [ "Passengers traveling with 0-3 Parch had the highest survival rates in the training data. But those traveling with 1, 2, or 3 Parch had about a 15% higher survival rate than those with 0 parents or children on board. " ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "8a180043-af5c-1f1d-7807-24fd2618e8f5" }, "source": [ "## Data Visualization" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "38d57642-051c-e03c-84b1-cc27fe3b0b9e" }, "source": [ "**Mean survival rate of passengers by Pclass**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "52671e5c-c179-735a-22f6-7b05edf456f6" }, "outputs": [], "source": [ "plotCategorical(train_df, 'Pclass', 'Survived')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "8ef39b77-d8d8-aeba-73b2-567c0bd99d16" }, "source": [ "**Distribution of passenger age by survival status**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "3f41ce0c-6fef-631d-940e-56ecaf3372ee" }, "outputs": [], "source": [ "plotHist(train_df, 'Age', 'Survived')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b74f671d-d10a-b7a0-0de4-fafb9c0950ff" }, "source": [ "**Mean survival rate of passengers by SibSp**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "b16c2669-a65d-490d-6c8c-461d94d9bedc" }, "outputs": [], "source": [ "plotCategorical(train_df, 'SibSp', 'Survived')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "aa6c912a-efce-607b-771c-8ffcdb2ee362" }, "source": [ "**Mean survival rate of passengers by Parch**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "0cb001e6-d3c3-0c8b-1666-e42557e4c1fc" }, "outputs": [], "source": [ "plotCategorical(train_df, 'Parch', 'Survived')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "e3095878-7657-35f5-4abb-8fcd5c1c5a4d" }, "source": [ "**Distribution of passenger age by Pclass and survival status**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5c6a7f4f-b20c-7e5b-0d81-16e62232f5a3" }, "outputs": [], "source": [ "plotHist(train_df, 'Age', 'Survived', 'Pclass', bins = 20)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "f98acb1e-48d2-2c46-8ff9-e78af72dc3c7" }, "source": [ "**Distribution of passenger age by sex and survival status**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "69f5a6b9-cd7a-c468-fb95-6a3691e63fd6" }, "outputs": [], "source": [ "plotHist(train_df, 'Age', 'Survived', row = 'Sex')\n", "plotDistribution(train_df, var = 'Age', target = 'Survived', rowVar = 'Sex')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "1e0045fd-51a6-1432-dbe2-e187aa62cfdf" }, "source": [ "**Mean Survival by Pclass of Males and Females**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e5b895a7-ee49-2eea-3dce-0d857c166ad6" }, "outputs": [], "source": [ "sns.pointplot(x = \"Pclass\", y = \"Survived\", hue = \"Sex\", data = train_df,\n", " palette={\"male\": \"blue\", \"female\": \"pink\"},\n", " markers=[\"*\", \"o\"], linestyles=[\"-\", \"--\"])" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "e4d28b40-09bc-d6df-fbfa-449d267cccf6" }, "source": [ "**Mean survival rate of passengers by Pclass, point of embarkation, and sex**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "f3538db3-9507-06ee-8510-55b0b226cd59" }, "outputs": [], "source": [ "grid = sns.FacetGrid(train_df, col='Embarked')\n", "grid.map(sns.pointplot, 'Pclass', 'Survived', 'Sex', palette='deep')\n", "grid.add_legend()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "f2169835-ebce-5094-aef7-5c4ad5a13d5c" }, "source": [ "**Price of fare by sex, point of embarkation, and survival**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1896b9ef-da4b-6e6b-9d23-8a2572bb6d71" }, "outputs": [], "source": [ "plotCategorical(train_df, 'Sex', 'Fare', rowVar = 'Embarked', colVar = 'Survived')\n", "\n", "# grid = sns.FacetGrid(train_df, row = 'Embarked', col = 'Survived', size = 2.2, aspect = 1.6)\n", "# grid.map(sns.barplot, 'Sex', 'Fare', ci = None)\n", "# grid.add_legend()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "847eb0b9-4dc3-a7c2-5f81-42edd03fae88" }, "source": [ "# Data Cleaning\n", "This section will consist of dropping extraneous variables/missing observations , imputing missing values where possible, manipulating old variables to make new ones, and pre-processing. " ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7f477508-3a3d-08a7-6348-7829a9c20441" }, "source": [ "## Dropping Ticket and Cabin Variables\n", "\n", "I am going to drop Ticket and Cabin. The ticket variable is alpha-numeric and 681 of the 891 passengers have a unique ticket. I am going to assume that some couples/families shared tickets since a ticket ID can be the same and there is only 1 missing fare in both the train and test datasets. Moreover, I am dropping Cabin because of the 891 observations in the training dataset there are only 204 observations where a cabin is listed and 147 of them are unique. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c86769d4-fd76-399c-187d-647f0828b16c" }, "outputs": [], "source": [ "train_df = train_df.drop(['Ticket', 'Cabin'], axis=1)\n", "test_df = test_df.drop(['Ticket', 'Cabin'], axis=1)\n", "combine = [train_df, test_df]\n", "\n", "print(\"After dropping Ticket and Cabin -- \", \n", "'Train:', train_df.shape, ', Test:', test_df.shape)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4fd32a2d-683e-9127-dffb-c9123f4d351f" }, "source": [ "##Creating Title Variable\n", "\n", "The following code uses a regular expression to extract a title ending in a period from the name variable for both training and test dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "0c077582-4e94-e295-d677-3fdfa987266e" }, "outputs": [], "source": [ "for dataset in combine:\n", " dataset['Title'] = dataset.Name.str.extract(' ([A-Za-z]+)\\.', expand=False)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "1d6b188d-b0b2-7ff0-c4e1-1775cb245c33" }, "source": [ "**Now let's look at the new title column for males and females:**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e0a44769-e1da-699d-f706-c67b1746b3ad" }, "outputs": [], "source": [ "pd.crosstab(train_df['Title'], train_df['Sex'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "3907fd59-4a18-7832-a3b9-d5b4a781d9d8" }, "outputs": [], "source": [ "analyzeByPivot(train_df, 'Title', 'Survived')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "47f6acd1-f76d-ab24-4a99-8d878b604914" }, "source": [ "Since there are a lot of similar titles and a few titles that fall into certain categories, let's group them accordingly. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "147e1be6-f0ec-f15d-a675-2ee2afbfff9c" }, "outputs": [], "source": [ "for dataset in combine:\n", " # Standardize officer titles \n", " dataset['Title'] = dataset['Title'].replace(['Capt', 'Col', 'Major', 'Dr'], 'Officer')\n", " # Standardize royal titles\n", " dataset['Title'] = dataset['Title'].replace(['Jonkheer', 'Don', 'Sir', 'Countess', 'Dona', 'Lady'],\n", " 'Royalty')\n", " # Mlle stands for Mademoiselle which is French for Miss\n", " dataset['Title'] = dataset['Title'].replace('Mlle', 'Miss')\n", " # Standardize Ms to Miss\n", " dataset['Title'] = dataset['Title'].replace('Ms', 'Miss')\n", " # Mme stands for Madame which is French for Mrs\n", " dataset['Title'] = dataset['Title'].replace('Mme', 'Mrs')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "133cc18b-330a-54b8-5ace-baf18cfe3dc9" }, "source": [ "Note: I left Rev on its own based on the assumption they will let others take their spot on the lifeboat\n", "\n", "**Mean survival rate by passenger title**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "928964cf-5cf6-dda9-597d-1d3b6d3935ac" }, "outputs": [], "source": [ "analyzeByPivot(train_df, 'Title', 'Survived')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "c5420ee6-8478-63bc-f85b-38fdce6d0037" }, "source": [ "Now that we have each passenger's title, we can drop Name and PassengerId from the training dataset and Name from the test dataset. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "703a8003-2ad1-a254-8bdb-ea44345ae467" }, "outputs": [], "source": [ "train_df = train_df.drop(['Name', 'PassengerId'], axis=1)\n", "test_df = test_df.drop(['Name'], axis=1)\n", "combine = [train_df, test_df]\n", "train_df.shape, test_df.shape" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "f82c9cc0-9479-f7a5-230b-535136febfe2" }, "source": [ "###Converting Title to Categorical" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "954f7b3f-fe15-0f2a-5b0e-fdf97fac19f2" }, "outputs": [], "source": [ "title_mapping = {\"Mr\": 1, \"Miss\": 2, \"Mrs\": 3, \"Master\": 4, \"Royalty\": 5, \"Officer\": 6, \"Rev\": 7}\n", "for dataset in combine:\n", " dataset['Title'] = dataset['Title'].map(title_mapping)\n", " # We only checked train_df for titles, so we don't know if there are other non-mapped titles in \n", " # test_df. For this reason we fill NA's with 0 and reserve that as an \"Other\" level.\n", " dataset['Title'] = dataset['Title'].fillna(0)\n", "train_df.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "e357d1fa-3aaf-0a0b-968e-4d17a67c1c61" }, "source": [ "##Converting Sex to Binary" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "8a69b4e4-bc69-a944-88e4-efa30bc68ad3" }, "outputs": [], "source": [ "for dataset in combine:\n", " dataset['Sex'] = dataset['Sex'].map({'female': 1, 'male': 0}).astype(int)\n", "train_df.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d06e6ef1-9908-723b-fc05-1208e47c211e" }, "source": [ "## Imputing Missing Age Values\n", "\n", "In accordance with Seghal's work on this topic, I impute the missing age values by iterating over Sex and Pclass. The goal is is to find the mean age (after excluding null values) for each combination of sex and Pclass. Recall that Sex has 2 levels (male, female) and Pclass has 3 levels (1, 2, 3) so we will have 6 combinations. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "40a5b95d-aa0a-9434-bd65-e6e5007ee3c2" }, "outputs": [], "source": [ "guess_ages = np.zeros((2,3))\n", "guess_ages" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "6877392c-de70-6cc7-127e-05bb848fd170" }, "source": [ "I will iterate over Sex and Pclass and populate the guess_ages array with the mean age of each combination at the end of each iteration. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "f136cc6d-615f-f049-01e2-080c698ffb67" }, "outputs": [], "source": [ "# For each dataset\n", "for dataset in combine:\n", " # For each sex\n", " for i in range(0, 2):\n", " # For each Pclass\n", " for j in range(0, 3):\n", " # Populate guess_age dataframe with each combo of Sex and Pclass for each iteration\n", " # and drop all NA values\n", " guess_df = dataset[(dataset['Sex'] == i) & \\\n", " (dataset['Pclass'] == j+1)]['Age'].dropna()\n", " # Set age_guess equal to the median age of each Sex and Pclass combo\n", " age_guess = guess_df.median()\n", " # Populate guess_ages array with each age_guess \n", " guess_ages[i,j] = age_guess \n", " # For each sex \n", " for i in range(0, 2):\n", " # For each Pclass\n", " for j in range(0, 3):\n", " # For each combo of Sex and Pclass, find the rows where Age is null and impute with \n", " # the corresponding age guess from the guess_ages array\n", " dataset.loc[ (dataset.Age.isnull()) & (dataset.Sex == i) & (dataset.Pclass == j+1),\\\n", " 'Age'] = guess_ages[i,j]\n", " # Convert age from float to int \n", " dataset['Age'] = dataset['Age'].astype(int)\n", "\n", "train_df.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "9699ca10-ed3c-73d4-40f7-46e2c9d5adc6" }, "source": [ "**Checking to see the distribution of age now that it has no missing values**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a8c30c8c-5dc7-9c13-9687-67f4360657c4" }, "outputs": [], "source": [ "plotDistribution(train_df, var = 'Age', target = None)\n", "train_df.Age.describe()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "880bb765-0723-3df8-c636-0b58b9b6aa36" }, "source": [ "It seems that age goes from about 0 to 80 and the median age is around 26. \n", "\n", "##Creating AgeBin Variable\n", "The AgeBin variable will separate ages into 5 separate bins " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "09310f4b-3cae-c11a-3cf6-20c2bb6bac27" }, "outputs": [], "source": [ "# Cuts Age into 5 bins\n", "train_df['AgeBin'] = pd.cut(train_df['Age'], 5)\n", "analyzeByPivot(train_df, 'AgeBin', 'Survived')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7591e6f6-3c28-9b8e-38ff-b69b7a10a351" }, "source": [ "**Now we use the AgeBin variable to encode the Age variable as a categorical variable**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d9c903eb-8dd8-439f-83f6-62c88962c75d" }, "outputs": [], "source": [ "combine = [train_df, test_df]\n", "\n", "for dataset in combine: \n", " dataset.loc[dataset['Age'] <= 16, 'Age'] = 0\n", " dataset.loc[(dataset['Age'] > 16) & (dataset['Age'] <= 32), 'Age'] = 1\n", " dataset.loc[(dataset['Age'] > 32) & (dataset['Age'] <= 48), 'Age'] = 2\n", " dataset.loc[(dataset['Age'] > 48) & (dataset['Age'] <= 64), 'Age'] = 3\n", " dataset.loc[ dataset['Age'] > 64, 'Age'] = 4\n", "train_df.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "09d8c871-503d-9314-0f88-aedbd87993fb" }, "source": [ "###Dropping AgeBin" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "6174c225-c069-d3f9-8248-477ae3895289" }, "outputs": [], "source": [ "train_df = train_df.drop(['AgeBin'], axis=1)\n", "combine = [train_df, test_df]\n", "train_df.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0c3eacb1-7ee0-44a6-34c8-d05a97d2f5ff" }, "source": [ "##Creating GroupSize Variable\n", "\n", "Note we have to add 1 to this number to account for the passenger. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d980c982-48ec-2a2a-e8ee-e2d3f4297152" }, "outputs": [], "source": [ "for dataset in combine:\n", " dataset['GroupSize'] = dataset['SibSp'] + dataset['Parch'] + 1\n", "analyzeByPivot(train_df, 'GroupSize', 'Survived')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "fadc40ee-7dff-a212-e0c4-3dc6c782dec3" }, "source": [ "**Next, I encode GroupSize as alone (GroupSize = 0), small (GroupSize = 1), medium (GroupSize = 2), and large (GroupSize = 3).**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "622a7581-08ba-5190-3256-7b94d46f3979" }, "outputs": [], "source": [ "for dataset in combine: \n", " # GroupSize = 1 --> Alone\n", " dataset.loc[dataset['GroupSize'] == 1, 'GroupSize'] = 0\n", " # GroupSize = 2, 3, or 4 --> Small \n", " dataset.loc[(dataset['GroupSize'] > 1) & (dataset['GroupSize'] <= 4), 'GroupSize'] = 1\n", " # GroupSize = 5, 6, or 7 --> Medium \n", " dataset.loc[(dataset['GroupSize'] > 4) & (dataset['GroupSize'] <= 7), 'GroupSize'] = 2\n", " # GroupSize = 8+ --> Large \n", " dataset.loc[dataset['GroupSize'] > 7, 'GroupSize'] = 3\n", "analyzeByPivot(train_df, 'GroupSize', 'Survived')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "df15dc4b-3685-80a3-f15e-b1d1d9e76969" }, "outputs": [], "source": [ "train_df.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "8df86db4-749f-9d99-c802-a62f5f513974" }, "source": [ "##Dropping SibSp and Parch" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "eb500ae3-cce3-7b72-7cef-b691e6268d0a" }, "outputs": [], "source": [ "train_df = train_df.drop(['Parch', 'SibSp'], axis=1)\n", "test_df = test_df.drop(['Parch', 'SibSp'], axis=1)\n", "combine = [train_df, test_df]\n", "\n", "train_df.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a0700b8e-fbc6-be22-17db-49c4d2a80eea" }, "source": [ "##Imputing Missing Values in Embarked\n", "\n", "Before, we noted that Embarked is missing 2 values in the train dataset and will use the mode to impute missing values. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a7e92a1f-198c-c863-2162-141e2ca63b4d" }, "outputs": [], "source": [ "# Drop NA values and report to mode of Embark\n", "freq_port = train_df.Embarked.dropna().mode()[0]\n", "freq_port" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "51cc72d9-7ff7-a668-7283-46108270eae4" }, "outputs": [], "source": [ "for dataset in combine:\n", " # Impute missing values with \"S\"\n", " dataset['Embarked'] = dataset['Embarked'].fillna(freq_port)\n", "analyzeByPivot(train_df, \"Embarked\", \"Survived\")" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "069110d6-66ca-e666-9e26-d5fed553e3eb" }, "source": [ "##Converting Embarked to Categorical" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ba56a88d-1410-a9ee-cb02-b382585dea92" }, "outputs": [], "source": [ "embarked_mapping = {'S': 0, 'C': 1, 'Q': 2}\n", "for dataset in combine:\n", " dataset['Embarked'] = dataset['Embarked'].map(embarked_mapping).astype(int)\n", "\n", "train_df.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "124e721a-d440-64e9-c901-1579eda89872" }, "source": [ "##Imputing Missing Values in Fare Variable (test_df)\n", "\n", "The code below specifies that we want to impute the single missing value with the median fare amount in the test dataset (inplace argument specifies that we want to update the Fare variable). " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "85439162-a9af-11aa-a351-cafcd57bb99a" }, "outputs": [], "source": [ "test_df['Fare'].fillna(test_df['Fare'].dropna().median(), inplace=True)\n", "test_df.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3bc560d4-7705-8969-4e50-368500284796" }, "source": [ "##Creating FareBin Variable\n", "\n", "Below is the distribution of Fare by percentiles. I eventually chose to break Fare up into quintiles." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c8e76fe7-8e9a-caf1-e613-2d5e520f25b4" }, "outputs": [], "source": [ "train_df.Fare.describe(percentiles = [.1, .2, .3, .4, .5, .6, .7, .8, .9])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "0b5dd58c-365c-0e6f-364e-5d28aab7f247" }, "outputs": [], "source": [ "train_df['FareBand'] = pd.qcut(train_df['Fare'], 5)\n", "analyzeByPivot(train_df, 'FareBand', 'Survived')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7aa18a4e-9cff-2847-ad59-b90b08ee4842" }, "source": [ "###Converting Fare to Categorical" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c3fab818-de5c-56df-f77b-acaa38bcf2b8" }, "outputs": [], "source": [ "for dataset in combine:\n", " dataset.loc[ dataset['Fare'] <= 7.854, 'Fare'] = 0\n", " dataset.loc[(dataset['Fare'] > 7.854) & (dataset['Fare'] <= 10.5), 'Fare'] = 1\n", " dataset.loc[(dataset['Fare'] > 10.5) & (dataset['Fare'] <= 21.679), 'Fare'] = 2\n", " dataset.loc[(dataset['Fare'] > 21.679) & (dataset['Fare'] <= 39.688), 'Fare'] = 3\n", " dataset.loc[dataset['Fare'] > 39.688, 'Fare'] = 4\n", " dataset['Fare'] = dataset['Fare'].astype(int)\n", "train_df = train_df.drop(['FareBand'], axis=1)\n", "combine = [train_df, test_df]" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "6eb2b53b-9d7f-0019-549b-29514a3eb8aa" }, "source": [ "##Cleaned Datasets" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "442dd2fd-91c7-c210-fda3-34a5b59528fd" }, "outputs": [], "source": [ "train_df.head(10)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "b88dd74d-50b9-bba7-fa78-6da3e722c729" }, "outputs": [], "source": [ "test_df.head(10)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "956955c7-bfe2-7f20-128b-33ec97eb2d20" }, "source": [ "**Correlation matrix of cleaned training dataset**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "01e6ab6a-0d6b-fa5c-5ed7-bf16c3dd23aa" }, "outputs": [], "source": [ "plotCorrelation(train_df)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "908415cb-cdda-43a0-5e6c-dfc09143e993" }, "source": [ "#Preparing Data for Modeling\n", "\n", "Drop predictor variable (Survived) from the list of independent variables (X). " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "014dbc43-f733-6694-fa7c-3040237ca0ea" }, "outputs": [], "source": [ "X_train = train_df.drop(\"Survived\", axis=1)\n", "Y_train = train_df[\"Survived\"]\n", "X_test = test_df.drop(\"PassengerId\", axis=1).copy()\n", "\n", "X_train.shape, Y_train.shape, X_test.shape" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3bea3c12-8a8a-46f7-eade-6c2cfea27c90" }, "source": [ "#Modeling\n", "\n", "##Logistic Regression" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "94d83104-121d-665a-aa08-99db288710da" }, "outputs": [], "source": [ "logreg = LogisticRegression() # Define Logistic Regression model\n", "logreg.fit(X_train, Y_train) # Run the model using the training data\n", "Y_pred = logreg.predict(X_test) # Use the model to predict Y values given the test data\n", "acc_log = round(logreg.score(X_train, Y_train) * 100, 2) # Accuracy \n", "print(\"Accuracy:\", acc_log)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "47c30f66-af26-73b4-1b4f-600d467cf2d1" }, "source": [ "**Optimal Number of Variables**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "854bdfb0-7efa-bcb6-ead0-ae183b98758d" }, "outputs": [], "source": [ "rfecv = RFECV(estimator = logreg, step = 1, cv = StratifiedKFold(Y_train, 2 ), scoring = 'accuracy')\n", "rfecv.fit(X_train,Y_train)\n", "print(\"Accuracy:\", round((rfecv.score(X_train, Y_train) * 100),2))\n", "print(\"Optimal number of variables: %d\" % rfecv.n_features_)\n", "\n", "# Plot number of variables VS. cross-validation scores\n", "plt.figure()\n", "plt.xlabel(\"Number of Variables Used\")\n", "plt.ylabel(\"Cross validation score (nb of correct classifications)\")\n", "plt.plot(range(1 , len(rfecv.grid_scores_) + 1), rfecv.grid_scores_ * 100)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "665bfed9-3ee4-517f-5bc5-c42b6d1b6e01" }, "source": [ "**Correlation Coefficients**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "3b9635b2-65fa-2c30-78f9-69118aa73cc5" }, "outputs": [], "source": [ "coeff_df = pd.DataFrame(train_df.columns.delete(0))\n", "coeff_df.columns = ['Variable']\n", "coeff_df[\"Correlation Coeff\"] = pd.Series(logreg.coef_[0])\n", "coeff_df.sort_values(by = 'Correlation Coeff', ascending=False)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d73cd713-0418-ec56-17e8-26281bf0a45f" }, "source": [ "##Support Vector Classifier (SVM)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "eaba47bc-4ba0-a590-b340-72125ea9b2e5" }, "outputs": [], "source": [ "svc = SVC()\n", "svc.fit(X_train, Y_train)\n", "Y_pred = svc.predict(X_test)\n", "acc_svc = round(svc.score(X_train, Y_train) * 100, 2)\n", "print(\"Accuracy:\", acc_svc)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "52067e2d-7bc5-ead6-ef13-a9e4a7130936" }, "source": [ "###Linear SVC" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "92c7f375-c21e-f987-16fc-15b597ef6369" }, "outputs": [], "source": [ "linear_svc = LinearSVC()\n", "linear_svc.fit(X_train, Y_train)\n", "Y_pred = linear_svc.predict(X_test)\n", "acc_linear_svc = round(linear_svc.score(X_train, Y_train) * 100, 2)\n", "print(\"Accuracy:\", acc_linear_svc)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4011af63-d5d0-2575-7c1c-f5e788b33c8e" }, "source": [ "**Optimal Number of Variables**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "83a7521a-abdf-896d-c4cd-4a4a4d623c5d" }, "outputs": [], "source": [ "rfecv = RFECV(estimator = linear_svc, step = 1, cv = StratifiedKFold(Y_train, 2 ), scoring = 'accuracy')\n", "rfecv.fit(X_train,Y_train)\n", "print(\"Accuracy:\", round((rfecv.score(X_train, Y_train) * 100),2))\n", "print(\"Optimal number of variables: %d\" % rfecv.n_features_)\n", "\n", "# Plot number of variables VS. cross-validation scores\n", "plt.figure()\n", "plt.xlabel(\"Number of Variables Used\")\n", "plt.ylabel(\"Cross validation score (nb of correct classifications)\")\n", "plt.plot(range(1 , len(rfecv.grid_scores_) + 1), rfecv.grid_scores_ * 100)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0fc37157-299e-74c0-d98d-8ea4649cc06a" }, "source": [ "##k-Nearest Neighbors (k-NN)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "408d5e25-c0f8-9b46-54e9-455e87696902" }, "outputs": [], "source": [ "knn = KNeighborsClassifier(n_neighbors = 3)\n", "knn.fit(X_train, Y_train)\n", "Y_pred = knn.predict(X_test)\n", "acc_knn = round(knn.score(X_train, Y_train) * 100, 2)\n", "print(\"Accuracy: \", acc_knn)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4b34d439-06d3-2f6c-456f-b5a3c28aaac2" }, "source": [ "##Naive Bayes" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "57b2e286-1bca-ec68-77b1-7229c556265f" }, "outputs": [], "source": [ "gaussian = GaussianNB()\n", "gaussian.fit(X_train, Y_train)\n", "Y_pred = gaussian.predict(X_test)\n", "acc_gaussian = round(gaussian.score(X_train, Y_train) * 100, 2)\n", "print(\"Accuracy: \", acc_gaussian)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "74c26833-d3e1-dd2e-b066-5312e4bacd11" }, "source": [ "##Perceptron" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "643a1b03-a65c-80ce-c11d-1b2c710c079e" }, "outputs": [], "source": [ "perceptron = Perceptron()\n", "perceptron.fit(X_train, Y_train)\n", "Y_pred = perceptron.predict(X_test)\n", "acc_perceptron = round(perceptron.score(X_train, Y_train) * 100, 2)\n", "print(\"Accuracy: \", acc_perceptron)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "ccd0b6ae-675f-3d81-4817-a786c98c0580" }, "source": [ "**Optimal Number of Variables**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "bf5027d3-2ef4-c088-b3a9-f16e74bb203f" }, "outputs": [], "source": [ "rfecv = RFECV(estimator = perceptron, step = 1, cv = StratifiedKFold(Y_train, 2 ), scoring = 'accuracy')\n", "rfecv.fit(X_train,Y_train)\n", "print(\"Accuracy:\", round((rfecv.score(X_train, Y_train) * 100),2))\n", "print(\"Optimal number of variables: %d\" % rfecv.n_features_)\n", "\n", "# Plot number of variables VS. cross-validation scores\n", "plt.figure()\n", "plt.xlabel(\"Number of Variables Used\")\n", "plt.ylabel(\"Cross validation score (nb of correct classifications)\")\n", "plt.plot(range(1 , len(rfecv.grid_scores_) + 1), rfecv.grid_scores_ * 100)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "99cef0e3-3a5f-3492-8625-f2fa3eca552f" }, "source": [ "##Stochastic Gradient Descent" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5a003a55-765a-cd7a-8dac-02628381509e" }, "outputs": [], "source": [ "sgd = SGDClassifier()\n", "sgd.fit(X_train, Y_train)\n", "Y_pred = sgd.predict(X_test)\n", "acc_sgd = round(sgd.score(X_train, Y_train) * 100, 2)\n", "print(\"Accuracy: \", acc_sgd)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "eb579fc0-289a-b74b-67bf-adef3f697b3b" }, "source": [ "**Optimal Number of Variables**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "6848b087-1238-bdc1-9657-4f45645c2307" }, "outputs": [], "source": [ "rfecv = RFECV(estimator = sgd, step = 1, cv = StratifiedKFold(Y_train, 2 ), scoring = 'accuracy')\n", "rfecv.fit(X_train,Y_train)\n", "print(\"Accuracy:\", round((rfecv.score(X_train, Y_train) * 100),2))\n", "print(\"Optimal number of variables: %d\" % rfecv.n_features_)\n", "\n", "# Plot number of variables VS. cross-validation scores\n", "plt.figure()\n", "plt.xlabel(\"Number of Variables Used\")\n", "plt.ylabel(\"Cross validation score (nb of correct classifications)\")\n", "plt.plot(range(1 , len(rfecv.grid_scores_) + 1), rfecv.grid_scores_ * 100)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3499247e-aa6f-592b-d256-62de3900f62f" }, "source": [ "##Decision Tree" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ed70ad38-cf88-76e6-87e2-4369f05bb5b2" }, "outputs": [], "source": [ "decision_tree = DecisionTreeClassifier()\n", "decision_tree.fit(X_train, Y_train)\n", "Y_pred = decision_tree.predict(X_test)\n", "acc_decision_tree = round(decision_tree.score(X_train, Y_train) * 100, 2)\n", "print(\"Accuracy: \", acc_decision_tree)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4f6ed38b-b4f4-b085-7266-f66b3afac51e" }, "source": [ "**Optimal Number of Variables**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "14c4bd39-f33b-1fc6-e6c0-fef81b6e579a" }, "outputs": [], "source": [ "rfecv = RFECV(estimator = decision_tree, step = 1, cv = StratifiedKFold(Y_train, 2 ), scoring = 'accuracy')\n", "rfecv.fit(X_train,Y_train)\n", "print(\"Accuracy:\", round((rfecv.score(X_train, Y_train) * 100),2))\n", "print(\"Optimal number of variables: %d\" % rfecv.n_features_)\n", "\n", "# Plot number of variables VS. cross-validation scores\n", "plt.figure()\n", "plt.xlabel(\"Number of Variables Used\")\n", "plt.ylabel(\"Cross validation score (nb of correct classifications)\")\n", "plt.plot(range(1 , len(rfecv.grid_scores_) + 1), rfecv.grid_scores_ * 100)\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "0a6d0636-c8cf-ef9b-b3a7-7ff1104f274f" }, "outputs": [], "source": [ "def plot_model_var_imp(model, X, y):\n", " imp = pd.DataFrame( \n", " model.feature_importances_ , \n", " columns = [ 'Importance' ] , \n", " index = X.columns \n", " )\n", " imp = imp.sort_values([ 'Importance'], ascending = True)\n", " imp[:10].plot(kind = 'barh')\n", "\n", "plot_model_var_imp(decision_tree, X_train, Y_train)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "e5002a61-6599-1ce9-51e4-0abb31724f15" }, "source": [ "##Random Forest\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "968a19bc-0ca2-7c92-ba98-b55b2d3b1f8f" }, "outputs": [], "source": [ "random_forest = RandomForestClassifier(n_estimators = 100)\n", "random_forest.fit(X_train, Y_train)\n", "Y_pred_rf = random_forest.predict(X_test)\n", "random_forest.score(X_train, Y_train)\n", "acc_random_forest = round(random_forest.score(X_train, Y_train) * 100, 2)\n", "print(\"Accuracy: \", acc_random_forest)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "51cd705d-c377-069f-f4bc-b5bb1122f121" }, "source": [ "**Optimal Number of Variables**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5c779e3f-80ac-4726-333f-cbdf4bee6c7a" }, "outputs": [], "source": [ "rfecv = RFECV(estimator = random_forest, step = 1, cv = StratifiedKFold(Y_train, 2 ), scoring = 'accuracy')\n", "rfecv.fit(X_train,Y_train)\n", "print(\"Accuracy:\", round((rfecv.score(X_train, Y_train) * 100),2))\n", "print(\"Optimal number of variables: %d\" % rfecv.n_features_)\n", "\n", "# Plot number of variables VS. cross-validation scores\n", "plt.figure()\n", "plt.xlabel(\"Number of Variables Used\")\n", "plt.ylabel(\"Cross validation score (nb of correct classifications)\")\n", "plt.plot(range(1 , len(rfecv.grid_scores_) + 1), rfecv.grid_scores_ * 100)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4d83137a-01f4-6170-1d7e-9ac2b1a6b813" }, "source": [ "##Gradient Boosting Classifer" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "65ae9faa-4b08-4e77-2332-8fabbb8127e0" }, "outputs": [], "source": [ "grad_boost = GradientBoostingClassifier()\n", "grad_boost.fit(X_train, Y_train)\n", "Y_pred = grad_boost.predict(X_test)\n", "grad_boost.score(X_train, Y_train)\n", "acc_grad_boost = round(grad_boost.score(X_train, Y_train) * 100, 2)\n", "print(\"Accuracy: \", acc_grad_boost)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7d4ce275-4477-57f5-2fb9-5ad63ec90466" }, "source": [ "**Optimal Number of Variables**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e018287b-571a-e3bc-c199-ee22a2ef27c0" }, "outputs": [], "source": [ "rfecv = RFECV(estimator = linear_svc, step = 1, cv = StratifiedKFold(Y_train, 2 ), scoring = 'accuracy')\n", "rfecv.fit(X_train,Y_train)\n", "print(\"Accuracy:\", round((rfecv.score(X_train, Y_train) * 100),2))\n", "print(\"Optimal number of variables: %d\" % rfecv.n_features_)\n", "\n", "# Plot number of variables VS. cross-validation scores\n", "plt.figure()\n", "plt.xlabel(\"Number of Variables Used\")\n", "plt.ylabel(\"Cross validation score (nb of correct classifications)\")\n", "plt.plot(range(1 , len(rfecv.grid_scores_) + 1), rfecv.grid_scores_ * 100)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7cfc9e00-5fe2-bd03-d176-d6f9b8bef137" }, "source": [ "##XGBoost" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e952fb03-d16a-6838-97ae-9bb8ea18e9aa" }, "outputs": [], "source": [ "xgboost = xgb.XGBClassifier(max_depth = 3, n_estimators = 300, learning_rate = 0.05)\n", "xgboost.fit(X_train, Y_train)\n", "Y_pred = xgboost.predict(X_test)\n", "xgboost.score(X_train, Y_train)\n", "acc_xgboost = round(xgboost.score(X_train, Y_train) * 100, 2)\n", "print(\"Accuracy: \", acc_xgboost)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "e762f7e6-5712-02c7-8088-a7933e8ce45d" }, "source": [ "#Ranking Models" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "32f82295-1d95-b879-9581-73df57b6b437" }, "outputs": [], "source": [ "models = pd.DataFrame({\n", " 'Model': ['Logistic Regression', 'Support Vector Classifier', 'Linear SVC', 'k-NN', \n", " 'Random Forest', 'Naive Bayes', 'Perceptron', \n", " 'Stochastic Gradient Decent', \n", " 'Decision Tree', \"Gradient Boosting Classifier\", \"XGBoost\"],\n", " 'Score': [acc_log, acc_svc, acc_linear_svc, acc_knn, \n", " acc_random_forest, acc_gaussian, acc_perceptron, \n", " acc_sgd, acc_decision_tree, acc_grad_boost, acc_xgboost]})\n", "models.sort_values(by = 'Score', ascending = False)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "337d9175-ca28-2edd-b66f-85d60b0db3c8" }, "source": [ "#Submission" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "12110d65-1b28-be9b-f930-7477fab80cdb" }, "outputs": [], "source": [ "submission = pd.DataFrame({\n", " \"PassengerId\": test_df[\"PassengerId\"],\n", " \"Survived\": Y_pred_rf})\n", "submission.to_csv('submission.csv', index = False)\n", "submission.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "83fc1215-96c0-a9a7-259b-d7d4a2070ca2" }, "source": [ "### Acknowledgements\n", "I would like to credit a number of people as their notebooks on this topic make up a large portion of the code and analysis used to produce this notebook: \n", "\n", "- [Titanic Data Science Solutions](https://www.kaggle.com/startupsci/titanic/titanic-data-science-solutions) - Manav Seghal\n", "- [Scikit-Learn ML from Start to Finish](https://www.kaggle.com/jeffd23/titanic/scikit-learn-ml-from-start-to-finish) - Jeff Delaney\n", "- [An Interactive Data Science Tutorial](https://www.kaggle.com/helgejo/titanic/an-interactive-data-science-tutorial) - Helge Bjorland\n", "- [XGBoost example (Python)](https://www.kaggle.com/datacanary/titanic/xgboost-example-python/code) - DataCanary\n" ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }