{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "3aa6612c-1ea7-8143-3691-1f606d5c0f0b" }, "source": [ "Imports\n", "-------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2fee9e4a-7daf-593e-b568-166e4e656bee" }, "outputs": [], "source": [ "import matplotlib\n", "from matplotlib import pyplot as plt\n", "matplotlib.style.use('ggplot')\n", "%matplotlib inline\n", "import seaborn as sns\n", "\n", "import numpy as np\n", "import pandas as pd\n", "pd.options.display.max_columns = 100\n", "pd.options.display.max_rows = 100\n", "\n", "from sklearn.pipeline import make_pipeline\n", "from sklearn.svm import SVC\n", "from sklearn.ensemble import RandomForestClassifier\n", "from sklearn.feature_selection import SelectFromModel\n", "from sklearn.feature_selection import SelectKBest\n", "from sklearn.cross_validation import StratifiedKFold\n", "from sklearn.grid_search import GridSearchCV\n", "from sklearn.ensemble.gradient_boosting import GradientBoostingClassifier\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.ensemble import ExtraTreesClassifier\n", "from sklearn import preprocessing\n", "from sklearn.cross_validation import cross_val_score" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "05623356-f7e5-f0e3-45b2-76464e5e6f34" }, "source": [ "Data Loading\n", "------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "df8ae86e-2d2d-3f48-8235-ac7503cd8adf" }, "outputs": [], "source": [ "base_folder = '../input/'\n", "data = pd.read_csv(base_folder + 'train.csv')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "db782418-bdd7-c69f-d763-a5c6b1d4a664" }, "outputs": [], "source": [ "data.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "f53e00b3-7281-5767-a6af-679628b0f07d" }, "source": [ "## Categorical/String variables:\n", "* Name\n", "* Sex\n", "* Ticket\n", "* Cabin\n", "* Embarked\n", "\n", "## Numerical variables:\n", "* PassengerId\n", "* Pclass\n", "* Age\n", "* SibSp\n", "* Parch\n", "* Fare" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "94c3d1b2-0923-9230-97bc-26b7f1d54f60" }, "outputs": [], "source": [ "data.describe()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a93b9cf7-bceb-e8a4-442f-a23b3f95f7c4" }, "source": [ "* From all the numerical variables Age has missing values in training set" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "ad3a2e54-6ede-bc6f-fc85-32988e7f1e58" }, "source": [ "# Sensible Value Imputation" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "950d883e-abac-3a9e-5394-b00bdf52634a" }, "source": [ "* There are many ways for filling missing values, just filling in the median of the values for now." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "7fc30eab-512b-afcc-cb0e-df9cfc04cab9" }, "outputs": [], "source": [ "# filled the empty age with median value of age\n", "data['Age'].fillna(data['Age'].median(), inplace=True)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "bf9f7937-6ea8-3a45-75b8-68f104dc09ce" }, "source": [ "# Explore Data" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "fc8fb8ab-c3f6-52b0-9f18-c8473345bb3a" }, "source": [ "### Sex versus survival" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "182dc33a-560f-bea9-2c06-559654772c2a" }, "outputs": [], "source": [ "survived_sex = data[data['Survived']==1]['Sex'].value_counts()\n", "dead_sex = data[data['Survived']==0]['Sex'].value_counts()\n", "#plot the survived male , female and dead male,female\n", "df = pd.DataFrame([survived_sex,dead_sex])\n", "df.index = ['Survived','Dead']\n", "df.plot(kind='bar', figsize=(15,8))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "2f23f92a-7881-3ccc-e495-a079588c9c7a" }, "source": [ "* Can be clearly seen from the above plot, males did not stand much of a chance :)\n", "* % of male survival seems much less than female survival" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4ad47786-2630-bfab-1367-1f8307bf9cee" }, "source": [ "### Age versus survival" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "48824ffd-b426-31f1-7bb2-edac91e618d6" }, "outputs": [], "source": [ "# dead and survived based on age of people\n", "figure = plt.figure(figsize=(15,8))\n", "plt.hist([data[data['Survived']==1]['Age'],data[data['Survived']==0]['Age']], color = ['g','r'],\n", " bins = 10,label = ['Survived','Dead'])\n", "plt.xlabel('Age')\n", "plt.ylabel('Number of passengers')\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7251ec76-e3e7-0b82-aa90-b36321cfb64c" }, "source": [ "* Those in the range 20-40 are more likely to be dead.\n", "* Those in the range 70-80 are almost always dead.\n", "* 0-20 there is not much diff i think\n", "* Making these as features would be a good idea?" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "c4a8a33d-b04b-aacf-1622-2b874f52d99a" }, "source": [ "### Fare versus survival" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "57cdcad5-734e-f01d-c5a4-02dccca5f451" }, "outputs": [], "source": [ "# plotting number of survivors based on the fare they gave\n", "figure = plt.figure(figsize=(15,8))\n", "plt.hist([data[data['Survived']==1]['Fare'],data[data['Survived']==0]['Fare']], color = ['g','r'],\n", " bins = 10,label = ['Survived','Dead'])\n", "plt.xlabel('Fare')\n", "plt.ylabel('Number of passengers')\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a5337384-e7bb-fbb5-886e-aa0ca8ebdab6" }, "source": [ "* Not exactly sure whether making <50 a feature will be a good idea? Although people less than 50 have high death rate!!\n", "* But over the complete data set we cannot say anything from the fare alone" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "cc14e4c8-35ab-5db6-de29-59e5ad21740f" }, "source": [ "### Age versus Fare\n", "* Survived\n", "* Not Survied" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "06367424-ac5e-580d-251a-a8ebdb1e37dc" }, "outputs": [], "source": [ "plt.figure(figsize=(15,8))\n", "ax = plt.subplot()\n", "ax.scatter(data[data['Survived']==1]['Age'],data[data['Survived']==1]['Fare'],c='green',s=40)\n", "ax.scatter(data[data['Survived']==0]['Age'],data[data['Survived']==0]['Fare'],c='red',s=40)\n", "ax.set_xlabel('Age')\n", "ax.set_ylabel('Fare')\n", "ax.legend(('survived','dead'),scatterpoints=1,loc='upper right',fontsize=20,)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "197f1f76-3065-85bb-6e10-ab7878b837c1" }, "source": [ "* Now i know that individually age between 20-40 are killed more.\n", "* Also i know individually those with lower fares are also killed more." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "600dfc8f-d5fb-8f1c-4566-4f64bd4e44d6" }, "source": [ "### Pclass versus survival" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "8631fe25-e90b-14f5-63d5-75d0ac12f8aa" }, "outputs": [], "source": [ "plt.figure(figsize=(15,8))\n", "ax = plt.subplot()\n", "ax.set_ylabel('Survived')\n", "ax.set_xlabel('Pclass')\n", "ax.hist([data[data['Survived']==1]['Pclass'],data[data['Survived']==0]['Pclass']],color = ['g','r'],)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "236c16c1-72be-5863-1724-190f22b014f9" }, "source": [ "* So from the above we see pclass3 is mostly dead. Class might correspond to the status here!?\n", "* That we can check from the fare they paid for a particular class!" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d6bce783-ea4e-fce1-fbdf-37134a9eb468" }, "source": [ "### Pclass versus fare" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "8da880bd-97ff-0c2d-7292-aae7de55183b" }, "outputs": [], "source": [ "# Plotting how fares versus pclass goes?\n", "ax = plt.subplot()\n", "ax.set_ylabel('Average fare')\n", "# we are plotting the mean cause the mean would show overall co-relation \n", "#rather than indivisual data points which may be unclear\n", "data.groupby('Pclass').mean()['Fare'].plot(kind='bar',figsize=(15,8), ax = ax)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "6e9c3368-9d63-559f-1e4e-deaf80f76ba1" }, "source": [ "### Embarkment versus survival" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "b8b0772a-794a-4920-d3e4-3382ae243396" }, "outputs": [], "source": [ "survived_embark = data[data['Survived']==1]['Embarked'].value_counts()\n", "dead_embark = data[data['Survived']==0]['Embarked'].value_counts()\n", "df = pd.DataFrame([survived_embark,dead_embark])\n", "df.index = ['Survived','Dead']\n", "df.plot(kind='bar',stacked=True, figsize=(15,8))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "f7dbb864-e723-96fe-a9e5-159dc229f54c" }, "source": [ "# Feature Engineering" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5c6441b0-be62-e018-d7d2-75cda7c99271" }, "outputs": [], "source": [ "# Combining both the test and trainig data so that all the manipulations which are done\n", "# happen on both the data sets.\n", "# Also if test set has any missing values, it will easily come to notice here\n", "def get_combined_data():\n", " train = pd.read_csv(base_folder + 'train.csv')\n", " test = pd.read_csv(base_folder + 'test.csv')\n", " # extracting and then removing the targets from the training data \n", " targets = train.Survived\n", " train.drop('Survived',1,inplace=True)\n", " \n", " combined = train.append(test)\n", " combined.reset_index(inplace=True)\n", " combined.drop('index',inplace=True,axis=1)\n", " return combined\n", "combined = get_combined_data()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "08392880-b717-d144-cd9d-3d4aa38f5370" }, "outputs": [], "source": [ "combined.info()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "fbd0c32a-fd8d-844c-2cbd-9013dd83d9d0" }, "source": [ "* We can see some variables have missing values\n", " * Fare\n", " * Cabin\n", " * Embarked\n", " * Age" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d17f9551-a838-2d3b-eb82-470eca552f4e" }, "source": [ "# Sensible Value Imputation" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "163a8cf7-5679-debe-20df-4005ff267406" }, "source": [ "* Filling missing values in both the test and the train data from those calculated from the training data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e17df5cc-45f6-2af6-5e92-0cf2d253ea45" }, "outputs": [], "source": [ "combined.Cabin.fillna('U',inplace=True)\n", "combined.Embarked.fillna('S',inplace=True)\n", "combined.Fare.fillna(data.Fare.mean(),inplace=True)\n", "combined.Age.fillna(data.Age.median(), inplace=True)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "cbe3aa76-1ccf-ab75-5477-522d4dc2f094" }, "source": [ "### Family Size feature" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d34834b2-d383-59df-a1ef-c9ba1e190291" }, "outputs": [], "source": [ "# The size of families (including the passenger)\n", "combined['FamilySize'] = combined['Parch'] + combined['SibSp'] + 1\n", "# Introducing other features based on the family size\n", "combined['Alone'] = combined['FamilySize'].map(lambda s : 1 if s == 1 else 0)\n", "combined['Small'] = combined['FamilySize'].map(lambda s : 1 if 2<=s<=4 else 0)\n", "combined['Large'] = combined['FamilySize'].map(lambda s : 1 if 5<=s else 0)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "4e5d6148-3b2d-12f7-4227-c490160aa4f9" }, "outputs": [], "source": [ "# The size of families on the training set\n", "data['FamilySize'] = data['Parch'] + data['SibSp'] + 1\n", "plt.figure(figsize=(15,8))\n", "ax = plt.subplot()\n", "ax.set_ylabel('Survived')\n", "ax.set_xlabel('FamilySize')\n", "ax.hist([data[data['Survived']==1]['FamilySize'],data[data['Survived']==0]['FamilySize']],color = ['g','r'],)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "921caaed-8e4f-ae4f-a17d-d791c1460333" }, "source": [ "* Seems like.. if you are alone: you are dead!" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b1a93c47-2ea1-c158-4f94-850a35a4dc93" }, "source": [ "### Title feature" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a4879438-234a-f123-6628-2874bd7307af" }, "outputs": [], "source": [ "if 'Title' not in combined.columns:\n", " combined['Title'] = combined['Name'].map(lambda name:name.split(',')[1].split('.')[0].strip())\n", " Title_Dictionary = {\n", " \"Capt\": \"Officer\",\n", " \"Col\": \"Officer\",\n", " \"Major\": \"Officer\",\n", " \"Jonkheer\": \"Royalty\",\n", " \"Don\": \"Royalty\",\n", " \"Sir\" : \"Royalty\",\n", " \"Dr\": \"Officer\",\n", " \"Rev\": \"Officer\",\n", " \"the Countess\":\"Royalty\",\n", " \"Dona\": \"Royalty\",\n", " \"Mme\": \"Mrs\",\n", " \"Mlle\": \"Miss\",\n", " \"Ms\": \"Mrs\",\n", " \"Mr\" : \"Mr\",\n", " \"Mrs\" : \"Mrs\",\n", " \"Miss\" : \"Miss\",\n", " \"Master\" : \"Master\",\n", " \"Lady\" : \"Royalty\"\n", "\n", " }\n", " combined['Title'] = combined.Title.map(Title_Dictionary)\n", " combined.drop('Name',axis=1,inplace=True)\n", " titles_dummies = pd.get_dummies(combined['Title'],prefix='Title')\n", " combined.drop('Title',axis=1,inplace=True)\n", " combined = pd.concat([combined,titles_dummies],axis=1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1e372e8f-3f7b-40d9-01fb-e92c98b5d247" }, "outputs": [], "source": [ "data['Title'] = data['Name'].map(lambda name:name.split(',')[1].split('.')[0].strip())\n", "data['Title'] = data.Title.map(Title_Dictionary)\n", "data = pd.concat([data,pd.get_dummies(data['Title'],prefix='Title')],axis=1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "3c466d64-905a-df80-6d38-f19fc9dc029e" }, "outputs": [], "source": [ "plt.figure(figsize=(15,8))\n", "ax = plt.subplot()\n", "ax.set_ylabel('Survived')\n", "ax.set_xlabel('Titles')\n", "ax.hist([data[data['Survived']==1]['Title_Officer'],\n", " data[data['Survived']==0]['Title_Officer']\n", " ],color = ['g','r'],)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "894a1372-f637-d45b-77d8-b140988193ef" }, "source": [ "* It can be seen that people with better titles survived more! This may be a good feature." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4ae26836-77da-8073-2d0c-1c2d9cea8141" }, "source": [ "### Adding some categorical features" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "cbec7987-a628-2d4c-e2d5-4d08bf1ce3bc" }, "outputs": [], "source": [ "# new columns m planning to create are age ranges\n", "# 10-20, 20-30 something like that\n", "combined['20-40'] = combined['Age'].apply(lambda x: 1 if x>=20 and x<=40 else 0)\n", "combined['70-80'] = combined['Age'].apply(lambda x: 1 if x>=70 and x<=80 else 0)\n", "combined['below-80'] = combined['Fare'].apply(lambda x: 1 if x<80 else 0)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "597a12ef-ac43-5df2-d028-194749e85030" }, "source": [ "### Categorical to one-hot encoding" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "8a5e3ba0-480f-d885-57dc-5a525730f531" }, "outputs": [], "source": [ "def get_one_hot_encoding(dt, features):\n", " for feature in features:\n", " if feature in dt.columns:\n", " dummies = pd.get_dummies(dt[feature],prefix=feature)\n", " dt = pd.concat([dt,dummies],axis=1)\n", " return dt" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2f559e92-cbb2-8ff0-66e9-5026dec3bd89" }, "outputs": [], "source": [ "combined = get_one_hot_encoding(combined,['Embarked','Cabin','Pclass','Embarked','Title'])\n", "combined['Sex'] = combined['Sex'].map({'male':0,'female':1})\n", "combined.drop(['Embarked','Cabin','Pclass','Embarked','Title'],inplace=True,axis=1)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "2e22ded2-fded-75d9-a7f5-012bebf37e0d" }, "source": [ "### Ticket feature extraction" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "72ef14ee-cc15-1971-33d2-e2bc96d891af" }, "outputs": [], "source": [ "def cleanTicket(ticket):\n", " ticket = ticket.replace('.','')\n", " ticket = ticket.replace('/','')\n", " ticket = ticket.split()\n", " ticket = map(lambda t : t.strip() , ticket)\n", " ticket = filter(lambda t : not t.isdigit(), ticket)\n", " ticket = list(ticket)\n", " if (len(ticket)) > 0:\n", " return ticket[0]\n", " else: \n", " return 'XXX'\n", "\n", "combined['Ticket'] = combined['Ticket'].map(cleanTicket)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "18a257a8-1636-ab1e-2a72-17e161a5040b" }, "outputs": [], "source": [ "combined = get_one_hot_encoding(combined,'Ticket')\n", "combined.drop('Ticket',axis=1,inplace=True)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3dbbe755-0ed0-6007-04bd-4241f30f0b75" }, "source": [ "# Normalise" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ea75a8c6-ef27-4bd7-4176-e643989746b2" }, "outputs": [], "source": [ "columns = combined.columns\n", "combined_new = pd.DataFrame(preprocessing.normalize(combined, axis=0, copy=True), columns=columns)\n", "combined_new['PassengerId'] = combined['PassengerId']\n", "combined = combined_new" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "7f8a91be-f26a-5fa1-e3bd-0b72ce7897d7" }, "outputs": [], "source": [ "combined.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "ba088ade-66c3-23cf-3b1c-f8677b1d7071" }, "source": [ "# Recover" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2e1202e1-d2b5-51e5-bb83-e2a3bd6d23b6" }, "outputs": [], "source": [ "train0 = pd.read_csv(base_folder + 'train.csv')\n", "targets = train0.Survived\n", "train = combined[0:891]\n", "test = combined[891:]" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "e3c5a0f5-da58-6f31-487b-2075dc3609be" }, "source": [ "# Feature Selection" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "57742055-56a9-b3a4-54a2-64a23d62bdb6" }, "outputs": [], "source": [ "clf = ExtraTreesClassifier(n_estimators=200)\n", "clf = clf.fit(train, targets)\n", "features = pd.DataFrame()\n", "features['feature'] = train.columns\n", "features['importance'] = clf.feature_importances_\n", "cols = features.sort(['importance'],ascending=False)['feature']\n", "model = SelectFromModel(clf, prefit=True)\n", "train_new = model.transform(train)\n", "test_new = model.transform(test)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a0cc3895-0209-8845-6320-06911e8607fd" }, "outputs": [], "source": [ "cols" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2bc927df-167d-0aab-f6b9-bbedc7be9105" }, "outputs": [], "source": [ "train_new.shape" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "68ed3caf-165c-7ead-6314-3784a97d2fc4" }, "source": [ "# Models" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e49e3d51-5cc8-9415-b455-43d3e358ca4a" }, "outputs": [], "source": [ "forest = RandomForestClassifier(max_features='sqrt')\n", "\n", "parameter_grid = {\n", " 'max_depth' : [4,5,6,7,8,9],\n", " 'n_estimators': [100, 200,210,240,250],\n", " 'criterion': ['gini','entropy']\n", " }\n", "\n", "cross_validation = StratifiedKFold(targets, n_folds=5)\n", "\n", "grid_search = GridSearchCV(forest,\n", " param_grid=parameter_grid,\n", " cv=cross_validation)\n", "\n", "grid_search.fit(train_new, targets)\n", "\n", "print('Best score: {}'.format(grid_search.best_score_))\n", "print('Best parameters: {}'.format(grid_search.best_params_))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "4b8e12e9-ce9e-22b3-5bb4-cfd85c36c494" }, "outputs": [], "source": [ "ext = ExtraTreesClassifier()\n", "\n", "parameter_grid = {\n", " 'max_depth' : [4,5,6,7,8,9],\n", " 'n_estimators': [100, 200,210,240,250],\n", " 'criterion': ['gini','entropy']\n", " }\n", "\n", "cross_validation = StratifiedKFold(targets, n_folds=5)\n", "\n", "grid_search = GridSearchCV(ext,\n", " param_grid=parameter_grid,\n", " cv=cross_validation)\n", "\n", "grid_search.fit(train_new, targets)\n", "\n", "print('Best score: {}'.format(grid_search.best_score_))\n", "print('Best parameters: {}'.format(grid_search.best_params_))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9e4cecbd-9401-0d2c-50f8-fbbaaba2adc0" }, "outputs": [], "source": [ "lr = LogisticRegression(penalty='l2')\n", "\n", "parameter_grid = {\n", " 'tol' : [0.1,0.01,0.001,10,1],\n", " 'max_iter': [100, 200,210,240,250],\n", " }\n", "\n", "cross_validation = StratifiedKFold(targets, n_folds=5)\n", "\n", "grid_search = GridSearchCV(lr,\n", " param_grid=parameter_grid,\n", " cv=cross_validation)\n", "\n", "grid_search.fit(train_new, targets)\n", "\n", "print('Best score: {}'.format(grid_search.best_score_))\n", "print('Best parameters: {}'.format(grid_search.best_params_))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "55225cb3-0ea1-a02f-fc37-c492d407bf91" }, "outputs": [], "source": [ "from sklearn.ensemble import AdaBoostClassifier\n", "adaboost = AdaBoostClassifier(n_estimators=100)\n", "\n", "cross_validation = StratifiedKFold(targets, n_folds=5)\n", "adaboost.fit(train_new, targets)\n", "\n", "print('Best score: {}'.format(cross_val_score(adaboost,train_new,targets,cv=10)))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "8d457fd8-0b6b-db21-624e-c9d91b060089" }, "source": [ "# Voting " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a436cc8c-e7c9-5dad-fc5b-5935a5096535" }, "outputs": [], "source": [ "from sklearn.ensemble import VotingClassifier\n", "eclf1 = VotingClassifier(estimators=[\n", " ('rf', forest),('etc',ext),('lr', lr), ('adb', adaboost)], voting='soft',\n", " weights=[2,1,1,1])\n", "eclf1 = eclf1.fit(train_new, targets)\n", "predictions=eclf1.predict(test_new)\n", "predictions\n", "\n", "test_predictions=eclf1.predict(test_new)\n", "test_predictions=test_predictions.astype(int)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a21504f6-81e5-6559-0a81-39f881dfca7d" }, "outputs": [], "source": [ "test_predictions = eclf1.predict(test_new)\n", "df_output = pd.DataFrame()\n", "df_output['PassengerId'] = test['PassengerId']\n", "df_output['Survived'] = test_predictions\n", "df_output[['PassengerId','Survived']].to_csv('output.csv',index=False)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "af7fb048-15f3-a040-6cd7-e979cdcddb79" }, "source": [ "# Ensembling and Stacking" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ca656df1-2cc9-f7a3-8e14-d9930ddfa3de" }, "outputs": [], "source": [ "# Put in our parameters for said classifiers\n", "# Random Forest parameters\n", "rf_params = {\n", " 'n_jobs': -1,\n", " 'n_estimators': 500,\n", " 'warm_start': True, \n", " #'max_features': 0.2,\n", " 'max_depth': 6,\n", " 'min_samples_leaf': 2,\n", " 'max_features' : 'sqrt',\n", " 'verbose': 0\n", "}\n", "\n", "# Extra Trees Parameters\n", "et_params = {\n", " 'n_jobs': -1,\n", " 'n_estimators':500,\n", " #'max_features': 0.5,\n", " 'max_depth': 8,\n", " 'min_samples_leaf': 2,\n", " 'verbose': 0\n", "}\n", "\n", "# AdaBoost parameters\n", "ada_params = {\n", " 'n_estimators': 500,\n", " 'learning_rate' : 0.75\n", "}\n", "\n", "# Gradient Boosting parameters\n", "gb_params = {\n", " 'n_estimators': 500,\n", " #'max_features': 0.2,\n", " 'max_depth': 5,\n", " 'min_samples_leaf': 2,\n", " 'verbose': 0\n", "}\n", "\n", "# Support Vector Classifier parameters \n", "svc_params = {\n", " 'kernel' : 'linear',\n", " 'C' : 0.025\n", " }\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "358ab7fc-4c4c-9153-fe34-4badc9450ffc" }, "outputs": [], "source": [ "NFOLDS = 5\n", "kf = StratifiedKFold(targets,n_folds= NFOLDS)\n", "\n", "# Class to extend the Sklearn classifier\n", "class SklearnHelper(object):\n", " def __init__(self, clf, seed=0, params=None):\n", " params['random_state'] = seed\n", " self.clf = clf(**params)\n", "\n", " def train(self, x_train, y_train):\n", " self.clf.fit(x_train, y_train)\n", "\n", " def predict(self, x):\n", " return self.clf.predict(x)\n", " \n", " def fit(self,x,y):\n", " return self.clf.fit(x,y)\n", " \n", " def feature_importances(self,x,y):\n", " return self.clf.fit(x,y).feature_importances_" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "135aad14-062b-8042-45d8-1c62adbec465" }, "outputs": [], "source": [ "# Create 5 objects that represent our 4 models\n", "SEED=0\n", "\n", "rf = SklearnHelper(clf=RandomForestClassifier, seed=SEED, params=rf_params)\n", "et = SklearnHelper(clf=ExtraTreesClassifier, seed=SEED, params=et_params)\n", "ada = SklearnHelper(clf=AdaBoostClassifier, seed=SEED, params=ada_params)\n", "gb = SklearnHelper(clf=GradientBoostingClassifier, seed=SEED, params=gb_params)\n", "svc = SklearnHelper(clf=SVC, seed=SEED, params=svc_params)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "2331a040-897b-9d13-c08a-7f6432298baa" }, "source": [ "### Out of fold predictions\n", "* First of all.. enumerate(kf). What does this do?\n", "* This function returns indices of the 90%train and 10%test set which it generated from each fold of the same training set data.\n", "* Using that data we can split the overall training data itself in to traing and test set for each fold... and then utilize it for generating our k fold results if required.\n", "\n", "* calculates mean predicts over the i folds in the test set\n", "* calculates predictions over all folds of the training set\n", "* finally returns both the predictions" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "36de685e-cd07-8d48-0f87-25b9eb8f91b7" }, "outputs": [], "source": [ "def get_oof(clf, x_train, y_train, x_test):\n", " ntrain = train.shape[0]\n", " ntest = test.shape[0]\n", " oof_train = np.zeros((ntrain,))\n", " oof_test = np.zeros((ntest,))\n", " oof_test_skf = np.empty((NFOLDS, ntest))\n", "\n", " for i, (train_index, test_index) in enumerate(kf):\n", " # get the training of fold number i from training set\n", " x_tr = train_new[train_index]\n", " # get the targets of fold i from training set\n", " y_tr = targets[train_index]\n", " # get the remaining 10% test set from the ith fold \n", " x_te = train_new[test_index]\n", "\n", " # train the classifier on the training set\n", " clf.train(x_tr, y_tr)\n", " \n", " # store results of predictions over the ith test set at proper locations\n", " # oof_train will contain all the predictions over the test set once all n_fold iterations are over\n", " oof_train[test_index] = clf.predict(x_te)\n", " # over the complete test set classifier trained so far will predict\n", " # ith entry of oof_test_skf will contain predictions from classifier trained till ith fold\n", " oof_test_skf[i, :] = clf.predict(x_test)\n", "\n", " # calculate mean of all the predictions done in the i folds and store them as final results in oof_test\n", " oof_test[:] = oof_test_skf.mean(axis=0)\n", " # predictions on training set, mean predictions on the test set\n", " return oof_train.reshape(-1, 1), oof_test.reshape(-1, 1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "86cd31d2-6120-3333-29f3-0192e815d378" }, "outputs": [], "source": [ "# Create our OOF train and test predictions. These base results will be used as new features\n", "et_oof_train, et_oof_test = get_oof(et, train_new, targets, test_new) # Extra Trees\n", "rf_oof_train, rf_oof_test = get_oof(rf,train_new, targets, test_new) # Random Forest\n", "ada_oof_train, ada_oof_test = get_oof(ada, train_new, targets, test_new) # AdaBoost \n", "gb_oof_train, gb_oof_test = get_oof(gb,train_new, targets, test_new) # Gradient Boost\n", "svc_oof_train, svc_oof_test = get_oof(svc,train_new, targets, test_new) # Support Vector Classifier\n", "\n", "print(\"Training is complete\")" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "ffdf1e91-0fea-73ff-c363-00e76a498df7" }, "source": [ "### Feature Importances" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "8bcf185e-e4f3-6911-b20b-c185584eed01" }, "outputs": [], "source": [ "rf_feature = rf.feature_importances(train_new,targets)\n", "et_feature = et.feature_importances(train_new, targets)\n", "ada_feature = ada.feature_importances(train_new, targets)\n", "gb_feature = gb.feature_importances(train_new,targets)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "030b1c1f-9d4f-b8a2-2a05-7b216271bf17" }, "source": [ "### Second Level Dataframe" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "b1fac48b-b65c-dba5-c562-4b7ea4af7b41" }, "outputs": [], "source": [ "cols_new = cols.values[0:25]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "18bd050e-66db-055f-cb59-9d9ce1017e15" }, "outputs": [], "source": [ "# Create a dataframe with features\n", "feature_dataframe = pd.DataFrame( {'features': cols_new,\n", " 'Random Forest feature importances': rf_feature,\n", " 'Extra Trees feature importances': et_feature,\n", " 'AdaBoost feature importances': ada_feature,\n", " 'Gradient Boost feature importances': gb_feature\n", " })" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ad770f64-5f8a-9fc4-ddd1-f0f5683e6b64" }, "outputs": [], "source": [ "# The final dataframe\n", "feature_dataframe.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a76c9f67-1b31-e43c-652c-80fb63a19ee9" }, "source": [ "* stored the predictions on training set of various classifiers into flattened array" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "dd4d1123-a810-d029-0332-d2ca10ff7090" }, "outputs": [], "source": [ "base_predictions_train = pd.DataFrame( {'RandomForest': rf_oof_train.ravel(),\n", " 'ExtraTrees': et_oof_train.ravel(),\n", " 'AdaBoost': ada_oof_train.ravel(),\n", " 'GradientBoost': gb_oof_train.ravel()\n", " })\n", "base_predictions_train.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "df45649b-e386-4ed9-7bbc-6222e59e80d2" }, "source": [ "### Visualise co-relation between classifiers\n", "* It has been seen that, lesser the co-relation, better the predictions from second level predictor" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d9f81b75-293e-9f6c-3259-448d2bba375b" }, "outputs": [], "source": [ "import plotly.graph_objs as go\n", "import plotly.offline as py\n", "py.init_notebook_mode(connected=True)\n", "data = [\n", " go.Heatmap(\n", " z= base_predictions_train.astype(float).corr().values ,\n", " x=base_predictions_train.columns.values,\n", " y= base_predictions_train.columns.values,\n", " colorscale='Portland',\n", " showscale=True,\n", " reversescale = True\n", " )\n", "]\n", "py.iplot(data, filename='labelled-heatmap')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "45c0ec2b-4914-8f6d-abc4-d46f9e1dfcfb" }, "source": [ "### Setup level 2" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "fbb9e72a-f2db-1b67-be71-93a26627fd1c" }, "outputs": [], "source": [ "#converted into a single array of training set(891) X 4 columns(number of classifiers)\n", "x_train = np.concatenate(( et_oof_train, rf_oof_train, ada_oof_train, gb_oof_train, svc_oof_train), axis=1)\n", "x_test = np.concatenate(( et_oof_test, rf_oof_test, ada_oof_test, gb_oof_test, svc_oof_test), axis=1)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "42ea89b6-6c5a-6614-fbcf-b0beecdc5061" }, "source": [ "#### FYI\n", "* x_train will contain predictions from all the classifiers\n", "* features for training for decision tree below is predictions of various classifiers\n", "* x_test contains the predictions on the test set " ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "359b77e8-b1f5-e20b-dcae-cebc1cf325d3" }, "source": [ "### Stack predictor" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "84868e43-086c-d5e9-d02f-052f335d2518" }, "outputs": [], "source": [ "from sklearn import tree\n", "clf = tree.DecisionTreeClassifier(max_depth=10,max_features='sqrt').fit(x_train, targets)\n", "predictions = clf.predict(x_test)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9cf82175-3efa-1395-577f-2b7efa974c0b" }, "outputs": [], "source": [ "# Just throw these to kaggle output :)\n", "predictions" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a314c35c-9ec8-572f-f7f2-ac3499a69eb4" }, "source": [ "# Output!!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d3325dc9-fc18-1d8c-e7ca-7a7b61ac6fcf" }, "outputs": [], "source": [ "%%bash\n", "kg submit output.csv -c titanic -u sp4658@nyu.edu -p **** -m \"voting classifier\"" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3a1c669a-6839-d34c-db19-e50d8ea93a2b" }, "source": [ "# Improvements" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "06518c04-fb7b-c1a2-d791-fa28248f8a3c" }, "source": [ "* A lot can be done in Missing Value Imputation\n", "* Wanted to try blending and other cool stuff!!\n", "* This is my first Kernel, i wanted to learn and try out many different stuff. Please advice with improvements." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "961f3d9d-7ade-399a-d695-78da326ce72f" }, "source": [ "# References\n", "* [Blending][1]\n", "* [Stacking][2]\n", "* [Ensembling][3]\n", "* [Voting][4]\n", "* [Stacking/Ensembling guide][5]\n", "* [Random Forest][6]\n", "* Lot more of them....\n", "\n", "\n", " [1]: https://github.com/emanuele/kaggle_pbr/blob/master/blend.py\n", " [2]: http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice\n", " [3]: http://mlwave.com/kaggle-ensembling-guide/\n", " [4]: https://www.kaggle.com/poonaml/titanic/titanic-survival-prediction-end-to-end-ml-pipeline\n", " [5]: https://www.kaggle.com/shivendra91/titanic/introduction-to-ensembling-stacking-in-python/editnb\n", " [6]: https://www.kaggle.com/benhamner/titanic/random-forest-benchmark-r/code" ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }