{"cells":[{"metadata":{"_cell_guid":"ea24845a-4a15-4235-9ed2-4823acbf4317","_uuid":"89022d89e1603ef3869a96b9da78da38b0b33b14"},"cell_type":"markdown","source":"\n>Hello and Welcome to Kaggle, the online Data Science Community to learn, share, and compete. Most beginners get lost in the field, because they fall into the black box approach, using libraries and algorithms they don't understand. This tutorial will give you a 1-2-year head start over your peers, by providing a framework that teaches you how-to think like a data scientist vs what to think/code. Not only will you be able to submit your first competition, but you’ll be able to solve any problem thrown your way. I provide clear explanations, clean code, and plenty of links to resources. *Please Note: This Kernel is still being improved. So check the Change Logs below for updates. Also, please be sure to upvote, fork, and comment and I'll continue to develop.* Thanks, and may you have \"statistically significant\" luck!\n\n\n# Table of Contents\n1. [Chapter 1 - How a Data Scientist Beat the Odds](#ch1)\n1. [Chapter 2 - A Data Science Framework](#ch2)\n1. [Chapter 3 - Step 1: Define the Problem and Step 2: Gather the Data](#ch3)\n1. [Chapter 4 - Step 3: Prepare Data for Consumption](#ch4)\n1. [Chapter 5 - The 4 C's of Data Cleaning: Correcting, Completing, Creating, and Converting](#ch5)\n1. [Chapter 6 - Step 4: Perform Exploratory Analysis with Statistics](#ch6)\n1. [Chapter 7 - Step 5: Model Data](#ch7)\n1. [Chapter 8 - Evaluate Model Performance](#ch8)\n1. [Chapter 9 - Tune Model with Hyper-Parameters](#ch9)\n1. [Chapter 10 - Tune Model with Feature Selection](#ch10)\n1. [Chapter 11 - Step 6: Validate and Implement](#ch11)\n1. [Chapter 12 - Conclusion and Step 7: Optimize and Strategize](#ch12)\n1. [Change Log](#ch90)\n1. [Credits](#ch91)\n\n**How-to Use this Tutorial:** Read the explanations provided in this Kernel and the links to developer documentation. The goal is to not just learn the whats, but the whys. If you don't understand something in the code the print() function is your best friend. In coding, it's okay to try, fail, and try again. If you do run into problems, Google is your second best friend, because 99.99% of the time, someone else had the same question/problem and already asked the coding community. If you've exhausted all your resources, the Kaggle Community via forums and comments can help too."},{"metadata":{"_cell_guid":"ec31d549-537e-4f35-8594-22ef03079081","_uuid":"ff6c70c884a8a5fb3258e7f3fefc9e55261df1df"},"cell_type":"markdown","source":"<a id=\"ch1\"></a>\n# How a Data Scientist Beat the Odds\nIt's the classical problem, predict the outcome of a binary event. In laymen terms this means, it either occurred or did not occur. For example, you won or did not win, you passed the test or did not pass the test, you were accepted or not accepted, and you get the point. A common business application is churn or customer retention. Another popular use case is, healthcare's mortality rate or survival analysis. Binary events create an interesting dynamic, because we know statistically, a random guess should achieve a 50% accuracy rate, without creating one single algorithm or writing one single line of code. However, just like autocorrect spellcheck technology, sometimes we humans can be too smart for our own good and actually underperform a coin flip. In this kernel, I use Kaggle's Getting Started Competition, Titanic: Machine Learning from Disaster, to walk the reader through, how-to use the data science framework to beat the odds.\n   \n*What happens when technology is too smart for its own good?*\n![Funny Autocorrect](http://15858-presscdn-0-65.pagely.netdna-cdn.com/wp-content/uploads/2016/03/hilarious-autocorrect-fails-20x.jpg)"},{"metadata":{"_cell_guid":"8514a868-6816-494a-9e9c-dba14cb36f42","_uuid":"bd2a68dcaace0c1371f2994f0204dce65800cc20"},"cell_type":"markdown","source":"<a id=\"ch2\"></a>\n# A Data Science Framework\n1. **Define the Problem:** If data science, big data, machine learning, predictive analytics, business intelligence, or any other buzzword is the solution, then what is the problem? As the saying goes, don't put the cart before the horse. Problems before requirements, requirements before solutions, solutions before design, and design before technology. Too often we are quick to jump on the new shiny technology, tool, or algorithm before determining the actual problem we are trying to solve.\n2. **Gather the Data:** John Naisbitt wrote in his 1984 (yes, 1984) book Megatrends, we are “drowning in data, yet staving for knowledge.\" So, chances are, the dataset(s) already exist somewhere, in some format. It may be external or internal, structured or unstructured, static or streamed, objective or subjective, etc. As the saying goes, you don't have to reinvent the wheel, you just have to know where to find it. In the next step, we will worry about transforming \"dirty data\" to \"clean data.\"\n3. **Prepare Data for Consumption:** This step is often referred to as data wrangling, a required process to turn “wild” data into “manageable” data. Data wrangling includes implementing data architectures for storage and processing, developing data governance standards for quality and control, data extraction (i.e. ETL and web scraping), and data cleaning to identify aberrant, missing, or outlier data points.\n4. **Perform Exploratory Analysis:** Anybody who has ever worked with data knows, garbage-in, garbage-out (GIGO). Therefore, it is important to deploy descriptive and graphical statistics to look for potential problems, patterns, classifications, correlations and comparisons in the dataset. In addition, data categorization (i.e. qualitative vs quantitative) is also important to understand and select the correct hypothesis test or data model.\n5. **Model Data:** Like descriptive and inferential statistics, data modeling can either summarize the data or predict future outcomes. Your dataset and expected results, will determine the algorithms available for use. It's important to remember, algorithms are tools and not magical wands or silver bullets. You must still be the master craft (wo)man that knows how-to select the right tool for the job. An analogy would be asking someone to hand you a Philip screwdriver, and they hand you a flathead screwdriver or worst a hammer. At best, it shows a complete lack of understanding. At worst, it makes completing the project impossible. The same is true in data modelling. The wrong model can lead to poor performance at best and the wrong conclusion (that’s used as actionable intelligence) at worst.\n6. **Validate and Implement Data Model:** After you've trained your model based on a subset of your data, it's time to test your model. This helps ensure you haven't overfit your model or made it so specific to the selected subset, that it does not accurately fit another subset from the same dataset. In this step we determine if our [model overfit, generalize, or underfit our dataset](http://docs.aws.amazon.com/machine-learning/latest/dg/model-fit-underfitting-vs-overfitting.html).\n7. **Optimize and Strategize:** This is the \"bionic man\" step, where you iterate back through the process to make it better...stronger...faster than it was before. As a data scientist, your strategy should be to outsource developer operations and application plumbing, so you have more time to focus on recommendations and design. Once you're able to package your ideas, this becomes your “currency exchange\" rate."},{"metadata":{"_cell_guid":"22ba49b5-b3b6-4129-8c57-623e0d863ab3","_uuid":"ff014109d795c5d1bfcc66c9761730406124ebe4"},"cell_type":"markdown","source":"<a id=\"ch3\"></a>\n# Step 1: Define the Problem\nFor this project, the problem statement is given to us on a golden plater, develop an algorithm to predict the survival outcome of passengers on the Titanic.\n\n......\n\n**Project Summary:**\nThe sinking of the RMS Titanic is one of the most infamous shipwrecks in history.  On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 passengers and crew. This sensational tragedy shocked the international community and led to better safety regulations for ships.\n\nOne of the reasons that the shipwreck led to such loss of life was that there were not enough lifeboats for the passengers and crew. Although there was some element of luck involved in surviving the sinking, some groups of people were more likely to survive than others, such as women, children, and the upper-class.\n\nIn this challenge, we ask you to complete the analysis of what sorts of people were likely to survive. In particular, we ask you to apply the tools of machine learning to predict which passengers survived the tragedy.\n\nPractice Skills\n* Binary classification\n* Python and R basics\n\n# Step 2: Gather the Data\n\nThe dataset is also given to us on a golden plater with test and train data at [Kaggle's Titanic: Machine Learning from Disaster](https://www.kaggle.com/c/titanic/data)\n"},{"metadata":{"_cell_guid":"ef4059c6-9397-4b72-85da-570f51ba139c","_uuid":"f1d8b59b37d2ce04a5396a0ab183dc3000817113"},"cell_type":"markdown","source":"<a id=\"ch4\"></a>\n# Step 3: Prepare Data for Consumption\nSince step 2 was provided to us on a golden plater, so is step 3. Therefore, normal processes in data wrangling, such as data architecture, governance, and extraction are out of scope. Thus, only data cleaning is in scope.\n\n## 3.1 Import Libraries\nThe following code is written in Python 3.x. Libraries provide pre-written functionality to perform necessary tasks. The idea is why write ten lines of code, when you can write one line. "},{"metadata":{"_cell_guid":"a33744a8-08c7-4158-a22f-e8f2008a25b0","collapsed":true,"_uuid":"412b006d19b0209bd10ee4c6d15fdf59f8d2af38","_kg_hide-output":false,"_kg_hide-input":false,"trusted":false},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n\n#load packages\nimport sys #access to system parameters https://docs.python.org/3/library/sys.html\nprint(\"Python version: {}\". format(sys.version))\n\nimport pandas as pd #collection of functions for data processing and analysis modeled after R dataframes with SQL like features\nprint(\"pandas version: {}\". format(pd.__version__))\n\nimport matplotlib #collection of functions for scientific and publication-ready visualization\nprint(\"matplotlib version: {}\". format(matplotlib.__version__))\n\nimport numpy as np #foundational package for scientific computing\nprint(\"NumPy version: {}\". format(np.__version__))\n\nimport scipy as sp #collection of functions for scientific computing and advance mathematics\nprint(\"SciPy version: {}\". format(sp.__version__)) \n\nimport IPython\nfrom IPython import display #pretty printing of dataframes in Jupyter notebook\nprint(\"IPython version: {}\". format(IPython.__version__)) \n\nimport sklearn #collection of machine learning algorithms\nprint(\"scikit-learn version: {}\". format(sklearn.__version__))\n\n#misc libraries\nimport random\nimport time\n\n\n#ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')\nprint('-'*25)\n\n\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a514e15b-8caa-4b2e-bb46-13521761ef27","_uuid":"2e4810f3c85ffdee7d7382167e455baf1c320bc8"},"cell_type":"markdown","source":"## 3.11 Load Data Modelling Libraries\n\nWe will use the popular *scikit-learn* library to develop our machine learning algorithms. In *sklearn,* algorithms are called Estimators and implemented in their own classes. For data visualization, we will use the *matplotlib* and *seaborn* library. Below are common classes to load."},{"metadata":{"_cell_guid":"b230c790-a3f5-46ed-b351-57e96cc8fa61","collapsed":true,"_uuid":"4a16d09256317518769f21bf9cc38726df8b0078","trusted":false},"cell_type":"code","source":"#Common Model Algorithms\nfrom sklearn import svm, tree, linear_model, neighbors, naive_bayes, ensemble, discriminant_analysis, gaussian_process\nfrom xgboost import XGBClassifier\n\n#Common Model Helpers\nfrom sklearn.preprocessing import OneHotEncoder, LabelEncoder\nfrom sklearn import feature_selection\nfrom sklearn import model_selection\nfrom sklearn import metrics\n\n#Visualization\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nimport matplotlib.pylab as pylab\nimport seaborn as sns\nfrom pandas.tools.plotting import scatter_matrix\n\n#Configure Visualization Defaults\n#%matplotlib inline = show plots in Jupyter Notebook browser\n%matplotlib inline\nmpl.style.use('ggplot')\nsns.set_style('white')\npylab.rcParams['figure.figsize'] = 12,8","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c6289058-157a-46ba-b5c3-f134b268f5cc","_uuid":"a0fe0965de515e8c82cabd1fbd82cce993279f16"},"cell_type":"markdown","source":"## 3.2 Meet and Greet Data\n\nThis is the meet and greet step. Get to know your data by first name and learn a little bit about it. What does it look like (datatype and values), what makes it tick (independent/feature variables(s)), what's its goals in life (dependent/target variable(s)). Think of it like a first date, before you jump in and start poking it in the bedroom.\n\nTo begin this step, we first import our data. Next we use the info() and sample() function, to get a quick and dirty overview of variable datatypes (i.e. qualitative vs quantitative). Click here for the [Source Data Dictionary](https://www.kaggle.com/c/titanic/data).\n\n1. The *Survived* variable is our outcome or dependent variable. It is a binary nominal datatype of 1 for survived and 0 for did not survive. All other variables are potential predictor or independent variables. **It's important to note, more predictor variables do not make a better model, but the right variables.**\n2. The *PassengerID* and *Ticket* variables are assumed to be random unique identifiers, that have no impact on the outcome variable. Thus, they will be excluded from analysis.\n3. The *Pclass* variable is an ordinal datatype for the ticket class, a proxy for socio-economic status (SES), representing 1 = upper class, 2 = middle class, and 3 = lower class.\n4. The *Name* variable is a nominal datatype. It could be used in feature engineering to derive the gender from title, family size from surname, and SES from titles like doctor or master. Since these variables already exist, we'll make use of it to see if title, like master, makes a difference.\n5. The *Sex* and *Embarked* variables are a nominal datatype. They will be converted to dummy variables for mathematical calculations.\n6. The *Age* and *Fare* variable are continuous quantitative datatypes.\n7. The *SibSp* represents number of related siblings/spouse aboard and *Parch* represents number of related parents/children aboard. Both are discrete quantitative datatypes. This can be used for feature engineering to create a family size and is alone variable.\n8. The *Cabin* variable is a nominal datatype that can be used in feature engineering for approximate position on ship when the incident occurred and SES from deck levels. However, since there are many null values, it does not add value and thus is excluded from analysis."},{"metadata":{"_cell_guid":"c0928d7d-043b-4aa5-8113-e96fb42dbdc7","collapsed":true,"_uuid":"d0afb79a714ea826208235d05afac4e3f60554db","trusted":false},"cell_type":"code","source":"#import data from file: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html\ndata_raw = pd.read_csv('../input/train.csv')\n\n\n#a dataset should be broken into 3 splits: train, test, and (final) validation\n#the test file provided is the validation file for competition submission\n#we will split the train set into train and test data in future sections\ndata_val  = pd.read_csv('../input/test.csv')\n\n\n#to play with our data we'll create a copy\n#remember python assignment or equal passes by reference vs values, so we use the copy function: https://stackoverflow.com/questions/46327494/python-pandas-dataframe-copydeep-false-vs-copydeep-true-vs\ndata1 = data_raw.copy(deep = True)\n\n#however passing by reference is convenient, because we can clean both datasets at once\ndata_cleaner = [data1, data_val]\n\n\n#preview data\nprint (data_raw.info()) #https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.info.html\n#data_raw.head() #https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.head.html\n#data_raw.tail() #https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.tail.html\ndata_raw.sample(10) #https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sample.html","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b575d2d8-b77d-4810-9232-3b2726145ca8","_uuid":"281ad467b485855c1a39116c5ed899ac3018689f"},"cell_type":"markdown","source":"<a id=\"ch5\"></a>\n## 3.21 The 4 C's of Data Cleaning: Correcting, Completing, Creating, and Converting\nIn this stage, we will clean our data by 1) correcting aberrant values and outliers, 2) completing missing information, 3) creating new features for analysis, and 4) converting fields to the correct format for calculations and presentation.\n\n1. **Correcting:** Reviewing the data, there does not appear to be any aberrant or non-acceptable data inputs. In addition, we see we may have potential outliers in age and fare. However, since they are reasonable values, we will wait until after we complete our exploratory analysis to determine if we should include or exclude from the dataset. It should be noted, that if they were unreasonable values, for example age = 800 instead of 80, then it's probably a safe decision to fix now. However, we want to use caution when we modify data from its original value, because it may be necessary to create an accurate model.\n2. **Completing:** There are null values or missing data in the age, cabin, and embarked field. Missing values can be bad, because some algorithms don't know how-to handle null values and will fail. While others, like decision trees, can handle null values. Thus, it's important to fix before we start modeling, because we will compare and contrast several models. There are two common methods, either delete the record or populate the missing value using a reasonable input. It is not recommended to delete the record, especially a large percentage of records, unless it truly represents an incomplete record. Instead, it's best to impute missing values. A basic methodology for qualitative data is impute using mode. A basic methodology for quantitative data is impute using mean, median, or mean + randomized standard deviation. An intermediate methodology is to use the basic methodology based on specific criteria; like the average age by class or embark port by fare and SES. There are more complex methodologies, however before deploying, it should be compared to the base model to determine if complexity truly adds value. For this dataset, age will be imputed with the median, the cabin attribute will be dropped, and embark will be imputed with mode. Subsequent model iterations may modify this decision to determine if it improves the model’s accuracy.\n3. **Creating:**  Feature engineering is when we use existing features to create new features to determine if they provide new signals to predict our outcome. For this dataset, we will create a title feature to determine if it played a role in survival.\n4. **Converting:** Last, but certainly not least, we'll deal with formatting. There are no date or currency formats, but datatype formats. Our categorical data imported as objects, which makes it difficult for mathematical calculations. For this dataset, we will convert object datatypes to categorical dummy variables."},{"metadata":{"_cell_guid":"e2181a68-3c33-4040-83a9-99adee8363a0","collapsed":true,"_uuid":"9a36681f0eda8ccc6396c64aa6bd6e8288461bcb","_kg_hide-output":false,"_kg_hide-input":false,"scrolled":false,"trusted":false},"cell_type":"code","source":"print('Train columns with null values:\\n', data1.isnull().sum())\nprint(\"-\"*10)\n\nprint('Test/Validation columns with null values:\\n', data_val.isnull().sum())\nprint(\"-\"*10)\n\ndata_raw.describe(include = 'all')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c6fc9f66-8779-454c-b6bd-47676c0dc81c","_uuid":"5002c97888cfdcb2b49ef76fd1505eed6a1721e4"},"cell_type":"markdown","source":"## 3.22 Clean Data\n\nNow that we know what to clean, let's execute our code.\n\n** Developer Documentation: **\n* [pandas.DataFrame](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html)\n* [pandas.DataFrame.info](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.info.html)\n* [pandas.DataFrame.describe](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html)\n* [Indexing and Selecting Data](https://pandas.pydata.org/pandas-docs/stable/indexing.html)\n* [pandas.isnull](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.isnull.html)\n* [pandas.DataFrame.sum](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sum.html)\n* [pandas.DataFrame.mode](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.mode.html)\n* [pandas.DataFrame.copy](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.copy.html)\n* [pandas.DataFrame.fillna](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.fillna.html)\n* [pandas.DataFrame.drop](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop.html)\n* [pandas.Series.value_counts](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.value_counts.html)\n* [pandas.DataFrame.loc](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.loc.html)"},{"metadata":{"_cell_guid":"56e3e982-a23a-4c84-b8a1-976adc94aa0b","collapsed":true,"_uuid":"160dd59e4ebe7a94521c65da2746ca370b4bd694","trusted":false},"cell_type":"code","source":"###COMPLETING: complete or delete missing values in train and test/validation dataset\nfor dataset in data_cleaner:    \n    #complete missing age with median\n    dataset['Age'].fillna(dataset['Age'].median(), inplace = True)\n\n    #complete embarked with mode\n    dataset['Embarked'].fillna(dataset['Embarked'].mode()[0], inplace = True)\n\n    #complete missing fare with median\n    dataset['Fare'].fillna(dataset['Fare'].median(), inplace = True)\n    \n#delete the cabin feature/column and others previously stated to exclude in train dataset\ndrop_column = ['PassengerId','Cabin', 'Ticket']\ndata1.drop(drop_column, axis=1, inplace = True)\n\nprint(data1.isnull().sum())\nprint(\"-\"*10)\nprint(data_val.isnull().sum())","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c66b14b7-e1e5-43d7-9260-840fe0ef46c7","collapsed":true,"_uuid":"e62f1b14f6f9758b66182e2476f3c57bb4a0734c","scrolled":false,"trusted":false},"cell_type":"code","source":"###CREATE: Feature Engineering for train and test/validation dataset\nfor dataset in data_cleaner:    \n    #Discrete variables\n    dataset['FamilySize'] = dataset ['SibSp'] + dataset['Parch'] + 1\n\n    dataset['IsAlone'] = 1 #initialize to yes/1 is alone\n    dataset['IsAlone'].loc[dataset['FamilySize'] > 1] = 0 # now update to no/0 if family size is greater than 1\n\n    #quick and dirty code split title from name: http://www.pythonforbeginners.com/dictionary/python-split\n    dataset['Title'] = dataset['Name'].str.split(\", \", expand=True)[1].str.split(\".\", expand=True)[0]\n\n\n    #Continuous variable bins; qcut vs cut: https://stackoverflow.com/questions/30211923/what-is-the-difference-between-pandas-qcut-and-pandas-cut\n    #Fare Bins/Buckets using qcut or frequency bins: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.qcut.html\n    dataset['FareBin'] = pd.qcut(dataset['Fare'], 4)\n\n    #Age Bins/Buckets using cut or value bins: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.cut.html\n    dataset['AgeBin'] = pd.cut(dataset['Age'].astype(int), 5)\n\n\n    \n#cleanup rare title names\n#print(data1['Title'].value_counts())\nstat_min = 10 #while small is arbitrary, we'll use the common minimum in statistics: http://nicholasjjackson.com/2012/03/08/sample-size-is-10-a-magic-number/\ntitle_names = (data1['Title'].value_counts() < stat_min) #this will create a true false series with title name as index\n\n#apply and lambda functions are quick and dirty code to find and replace with fewer lines of code: https://community.modeanalytics.com/python/tutorial/pandas-groupby-and-python-lambda-functions/\ndata1['Title'] = data1['Title'].apply(lambda x: 'Misc' if title_names.loc[x] == True else x)\nprint(data1['Title'].value_counts())\nprint(\"-\"*10)\n\n\n#preview data again\ndata1.info()\ndata_val.info()\ndata1.sample(10)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f7972928-84b5-47b1-98b6-3e47bc7ddb17","_uuid":"eb0a15c2065a827c3c5431622c04707f3e9d2e52"},"cell_type":"markdown","source":"## 3.23 Convert Formats\n\nWe will convert categorical data to dummy variables for mathematical analysis. There are multiple ways to encode categorical variables; we will use the sklearn and pandas functions.\n\nIn this step, we will also define our x (independent/features/explanatory/predictor/etc.) and y (dependent/target/outcome/response/etc.) variables for data modeling.\n\n** Developer Documentation: **\n* [Categorical Encoding](http://pbpython.com/categorical-encoding.html)\n* [Sklearn LabelEncoder](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html)\n* [Sklearn OneHotEncoder](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html)\n* [Pandas Categorical dtype](https://pandas.pydata.org/pandas-docs/stable/categorical.html)\n* [pandas.get_dummies](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html)"},{"metadata":{"_cell_guid":"95f9a05d-9f6e-46b3-8a3c-95a78f0c0834","collapsed":true,"_uuid":"7f746d8c6f7d0214253919185fa16f2f6a447a15","scrolled":true,"trusted":false},"cell_type":"code","source":"#CONVERT: convert objects to category using Label Encoder for train and test/validation dataset\n\n#code categorical data\nlabel = LabelEncoder()\nfor dataset in data_cleaner:    \n    dataset['Sex_Code'] = label.fit_transform(dataset['Sex'])\n    dataset['Embarked_Code'] = label.fit_transform(dataset['Embarked'])\n    dataset['Title_Code'] = label.fit_transform(dataset['Title'])\n    dataset['AgeBin_Code'] = label.fit_transform(dataset['AgeBin'])\n    dataset['FareBin_Code'] = label.fit_transform(dataset['FareBin'])\n\n\n#define y variable aka target/outcome\nTarget = ['Survived']\n\n#define x variables for original features aka feature selection\ndata1_x = ['Sex','Pclass', 'Embarked', 'Title','SibSp', 'Parch', 'Age', 'Fare', 'FamilySize', 'IsAlone'] #pretty name/values for charts\ndata1_x_calc = ['Sex_Code','Pclass', 'Embarked_Code', 'Title_Code','SibSp', 'Parch', 'Age', 'Fare'] #coded for algorithm calculation\ndata1_xy =  Target + data1_x\nprint('Original X Y: ', data1_xy, '\\n')\n\n\n#define x variables for original w/bin features to remove continuous variables\ndata1_x_bin = ['Sex_Code','Pclass', 'Embarked_Code', 'Title_Code', 'FamilySize', 'AgeBin_Code', 'FareBin_Code']\ndata1_xy_bin = Target + data1_x_bin\nprint('Bin X Y: ', data1_xy_bin, '\\n')\n\n\n#define x and y variables for dummy features original\ndata1_dummy = pd.get_dummies(data1[data1_x])\ndata1_x_dummy = data1_dummy.columns.tolist()\ndata1_xy_dummy = Target + data1_x_dummy\nprint('Dummy X Y: ', data1_xy_dummy, '\\n')\n\n\n\ndata1_dummy.head()\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"43ae9ddf-85b6-4abd-89a7-cfb0ff70ebad","_uuid":"9f4b880b2e485e6bc4b6a64113b8f816e2319e85"},"cell_type":"markdown","source":"## 3.24 Da-Double Check Cleaned Data\n\nNow that we've cleaned our data, let's do a discount da-double check!"},{"metadata":{"_cell_guid":"00d10c37-5d8a-4eaf-8fe8-25a0b58e0bfd","collapsed":true,"_uuid":"e2ef29db7e31dc83c10238ee33ad4ad0ad426183","scrolled":false,"trusted":false},"cell_type":"code","source":"print('Train columns with null values: \\n', data1.isnull().sum())\nprint(\"-\"*10)\nprint (data1.info())\nprint(\"-\"*10)\n\nprint('Test/Validation columns with null values: \\n', data_val.isnull().sum())\nprint(\"-\"*10)\nprint (data_val.info())\nprint(\"-\"*10)\n\ndata_raw.describe(include = 'all')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e1ffcb68-17a5-43e0-b7e1-4a0b9d33dbdd","_uuid":"9b4b6e1f40e310274447c880760b87ec6a0c7c77"},"cell_type":"markdown","source":"## 3.25 Split Training and Testing Data\n\nAs mentioned previously, the test file provided is really validation data for competition submission. So, we will use *sklearn* function to split the training data in two datasets; 75/25 split. This is important, so we don't [overfit our model](https://www.coursera.org/learn/python-machine-learning/lecture/fVStr/overfitting-and-underfitting). Meaning, the algorithm is so specific to a given subset, it cannot accurately generalize another subset, from the same dataset. It's important our algorithm has not seen the subset we will use to test, so it doesn't \"cheat\" by memorizing the answers. We will use [*sklearn's* train_test_split function](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html). In later sections we will also use [*sklearn's* cross validation functions](http://scikit-learn.org/stable/modules/cross_validation.html#cross-validation), that splits our dataset into train and test for data modeling comparison."},{"metadata":{"_cell_guid":"15e84f99-e015-4e0a-bd0a-2f856ffad1f6","collapsed":true,"_uuid":"f9eb9f6e78235a38580ee3125ede17b836edabb3","trusted":false},"cell_type":"code","source":"#split train and test data with function defaults\n#random_state -> seed or control random number generator: https://www.quora.com/What-is-seed-in-random-number-generation\ntrain1_x, test1_x, train1_y, test1_y = model_selection.train_test_split(data1[data1_x_calc], data1[Target], random_state = 0)\ntrain1_x_bin, test1_x_bin, train1_y_bin, test1_y_bin = model_selection.train_test_split(data1[data1_x_bin], data1[Target] , random_state = 0)\ntrain1_x_dummy, test1_x_dummy, train1_y_dummy, test1_y_dummy = model_selection.train_test_split(data1_dummy[data1_x_dummy], data1[Target], random_state = 0)\n\n\nprint(\"Data1 Shape: {}\".format(data1.shape))\nprint(\"Train1 Shape: {}\".format(train1_x.shape))\nprint(\"Test1 Shape: {}\".format(test1_x.shape))\n\ntrain1_x_bin.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8a13c80c-9f6b-405a-bba2-d3fb9b1cf19d","_uuid":"d3d49dc8106082d149dc4058b002355fc809726b"},"cell_type":"markdown","source":"<a id=\"ch6\"></a>\n# Step 4: Perform Exploratory Analysis with Statistics\nNow that our data is cleaned, we will explore our data with descriptive and graphical statistics to describe and summarize our variables. In this stage, you will find yourself classifying features and determining their correlation with the target variable and each other."},{"metadata":{"_cell_guid":"f14819d3-e23c-40fa-a281-375e8777a47b","collapsed":true,"_uuid":"5355b0e45b130a1fd574510141f65e62ebfe2148","scrolled":false,"trusted":false},"cell_type":"code","source":"#Discrete Variable Correlation by Survival using\n#group by aka pivot table: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html\nfor x in data1_x:\n    if data1[x].dtype != 'float64' :\n        print('Survival Correlation by:', x)\n        print(data1[[x, Target[0]]].groupby(x, as_index=False).mean())\n        print('-'*10, '\\n')\n        \n\n#using crosstabs: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.crosstab.html\nprint(pd.crosstab(data1['Title'],data1[Target[0]]))\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ada8f356-f5bf-4629-81c9-b5a95e16a09d","collapsed":true,"_uuid":"a8485baedcf8d61ab13c1cb19d3654a835ce1765","scrolled":false,"trusted":false},"cell_type":"code","source":"#IMPORTANT: Intentionally plotted different ways for learning purposes only. \n\n#optional plotting w/pandas: https://pandas.pydata.org/pandas-docs/stable/visualization.html\n\n#we will use matplotlib.pyplot: https://matplotlib.org/api/pyplot_api.html\n\n#to organize our graphics will use figure: https://matplotlib.org/api/_as_gen/matplotlib.pyplot.figure.html#matplotlib.pyplot.figure\n#subplot: https://matplotlib.org/api/_as_gen/matplotlib.pyplot.subplot.html#matplotlib.pyplot.subplot\n#and subplotS: https://matplotlib.org/api/_as_gen/matplotlib.pyplot.subplots.html?highlight=matplotlib%20pyplot%20subplots#matplotlib.pyplot.subplots\n\n#graph distribution of quantitative data\nplt.figure(figsize=[16,12])\n\nplt.subplot(231)\nplt.boxplot(x=data1['Fare'], showmeans = True, meanline = True)\nplt.title('Fare Boxplot')\nplt.ylabel('Fare ($)')\n\nplt.subplot(232)\nplt.boxplot(data1['Age'], showmeans = True, meanline = True)\nplt.title('Age Boxplot')\nplt.ylabel('Age (Years)')\n\nplt.subplot(233)\nplt.boxplot(data1['FamilySize'], showmeans = True, meanline = True)\nplt.title('Family Size Boxplot')\nplt.ylabel('Family Size (#)')\n\nplt.subplot(234)\nplt.hist(x = [data1[data1['Survived']==1]['Fare'], data1[data1['Survived']==0]['Fare']], \n         stacked=True, color = ['g','r'],label = ['Survived','Dead'])\nplt.title('Fare Histogram by Survival')\nplt.xlabel('Fare ($)')\nplt.ylabel('# of Passengers')\nplt.legend()\n\nplt.subplot(235)\nplt.hist(x = [data1[data1['Survived']==1]['Age'], data1[data1['Survived']==0]['Age']], \n         stacked=True, color = ['g','r'],label = ['Survived','Dead'])\nplt.title('Age Histogram by Survival')\nplt.xlabel('Age (Years)')\nplt.ylabel('# of Passengers')\nplt.legend()\n\nplt.subplot(236)\nplt.hist(x = [data1[data1['Survived']==1]['FamilySize'], data1[data1['Survived']==0]['FamilySize']], \n         stacked=True, color = ['g','r'],label = ['Survived','Dead'])\nplt.title('Family Size Histogram by Survival')\nplt.xlabel('Family Size (#)')\nplt.ylabel('# of Passengers')\nplt.legend()\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"be8b849a-fdf8-4491-b5e2-e70fee8856c3","collapsed":true,"_uuid":"e8da47b89495ad91f35daab26c153d90099afe7e","trusted":false},"cell_type":"code","source":"#we will use seaborn graphics for multi-variable comparison: https://seaborn.pydata.org/api.html\n\n#graph individual features by survival\nfig, saxis = plt.subplots(2, 3,figsize=(16,12))\n\nsns.barplot(x = 'Embarked', y = 'Survived', data=data1, ax = saxis[0,0])\nsns.barplot(x = 'Pclass', y = 'Survived', order=[1,2,3], data=data1, ax = saxis[0,1])\nsns.barplot(x = 'IsAlone', y = 'Survived', order=[1,0], data=data1, ax = saxis[0,2])\n\nsns.pointplot(x = 'FareBin', y = 'Survived',  data=data1, ax = saxis[1,0])\nsns.pointplot(x = 'AgeBin', y = 'Survived',  data=data1, ax = saxis[1,1])\nsns.pointplot(x = 'FamilySize', y = 'Survived', data=data1, ax = saxis[1,2])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"d5f64ab0-2ad1-4b2d-bb5b-b3e8196df09a","collapsed":true,"_uuid":"9285ac3c47272d97e1b5a3e9d5f693b285a7bec2","trusted":false},"cell_type":"code","source":"#graph distribution of qualitative data: Pclass\n#we know class mattered in survival, now let's compare class and a 2nd feature\nfig, (axis1,axis2,axis3) = plt.subplots(1,3,figsize=(14,12))\n\nsns.boxplot(x = 'Pclass', y = 'Fare', hue = 'Survived', data = data1, ax = axis1)\naxis1.set_title('Pclass vs Fare Survival Comparison')\n\nsns.violinplot(x = 'Pclass', y = 'Age', hue = 'Survived', data = data1, split = True, ax = axis2)\naxis2.set_title('Pclass vs Age Survival Comparison')\n\nsns.boxplot(x = 'Pclass', y ='FamilySize', hue = 'Survived', data = data1, ax = axis3)\naxis3.set_title('Pclass vs Family Size Survival Comparison')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c64adbcf-bb9e-4ac7-81fe-cc32dbec9984","collapsed":true,"_uuid":"7c98f887cebba99c541ac1af747d48ce300532ed","trusted":false},"cell_type":"code","source":"#graph distribution of qualitative data: Sex\n#we know sex mattered in survival, now let's compare sex and a 2nd feature\nfig, qaxis = plt.subplots(1,3,figsize=(14,12))\n\nsns.barplot(x = 'Sex', y = 'Survived', hue = 'Embarked', data=data1, ax = qaxis[0])\naxis1.set_title('Sex vs Embarked Survival Comparison')\n\nsns.barplot(x = 'Sex', y = 'Survived', hue = 'Pclass', data=data1, ax  = qaxis[1])\naxis1.set_title('Sex vs Pclass Survival Comparison')\n\nsns.barplot(x = 'Sex', y = 'Survived', hue = 'IsAlone', data=data1, ax  = qaxis[2])\naxis1.set_title('Sex vs IsAlone Survival Comparison')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"42909c80-4f4e-4573-9489-0e8b56578a5f","collapsed":true,"_uuid":"0a9c3a9367fe66193f762090d74fa03ad8f92b76","trusted":false},"cell_type":"code","source":"#more side-by-side comparisons\nfig, (maxis1, maxis2) = plt.subplots(1, 2,figsize=(14,12))\n\n#how does family size factor with sex & survival compare\nsns.pointplot(x=\"FamilySize\", y=\"Survived\", hue=\"Sex\", data=data1,\n              palette={\"male\": \"blue\", \"female\": \"pink\"},\n              markers=[\"*\", \"o\"], linestyles=[\"-\", \"--\"], ax = maxis1)\n\n#how does class factor with sex & survival compare\nsns.pointplot(x=\"Pclass\", y=\"Survived\", hue=\"Sex\", data=data1,\n              palette={\"male\": \"blue\", \"female\": \"pink\"},\n              markers=[\"*\", \"o\"], linestyles=[\"-\", \"--\"], ax = maxis2)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"23072dbf-629d-46d1-b0d3-61affd805ffc","collapsed":true,"_uuid":"9c198c6cc80f325db06fabbc79f197c7524595e3","trusted":false},"cell_type":"code","source":"#how does embark port factor with class, sex, and survival compare\n#facetgrid: https://seaborn.pydata.org/generated/seaborn.FacetGrid.html\ne = sns.FacetGrid(data1, col = 'Embarked')\ne.map(sns.pointplot, 'Pclass', 'Survived', 'Sex', ci=95.0, palette = 'deep')\ne.add_legend()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"d1749e02-355d-4209-af23-6c326e426a55","collapsed":true,"_uuid":"3ede990a7e35035d112aff7bc0deb1b936009756","trusted":false},"cell_type":"code","source":"#plot distributions of age of passengers who survived or did not survive\na = sns.FacetGrid( data1, hue = 'Survived', aspect=4 )\na.map(sns.kdeplot, 'Age', shade= True )\na.set(xlim=(0 , data1['Age'].max()))\na.add_legend()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"4b18d4b5-7a5d-4a80-8b63-06ea48ee2b7a","collapsed":true,"_uuid":"9217b8fd04d2070a3588f507fe7a93fbdf962abe","trusted":false},"cell_type":"code","source":"#histogram comparison of sex, class, and age by survival\nh = sns.FacetGrid(data1, row = 'Sex', col = 'Pclass', hue = 'Survived')\nh.map(plt.hist, 'Age', alpha = .75)\nh.add_legend()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"20cde1bc-4836-4d29-b3cd-eddb97d69838","collapsed":true,"_uuid":"85e1507db880d6eafd2ee6bc7fef8845f79f949d","trusted":false},"cell_type":"code","source":"#pair plots of entire dataset\npp = sns.pairplot(data1, hue = 'Survived', palette = 'deep', size=1.2, diag_kind = 'kde', diag_kws=dict(shade=True), plot_kws=dict(s=10) )\npp.set(xticklabels=[])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"fbfaf416-4eb7-45fc-8ee2-123852ee6a7d","collapsed":true,"_uuid":"d599e6d35f40e66f3f2bd162b2d9087d10ed443d","trusted":false},"cell_type":"code","source":"#correlation heatmap of dataset\ndef correlation_heatmap(df):\n    _ , ax = plt.subplots(figsize =(14, 12))\n    colormap = sns.diverging_palette(220, 10, as_cmap = True)\n    \n    _ = sns.heatmap(\n        df.corr(), \n        cmap = colormap,\n        square=True, \n        cbar_kws={'shrink':.9 }, \n        ax=ax,\n        annot=True, \n        linewidths=0.1,vmax=1.0, linecolor='white',\n        annot_kws={'fontsize':12 }\n    )\n    \n    plt.title('Pearson Correlation of Features', y=1.05, size=15)\n\ncorrelation_heatmap(data1)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"079d255c-e55e-482b-be9f-ae6b23015790","_uuid":"1f4f5f75093fdcec864a835d94cd5d9a05b591c5"},"cell_type":"markdown","source":"<a id=\"ch7\"></a>\n# Step 5: Model Data\nData Science is a multi-disciplinary field between mathematics (i.e. statistics, linear algebra, etc.), computer science (i.e. programming languages, computer systems, etc.) and business management (i.e. communication, subject-matter knowledge, etc.). Most data scientist come from one of the three fields, so they tend to lean towards that discipline. However, data science is like a three-legged stool, with no one leg being more important than the other. So, this step will require advanced knowledge in mathematics. But don’t worry, we only need a high-level overview, which we’ll cover in this Kernel. Also, thanks to computer science, a lot of the heavy lifting is done for you. So, problems that once required graduate degrees in mathematics or statistics, now only take a few lines of code. Last, we’ll need some business acumen to think through the problem. After all, like training a sight-seeing dog, it’s learning from us and not the other way around.\n\nMachine Learning (ML), as the name suggest, is teaching the machine how-to think and not what to think. While this topic and big data has been around for decades, it is becoming more popular than ever because the barrier to entry is lower, for businesses and professionals alike. This is both good and bad. It’s good because these algorithms are now accessible to more people that can solve more problems in the real-world. It’s bad because a lower barrier to entry means, more people will not know the tools they are using and can come to incorrect conclusions. That’s why I focus on teaching you, not just what to do, but why you’re doing it. Previously, I used the analogy of asking someone to hand you a Philip screwdriver, and they hand you a flathead screwdriver or worst a hammer. At best, it shows a complete lack of understanding. At worst, it makes completing the project impossible; or even worst, implements incorrect actionable intelligence. So now that I’ve hammered (no pun intended) my point, I’ll show you what to do and most importantly, WHY you do it.\n\nFirst, you must understand, that the purpose of machine learning is to solve human problems. Machine learning can be categorized as: supervised learning, unsupervised learning, and reinforced learning. Supervised learning is where you train the model by presenting it a training dataset that includes the correct answer. Unsupervised learning is where you train the model using a training dataset that does not include the correct answer. And reinforced learning is a hybrid of the previous two, where the model is not given the correct answer immediately, but later after a sequence of events to reinforce learning. We are doing supervised machine learning, because we are training our algorithm by presenting it with a set of features and their corresponding target. We then hope to present it a new subset from the same dataset and have similar results in prediction accuracy.\n\nThere are many machine learning algorithms, however they can be reduced to four categories: classification, regression, clustering, or dimensionality reduction, depending on your target variable and data modeling goals. We'll save clustering and dimension reduction for another day, and focus on classification and regression. We can generalize that a continuous target variable requires a regression algorithm and a discrete target variable requires a classification algorithm. One side note, logistic regression, while it has regression in the name, is really a classification algorithm. Since our problem is predicting if a passenger survived or did not survive, this is a discrete target variable. We will use a classification algorithm from the *sklearn* library to begin our analysis. We will use cross validation and scoring metrics, discussed in later sections, to rank and compare our algorithms’ performance.\n\n**Machine Learning Selection:**\n* [Sklearn Estimator Overview](http://scikit-learn.org/stable/user_guide.html)\n* [Sklearn Estimator Detail](http://scikit-learn.org/stable/modules/classes.html)\n* [Choosing Estimator Mind Map](http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html)\n* [Choosing Estimator Cheat Sheet](https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Scikit_Learn_Cheat_Sheet_Python.pdf)\n\n\nNow that we identified our solution as a supervised learning classification algorithm. We can narrow our list of choices.\n\n**Machine Learning Classification Algorithms:**\n* [Ensemble Methods](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.ensemble)\n* [Generalized Linear Models (GLM)](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.linear_model)\n* [Naive Bayes](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.naive_bayes)\n* [Nearest Neighbors](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.neighbors)\n* [Support Vector Machines (SVM)](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.svm)\n* [Decision Trees](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.tree)\n* [Discriminant Analysis](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.discriminant_analysis)\n\n\n### Data Science 101: How to Choose a Machine Learning Algorithm (MLA)\n**IMPORTANT:** When it comes to data modeling, the beginner’s question is always, \"what is the best machine learning algorithm?\" To this the beginner must learn, the [No Free Lunch Theorem (NFLT)](http://robertmarks.org/Classes/ENGR5358/Papers/NFL_4_Dummies.pdf) of Machine Learning. In short, NFLT states, there is no super algorithm, that works best in all situations, for all datasets. So the best approach is to try multiple MLAs, tune them, and compare them for your specific scenario. With that being said, some good research has been done to compare algorithms, such as [Caruana & Niculescu-Mizil 2006](https://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icml06.pdf) watch [video lecture here](http://videolectures.net/solomon_caruana_wslmw/) of MLA comparisons, [Ogutu et al. 2011](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3103196/) done by the NIH for genomic selection, [Fernandez-Delgado et al. 2014](http://jmlr.org/papers/volume15/delgado14a/delgado14a.pdf) comparing 179 classifiers from 17 families, [Thoma 2016 sklearn comparison](https://martin-thoma.com/comparing-classifiers/), and there is also a school of thought that says, [more data beats a better algorithm](https://www.kdnuggets.com/2015/06/machine-learning-more-data-better-algorithms.html). \n\nSo with all this information, where is a beginner to start? I recommend starting with [Trees, Bagging, Random Forests, and Boosting](http://jessica2.msri.org/attachments/10778/10778-boost.pdf). They are basically different implementations of a decision tree, which is the easiest concept to learn and understand. They are also easier to tune, discussed in the next section, than something like SVC. Below, I'll give an overview of how-to run and compare several MLAs, but the rest of this Kernel will focus on learning data modeling via decision trees and its derivatives."},{"metadata":{"_cell_guid":"68816e33-19bc-482e-b4e8-062098b18798","collapsed":true,"_uuid":"979ae81e95cf760f05e94853b932b5101198b4e9","trusted":false},"cell_type":"code","source":"#Machine Learning Algorithm (MLA) Selection and Initialization\nMLA = [\n    #Ensemble Methods\n    ensemble.AdaBoostClassifier(),\n    ensemble.BaggingClassifier(),\n    ensemble.ExtraTreesClassifier(),\n    ensemble.GradientBoostingClassifier(),\n    ensemble.RandomForestClassifier(),\n\n    #Gaussian Processes\n    gaussian_process.GaussianProcessClassifier(),\n    \n    #GLM\n    linear_model.LogisticRegressionCV(),\n    linear_model.PassiveAggressiveClassifier(),\n    linear_model.RidgeClassifierCV(),\n    linear_model.SGDClassifier(),\n    linear_model.Perceptron(),\n    \n    #Navies Bayes\n    naive_bayes.BernoulliNB(),\n    naive_bayes.GaussianNB(),\n    \n    #Nearest Neighbor\n    neighbors.KNeighborsClassifier(),\n    \n    #SVM\n    svm.SVC(probability=True),\n    svm.NuSVC(probability=True),\n    svm.LinearSVC(),\n    \n    #Trees    \n    tree.DecisionTreeClassifier(),\n    tree.ExtraTreeClassifier(),\n    \n    #Discriminant Analysis\n    discriminant_analysis.LinearDiscriminantAnalysis(),\n    discriminant_analysis.QuadraticDiscriminantAnalysis(),\n\n    \n    #xgboost: http://xgboost.readthedocs.io/en/latest/model.html\n    XGBClassifier()    \n    ]\n\n\n\n#split dataset in cross-validation with this splitter class: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.ShuffleSplit.html#sklearn.model_selection.ShuffleSplit\n#note: this is an alternative to train_test_split\ncv_split = model_selection.ShuffleSplit(n_splits = 10, test_size = .3, train_size = .6, random_state = 0 ) # run model 10x with 60/30 split intentionally leaving out 10%\n\n#create table to compare MLA metrics\nMLA_columns = ['MLA Name', 'MLA Parameters','MLA Train Accuracy Mean', 'MLA Test Accuracy Mean', 'MLA Test Accuracy 3*STD' ,'MLA Time']\nMLA_compare = pd.DataFrame(columns = MLA_columns)\n\n#create table to compare MLA predictions\nMLA_predict = data1[Target]\n\n#index through MLA and save performance to table\nrow_index = 0\nfor alg in MLA:\n\n    #set name and parameters\n    MLA_name = alg.__class__.__name__\n    MLA_compare.loc[row_index, 'MLA Name'] = MLA_name\n    MLA_compare.loc[row_index, 'MLA Parameters'] = str(alg.get_params())\n    \n    #score model with cross validation: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html#sklearn.model_selection.cross_validate\n    cv_results = model_selection.cross_validate(alg, data1[data1_x_bin], data1[Target], cv  = cv_split)\n\n    MLA_compare.loc[row_index, 'MLA Time'] = cv_results['fit_time'].mean()\n    MLA_compare.loc[row_index, 'MLA Train Accuracy Mean'] = cv_results['train_score'].mean()\n    MLA_compare.loc[row_index, 'MLA Test Accuracy Mean'] = cv_results['test_score'].mean()   \n    #if this is a non-bias random sample, then +/-3 standard deviations (std) from the mean, should statistically capture 99.7% of the subsets\n    MLA_compare.loc[row_index, 'MLA Test Accuracy 3*STD'] = cv_results['test_score'].std()*3   #let's know the worst that can happen!\n    \n\n    #save MLA predictions - see section 6 for usage\n    alg.fit(data1[data1_x_bin], data1[Target])\n    MLA_predict[MLA_name] = alg.predict(data1[data1_x_bin])\n    \n    row_index+=1\n\n    \n#print and sort table: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sort_values.html\nMLA_compare.sort_values(by = ['MLA Test Accuracy Mean'], ascending = False, inplace = True)\nMLA_compare\n#MLA_predict\n\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b9653051-8e79-4ae3-ac31-497bf0a5caed","collapsed":true,"_uuid":"b7acbf3f7ad6e812a2b5d60794bc3c5ad7d3fd0c","trusted":false},"cell_type":"code","source":"#barplot using https://seaborn.pydata.org/generated/seaborn.barplot.html\nsns.barplot(x='MLA Test Accuracy Mean', y = 'MLA Name', data = MLA_compare, color = 'm')\n\n#prettify using pyplot: https://matplotlib.org/api/pyplot_api.html\nplt.title('Machine Learning Algorithm Accuracy Score \\n')\nplt.xlabel('Accuracy Score (%)')\nplt.ylabel('Algorithm')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c90ba48e-3c44-47c8-9bd2-d46187efb9d8","_uuid":"9e3ac7492a21e7f60b0656008f7873cab882f196"},"cell_type":"markdown","source":"<a id=\"ch8\"></a>\n## 5.1 Evaluate Model Performance\nLet's recap, with some basic data cleaning, analysis, and machine learning algorithms (MLA), we are able to predict passenger survival with ~82% accuracy. Not bad for a few lines of code. But the question we always ask is, can we do better and more importantly get an ROI (return on investment) for our time invested? For example, if we're only going to increase our accuracy by 1/10th of a percent, is it really worth 3-months of development. If you work in research maybe the answer is yes, but if you work in business mostly the answer is no. So, keep that in mind when improving your model.\n\n### Data Science 101: Determine a Baseline Accuracy ###\nBefore we decide how-to make our model better, let's determine if our model is even worth keeping. To do that, we have to go back to the basics of data science 101. We know this is a binary problem, because there are only two possible outcomes; passengers survived or died. So, think of it like a coin flip problem. If you have a fair coin and you guessed heads or tail, then you have a 50-50 chance of guessing correct. So, let's set 50% as the worst model performance; because anything lower than that, then why do I need you when I can just flip a coin?\n\nOkay, so with no information about the dataset, we can always get 50% with a binary problem. But we have information about the dataset, so we should be able to do better. We know that 1,502/2,224 or 67.5% of people died. Therefore, if we just predict the most frequent occurrence, that 100% of people died, then we would be right 67.5% of the time. So, let's set 68% as bad model performance, because again, anything lower than that, then why do I need you, when I can just predict using the most frequent occurrence.\n\n### Data Science 101: How-to Create Your Own Model ###\nOur accuracy is increasing, but can we do better? Are there any signals in our data? To illustrate this, we're going to build our own decision tree model, because it is the easiest to conceptualize and requires simple addition and multiplication calculations. When creating a decision tree, you want to ask questions that segment your target response, placing the survived/1 and dead/0 into homogeneous subgroups. This is part science and part art, so let's just play the 21-question game to show you how it works. If you want to follow along on your own, download the train dataset and import into Excel. Create a pivot table with survival in the columns, count and % of row count in the values, and the features described below in the rows.\n\nRemember, the name of the game is to create subgroups using a decision tree model to get survived/1 in one bucket and dead/0 in another bucket. Our rule of thumb will be the majority rules. Meaning, if the majority or 50% or more survived, then everybody in our subgroup survived/1, but if 50% or less survived then if everybody in our subgroup died/0. Also, we will stop if the subgroup is less than 10 and/or our model accuracy plateaus or decreases. Got it? Let's go!\n\n***Question 1: Were you on the Titanic?*** If Yes, then majority (62%) died. Note our sample survival is different than our population of 68%. Nonetheless, if we assumed everybody died, our sample accuracy is 62%.\n\n***Question 2: Are you male or female?*** Male, majority (81%) died. Female, majority (74%) survived. Giving us an accuracy of 79%.\n\n***Question 3A (going down the female branch with count = 314): Are you in class 1, 2, or 3?*** Class 1, majority (97%) survived and Class 2, majority (92%) survived. Since the dead subgroup is less than 10, we will stop going down this branch. Class 3, is even at a 50-50 split. No new information to improve our model is gained.\n\n***Question 4A (going down the female class 3 branch with count = 144): Did you embark from port C, Q, or S?*** We gain a little information. C and Q, the majority still survived, so no change. Also, the dead subgroup is less than 10, so we will stop. S, the majority (63%) died. So, we will change females, class 3, embarked S from assuming they survived, to assuming they died. Our model accuracy increases to 81%. \n\n***Question 5A (going down the female class 3 embarked S branch with count = 88):*** So far, it looks like we made good decisions. Adding another level does not seem to gain much more information. This subgroup 55 died and 33 survived, since majority died we need to find a signal to identify the 33 or a subgroup to change them from dead to survived and improve our model accuracy. We can play with our features. One I found was fare 0-8, majority survived. It's a small sample size 11-9, but one often used in statistics. We slightly improve our accuracy, but not much to move us past 82%. So, we'll stop here.\n\n***Question 3B (going down the male branch with count = 577):*** Going back to question 2, we know the majority of males died. So, we are looking for a feature that identifies a subgroup that majority survived. Surprisingly, class or even embarked didn't matter like it did for females, but title does and gets us to 82%. Guess and checking other features, none seem to push us past 82%. So, we'll stop here for now.\n\nYou did it, with very little information, we get to 82% accuracy. On a worst, bad, good, better, and best scale, we'll set 82% to good, since it's a simple model that yields us decent results. But the question still remains, can we do better than our handmade model? \n\nBefore we do, let's code what we just wrote above. Please note, this is a manual process created by \"hand.\" You won't have to do this, but it's important to understand it before you start working with MLA. Think of MLA like a TI-89 calculator on a Calculus Exam. It's very powerful and helps you with a lot of the grunt work. But if you don't know what you're doing on the exam, a calculator, even a TI-89, is not going to help you pass. So, study the next section wisely.\n\nReference: [Cross-Validation and Decision Tree Tutorial](http://www.cs.utoronto.ca/~fidler/teaching/2015/slides/CSC411/tutorial3_CrossVal-DTs.pdf)"},{"metadata":{"_cell_guid":"7927b0e9-d000-4a67-a090-1fa19579e3ee","collapsed":true,"_uuid":"fba1acea71030c07c882deff78b917c19c8b7585","trusted":false},"cell_type":"code","source":"#IMPORTANT: This is a handmade model for learning purposes only.\n#However, it is possible to create your own predictive model without a fancy algorithm :)\n\n#coin flip model with random 1/survived 0/died\n\n#iterate over dataFrame rows as (index, Series) pairs: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.iterrows.html\nfor index, row in data1.iterrows(): \n    #random number generator: https://docs.python.org/2/library/random.html\n    if random.random() > .5:     # Random float x, 0.0 <= x < 1.0    \n        data1.set_value(index, 'Random_Predict', 1) #predict survived/1\n    else: \n        data1.set_value(index, 'Random_Predict', 0) #predict died/0\n    \n\n#score random guess of survival. Use shortcut 1 = Right Guess and 0 = Wrong Guess\n#the mean of the column will then equal the accuracy\ndata1['Random_Score'] = 0 #assume prediction wrong\ndata1.loc[(data1['Survived'] == data1['Random_Predict']), 'Random_Score'] = 1 #set to 1 for correct prediction\nprint('Coin Flip Model Accuracy: {:.2f}%'.format(data1['Random_Score'].mean()*100))\n\n#we can also use scikit's accuracy_score function to save us a few lines of code\n#http://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score\nprint('Coin Flip Model Accuracy w/SciKit: {:.2f}%'.format(metrics.accuracy_score(data1['Survived'], data1['Random_Predict'])*100))\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"cde02521-1ec1-4293-b8de-064c49118d91","collapsed":true,"_uuid":"c371166144bb9cd7925c3b8a3a00baf90bcebfb9","trusted":false},"cell_type":"code","source":"#group by or pivot table: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html\npivot_female = data1[data1.Sex=='female'].groupby(['Sex','Pclass', 'Embarked','FareBin'])['Survived'].mean()\nprint('Survival Decision Tree w/Female Node: \\n',pivot_female)\n\npivot_male = data1[data1.Sex=='male'].groupby(['Sex','Title'])['Survived'].mean()\nprint('\\n\\nSurvival Decision Tree w/Male Node: \\n',pivot_male)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"330582d5-8034-4c4a-93f0-60067939c5c0","collapsed":true,"_uuid":"82a3b4a54c2cb510f790ab2f7e3c52c46b63206b","trusted":false},"cell_type":"code","source":"#handmade data model using brain power (and Microsoft Excel Pivot Tables for quick calculations)\ndef mytree(df):\n    \n    #initialize table to store predictions\n    Model = pd.DataFrame(data = {'Predict':[]})\n    male_title = ['Master'] #survived titles\n\n    for index, row in df.iterrows():\n\n        #Question 1: Were you on the Titanic; majority died\n        Model.loc[index, 'Predict'] = 0\n\n        #Question 2: Are you female; majority survived\n        if (df.loc[index, 'Sex'] == 'female'):\n                  Model.loc[index, 'Predict'] = 1\n\n        #Question 3A Female - Class and Question 4 Embarked gain minimum information\n\n        #Question 5B Female - FareBin; set anything less than .5 in female node decision tree back to 0       \n        if ((df.loc[index, 'Sex'] == 'female') & \n            (df.loc[index, 'Pclass'] == 3) & \n            (df.loc[index, 'Embarked'] == 'S')  &\n            (df.loc[index, 'Fare'] > 8)\n\n           ):\n                  Model.loc[index, 'Predict'] = 0\n\n        #Question 3B Male: Title; set anything greater than .5 to 1 for majority survived\n        if ((df.loc[index, 'Sex'] == 'male') &\n            (df.loc[index, 'Title'] in male_title)\n            ):\n            Model.loc[index, 'Predict'] = 1\n        \n        \n    return Model\n\n\n#model data\nTree_Predict = mytree(data1)\nprint('Decision Tree Model Accuracy/Precision Score: {:.2f}%\\n'.format(metrics.accuracy_score(data1['Survived'], Tree_Predict)*100))\n\n\n#Accuracy Summary Report with http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html#sklearn.metrics.classification_report\n#Where recall score = (true positives)/(true positive + false negative) w/1 being best:http://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score\n#And F1 score = weighted average of precision and recall w/1 being best: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score\nprint(metrics.classification_report(data1['Survived'], Tree_Predict))\n\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"12c7bfa9-6876-4df7-adf9-147eac1bd43c","collapsed":true,"_uuid":"22413208c3179422ff603c059c695d78b4dd0db1","trusted":false},"cell_type":"code","source":"#Plot Accuracy Summary\n#Credit: http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html\nimport itertools\ndef plot_confusion_matrix(cm, classes,\n                          normalize=False,\n                          title='Confusion matrix',\n                          cmap=plt.cm.Blues):\n    \"\"\"\n    This function prints and plots the confusion matrix.\n    Normalization can be applied by setting `normalize=True`.\n    \"\"\"\n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n        print(\"Normalized confusion matrix\")\n    else:\n        print('Confusion matrix, without normalization')\n\n    print(cm)\n\n    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n    plt.title(title)\n    plt.colorbar()\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n\n    fmt = '.2f' if normalize else 'd'\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, format(cm[i, j], fmt),\n                 horizontalalignment=\"center\",\n                 color=\"white\" if cm[i, j] > thresh else \"black\")\n\n    plt.tight_layout()\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')\n\n# Compute confusion matrix\ncnf_matrix = metrics.confusion_matrix(data1['Survived'], Tree_Predict)\nnp.set_printoptions(precision=2)\n\nclass_names = ['Dead', 'Survived']\n# Plot non-normalized confusion matrix\nplt.figure()\nplot_confusion_matrix(cnf_matrix, classes=class_names,\n                      title='Confusion matrix, without normalization')\n\n# Plot normalized confusion matrix\nplt.figure()\nplot_confusion_matrix(cnf_matrix, classes=class_names, normalize=True, \n                      title='Normalized confusion matrix')\n\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ec85d77e-0be8-457e-b876-ccf3545d35e5","_uuid":"9e6bf62b0e2a7f8ce2cccd68310338dfc5d2c156"},"cell_type":"markdown","source":"## 5.11 Model Performance with Cross-Validation (CV)\nIn step 5.0, we used [sklearn cross_validate](http://scikit-learn.org/stable/modules/cross_validation.html#multimetric-cross-validation) function to train, test, and score our model performance.\n\nRemember, it's important we use a different subset for train data to build our model and test data to evaluate our model. Otherwise, our model will be overfitted. Meaning it's great at \"predicting\" data it's already seen, but terrible at predicting data it has not seen; which is not prediction at all. It's like cheating on a school quiz to get 100%, but then when you go to take the exam, you fail because you never truly learned anything. The same is true with machine learning.\n\nCV is basically a shortcut to split and score our model multiple times, so we can get an idea of how well it will perform on unseen data. It’s a little more expensive in computer processing, but it's important so we don't gain false confidence. This is helpful in a Kaggle Competition or any use case where consistency matters and surprises should be avoided.\n \nIn addition to CV, we used a customized [sklearn train test splitter](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection), to allow a little more randomness in our test scoring. Below is an image of the default CV split.\n\n![CV](http://blog-test.goldenhelix.com/wp-content/uploads/2015/04/B-fig-1.jpg)"},{"metadata":{"_cell_guid":"1ae78385-67ec-431e-98ad-1c34ab10499b","_uuid":"68978ae17037762ad30405bb51ec67326fecfdec"},"cell_type":"markdown","source":"<a id=\"ch9\"></a>\n# 5.12 Tune Model with Hyper-Parameters\nWhen we used [sklearn Decision Tree (DT) Classifier](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier), we accepted all the function defaults. This leaves opportunity to see how various hyper-parameter settings will change the model accuracy.  [(Click here to learn more about parameters vs hyper-parameters.)](https://www.youtube.com/watch?v=EJtTNboTsm8)\n\nHowever, in order to tune a model, we need to actually understand it. That's why I took the time in the previous sections to show you how predictions work. Now let's learn a little bit more about our DT algorithm.\n\nCredit: [sklearn](http://scikit-learn.org/stable/modules/tree.html#classification)\n\n>**Some advantages of decision trees are:**\n* Simple to understand and to interpret. Trees can be visualized.\n* Requires little data preparation. Other techniques often require data normalization, dummy variables need to be created and blank values to be removed. Note however that this module does not support missing values.\n* The cost of using the tree (i.e., predicting data) is logarithmic in the number of data points used to train the tree.\n* Able to handle both numerical and categorical data. Other techniques are usually specialized in analyzing datasets that have only one type of variable. See algorithms for more information.\n* Able to handle multi-output problems.\n* Uses a white box model. If a given situation is observable in a model, the explanation for the condition is easily explained by Boolean logic. By contrast, in a black box model (e.g., in an artificial neural network), results may be more difficult to interpret.\n* Possible to validate a model using statistical tests. That makes it possible to account for the reliability of the model.\n* Performs well even if its assumptions are somewhat violated by the true model from which the data were generated.\n\n> **The disadvantages of decision trees include:**\n* Decision-tree learners can create over-complex trees that do not generalize the data well. This is called overfitting. Mechanisms such as pruning (not currently supported), setting the minimum number of samples required at a leaf node or setting the maximum depth of the tree are necessary to avoid this problem.\n* Decision trees can be unstable because small variations in the data might result in a completely different tree being generated. This problem is mitigated by using decision trees within an ensemble.\n* The problem of learning an optimal decision tree is known to be NP-complete under several aspects of optimality and even for simple concepts. Consequently, practical decision-tree learning algorithms are based on heuristic algorithms such as the greedy algorithm where locally optimal decisions are made at each node. Such algorithms cannot guarantee to return the globally optimal decision tree. This can be mitigated by training multiple trees in an ensemble learner, where the features and samples are randomly sampled with replacement.\n* There are concepts that are hard to learn because decision trees do not express them easily, such as XOR, parity or multiplexer problems.\n* Decision tree learners create biased trees if some classes dominate. It is therefore recommended to balance the dataset prior to fitting with the decision tree.\n\n\n\nBelow are available hyper-parameters and [defintions](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier):\n> class sklearn.tree.DecisionTreeClassifier(criterion=’gini’, splitter=’best’, max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features=None, random_state=None, max_leaf_nodes=None, min_impurity_decrease=0.0, min_impurity_split=None, class_weight=None, presort=False)\n\n\nWe will tune our model using [ParameterGrid](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.ParameterGrid.html#sklearn.model_selection.ParameterGrid), [GridSearchCV](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV), and customized [sklearn scoring](http://scikit-learn.org/stable/modules/model_evaluation.html); [click here to learn more about ROC_AUC scores](http://www.dataschool.io/roc-curves-and-auc-explained/). We will then visualize our tree with [graphviz](http://scikit-learn.org/stable/modules/generated/sklearn.tree.export_graphviz.html#sklearn.tree.export_graphviz). [Click here to learn more about ROC_AUC scores](http://www.dataschool.io/roc-curves-and-auc-explained/).\n"},{"metadata":{"_cell_guid":"a614a6a0-1861-4d87-aaab-cc8f9b54bd82","collapsed":true,"_uuid":"4cfe6cb996dd39e1145effffa51a6953c396278d","trusted":false},"cell_type":"code","source":"#base model\ndtree = tree.DecisionTreeClassifier(random_state = 0)\nbase_results = model_selection.cross_validate(dtree, data1[data1_x_bin], data1[Target], cv  = cv_split)\ndtree.fit(data1[data1_x_bin], data1[Target])\n\nprint('BEFORE DT Parameters: ', dtree.get_params())\nprint(\"BEFORE DT Training w/bin score mean: {:.2f}\". format(base_results['train_score'].mean()*100)) \nprint(\"BEFORE DT Test w/bin score mean: {:.2f}\". format(base_results['test_score'].mean()*100))\nprint(\"BEFORE DT Test w/bin score 3*std: +/- {:.2f}\". format(base_results['test_score'].std()*100*3))\n#print(\"BEFORE DT Test w/bin set score min: {:.2f}\". format(base_results['test_score'].min()*100))\nprint('-'*10)\n\n\n#tune hyper-parameters: http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier\nparam_grid = {'criterion': ['gini', 'entropy'],  #scoring methodology; two supported formulas for calculating information gain - default is gini\n              #'splitter': ['best', 'random'], #splitting methodology; two supported strategies - default is best\n              'max_depth': [2,4,6,8,10,None], #max depth tree can grow; default is none\n              #'min_samples_split': [2,5,10,.03,.05], #minimum subset size BEFORE new split (fraction is % of total); default is 2\n              #'min_samples_leaf': [1,5,10,.03,.05], #minimum subset size AFTER new split split (fraction is % of total); default is 1\n              #'max_features': [None, 'auto'], #max features to consider when performing split; default none or all\n              'random_state': [0] #seed or control random number generator: https://www.quora.com/What-is-seed-in-random-number-generation\n             }\n\n#print(list(model_selection.ParameterGrid(param_grid)))\n\n#choose best model with grid_search: #http://scikit-learn.org/stable/modules/grid_search.html#grid-search\n#http://scikit-learn.org/stable/auto_examples/model_selection/plot_grid_search_digits.html\ntune_model = model_selection.GridSearchCV(tree.DecisionTreeClassifier(), param_grid=param_grid, scoring = 'roc_auc', cv = cv_split)\ntune_model.fit(data1[data1_x_bin], data1[Target])\n\n#print(tune_model.cv_results_.keys())\n#print(tune_model.cv_results_['params'])\nprint('AFTER DT Parameters: ', tune_model.best_params_)\n#print(tune_model.cv_results_['mean_train_score'])\nprint(\"AFTER DT Training w/bin score mean: {:.2f}\". format(tune_model.cv_results_['mean_train_score'][tune_model.best_index_]*100)) \n#print(tune_model.cv_results_['mean_test_score'])\nprint(\"AFTER DT Test w/bin score mean: {:.2f}\". format(tune_model.cv_results_['mean_test_score'][tune_model.best_index_]*100))\nprint(\"AFTER DT Test w/bin score 3*std: +/- {:.2f}\". format(tune_model.cv_results_['std_test_score'][tune_model.best_index_]*100*3))\nprint('-'*10)\n\n\n#duplicates gridsearchcv\n#tune_results = model_selection.cross_validate(tune_model, data1[data1_x_bin], data1[Target], cv  = cv_split)\n\n#print('AFTER DT Parameters: ', tune_model.best_params_)\n#print(\"AFTER DT Training w/bin set score mean: {:.2f}\". format(tune_results['train_score'].mean()*100)) \n#print(\"AFTER DT Test w/bin set score mean: {:.2f}\". format(tune_results['test_score'].mean()*100))\n#print(\"AFTER DT Test w/bin set score min: {:.2f}\". format(tune_results['test_score'].min()*100))\n#print('-'*10)\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"3e84f865-d316-4268-a422-c0062e4069c2","_uuid":"cf8c9846dfcbd7277756f1a573d55642d3f2f2be"},"cell_type":"markdown","source":"<a id=\"ch10\"></a>\n## 5.13 Tune Model with Feature Selection\nAs stated in the beginning, more predictor variables do not make a better model, but the right predictors do. So another step in data modeling is feature selection. [Sklearn](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.feature_selection) has several options, we will use [recursive feature elimination (RFE) with cross validation (CV)](http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html#sklearn.feature_selection.RFECV)."},{"metadata":{"_cell_guid":"df54f00c-e5a2-4994-aab8-f5b19961eafb","collapsed":true,"_uuid":"d35bf5c10d2dc9c582e5fea4956942182c6fd206","trusted":false},"cell_type":"code","source":"#base model\nprint('BEFORE DT RFE Training Shape Old: ', data1[data1_x_bin].shape) \nprint('BEFORE DT RFE Training Columns Old: ', data1[data1_x_bin].columns.values)\n\nprint(\"BEFORE DT RFE Training w/bin score mean: {:.2f}\". format(base_results['train_score'].mean()*100)) \nprint(\"BEFORE DT RFE Test w/bin score mean: {:.2f}\". format(base_results['test_score'].mean()*100))\nprint(\"BEFORE DT RFE Test w/bin score 3*std: +/- {:.2f}\". format(base_results['test_score'].std()*100*3))\nprint('-'*10)\n\n\n\n#feature selection\ndtree_rfe = feature_selection.RFECV(dtree, step = 1, scoring = 'accuracy', cv = cv_split)\ndtree_rfe.fit(data1[data1_x_bin], data1[Target])\n\n#transform x&y to reduced features and fit new model\n#alternative: can use pipeline to reduce fit and transform steps: http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html\nX_rfe = data1[data1_x_bin].columns.values[dtree_rfe.get_support()]\nrfe_results = model_selection.cross_validate(dtree, data1[X_rfe], data1[Target], cv  = cv_split)\n\n#print(dtree_rfe.grid_scores_)\nprint('AFTER DT RFE Training Shape New: ', data1[X_rfe].shape) \nprint('AFTER DT RFE Training Columns New: ', X_rfe)\n\nprint(\"AFTER DT RFE Training w/bin score mean: {:.2f}\". format(rfe_results['train_score'].mean()*100)) \nprint(\"AFTER DT RFE Test w/bin score mean: {:.2f}\". format(rfe_results['test_score'].mean()*100))\nprint(\"AFTER DT RFE Test w/bin score 3*std: +/- {:.2f}\". format(rfe_results['test_score'].std()*100*3))\nprint('-'*10)\n\n\n#tune rfe model\nrfe_tune_model = model_selection.GridSearchCV(tree.DecisionTreeClassifier(), param_grid=param_grid, scoring = 'roc_auc', cv = cv_split)\nrfe_tune_model.fit(data1[X_rfe], data1[Target])\n\n#print(rfe_tune_model.cv_results_.keys())\n#print(rfe_tune_model.cv_results_['params'])\nprint('AFTER DT RFE Tuned Parameters: ', rfe_tune_model.best_params_)\n#print(rfe_tune_model.cv_results_['mean_train_score'])\nprint(\"AFTER DT RFE Tuned Training w/bin score mean: {:.2f}\". format(rfe_tune_model.cv_results_['mean_train_score'][tune_model.best_index_]*100)) \n#print(rfe_tune_model.cv_results_['mean_test_score'])\nprint(\"AFTER DT RFE Tuned Test w/bin score mean: {:.2f}\". format(rfe_tune_model.cv_results_['mean_test_score'][tune_model.best_index_]*100))\nprint(\"AFTER DT RFE Tuned Test w/bin score 3*std: +/- {:.2f}\". format(rfe_tune_model.cv_results_['std_test_score'][tune_model.best_index_]*100*3))\nprint('-'*10)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a5737627-e5ea-480f-883b-eea1caaa0e46","collapsed":true,"_uuid":"7337e6c91797354d8fd887a538bf916a80619069","trusted":false},"cell_type":"code","source":"#Graph MLA version of Decision Tree: http://scikit-learn.org/stable/modules/generated/sklearn.tree.export_graphviz.html\nimport graphviz \ndot_data = tree.export_graphviz(dtree, out_file=None, \n                                feature_names = data1_x_bin, class_names = True,\n                                filled = True, rounded = True)\ngraph = graphviz.Source(dot_data) \ngraph","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"5f9bd02f-93ae-4cfa-992c-38b24c322011","_uuid":"6e2d288d4e477837f513e37461754256e1eef44a"},"cell_type":"markdown","source":"<a id=\"ch11\"></a>\n# Step 6: Validate and Implement\nThe next step is to prepare for submission using the validation data. "},{"metadata":{"_cell_guid":"336c5075-9077-479d-9d95-38b6c9b7b0c8","collapsed":true,"_uuid":"3648c38f48305bacd85ee8ee3048404631a03ad5","trusted":false},"cell_type":"code","source":"#compare algorithm predictions with each other, where 1 = exactly similar and 0 = exactly opposite\n#there are some 1's, but enough blues and light reds to create a \"super algorithm\" by combining them\ncorrelation_heatmap(MLA_predict)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"82d84146-6a2a-4b61-be81-922b3f35542f","collapsed":true,"_uuid":"80c3637e384b48e64f00462eb8931196457716d2","trusted":false},"cell_type":"code","source":"#why choose one model, when you can pick them all with voting classifier\n#http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html\n#removed models w/o attribute 'predict_proba' required for vote classifier and models with a 1.0 correlation to another model\nvote_est = [\n    #Ensemble Methods: http://scikit-learn.org/stable/modules/ensemble.html\n    ('ada', ensemble.AdaBoostClassifier()),\n    ('bc', ensemble.BaggingClassifier()),\n    ('etc',ensemble.ExtraTreesClassifier()),\n    ('gbc', ensemble.GradientBoostingClassifier()),\n    ('rfc', ensemble.RandomForestClassifier()),\n\n    #Gaussian Processes: http://scikit-learn.org/stable/modules/gaussian_process.html#gaussian-process-classification-gpc\n    ('gpc', gaussian_process.GaussianProcessClassifier()),\n    \n    #GLM: http://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n    ('lr', linear_model.LogisticRegressionCV()),\n    \n    #Navies Bayes: http://scikit-learn.org/stable/modules/naive_bayes.html\n    ('bnb', naive_bayes.BernoulliNB()),\n    ('gnb', naive_bayes.GaussianNB()),\n    \n    #Nearest Neighbor: http://scikit-learn.org/stable/modules/neighbors.html\n    ('knn', neighbors.KNeighborsClassifier()),\n    \n    #SVM: http://scikit-learn.org/stable/modules/svm.html\n    ('svc', svm.SVC(probability=True)),\n    \n    #xgboost: http://xgboost.readthedocs.io/en/latest/model.html\n   ('xgb', XGBClassifier())\n\n]\n\n\n#Hard Vote or majority rules\nvote_hard = ensemble.VotingClassifier(estimators = vote_est , voting = 'hard')\nvote_hard_cv = model_selection.cross_validate(vote_hard, data1[data1_x_bin], data1[Target], cv  = cv_split)\nvote_hard.fit(data1[data1_x_bin], data1[Target])\n\nprint(\"Hard Voting Training w/bin score mean: {:.2f}\". format(vote_hard_cv['train_score'].mean()*100)) \nprint(\"Hard Voting Test w/bin score mean: {:.2f}\". format(vote_hard_cv['test_score'].mean()*100))\nprint(\"Hard Voting Test w/bin score 3*std: +/- {:.2f}\". format(vote_hard_cv['test_score'].std()*100*3))\nprint('-'*10)\n\n\n#Soft Vote or weighted probabilities\nvote_soft = ensemble.VotingClassifier(estimators = vote_est , voting = 'soft')\nvote_soft_cv = model_selection.cross_validate(vote_soft, data1[data1_x_bin], data1[Target], cv  = cv_split)\nvote_soft.fit(data1[data1_x_bin], data1[Target])\n\nprint(\"Soft Voting Training w/bin score mean: {:.2f}\". format(vote_soft_cv['train_score'].mean()*100)) \nprint(\"Soft Voting Test w/bin score mean: {:.2f}\". format(vote_soft_cv['test_score'].mean()*100))\nprint(\"Soft Voting Test w/bin score 3*std: +/- {:.2f}\". format(vote_soft_cv['test_score'].std()*100*3))\nprint('-'*10)\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c62f344d-24c3-4c3d-8a7a-7d1013390091","collapsed":true,"_uuid":"20d362a06ca426b4bc8b2413209885f9f7853a3f","_kg_hide-output":true,"_kg_hide-input":true,"trusted":false},"cell_type":"code","source":"#IMPORTANT: THIS SECTION IS UNDER CONSTRUCTION!!!! 12.24.17\n#UPDATE: This section was scrapped for the next section; as it's more computational friendly.\n\n#WARNING: Running is very computational intensive and time expensive\n#code is written for experimental/developmental purposes and not production ready\n\n\n#tune each estimator before creating a super model\n#http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html\ngrid_n_estimator = [50,100,300]\ngrid_ratio = [.1,.25,.5,.75,1.0]\ngrid_learn = [.01,.03,.05,.1,.25]\ngrid_max_depth = [2,4,6,None]\ngrid_min_samples = [5,10,.03,.05,.10]\ngrid_criterion = ['gini', 'entropy']\ngrid_bool = [True, False]\ngrid_seed = [0]\n\nvote_param = [{\n#            #http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html\n            'ada__n_estimators': grid_n_estimator,\n            'ada__learning_rate': grid_ratio,\n            'ada__algorithm': ['SAMME', 'SAMME.R'],\n            'ada__random_state': grid_seed,\n    \n            #http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html#sklearn.ensemble.BaggingClassifier\n            'bc__n_estimators': grid_n_estimator,\n            'bc__max_samples': grid_ratio,\n            'bc__oob_score': grid_bool, \n            'bc__random_state': grid_seed,\n            \n            #http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html#sklearn.ensemble.ExtraTreesClassifier\n            'etc__n_estimators': grid_n_estimator,\n            'etc__criterion': grid_criterion,\n            'etc__max_depth': grid_max_depth,\n            'etc__random_state': grid_seed,\n\n\n            #http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn.ensemble.GradientBoostingClassifier\n            'gbc__loss': ['deviance', 'exponential'],\n            'gbc__learning_rate': grid_ratio,\n            'gbc__n_estimators': grid_n_estimator,\n            'gbc__criterion': ['friedman_mse', 'mse', 'mae'],\n            'gbc__max_depth': grid_max_depth,\n            'gbc__min_samples_split': grid_min_samples,\n            'gbc__min_samples_leaf': grid_min_samples,      \n            'gbc__random_state': grid_seed,\n    \n            #http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier\n            'rfc__n_estimators': grid_n_estimator,\n            'rfc__criterion': grid_criterion,\n            'rfc__max_depth': grid_max_depth,\n            'rfc__min_samples_split': grid_min_samples,\n            'rfc__min_samples_leaf': grid_min_samples,   \n            'rfc__bootstrap': grid_bool,\n            'rfc__oob_score': grid_bool, \n            'rfc__random_state': grid_seed,\n        \n            #http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html#sklearn.linear_model.LogisticRegressionCV\n            'lr__fit_intercept': grid_bool,\n            'lr__penalty': ['l1','l2'],\n            'lr__solver': ['newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'],\n            'lr__random_state': grid_seed,\n            \n            #http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.BernoulliNB.html#sklearn.naive_bayes.BernoulliNB\n            'bnb__alpha': grid_ratio,\n            'bnb__prior': grid_bool,\n            'bnb__random_state': grid_seed,\n    \n            #http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn.neighbors.KNeighborsClassifier\n            'knn__n_neighbors': [1,2,3,4,5,6,7],\n            'knn__weights': ['uniform', 'distance'],\n            'knn__algorithm': ['auto', 'ball_tree', 'kd_tree', 'brute'],\n            'knn__random_state': grid_seed,\n            \n            #http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC\n            #http://blog.hackerearth.com/simple-tutorial-svm-parameter-tuning-python-r\n            'svc__kernel': ['linear', 'poly', 'rbf', 'sigmoid'],\n            'svc__C': grid_max_depth,\n            'svc__gamma': grid_ratio,\n            'svc__decision_function_shape': ['ovo', 'ovr'],\n            'svc__probability': [True],\n            'svc__random_state': grid_seed,\n    \n    \n            #http://xgboost.readthedocs.io/en/latest/parameter.html\n            'xgb__learning_rate': grid_ratio,\n            'xgb__max_depth': [2,4,6,8,10],\n            'xgb__tree_method': ['exact', 'approx', 'hist'],\n            'xgb__objective': ['reg:linear', 'reg:logistic', 'binary:logistic'],\n            'xgb__seed': grid_seed    \n\n        }]\n\n\n\n\n#Soft Vote with tuned models\n#grid_soft = model_selection.GridSearchCV(estimator = vote_soft, param_grid = vote_param, cv = 2, scoring = 'roc_auc')\n#grid_soft.fit(data1[data1_x_bin], data1[Target])\n\n#print(grid_soft.cv_results_.keys())\n#print(grid_soft.cv_results_['params'])\n#print('Soft Vote Tuned Parameters: ', grid_soft.best_params_)\n#print(grid_soft.cv_results_['mean_train_score'])\n#print(\"Soft Vote Tuned Training w/bin set score mean: {:.2f}\". format(grid_soft.cv_results_['mean_train_score'][tune_model.best_index_]*100)) \n#print(grid_soft.cv_results_['mean_test_score'])\n#print(\"Soft Vote Tuned Test w/bin set score mean: {:.2f}\". format(grid_soft.cv_results_['mean_test_score'][tune_model.best_index_]*100))\n#print(\"Soft Vote Tuned Test w/bin score 3*std: +/- {:.2f}\". format(grid_soft.cv_results_['std_test_score'][tune_model.best_index_]*100*3))\n#print('-'*10)\n\n\n#credit: https://rasbt.github.io/mlxtend/user_guide/classifier/EnsembleVoteClassifier/\n#cv_keys = ('mean_test_score', 'std_test_score', 'params')\n#for r, _ in enumerate(grid_soft.cv_results_['mean_test_score']):\n#    print(\"%0.3f +/- %0.2f %r\"\n#          % (grid_soft.cv_results_[cv_keys[0]][r],\n#             grid_soft.cv_results_[cv_keys[1]][r] / 2.0,\n#             grid_soft.cv_results_[cv_keys[2]][r]))\n\n\n#print('-'*10)\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"7b605629-06f5-4bdb-b7da-c0f842ef678f","collapsed":true,"_uuid":"77e83a4e5a31b9276767f550cca0de1bc53f5f70","trusted":false},"cell_type":"code","source":"#WARNING: Running is very computational intensive and time expensive.\n#Code is written for experimental/developmental purposes and not production ready!\n\n\n#Hyperparameter Tune with GridSearchCV: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html\ngrid_n_estimator = [10, 50, 100, 300]\ngrid_ratio = [.1, .25, .5, .75, 1.0]\ngrid_learn = [.01, .03, .05, .1, .25]\ngrid_max_depth = [2, 4, 6, 8, 10, None]\ngrid_min_samples = [5, 10, .03, .05, .10]\ngrid_criterion = ['gini', 'entropy']\ngrid_bool = [True, False]\ngrid_seed = [0]\n\n\ngrid_param = [\n            [{\n            #AdaBoostClassifier - http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html\n            'n_estimators': grid_n_estimator, #default=50\n            'learning_rate': grid_learn, #default=1\n            #'algorithm': ['SAMME', 'SAMME.R'], #default=’SAMME.R\n            'random_state': grid_seed\n            }],\n       \n    \n            [{\n            #BaggingClassifier - http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html#sklearn.ensemble.BaggingClassifier\n            'n_estimators': grid_n_estimator, #default=10\n            'max_samples': grid_ratio, #default=1.0\n            'random_state': grid_seed\n             }],\n\n    \n            [{\n            #ExtraTreesClassifier - http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html#sklearn.ensemble.ExtraTreesClassifier\n            'n_estimators': grid_n_estimator, #default=10\n            'criterion': grid_criterion, #default=”gini”\n            'max_depth': grid_max_depth, #default=None\n            'random_state': grid_seed\n             }],\n\n\n            [{\n            #GradientBoostingClassifier - http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn.ensemble.GradientBoostingClassifier\n            #'loss': ['deviance', 'exponential'], #default=’deviance’\n            'learning_rate': [.05], #default=0.1 -- 12/31/17 set to reduce runtime -- The best parameter for GradientBoostingClassifier is {'learning_rate': 0.05, 'max_depth': 2, 'n_estimators': 300, 'random_state': 0} with a runtime of 264.45 seconds.\n            'n_estimators': [300], #default=100 -- 12/31/17 set to reduce runtime -- The best parameter for GradientBoostingClassifier is {'learning_rate': 0.05, 'max_depth': 2, 'n_estimators': 300, 'random_state': 0} with a runtime of 264.45 seconds.\n            #'criterion': ['friedman_mse', 'mse', 'mae'], #default=”friedman_mse”\n            'max_depth': grid_max_depth, #default=3   \n            'random_state': grid_seed\n             }],\n\n    \n            [{\n            #RandomForestClassifier - http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier\n            'n_estimators': grid_n_estimator, #default=10\n            'criterion': grid_criterion, #default=”gini”\n            'max_depth': grid_max_depth, #default=None\n            'oob_score': [True], #default=False -- 12/31/17 set to reduce runtime -- The best parameter for RandomForestClassifier is {'criterion': 'entropy', 'max_depth': 6, 'n_estimators': 100, 'oob_score': True, 'random_state': 0} with a runtime of 146.35 seconds.\n            'random_state': grid_seed\n             }],\n    \n            [{    \n            #GaussianProcessClassifier\n            'max_iter_predict': grid_n_estimator, #default: 100\n            'random_state': grid_seed\n            }],\n        \n    \n            [{\n            #LogisticRegressionCV - http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html#sklearn.linear_model.LogisticRegressionCV\n            'fit_intercept': grid_bool, #default: True\n            #'penalty': ['l1','l2'],\n            'solver': ['newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'], #default: lbfgs\n            'random_state': grid_seed\n             }],\n            \n    \n            [{\n            #BernoulliNB - http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.BernoulliNB.html#sklearn.naive_bayes.BernoulliNB\n            'alpha': grid_ratio, #default: 1.0\n             }],\n    \n    \n            #GaussianNB - \n            [{}],\n    \n            [{\n            #KNeighborsClassifier - http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn.neighbors.KNeighborsClassifier\n            'n_neighbors': [1,2,3,4,5,6,7], #default: 5\n            'weights': ['uniform', 'distance'], #default = ‘uniform’\n            'algorithm': ['auto', 'ball_tree', 'kd_tree', 'brute']\n            }],\n            \n    \n            [{\n            #SVC - http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC\n            #http://blog.hackerearth.com/simple-tutorial-svm-parameter-tuning-python-r\n            #'kernel': ['linear', 'poly', 'rbf', 'sigmoid'],\n            'C': [1,2,3,4,5], #default=1.0\n            'gamma': grid_ratio, #edfault: auto\n            'decision_function_shape': ['ovo', 'ovr'], #default:ovr\n            'probability': [True],\n            'random_state': grid_seed\n             }],\n\n    \n            [{\n            #XGBClassifier - http://xgboost.readthedocs.io/en/latest/parameter.html\n            'learning_rate': grid_learn, #default: .3\n            'max_depth': [1,2,4,6,8,10], #default 2\n            'n_estimators': grid_n_estimator, \n            'seed': grid_seed  \n             }]   \n        ]\n\n\n\nstart_total = time.perf_counter() #https://docs.python.org/3/library/time.html#time.perf_counter\nfor clf, param in zip (vote_est, grid_param): #https://docs.python.org/3/library/functions.html#zip\n\n    #print(clf[1]) #vote_est is a list of tuples, index 0 is the name and index 1 is the algorithm\n    #print(param)\n    \n    \n    start = time.perf_counter()        \n    best_search = model_selection.GridSearchCV(estimator = clf[1], param_grid = param, cv = cv_split, scoring = 'roc_auc')\n    best_search.fit(data1[data1_x_bin], data1[Target])\n    run = time.perf_counter() - start\n\n    best_param = best_search.best_params_\n    print('The best parameter for {} is {} with a runtime of {:.2f} seconds.'.format(clf[1].__class__.__name__, best_param, run))\n    clf[1].set_params(**best_param) \n\n\nrun_total = time.perf_counter() - start_total\nprint('Total optimization time was {:.2f} minutes.'.format(run_total/60))\n\nprint('-'*10)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"309fed2c-44d2-481c-807a-bd062f5a329c","collapsed":true,"_uuid":"b09f7cbe41527b36e4d3754febda6c1b395bd7e9","trusted":false},"cell_type":"code","source":"#Hard Vote or majority rules w/Tuned Hyperparameters\ngrid_hard = ensemble.VotingClassifier(estimators = vote_est , voting = 'hard')\ngrid_hard_cv = model_selection.cross_validate(grid_hard, data1[data1_x_bin], data1[Target], cv  = cv_split)\ngrid_hard.fit(data1[data1_x_bin], data1[Target])\n\nprint(\"Hard Voting w/Tuned Hyperparameters Training w/bin score mean: {:.2f}\". format(grid_hard_cv['train_score'].mean()*100)) \nprint(\"Hard Voting w/Tuned Hyperparameters Test w/bin score mean: {:.2f}\". format(grid_hard_cv['test_score'].mean()*100))\nprint(\"Hard Voting w/Tuned Hyperparameters Test w/bin score 3*std: +/- {:.2f}\". format(grid_hard_cv['test_score'].std()*100*3))\nprint('-'*10)\n\n#Soft Vote or weighted probabilities w/Tuned Hyperparameters\ngrid_soft = ensemble.VotingClassifier(estimators = vote_est , voting = 'soft')\ngrid_soft_cv = model_selection.cross_validate(grid_soft, data1[data1_x_bin], data1[Target], cv  = cv_split)\ngrid_soft.fit(data1[data1_x_bin], data1[Target])\n\nprint(\"Soft Voting w/Tuned Hyperparameters Training w/bin score mean: {:.2f}\". format(grid_soft_cv['train_score'].mean()*100)) \nprint(\"Soft Voting w/Tuned Hyperparameters Test w/bin score mean: {:.2f}\". format(grid_soft_cv['test_score'].mean()*100))\nprint(\"Soft Voting w/Tuned Hyperparameters Test w/bin score 3*std: +/- {:.2f}\". format(grid_soft_cv['test_score'].std()*100*3))\nprint('-'*10)\n\n\n#12/31/17 tuned with data1_x_bin\n#The best parameter for AdaBoostClassifier is {'learning_rate': 0.1, 'n_estimators': 300, 'random_state': 0} with a runtime of 33.39 seconds.\n#The best parameter for BaggingClassifier is {'max_samples': 0.25, 'n_estimators': 300, 'random_state': 0} with a runtime of 30.28 seconds.\n#The best parameter for ExtraTreesClassifier is {'criterion': 'entropy', 'max_depth': 6, 'n_estimators': 100, 'random_state': 0} with a runtime of 64.76 seconds.\n#The best parameter for GradientBoostingClassifier is {'learning_rate': 0.05, 'max_depth': 2, 'n_estimators': 300, 'random_state': 0} with a runtime of 34.35 seconds.\n#The best parameter for RandomForestClassifier is {'criterion': 'entropy', 'max_depth': 6, 'n_estimators': 100, 'oob_score': True, 'random_state': 0} with a runtime of 76.32 seconds.\n#The best parameter for GaussianProcessClassifier is {'max_iter_predict': 10, 'random_state': 0} with a runtime of 6.01 seconds.\n#The best parameter for LogisticRegressionCV is {'fit_intercept': True, 'random_state': 0, 'solver': 'liblinear'} with a runtime of 8.04 seconds.\n#The best parameter for BernoulliNB is {'alpha': 0.1} with a runtime of 0.19 seconds.\n#The best parameter for GaussianNB is {} with a runtime of 0.04 seconds.\n#The best parameter for KNeighborsClassifier is {'algorithm': 'brute', 'n_neighbors': 7, 'weights': 'uniform'} with a runtime of 4.84 seconds.\n#The best parameter for SVC is {'C': 2, 'decision_function_shape': 'ovo', 'gamma': 0.1, 'probability': True, 'random_state': 0} with a runtime of 29.39 seconds.\n#The best parameter for XGBClassifier is {'learning_rate': 0.01, 'max_depth': 4, 'n_estimators': 300, 'seed': 0} with a runtime of 46.23 seconds.\n#Total optimization time was 5.56 minutes.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"057f369f-a3c6-4718-94f4-db8cb6d59777","collapsed":true,"_uuid":"29ad86c5413910763316df7fda7753c4b0d0a97d","trusted":false},"cell_type":"code","source":"#prepare data for modeling\nprint(data_val.info())\nprint(\"-\"*10)\n#data_val.sample(10)\n\n\n\n#handmade decision tree - submission score = 0.77990\ndata_val['Survived'] = mytree(data_val).astype(int)\n\n\n#decision tree w/full dataset modeling submission score: defaults= 0.76555, tuned= 0.77990\n#submit_dt = tree.DecisionTreeClassifier()\n#submit_dt = model_selection.GridSearchCV(tree.DecisionTreeClassifier(), param_grid=param_grid, scoring = 'roc_auc', cv = cv_split)\n#submit_dt.fit(data1[data1_x_bin], data1[Target])\n#print('Best Parameters: ', submit_dt.best_params_) #Best Parameters:  {'criterion': 'gini', 'max_depth': 4, 'random_state': 0}\n#data_val['Survived'] = submit_dt.predict(data_val[data1_x_bin])\n\n\n#bagging w/full dataset modeling submission score: defaults= 0.75119, tuned= 0.77990\n#submit_bc = ensemble.BaggingClassifier()\n#submit_bc = model_selection.GridSearchCV(ensemble.BaggingClassifier(), param_grid= {'n_estimators':grid_n_estimator, 'max_samples': grid_ratio, 'oob_score': grid_bool, 'random_state': grid_seed}, scoring = 'roc_auc', cv = cv_split)\n#submit_bc.fit(data1[data1_x_bin], data1[Target])\n#print('Best Parameters: ', submit_bc.best_params_) #Best Parameters:  {'max_samples': 0.25, 'n_estimators': 500, 'oob_score': True, 'random_state': 0}\n#data_val['Survived'] = submit_bc.predict(data_val[data1_x_bin])\n\n\n#extra tree w/full dataset modeling submission score: defaults= 0.76555, tuned= 0.77990\n#submit_etc = ensemble.ExtraTreesClassifier()\n#submit_etc = model_selection.GridSearchCV(ensemble.ExtraTreesClassifier(), param_grid={'n_estimators': grid_n_estimator, 'criterion': grid_criterion, 'max_depth': grid_max_depth, 'random_state': grid_seed}, scoring = 'roc_auc', cv = cv_split)\n#submit_etc.fit(data1[data1_x_bin], data1[Target])\n#print('Best Parameters: ', submit_etc.best_params_) #Best Parameters:  {'criterion': 'entropy', 'max_depth': 6, 'n_estimators': 100, 'random_state': 0}\n#data_val['Survived'] = submit_etc.predict(data_val[data1_x_bin])\n\n\n#random foreset w/full dataset modeling submission score: defaults= 0.71291, tuned= 0.73205\n#submit_rfc = ensemble.RandomForestClassifier()\n#submit_rfc = model_selection.GridSearchCV(ensemble.RandomForestClassifier(), param_grid={'n_estimators': grid_n_estimator, 'criterion': grid_criterion, 'max_depth': grid_max_depth, 'random_state': grid_seed}, scoring = 'roc_auc', cv = cv_split)\n#submit_rfc.fit(data1[data1_x_bin], data1[Target])\n#print('Best Parameters: ', submit_rfc.best_params_) #Best Parameters:  {'criterion': 'entropy', 'max_depth': 6, 'n_estimators': 100, 'random_state': 0}\n#data_val['Survived'] = submit_rfc.predict(data_val[data1_x_bin])\n\n\n\n#ada boosting w/full dataset modeling submission score: defaults= 0.74162, tuned= 0.75119\n#submit_abc = ensemble.AdaBoostClassifier()\n#submit_abc = model_selection.GridSearchCV(ensemble.AdaBoostClassifier(), param_grid={'n_estimators': grid_n_estimator, 'learning_rate': grid_ratio, 'algorithm': ['SAMME', 'SAMME.R'], 'random_state': grid_seed}, scoring = 'roc_auc', cv = cv_split)\n#submit_abc.fit(data1[data1_x_bin], data1[Target])\n#print('Best Parameters: ', submit_abc.best_params_) #Best Parameters:  {'algorithm': 'SAMME.R', 'learning_rate': 0.1, 'n_estimators': 300, 'random_state': 0}\n#data_val['Survived'] = submit_abc.predict(data_val[data1_x_bin])\n\n\n#gradient boosting w/full dataset modeling submission score: defaults= 0.75119, tuned= 0.77033\n#submit_gbc = ensemble.GradientBoostingClassifier()\n#submit_gbc = model_selection.GridSearchCV(ensemble.GradientBoostingClassifier(), param_grid={'learning_rate': grid_ratio, 'n_estimators': grid_n_estimator, 'max_depth': grid_max_depth, 'random_state':grid_seed}, scoring = 'roc_auc', cv = cv_split)\n#submit_gbc.fit(data1[data1_x_bin], data1[Target])\n#print('Best Parameters: ', submit_gbc.best_params_) #Best Parameters:  {'learning_rate': 0.25, 'max_depth': 2, 'n_estimators': 50, 'random_state': 0}\n#data_val['Survived'] = submit_gbc.predict(data_val[data1_x_bin])\n\n#extreme boosting w/full dataset modeling submission score: defaults= 0.73684, tuned= 0.77990\n#submit_xgb = XGBClassifier()\n#submit_xgb = model_selection.GridSearchCV(XGBClassifier(), param_grid= {'learning_rate': grid_learn, 'max_depth': [0,2,4,6,8,10], 'n_estimators': grid_n_estimator, 'seed': grid_seed}, scoring = 'roc_auc', cv = cv_split)\n#submit_xgb.fit(data1[data1_x_bin], data1[Target])\n#print('Best Parameters: ', submit_xgb.best_params_) #Best Parameters:  {'learning_rate': 0.01, 'max_depth': 4, 'n_estimators': 300, 'seed': 0}\n#data_val['Survived'] = submit_xgb.predict(data_val[data1_x_bin])\n\n\n#hard voting classifier w/full dataset modeling submission score: defaults= 0.75598, tuned = 0.77990\n#data_val['Survived'] = vote_hard.predict(data_val[data1_x_bin])\ndata_val['Survived'] = grid_hard.predict(data_val[data1_x_bin])\n\n\n#soft voting classifier w/full dataset modeling submission score: defaults= 0.73684, tuned = 0.74162\n#data_val['Survived'] = vote_soft.predict(data_val[data1_x_bin])\n#data_val['Survived'] = grid_soft.predict(data_val[data1_x_bin])\n\n\n#submit file\nsubmit = data_val[['PassengerId','Survived']]\nsubmit.to_csv(\"../working/submit.csv\", index=False)\n\nprint('Validation Data Distribution: \\n', data_val['Survived'].value_counts(normalize = True))\nsubmit.sample(10)\n\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a6d34edc-1265-41e8-b163-0b817cd3b6a5","_uuid":"c476df95925d67809975caca4ea4dcb4b2a0b277"},"cell_type":"markdown","source":"<a id=\"ch12\"></a>\n# Step 7: Optimize and Strategize\n## Conclusion\nIteration one of the Data Science Framework, seems to converge on 0.77990 submission accuracy. Using the same dataset and different implementation of a decision tree (adaboost, random forest, gradient boost, xgboost, etc.) with tuning does not exceed the 0.77990 submission accuracy. Interesting for this dataset, the simple decision tree algorithm had the best default submission score and with tuning achieved the same best accuracy score.\n\nWhile no general conclusions can be made from testing a handful of algorithms on a single dataset, there are several observations on the mentioned dataset. \n1. The train dataset has a different distribution than the test/validation dataset and population. This created wide margins between the cross validation (CV) accuracy score and Kaggle submission accuracy score.\n2. Given the same dataset, decision tree based algorithms, seemed to converge on the same accuracy score after proper tuning.\n3. Despite tuning, no machine learning algorithm, exceeded the homemade algorithm. The author will theorize, that for small datasets, a manmade algorithm is the bar to beat. \n\nWith that in mind, for iteration two, I would spend more time on preprocessing and feature engineering. In order to better align the CV score and Kaggle score and improve the overall accuracy.\n\n"},{"metadata":{"_cell_guid":"973396d8-c37f-4f74-baec-a0c0653c975e","_uuid":"4c6df5d0a4ecf018846830f1ab4310549977a51d"},"cell_type":"markdown","source":"<a id=\"ch90\"></a>\n## Change Log:\n11/22/17 Please note, this kernel is currently in progress, but open to feedback. Thanks!  \n11/23/17 Cleaned up published notebook and updated through step 3.  \n11/25/17 Added enhancements to published notebook and started step 4.  \n11/26/17 Skipped ahead to data model, since this is a published notebook. Accuracy with (very) simple data cleaning and logistic regression is **~82%**. Continue to up vote and I will continue to develop this notebook. Thanks!  \n12/2/17 Updated section 4 with exploratory analysis and section 5 with more classifiers. Improved model to **~85%** accuracy.  \n12/3/17 Update section 4 with improved graphical statistics.  \n12/7/17 Updated section 5 with Data Science 101 Lesson.  \n12/8/17 Reorganized section 3 & 4 with cleaner code.  \n12/9/17 Updated section 5 with model optimization how-tos. Initial competition submission with Decision Tree; will update with better algorithm later.  \n12/10/17 Updated section 3 & 4 with cleaner code and better datasets.  \n12/11/17 Updated section 5 with better how-tos.  \n12/12/17 Cleaned section 5 to prep for hyper-parameter tuning.  \n12/13/17 Updated section 5 to focus on learning data modeling via decision tree.  \n12/20/17 Updated section 4 - Thanks @Daniel M. for suggestion to split up visualization code. Started working on section 6 for \"super\" model.  \n12/23/17 Edited section 1-5 for clarity and more concise code.  \n12/24/17 Updated section 5 with random_state and score for more consistent results.  \n12/31/17 Completed data science framework iteration 1 and added section 7 with conclusion.  "},{"metadata":{"_cell_guid":"5678e859-6610-4895-8fe8-0d28f0767b25","_uuid":"b04d314d7ffa7e1d48b3ab6a90004fcee56b0ea9"},"cell_type":"markdown","source":"<a id=\"ch91\"></a>\n# Credits\nProgramming is all about \"borrowing\" code, because knife sharpens knife. Nonetheless, I want to give credit, where credit is due. \n\n* [Introduction to Machine Learning with Python: A Guide for Data Scientists by Andreas Müller and Sarah Guido](https://www.amazon.com/gp/product/1449369413/ref=as_li_tl?ie=UTF8&tag=kaggle-20&camp=1789&creative=9325&linkCode=as2&creativeASIN=1449369413&linkId=740510c3199892cca1632fe738fb8d08) - Machine Learning 101 written by a core developer of sklearn\n* [Visualize This: The Flowing Data Guide to Design, Visualization, and Statistics by Nathan Yau](https://www.amazon.com/gp/product/0470944889/ref=as_li_tl?ie=UTF8&tag=kaggle-20&camp=1789&creative=9325&linkCode=as2&creativeASIN=0470944889&linkId=f797da48813ed5cfc762ce5df8ef957f) - Learn the art and science of data visualization\n* [Machine Learning for Dummies by John Mueller and Luca Massaron ](https://www.amazon.com/gp/product/1119245516/ref=as_li_tl?ie=UTF8&tag=kaggle-20&camp=1789&creative=9325&linkCode=as2&creativeASIN=1119245516&linkId=5b4ac9a6fd1da198d82f9ca841d1af9f) - Easy to understand for a beginner book, but detailed to actually learn the fundamentals of the topic\n"},{"metadata":{"_cell_guid":"51f197c2-76d7-4503-ade5-f9f35323c4c5","_uuid":"2e95a403934cf5c0ab1ff9148b62ef14590e84c9"},"cell_type":"markdown","source":"\n"}],"metadata":{"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}