{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Titanic - Doing it right! Pipeline and Evaluation\n### Introduction\nHi! Welcome to this notebook. I will walk you through several stages regarding the Titanic machine learning competition provided by kaggle. The general aim is to predict which passengers would have survived the shipwreck using various data points from passengers on the Titanic. The focus of this notebook lies on two important concepts people can get wrong:\n* Correct training in connection to **Data Leakage**\n* Correct evaluation in connection to Train Set, Validation Set and Test Set\n\nPlease note that I am not a professional or have detailed explanations for most of the techniques I try to describe! If you want to learn more I strongly recommend the book **\"Data Science for Business\"** by Foster Provost and Tom Fawcett.\n\n### Workflow\n1. Libraries\n2. Acquire Data\n3. Analysis\n4. Preprocessing\n    * Feature Engineering\n    * Pipeline\n5. Training\n6. Evaluation\n7. Prediction","metadata":{}},{"cell_type":"markdown","source":"# Libraries\nBesides the standard libraries like numpy or pandas we use multiple libraries to build our pipeline and acquire our model.","metadata":{}},{"cell_type":"code","source":"#Path\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n#Standard\nimport numpy as np # linear algebra\nimport pandas as pd # data processing\nimport matplotlib.pyplot as plt # data visualization\n\n#Preprocessing\nfrom sklearn.impute import KNNImputer # nearest neighbor imputer\nfrom sklearn.preprocessing import StandardScaler # scaler\n\n#Pipeline\nfrom sklearn.pipeline import Pipeline # pipeline\nfrom sklearn.base import BaseEstimator, TransformerMixin # custom pipeline methods\nfrom sklearn.compose import ColumnTransformer # pipeline unit\n\n#Training\nfrom sklearn.model_selection import train_test_split # train test split\nfrom sklearn.ensemble import RandomForestClassifier # random forest model\nfrom sklearn.ensemble import GradientBoostingClassifier # gradient boosting model\nfrom sklearn.model_selection import GridSearchCV # grid search technique\n\n#Evaluation\nfrom sklearn.metrics import accuracy_score # evaluation method","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Acquire Data & General Information\nWe acquire the labelled Train Data (train_data in code) and unlabeled Test Data which we call the Score Set ('score_set' in code) to avoid confusions later on. We then look at the columns and data types, the first few lines of each set, and the general distribution of each column. It provides us with a general overview and a first insight into the data.\nOne important aspect is that both the Train Data and Score Set have the same columns and that a column has the same type in both sets.","metadata":{}},{"cell_type":"code","source":"train_data = pd.read_csv(\"/kaggle/input/titanic/train.csv\") # Train Data\nscore_set = pd.read_csv(\"/kaggle/input/titanic/test.csv\") # Score Set","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One of the most important information sources is the description of the features provided by the challenge itself.\n\n| Variable        | Definition           | Key  |\n| ------------- |:-------------:| -----:|\n| survival      | Survival      | 0 = No, 1 = Yes |\n| pclass        | Ticket class  | 1 = 1st, 2 = 2nd, 3 = rd |\n| sex           | Sex           |  |\n| Age          | Age in years           |  |\n| sibsp           | # of siblings / spouses aboard the Titanic         |  |\n| parch          | # of parents / children aboard the Titanic          |  |\n| ticket           | Ticket number           |  |\n| fare           | Passenger fare           |  |\n| cabin           | Cabin number           |  |\n| embarked           | Port of Embarkation           | C = Cherbourg, Q = Queenstown, S = Southampton |","metadata":{}},{"cell_type":"code","source":"train_data.info() # Feature Information","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.head() # First five rows","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.describe(include='all') # Descriptive Statistics","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"score_set.info() # Feature Information","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"score_set.head() # First five rows","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"score_set.describe(include='all') # Descriptive Statistics","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Analysis\n## Total distribution of each discrete variable\nWe can visualize the distribution of the discrete variables to look into the different possible values each can take and the distribution of these values. ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(20, 15))\n\n# Survived Plot\nplt.subplot(3, 2, 1)\ntrain_data['Survived'].value_counts().plot(kind='bar')\nplt.title('Survived')\n\n# Pclass Plot\nplt.subplot(3, 2, 2)\ntrain_data['Pclass'].value_counts().plot(kind='bar')\nplt.title('Pclass')\n\n# Sex Plot\nplt.subplot(3, 2, 3)\ntrain_data['Sex'].value_counts().plot(kind='bar')\nplt.title('Sex')\n\n# Siblings and Spouses Plot\nplt.subplot(3, 2, 4)\ntrain_data['SibSp'].value_counts().plot(kind='bar')\nplt.title('SibSp')\n\n# Parents and Children Plot\nplt.subplot(3, 2, 5)\ntrain_data['Parch'].value_counts().plot(kind='bar')\nplt.title('Parch')\n\n# Embarked Plot\nplt.subplot(3, 2, 6)\ntrain_data['Embarked'].value_counts().plot(kind='bar')\nplt.title('Embarked')\n\nplt.tight_layout() ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below we can see that the Ticket and Cabin column have a lot of unique values which we have to clean up later.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\n\n# Ticket Plot\nplt.subplot(1, 2, 1)\ntrain_data['Ticket'].value_counts().plot(kind='bar')\nplt.title('Ticket')\n\n# Cabin Plot\nplt.subplot(1, 2, 2)\ntrain_data['Cabin'].value_counts().plot(kind='bar')\nplt.title('Cabin')\n\nplt.tight_layout() ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Distribution for each continuous variable","metadata":{}},{"cell_type":"markdown","source":"We do the same for continuous variables to detect potential outliers. Age doesn't seem to have any outliers. Fare could have one at 500, however, a quick google search shows that the prices were as high as 4300$ meaning we have not an outlier here as well.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10, 10))\n\n# Age Boxplot\nplt.subplot(2, 2, 1)\ntrain_data.boxplot(column=['Age'])\nplt.title(\"Age\")\n\n# Age Histogram\nplt.subplot(2, 2, 2)\ntrain_data['Age'].hist()\nplt.title(\"Age\")\nplt.ylabel(\"(Absolute) Frequency\")\n\n# Fare Boxplot\nplt.subplot(2, 2, 3)\ntrain_data.boxplot(column=['Fare'])\nplt.title(\"Fare\")\nplt.subplot(2, 2, 4)\n\n# Fare Histogram\ntrain_data['Fare'].hist()\nplt.title(\"Fare\")\nplt.ylabel(\"(Absolute) Frequency\")\n\nplt.tight_layout() ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Calculate missing data for Train Data and Score Set\nCalculating the percentages of missing data for each column we see that we have to impute values for the Age and Cabin column. The Cabin column could even be dropped as it has a big percentage of its values missing.","metadata":{}},{"cell_type":"code","source":"# Missing Data Train Data\nmissing_data_train = pd.DataFrame({'total_missing': train_data.isnull().sum(), 'perc_missing': (train_data.isnull().sum()/891)*100})\nprint('Missing Data Train Data \\n',missing_data_train,'\\n')\n\n# Missing Data Score Set\nmissing_data_score = pd.DataFrame({'total_missing': score_set.isnull().sum(), 'perc_missing': (score_set.isnull().sum()/418)*100})\nprint('Missing Data Score Set \\n',missing_data_score)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Show distribution given Target variable\nIn correlation to the target variable, we can see for example that people that were in the first class and female had higher survival rates than others males or people travelling third class.","metadata":{}},{"cell_type":"code","source":"# Target Variable\nStatus = train_data['Survived']\n\n# Correlation Pclass\ntrain_data['Pclass'].value_counts()\npd.crosstab(Status, train_data['Pclass'], margins=True)\nprint(pd.crosstab(Status, train_data['Pclass'], margins=True, normalize='columns'),'\\n')\n\n# Correlation Sex\ntrain_data['Sex'].value_counts()\npd.crosstab(Status, train_data['Sex'], margins=True)\nprint(pd.crosstab(Status, train_data['Sex'], margins=True, normalize='columns'),'\\n')\n\n# Correlation Parents and Children\ntrain_data['Parch'].value_counts()\npd.crosstab(Status, train_data['Parch'], margins=True)\nprint(pd.crosstab(Status, train_data['Parch'], margins=True, normalize='columns'),'\\n')\n\n# Correlation Siblings and Spouses\ntrain_data['SibSp'].value_counts()\npd.crosstab(Status, train_data['SibSp'], margins=True)\nprint(pd.crosstab(Status, train_data['SibSp'], margins=True, normalize='columns'),'\\n')\n\n# Correlation Embarked\ntrain_data['Embarked'].value_counts()\npd.crosstab(Status, train_data['Embarked'], margins=True)\nprint(pd.crosstab(Status, train_data['Embarked'], margins=True, normalize='columns'),'\\n')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plot correlation of variables\nPclass and Fare have the highest correlation to the target variable. One important point is that it doesn't matter if and how strong variables correlate if you want to predict something! It only matters if you want to explain certain phenomena influenced by variables.","metadata":{}},{"cell_type":"code","source":"# Compute correlation matrix\ncorr = train_data.corr()\n\n# Drop PassengerId\ncorr_adjusted = train_data.drop(columns=['PassengerId']).corr()\n\n# Visualization of the correlation matrix with Seaborn library\nimport seaborn as sns\nheat = sns.heatmap(corr_adjusted, \n            xticklabels=corr_adjusted.columns.values,\n            yticklabels=corr_adjusted.columns.values)\n\n# Correlation Values\nprint(train_data.corrwith(train_data[\"Survived\"]).sort_values())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Preprocessing","metadata":{}},{"cell_type":"markdown","source":"## Feature Engineering","metadata":{}},{"cell_type":"markdown","source":"For the Train Data and Score Set, we map the Sex column to numbers and create the Family Size and Is Alone Feature. For Family Size we take Siblings and Spouses column (SibSp) and the Parents and Children Column (Parch) and add one to it to generate the whole Family Size on the Titanic. If Family Size is 1 it means the person is travelling alone and we set Is Alone to one indicating Is Alone is true.","metadata":{}},{"cell_type":"code","source":"#Feature Engineering\n\n#Train Data\n\n#Map Sex to numbers\ntrain_data['Sex'] = train_data['Sex'].map( {'female': 1, 'male': 0} ).astype(int)\n\n#Engineer FamiliySize feature\ntrain_data['FamilySize'] = train_data['SibSp'] + train_data['Parch'] + 1\n\n#Engineer IsAlone feature\ntrain_data['IsAlone'] = 0\ntrain_data.loc[train_data['FamilySize'] == 1, 'IsAlone'] = 1\n\n#Score Set\n\n#Map Sex to numbers\nscore_set['Sex'] = score_set['Sex'].map( {'female': 1, 'male': 0} ).astype(int)\n\n#Engineer FamiliySize feature\nscore_set['FamilySize'] = score_set['SibSp'] + score_set['Parch'] + 1\n\n#Engineer IsAlone feature\nscore_set['IsAlone'] = 0\nscore_set.loc[score_set['FamilySize'] == 1, 'IsAlone'] = 1","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Pipeline and Data Leakage\n### General Important Concepts\nBefore we start to explaining **Pipelines** and **Data Leakage** we should look at the overall principles of Data Science and Predictive Modelling. In Data Science Projects often labelled **Train Data** and unlabeled **Test Data** (which we named Score Set) is given. The goal is to learn patterns from the Train Data and apply these learned patterns and rules to the Score Set to predict as best as possible the correct labels. Most simply we take the whole Train Data, learn the patterns and predict the labels of the Score Set. This is however a very bad way to do it because we have no information on how good our model performs. Sure we can just take our model and predict on the Train Data to get an accuracy score, but just let me show you some pictures from Wikimedia explaining the problem with that:\n\n<img src=\"https://upload.wikimedia.org/wikipedia/commons/1/19/Overfitting.svg\" width=\"300\">\n\nChabacano, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, via Wikimedia Commons\n\n<img src=\"https://upload.wikimedia.org/wikipedia/commons/5/52/2d-epochs-overfitting.svg\" width=\"300\">\n\nMartinThoma, CC0, via Wikimedia Commons\n\nAs you can see the Performance / Predictive Power on your Train Data gets better and better the more we train and the more complex our model gets, but on the Score Set, there is a point where the performance goes down. This is called **Overfitting** and a very very important concept. It shows that we memorize the Train Data too much and don't generalize any more. To avoid this we introduce a **Test Set**. A Test Set is a standalone part of the Train Data we don't train on but test the model after the training phase to get an unbiased evaluation of our model. We now have a Train Set and Test Set. Usually, it is around 20 % of the Train Data, but it depends on the size of the data. If the performance on the Test Set gets down we stop training as we've found the sweet spot with the highest performance on unseen data. \n\nWhat if we want to tune the model in the training phase? Trying different parameters and techniques? If we evaluate all these possible configurations on the Test Set we no longer have an unbiased data set on which we can evaluate the final model with the best parameters. We simply create another standalone set of the Train Data, the **Validation Set**. It is usually around 20 % of the Train Data we have left after have it split into Train Set and Test Set. The logical workflow is as follows. Split the Train Data into Train Set, Validation Set and Test set. Train the model on our Train Set and try different parameters. For each model, the configuration takes the performance on the Validation Set. After training, take the best model configuration given the different performances on the Validation Set and evaluate the final model on the Test Set to get an unbiased opinion on how well our final model predicts unseen data.\n\n<img src=\"https://upload.wikimedia.org/wikipedia/commons/b/bb/ML_dataset_training_validation_test_sets.png\" width=\"600\">\n\nKm121220, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, via Wikimedia Commons\n\n### Data Leakage\nTo get evaluation values that are the closest to how our model will perform on unseen data we should take a look at **Data Leakage**. Data leakage describes if the information of the Test Set and Validation Set leak into our Train Set and model, meaning both sets are not 100 % unseen any longer. A typical example is if we calculate the mean of the whole Train Data and impute it where data is missing. The right way to do it would be to split the Train Data into Train Set, Validation Set and Test Set, calculate the mean of the only Train Set and impute it into the three sets where data is missing. In this way, there is no information flow from the Validation Set and Test Set into the Train Set. You should always think of the information you derive and compute for the Train Data as a one-way lane from the Train Set to the Validation Set and Test Set.\n### The Pipeline and Cross-Fold-Validation\nA **Pipeline** can be seen as a set of sequential steps transforming given data when calling the Pipeline. The transformations can be normalizations, imputations or custom mappings to other datatypes. Why are we using it? To prevent data leakage when using another method: **Cross-Fold-Validation**. Using Cross-Fold-Validation we part the Train Data in a Train Set and Test Set. The method then parts the Train Set into n-folds (like 5 or 10), trains on n-1 folds and validates on the last fold. It does that n-times for every possible train folds - validation fold configuration and takes the mean of all calculated evaluation metrics. Imagine we have three folds and want to impute missing data. To prevent Data Leakage we have to calculate the mean of a column from the first two folds and impute the solution in all three folds. We then train and evaluate. The next step is to do it again, but taking the second and third fold for calculation and training and impute and evaluate on the first and so on. This would mean a lot of code. With a pipeline, we can define the data preprocessing before, pass the pipeline to Cross-Fold-Validation where the preprocessing is run automatically for every single fold configuration. In short: We prevent Data Leakage!\n\nI could only find a picture of Cross-Fold-Validation in another language than English on Wikimedia. The first block 'conjunto de dados' stands here for the Train Set from option 'A' in the picture above. As you can see the Train Set gets parted into four-folds. The blue folds are the Train Set for the respective fold, the yellow folds the Validation Set for the respective fold.\n\n<img src=\"https://upload.wikimedia.org/wikipedia/commons/5/5c/K-Fold_Cross-Validation.png\" width=\"600\">\n\nLeomaurodesenv, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, via Wikimedia Commons","metadata":{}},{"cell_type":"markdown","source":"### Custom Methods\nAt first, we look at some custom methods for our Pipeline transforming the columns Fare, Age, Embarked, Cabin and Name. In short for the Fare and Age column we use binning to introduce non-linearity. The Embarked, Cabin and Name columns get mapped to numbers. Each class consists of a init, fit and transform method. While we can ignore the init method, the fit method extracts and saves information of the Train Set. This information is then used to transform the Train Set and Validation Set inside the transform method. ","metadata":{}},{"cell_type":"markdown","source":"#### Fare Feature\nFirst, we look at the Fare Feature. We part the feature beforehand into equal size buckets and use this information to encode the feature to introduce non-linearity. You should notice something here! The way I did it is a case of Data Leakage. We calculate the bucket boundaries on the whole Train Data and not just on the Train Set after the split. The right way to do it would be inside the fit method inside the FareFeatureTransfomer class below, but I didn't find a way to extract and save the boundary information (yet).","metadata":{}},{"cell_type":"code","source":"# Binning\npd.qcut(train_data['Fare'], 4).unique()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Transform Fare Feature\nclass FareFeatureTransformer(BaseEstimator, TransformerMixin):\n    \n    def __init__(self):\n        None\n        \n    def fit(self, x, y=None):\n        return self\n    \n    def transform(self, x):\n        \n        # Transform to DataFrame\n        if isinstance(x, np.ndarray):\n            x = pd.DataFrame(x, columns=['Fare'])\n        \n        # Map to Bucket Boundaries\n        x.loc[ x['Fare'] <= 7.91, 'Fare'] = 0\n        x.loc[(x['Fare'] > 7.91) & (x['Fare'] <= 14.454), 'Fare'] = 1\n        x.loc[(x['Fare'] > 14.454) & (x['Fare'] <= 31), 'Fare']   = 2\n        x.loc[ x['Fare'] > 31, 'Fare'] = 3    \n        \n        return x","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Age Feature\nThis is the same as above. The only difference is that the buckets are of equal width and not of equal size.","metadata":{}},{"cell_type":"code","source":"# Binning\npd.cut(train_data['Age'], 5).unique()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Transform Age Feature\nclass AgeFeatureTransformer(BaseEstimator, TransformerMixin):\n    \n    def __init__(self):\n        None\n    \n    def fit(self, x, y=None):\n        return self\n    \n    def transform(self, x):\n        \n        # Transform to DataFrame\n        if isinstance(x, np.ndarray):\n            x = pd.DataFrame(x, columns=['Age'])\n        \n        # Map to Bucket Boundaries\n        x.loc[ x['Age'] <= 16, 'Age'] = 0\n        x.loc[(x['Age'] > 16) & (x['Age'] <= 32), 'Age'] = 1\n        x.loc[(x['Age'] > 32) & (x['Age'] <= 48), 'Age'] = 2\n        x.loc[(x['Age'] > 48) & (x['Age'] <= 64), 'Age'] = 3\n        x.loc[ x['Age'] > 64, 'Age'] = 4  \n        \n        return x","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Embarked Feature\nThis class performs a simple mapping operation for each of the possible values of the Embarked Feature. We encode the missing values as well to be able to use the KNN imputation later on.","metadata":{}},{"cell_type":"code","source":"# Transform Embarked Feature\nclass EmbarkedFeatureTransformer(BaseEstimator, TransformerMixin):\n    \n    def __init__(self):\n        None\n    \n    def fit(self, x, y=None):\n        return self\n    \n    def transform(self, x):\n        \n        # Transform to DataFrame\n        if isinstance(x, np.ndarray):\n            x = pd.DataFrame(x, columns=['Embarked'])\n        \n        # Map to numbers\n        x['Embarked'] = x['Embarked'].replace(np.nan, 'NaN', regex=True)\n        x['Embarked'] = x['Embarked'].map( {'S': 0, 'Q': 1, 'C': 2, 'NaN': 3} ).astype(int)\n        \n        return x","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Cabin Feature\nInside the fit method, we extract the dummies of the Train Set. We then use these dummies to transform both the Train Set and Validation set.","metadata":{}},{"cell_type":"code","source":"# Transform Cabin Feature\nclass CabinFeatureTransformer(BaseEstimator, TransformerMixin):\n    \n    def __init__(self):\n        None\n    \n    def fit(self, x, y=None):\n        \n        # Impute missing values\n        x.Cabin.fillna('U', inplace=True)\n        x['Cabin'] = x['Cabin'].map(lambda c: c[0])\n        \n        # Get dummies\n        cabin_dummies = pd.get_dummies(x['Cabin'], prefix='Cabin')\n        self.cabin_columns = cabin_dummies.columns\n        \n        return self\n    \n    def transform(self, x):\n        \n        # Impute missing values\n        x.Cabin.fillna('U', inplace=True)\n        x['Cabin'] = x['Cabin'].map(lambda c: c[0])\n        \n        # Get dummies and reindex them with fitted ones\n        cabin_dummies = pd.get_dummies(x['Cabin'], prefix='Cabin')\n        cabin_dummies = cabin_dummies.reindex(columns = self.cabin_columns, fill_value=0)\n        \n        # Concat new columns\n        x = pd.concat([x, cabin_dummies], axis=1) \n        \n        # Drop original column\n        x.drop('Cabin', axis=1, inplace=True)\n             \n        return x","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Name Feature\nHere we do the same thing as above. The only difference is that we only create a dummy for a value if that value has a count higher or equal to ten.","metadata":{}},{"cell_type":"code","source":"# Transform Name Feature\nclass NameFeatureTransformer(BaseEstimator, TransformerMixin):\n    \n    def __init__(self):\n        None\n    \n    def fit(self, x, y=None):\n        \n        # Get Titles\n        x['Title'] = x['Name'].str.split(\", \", expand=True)[1].str.split(\".\", expand=True)[0]\n        \n        # Map to Misc if value count < 10\n        stat_min = 10\n        title_names = (x['Title'].value_counts() < stat_min)\n        x['Title'] = x['Title'].apply(lambda x: 'Misc' if title_names.loc[x] == True else x)\n        \n        # Get dummies\n        name_dummies = pd.get_dummies(x['Title'], prefix='Title')\n        self.name_columns = name_dummies.columns\n\n        return self\n    \n    def transform(self, x):\n        \n        # Get Titles\n        x['Title'] = x['Name'].str.split(\", \", expand=True)[1].str.split(\".\", expand=True)[0]\n        \n        # Map to Misc if value count < 2\n        stat_min = 2\n        title_names = (x['Title'].value_counts() < stat_min)\n        x['Title'] = x['Title'].apply(lambda x: 'Misc' if title_names.loc[x] == True else x)\n        \n        # Get dummies and reindex them with fitted ones\n        name_dummies = pd.get_dummies(x['Title'], prefix='Title')\n        name_dummies = name_dummies.reindex(columns = self.name_columns, fill_value=0)\n        \n        # Concat new columns\n        x = pd.concat([x, name_dummies], axis=1)\n        \n        # Drop original columns\n        x.drop('Title', axis=1, inplace=True)\n        x.drop('Name', axis=1, inplace=True)\n\n        return x","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Column Transformer \nWe can use a transformer to bundle all preprocessing steps. We define how each column gets preprocessed and add additional steps outside our classes.","metadata":{}},{"cell_type":"code","source":"# Age Feature\nage = ['Age']\nage_transformer = Pipeline(steps=[\n    ('imputer', KNNImputer(n_neighbors = 100)), # Impute missing values with KNN\n    ('binning', AgeFeatureTransformer()) # Use the Age Feature Transformer class\n])\n\n# Fare Feature\nfare = ['Fare']\nfare_transformer = Pipeline(steps=[\n    ('imputer', KNNImputer(n_neighbors = 100)), # Impute missing values with KNN\n    ('binning', FareFeatureTransformer()) # Use the Fare Feature Transformer class\n])\n\n# Discrete Features\ndiscrete = ['Pclass', 'Sex', 'Parch', 'SibSp', 'FamilySize', 'IsAlone']\ndiscrete_transformer = Pipeline(steps=[\n    ('imputer', KNNImputer(n_neighbors = 100)) # Impute missing values with KNN\n])\n\n# Embarked Feature\nembarked = ['Embarked']\nembarked_transformer = Pipeline(steps=[\n    ('map', EmbarkedFeatureTransformer()), # Use the Embarked Feature Transformer class\n    ('imputer', KNNImputer(n_neighbors = 100, missing_values = 3)) # Impute missing values with KNN\n])\n\n# Cabin Feature\ncabin = ['Cabin']\n\n# Name Feature\nname = ['Name']\n\n# Transformer\ntransformer = ColumnTransformer(\n    transformers = [\n        ('name_data', NameFeatureTransformer(), name), # Name Feature\n        ('embarked_data', embarked_transformer, embarked), # Embarked Feature\n        ('age_data', age_transformer, age), # Age Feature\n        ('fare_data', fare_transformer, fare), # Fare Feature\n        ('cabin_data', CabinFeatureTransformer(), cabin), # Cabin Feature\n        ('discrete_data', discrete_transformer, discrete) # Discrete Features\n    ]\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Final Pipeline\nFinally, we can assemble the pipeline defining the transformer and the used model.","metadata":{}},{"cell_type":"code","source":"# Random Forest Pipeline\nforest_pipe = Pipeline(steps=[\n    ('transformer', transformer), # Transfomer\n    ('rf', RandomForestClassifier(random_state=0)) # Random Forest\n])\n\n# Gradient Boosting Classifier\nboost_pipe = Pipeline(steps=[\n    ('transformer', transformer), # Transfomer\n    ('gb', GradientBoostingClassifier(random_state=0)) # Gradient Boosting\n])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training & Evaluation\n## Split\nNow it is time to perform the split on the Train Data into the Train Set and Test Set. Remember that the Validation Set gets created automatically when using Cross-Fold-Validation later.","metadata":{}},{"cell_type":"code","source":"# Used Features\nfeatures = [\"Pclass\", \"Sex\", \"Age\", \"Fare\", \"Embarked\", \"Name\", \"Parch\", \"SibSp\", \"Cabin\", \"FamilySize\", \"IsAlone\"]\n\n# Score Set\nX_score = score_set[features]\n\n# Train Data\nX = train_data[features]\ny = train_data['Survived']\n\n# Split of Train Data into Train Set and Test Set\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Simple Training\nThen we can perform a 'simple' training. Meaning running the pipeline only once without tuning any hyperparameters.","metadata":{}},{"cell_type":"code","source":"forest_pipe.fit(X_train,y_train) # Fit to Train Set\ny_pred = forest_pipe.predict(X_test) # Predict on Test Set\nprint(accuracy_score(y_test, y_pred)) # Get Accuracy","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"boost_pipe.fit(X_train,y_train) # Fit to Train Set\ny_pred = boost_pipe.predict(X_test) # Predict on Test Set\nprint(accuracy_score(y_test, y_pred)) # Get Accuracy","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Many other notebooks train on the Train Data and evaluate the accuracy of it as well. Like I did here:","metadata":{}},{"cell_type":"code","source":"forest_pipe.fit(X_train,y_train) # Fit to Train Set\ny_pred = forest_pipe.predict(X_train) # Predict on Train Set\nprint(accuracy_score(y_train, y_pred)) # Get Accuracy","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You see how much better my score is and how simple it is to 'achieve' a nice score? However, **Data Science and predictive modelling are not about getting a good score on seen data!** It is about getting a good score on unseen data. Imagine working on for a company to predict churn rates and claiming the model can predict churners with an accuracy of around 91% while in reality, the model can only predict churners with an accuracy of 78%. Remember the graph from above. The 91% reflect the red training line, the 78% reflect the blue scoring line. While it is nice to present a high number, in the end when scoring the new unseen data you will achieve the 78%. There is no meaning, need or sense in evaluating your model on the data you trained it on. This could mean a loss in business value and is just not good work. \n\nThis is the same for data science competitions. You can predict with a 91% accuracy on your training data, but when you predict on your Scoring Set and turn it in you get the 78%.\n\nAlways remember to increase the score on the Test Set, the truly unseen data and not on the data you trained on. One of your main goals is to get a number from the evaluation that reflects the score on new unseen data as close as possible.","metadata":{}},{"cell_type":"markdown","source":"## Hyperparametertuning (Cross-Fold-Validation)\nNow we can tune some of the hyperparameters of each model performing Cross-Fold-Validation. It runs n-folds for each possible configuration of the defined hyperparameters processing the Pipeline every single time.","metadata":{}},{"cell_type":"code","source":"# Define the parameters and values\nparam_grid = {\n    'rf__bootstrap': [True, False],\n    'rf__max_depth': [0,5,10],\n    'rf__criterion': ['gini','entropy'],\n    'rf__min_samples_leaf': [0,2,5],\n    'rf__min_samples_split': [0,2,5],\n    'rf__n_estimators': [100,500,1000]\n}\n\n# Define the grid search model\nforest_model = GridSearchCV(estimator = forest_pipe, param_grid = param_grid, scoring='accuracy', cv = 5, n_jobs = -1, verbose = 8)\n\n# Fit to Train Set\nforest_model.fit(X_train, y_train)\n\n# Get best values for params\nprint(forest_model.best_params_)\n\n# Predict on Test Set\ny_pred = forest_model.predict(X_test)\n\n# Get Accuracy\nprint(accuracy_score(y_test, y_pred))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define the parameters and values\nparam_grid = {\n    'gb__learning_rate': [0.001, 0.01, 0.1, 1],\n    'gb__n_estimators': [100,500,1000],\n    'gb__min_samples_leaf': [0,1,2],\n    'gb__min_samples_split': [0,2,5],\n    'gb__max_depth': [0,3,5],\n    'gb__warm_start': [True, False]\n}\n\n# Define the grid search model\nboost_model = GridSearchCV(estimator = boost_pipe, param_grid = param_grid, scoring='accuracy', cv = 5, n_jobs = -1, verbose = 8)\n\n# Fit to Train Set\nboost_model.fit(X_train, y_train)\n\n# Get best values for params\nprint(boost_model.best_params_)\n\n# Predict on Test Set\ny_pred = boost_model.predict(X_test)\n\n# Get Accuracy\nprint(accuracy_score(y_test, y_pred))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Final Prediction\nAs we can see the Random Forest Model provided the most accurate outputs with an accuracy score of 0.7988 on the Test Set. We take the model and predict on the Score Set and see that our evaluated accuracy score from the Test Set (79.88%) and the accuracy score on the Score Set (77.27%) are very close. This is a good thing. We can communicate clearly how good our model performs with unseen data.","metadata":{}},{"cell_type":"code","source":"# Predict on Score Set\npredictions = forest_model.predict(X_score)\n\n# Generate Submission\noutput = pd.DataFrame({'PassengerId':score_set.PassengerId, 'Survived':predictions})\noutput.to_csv('my_submission.csv', index=False)\nprint(\"Submission successfully saved\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}