{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\nThe sinking of Titanic is one of the most notoririous in the history. In 1912, during her voyege, the Titanic sank after colliding with an icebarg, killing 1502 out of 2224 passengers and crew.\n\n<font color = 'blue'>\nContent:\n\n1. [Load and Check Data](#1)   \n1. [Variable Description](#2) \n    * [Univariate Variable Analysis](#3)\n        * [Categorical Variable](#4)\n        * [Numerical Variable](#5)\n1. [Basic Data Analysis](#6)\n1. [Outlier Detection](#7)\n1. [Missing Value](#8)\n    * [Find Missing Value](#9)\n    * [Fill Missing Value](#10)\n1. [Visualization](#11)\n    * [Correlation Between SibSp -- Parch -- Age -- Fare -- Survived](#12)\n    * [SibSp -- Survived](#13)\n    * [Parch -- Survived](#14)\n    * [Pclass -- Survived](#15)\n    * [Age -- Survived](#16)\n    * [Pclass -- Survived -- Age](#17)\n    * [Embarked -- Sex -- Pclass -- Survived](#18)\n    * [Embarked -- Sex -- Fare -- Survived](#19)\n    * [Fill Missing: Age Feature](#20)\n1. [Feature Engineering](#21)\n    * [Name -- Title](#22)\n    * [Family Size](#23)\n    * [Embarked](#24)\n    * [Ticket](#25)\n    * [Pclass](#26)\n    * [Sex](#27)\n    * [Drop Passenger ID and Cabin](#28)\n1. [Modeling](#29)\n    * [Train - Test Split](#30)\n    * [Simple Logistic Regression](#31)\n    * [Hyperparameter Tuning -- Grid Search -- Cross Validation](#32)\n    * [Ensemble Modeling](#33)\n    * [Prediction and Submission](#34)","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt\nplt.style.use('seaborn-whitegrid')\n\nimport seaborn as sns\n\nfrom collections import Counter\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:33.069766Z","iopub.execute_input":"2022-08-09T22:41:33.070221Z","iopub.status.idle":"2022-08-09T22:41:33.740156Z","shell.execute_reply.started":"2022-08-09T22:41:33.070128Z","shell.execute_reply":"2022-08-09T22:41:33.738746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = '1'></a><br>\n# Load and Check Data","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(\"/kaggle/input/titanic/train.csv\")\ntest_df = pd.read_csv(\"/kaggle/input/titanic/test.csv\")\ntest_Passenger_Id = test_df['PassengerId']","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:33.741833Z","iopub.execute_input":"2022-08-09T22:41:33.742197Z","iopub.status.idle":"2022-08-09T22:41:33.775266Z","shell.execute_reply.started":"2022-08-09T22:41:33.742166Z","shell.execute_reply":"2022-08-09T22:41:33.773853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.columns","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:33.777163Z","iopub.execute_input":"2022-08-09T22:41:33.777608Z","iopub.status.idle":"2022-08-09T22:41:33.788274Z","shell.execute_reply.started":"2022-08-09T22:41:33.777568Z","shell.execute_reply":"2022-08-09T22:41:33.787350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:33.790827Z","iopub.execute_input":"2022-08-09T22:41:33.791748Z","iopub.status.idle":"2022-08-09T22:41:33.811889Z","shell.execute_reply.started":"2022-08-09T22:41:33.791703Z","shell.execute_reply":"2022-08-09T22:41:33.811098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:33.813142Z","iopub.execute_input":"2022-08-09T22:41:33.813622Z","iopub.status.idle":"2022-08-09T22:41:33.851699Z","shell.execute_reply.started":"2022-08-09T22:41:33.813592Z","shell.execute_reply":"2022-08-09T22:41:33.850866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = '2'></a><br>\n# Variable Description","metadata":{}},{"cell_type":"markdown","source":"1. PassengerId: Unique id number to each passenger\n1. Survived: Passanger survived or died \n1. Pclass: Passenger class\n1. Name: Name \n1. Sex: Gender of passenger \n1. Age: Age of passenger \n1. SibSp: Number of siplings/spouses \n1. Parch: Number of parent/childiren \n1. Ticket: Ticket number \n1. Fare: Amount of money spent on ticket  \n1. Cabin: Cabin category\n1. Embarked: Port passenger embarked (C = Cherbourg, Q = Queenstown, S = Southampton)","metadata":{}},{"cell_type":"code","source":"train_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:33.852912Z","iopub.execute_input":"2022-08-09T22:41:33.853431Z","iopub.status.idle":"2022-08-09T22:41:33.869420Z","shell.execute_reply.started":"2022-08-09T22:41:33.853400Z","shell.execute_reply":"2022-08-09T22:41:33.868386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* float64(2): Age, Fare\n* int64(5): PassengerId, Survived, Pclass, SibSp, Parch \n* object(5): Name, Sex, Ticket, Cabin, Embarked ","metadata":{}},{"cell_type":"markdown","source":"<a id = '3'></a><b>    \n# Univariate Variable Analysis\n* Categorical Variable: Survived, Pclass, Sex, Cabin, Name, Ticket, SibSp, Parch\n* Numerical Variable: Age, Fare, PassengerId","metadata":{}},{"cell_type":"markdown","source":"<a id = '4'></a><b>\n## Categorical Variable","metadata":{}},{"cell_type":"code","source":"def bar_plot(variable):\n    # get feature\n    var = train_df[variable]\n    \n    # count number of categorical variable \n    varValue = var.value_counts()\n    \n    # visualize\n    plt.figure(figsize = (9, 3))\n    plt.bar(varValue.index, varValue)\n    plt.xticks(varValue.index, varValue.index.values)\n    plt.ylabel(\"Frequency\")\n    plt.title(variable)\n    plt.show()\n    print(\"{}: \\n {}\".format(variable, varValue))","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:33.871043Z","iopub.execute_input":"2022-08-09T22:41:33.871622Z","iopub.status.idle":"2022-08-09T22:41:33.877688Z","shell.execute_reply.started":"2022-08-09T22:41:33.871589Z","shell.execute_reply":"2022-08-09T22:41:33.876681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"category1 = [\"Survived\", \"Pclass\", \"Sex\", \"Embarked\", \"SibSp\", \"Parch\"]\nfor i in category1:\n    bar_plot(i)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:33.879408Z","iopub.execute_input":"2022-08-09T22:41:33.880233Z","iopub.status.idle":"2022-08-09T22:41:34.918692Z","shell.execute_reply.started":"2022-08-09T22:41:33.880191Z","shell.execute_reply":"2022-08-09T22:41:34.917375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"5\"></a><b>\n## Numerical Variable","metadata":{}},{"cell_type":"code","source":"def plot_hist(variable):\n    plt.figure(figsize = (9, 3))\n    plt.hist(train_df[variable], bins = 50)\n    plt.xlabel(variable)\n    plt.ylabel(\"Frequency\")\n    plt.title(\"{} distribution with hist\".format(variable))\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:34.923698Z","iopub.execute_input":"2022-08-09T22:41:34.924976Z","iopub.status.idle":"2022-08-09T22:41:34.933104Z","shell.execute_reply.started":"2022-08-09T22:41:34.924908Z","shell.execute_reply":"2022-08-09T22:41:34.931721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"numerical1 = [\"Fare\", \"Age\", \"PassengerId\"]\n\nfor n in numerical1:\n    plot_hist(n)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:34.934499Z","iopub.execute_input":"2022-08-09T22:41:34.935399Z","iopub.status.idle":"2022-08-09T22:41:35.949601Z","shell.execute_reply.started":"2022-08-09T22:41:34.935356Z","shell.execute_reply":"2022-08-09T22:41:35.948625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"6\"></a><b>\n# Basic Data Analysis \n\n* Pclass - Survived\n* Sex - Survived\n* SibSp - Survived\n* Parch - Survived","metadata":{}},{"cell_type":"code","source":"# Pclass - Survived\ntrain_df[[\"Pclass\", \"Survived\"]].groupby([\"Pclass\"], as_index = False).mean().sort_values(by = \"Survived\", ascending = False)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:35.951178Z","iopub.execute_input":"2022-08-09T22:41:35.951704Z","iopub.status.idle":"2022-08-09T22:41:35.969542Z","shell.execute_reply.started":"2022-08-09T22:41:35.951671Z","shell.execute_reply":"2022-08-09T22:41:35.968198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Sex - Survived\ntrain_df[[\"Sex\", \"Survived\"]].groupby([\"Sex\"], as_index = False).mean().sort_values(by = \"Survived\", ascending = False)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:35.972167Z","iopub.execute_input":"2022-08-09T22:41:35.972774Z","iopub.status.idle":"2022-08-09T22:41:35.990203Z","shell.execute_reply.started":"2022-08-09T22:41:35.972739Z","shell.execute_reply":"2022-08-09T22:41:35.989105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# SibSp - Survived\ntrain_df[[\"SibSp\", \"Survived\"]].groupby([\"SibSp\"], as_index = False).mean().sort_values(by = \"Survived\", ascending = False)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:35.991849Z","iopub.execute_input":"2022-08-09T22:41:35.992484Z","iopub.status.idle":"2022-08-09T22:41:36.008357Z","shell.execute_reply.started":"2022-08-09T22:41:35.992450Z","shell.execute_reply":"2022-08-09T22:41:36.007389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Parch - Survived\ntrain_df[[\"Parch\", \"Survived\"]].groupby([\"Parch\"], as_index = False).mean().sort_values(by = \"Survived\", ascending = False)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.009676Z","iopub.execute_input":"2022-08-09T22:41:36.010069Z","iopub.status.idle":"2022-08-09T22:41:36.024669Z","shell.execute_reply.started":"2022-08-09T22:41:36.010038Z","shell.execute_reply":"2022-08-09T22:41:36.023666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Parch - SibSp - Survived\ntrain_df[[\"Parch\", \"SibSp\", \"Survived\"]].groupby([\"Parch\", \"SibSp\"], as_index = False).mean().sort_values(by = \"Survived\", ascending = False)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.026306Z","iopub.execute_input":"2022-08-09T22:41:36.026986Z","iopub.status.idle":"2022-08-09T22:41:36.054239Z","shell.execute_reply.started":"2022-08-09T22:41:36.026944Z","shell.execute_reply":"2022-08-09T22:41:36.052830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"7\"></a><b>\n# Outlier Detection","metadata":{}},{"cell_type":"code","source":"def detect_outliers(df,features):\n    outlier_indices = []\n    \n    for c in features:\n        # 1st quartile\n        Q1 = np.percentile(df[c],25)\n        \n        # 3rd quartile\n        Q3 = np.percentile(df[c],75)\n        \n        # IQR\n        IQR = Q3 - Q1\n        \n        # Outlier step\n        outlier_step = IQR * 1.5\n        \n        # detect outlier and their indeces\n        outlier_list_col = df[(df[c] < Q1 - outlier_step) | (df[c] > Q3 + outlier_step)].index\n        \n        # store indeces\n        outlier_indices.extend(outlier_list_col)\n    \n    outlier_indices = Counter(outlier_indices)\n    multiple_outliers = list(i for i, v in outlier_indices.items() if v > 2)\n    \n    return multiple_outliers","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.055753Z","iopub.execute_input":"2022-08-09T22:41:36.056589Z","iopub.status.idle":"2022-08-09T22:41:36.066914Z","shell.execute_reply.started":"2022-08-09T22:41:36.056544Z","shell.execute_reply":"2022-08-09T22:41:36.065683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.loc[detect_outliers(train_df,[\"Age\",\"SibSp\",\"Parch\",\"Fare\"])]","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.068495Z","iopub.execute_input":"2022-08-09T22:41:36.069146Z","iopub.status.idle":"2022-08-09T22:41:36.107759Z","shell.execute_reply.started":"2022-08-09T22:41:36.068979Z","shell.execute_reply":"2022-08-09T22:41:36.106462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# drop outliers\ntrain_df = train_df.drop(detect_outliers(train_df,[\"Age\",\"SibSp\",\"Parch\",\"Fare\"]),axis = 0).reset_index(drop = True)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.109389Z","iopub.execute_input":"2022-08-09T22:41:36.110452Z","iopub.status.idle":"2022-08-09T22:41:36.126263Z","shell.execute_reply.started":"2022-08-09T22:41:36.110398Z","shell.execute_reply":"2022-08-09T22:41:36.125083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.127833Z","iopub.execute_input":"2022-08-09T22:41:36.128267Z","iopub.status.idle":"2022-08-09T22:41:36.152799Z","shell.execute_reply.started":"2022-08-09T22:41:36.128227Z","shell.execute_reply":"2022-08-09T22:41:36.151665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"8\"></a><b>\n# Missing Value\n* Find Missing Value\n* Fill Missing Value","metadata":{}},{"cell_type":"code","source":"train_df_len = len(train_df)\ntrain_df = pd.concat([train_df,test_df],axis = 0).reset_index(drop = True)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.154447Z","iopub.execute_input":"2022-08-09T22:41:36.155438Z","iopub.status.idle":"2022-08-09T22:41:36.168192Z","shell.execute_reply.started":"2022-08-09T22:41:36.155393Z","shell.execute_reply":"2022-08-09T22:41:36.166772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.170180Z","iopub.execute_input":"2022-08-09T22:41:36.170964Z","iopub.status.idle":"2022-08-09T22:41:36.196599Z","shell.execute_reply.started":"2022-08-09T22:41:36.170904Z","shell.execute_reply":"2022-08-09T22:41:36.195222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"9\"></a><b>\n## Find Missing Value","metadata":{}},{"cell_type":"code","source":"train_df.columns[train_df.isnull().any()]","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.198729Z","iopub.execute_input":"2022-08-09T22:41:36.199622Z","iopub.status.idle":"2022-08-09T22:41:36.211390Z","shell.execute_reply.started":"2022-08-09T22:41:36.199576Z","shell.execute_reply":"2022-08-09T22:41:36.210133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.217781Z","iopub.execute_input":"2022-08-09T22:41:36.218711Z","iopub.status.idle":"2022-08-09T22:41:36.229764Z","shell.execute_reply.started":"2022-08-09T22:41:36.218665Z","shell.execute_reply":"2022-08-09T22:41:36.228496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"10\"></a><b>\n## Fill Missing Value\n* Embarked has 2 missing value\n* Fare has only 1    ","metadata":{}},{"cell_type":"code","source":"train_df[train_df[\"Embarked\"].isnull()]","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.231662Z","iopub.execute_input":"2022-08-09T22:41:36.233046Z","iopub.status.idle":"2022-08-09T22:41:36.258269Z","shell.execute_reply.started":"2022-08-09T22:41:36.233002Z","shell.execute_reply":"2022-08-09T22:41:36.256979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.boxplot(column = \"Pclass\", by = \"Embarked\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.259669Z","iopub.execute_input":"2022-08-09T22:41:36.260836Z","iopub.status.idle":"2022-08-09T22:41:36.498211Z","shell.execute_reply.started":"2022-08-09T22:41:36.260793Z","shell.execute_reply":"2022-08-09T22:41:36.495518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"Embarked\"] = train_df[\"Embarked\"].fillna(\"C\") ","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.499867Z","iopub.execute_input":"2022-08-09T22:41:36.500654Z","iopub.status.idle":"2022-08-09T22:41:36.508382Z","shell.execute_reply.started":"2022-08-09T22:41:36.500608Z","shell.execute_reply":"2022-08-09T22:41:36.507061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[train_df[\"Embarked\"].isnull()]","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.510033Z","iopub.execute_input":"2022-08-09T22:41:36.511106Z","iopub.status.idle":"2022-08-09T22:41:36.529136Z","shell.execute_reply.started":"2022-08-09T22:41:36.511062Z","shell.execute_reply":"2022-08-09T22:41:36.527894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[train_df[\"Fare\"].isnull()]","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.530847Z","iopub.execute_input":"2022-08-09T22:41:36.531282Z","iopub.status.idle":"2022-08-09T22:41:36.554314Z","shell.execute_reply.started":"2022-08-09T22:41:36.531242Z","shell.execute_reply":"2022-08-09T22:41:36.553066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"Fare\"] = train_df[\"Fare\"].fillna(np.mean(train_df[train_df[\"Sex\"] == \"male\"][\"Fare\"]))","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.556249Z","iopub.execute_input":"2022-08-09T22:41:36.557517Z","iopub.status.idle":"2022-08-09T22:41:36.566209Z","shell.execute_reply.started":"2022-08-09T22:41:36.557473Z","shell.execute_reply":"2022-08-09T22:41:36.564971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[train_df[\"Fare\"].isnull()]","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.568156Z","iopub.execute_input":"2022-08-09T22:41:36.569104Z","iopub.status.idle":"2022-08-09T22:41:36.584640Z","shell.execute_reply.started":"2022-08-09T22:41:36.569058Z","shell.execute_reply":"2022-08-09T22:41:36.583516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"11\"></a><b>\n# Visualization ","metadata":{}},{"cell_type":"markdown","source":"<a id = \"12\"></a><b>\n## Correlation Between SibSp -- Parch -- Age -- Fare -- Survived","metadata":{}},{"cell_type":"code","source":"list1 = [\"SibSp\", \"Parch\", \"Age\", \"Fare\", \"Survived\"]\nsns.heatmap(train_df[list1].corr(), annot = True, fmt = \".2f\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.586442Z","iopub.execute_input":"2022-08-09T22:41:36.587194Z","iopub.status.idle":"2022-08-09T22:41:36.911057Z","shell.execute_reply.started":"2022-08-09T22:41:36.587153Z","shell.execute_reply":"2022-08-09T22:41:36.910010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"13\"></a><b>\n## SibSp -- Survived","metadata":{}},{"cell_type":"code","source":"g = sns.factorplot(x = \"SibSp\", y = \"Survived\", data = train_df, kind = \"bar\", size = 6)\ng.set_ylabels(\"Survived Probability\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:36.913198Z","iopub.execute_input":"2022-08-09T22:41:36.913892Z","iopub.status.idle":"2022-08-09T22:41:37.387092Z","shell.execute_reply.started":"2022-08-09T22:41:36.913847Z","shell.execute_reply":"2022-08-09T22:41:37.385903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Having a lot of SibSp have less change Survived\n* If SibSp == 0 or 1 or 2, passenger has more change to survived\n* We can consider a new feature describing these categories","metadata":{}},{"cell_type":"markdown","source":"<a id = \"14\"></a><b>\n## Parch -- Survived","metadata":{}},{"cell_type":"code","source":"g = sns.factorplot(x = \"Parch\", y = \"Survived\", data = train_df, kind = \"bar\", size = 6)\ng.set_ylabels(\"Survived Probability\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:37.388728Z","iopub.execute_input":"2022-08-09T22:41:37.389205Z","iopub.status.idle":"2022-08-09T22:41:37.835315Z","shell.execute_reply.started":"2022-08-09T22:41:37.389164Z","shell.execute_reply":"2022-08-09T22:41:37.834382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* SibSp and Parch can be used for new feature extraction th = 3\n* Small familes have more chance to Survived\n* There is a std in Survived of passenger with Parch = 3","metadata":{}},{"cell_type":"markdown","source":"<a id = \"15\"></a><b>\n# Pclass -- Survived","metadata":{}},{"cell_type":"code","source":"g = sns.factorplot(x = \"Pclass\", y = \"Survived\", data = train_df, kind = \"bar\", size = 6)\ng.set_ylabels(\"Survived Probability\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:37.836427Z","iopub.execute_input":"2022-08-09T22:41:37.837184Z","iopub.status.idle":"2022-08-09T22:41:38.300256Z","shell.execute_reply.started":"2022-08-09T22:41:37.837151Z","shell.execute_reply":"2022-08-09T22:41:38.299155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"16\"></a><b>\n## Age -- Survived","metadata":{}},{"cell_type":"code","source":"g = sns.FacetGrid(train_df, col = \"Survived\")\ng.map(sns.distplot, \"Age\", bins = 25)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:38.301747Z","iopub.execute_input":"2022-08-09T22:41:38.302435Z","iopub.status.idle":"2022-08-09T22:41:38.795003Z","shell.execute_reply.started":"2022-08-09T22:41:38.302402Z","shell.execute_reply":"2022-08-09T22:41:38.793722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* age <= 10 has a high survive rate \n* oldest passengers (80) survived\n* large number of 20 years old didn't survive\n* most passenger are 15 - 35 age range\n* use age feature in training \n* use age distribution for missinge value of age","metadata":{}},{"cell_type":"markdown","source":"<a id = \"17\"></a><b>\n## Pclass -- Survived -- Age","metadata":{}},{"cell_type":"code","source":"g = sns.FacetGrid(train_df, col = \"Survived\", row = \"Pclass\", size = 2)\ng.map(plt.hist, \"Age\", bins = 25)\ng.add_legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:38.796436Z","iopub.execute_input":"2022-08-09T22:41:38.796854Z","iopub.status.idle":"2022-08-09T22:41:40.142076Z","shell.execute_reply.started":"2022-08-09T22:41:38.796820Z","shell.execute_reply":"2022-08-09T22:41:40.141008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Pclass is important feature for model training","metadata":{}},{"cell_type":"markdown","source":"<a id = '18'></a><b>\n## Embarked -- Sex -- Pclass -- Survived","metadata":{}},{"cell_type":"code","source":"g = sns.FacetGrid(train_df, row = \"Embarked\", size = 2)\ng.map(sns.pointplot, \"Pclass\", \"Survived\", \"Sex\")\ng.add_legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:40.143985Z","iopub.execute_input":"2022-08-09T22:41:40.144427Z","iopub.status.idle":"2022-08-09T22:41:41.214405Z","shell.execute_reply.started":"2022-08-09T22:41:40.144386Z","shell.execute_reply":"2022-08-09T22:41:41.213320Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Famele passengers have much better survive rate than male\n* male have more bettersurvive rate in Pclass 3 in C ","metadata":{}},{"cell_type":"markdown","source":"<a id = '19'></a><b>\n## Embarked -- Sex -- Fare -- Survived","metadata":{}},{"cell_type":"code","source":"g = sns.FacetGrid(train_df, row = \"Embarked\", col = \"Survived\", size = 2.5)\ng.map(sns.barplot, \"Sex\", \"Fare\")\ng.add_legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:41.215695Z","iopub.execute_input":"2022-08-09T22:41:41.216056Z","iopub.status.idle":"2022-08-09T22:41:42.454054Z","shell.execute_reply.started":"2022-08-09T22:41:41.216025Z","shell.execute_reply":"2022-08-09T22:41:42.452981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Passenger who pay higher fare have better survival. Fare can be used as categorical for training. ","metadata":{}},{"cell_type":"markdown","source":"<a id = \"20\"></a><b>\n## Fill Missing: Age Feature","metadata":{}},{"cell_type":"code","source":"train_df[train_df[\"Age\"].isnull()]","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:42.455453Z","iopub.execute_input":"2022-08-09T22:41:42.455779Z","iopub.status.idle":"2022-08-09T22:41:42.483034Z","shell.execute_reply.started":"2022-08-09T22:41:42.455748Z","shell.execute_reply":"2022-08-09T22:41:42.482159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.factorplot(x = \"Sex\", y = \"Age\", data = train_df, kind = \"box\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:42.490320Z","iopub.execute_input":"2022-08-09T22:41:42.490886Z","iopub.status.idle":"2022-08-09T22:41:42.735561Z","shell.execute_reply.started":"2022-08-09T22:41:42.490844Z","shell.execute_reply":"2022-08-09T22:41:42.734249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Sex is not informative for age prediction, age distribution seems to be same","metadata":{}},{"cell_type":"code","source":"sns.factorplot(x = \"Sex\", y = \"Age\", hue = \"Pclass\", data = train_df, kind = \"box\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:42.736822Z","iopub.execute_input":"2022-08-09T22:41:42.737160Z","iopub.status.idle":"2022-08-09T22:41:43.269083Z","shell.execute_reply.started":"2022-08-09T22:41:42.737130Z","shell.execute_reply":"2022-08-09T22:41:43.268048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* 1st class passenger are older than 2nd, 2nd is older than 3rd class.","metadata":{}},{"cell_type":"code","source":"sns.factorplot(x = \"Parch\", y = \"Age\", data = train_df, kind = \"box\")\nsns.factorplot(x = \"SibSp\", y = \"Age\", data = train_df, kind = \"box\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:43.270254Z","iopub.execute_input":"2022-08-09T22:41:43.270546Z","iopub.status.idle":"2022-08-09T22:41:43.908909Z","shell.execute_reply.started":"2022-08-09T22:41:43.270518Z","shell.execute_reply":"2022-08-09T22:41:43.907759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"Sex\"] = [1 if i == \"male\" else 0 for i in train_df[\"Sex\"]]","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:43.910176Z","iopub.execute_input":"2022-08-09T22:41:43.910467Z","iopub.status.idle":"2022-08-09T22:41:43.916689Z","shell.execute_reply.started":"2022-08-09T22:41:43.910440Z","shell.execute_reply":"2022-08-09T22:41:43.915635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.heatmap(train_df[[\"Age\", \"Sex\", \"SibSp\", \"Parch\", \"Pclass\"]].corr(), annot = True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:43.918323Z","iopub.execute_input":"2022-08-09T22:41:43.918644Z","iopub.status.idle":"2022-08-09T22:41:44.209045Z","shell.execute_reply.started":"2022-08-09T22:41:43.918616Z","shell.execute_reply":"2022-08-09T22:41:44.207974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Age is not correlated with sex, but it is correlated with parch, sibsp, pclass","metadata":{}},{"cell_type":"code","source":"index_nan_age = list(train_df[\"Age\"][train_df[\"Age\"].isnull()].index)\n\nfor i in index_nan_age:\n    age_pred = train_df[\"Age\"][((train_df[\"SibSp\"] == train_df.iloc[i][\"SibSp\"]) &(train_df[\"Parch\"] == train_df.iloc[i][\"Parch\"])& (train_df[\"Pclass\"] == train_df.iloc[i][\"Pclass\"]))].median()\n    age_med = train_df[\"Age\"].median()\n    \n    if not np.isnan(age_pred):\n        train_df[\"Age\"].iloc[i] = age_pred\n    else:\n        train_df[\"Age\"].iloc[i] = age_med","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:44.211889Z","iopub.execute_input":"2022-08-09T22:41:44.213236Z","iopub.status.idle":"2022-08-09T22:41:44.758741Z","shell.execute_reply.started":"2022-08-09T22:41:44.213191Z","shell.execute_reply":"2022-08-09T22:41:44.757493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[train_df[\"Age\"].isnull()]","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-08-09T22:41:44.760420Z","iopub.execute_input":"2022-08-09T22:41:44.760747Z","iopub.status.idle":"2022-08-09T22:41:44.772935Z","shell.execute_reply.started":"2022-08-09T22:41:44.760714Z","shell.execute_reply":"2022-08-09T22:41:44.771723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"21\"></a><b>\n# Feature Engineering ","metadata":{}},{"cell_type":"markdown","source":"<a id = \"22\"></a><b>\n## Name -- Title","metadata":{}},{"cell_type":"code","source":"train_df[\"Name\"].head(10)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:44.774275Z","iopub.execute_input":"2022-08-09T22:41:44.774738Z","iopub.status.idle":"2022-08-09T22:41:44.786669Z","shell.execute_reply.started":"2022-08-09T22:41:44.774708Z","shell.execute_reply":"2022-08-09T22:41:44.785651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"name = train_df[\"Name\"]","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:44.788048Z","iopub.execute_input":"2022-08-09T22:41:44.789006Z","iopub.status.idle":"2022-08-09T22:41:44.794584Z","shell.execute_reply.started":"2022-08-09T22:41:44.788970Z","shell.execute_reply":"2022-08-09T22:41:44.793709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"Title\"] = [i.split(\".\")[0].split(\",\")[-1].strip() for i in name]","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:44.795838Z","iopub.execute_input":"2022-08-09T22:41:44.796339Z","iopub.status.idle":"2022-08-09T22:41:44.806305Z","shell.execute_reply.started":"2022-08-09T22:41:44.796310Z","shell.execute_reply":"2022-08-09T22:41:44.805307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"Title\"].head(10)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:44.807699Z","iopub.execute_input":"2022-08-09T22:41:44.808370Z","iopub.status.idle":"2022-08-09T22:41:44.818388Z","shell.execute_reply.started":"2022-08-09T22:41:44.808329Z","shell.execute_reply":"2022-08-09T22:41:44.817385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(x = \"Title\", data = train_df)\nplt.xticks(rotation = 60)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:44.819774Z","iopub.execute_input":"2022-08-09T22:41:44.820479Z","iopub.status.idle":"2022-08-09T22:41:45.070818Z","shell.execute_reply.started":"2022-08-09T22:41:44.820439Z","shell.execute_reply":"2022-08-09T22:41:45.070021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert to categorical\ntrain_df[\"Title\"] = train_df[\"Title\"].replace([\"Lady\", \"Sir\", \"Don\", \"the Countess\", \"Capt\", \"Col\", \"Dr\", \"Major\", \"Rev\", \"Jonkheer\", \"Dona\"], \"other\")\ntrain_df[\"Title\"] = [0 if i == \"Master\" else 1 if i == \"Miss\" or i == \"Ms\" or i == \"Mlle\" or i == \"Mrs\" else 2 if i == \"Mr\" else 3 for i in train_df[\"Title\"]]","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:45.072404Z","iopub.execute_input":"2022-08-09T22:41:45.073096Z","iopub.status.idle":"2022-08-09T22:41:45.083707Z","shell.execute_reply.started":"2022-08-09T22:41:45.073053Z","shell.execute_reply":"2022-08-09T22:41:45.082674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"Title\"].head(10)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:45.085247Z","iopub.execute_input":"2022-08-09T22:41:45.085725Z","iopub.status.idle":"2022-08-09T22:41:45.094757Z","shell.execute_reply.started":"2022-08-09T22:41:45.085685Z","shell.execute_reply":"2022-08-09T22:41:45.093724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(x = \"Title\", data = train_df)\nplt.xticks(rotation = 60)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:45.096037Z","iopub.execute_input":"2022-08-09T22:41:45.096810Z","iopub.status.idle":"2022-08-09T22:41:45.259847Z","shell.execute_reply.started":"2022-08-09T22:41:45.096778Z","shell.execute_reply":"2022-08-09T22:41:45.259002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"g = sns.factorplot(x = \"Title\", y = \"Survived\", data = train_df, kind = \"bar\")\ng.set_xticklabels([\"Master\", \"Mrs\", \"Mr\", \"Other\"])\ng.set_ylabels(\"Survived Probability\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:45.260954Z","iopub.execute_input":"2022-08-09T22:41:45.261884Z","iopub.status.idle":"2022-08-09T22:41:45.607564Z","shell.execute_reply.started":"2022-08-09T22:41:45.261841Z","shell.execute_reply":"2022-08-09T22:41:45.606484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.drop(labels = [\"Name\"], axis = 1, inplace = True)\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:45.609006Z","iopub.execute_input":"2022-08-09T22:41:45.609301Z","iopub.status.idle":"2022-08-09T22:41:45.626703Z","shell.execute_reply.started":"2022-08-09T22:41:45.609274Z","shell.execute_reply":"2022-08-09T22:41:45.625879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.get_dummies(train_df, columns = [\"Title\"])\ntrain_df.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:45.627918Z","iopub.execute_input":"2022-08-09T22:41:45.628287Z","iopub.status.idle":"2022-08-09T22:41:45.656118Z","shell.execute_reply.started":"2022-08-09T22:41:45.628253Z","shell.execute_reply":"2022-08-09T22:41:45.655109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"23\"></a><b>\n## Family Size","metadata":{}},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:45.657599Z","iopub.execute_input":"2022-08-09T22:41:45.657903Z","iopub.status.idle":"2022-08-09T22:41:45.675041Z","shell.execute_reply.started":"2022-08-09T22:41:45.657876Z","shell.execute_reply":"2022-08-09T22:41:45.673857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"f_size\"] = train_df[\"SibSp\"] + train_df[\"Parch\"] + 1","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:45.676835Z","iopub.execute_input":"2022-08-09T22:41:45.677325Z","iopub.status.idle":"2022-08-09T22:41:45.685514Z","shell.execute_reply.started":"2022-08-09T22:41:45.677283Z","shell.execute_reply":"2022-08-09T22:41:45.684480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:45.686830Z","iopub.execute_input":"2022-08-09T22:41:45.687463Z","iopub.status.idle":"2022-08-09T22:41:45.708754Z","shell.execute_reply.started":"2022-08-09T22:41:45.687419Z","shell.execute_reply":"2022-08-09T22:41:45.707956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"g = sns.factorplot(x = \"f_size\", y = \"Survived\", data = train_df, kind = \"bar\")\ng.set_ylabels(\"Survived Probability\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:45.710026Z","iopub.execute_input":"2022-08-09T22:41:45.710789Z","iopub.status.idle":"2022-08-09T22:41:46.158306Z","shell.execute_reply.started":"2022-08-09T22:41:45.710755Z","shell.execute_reply":"2022-08-09T22:41:46.157260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"family_size\"] = [1 if i < 5 else 0 for i in train_df[\"f_size\"]]","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:46.159861Z","iopub.execute_input":"2022-08-09T22:41:46.160959Z","iopub.status.idle":"2022-08-09T22:41:46.167482Z","shell.execute_reply.started":"2022-08-09T22:41:46.160913Z","shell.execute_reply":"2022-08-09T22:41:46.166308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:46.169143Z","iopub.execute_input":"2022-08-09T22:41:46.169445Z","iopub.status.idle":"2022-08-09T22:41:46.194519Z","shell.execute_reply.started":"2022-08-09T22:41:46.169417Z","shell.execute_reply":"2022-08-09T22:41:46.193273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(x = \"family_size\", data = train_df)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:46.196264Z","iopub.execute_input":"2022-08-09T22:41:46.197166Z","iopub.status.idle":"2022-08-09T22:41:46.340273Z","shell.execute_reply.started":"2022-08-09T22:41:46.197120Z","shell.execute_reply":"2022-08-09T22:41:46.339108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"g = sns.factorplot(x = \"family_size\", y = \"Survived\", data = train_df, kind = \"bar\")\ng.set_ylabels(\"Survived Probabily\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:46.342346Z","iopub.execute_input":"2022-08-09T22:41:46.343212Z","iopub.status.idle":"2022-08-09T22:41:46.610246Z","shell.execute_reply.started":"2022-08-09T22:41:46.343162Z","shell.execute_reply":"2022-08-09T22:41:46.609139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Small families have more chance to survive than large families","metadata":{}},{"cell_type":"code","source":"train_df = pd.get_dummies(train_df, columns = [\"family_size\"])","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:46.611719Z","iopub.execute_input":"2022-08-09T22:41:46.612358Z","iopub.status.idle":"2022-08-09T22:41:46.623207Z","shell.execute_reply.started":"2022-08-09T22:41:46.612312Z","shell.execute_reply":"2022-08-09T22:41:46.621987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:46.624423Z","iopub.execute_input":"2022-08-09T22:41:46.624759Z","iopub.status.idle":"2022-08-09T22:41:46.650001Z","shell.execute_reply.started":"2022-08-09T22:41:46.624728Z","shell.execute_reply":"2022-08-09T22:41:46.648820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"24\"></a><b>\n## Embarked","metadata":{}},{"cell_type":"code","source":"train_df[\"Embarked\"].head()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:46.651571Z","iopub.execute_input":"2022-08-09T22:41:46.652007Z","iopub.status.idle":"2022-08-09T22:41:46.660760Z","shell.execute_reply.started":"2022-08-09T22:41:46.651964Z","shell.execute_reply":"2022-08-09T22:41:46.659894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(x = \"Embarked\", data = train_df)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:46.662554Z","iopub.execute_input":"2022-08-09T22:41:46.663351Z","iopub.status.idle":"2022-08-09T22:41:46.799807Z","shell.execute_reply.started":"2022-08-09T22:41:46.663309Z","shell.execute_reply":"2022-08-09T22:41:46.798675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.get_dummies(train_df, columns = [\"Embarked\"])\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:46.801815Z","iopub.execute_input":"2022-08-09T22:41:46.802325Z","iopub.status.idle":"2022-08-09T22:41:46.830468Z","shell.execute_reply.started":"2022-08-09T22:41:46.802282Z","shell.execute_reply":"2022-08-09T22:41:46.829392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"25\"></a><b>\n## Ticket","metadata":{}},{"cell_type":"code","source":"train_df[\"Ticket\"].head(10) ","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:46.831899Z","iopub.execute_input":"2022-08-09T22:41:46.832561Z","iopub.status.idle":"2022-08-09T22:41:46.843470Z","shell.execute_reply.started":"2022-08-09T22:41:46.832519Z","shell.execute_reply":"2022-08-09T22:41:46.842663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tickets = []\n\nfor i in list(train_df[\"Ticket\"]):\n    if not i.isdigit():\n        tickets.append(i.replace(\".\", \"\").replace(\"/\", \"\").strip().split(\" \")[0])\n    else:\n        tickets.append(\"x\")\n\ntrain_df[\"Ticket\"] = tickets","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:46.844862Z","iopub.execute_input":"2022-08-09T22:41:46.846886Z","iopub.status.idle":"2022-08-09T22:41:46.853987Z","shell.execute_reply.started":"2022-08-09T22:41:46.846853Z","shell.execute_reply":"2022-08-09T22:41:46.853088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:46.855175Z","iopub.execute_input":"2022-08-09T22:41:46.855796Z","iopub.status.idle":"2022-08-09T22:41:46.879405Z","shell.execute_reply.started":"2022-08-09T22:41:46.855765Z","shell.execute_reply":"2022-08-09T22:41:46.878339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.get_dummies(train_df, columns = [\"Ticket\"], prefix = \"T\")\ntrain_df.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:46.881733Z","iopub.execute_input":"2022-08-09T22:41:46.882869Z","iopub.status.idle":"2022-08-09T22:41:46.912762Z","shell.execute_reply.started":"2022-08-09T22:41:46.882824Z","shell.execute_reply":"2022-08-09T22:41:46.911599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"26\"></a><b>\n## Pclass","metadata":{}},{"cell_type":"code","source":"sns.countplot(\"Pclass\", data = train_df)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:46.914346Z","iopub.execute_input":"2022-08-09T22:41:46.915375Z","iopub.status.idle":"2022-08-09T22:41:47.033601Z","shell.execute_reply.started":"2022-08-09T22:41:46.915330Z","shell.execute_reply":"2022-08-09T22:41:47.032434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"Pclass\"] = train_df[\"Pclass\"].astype(\"category\")","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.035346Z","iopub.execute_input":"2022-08-09T22:41:47.036054Z","iopub.status.idle":"2022-08-09T22:41:47.042506Z","shell.execute_reply.started":"2022-08-09T22:41:47.036012Z","shell.execute_reply":"2022-08-09T22:41:47.041657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.get_dummies(train_df, columns = [\"Pclass\"])","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.043881Z","iopub.execute_input":"2022-08-09T22:41:47.044518Z","iopub.status.idle":"2022-08-09T22:41:47.059891Z","shell.execute_reply.started":"2022-08-09T22:41:47.044472Z","shell.execute_reply":"2022-08-09T22:41:47.058477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.061362Z","iopub.execute_input":"2022-08-09T22:41:47.062297Z","iopub.status.idle":"2022-08-09T22:41:47.085138Z","shell.execute_reply.started":"2022-08-09T22:41:47.062255Z","shell.execute_reply":"2022-08-09T22:41:47.083952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"27\"></a><b>\n## Sex","metadata":{}},{"cell_type":"code","source":"train_df[\"Sex\"] = train_df[\"Sex\"].astype(\"category\")\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.086699Z","iopub.execute_input":"2022-08-09T22:41:47.087391Z","iopub.status.idle":"2022-08-09T22:41:47.111122Z","shell.execute_reply.started":"2022-08-09T22:41:47.087349Z","shell.execute_reply":"2022-08-09T22:41:47.110319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.get_dummies(train_df, columns = [\"Sex\"])","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.112332Z","iopub.execute_input":"2022-08-09T22:41:47.113243Z","iopub.status.idle":"2022-08-09T22:41:47.123158Z","shell.execute_reply.started":"2022-08-09T22:41:47.113197Z","shell.execute_reply":"2022-08-09T22:41:47.122131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.124571Z","iopub.execute_input":"2022-08-09T22:41:47.125356Z","iopub.status.idle":"2022-08-09T22:41:47.147497Z","shell.execute_reply.started":"2022-08-09T22:41:47.125313Z","shell.execute_reply":"2022-08-09T22:41:47.146314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"28\"></a><b>\n## Drop Passenger ID and Cabin","metadata":{}},{"cell_type":"code","source":"train_df.drop([\"PassengerId\", \"Cabin\"], axis = 1, inplace = True)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.148797Z","iopub.execute_input":"2022-08-09T22:41:47.149770Z","iopub.status.idle":"2022-08-09T22:41:47.158994Z","shell.execute_reply.started":"2022-08-09T22:41:47.149732Z","shell.execute_reply":"2022-08-09T22:41:47.158084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.columns","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.160290Z","iopub.execute_input":"2022-08-09T22:41:47.161181Z","iopub.status.idle":"2022-08-09T22:41:47.171279Z","shell.execute_reply.started":"2022-08-09T22:41:47.161148Z","shell.execute_reply":"2022-08-09T22:41:47.170284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"29\"></a><b>\n# Modeling","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split, StratifiedKFold, GridSearchCV\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.svm import SVC\nfrom sklearn.ensemble import RandomForestClassifier, VotingClassifier\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.metrics import accuracy_score","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.172361Z","iopub.execute_input":"2022-08-09T22:41:47.173168Z","iopub.status.idle":"2022-08-09T22:41:47.488748Z","shell.execute_reply.started":"2022-08-09T22:41:47.173125Z","shell.execute_reply":"2022-08-09T22:41:47.487828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"30\"></a><b>\n## Train - Test Split","metadata":{}},{"cell_type":"code","source":"train_df_len","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.490230Z","iopub.execute_input":"2022-08-09T22:41:47.490541Z","iopub.status.idle":"2022-08-09T22:41:47.495777Z","shell.execute_reply.started":"2022-08-09T22:41:47.490512Z","shell.execute_reply":"2022-08-09T22:41:47.494945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = train_df[train_df_len:]\ntest.drop([\"Survived\"], inplace = True, axis = 1)\n\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.497149Z","iopub.execute_input":"2022-08-09T22:41:47.497688Z","iopub.status.idle":"2022-08-09T22:41:47.523668Z","shell.execute_reply.started":"2022-08-09T22:41:47.497595Z","shell.execute_reply":"2022-08-09T22:41:47.522839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train_df[:train_df_len]\nX_train = train.drop([\"Survived\"], axis = 1)\ny_train = train[\"Survived\"]","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.524845Z","iopub.execute_input":"2022-08-09T22:41:47.525352Z","iopub.status.idle":"2022-08-09T22:41:47.530708Z","shell.execute_reply.started":"2022-08-09T22:41:47.525321Z","shell.execute_reply":"2022-08-09T22:41:47.529901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(X_train, y_train, test_size = 0.33, random_state = 42)\n\nprint(\"x_train: \", len(X_train))\nprint(\"X_test: \", len(X_test))\nprint(\"y_train: \", len(y_train))\nprint(\"y_test: \", len(y_test))\nprint(\"test: \", len(test))","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.532308Z","iopub.execute_input":"2022-08-09T22:41:47.532830Z","iopub.status.idle":"2022-08-09T22:41:47.550435Z","shell.execute_reply.started":"2022-08-09T22:41:47.532795Z","shell.execute_reply":"2022-08-09T22:41:47.549463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"31\"></a><b>\n## Simple Logistic Regression","metadata":{}},{"cell_type":"code","source":"logreg = LogisticRegression()\nlogreg.fit(X_train, y_train)\n\nacc_log_train = round(logreg.score(X_train, y_train) * 100,2)\nacc_log_test = round(logreg.score(X_test, y_test) * 100,2)\n\nprint(\"Training Accuracy: % {}\".format(acc_log_train))\nprint(\"Test Accuracy: % {}\".format(acc_log_test))","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.560693Z","iopub.execute_input":"2022-08-09T22:41:47.561290Z","iopub.status.idle":"2022-08-09T22:41:47.630920Z","shell.execute_reply.started":"2022-08-09T22:41:47.561255Z","shell.execute_reply":"2022-08-09T22:41:47.629743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"32\"></a><b>\n## Hyperparameter Tuning -- Grid Search -- Cross Validation\nWe will compare 5 ml classifier and evaluate mean accuracy of each of them by stratified cross validation\n    \n* Decision Tree\n* SVM\n* Random Forest\n* KNN\n* Logistic Regression","metadata":{}},{"cell_type":"code","source":"random_state = 42\nclassifier = [DecisionTreeClassifier(random_state = random_state),\n             SVC(random_state = random_state),\n             RandomForestClassifier(random_state = random_state),\n             LogisticRegression(random_state = random_state),\n             KNeighborsClassifier()]\n\ndt_param_grid = {\"min_samples_split\" : range(10,500,20),\n                \"max_depth\": range(1,20,2)}\n\nsvc_param_grid = {\"kernel\" : [\"rbf\"],\n                 \"gamma\": [0.001, 0.01, 0.1, 1],\n                 \"C\": [1,10,50,100,200,300,1000]}\n\nrf_param_grid = {\"max_features\": [1,3,10],\n                \"min_samples_split\":[2,3,10],\n                \"min_samples_leaf\":[1,3,10],\n                \"bootstrap\":[False],\n                \"n_estimators\":[100,300],\n                \"criterion\":[\"gini\"]}\n\nlogreg_param_grid = {\"C\":np.logspace(-3,3,7),\n                    \"penalty\": [\"l1\",\"l2\"]}\n\nknn_param_grid = {\"n_neighbors\": np.linspace(1,19,10, dtype = int).tolist(),\n                 \"weights\": [\"uniform\",\"distance\"],\n                 \"metric\":[\"euclidean\",\"manhattan\"]}\n\nclassifier_param = [dt_param_grid,\n                   svc_param_grid,\n                   rf_param_grid,\n                   logreg_param_grid,\n                   knn_param_grid]","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.632716Z","iopub.execute_input":"2022-08-09T22:41:47.633427Z","iopub.status.idle":"2022-08-09T22:41:47.647984Z","shell.execute_reply.started":"2022-08-09T22:41:47.633383Z","shell.execute_reply":"2022-08-09T22:41:47.646704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cv_result = []\nbest_estimators = []\n\nfor i in range(len(classifier)):\n    clf = GridSearchCV(classifier[i], param_grid = classifier_param[i], cv = StratifiedKFold(n_splits = 10), scoring = \"accuracy\", n_jobs = -1,verbose = 1)\n    clf.fit(X_train, y_train)\n    cv_result.append(clf.best_score_)\n    best_estimators.append(clf.best_estimator_)\n    print(cv_result[i])","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:41:47.649783Z","iopub.execute_input":"2022-08-09T22:41:47.650677Z","iopub.status.idle":"2022-08-09T22:43:03.802953Z","shell.execute_reply.started":"2022-08-09T22:41:47.650627Z","shell.execute_reply":"2022-08-09T22:43:03.801558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cv_results = pd.DataFrame({\"Cross Validation Means\":cv_result, \"ML Models\":[\"DecisionTreeClassifier\", \"SVM\",\"RandomForestClassifier\",\n             \"LogisticRegression\",\n             \"KNeighborsClassifier\"]})\n\ng = sns.barplot(\"Cross Validation Means\", \"ML Models\", data = cv_results)\ng.set_xlabel(\"Mean Accuracy\")\ng.set_title(\"Cross Validation Scores\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:43:03.804645Z","iopub.execute_input":"2022-08-09T22:43:03.805751Z","iopub.status.idle":"2022-08-09T22:43:04.003433Z","shell.execute_reply.started":"2022-08-09T22:43:03.805710Z","shell.execute_reply":"2022-08-09T22:43:04.002455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"33\"></a><b>\n## Ensemble Modeling","metadata":{}},{"cell_type":"code","source":"votingC = VotingClassifier(estimators = [(\"dt\", best_estimators[0]),\n                                        (\"rfc\", best_estimators[2]),\n                                        (\"lr\", best_estimators[3])],\n                                        voting = \"soft\", n_jobs = -1)\n\nvotingC = votingC.fit(X_train, y_train)\nprint(accuracy_score(votingC.predict(X_test), y_test))","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:43:04.004834Z","iopub.execute_input":"2022-08-09T22:43:04.005179Z","iopub.status.idle":"2022-08-09T22:43:04.637708Z","shell.execute_reply.started":"2022-08-09T22:43:04.005150Z","shell.execute_reply":"2022-08-09T22:43:04.636295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"34\"></a><b>\n## Prediction and Submission","metadata":{}},{"cell_type":"code","source":"test_survived = pd.Series(votingC.predict(test), name = \"Survived\").astype(int)\nresult = pd.concat([test_Passenger_Id, test_survived], axis = 1)\nresult.to_csv(\"titanic.csv\", index = False)","metadata":{"execution":{"iopub.status.busy":"2022-08-09T22:43:04.639433Z","iopub.execute_input":"2022-08-09T22:43:04.639962Z","iopub.status.idle":"2022-08-09T22:43:04.738995Z","shell.execute_reply.started":"2022-08-09T22:43:04.639908Z","shell.execute_reply":"2022-08-09T22:43:04.737647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}