{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!pip -q --disable-pip-version-check install mplcyberpunk","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-06T07:40:57.086152Z","iopub.execute_input":"2022-07-06T07:40:57.087679Z","iopub.status.idle":"2022-07-06T07:41:10.042581Z","shell.execute_reply.started":"2022-07-06T07:40:57.087504Z","shell.execute_reply":"2022-07-06T07:41:10.040954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

SPACESHIP TITANIC

","metadata":{}},{"cell_type":"markdown","source":"

Introduction 📝

\nThe goal of the competition is to predict whether a passenger was transported to an alternate dimension during the Spaceship Titanic's collision with the spacetime anomaly.This notebook will contain almost all the necessary steps and methods which will be helpful in this competition.","metadata":{}},{"cell_type":"markdown","source":"

Dataset Info 📈

\nColumns of the train data- \n\n* ```PassengerId``` - A unique Id for each passenger. Each Id takes the form gggg_pp where gggg indicates a group the passenger is travelling with and pp is their number within the group. People in a group are often family members, but not always.\n* ```HomePlanet``` - The planet the passenger departed from, typically their planet of permanent residence.\n* ```CryoSleep``` - Indicates whether the passenger elected to be put into suspended animation for the duration of the voyage. Passengers in cryosleep are confined to their cabins.\n* ```Cabin``` - The cabin number where the passenger is staying. Takes the form deck/num/side, where side can be either P for Port or S for Starboard.\n* ```Destination``` - The planet the passenger will be debarking to.\n* ```Age``` - The age of the passenger.\n* ```VIP``` - Whether the passenger has paid for special VIP service during the voyage.\n* ```RoomService, FoodCourt, ShoppingMall, Spa, VRDeck``` - Amount the passenger has billed at each of the Spaceship Titanic's many luxury amenities.\n* ```Name``` - The first and last names of the passenger.\n* ```Transported``` - Whether the passenger was transported to another dimension. This is the target, the column you are trying to predict.","metadata":{}},{"cell_type":"markdown","source":"

Evaluation Metric 📐

\nSubmissions are evaluated based on their classification accuracy, the percentage of predicted labels that are correct.\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n

Please consider upvoting the kernel if you found it useful.

\n
","metadata":{}},{"cell_type":"markdown","source":"#

TABLE OF CONTENTS

\n","metadata":{}},{"cell_type":"markdown","source":"\n#

IMPORTING LIBRARIES

","metadata":{}},{"cell_type":"code","source":"import shap\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport mplcyberpunk\nfrom termcolor import colored\nplt.style.use('cyberpunk')\n\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.svm import SVC\nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier\nfrom xgboost import XGBClassifier\nfrom sklearn.metrics import accuracy_score\n\nimport warnings\nwarnings.simplefilter('ignore')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-06T07:41:10.049606Z","iopub.execute_input":"2022-07-06T07:41:10.050071Z","iopub.status.idle":"2022-07-06T07:41:13.635874Z","shell.execute_reply.started":"2022-07-06T07:41:10.050018Z","shell.execute_reply":"2022-07-06T07:41:13.635075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#

READING THE DATA

","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv('../input/spaceship-titanic/train.csv')\ndf_test = pd.read_csv('../input/spaceship-titanic/test.csv')","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:13.637315Z","iopub.execute_input":"2022-07-06T07:41:13.638255Z","iopub.status.idle":"2022-07-06T07:41:13.733466Z","shell.execute_reply.started":"2022-07-06T07:41:13.638182Z","shell.execute_reply":"2022-07-06T07:41:13.732738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:13.738305Z","iopub.execute_input":"2022-07-06T07:41:13.739774Z","iopub.status.idle":"2022-07-06T07:41:13.773170Z","shell.execute_reply.started":"2022-07-06T07:41:13.739714Z","shell.execute_reply":"2022-07-06T07:41:13.772309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:13.775969Z","iopub.execute_input":"2022-07-06T07:41:13.777186Z","iopub.status.idle":"2022-07-06T07:41:13.801962Z","shell.execute_reply.started":"2022-07-06T07:41:13.777115Z","shell.execute_reply":"2022-07-06T07:41:13.800965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(colored(f'Number of rows in train data: {df_train.shape[0]}', 'red'))\nprint(colored(f'Number of columns in train data: {df_train.shape[1]}', 'red'))\nprint(colored(f'Number of rows in test data: {df_test.shape[0]}', 'blue'))\nprint(colored(f'Number of columns in test data: {df_test.shape[1]}', 'blue'))","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:13.804153Z","iopub.execute_input":"2022-07-06T07:41:13.804725Z","iopub.status.idle":"2022-07-06T07:41:13.813999Z","shell.execute_reply.started":"2022-07-06T07:41:13.804676Z","shell.execute_reply":"2022-07-06T07:41:13.812306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.describe()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:13.816260Z","iopub.execute_input":"2022-07-06T07:41:13.817126Z","iopub.status.idle":"2022-07-06T07:41:13.881366Z","shell.execute_reply.started":"2022-07-06T07:41:13.817070Z","shell.execute_reply":"2022-07-06T07:41:13.880306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test.describe()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:13.882944Z","iopub.execute_input":"2022-07-06T07:41:13.883904Z","iopub.status.idle":"2022-07-06T07:41:13.920398Z","shell.execute_reply.started":"2022-07-06T07:41:13.883849Z","shell.execute_reply":"2022-07-06T07:41:13.919436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#

EXPLORATORY DATA ANALYSIS

\n### We perform EDA to analyse and gain insights of the data which will help in better understanding the problem and will bring an advantage when creating models.","metadata":{}},{"cell_type":"markdown","source":"\n## 1. Missing Values\n### First of all, we will start by analying the missing values in the dataset. ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(20,6))\n\nna = pd.DataFrame(df_train.isna().sum())\n\nsns.barplot(y=na[0], x=na.index)\nplt.title('Missing Values Distribution', size = 20, weight='bold')\nprint(colored(\"Missing values column wise -\", 'magenta'))\nprint(colored(df_train.isna().sum(), 'magenta'))\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:13.921890Z","iopub.execute_input":"2022-07-06T07:41:13.922191Z","iopub.status.idle":"2022-07-06T07:41:14.335715Z","shell.execute_reply.started":"2022-07-06T07:41:13.922157Z","shell.execute_reply":"2022-07-06T07:41:14.334278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

INSIGHTS FROM THE GRAPH

\n\n* We can observe that almost all the columns have Null values. This is a problem that can be solved in multiple ways depeding on the situation which will we see later.\n* Let us also analyse how much these Null values of each column affects Transported.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(4, 3, figsize=(20, 20))\nfig.suptitle(\"Missing Values Distribution By Transported\", size = 20, weight='bold')\nfig.subplots_adjust(top=0.95)\ni = 0\nfor x in df_train.columns:\n if len(df_train[df_train[x].isna()==True])>0:\n sns.countplot(x='Transported', data=df_train[df_train[x].isna()==True], ax=fig.axes[i], palette='turbo')\n fig.axes[i].set_title(x, weight='bold')\n i += 1\n \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:14.339667Z","iopub.execute_input":"2022-07-06T07:41:14.340377Z","iopub.status.idle":"2022-07-06T07:41:15.630814Z","shell.execute_reply.started":"2022-07-06T07:41:14.340335Z","shell.execute_reply":"2022-07-06T07:41:15.629838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

INSIGHTS FROM THE GRAPH

\n\n* We can observe that all the columns have almost equal distribution of target class. \n* So one thing we can say that even if we drop rows with missing values according to a particular column, it would not create any bias in the data(but we will not drop the rows right now).","metadata":{}},{"cell_type":"markdown","source":"\n## 2. Transported\n### Transported is the target column which we have to predict. Let us analyse its distribution.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(9,6))\n\nsns.countplot(x='Transported', data=df_train, palette = 'winter')\nplt.title(\"Transported Distribution\", size = 20, weight='bold')\n\nprint(colored(f\"Percentage of Passengers Transported - {(len(df_train[df_train['Transported']==True]) / df_train.shape[0])*100:.2f}%\", 'cyan'))\nprint(colored(f\"Percentage of Passengers Not Transported - {(len(df_train[df_train['Transported']==False]) / df_train.shape[0])*100:.2f}%\", 'cyan'))\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:15.632413Z","iopub.execute_input":"2022-07-06T07:41:15.632717Z","iopub.status.idle":"2022-07-06T07:41:15.818703Z","shell.execute_reply.started":"2022-07-06T07:41:15.632688Z","shell.execute_reply":"2022-07-06T07:41:15.817442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

INSIGHTS FROM THE GRAPH

\n\n* We can observe that the target column is almost perfectly balanced, so we don't have to worry about unequal distribution.","metadata":{}},{"cell_type":"markdown","source":"\n## 3. PassengerId\nAll the passengers have unique id so we can't use this feature directly for modelling. But we will not discard this feature as we can perform some feature engineering to extract useful information from it. Also it does not contain any null values so extracting features from it might be very helpful.\n\nEach Id takes the form gggg_pp where gggg indicates a group the passenger is travelling with and pp is their number within the group, so we can get to know the size of the group and can check how it affects Transported.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(20,7))\nfig.suptitle(\"GroupSize Distribution\", size = 20, weight='bold')\n\ndf_train['Group'] = df_train['PassengerId'].apply(lambda x: x.split('_')[0]).astype(int)\ndf_train['GroupSize']=df_train['Group'].map(lambda x: df_train['Group'].value_counts()[x])\n\ndf_temp = df_train.drop_duplicates(subset = [\"Group\"], keep='last')\n\nsns.countplot(x='GroupSize', data=df_temp, ax=ax[0])\nsns.countplot(x='GroupSize', data=df_temp, hue='Transported', ax=ax[1])\n\nprint(colored(f\"Number of unique groups - {len(df_temp)}\",'blue'))\ndata = pd.DataFrame(df_temp['GroupSize'].value_counts()).reset_index().rename(columns={'index': 'GroupSize', 'GroupSize':'Count'})\nprint(colored(\"Group Size Distribution - \",'blue'))\nprint(colored(data, 'blue'))\nplt.show()\n\nprint(colored(f\"Total number of individual passengers - {len(df_temp[df_temp['GroupSize']==1])}\", 'blue'))\nprint(colored(f\"Number of individual passengers transported - {len(df_temp[(df_temp['GroupSize']==1) & (df_temp['Transported']==True)])}\", 'blue'))\nprint(colored(f\"Number of individual passengers not transported - {len(df_temp[(df_temp['GroupSize']==1) & (df_temp['Transported']==False)])}\", 'blue'))\nprint(colored(f\"Toal number of non individual passengers - {len(df_temp[df_temp['GroupSize']!=1])}\", 'red'))\nprint(colored(f\"Number of non individual passengers transported - {len(df_temp[(df_temp['GroupSize']!=1) & (df_temp['Transported']==True)])}\", 'red'))\nprint(colored(f\"Number of non individual passengers not transported - {len(df_temp[(df_temp['GroupSize']!=1) & (df_temp['Transported']==False)])}\", 'red'))","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:15.820472Z","iopub.execute_input":"2022-07-06T07:41:15.821132Z","iopub.status.idle":"2022-07-06T07:41:23.828468Z","shell.execute_reply.started":"2022-07-06T07:41:15.821083Z","shell.execute_reply":"2022-07-06T07:41:23.826942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

INSIGHTS FROM THE GRAPH

\n\n* We can observe that most of the passengers are individual passengers. \n* Apart from that the maximum size of the group is 8. \n* An interesting observation is that there is lesser chance of passenger to be transported if he/she is an individual than in a group.","metadata":{}},{"cell_type":"markdown","source":"\n## 4. HomePlanet\n This is one of the categorical feature. There are 3 unique values and this feature contains some null values which we need to take care but first let's check its distribution and dependence on Transported.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(20,6))\nfig.suptitle('HomePlanet Distribution', size = 20, weight='bold')\n\nsizes = list(df_train['HomePlanet'].value_counts(sort=False))\n\nlabels = df_train['HomePlanet'].dropna().unique()\ncolors = ['#099FFF', '#CC00FF', '#13CA91']\nexplode = (0.05,0.05,0.05) \n\nax[0].pie(sizes, colors=colors, explode=explode, startangle=90, labels=labels,\n autopct='%1.2f%%', pctdistance=0.6,textprops={'fontsize':12})\nsns.countplot(x='HomePlanet', data=df_train, hue='Transported', ax=ax[1])\n\nprint(colored(\"HomePlanet Distribution - \",'green'))\ndata = pd.DataFrame(df_train['HomePlanet'].value_counts()).reset_index().rename(columns={'index': 'HomePlanet', 'HomePlanet':'Count'})\nprint(colored(data, 'green'))\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:23.829733Z","iopub.execute_input":"2022-07-06T07:41:23.830582Z","iopub.status.idle":"2022-07-06T07:41:24.166621Z","shell.execute_reply.started":"2022-07-06T07:41:23.830479Z","shell.execute_reply":"2022-07-06T07:41:24.165439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

INSIGHTS FROM THE GRAPH

\n\n* We can observe that most of the passengers are from Earth. \n* There is a higher chance of residents from Europa to be transported than others.\n* The chances of the residents of Earth of getting transported is less.\n* There is equal probability for the residents of Mars.","metadata":{}},{"cell_type":"markdown","source":"\n## 5. CyroSleep\nThis is another categorical feature with value either True or False and this feature contains maximum null values which we need to take care but first let's check its distribution and dependence on Transported.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(20,6))\nfig.suptitle('CryoSleep Distribution', size = 20, weight='bold')\n\nsizes = list(df_train['CryoSleep'].value_counts())\n\nlabels = df_train['CryoSleep'].dropna().unique()\ncolors = ['#099FFF', '#CC00FF']\nexplode = (0.05,0.05) \n\nax[0].pie(sizes, colors=colors, explode=explode, startangle=90, labels=labels,\n autopct='%1.2f%%', pctdistance=0.6,textprops={'fontsize':12})\nsns.countplot(x='CryoSleep', data=df_train, hue='Transported', ax=ax[1])\n\nprint(colored(\"CryoSleep Distribution - \",'magenta'))\ndata = pd.DataFrame(df_train['CryoSleep'].value_counts()).reset_index().rename(columns={'index': 'CryoSleep', 'CryoSleep':'Count'})\nprint(colored(data, 'magenta'))\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:24.169329Z","iopub.execute_input":"2022-07-06T07:41:24.169733Z","iopub.status.idle":"2022-07-06T07:41:24.680965Z","shell.execute_reply.started":"2022-07-06T07:41:24.169686Z","shell.execute_reply":"2022-07-06T07:41:24.680050Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

INSIGHTS FROM THE GRAPH

\n\n* We can observe that CyroSleep has a high false percentage. \n* One of the best thing is that this feature has a direct relationship with the Transported. \n* If CyroSleep is False, then the chances of Transported is less (Transported=False) whereas if CyroSleep is True, then the chances of Transported is high (Transported=True).","metadata":{}},{"cell_type":"markdown","source":"\n## 6. Cabin\nJust like PassengerId, we can't directly use Cabin feature. But we will not discard this feature as we can perform some feature engineering to extract useful information from it. \n\nEach cabin takes the form deck/num/side, where there are 8 unique deck values and side can be either P for Port or S for Starboard. So we can extract these features to know their distribution and can check how it affects Transported.","metadata":{}},{"cell_type":"code","source":"df_train['Cabin'].fillna('Z/9999/Z', inplace=True)\ndf_train['deck'] = df_train['Cabin'].apply(lambda x : x.split('/')[0])\ndf_train['side'] = df_train['Cabin'].apply(lambda x : x.split('/')[2])\n\nfig, ax = plt.subplots(2, 2, figsize=(20,12))\nfig.suptitle('Cabin Distribution', size = 20, weight='bold')\n\nsns.countplot(x='deck', data=df_train, order=['A','B','C','D','E','F','G','T','Z'], ax=ax[0][0], palette='turbo')\nsns.countplot(x='deck', data=df_train, order=['A','B','C','D','E','F','G','T','Z'], hue='Transported', ax=ax[0][1])\n\nsns.countplot(x='side', data=df_train, ax=ax[1][0], palette='turbo')\nsns.countplot(x='side', data=df_train, hue='Transported', ax=ax[1][1])\n\nprint(colored(\"Cabin Deck Distribution - \",'red'))\ndata = pd.DataFrame(df_train['deck'].value_counts()).reset_index().rename(columns={'index': 'Deck', 'deck':'Count'})\nprint(colored(data, 'red'))\n\nprint(colored(\"Cabin Side Distribution - \",'blue'))\ndata = pd.DataFrame(df_train['side'].value_counts()).reset_index().rename(columns={'index': 'Side', 'side':'Count'})\nprint(colored(data, 'blue'))\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:24.682647Z","iopub.execute_input":"2022-07-06T07:41:24.683215Z","iopub.status.idle":"2022-07-06T07:41:25.361052Z","shell.execute_reply.started":"2022-07-06T07:41:24.683168Z","shell.execute_reply":"2022-07-06T07:41:25.359598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

INSIGHTS FROM THE GRAPH

\n\n* The distribution of deck is very unequal whereas there is an almost equal distribution of side.\n* Also, there are only 5 samples of deck 'T'.\n* Apart from that, there is no proper conclusion on how the deck affects Transported as few classes have almost equal distribution whereas some has huge difference.\n* But if you look at the Cabin's Side, you’ll notice that passenger with side 'S' has higher chance of getting transported than side 'P'.","metadata":{}},{"cell_type":"markdown","source":"\n## 7. Destination\n This is also a categorical feature. There are 3 unique values and this feature also contains some null values which we need to take care but first let's check its distribution and dependence on Transported.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(20,6))\nfig.suptitle('Destination Distribution', size = 20, weight='bold')\n\nsizes = list(df_train['Destination'].value_counts(sort=False))\n\nlabels = df_train['Destination'].dropna().unique()\ncolors = ['#099FFF', '#CC00FF', '#13CA91']\nexplode = (0.05,0.05,0.05) \n\nax[0].pie(sizes, colors=colors, explode=explode, startangle=90, labels=labels,\n autopct='%1.2f%%', pctdistance=0.6,textprops={'fontsize':12})\nsns.countplot(x='Destination', data=df_train, hue='Transported', ax=ax[1])\n\nprint(colored(\"Destination Distribution - \",'cyan'))\ndata = pd.DataFrame(df_train['Destination'].value_counts()).reset_index().rename(columns={'index': 'Destination', 'Destination':'Count'})\nprint(colored(data, 'cyan'))\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:25.362918Z","iopub.execute_input":"2022-07-06T07:41:25.363309Z","iopub.status.idle":"2022-07-06T07:41:25.698682Z","shell.execute_reply.started":"2022-07-06T07:41:25.363266Z","shell.execute_reply":"2022-07-06T07:41:25.697381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

INSIGHTS FROM THE GRAPH

\n\n* We can observe that most of the passenger's destination is TRAPPIST-1e.\n* Apart from that, the chances of getting transported is maximum for the passengers having destination as 55 Cancri e, but the distribution is very much equal for the other two destinations.","metadata":{}},{"cell_type":"markdown","source":"\n## 8. VIP\nThis is the last categorical feature with value either True or False and this feature also contains some null values which we need to take care but first let's check its distribution and dependence on Transported.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(20,6))\nfig.suptitle('VIP Distribution', size = 20, weight='bold')\n\nsizes = list(df_train['VIP'].value_counts(sort=False))\n\nlabels = df_train['VIP'].dropna().unique()\ncolors = ['#099FFF', '#CC00FF']\nexplode = (0.25,0.25) \n\nax[0].pie(sizes, colors=colors, explode=explode, startangle=90, labels=labels,\n autopct='%1.2f%%', pctdistance=0.6,textprops={'fontsize':12})\nsns.countplot(x='VIP', data=df_train, hue='Transported', ax=ax[1])\n\nprint(colored(\"VIP Distribution - \",'green'))\ndata = pd.DataFrame(df_train['VIP'].value_counts()).reset_index().rename(columns={'index': 'VIP', 'VIP':'Count'})\nprint(colored(data, 'green'))\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:25.700208Z","iopub.execute_input":"2022-07-06T07:41:25.700503Z","iopub.status.idle":"2022-07-06T07:41:26.049478Z","shell.execute_reply.started":"2022-07-06T07:41:25.700473Z","shell.execute_reply":"2022-07-06T07:41:26.048454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

INSIGHTS FROM THE GRAPH

\n\n* We can observe that VIP has a high false percentage. \n* Also, this feature doesn't look useful as the transported distribution is almost equal for both VIP and non VIP passengers.","metadata":{}},{"cell_type":"markdown","source":"\n## 9. Age\nThis is a continuos value feature with values ranging from 0 to 79. This feature also contains some null values which we need to take care but first let's check its distribution and dependence on Transported.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(20,6))\nfig.suptitle('Age Distribution', size = 20, weight='bold')\n\nsns.boxplot(x='Transported', y='Age', data=df_train, ax=ax[0])\nsns.histplot(x='Age', element=\"step\", kde=True, data=df_train, hue='Transported', ax=ax[1])\n\nprint(colored(\"Transported Passengers Age Distribution - \", 'magenta'))\nprint(colored(f\"Minimum Age - {df_train[df_train['Transported']==True]['Age'].describe()['min']}\", 'magenta'))\nprint(colored(f\"Maximum Age - {df_train[df_train['Transported']==True]['Age'].describe()['max']}\", 'magenta'))\nprint(colored(f\"Average Age - {df_train[df_train['Transported']==True]['Age'].describe()['mean']}\", 'magenta'))\n\nprint(colored(\"Non Transported Passengers Age Distribution - \", 'blue'))\nprint(colored(f\"Minimum Age - {df_train[df_train['Transported']==False]['Age'].describe()['min']}\", 'blue'))\nprint(colored(f\"Maximum Age - {df_train[df_train['Transported']==False]['Age'].describe()['max']}\", 'blue'))\nprint(colored(f\"Average Age - {df_train[df_train['Transported']==False]['Age'].describe()['mean']}\", 'blue'))\n\nmplcyberpunk.make_lines_glow()\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:26.051150Z","iopub.execute_input":"2022-07-06T07:41:26.051730Z","iopub.status.idle":"2022-07-06T07:41:26.539345Z","shell.execute_reply.started":"2022-07-06T07:41:26.051688Z","shell.execute_reply":"2022-07-06T07:41:26.538418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

INSIGHTS FROM THE GRAPH

\n\n* From the first look we can observe that the distribution of age is almost similar for both transported and non transported passengers. \n* But if we look more carefully, we can oberve that initially the age distribution of transported passengers is high indicating that passengers having age less than 10 have higher chances of getting transported but it is quite opposite for the passengers who are in their 20s.\n* But rest the age distribution is quite similar for both classes. ","metadata":{}},{"cell_type":"markdown","source":"\n## 10. RoomService, FoodCourt, ShoppingMall, Spa, VRDeck\nThese are the columns that contains the amount the passenger has billed at each of the Spaceship Titanic's many luxury amenities.These all are continuos feature and we will check their distribution and dependence on Transported.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(20,6))\nfig.suptitle('RoomService Distribution', size = 20, weight='bold')\n\nsns.boxplot(x='Transported', y='RoomService', data=df_train, ax=ax[0])\nsns.histplot(x='RoomService', element=\"step\", kde=True, data=df_train, hue='Transported', bins=100, ax=ax[1])\n\nprint(colored(f\"Percentage of Passengers with no RoomService Billing - {len(df_train[df_train['RoomService']==0.0])/len(df_train)*100:.2f}%\", 'red'))\nprint(colored(\"Transported Passengers RoomService Billing Distribution - \", 'magenta'))\nprint(colored(f\"Minimum RoomService Billing - {df_train[df_train['Transported']==True]['RoomService'].describe()['min']}\", 'magenta'))\nprint(colored(f\"Maximum RoomService Billing - {df_train[df_train['Transported']==True]['RoomService'].describe()['max']}\", 'magenta'))\nprint(colored(f\"Average RoomService Billing - {df_train[df_train['Transported']==True]['RoomService'].describe()['mean']}\", 'magenta'))\n\nprint(colored(\"Non Transported Passengers RoomService Billing Distribution - \", 'blue'))\nprint(colored(f\"Minimum RoomService Billing - {df_train[df_train['Transported']==False]['RoomService'].describe()['min']}\", 'blue'))\nprint(colored(f\"Maximum RoomService Billing - {df_train[df_train['Transported']==False]['RoomService'].describe()['max']}\", 'blue'))\nprint(colored(f\"Average RoomService Billing - {df_train[df_train['Transported']==False]['RoomService'].describe()['mean']}\", 'blue'))\n\nmplcyberpunk.make_lines_glow()\n\nplt.show()\n\nfig, ax = plt.subplots(1, 2, figsize=(20,6))\nfig.suptitle('FoodCourt Distribution', size = 20, weight='bold')\n\nsns.boxplot(x='Transported', y='FoodCourt', data=df_train, ax=ax[0])\nsns.histplot(x='FoodCourt', element=\"step\", kde=True, data=df_train, hue='Transported', bins=100, ax=ax[1])\n\nprint(colored(f\"Percentage of Passengers with no FoodCourt Billing - {len(df_train[df_train['FoodCourt']==0.0])/len(df_train)*100:.2f}%\", 'red'))\nprint(colored(\"Transported Passengers FoodCourt Billing Distribution - \", 'magenta'))\nprint(colored(f\"Minimum FoodCourt Billing - {df_train[df_train['Transported']==True]['FoodCourt'].describe()['min']}\", 'magenta'))\nprint(colored(f\"Maximum FoodCourt Billing - {df_train[df_train['Transported']==True]['FoodCourt'].describe()['max']}\", 'magenta'))\nprint(colored(f\"Average FoodCourt Billing - {df_train[df_train['Transported']==True]['FoodCourt'].describe()['mean']}\", 'magenta'))\n\nprint(colored(\"Non Transported Passengers FoodCourt Billing Distribution - \", 'blue'))\nprint(colored(f\"Minimum FoodCourt Billing - {df_train[df_train['Transported']==False]['FoodCourt'].describe()['min']}\", 'blue'))\nprint(colored(f\"Maximum FoodCourt Billing - {df_train[df_train['Transported']==False]['FoodCourt'].describe()['max']}\", 'blue'))\nprint(colored(f\"Average FoodCourt Billing - {df_train[df_train['Transported']==False]['FoodCourt'].describe()['mean']}\", 'blue'))\n\n\nmplcyberpunk.make_lines_glow()\n\nplt.show()\n\nfig, ax = plt.subplots(1, 2, figsize=(20,6))\nfig.suptitle('ShoppingMall Distribution', size = 20, weight='bold')\n\nsns.boxplot(x='Transported', y='ShoppingMall', data=df_train, ax=ax[0])\nsns.histplot(x='ShoppingMall', element=\"step\", kde=True, data=df_train, hue='Transported', bins=100, ax=ax[1])\n\nprint(colored(f\"Percentage of Passengers with no ShoppingMall Billing - {len(df_train[df_train['ShoppingMall']==0.0])/len(df_train)*100:.2f}%\", 'red'))\nprint(colored(\"Transported Passengers ShoppingMall Billing Distribution - \", 'magenta'))\nprint(colored(f\"Minimum ShoppingMall Billing - {df_train[df_train['Transported']==True]['ShoppingMall'].describe()['min']}\", 'magenta'))\nprint(colored(f\"Maximum ShoppingMall Billing - {df_train[df_train['Transported']==True]['ShoppingMall'].describe()['max']}\", 'magenta'))\nprint(colored(f\"Average ShoppingMall Billing - {df_train[df_train['Transported']==True]['ShoppingMall'].describe()['mean']}\", 'magenta'))\n\nprint(colored(\"Non Transported Passengers ShoppingMall Billing Distribution - \", 'blue'))\nprint(colored(f\"Minimum ShoppingMall Billing - {df_train[df_train['Transported']==False]['ShoppingMall'].describe()['min']}\", 'blue'))\nprint(colored(f\"Maximum ShoppingMall Billing - {df_train[df_train['Transported']==False]['ShoppingMall'].describe()['max']}\", 'blue'))\nprint(colored(f\"Average ShoppingMall Billing - {df_train[df_train['Transported']==False]['ShoppingMall'].describe()['mean']}\", 'blue'))\n\n\nmplcyberpunk.make_lines_glow()\n\nplt.show()\n\nfig, ax = plt.subplots(1, 2, figsize=(20,6))\nfig.suptitle('Spa Distribution', size = 20, weight='bold')\n\nsns.boxplot(x='Transported', y='Spa', data=df_train, ax=ax[0])\nsns.histplot(x='Spa', element=\"step\", kde=True, data=df_train, hue='Transported', bins=100, ax=ax[1])\n\nprint(colored(f\"Percentage of Passengers with no Spa Billing - {len(df_train[df_train['Spa']==0.0])/len(df_train)*100:.2f}%\", 'red'))\nprint(colored(\"Transported Passengers Spa Billing Distribution - \", 'magenta'))\nprint(colored(f\"Minimum Spa Billing - {df_train[df_train['Transported']==True]['Spa'].describe()['min']}\", 'magenta'))\nprint(colored(f\"Maximum Spa Billing - {df_train[df_train['Transported']==True]['Spa'].describe()['max']}\", 'magenta'))\nprint(colored(f\"Average Spa Billing - {df_train[df_train['Transported']==True]['Spa'].describe()['mean']}\", 'magenta'))\n\nprint(colored(\"Non Transported Passengers Spa Billing Distribution - \", 'blue'))\nprint(colored(f\"Minimum Spa Billing - {df_train[df_train['Transported']==False]['Spa'].describe()['min']}\", 'blue'))\nprint(colored(f\"Maximum Spa Billing - {df_train[df_train['Transported']==False]['Spa'].describe()['max']}\", 'blue'))\nprint(colored(f\"Average Spa Billing - {df_train[df_train['Transported']==False]['Spa'].describe()['mean']}\", 'blue'))\n\nmplcyberpunk.make_lines_glow()\n\nplt.show()\n\nfig, ax = plt.subplots(1, 2, figsize=(20,6))\nfig.suptitle('VRDeck Distribution', size = 20, weight='bold')\n\nsns.boxplot(x='Transported', y='VRDeck', data=df_train, ax=ax[0])\nsns.histplot(x='VRDeck', element=\"step\", kde=True, data=df_train, hue='Transported', bins=100, ax=ax[1])\n\nprint(colored(f\"Percentage of Passengers with no VRDeck Billing - {len(df_train[df_train['VRDeck']==0.0])/len(df_train)*100:.2f}%\", 'red'))\nprint(colored(\"Transported Passengers VRDeck Billing Distribution - \", 'magenta'))\nprint(colored(f\"Minimum VRDeck Billing - {df_train[df_train['Transported']==True]['VRDeck'].describe()['min']}\", 'magenta'))\nprint(colored(f\"Maximum VRDeck Billing - {df_train[df_train['Transported']==True]['VRDeck'].describe()['max']}\", 'magenta'))\nprint(colored(f\"Average VRDeck Billing - {df_train[df_train['Transported']==True]['VRDeck'].describe()['mean']}\", 'magenta'))\n\nprint(colored(\"Non Transported Passengers VRDeck Billing Distribution - \", 'blue'))\nprint(colored(f\"Minimum VRDeck Billing - {df_train[df_train['Transported']==False]['VRDeck'].describe()['min']}\", 'blue'))\nprint(colored(f\"Maximum VRDeck Billing - {df_train[df_train['Transported']==False]['VRDeck'].describe()['max']}\", 'blue'))\nprint(colored(f\"Average VRDeck Billing - {df_train[df_train['Transported']==False]['VRDeck'].describe()['mean']}\", 'blue'))\n\nmplcyberpunk.make_lines_glow()\n\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-06T07:41:26.540982Z","iopub.execute_input":"2022-07-06T07:41:26.541314Z","iopub.status.idle":"2022-07-06T07:41:28.990806Z","shell.execute_reply.started":"2022-07-06T07:41:26.541270Z","shell.execute_reply":"2022-07-06T07:41:28.989579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

INSIGHTS FROM THE GRAPH

\n\n* We can clearly observe that all the distributions are centered towards zero. \n* There are more than 60% passengers in each distribution who have not paid for that service.\n* Also, there are few cases with very high billings (looks like an outlier)\n* It looks like using these features directly for modelling won't help and we might need to create new features from these to have better performance.","metadata":{}},{"cell_type":"markdown","source":"### We have observed the distribution of these continuos features and looked how these affect the Transported. But still we need more clarity on these features. So next we will check their correlation with some other features like Age and VIP and can get to know how much they are correlated and affect Transported.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(3, 2, figsize=(20, 15))\nfig.suptitle('Continuous Features VS Age', size = 20, weight='bold')\nfig.delaxes(ax[2][1])\n\ndf_temp = df_train.iloc[:, 5:12]\ncolumns = df_temp.columns[2:]\nfor i, col in enumerate(columns):\n sns.scatterplot(x='Age', y=col, hue='Transported', data=df_train, ax=fig.axes[i], palette='turbo')\n fig.axes[i].set_title(f'{col} VS Age', weight='bold')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:28.992585Z","iopub.execute_input":"2022-07-06T07:41:28.993358Z","iopub.status.idle":"2022-07-06T07:41:31.379870Z","shell.execute_reply.started":"2022-07-06T07:41:28.993290Z","shell.execute_reply":"2022-07-06T07:41:31.377697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

INSIGHTS FROM THE GRAPH

\n\n* As expected, the expendicture of each feature is almost zero for passengers with age less than 10.\n* The distribution of the transported and non-transported is quite similar with respect to Age in the case FoodCourt and ShoppingMall.\n* Whereas we can observe that non-transported passengers have spent more in the case of RoomService, Spa and VRDeck.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(3, 2, figsize=(20, 15))\nfig.suptitle('Continuous Features VS VIP', size = 20, weight='bold')\nfig.delaxes(ax[2][1])\n\nfor i, col in enumerate(columns):\n sns.stripplot(x=\"VIP\", y=col, hue='Transported', data=df_train, dodge=True, ax=fig.axes[i], palette='winter')\n fig.axes[i].set_title(f'{col} VS VIP', weight='bold')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:31.381353Z","iopub.execute_input":"2022-07-06T07:41:31.382243Z","iopub.status.idle":"2022-07-06T07:41:33.040791Z","shell.execute_reply.started":"2022-07-06T07:41:31.382184Z","shell.execute_reply":"2022-07-06T07:41:33.039436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

INSIGHTS FROM THE GRAPH

\n\n* Although, it is expected that VIP passengers must have spent more than normal passengers, but as the count of VIP passengers is very less, we can't observe the same from the above plots.\n* Apart from that the distribution looks quite similar for each feature for both VIP and non-VIP passengers.","metadata":{}},{"cell_type":"markdown","source":"### As I stated before, we can create new features from these 5 continuous features to get more valuable dataset. \n### New Features :- \n1) Total Expenses - Sum of all the 5 expenses.
\n2) NoSpent - Whether the passenger has spent anything or not.","metadata":{}},{"cell_type":"code","source":"df_train['Total_Expenses'] = df_train[df_temp.columns[2:]].sum(axis=1)\ndf_train['NoSpent'] = df_train['Total_Expenses']==0\n\nfig, ax = plt.subplots(1, 2, figsize=(20, 6))\nfig.suptitle('Total Expenses Distribution', size = 20, weight='bold')\n\nsns.boxplot(x='Transported', y='Total_Expenses', data=df_train, ax=ax[0], palette='turbo')\nsns.histplot(x='Total_Expenses', element=\"step\", kde=True, data=df_train, hue='Transported', bins=100, ax=ax[1], palette='turbo')\n\nprint(colored(\"Total_Expenses Distribution - \", 'cyan'))\nprint(colored(f\"Minimum Total_Expenses - {df_train['Total_Expenses'].describe()['min']}\", 'cyan'))\nprint(colored(f\"Maximum Total_Expenses - {df_train['Total_Expenses'].describe()['max']}\", 'cyan'))\nprint(colored(f\"Average Total_Expenses - {df_train['Total_Expenses'].describe()['mean']}\", 'cyan'))\n\nplt.show()\n\nfig, ax = plt.subplots(1, 2, figsize=(20,6))\nfig.suptitle('NoSpent Distribution', size = 20, weight='bold')\n\nsizes = list(df_train['NoSpent'].value_counts(sort=False))\n\nlabels = df_train['NoSpent'].dropna().unique()\ncolors = ['#13CA91', '#e5ab09']\nexplode = (0.0,0.05) \n\nax[0].pie(sizes, colors=colors, explode=explode, startangle=90, labels=labels,\n autopct='%1.2f%%', pctdistance=0.6,textprops={'fontsize':12})\nsns.countplot(x='NoSpent', data=df_train, hue='Transported', ax=ax[1], palette='turbo')\n\nprint(colored(\"NoSpent Distribution - \",'cyan'))\ndata = pd.DataFrame(df_train['NoSpent'].value_counts()).reset_index().rename(columns={'index': 'NoSpent', 'NoSpent':'Count'})\nprint(colored(data, 'cyan'))\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:33.042242Z","iopub.execute_input":"2022-07-06T07:41:33.042540Z","iopub.status.idle":"2022-07-06T07:41:33.799096Z","shell.execute_reply.started":"2022-07-06T07:41:33.042491Z","shell.execute_reply":"2022-07-06T07:41:33.797922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

INSIGHTS FROM THE GRAPH

\n\n* We can observe that the total expenses is still mostly near to zero and the distribution is almost same for both transported and non-transported passengers.\n* On the other hand we can observe a direct relationship with the NoSpent and Transported feature such that the person who hasn't spent anything has higher chance of getting transported.","metadata":{}},{"cell_type":"markdown","source":"\n#

DATA PRE-PROCESSING

\nAfter exploring the data, now it's time to process the data so that it can be used to train the model.
\nI have already performed all the feature engineering in the EDA, so it is not included in the processing.","metadata":{}},{"cell_type":"code","source":"# Let's have a quick look at the data\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:33.800916Z","iopub.execute_input":"2022-07-06T07:41:33.801217Z","iopub.status.idle":"2022-07-06T07:41:33.828403Z","shell.execute_reply.started":"2022-07-06T07:41:33.801184Z","shell.execute_reply":"2022-07-06T07:41:33.827239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## Handling Missing Values\nAs we already know, there are missing values in almost all the columns. So either we can drop all the rows with the null values or we can fill these null values.
\nThere aren't many missing values, but it's a good idea to enter these values. This gives you more data and allows you to create better models. There are multiple ways to fill these null values depending upon each column. But I will follow an easier approach by filling the null values by either it's median or mode depending upon the column.","metadata":{}},{"cell_type":"code","source":"print(colored(f\"Number of missing values before - {df_train.isna().sum().sum()}\", 'red'))\n\nfor col in df_train.columns:\n if col == 'Age':\n df_train[col].fillna(df_train[col].median(), inplace=True)\n else:\n df_train[col].fillna(df_train[col].mode()[0], inplace=True)\n \nprint(colored(f\"Number of missing values after - {df_train.isna().sum().sum()}\", 'blue'))","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:33.829719Z","iopub.execute_input":"2022-07-06T07:41:33.830264Z","iopub.status.idle":"2022-07-06T07:41:33.889624Z","shell.execute_reply.started":"2022-07-06T07:41:33.830217Z","shell.execute_reply":"2022-07-06T07:41:33.888194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## Dropping unwanted columns.","metadata":{}},{"cell_type":"code","source":"df_train.drop(['PassengerId', 'Cabin', 'Group', 'Name'], axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:33.891136Z","iopub.execute_input":"2022-07-06T07:41:33.891803Z","iopub.status.idle":"2022-07-06T07:41:33.903646Z","shell.execute_reply.started":"2022-07-06T07:41:33.891748Z","shell.execute_reply":"2022-07-06T07:41:33.902076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## Encoding\nIn machine learning models, all input and output variables must be numeric. This means that if our data contains values in string or boolean format, they must be converted into numerical format before training the model.","metadata":{}},{"cell_type":"code","source":"for col in df_train.columns[df_train.dtypes == object]:\n encoder = LabelEncoder()\n df_train[col] = encoder.fit_transform(df_train[col])\n \nfor col in df_train.columns[df_train.dtypes == bool]:\n df_train[col] = df_train[col].astype('int')","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:33.905087Z","iopub.execute_input":"2022-07-06T07:41:33.906201Z","iopub.status.idle":"2022-07-06T07:41:33.931143Z","shell.execute_reply.started":"2022-07-06T07:41:33.906059Z","shell.execute_reply":"2022-07-06T07:41:33.930056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## Splitting the data\nLastly, we assign our input and the target variables and then split the data into train and validation set.","metadata":{}},{"cell_type":"code","source":"X = df_train.drop('Transported', axis=1)\ny = df_train['Transported']\nX_train, X_valid, y_train, y_valid = train_test_split(X, y, stratify=y, test_size=0.2, random_state=5)\n\nprint(colored(f\"Number of rows in training set - {len(X_train)}\", 'cyan'))\nprint(colored(f\"Number of rows in validation set - {len(X_valid)}\", 'magenta'))","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:33.935262Z","iopub.execute_input":"2022-07-06T07:41:33.937033Z","iopub.status.idle":"2022-07-06T07:41:33.958664Z","shell.execute_reply.started":"2022-07-06T07:41:33.936975Z","shell.execute_reply":"2022-07-06T07:41:33.957416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#

ML MODELS

\nIt is now time to develop some models following all of the experiments and preprocessing. Let's begin by developing a variety of primary machine learning models.","metadata":{}},{"cell_type":"code","source":"acc_plot = {} # For plotting purpose\npreds = [] # For ensembling","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:33.965173Z","iopub.execute_input":"2022-07-06T07:41:33.965483Z","iopub.status.idle":"2022-07-06T07:41:33.971285Z","shell.execute_reply.started":"2022-07-06T07:41:33.965450Z","shell.execute_reply":"2022-07-06T07:41:33.969930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### Logistic Regression","metadata":{}},{"cell_type":"code","source":"model_lor = LogisticRegression(random_state=5).fit(X_train, y_train)\ny_pred = model_lor.predict(X_valid)\nacc = accuracy_score(y_valid, y_pred)\nacc_plot['LogisticRegression'] = acc\nprint(f\"Model Name: LogisticRegression ====>>> Validation Accuracy: {acc}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:33.973715Z","iopub.execute_input":"2022-07-06T07:41:33.975318Z","iopub.status.idle":"2022-07-06T07:41:34.145122Z","shell.execute_reply.started":"2022-07-06T07:41:33.975249Z","shell.execute_reply":"2022-07-06T07:41:34.144010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### Support Vector Classifier","metadata":{}},{"cell_type":"code","source":"model_svc = SVC(random_state=5).fit(X_train, y_train)\ny_pred = model_svc.predict(X_valid)\nacc = accuracy_score(y_valid, y_pred)\nacc_plot['SVC'] = acc\nprint(f\"Model Name: SVC ====>>> Validation Accuracy: {acc}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:34.147160Z","iopub.execute_input":"2022-07-06T07:41:34.147967Z","iopub.status.idle":"2022-07-06T07:41:36.550683Z","shell.execute_reply.started":"2022-07-06T07:41:34.147910Z","shell.execute_reply":"2022-07-06T07:41:36.549674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### Naive Bayes","metadata":{}},{"cell_type":"code","source":"model_nb = GaussianNB().fit(X_train, y_train)\ny_pred = model_nb.predict(X_valid)\nacc = accuracy_score(y_valid, y_pred)\nacc_plot['NaiveBayes'] = acc\nprint(f\"Model Name: NaiveBayes ====>>> Validation Accuracy: {acc}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:36.551964Z","iopub.execute_input":"2022-07-06T07:41:36.552287Z","iopub.status.idle":"2022-07-06T07:41:36.567109Z","shell.execute_reply.started":"2022-07-06T07:41:36.552257Z","shell.execute_reply":"2022-07-06T07:41:36.565873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### Random Forest Classifier","metadata":{}},{"cell_type":"code","source":"model_rfc = RandomForestClassifier(random_state=5).fit(X_train, y_train)\ny_pred = model_rfc.predict(X_valid)\nacc = accuracy_score(y_valid, y_pred)\nacc_plot['RFClassifier'] = acc\nprint(f\"Model Name: RFClassifier ====>>> Validation Accuracy: {acc}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:36.568819Z","iopub.execute_input":"2022-07-06T07:41:36.569760Z","iopub.status.idle":"2022-07-06T07:41:37.495654Z","shell.execute_reply.started":"2022-07-06T07:41:36.569707Z","shell.execute_reply":"2022-07-06T07:41:37.494553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### Gradient Boosting Classifier","metadata":{}},{"cell_type":"code","source":"model_gbc = GradientBoostingClassifier(random_state=5).fit(X_train, y_train)\ny_pred = model_gbc.predict(X_valid)\nacc = accuracy_score(y_valid, y_pred)\nacc_plot['GBTClassifier'] = acc\nprint(f\"Model Name: GBTClassifier ====>>> Validation Accuracy: {acc}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:37.496875Z","iopub.execute_input":"2022-07-06T07:41:37.497134Z","iopub.status.idle":"2022-07-06T07:41:38.544017Z","shell.execute_reply.started":"2022-07-06T07:41:37.497106Z","shell.execute_reply":"2022-07-06T07:41:38.542888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### XGBoost Classifier","metadata":{}},{"cell_type":"code","source":"model_xgb = XGBClassifier(eval_metric='logloss', random_state=5).fit(X_train, y_train)\ny_pred = model_xgb.predict(X_valid)\nacc = accuracy_score(y_valid, y_pred)\nacc_plot['XGBoost'] = acc\nprint(f\"Model Name: XGBoost Classifier ====>>> Validation Accuracy: {acc}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:38.545194Z","iopub.execute_input":"2022-07-06T07:41:38.545999Z","iopub.status.idle":"2022-07-06T07:41:39.428612Z","shell.execute_reply.started":"2022-07-06T07:41:38.545963Z","shell.execute_reply":"2022-07-06T07:41:39.427279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 6))\nsns.barplot(list(acc_plot.keys()), list(acc_plot.values()))\nplt.xlabel(\"ML Models\")\nplt.ylabel(\"Validation Accuracy\")\nplt.title(\"Comparison Graph\", fontsize=20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:39.430098Z","iopub.execute_input":"2022-07-06T07:41:39.430387Z","iopub.status.idle":"2022-07-06T07:41:39.663350Z","shell.execute_reply.started":"2022-07-06T07:41:39.430356Z","shell.execute_reply":"2022-07-06T07:41:39.662617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### SHAP (SHapley Additive exPlanations)","metadata":{}},{"cell_type":"markdown","source":"Let's now use SHAP to depict the summary plot to understand which feature was most important to the model. The shap value is a measure of how well that single data point was able to anticipate the results. The data point makes relatively little contribution to predictions if the shap value is extremely close to zero. Depending on whether the shap value is very positive or strongly negative, we may claim that the data point significantly influences predictions.","metadata":{}},{"cell_type":"code","source":"# Initialize object that can calculate shap values\nexplainer = shap.TreeExplainer(model_xgb)\n\n# Calculate Shap values\nshap_values = explainer.shap_values(X_valid)","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:39.664674Z","iopub.execute_input":"2022-07-06T07:41:39.665656Z","iopub.status.idle":"2022-07-06T07:41:40.279231Z","shell.execute_reply.started":"2022-07-06T07:41:39.665608Z","shell.execute_reply":"2022-07-06T07:41:40.278088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"shap.summary_plot(shap_values, X_valid, plot_size=(20, 15), axis_color='#ff0090')","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:40.281265Z","iopub.execute_input":"2022-07-06T07:41:40.282038Z","iopub.status.idle":"2022-07-06T07:41:41.259414Z","shell.execute_reply.started":"2022-07-06T07:41:40.281985Z","shell.execute_reply":"2022-07-06T07:41:41.258327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"shap.summary_plot(shap_values, X_valid, plot_type='bar', plot_size=(20, 15), axis_color='#ff0090')","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:41.262732Z","iopub.execute_input":"2022-07-06T07:41:41.263057Z","iopub.status.idle":"2022-07-06T07:41:41.636955Z","shell.execute_reply.started":"2022-07-06T07:41:41.263022Z","shell.execute_reply":"2022-07-06T07:41:41.635805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### Ensembling\nIn order to deliver the most precise predictions for a specific situation, we create machine learning models. However, a single model might not produce the most accurate results and might make certain mistakes. We may mix the findings of several models to obtain better outcomes in order to decrease these mistakes and enhance the predictions. This is referred to as ensemble learning. The aggregate findings balance out the noise and can be more accurate than any one approach. In this case, I've shown this by ensembling three of the best performing models and evaluated their valid set results.","metadata":{}},{"cell_type":"code","source":"y_pred = model_svc.predict(X_valid)\npreds.append(y_pred)\n\ny_pred = model_gbc.predict(X_valid)\npreds.append(y_pred)\n\ny_pred = model_xgb.predict(X_valid)\npreds.append(y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:41.638917Z","iopub.execute_input":"2022-07-06T07:41:41.639799Z","iopub.status.idle":"2022-07-06T07:41:42.169289Z","shell.execute_reply.started":"2022-07-06T07:41:41.639746Z","shell.execute_reply":"2022-07-06T07:41:42.168569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_preds = np.array(preds).mean(axis = 0)\nfinal_preds = np.where(final_preds > 0.5, 1, 0)","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:42.170434Z","iopub.execute_input":"2022-07-06T07:41:42.170916Z","iopub.status.idle":"2022-07-06T07:41:42.178514Z","shell.execute_reply.started":"2022-07-06T07:41:42.170870Z","shell.execute_reply":"2022-07-06T07:41:42.177396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"acc = accuracy_score(y_valid, final_preds)\nprint(f\"Ensembling ====>>> Validation Accuracy: {acc}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-06T07:41:42.179717Z","iopub.execute_input":"2022-07-06T07:41:42.180300Z","iopub.status.idle":"2022-07-06T07:41:42.194228Z","shell.execute_reply.started":"2022-07-06T07:41:42.180249Z","shell.execute_reply":"2022-07-06T07:41:42.193052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

⚠ WORK IN PROGRESS ⚠

\n

Please consider upvoting the kernel if you found it useful.

\n
","metadata":{}}]}