{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-18T12:39:43.691022Z","iopub.execute_input":"2022-03-18T12:39:43.691424Z","iopub.status.idle":"2022-03-18T12:39:43.709977Z","shell.execute_reply.started":"2022-03-18T12:39:43.691385Z","shell.execute_reply":"2022-03-18T12:39:43.708876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
","metadata":{}},{"cell_type":"markdown","source":"\n## 1. Feature Observation (Visualization)\n* Dividing the columns according to their data types \n* Using Seaborn for visualization\n\n## 2. Feature Engineering\n* Extracting the Cabin column\n* Perform one hot encoding for necessary features\n* Getting features ready for our model (train + validation split)\n\n## 3. Training + Validation\n* Random Forest Model Implementation\n* XGBoost Model Implementation\n* Hyperparameter Tuning through Random Search\n\n## 4. Machine Learning Model Report (Random Forest)\n* Confusion Matrix\n* Feature Importances\n\n## 5. Summary\n* Making our submission file\n* Possible areas of improvement\n* acknowledgements\n\n## 6. PyTorch","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv('../input/spaceship-titanic/train.csv')\ntest_df = pd.read_csv('../input/spaceship-titanic/test.csv')\nsample_df = pd.read_csv('../input/spaceship-titanic/sample_submission.csv')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-18T12:39:44.280287Z","iopub.execute_input":"2022-03-18T12:39:44.281302Z","iopub.status.idle":"2022-03-18T12:39:44.359656Z","shell.execute_reply.started":"2022-03-18T12:39:44.281249Z","shell.execute_reply":"2022-03-18T12:39:44.358695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Feature Observation","metadata":{"execution":{"iopub.status.busy":"2022-02-27T04:58:55.25585Z","iopub.execute_input":"2022-02-27T04:58:55.256143Z","iopub.status.idle":"2022-02-27T04:58:55.294303Z","shell.execute_reply.started":"2022-02-27T04:58:55.256115Z","shell.execute_reply":"2022-02-27T04:58:55.293568Z"}}},{"cell_type":"code","source":"train_df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-18T12:39:44.361277Z","iopub.execute_input":"2022-03-18T12:39:44.361533Z","iopub.status.idle":"2022-03-18T12:39:44.400332Z","shell.execute_reply.started":"2022-03-18T12:39:44.361502Z","shell.execute_reply":"2022-03-18T12:39:44.399226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Let us first divide the features into 'objects', 'boolean', 'float' datatypes","metadata":{}},{"cell_type":"code","source":"objects = train_df.select_dtypes(include='object')\nboolean = train_df.select_dtypes(include='bool')\nnumbers = train_df.select_dtypes(include='float')","metadata":{"execution":{"iopub.status.busy":"2022-03-18T12:39:44.405459Z","iopub.execute_input":"2022-03-18T12:39:44.406342Z","iopub.status.idle":"2022-03-18T12:39:44.425443Z","shell.execute_reply.started":"2022-03-18T12:39:44.406285Z","shell.execute_reply":"2022-03-18T12:39:44.424182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.1 Numerical Feature Observation","metadata":{"execution":{"iopub.status.busy":"2022-02-27T05:08:20.3948Z","iopub.execute_input":"2022-02-27T05:08:20.395403Z","iopub.status.idle":"2022-02-27T05:08:20.414595Z","shell.execute_reply.started":"2022-02-27T05:08:20.395344Z","shell.execute_reply":"2022-02-27T05:08:20.413926Z"}}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\nplt.figure(figsize=(20,20))\n \n# Draw the plot\nfor count, feature in enumerate(numbers.columns.tolist()):\n ax = plt.subplot(3, 2, count+1) \n ax.set_title(\"{}\".format(feature), size = 30)\n ax.hist(numbers[feature],color = 'pink', edgecolor = 'black', bins = 50)\n \nplt.suptitle(\"Distribution of Numerical Features\", fontsize = 50)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-18T12:39:44.49019Z","iopub.execute_input":"2022-03-18T12:39:44.4906Z","iopub.status.idle":"2022-03-18T12:40:06.99667Z","shell.execute_reply.started":"2022-03-18T12:39:44.490549Z","shell.execute_reply":"2022-03-18T12:40:06.995753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> #### Seems like majority of the people onboard did not spend much at each of the Spaceship Titanic's many luxury amenities. \n> #### For the age wise, it seems like majority of our passengers were between the ages of 20 to 30 while there was a pretty large number of infants onboard too","metadata":{}},{"cell_type":"markdown","source":"## 1.2 Object Feature Observation","metadata":{}},{"cell_type":"code","source":"objects","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-18T12:40:06.998704Z","iopub.execute_input":"2022-03-18T12:40:06.999784Z","iopub.status.idle":"2022-03-18T12:40:07.01985Z","shell.execute_reply.started":"2022-03-18T12:40:06.99973Z","shell.execute_reply":"2022-03-18T12:40:07.018743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in objects.columns.tolist()[1:-1]:\n print(objects[i].describe(),\"\\n\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-18T12:40:07.021548Z","iopub.execute_input":"2022-03-18T12:40:07.021875Z","iopub.status.idle":"2022-03-18T12:40:07.058898Z","shell.execute_reply.started":"2022-03-18T12:40:07.021835Z","shell.execute_reply":"2022-03-18T12:40:07.057902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_theme(style=\"darkgrid\")\nfig, ax =plt.subplots(2, 2,figsize=(15, 15))\nsns.countplot(x = 'HomePlanet', data = objects, ax=ax[0][0])\nsns.countplot(x = 'CryoSleep', data=objects, ax=ax[0][1])\nsns.countplot(x = 'Destination', data = objects, ax=ax[1][0])\nsns.countplot(x = 'VIP', data = objects, ax=ax[1][1])\nfig.suptitle('Comparisons of different Categorical Features', fontsize=20)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-18T12:40:07.061571Z","iopub.execute_input":"2022-03-18T12:40:07.061834Z","iopub.status.idle":"2022-03-18T12:40:07.727369Z","shell.execute_reply.started":"2022-03-18T12:40:07.061801Z","shell.execute_reply":"2022-03-18T12:40:07.726229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> #### Seems like most of the people on board were from Earth and a vast majority of them were headed for the TRAPPIST-1e. \n> #### It can also be seen that almost all of them were not VIP and about 5500 of them did not opt for cryosleep for the journey\n> #### For Cabin, we will leave that to the feature engineering section to accomplish and we wound visualize the outcomes then","metadata":{}},{"cell_type":"markdown","source":"# 2. Feature Engineering","metadata":{}},{"cell_type":"markdown","source":"#### I will first engineer the **Cabin** column which seems pretty important\n#### The column contains these following information\n* deck\n* num\n* side\n\n#### If we can extract out those informations to three different columns, our model would be able to extract weights from those individual columns for a possibly more accurate prediction","metadata":{}},{"cell_type":"code","source":"objects['Cabin']","metadata":{"execution":{"iopub.status.busy":"2022-03-18T12:40:07.728542Z","iopub.execute_input":"2022-03-18T12:40:07.728746Z","iopub.status.idle":"2022-03-18T12:40:07.738446Z","shell.execute_reply.started":"2022-03-18T12:40:07.72872Z","shell.execute_reply":"2022-03-18T12:40:07.737215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"objects['Cabin'].fillna(0, inplace=True)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-18T12:40:07.740355Z","iopub.execute_input":"2022-03-18T12:40:07.740677Z","iopub.status.idle":"2022-03-18T12:40:07.750978Z","shell.execute_reply.started":"2022-03-18T12:40:07.740631Z","shell.execute_reply":"2022-03-18T12:40:07.749247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import math\ndeck = []\nnum = []\nside = []\n\nfor i in range(len(objects['Cabin'])):\n if objects['Cabin'].iloc[i] == 0:\n result = [math.nan, math.nan, math.nan]\n else:\n result = objects['Cabin'].iloc[i].split('/')\n deck.append(result[0])\n num.append(result[1])\n side.append(result[2])\n \nobjects['Deck'] = deck\nobjects['Num'] = num\nobjects['Side'] = side\n\nobjects","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-18T12:40:07.752691Z","iopub.execute_input":"2022-03-18T12:40:07.752987Z","iopub.status.idle":"2022-03-18T12:40:08.000395Z","shell.execute_reply.started":"2022-03-18T12:40:07.752946Z","shell.execute_reply":"2022-03-18T12:40:07.999393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_theme(style=\"darkgrid\")\nfig, ax =plt.subplots(1,2,figsize=(15, 15))\nsns.countplot(x = 'Deck', data = objects, ax=ax[0])\nsns.countplot(x = 'Side', data=objects, ax=ax[1])\n\nfig.suptitle('Comparisons of elements of Cabin feature', fontsize=20)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-18T12:40:08.002127Z","iopub.execute_input":"2022-03-18T12:40:08.002467Z","iopub.status.idle":"2022-03-18T12:40:08.42626Z","shell.execute_reply.started":"2022-03-18T12:40:08.002427Z","shell.execute_reply":"2022-03-18T12:40:08.425096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Changing datatype of 'Num' into float for visualization as shown below\nobjects = objects.astype({\"Num\": float})","metadata":{"execution":{"iopub.status.busy":"2022-03-18T12:40:08.427655Z","iopub.execute_input":"2022-03-18T12:40:08.427865Z","iopub.status.idle":"2022-03-18T12:40:08.437989Z","shell.execute_reply.started":"2022-03-18T12:40:08.427838Z","shell.execute_reply":"2022-03-18T12:40:08.436999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.displot(objects, x=\"Num\",kind=\"kde\", height=10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-18T12:40:08.441875Z","iopub.execute_input":"2022-03-18T12:40:08.443347Z","iopub.status.idle":"2022-03-18T12:40:08.824856Z","shell.execute_reply.started":"2022-03-18T12:40:08.443296Z","shell.execute_reply":"2022-03-18T12:40:08.824038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> #### It seems like for the Decks, majority of the people were in F and G while the number on the portside was similar to the ones on the starboard side. \n> #### For the Num column, seems like the majority is around 100 to 200. This Cabin number at first might seem like it doesn't pose much information but since we do not know the layout of the spaceship, leaving the learning to the model might help the model predict the room number in which the occupants had a higher rate of survival possibility, probably due to the distance to the exit","metadata":{}},{"cell_type":"markdown","source":"#### Now let us ensure our feature columns are in one single dataframe and split them into 70% training data and 30% validation data","metadata":{}},{"cell_type":"markdown","source":"## 2.1 Data Preparation","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv('../input/spaceship-titanic/train.csv')\ntest_df = pd.read_csv('../input/spaceship-titanic/test.csv')\n\ntrain_df['label'] = 'train'\ntest_df['label'] = 'test'\n\n# Concat\nconcat_df = pd.concat([train_df , test_df])","metadata":{"execution":{"iopub.status.busy":"2022-03-18T12:40:08.8262Z","iopub.execute_input":"2022-03-18T12:40:08.826439Z","iopub.status.idle":"2022-03-18T12:40:08.886341Z","shell.execute_reply.started":"2022-03-18T12:40:08.826409Z","shell.execute_reply":"2022-03-18T12:40:08.885334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Extracting out the 'Cabin' column\nconcat_df['Cabin'].fillna(0, inplace=True)\n\ndeck = []\nnum = []\nside = []\n\nfor i in range(len(concat_df['Cabin'])):\n if concat_df['Cabin'].iloc[i] == 0:\n result = [math.nan, math.nan, math.nan]\n else:\n result = concat_df['Cabin'].iloc[i].split('/')\n deck.append(result[0])\n num.append(result[1])\n side.append(result[2])\n \nconcat_df['Deck'] = deck\nconcat_df['Num'] = num\nconcat_df['Side'] = side","metadata":{"execution":{"iopub.status.busy":"2022-03-18T12:40:08.888354Z","iopub.execute_input":"2022-03-18T12:40:08.888712Z","iopub.status.idle":"2022-03-18T12:40:09.225827Z","shell.execute_reply.started":"2022-03-18T12:40:08.88867Z","shell.execute_reply":"2022-03-18T12:40:09.224861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"concat_df.drop(['PassengerId', 'Name', 'Cabin'], axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-03-18T12:40:09.227407Z","iopub.execute_input":"2022-03-18T12:40:09.227659Z","iopub.status.idle":"2022-03-18T12:40:09.239223Z","shell.execute_reply.started":"2022-03-18T12:40:09.227628Z","shell.execute_reply":"2022-03-18T12:40:09.238196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> #### Let's proceed on with one-hot encoding for our categorical features","metadata":{}},{"cell_type":"code","source":"onehotencoding = pd.get_dummies(concat_df.select_dtypes(include='object').drop(['Transported', 'label'], axis=1))\nconcat_df.drop(concat_df.select_dtypes(include='object').drop(['Transported', 'label'], axis=1), axis=1, inplace=True)\nconcat_set = pd.concat([concat_df, onehotencoding], axis=1)\n\n# Split your data\ntrain_df = concat_set[concat_set['label'] == 'train']\ntest_df = concat_set[concat_set['label'] == 'test']\n\n# Drop your labels\ntrain_df = train_df.drop('label', axis=1)\ntest_df = test_df.drop('label', axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-03-18T12:40:09.240978Z","iopub.execute_input":"2022-03-18T12:40:09.241356Z","iopub.status.idle":"2022-03-18T12:40:09.686164Z","shell.execute_reply.started":"2022-03-18T12:40:09.241311Z","shell.execute_reply":"2022-03-18T12:40:09.685424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> #### One hot encoding is done\n> #### We won't carry out any feature scaling and normalization as we would be using Random Forest model for this classification problem\n> #### However, we would need to fill in the missing (NaN) values for numerical columns for this model to work","metadata":{}},{"cell_type":"code","source":"train_df.isna().sum().head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-18T12:40:09.687162Z","iopub.execute_input":"2022-03-18T12:40:09.687389Z","iopub.status.idle":"2022-03-18T12:40:09.728633Z","shell.execute_reply.started":"2022-03-18T12:40:09.687361Z","shell.execute_reply":"2022-03-18T12:40:09.728086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### For 'Age', we would use MissForest Imputation.\n\n#### For the rest of the missing numerical columns, I will just impute it with the mode value as the data is very skewed towards the low values. ","metadata":{}},{"cell_type":"code","source":"pip install missingpy","metadata":{"execution":{"iopub.status.busy":"2022-03-18T12:40:09.729618Z","iopub.execute_input":"2022-03-18T12:40:09.730257Z","iopub.status.idle":"2022-03-18T12:40:18.620995Z","shell.execute_reply.started":"2022-03-18T12:40:09.730219Z","shell.execute_reply":"2022-03-18T12:40:18.620054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from missingpy import MissForest\n\n# Impute\nimputer = MissForest()\ndata_imputed = imputer.fit_transform(train_df.iloc[: , :6])\ndata_imputed = pd.DataFrame(data=data_imputed, columns=train_df.iloc[: , :6].columns)\n\ndata_imputed","metadata":{"execution":{"iopub.status.busy":"2022-03-18T12:40:18.62303Z","iopub.execute_input":"2022-03-18T12:40:18.6234Z","iopub.status.idle":"2022-03-18T12:40:47.574921Z","shell.execute_reply.started":"2022-03-18T12:40:18.623354Z","shell.execute_reply":"2022-03-18T12:40:47.573881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.drop(['Age', 'RoomService', 'FoodCourt', 'ShoppingMall', 'Spa', 'VRDeck'], axis = 1, inplace=True)\ntrain_df = pd.concat([train_df, data_imputed], axis=1)\ntrain_df.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-03-18T12:40:47.576718Z","iopub.execute_input":"2022-03-18T12:40:47.577029Z","iopub.status.idle":"2022-03-18T12:40:47.645653Z","shell.execute_reply.started":"2022-03-18T12:40:47.576988Z","shell.execute_reply":"2022-03-18T12:40:47.644596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.Transported = train_df.Transported.astype(bool)","metadata":{"execution":{"iopub.status.busy":"2022-03-18T12:40:47.6475Z","iopub.execute_input":"2022-03-18T12:40:47.647798Z","iopub.status.idle":"2022-03-18T12:40:47.654368Z","shell.execute_reply.started":"2022-03-18T12:40:47.64776Z","shell.execute_reply":"2022-03-18T12:40:47.653218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### All the work is done for imputation ","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\ny = train_df['Transported']\nX = train_df.drop('Transported', axis=1)\n\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.30, random_state=42)","metadata":{"execution":{"iopub.status.busy":"2022-03-14T15:14:36.162538Z","iopub.status.idle":"2022-03-14T15:14:36.162833Z","shell.execute_reply.started":"2022-03-14T15:14:36.162686Z","shell.execute_reply":"2022-03-14T15:14:36.162701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Training + Validation","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"##### Image taken from https://www.youtube.com/watch?v=goPiwckWE9M","metadata":{}},{"cell_type":"markdown","source":"#### For this dataset, I would be using **Random Forest Classifier**. \n#### For those of you not familiar with this model, it is basically a ensemble method where there are numerous numbers of decision trees and averaging is used to improve the predictive accuracy and control over-fitting. \n#### This model heavily relies on the \"Wisdom of Crowds\"\n#### As we know, decision trees can prove to be extremely accurate and does not need its features to be scaled nor normalized \n#### Furthermore, overfitting will be controlled later in the hyperparameter tuning section via hyperparameters such as n_estimators and max_depth\n#### For more information, please visit this website https://towardsdatascience.com/understanding-random-forest-58381e0602d2 which I think does a terrific job of explaining this model","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import roc_auc_score\n\nmodel = RandomForestClassifier(n_jobs=-1, random_state=42)\n\n#Fitting the model\nmodel.fit(X_train,y_train)\n\n#Prediction\npred = model.predict(X_val)\n\n#Evaluation\naccuracy = accuracy_score(y_val, pred)\nauc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])\n\n#Statement\nprint(\"The accuracy is {} and roc_auc_score is {}\".format(accuracy, auc))","metadata":{"execution":{"iopub.status.busy":"2022-03-05T11:43:03.041972Z","iopub.execute_input":"2022-03-05T11:43:03.04236Z","iopub.status.idle":"2022-03-05T11:43:06.018552Z","shell.execute_reply.started":"2022-03-05T11:43:03.04231Z","shell.execute_reply":"2022-03-05T11:43:06.017746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Our model engineering won't be complete without some Hyperparameter Tuning which I will be performing using GridSearch with Cross Validation","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\n# Create the parameter grid based on the results of random search \nparam_grid = {\n 'max_depth': [70,90, 110, 130],\n 'min_samples_leaf': [1,2,3],\n 'min_samples_split': [2,4,6],\n 'n_estimators': [100, 150, 200, 250, 300]\n}\n# Create a based model\nrf = RandomForestClassifier(n_jobs=-1, random_state=42)\n# Instantiate the grid search model\ngrid_search = GridSearchCV(estimator = rf, param_grid = param_grid, \n cv = 3, n_jobs = -1, verbose = 2)\n\n# Fit the grid search to the data\ngrid_search.fit(X_train,y_train)\n","metadata":{"execution":{"iopub.status.busy":"2022-03-05T11:43:06.01997Z","iopub.execute_input":"2022-03-05T11:43:06.020912Z","iopub.status.idle":"2022-03-05T11:57:43.577554Z","shell.execute_reply.started":"2022-03-05T11:43:06.020857Z","shell.execute_reply":"2022-03-05T11:57:43.576554Z"},"_kg_hide-output":true,"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### So we successfully obtained our best performing parameters","metadata":{}},{"cell_type":"code","source":"grid_search.best_params_","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-05T11:57:43.579206Z","iopub.execute_input":"2022-03-05T11:57:43.579467Z","iopub.status.idle":"2022-03-05T11:57:43.586701Z","shell.execute_reply.started":"2022-03-05T11:57:43.579435Z","shell.execute_reply":"2022-03-05T11:57:43.586067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Let us check the validation metrics with our improved model after some personal tuning too","metadata":{}},{"cell_type":"code","source":"rf_model = RandomForestClassifier(n_jobs=-1, \n random_state=42, \n n_estimators = 230, \n min_samples_split = 5, \n min_samples_leaf = 1, \n max_depth = 110, \n bootstrap = True)\n\n#Fitting the model\nrf_model.fit(X_train,y_train)\n\n#Prediction\npred = rf_model.predict(X_val)\n\n#Evaluation\nimproved_accuracy = accuracy_score(y_val, pred)\nimproved_auc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])\n\nimprovement_in_accuracy = ((improved_accuracy-accuracy)/accuracy)*100\nimporvement_in_auc = ((improved_auc-auc)/auc)*100\n\n#Statement\nprint(\"After Hyperparameter Tuning, the accuracy is {} and roc_auc_score is {}\".format(improved_accuracy, improved_auc))\nprint(\"Improvement in accuracy is {:.2f}% and in roc_auc_score is {:.2f}%\".format(improvement_in_accuracy,imporvement_in_auc))","metadata":{"execution":{"iopub.status.busy":"2022-03-05T11:57:43.588051Z","iopub.execute_input":"2022-03-05T11:57:43.588804Z","iopub.status.idle":"2022-03-05T11:57:49.512889Z","shell.execute_reply.started":"2022-03-05T11:57:43.588766Z","shell.execute_reply":"2022-03-05T11:57:49.511947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### The tuned model has seen an improvement but not by a significant extent","metadata":{}},{"cell_type":"markdown","source":"#### For the second model, I will be using XGBoost. XGBoost is also an ensemble method but this algorithm is focused on limiting the error/loss that it's predictions bring","metadata":{}},{"cell_type":"code","source":"from xgboost import XGBClassifier\n\nboost_model = XGBClassifier(n_jobs=-1, random_state=42)\n\n#Fitting the model\nboost_model.fit(X_train,y_train)\n\n#Prediction\npred = boost_model.predict(X_val)\n\n#Evaluation\naccuracy = accuracy_score(y_val, pred)\nauc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])\n\n#Statement\nprint(\"The accuracy is {} and roc_auc_score is {}\".format(accuracy, auc))","metadata":{"execution":{"iopub.status.busy":"2022-03-05T11:57:49.514303Z","iopub.execute_input":"2022-03-05T11:57:49.514565Z","iopub.status.idle":"2022-03-05T11:58:13.088571Z","shell.execute_reply.started":"2022-03-05T11:57:49.514532Z","shell.execute_reply":"2022-03-05T11:58:13.087519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### It seems like our baseline Gradient Boosting model is not as accurate as our Random Forest Model. Let us try with some hyperparameters","metadata":{}},{"cell_type":"code","source":"boost_model = XGBClassifier(n_jobs=-1, random_state=42, max_depth = 5)\n\n#Fitting the model\nboost_model.fit(X_train,y_train)\n\n#Prediction\npred = boost_model.predict(X_val)\n\n#Evaluation\nimproved_accuracy = accuracy_score(y_val, pred)\nimproved_auc = roc_auc_score(y_val, boost_model.predict_proba(X_val)[:, 1])\n\nimprovement_in_accuracy = ((improved_accuracy-accuracy)/accuracy)*100\nimporvement_in_auc = ((improved_auc-auc)/auc)*100\n\n#Statement\nprint(\"After Hyperparameter Tuning, the accuracy is {} and roc_auc_score is {}\".format(improved_accuracy, improved_auc))\nprint(\"Improvement in accuracy is {:.2f}% and in roc_auc_score is {:.2f}%\".format(improvement_in_accuracy,imporvement_in_auc))","metadata":{"execution":{"iopub.status.busy":"2022-03-05T11:58:13.090591Z","iopub.execute_input":"2022-03-05T11:58:13.090955Z","iopub.status.idle":"2022-03-05T11:58:32.964141Z","shell.execute_reply.started":"2022-03-05T11:58:13.090894Z","shell.execute_reply":"2022-03-05T11:58:32.963342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Seems like our tuned Gradient Boosting Model has a very slight advantage over our Random Forest Model","metadata":{}},{"cell_type":"markdown","source":"# 4. Machine Learning Model Report (Random Forest)","metadata":{}},{"cell_type":"markdown","source":"#### Let us first look at our confusion matrix for an overview of our Random Forest model's performance","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\n\ncf_matrix = confusion_matrix(y_val, pred)\n\nax = sns.heatmap(cf_matrix/np.sum(cf_matrix), annot=True, \n fmt='.2%', cmap='Blues')\n\nax.set_title('Seaborn Confusion Matrix with labels\\n\\n');\nax.set_xlabel('\\nPredicted Values')\nax.set_ylabel('Actual Values ');\n\n## Ticket labels - List must be in alphabetical order\nax.xaxis.set_ticklabels(['False','True'])\nax.yaxis.set_ticklabels(['False','True'])\n\nsns.set(rc={'figure.figsize':(10,10)})\n## Display the visualization of the Confusion Matrix.\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-05T11:58:32.968662Z","iopub.execute_input":"2022-03-05T11:58:32.97075Z","iopub.status.idle":"2022-03-05T11:58:33.303055Z","shell.execute_reply.started":"2022-03-05T11:58:32.97069Z","shell.execute_reply":"2022-03-05T11:58:33.302028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Seems like 80% of our predictions lies in the True Positive and True Negative portion which seems like a decent number\n#### The False Positive and False Negative proportions are also balanced","metadata":{}},{"cell_type":"markdown","source":"#### Now let us look at the feature importances which may give us a good insight as to which features were important and which were not as much","metadata":{}},{"cell_type":"code","source":"feats = {} \nfor feature, importance in zip(X.columns, model.feature_importances_):\n feats[feature] = importance\n \nfeature_importances = sorted(feats.items(), key=lambda x: x[1], reverse=True)\n\ndf = {\"Feature\": [], \"Importance\": []}\nfor i in feature_importances[:10]:\n df[\"Feature\"].append(i[0])\n df['Importance'].append(i[1])\n \nsns.set(rc={'figure.figsize':(20,15)})\nsns.barplot(x='Feature', y='Importance', data = df)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-05T11:58:33.304676Z","iopub.execute_input":"2022-03-05T11:58:33.305115Z","iopub.status.idle":"2022-03-05T11:58:33.786427Z","shell.execute_reply.started":"2022-03-05T11:58:33.30507Z","shell.execute_reply":"2022-03-05T11:58:33.785367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### The numerical columns seems to have had the highest importances. However, I have my doubts as I do not believe the amount spent at these amenities would greatly influence whether the person was transported. This may be due to several reasons. \n#### 1. The data for these numerical columns are very skewed to the lower values and most of the data points are 0\n#### 2. Since values for these categories are very low, even a small change in these values might greatly influence our model's predictions (without much meaning), thus explaining the high feature importance placed by our model\n#### The following features with the highest importances are whether the person opted for CryoSleep and their planet of Origin. ","metadata":{}},{"cell_type":"markdown","source":"# 5. Summary","metadata":{}},{"cell_type":"markdown","source":"#### Let us create a file to submit to the leaderboard","metadata":{}},{"cell_type":"code","source":"# Impute\nimputer = MissForest()\ndata_imputed = imputer.fit_transform(test_df.iloc[: , :6])\ndata_imputed = pd.DataFrame(data=data_imputed, columns=test_df.iloc[: , :6].columns)\ntest_df.drop(['Age', 'RoomService', 'FoodCourt', 'ShoppingMall', 'Spa', 'VRDeck'], axis = 1, inplace=True)\ntest_df = pd.concat([test_df, data_imputed], axis=1)\n\ntest_df.drop('Transported', axis=1, inplace=True)\n\n#submission\nrf_submission = rf_model.predict(test_df)\nboost_submission = boost_model.predict(test_df)\n\n#submission file\nsample_df['Transported'] = rf_submission\nsample_df.to_csv('rf_submission.csv', index=False)\n\nsample_df['Transported'] = boost_submission\nsample_df.to_csv('boost_submission.csv', index=False)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-03-05T11:58:33.788102Z","iopub.execute_input":"2022-03-05T11:58:33.788378Z","iopub.status.idle":"2022-03-05T11:58:50.215104Z","shell.execute_reply.started":"2022-03-05T11:58:33.788348Z","shell.execute_reply":"2022-03-05T11:58:50.214108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_df['Transported'] = boost_submission\nsample_df.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-03-05T12:10:33.036465Z","iopub.execute_input":"2022-03-05T12:10:33.036817Z","iopub.status.idle":"2022-03-05T12:10:33.054064Z","shell.execute_reply.started":"2022-03-05T12:10:33.036779Z","shell.execute_reply":"2022-03-05T12:10:33.053037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Our model seems to have performed decently with a rough accuracy of 79~80%. \n#### However, I do believe attempting other models to our data and selecting which feature to feed into our models might bring a higher accuracy. \n#### In the near future, I am planning on fitting **Gradient Boosting Model** (completed above) to our data and as to the feature engineering wise, I plan to try out different data imputation methods such as **KNNImputer**. Even different methods of hyperparameter tuning might help out, possibly using **Bayesian Optimization** which I am very interested in\n#### So do check out my profile for more informative notebooks like this! \n#### Any suggestions and comments are very much welcomed! Thank you for reading :)","metadata":{}},{"cell_type":"markdown","source":"## Acknowledgements\n### https://www.stackvidhya.com/plot-confusion-matrix-in-python-and-why/\n### https://towardsdatascience.com/hyperparameter-tuning-the-random-forest-in-python-using-scikit-learn-28d2aa77dd74\n### https://towardsdatascience.com/imputing-numerical-data-top-5-techniques-every-data-scientist-must-know-587c0f51552a","metadata":{}}]}
![](\"https://cdn.pixabay.com/photo/2017/08/30/01/05/milky-way-2695569__480.jpg\")
\n