{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Problem statement","metadata":{"cell_id":"00001-f42bccd6-adb6-4757-999f-c6810f41a029","deepnote_cell_type":"markdown","deepnote_cell_height":82}},{"cell_type":"markdown","source":"The sinking of the Titanic on April 15th, 1912 is one of the most tragic tragedies in history. The Titanic sank after colliding with an iceberg, killing 1502 out of 2224 passengers. The numbers of survivors were low due to the lack of lifeboats for all passengers and crew. Some passengers were more likely to survive than others, such as women, children, and upper-class. This case study analyzes what sorts of people were likely to survive this tragedy. The dataset includes the following: \n\n- Pclass:\tTicket class (1 = 1st, 2 = 2nd, 3 = 3rd)\n- Sex: Sex\t\n- Age: Age in years\t\n- Sibsp:\t# of siblings / spouses aboard the Titanic\t\n- Parch:\t# of parents / children aboard the Titanic\t\n- Ticket:\tTicket number\t\n- Fare:\tPassenger fare\t\n- Cabin:\tCabin number\t\n- Embarked:\tPort of Embarkation\tC = Cherbourg, Q = Queenstown, S = Southampton\n\n\n- Target class: Survived: Survival\t(0 = No, 1 = Yes)\n","metadata":{"cell_id":"00002-59bffafe-49e5-4f29-bf1e-6de867e48ad2","deepnote_cell_type":"markdown","deepnote_cell_height":420.859375}},{"cell_type":"markdown","source":"# Step 1: Data reading and insight","metadata":{"cell_id":"00007-176d004a-344a-42d2-8e28-9caa6827de70","deepnote_cell_type":"markdown","deepnote_cell_height":82}},{"cell_type":"code","source":"pip install lightgbm","metadata":{"cell_id":"29b7797321754d5780dadde5ba01ac5c","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"a343fc1d","execution_start":1658388880731,"execution_millis":8990,"deepnote_cell_type":"code","deepnote_cell_height":444,"execution":{"iopub.status.busy":"2022-07-21T21:46:11.282780Z","iopub.execute_input":"2022-07-21T21:46:11.283818Z","iopub.status.idle":"2022-07-21T21:46:24.662379Z","shell.execute_reply.started":"2022-07-21T21:46:11.283709Z","shell.execute_reply":"2022-07-21T21:46:24.660980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pip install imblearn","metadata":{"cell_id":"5ef97eafd9844cf1b2e599072c034fbd","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"564ed77b","execution_start":1658388933720,"execution_millis":5024,"deepnote_cell_type":"code","deepnote_cell_height":464,"execution":{"iopub.status.busy":"2022-07-21T21:46:31.944085Z","iopub.execute_input":"2022-07-21T21:46:31.944543Z","iopub.status.idle":"2022-07-21T21:46:43.074818Z","shell.execute_reply.started":"2022-07-21T21:46:31.944504Z","shell.execute_reply":"2022-07-21T21:46:43.073342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nsns.set_style('whitegrid')\nplt.style.use('seaborn-deep')\n\nimport warnings\nwarnings.filterwarnings('ignore')\nimport sklearn.metrics as skm\nimport sklearn.model_selection as skms\nimport sklearn.preprocessing as skp\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.metrics import accuracy_score, precision_score, recall_score, roc_auc_score, f1_score, confusion_matrix, precision_recall_curve, roc_curve\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.ensemble import RandomForestClassifier\nimport lightgbm as lgb\nfrom imblearn.under_sampling import RandomUnderSampler\nfrom imblearn.over_sampling import RandomOverSampler\nfrom imblearn.over_sampling import SMOTE\nimport random\n\n\nseed = 12\nnp.random.seed(seed)\n\nfrom datetime import date","metadata":{"cell_id":"3260d55accf94b31a74c36f7d88aa74a","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"4a3d3869","execution_start":1658388945295,"execution_millis":3606,"deepnote_cell_type":"code","deepnote_cell_height":585,"execution":{"iopub.status.busy":"2022-07-21T21:47:16.389021Z","iopub.execute_input":"2022-07-21T21:47:16.389427Z","iopub.status.idle":"2022-07-21T21:47:18.593043Z","shell.execute_reply.started":"2022-07-21T21:47:16.389396Z","shell.execute_reply":"2022-07-21T21:47:18.591457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# read the data using pandas dataframe\ntitanic = pd.read_csv('../input/titanic/train.csv')\ntitanic_test = pd.read_csv('../input/titanic/test.csv')\n\ntitanic.head(5)","metadata":{"cell_id":"00008-f4e8a86f-da69-4324-842b-94ed4f10f958","deepnote_to_be_reexecuted":false,"source_hash":"72474ca8","execution_start":1658388958990,"execution_millis":70,"deepnote_cell_type":"code","deepnote_cell_height":484,"execution":{"iopub.status.busy":"2022-07-21T21:47:29.209898Z","iopub.execute_input":"2022-07-21T21:47:29.210372Z","iopub.status.idle":"2022-07-21T21:47:29.258738Z","shell.execute_reply.started":"2022-07-21T21:47:29.210332Z","shell.execute_reply":"2022-07-21T21:47:29.257439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# important funtions\ndef datasetShape(df):\n rows, cols = df.shape\n print(\"The dataframe has\",rows,\"rows and\",cols,\"columns.\")\n \n# select numerical and categorical features\ndef divideFeatures(df):\n numerical_features = df.select_dtypes(include=[np.number]).drop('Survived', axis=1)\n categorical_features = df.select_dtypes(include=[np.object])\n return numerical_features, categorical_features\n","metadata":{"cell_id":"00009-7a265857-8dab-431a-9492-8617221a2130","deepnote_to_be_reexecuted":false,"source_hash":"32a7971f","execution_start":1658388967360,"execution_millis":3,"deepnote_cell_type":"code","deepnote_cell_height":261,"deepnote_output_heights":[177],"execution":{"iopub.status.busy":"2022-07-21T21:47:34.744593Z","iopub.execute_input":"2022-07-21T21:47:34.745002Z","iopub.status.idle":"2022-07-21T21:47:34.752412Z","shell.execute_reply.started":"2022-07-21T21:47:34.744968Z","shell.execute_reply":"2022-07-21T21:47:34.751226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check dataset shape\ndatasetShape(titanic)\n\n# check for duplicates\nif(len(titanic) == len(titanic['PassengerId'].unique())):\n print(\"No duplicates found!!\")\nelse:\n print(\"Duplicates occuring\")","metadata":{"cell_id":"00010-574b1f41-4ceb-4af2-b6af-f05a54dbf516","deepnote_to_be_reexecuted":false,"source_hash":"435bdf51","execution_start":1658389012896,"execution_millis":2,"deepnote_cell_type":"code","deepnote_cell_height":258.375,"deepnote_output_heights":[177],"execution":{"iopub.status.busy":"2022-07-21T21:47:37.951679Z","iopub.execute_input":"2022-07-21T21:47:37.952053Z","iopub.status.idle":"2022-07-21T21:47:37.965726Z","shell.execute_reply.started":"2022-07-21T21:47:37.952022Z","shell.execute_reply":"2022-07-21T21:47:37.964429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"titanic.info()","metadata":{"cell_id":"d455f6819c554a5d90987aac5b2f61a6","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"541f226c","execution_start":1658389018900,"execution_millis":14,"deepnote_cell_type":"code","deepnote_cell_height":475.5625,"execution":{"iopub.status.busy":"2022-07-21T21:47:40.983137Z","iopub.execute_input":"2022-07-21T21:47:40.983694Z","iopub.status.idle":"2022-07-21T21:47:41.012873Z","shell.execute_reply.started":"2022-07-21T21:47:40.983630Z","shell.execute_reply":"2022-07-21T21:47:41.011901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"titanic.describe()","metadata":{"cell_id":"d7a437051566487bb27ef7152f356209","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"5eea73d8","execution_start":1658389025412,"execution_millis":87,"deepnote_cell_type":"code","deepnote_cell_height":535,"execution":{"iopub.status.busy":"2022-07-21T21:47:44.165134Z","iopub.execute_input":"2022-07-21T21:47:44.165510Z","iopub.status.idle":"2022-07-21T21:47:44.206533Z","shell.execute_reply.started":"2022-07-21T21:47:44.165466Z","shell.execute_reply":"2022-07-21T21:47:44.205401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 2: EDA","metadata":{"cell_id":"00011-041e899f-fe4c-4971-a40e-14be865417e6","deepnote_cell_type":"markdown","deepnote_cell_height":82}},{"cell_type":"code","source":"# check null values\n\npd.DataFrame(titanic.isnull().sum(), columns=[\"Null Count\"]).style.background_gradient(cmap='Blues')\n","metadata":{"cell_id":"00012-0eac5509-7bfc-4004-a3c3-8aa91e8beb5e","deepnote_to_be_reexecuted":false,"source_hash":"ae69b5f0","execution_start":1658389033963,"execution_millis":1258,"deepnote_cell_type":"code","deepnote_cell_height":531,"deepnote_output_heights":[380],"execution":{"iopub.status.busy":"2022-07-21T21:47:49.287304Z","iopub.execute_input":"2022-07-21T21:47:49.287723Z","iopub.status.idle":"2022-07-21T21:47:49.383772Z","shell.execute_reply.started":"2022-07-21T21:47:49.287687Z","shell.execute_reply":"2022-07-21T21:47:49.382547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"numerical_features, categorical_features = divideFeatures(titanic)","metadata":{"cell_id":"00013-b3900328-c9a7-4b5c-a61e-a7abb0f2b7d5","deepnote_to_be_reexecuted":false,"source_hash":"bfbb472d","execution_start":1658389041548,"execution_millis":16,"deepnote_cell_type":"code","deepnote_cell_height":81,"execution":{"iopub.status.busy":"2022-07-21T21:47:55.175266Z","iopub.execute_input":"2022-07-21T21:47:55.175645Z","iopub.status.idle":"2022-07-21T21:47:55.185483Z","shell.execute_reply.started":"2022-07-21T21:47:55.175616Z","shell.execute_reply":"2022-07-21T21:47:55.183447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# boxplots of numerical features for outlier detection\n\nfig = plt.figure(figsize=(16,30))\nfor i in range(len(numerical_features.columns)):\n fig.add_subplot(9, 5, i+1)\n sns.boxplot(y=numerical_features.iloc[:,i])\nplt.tight_layout()\nplt.show()","metadata":{"cell_id":"becb56e7e0114fbab4a40075f3df6b0d","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"2a8ec006","execution_start":1658389048553,"execution_millis":837,"deepnote_cell_type":"code","deepnote_cell_height":527.71875,"deepnote_output_heights":[304.71875],"execution":{"iopub.status.busy":"2022-07-21T21:47:58.991228Z","iopub.execute_input":"2022-07-21T21:47:58.991660Z","iopub.status.idle":"2022-07-21T21:47:59.758536Z","shell.execute_reply.started":"2022-07-21T21:47:58.991621Z","shell.execute_reply":"2022-07-21T21:47:59.757328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# distplots for categorical data\n\nfig = plt.figure(figsize=(16,30))\nfor i in range(len(categorical_features.columns)):\n fig.add_subplot(9, 5, i+1)\n categorical_features.iloc[:,i].hist()\n plt.xlabel(categorical_features.columns[i])\nplt.tight_layout()\nplt.show()","metadata":{"cell_id":"4ec56c9908a04d04b6e706eac48ac9eb","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"267f12bc","execution_start":1658355531740,"execution_millis":146470,"deepnote_cell_type":"code","deepnote_cell_height":394.46875,"deepnote_output_heights":[153.46875],"execution":{"iopub.status.busy":"2022-07-21T21:48:04.647393Z","iopub.execute_input":"2022-07-21T21:48:04.647812Z","iopub.status.idle":"2022-07-21T21:49:09.660629Z","shell.execute_reply.started":"2022-07-21T21:48:04.647780Z","shell.execute_reply":"2022-07-21T21:49:09.659430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"plot the % of the target variable","metadata":{"cell_id":"b9f688cb3b20431cbc595f88bb4c52d8","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"sns.color_palette(\"Blues\", as_cmap=True)\nGnBu_palette = sns.color_palette(\"GnBu\",10)\nBlues_palette = sns.color_palette(\"Blues\",10)\nsns.palplot(Blues_palette)\nsns.palplot(GnBu_palette)","metadata":{"cell_id":"0ab6947d009948438d91ca46a2b1e93b","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"b3d3b756","execution_start":1658389057820,"execution_millis":225,"deepnote_cell_type":"code","deepnote_cell_height":340,"deepnote_output_heights":[70,70],"execution":{"iopub.status.busy":"2022-07-21T21:49:14.258094Z","iopub.execute_input":"2022-07-21T21:49:14.258522Z","iopub.status.idle":"2022-07-21T21:49:14.478868Z","shell.execute_reply.started":"2022-07-21T21:49:14.258464Z","shell.execute_reply":"2022-07-21T21:49:14.477572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(1,2,figsize=(10,4))\n\ntitanic['Survived'].value_counts().plot.pie(\n explode=[0,0.1],autopct='%1.1f%%',ax=axes[0],shadow=True, colors=[Blues_palette[1],Blues_palette[3]]\n)\n\nsns.countplot('Survived',data=titanic,ax=axes[1], palette=[GnBu_palette[6],GnBu_palette[7]])\naxes[1].patch.set_alpha(0)\n\nfig.text(0.28,0.92,\"Distribution of Outcome percent\", fontweight=\"bold\", fontfamily='serif', fontsize=17)\n\nplt.show()","metadata":{"cell_id":"f721b313e5464cb9acd8a6da14028842","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"61bd8229","execution_start":1658389068300,"execution_millis":318,"deepnote_cell_type":"code","deepnote_cell_height":584,"deepnote_output_heights":[289,254],"execution":{"iopub.status.busy":"2022-07-21T21:49:18.345456Z","iopub.execute_input":"2022-07-21T21:49:18.345928Z","iopub.status.idle":"2022-07-21T21:49:18.632139Z","shell.execute_reply.started":"2022-07-21T21:49:18.345891Z","shell.execute_reply":"2022-07-21T21:49:18.630905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Distribution of continue data over output column\ncontinue_features = numerical_features.select_dtypes(include=['float64'])\nsns.pairplot(titanic, hue = 'Survived', vars = continue_features )","metadata":{"cell_id":"efea4f47c37c405f8719e21fec2b3d67","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"8fc1842e","execution_start":1658389076120,"execution_millis":3812,"deepnote_cell_type":"code","deepnote_cell_height":544.1875,"deepnote_output_heights":[21.1875,359,359],"execution":{"iopub.status.busy":"2022-07-21T21:49:24.390187Z","iopub.execute_input":"2022-07-21T21:49:24.390777Z","iopub.status.idle":"2022-07-21T21:49:25.952390Z","shell.execute_reply.started":"2022-07-21T21:49:24.390727Z","shell.execute_reply":"2022-07-21T21:49:25.951534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Most of the dead are close to 25 years of age\n- The chance of survival increases if you pay passenger fares","metadata":{"cell_id":"9bad71bd33d94715b416656935c6c2dc","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":94.78125}},{"cell_type":"code","source":"# Distribution of discrete data over output column\ndiscrete_features = numerical_features.select_dtypes(include=['integer']).drop('PassengerId', axis=1)\n\nfig, axes = plt.subplots(1,3, figsize=(14,6) )\n#fig, ax =plt.subplots(1,2)\nsns.countplot(x = 'Pclass', hue = 'Survived', data=titanic, ax=axes[0])\nsns.countplot(x = 'SibSp', hue = 'Survived', data=titanic, ax=axes[1])\nsns.countplot(x = 'Parch', hue = 'Survived', data=titanic, ax=axes[2])\nfig.show()","metadata":{"cell_id":"045a6a29ac514953a9864da05bf56a53","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"8c928f93","execution_start":1658389105766,"execution_millis":1430,"deepnote_cell_type":"code","deepnote_cell_height":562.890625,"deepnote_output_heights":[321.890625,611],"execution":{"iopub.status.busy":"2022-07-21T21:49:28.456165Z","iopub.execute_input":"2022-07-21T21:49:28.457072Z","iopub.status.idle":"2022-07-21T21:49:29.045332Z","shell.execute_reply.started":"2022-07-21T21:49:28.457030Z","shell.execute_reply":"2022-07-21T21:49:29.043989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- The chance of survival in first class is higher \n- The chance of survival is higher with one sibling on board\n- The chance of survival is higher with 1-3 children on board","metadata":{"cell_id":"10098256adfe415abf6d2e7018d5717e","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":120.171875}},{"cell_type":"code","source":"# Distribution of categorical data over output column\n\nfig, axes = plt.subplots(1,2, figsize=(14,6) )\nsns.countplot(x = 'Sex', hue = 'Survived', data=titanic, ax=axes[0])\nsns.countplot(x = 'Embarked', hue = 'Survived', data=titanic, ax=axes[1])\nfig.show()\n","metadata":{"cell_id":"00014-692a4d30-3634-4cda-ac9a-318d88410153","deepnote_to_be_reexecuted":false,"source_hash":"98597c11","execution_start":1658389116416,"execution_millis":959,"deepnote_cell_type":"code","deepnote_cell_height":526.890625,"deepnote_output_heights":[321.890625,611],"execution":{"iopub.status.busy":"2022-07-21T21:49:33.105379Z","iopub.execute_input":"2022-07-21T21:49:33.105796Z","iopub.status.idle":"2022-07-21T21:49:33.412258Z","shell.execute_reply.started":"2022-07-21T21:49:33.105767Z","shell.execute_reply":"2022-07-21T21:49:33.409754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- The chance of survival is higher for female\n- The chance of survival is higher for passengers embarked in the port of Cherbourg","metadata":{"cell_id":"d074cac87d81412db71dd5e8d0b99c79","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":94.78125}},{"cell_type":"code","source":"titanic[titanic.columns[:13]].corr().style.background_gradient(cmap='Blues')","metadata":{"cell_id":"2128a3ed3d184ab083bd027b5acf6f13","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"566c0731","execution_start":1658389127564,"execution_millis":646,"deepnote_cell_type":"code","deepnote_cell_height":332,"deepnote_output_heights":[235],"execution":{"iopub.status.busy":"2022-07-21T21:49:37.535327Z","iopub.execute_input":"2022-07-21T21:49:37.535734Z","iopub.status.idle":"2022-07-21T21:49:37.566551Z","shell.execute_reply.started":"2022-07-21T21:49:37.535701Z","shell.execute_reply":"2022-07-21T21:49:37.565242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 3: Data Cleaning ","metadata":{"cell_id":"00023-6c4e87eb-9254-4647-998b-45ceaa26a0da","deepnote_cell_type":"markdown","deepnote_cell_height":82}},{"cell_type":"code","source":"# drop the PassengerId, Cabin, Ticket and Name \ntitanic.drop(['PassengerId', 'Cabin', 'Ticket', 'Name'], axis=1, inplace=True)\n\n# drop the duplicate rows\ntitanic.drop_duplicates(inplace=True)\ndatasetShape(titanic)","metadata":{"cell_id":"7924ac398a004755909ad3542bfe9a6c","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"3e344f95","execution_start":1658389134418,"execution_millis":36,"deepnote_cell_type":"code","deepnote_cell_height":202.1875,"execution":{"iopub.status.busy":"2022-07-21T21:49:41.515706Z","iopub.execute_input":"2022-07-21T21:49:41.516096Z","iopub.status.idle":"2022-07-21T21:49:41.530136Z","shell.execute_reply.started":"2022-07-21T21:49:41.516065Z","shell.execute_reply":"2022-07-21T21:49:41.528897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# remove all columns having no values\ntitanic.dropna(axis=1, how=\"all\", inplace=True)\ntitanic.dropna(axis=0, how=\"all\", inplace=True)\ndatasetShape(titanic)","metadata":{"cell_id":"b1e274b240204ceb82000be10e15a4e4","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"f92c9ae6","execution_start":1658389148735,"execution_millis":36,"deepnote_cell_type":"code","deepnote_cell_height":166.1875,"execution":{"iopub.status.busy":"2022-07-21T21:49:44.855769Z","iopub.execute_input":"2022-07-21T21:49:44.856131Z","iopub.status.idle":"2022-07-21T21:49:44.869958Z","shell.execute_reply.started":"2022-07-21T21:49:44.856103Z","shell.execute_reply":"2022-07-21T21:49:44.868745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# remove columns having null values more than 30%\ntitanic.dropna(thresh=titanic.shape[0]*0.7,how='all',axis=1, inplace=True)\ndatasetShape(titanic)","metadata":{"cell_id":"14264dc9799a48268757853ee515deaf","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"4052d3d4","execution_start":1658389156817,"execution_millis":32,"deepnote_cell_type":"code","deepnote_cell_height":148.1875,"execution":{"iopub.status.busy":"2022-07-21T21:49:48.495900Z","iopub.execute_input":"2022-07-21T21:49:48.496348Z","iopub.status.idle":"2022-07-21T21:49:48.507230Z","shell.execute_reply.started":"2022-07-21T21:49:48.496310Z","shell.execute_reply":"2022-07-21T21:49:48.506025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Missing Value Imputation","metadata":{"cell_id":"0ad827552c294d0a931e3bd83ce7ddca","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"# missing values with percentage\n\ndef calc_missing(df):\n missing = df.isna().sum().sort_values(ascending=False)\n missing = missing[missing != 0]\n missing_perc = missing/df.shape[0]*100\n return missing, missing_perc\n \n\nmissing, missing_perc = calc_missing(titanic)\npd.concat([missing, missing_perc], axis=1, keys=['Total','Percent'])","metadata":{"cell_id":"00028-e9048e0f-206d-4d70-b58a-19f03cf5dca5","deepnote_to_be_reexecuted":false,"source_hash":"cffe9fb0","execution_start":1658389161852,"execution_millis":562,"deepnote_cell_type":"code","deepnote_cell_height":452,"deepnote_output_heights":[382.1875],"execution":{"iopub.status.busy":"2022-07-21T21:49:51.071132Z","iopub.execute_input":"2022-07-21T21:49:51.071542Z","iopub.status.idle":"2022-07-21T21:49:51.088810Z","shell.execute_reply.started":"2022-07-21T21:49:51.071506Z","shell.execute_reply":"2022-07-21T21:49:51.087475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Imputing Age with mean value as its on the average of the distribution and also relevant.","metadata":{"cell_id":"cb974c90bc7f431eb11bf09e5a3da7f4","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"titanic.loc[titanic.Age.isna(), 'Age'] = titanic.Age.mean()\nprint(\"Missing values in Age:\",titanic.Age.isna().sum())","metadata":{"cell_id":"00029-4b10582c-a2eb-41d2-aa05-7804c5fec947","deepnote_to_be_reexecuted":false,"source_hash":"b2b13bcb","execution_start":1658389483323,"execution_millis":135,"deepnote_cell_type":"code","deepnote_cell_height":130.1875,"deepnote_output_heights":[21.1875,250],"execution":{"iopub.status.busy":"2022-07-21T21:49:54.760379Z","iopub.execute_input":"2022-07-21T21:49:54.761617Z","iopub.status.idle":"2022-07-21T21:49:54.771321Z","shell.execute_reply.started":"2022-07-21T21:49:54.761556Z","shell.execute_reply":"2022-07-21T21:49:54.770193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"titanic_test.loc[titanic_test.Age.isna(), 'Age'] = titanic_test.Age.mean()\nprint(\"Missing values in Age:\",titanic_test.Age.isna().sum())","metadata":{"cell_id":"00030-401ca761-65d2-4fe1-b8d8-3d294b414ab5","deepnote_to_be_reexecuted":false,"source_hash":"510b4642","execution_start":1658389487833,"execution_millis":8,"deepnote_cell_type":"code","deepnote_cell_height":130.1875,"deepnote_output_heights":[21.1875,481.171875],"execution":{"iopub.status.busy":"2022-07-21T21:49:57.922740Z","iopub.execute_input":"2022-07-21T21:49:57.923118Z","iopub.status.idle":"2022-07-21T21:49:57.930874Z","shell.execute_reply.started":"2022-07-21T21:49:57.923087Z","shell.execute_reply":"2022-07-21T21:49:57.929560Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Imputing Embarked with mode value ","metadata":{"cell_id":"6defe03043674958a080770eb64ad5f0","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"titanic.Embarked.mode()","metadata":{"cell_id":"4792c3cec2ac44c9939e581adef0d050","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"f80d41bc","execution_start":1658389492045,"execution_millis":58,"deepnote_cell_type":"code","deepnote_cell_height":137.375,"deepnote_output_heights":[40.375],"execution":{"iopub.status.busy":"2022-07-21T21:50:04.318381Z","iopub.execute_input":"2022-07-21T21:50:04.318796Z","iopub.status.idle":"2022-07-21T21:50:04.328061Z","shell.execute_reply.started":"2022-07-21T21:50:04.318762Z","shell.execute_reply":"2022-07-21T21:50:04.326869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"titanic.loc[titanic.Embarked.isna(), 'Embarked'] = 'S'\nprint(\"Missing values in Embarked:\",titanic.Embarked.isna().sum())","metadata":{"cell_id":"3521592c3f024d4c86e7de10b2b18da6","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"e4ea5bc1","execution_start":1658389506948,"execution_millis":7,"deepnote_cell_type":"code","deepnote_cell_height":130.1875,"execution":{"iopub.status.busy":"2022-07-21T21:50:09.612243Z","iopub.execute_input":"2022-07-21T21:50:09.612736Z","iopub.status.idle":"2022-07-21T21:50:09.621217Z","shell.execute_reply.started":"2022-07-21T21:50:09.612691Z","shell.execute_reply":"2022-07-21T21:50:09.619999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"titanic_test.loc[titanic_test.Embarked.isna(), 'Embarked'] = 'S'\nprint(\"Missing values in Embarked:\",titanic_test.Embarked.isna().sum())","metadata":{"cell_id":"1fac4ce7a8734fc1a7cfdc6ef29d41c8","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"ae3b611d","execution_start":1658389514713,"execution_millis":802,"deepnote_cell_type":"code","deepnote_cell_height":130.1875,"execution":{"iopub.status.busy":"2022-07-21T21:50:13.093259Z","iopub.execute_input":"2022-07-21T21:50:13.093690Z","iopub.status.idle":"2022-07-21T21:50:13.101219Z","shell.execute_reply.started":"2022-07-21T21:50:13.093654Z","shell.execute_reply":"2022-07-21T21:50:13.100333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 4: Data Preparation","metadata":{"cell_id":"b96091e517b042e7a5b63d6c577ba59e","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":82}},{"cell_type":"markdown","source":"Outlier Treatment\n\nTreating with the Survived target feature and other numerical features, which are skewed. We will take log of the feature values using np.log1p()","metadata":{"cell_id":"fbacc4b1c7794d04ac0d74c3a606a5fa","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":111.171875}},{"cell_type":"code","source":"# extract all skewed features\ntemp_numerical_features, temp_categorical_features = divideFeatures(titanic)\n# remove categorical features stored as int\ntemp_numerical_features.drop(['Pclass'], axis=1, inplace=True)\nskewed_features = temp_numerical_features.apply(lambda x: x.skew()).sort_values(ascending=False)","metadata":{"cell_id":"00032-5809f75e-74bb-4490-9aa9-e29616a0d62a","deepnote_to_be_reexecuted":false,"source_hash":"91258632","execution_start":1658389524302,"execution_millis":7,"deepnote_cell_type":"code","deepnote_cell_height":153,"execution":{"iopub.status.busy":"2022-07-21T21:50:17.738561Z","iopub.execute_input":"2022-07-21T21:50:17.738960Z","iopub.status.idle":"2022-07-21T21:50:17.752413Z","shell.execute_reply.started":"2022-07-21T21:50:17.738925Z","shell.execute_reply":"2022-07-21T21:50:17.751533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# transform skewed features\nfor feat in skewed_features.index:\n if skewed_features.loc[feat] > 0.5:\n titanic[feat] = np.log1p(titanic[feat])\n titanic_test[feat] = np.log1p(titanic_test[feat])\n","metadata":{"cell_id":"00033-c726b7db-cf4c-4076-94b2-e6f07d906890","deepnote_to_be_reexecuted":false,"source_hash":"42711fdc","execution_start":1658389530249,"execution_millis":4,"deepnote_cell_type":"code","deepnote_cell_height":171,"deepnote_output_heights":[21.1875,250],"execution":{"iopub.status.busy":"2022-07-21T21:50:20.881575Z","iopub.execute_input":"2022-07-21T21:50:20.881938Z","iopub.status.idle":"2022-07-21T21:50:20.891810Z","shell.execute_reply.started":"2022-07-21T21:50:20.881910Z","shell.execute_reply":"2022-07-21T21:50:20.890672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# outlier treatment for categorical features\ndef getCategoricalSkewed(categories, threshold):\n tempSkewedFeatures = []\n for feat in categories:\n for featValuePerc in list(titanic[feat].value_counts()/titanic.shape[0]):\n if featValuePerc > threshold:\n tempSkewedFeatures.append(feat)\n return list(set(tempSkewedFeatures))\n\n# display all categorical skewed features which have value_counts > 90%\ncategoricalSkewed = getCategoricalSkewed(temp_categorical_features.columns, .90)\nfor feat in categoricalSkewed:\n print(titanic[feat].value_counts()/len(titanic))\n print()","metadata":{"cell_id":"00034-78759df6-6308-47f7-8afa-b63f4405d0ce","deepnote_to_be_reexecuted":false,"source_hash":"39185b68","execution_start":1658389539121,"execution_millis":4,"deepnote_cell_type":"code","deepnote_cell_height":315,"deepnote_output_heights":[382.1875],"execution":{"iopub.status.busy":"2022-07-21T21:50:24.276079Z","iopub.execute_input":"2022-07-21T21:50:24.276530Z","iopub.status.idle":"2022-07-21T21:50:24.286845Z","shell.execute_reply.started":"2022-07-21T21:50:24.276483Z","shell.execute_reply":"2022-07-21T21:50:24.285745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"No skewed categorical features","metadata":{"cell_id":"528da422560445d1911cc4881aeb6595","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"#print(\"Before Removing:\")\n#datasetShape(titanic)\n\n# removing skewed categorical data \n#titanic.drop(categoricalSkewed, axis=1, inplace=True)\n#print(\"After Removing:\")\n#datasetShape(titanic)","metadata":{"cell_id":"00036-eeb2f447-7988-4638-9a32-c7ffe3e98795","deepnote_cell_type":"code","deepnote_cell_height":174,"execution":{"iopub.status.busy":"2022-07-21T12:58:48.827052Z","iopub.execute_input":"2022-07-21T12:58:48.827384Z","iopub.status.idle":"2022-07-21T12:58:48.836258Z","shell.execute_reply.started":"2022-07-21T12:58:48.827353Z","shell.execute_reply":"2022-07-21T12:58:48.835142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Binning Features","metadata":{"cell_id":"2a9bd61ef9a84f3bb4ceaef6810a5506","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"# Titanic Pclass in three bins\ntitanic['Pclass'].replace([1,2,3], ['First', 'Second', 'Third'], inplace=True)\ntitanic_test['Pclass'].replace([1,2,3], ['First', 'Second', 'Third'], inplace=True)","metadata":{"cell_id":"9278ddaf68ec41009c0fc4c3b10bb01a","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"653215c1","execution_start":1658389557931,"execution_millis":3,"deepnote_cell_type":"code","deepnote_cell_height":117,"execution":{"iopub.status.busy":"2022-07-21T21:50:50.251914Z","iopub.execute_input":"2022-07-21T21:50:50.252330Z","iopub.status.idle":"2022-07-21T21:50:50.261335Z","shell.execute_reply.started":"2022-07-21T21:50:50.252296Z","shell.execute_reply":"2022-07-21T21:50:50.259984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create Dummy Features","metadata":{"cell_id":"dc47f8c39d804c56aabc8d926b0556b1","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"# extract numerical and categorical for dummy and scaling later\n\ncategorical_features.drop(['Name','Ticket', 'Cabin'], axis=1, inplace=True)\n\nnumerical_features, categorical_features = divideFeatures(titanic)\nfor feat in categorical_features.columns:\n dummyVars = pd.get_dummies(titanic[feat], drop_first=True, prefix=feat+\"_\")\n titanic = pd.concat([titanic, dummyVars], axis=1)\n titanic.drop(feat, axis=1, inplace=True)\n\n dummyVars_test = pd.get_dummies(titanic_test[feat], drop_first=True, prefix=feat+\"_\")\n titanic_test = pd.concat([titanic_test, dummyVars_test], axis=1)\n titanic_test.drop(feat, axis=1, inplace=True)\n\ndatasetShape(titanic)","metadata":{"cell_id":"09897b54c5f140a9a5aaace7e2fe914d","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"b86b3c04","execution_start":1658390080912,"execution_millis":286,"deepnote_cell_type":"code","deepnote_cell_height":364.1875,"execution":{"iopub.status.busy":"2022-07-21T21:50:56.882456Z","iopub.execute_input":"2022-07-21T21:50:56.882901Z","iopub.status.idle":"2022-07-21T21:50:56.915667Z","shell.execute_reply.started":"2022-07-21T21:50:56.882862Z","shell.execute_reply":"2022-07-21T21:50:56.914312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 5: Data Modelling","metadata":{"cell_id":"00040-75bba0a1-7d35-4561-a8cb-36d68cd5f617","deepnote_cell_type":"markdown","deepnote_cell_height":82}},{"cell_type":"markdown","source":"Split Train-Test Data","metadata":{"cell_id":"f19396eac3e24292a6a2b67d20b458ec","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"# shuffle samples\ndf_shuffle = titanic.sample(frac=1, random_state=seed).reset_index(drop=True)","metadata":{"cell_id":"00041-c0899813-fb5f-4ebc-bffa-cb24d0a8c9e5","deepnote_to_be_reexecuted":false,"source_hash":"972f99e2","execution_start":1658390220390,"execution_millis":6,"deepnote_cell_type":"code","deepnote_cell_height":99,"execution":{"iopub.status.busy":"2022-07-21T21:51:09.678294Z","iopub.execute_input":"2022-07-21T21:51:09.678728Z","iopub.status.idle":"2022-07-21T21:51:09.687761Z","shell.execute_reply.started":"2022-07-21T21:51:09.678690Z","shell.execute_reply":"2022-07-21T21:51:09.686509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_y = df_shuffle.pop('Survived')\ndf_X = df_shuffle\n\n# split into train dev and test\nX_train, X_test, y_train, y_test = skms.train_test_split(df_X, df_y, train_size=0.7, random_state=seed)\nprint(f\"Train set has {X_train.shape[0]} records out of {len(df_shuffle)} which is {round(X_train.shape[0]/len(df_shuffle)*100)}%\")\nprint(f\"Test set has {X_test.shape[0]} records out of {len(df_shuffle)} which is {round(X_test.shape[0]/len(df_shuffle)*100)}%\")","metadata":{"cell_id":"00042-93813810-ea34-43ee-87c3-fbc972e472ab","deepnote_to_be_reexecuted":false,"source_hash":"77d56b8a","execution_start":1658390261056,"execution_millis":1,"deepnote_cell_type":"code","deepnote_cell_height":240.375,"execution":{"iopub.status.busy":"2022-07-21T21:51:14.606556Z","iopub.execute_input":"2022-07-21T21:51:14.606966Z","iopub.status.idle":"2022-07-21T21:51:14.619128Z","shell.execute_reply.started":"2022-07-21T21:51:14.606931Z","shell.execute_reply":"2022-07-21T21:51:14.617939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Feature Scaling","metadata":{"cell_id":"3d4c750c2e914777915bb838d29cfc4b","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"scaler = skp.StandardScaler()\n\n# apply scaling to all numerical variables (float64) except dummy variables as they are already between 0 and 1\n\nX_train[continue_features.columns] = scaler.fit_transform(X_train[continue_features.columns])\n\n# scale test data with transform()\nX_test[continue_features.columns] = scaler.transform(X_test[continue_features.columns])\n\ntitanic_test[continue_features.columns] = scaler.transform(titanic_test[continue_features.columns])\n\n\n# view sample data\nX_train.describe()","metadata":{"cell_id":"00043-728ea120-d452-49b6-978f-7daa74e07b89","deepnote_to_be_reexecuted":false,"source_hash":"eef521ca","execution_start":1658390948372,"execution_millis":592,"deepnote_cell_type":"code","deepnote_cell_height":769,"deepnote_output_heights":[21.1875],"execution":{"iopub.status.busy":"2022-07-21T21:51:24.679101Z","iopub.execute_input":"2022-07-21T21:51:24.679530Z","iopub.status.idle":"2022-07-21T21:51:24.729459Z","shell.execute_reply.started":"2022-07-21T21:51:24.679472Z","shell.execute_reply":"2022-07-21T21:51:24.727930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"smote = SMOTE()\nX_over, y_over = smote.fit_resample(X_train,y_train)","metadata":{"cell_id":"b737e02ea00848c1958aaefc5eaa0066","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"b5c63757","execution_start":1658391172286,"execution_millis":1,"deepnote_cell_type":"code","deepnote_cell_height":99,"execution":{"iopub.status.busy":"2022-07-21T21:51:37.981475Z","iopub.execute_input":"2022-07-21T21:51:37.982636Z","iopub.status.idle":"2022-07-21T21:51:37.996627Z","shell.execute_reply.started":"2022-07-21T21:51:37.982576Z","shell.execute_reply":"2022-07-21T21:51:37.995237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_over.shape","metadata":{"cell_id":"a8dd2fc8a15d49c9b28752d99ab4d0a2","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"5add5d25","execution_start":1658391206969,"execution_millis":364,"deepnote_cell_type":"code","deepnote_cell_height":118.1875,"deepnote_output_heights":[21.1875],"execution":{"iopub.status.busy":"2022-07-21T21:51:40.614873Z","iopub.execute_input":"2022-07-21T21:51:40.615390Z","iopub.status.idle":"2022-07-21T21:51:40.621978Z","shell.execute_reply.started":"2022-07-21T21:51:40.615342Z","shell.execute_reply":"2022-07-21T21:51:40.620822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(1,2,figsize=(10,4))\n\ny_over.value_counts().plot.pie(\n explode=[0,0.1],autopct='%1.1f%%',ax=axes[0],shadow=True, colors=[Blues_palette[1],Blues_palette[3]]\n)\n\nsns.countplot(y_over, ax=axes[1], palette=[GnBu_palette[6],GnBu_palette[7]])\naxes[1].patch.set_alpha(0)\n\nfig.text(0.28,0.92,\"Distribution of After SMOTE Output percent\", fontweight=\"bold\", fontfamily='serif', fontsize=17)\n\nplt.show()","metadata":{"cell_id":"4da6270529894d7e81f314854717e7e3","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"6fad88fa","execution_start":1658391295412,"execution_millis":253,"deepnote_cell_type":"code","deepnote_cell_height":584,"deepnote_output_heights":[289],"execution":{"iopub.status.busy":"2022-07-21T21:51:43.227948Z","iopub.execute_input":"2022-07-21T21:51:43.228353Z","iopub.status.idle":"2022-07-21T21:51:43.492911Z","shell.execute_reply.started":"2022-07-21T21:51:43.228319Z","shell.execute_reply":"2022-07-21T21:51:43.491528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Before SMOTE","metadata":{"cell_id":"d7370c25f8894479b4e723723d992131","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":82}},{"cell_type":"code","source":"# Evaluation function definition\n\ndef get_clf_eval(y_test, pred = None, pred_proba = None):\n confusion = confusion_matrix(y_test, pred)\n accuacy = accuracy_score(y_test, pred)\n precision = precision_score(y_test, pred)\n recall = recall_score(y_test, pred)\n f1 = f1_score(y_test, pred)\n roc_auc = roc_auc_score(y_test, pred_proba)\n \n print('confusion')\n print(confusion)\n print('accuacy : {}'.format(np.around(accuacy,4)))\n print('precision: {}'.format(np.around(precision,4)))\n print('recall : {}'.format(np.around(recall,4)))\n print('F1 : {}'.format(np.around(f1,4))) \n print('ROC_AUC : {}'.format(np.around(roc_auc,4)))","metadata":{"cell_id":"f7961210a57d44ef9a1437408c803bb6","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"d7423eb","execution_start":1658391550941,"execution_millis":5,"deepnote_cell_type":"code","deepnote_cell_height":369,"execution":{"iopub.status.busy":"2022-07-21T21:51:47.580110Z","iopub.execute_input":"2022-07-21T21:51:47.580555Z","iopub.status.idle":"2022-07-21T21:51:47.590362Z","shell.execute_reply.started":"2022-07-21T21:51:47.580518Z","shell.execute_reply":"2022-07-21T21:51:47.588977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Logistic Regression Model","metadata":{"cell_id":"63b441d795b34b66b5cbdf25fcd74d43","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"lg_reg = LogisticRegression()\n\nlg_reg.fit(X_train, y_train)\npred = lg_reg.predict(X_test)\npred_proba = lg_reg.predict_proba(X_test)[:,1]\nget_clf_eval(y_test, pred, pred_proba)","metadata":{"cell_id":"5ea57a077fa6420ea278c7973c6447c5","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"8720cb92","execution_start":1658391559725,"execution_millis":508,"deepnote_cell_type":"code","deepnote_cell_height":343.5,"execution":{"iopub.status.busy":"2022-07-21T21:51:51.200199Z","iopub.execute_input":"2022-07-21T21:51:51.200638Z","iopub.status.idle":"2022-07-21T21:51:51.233218Z","shell.execute_reply.started":"2022-07-21T21:51:51.200601Z","shell.execute_reply":"2022-07-21T21:51:51.231973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Classification Accuracy = (TP+TN) / (TP + TN + FP + FN)\n\nMisclassification rate (Error Rate) = (FP + FN) / (TP + TN + FP + FN)\n\nPrecision = TP/Total TRUE Predictions = TP/ (TP+FP) (When model predicted TRUE class, how often was it right?)\n\nRecall = TP/ Actual TRUE = TP/ (TP+FN) (when the class was actually TRUE, how often did the classifier get it right?)\n\nF1-Score = 2/(1/Recall + 1/Precision) (is a harmonic mean of Precision and Recall, and so it gives a combined idea about these two metrics. It is maximum when Precision is equal to Recall.)","metadata":{"cell_id":"ab007bc0de63491fba34e36f02917d21","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":265.125}},{"cell_type":"markdown","source":"Random Forest Model","metadata":{"cell_id":"ea63b6c3a0c143689be43bcf95b85609","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"rf_clf = RandomForestClassifier()\nparam = {'n_estimators' : [100],\n 'max_depth':[8,9,10],\n 'min_samples_split':[2,5,7],\n 'min_samples_leaf':[6.5,7,7.5]\n }","metadata":{"cell_id":"4d5e3e2f776549058a9151c443260a2e","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"d4ff9bc0","execution_start":1658391626863,"execution_millis":13,"deepnote_cell_type":"code","deepnote_cell_height":171,"execution":{"iopub.status.busy":"2022-07-21T21:52:07.919041Z","iopub.execute_input":"2022-07-21T21:52:07.919454Z","iopub.status.idle":"2022-07-21T21:52:07.925906Z","shell.execute_reply.started":"2022-07-21T21:52:07.919418Z","shell.execute_reply":"2022-07-21T21:52:07.924768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grid = GridSearchCV(rf_clf,param_grid = param,scoring = 'accuracy',cv=5)\ngrid.fit(X_train ,y_train)","metadata":{"cell_id":"c449ddc44a234e81854e9f3d90c65c80","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"7cdcd902","execution_start":1658391643757,"execution_millis":29036,"deepnote_cell_type":"code","deepnote_cell_height":208.328125,"deepnote_output_heights":[93.328125],"execution":{"iopub.status.busy":"2022-07-21T21:52:10.829401Z","iopub.execute_input":"2022-07-21T21:52:10.830684Z","iopub.status.idle":"2022-07-21T21:52:24.361076Z","shell.execute_reply.started":"2022-07-21T21:52:10.830639Z","shell.execute_reply":"2022-07-21T21:52:24.360057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grid.best_params_","metadata":{"cell_id":"d60232356dc645a68c7e9d8a209908d5","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"bc2baa2b","execution_start":1658391730076,"execution_millis":383,"deepnote_cell_type":"code","deepnote_cell_height":175.75,"deepnote_output_heights":[78.75],"execution":{"iopub.status.busy":"2022-07-21T21:52:27.439085Z","iopub.execute_input":"2022-07-21T21:52:27.439532Z","iopub.status.idle":"2022-07-21T21:52:27.447819Z","shell.execute_reply.started":"2022-07-21T21:52:27.439481Z","shell.execute_reply":"2022-07-21T21:52:27.446657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grid.best_score_","metadata":{"cell_id":"606ae2b4455c409a9f386e9bc4bfdfce","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"4fc6697","execution_start":1658391754727,"execution_millis":14,"deepnote_cell_type":"code","deepnote_cell_height":118.1875,"deepnote_output_heights":[21.1875],"execution":{"iopub.status.busy":"2022-07-21T21:52:33.348682Z","iopub.execute_input":"2022-07-21T21:52:33.349063Z","iopub.status.idle":"2022-07-21T21:52:33.356164Z","shell.execute_reply.started":"2022-07-21T21:52:33.349032Z","shell.execute_reply":"2022-07-21T21:52:33.354990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred = grid.predict(X_test)\npred_proba = grid.predict_proba(X_test)[:,1]\nget_clf_eval(y_test, pred, pred_proba)","metadata":{"cell_id":"6cabc2190542418abb984f0aab38aa1d","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"33f17c09","execution_start":1658391778874,"execution_millis":95,"deepnote_cell_type":"code","deepnote_cell_height":289.5,"execution":{"iopub.status.busy":"2022-07-21T21:52:36.305999Z","iopub.execute_input":"2022-07-21T21:52:36.306415Z","iopub.status.idle":"2022-07-21T21:52:36.352837Z","shell.execute_reply.started":"2022-07-21T21:52:36.306373Z","shell.execute_reply":"2022-07-21T21:52:36.351923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"LightGBM Classification Modeling","metadata":{"cell_id":"dd65f2dfff23438db6e3c9d2f4e63a23","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"model = lgb.LGBMClassifier(\n n_estimators=400,\n num_leaves=20,\n min_data_in_leaf=60,\n learning_rate=0.01,\n boosting='gbdt',\n objective='binary',\n metric='auc',\n Is_training_metric=True,\n n_jobs=-1\n)","metadata":{"cell_id":"9c93512c0f5d48fcbafc4d05b91a5900","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"c3f9c8ba","execution_start":1658391904904,"execution_millis":0,"deepnote_cell_type":"code","deepnote_cell_height":261,"execution":{"iopub.status.busy":"2022-07-21T21:52:43.649544Z","iopub.execute_input":"2022-07-21T21:52:43.649971Z","iopub.status.idle":"2022-07-21T21:52:43.656394Z","shell.execute_reply.started":"2022-07-21T21:52:43.649922Z","shell.execute_reply":"2022-07-21T21:52:43.655054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.fit(X_train,y_train)","metadata":{"cell_id":"4bd382acd9204d07b382f3e40a7160cb","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"6982db56","execution_start":1658391915849,"execution_millis":2220,"deepnote_cell_type":"code","deepnote_cell_height":286.28125,"deepnote_output_heights":[null,98.71875],"execution":{"iopub.status.busy":"2022-07-21T21:52:46.413449Z","iopub.execute_input":"2022-07-21T21:52:46.413841Z","iopub.status.idle":"2022-07-21T21:52:46.562135Z","shell.execute_reply.started":"2022-07-21T21:52:46.413809Z","shell.execute_reply":"2022-07-21T21:52:46.560542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred = model.predict(X_test)\npred_proba = model.predict_proba(X_test)[:,1]\nget_clf_eval(y_test, pred, pred_proba)","metadata":{"cell_id":"d636bc73a5134372a60b96060261bb62","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"de1d0dc5","execution_start":1658391965419,"execution_millis":865,"deepnote_cell_type":"code","deepnote_cell_height":289.5,"execution":{"iopub.status.busy":"2022-07-21T21:52:54.070991Z","iopub.execute_input":"2022-07-21T21:52:54.071397Z","iopub.status.idle":"2022-07-21T21:52:54.099345Z","shell.execute_reply.started":"2022-07-21T21:52:54.071366Z","shell.execute_reply":"2022-07-21T21:52:54.098291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"SGD Classification Model","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import SGDClassifier\n\nclf_SGD = SGDClassifier(loss=\"squared_loss\", penalty=\"l2\", max_iter=4500,tol=-1000, random_state=1)\nclf_SGD.fit(X_train,y_train)\npred = clf_SGD.predict(X_test)\npred_proba = lg_reg.predict_proba(X_test)[:,1]\nget_clf_eval(y_test, pred, pred_proba)","metadata":{"execution":{"iopub.status.busy":"2022-07-21T22:06:08.000412Z","iopub.execute_input":"2022-07-21T22:06:08.000830Z","iopub.status.idle":"2022-07-21T22:06:08.179647Z","shell.execute_reply.started":"2022-07-21T22:06:08.000798Z","shell.execute_reply":"2022-07-21T22:06:08.178418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# After SMOTE","metadata":{"cell_id":"a1b9884f85e145ac93dfff56852bd6f5","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":82}},{"cell_type":"markdown","source":"Logistic Regression Model","metadata":{"cell_id":"e2b13f52704f45ffbd57a3562be5f53d","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"lg_reg = LogisticRegression()\n\nlg_reg.fit(X_over, y_over)\npred = lg_reg.predict(X_test)\npred_proba = lg_reg.predict_proba(X_test)[:,1]\nget_clf_eval(y_test, pred, pred_proba)","metadata":{"cell_id":"986688653bbe4ff09cd4ebeaebbdce51","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"632f6386","execution_start":1658393684164,"execution_millis":1,"deepnote_cell_type":"code","deepnote_cell_height":343.5,"execution":{"iopub.status.busy":"2022-07-21T21:52:58.324475Z","iopub.execute_input":"2022-07-21T21:52:58.324858Z","iopub.status.idle":"2022-07-21T21:52:58.351882Z","shell.execute_reply.started":"2022-07-21T21:52:58.324829Z","shell.execute_reply":"2022-07-21T21:52:58.350762Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Random Forest Model","metadata":{"cell_id":"c4f720aebd364e50bb295ffaa4aacb8d","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"rf_clf = RandomForestClassifier()\nparam = {'n_estimators' : [200],\n 'max_depth':[10],\n 'min_samples_split':[2],\n 'min_samples_leaf':[7]\n }","metadata":{"cell_id":"ce4db165a9d9472bb3170204e6d5f86c","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"b5620e73","execution_start":1658392259969,"execution_millis":1,"deepnote_cell_type":"code","deepnote_cell_height":171,"execution":{"iopub.status.busy":"2022-07-21T21:53:02.028275Z","iopub.execute_input":"2022-07-21T21:53:02.028688Z","iopub.status.idle":"2022-07-21T21:53:02.035144Z","shell.execute_reply.started":"2022-07-21T21:53:02.028653Z","shell.execute_reply":"2022-07-21T21:53:02.033918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grid = GridSearchCV(rf_clf,param_grid = param,scoring = 'accuracy',cv=5)\ngrid.fit(X_over ,y_over)\n\npred = grid.predict(X_test)\npred_proba = grid.predict_proba(X_test)[:,1]\nget_clf_eval(y_test, pred, pred_proba)","metadata":{"cell_id":"9a13e1e94d234087af206c36bd497aab","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"ae9a2fd9","execution_start":1658392287955,"execution_millis":3056,"deepnote_cell_type":"code","deepnote_cell_height":344,"execution":{"iopub.status.busy":"2022-07-21T21:53:06.547084Z","iopub.execute_input":"2022-07-21T21:53:06.547510Z","iopub.status.idle":"2022-07-21T21:53:09.017224Z","shell.execute_reply.started":"2022-07-21T21:53:06.547455Z","shell.execute_reply":"2022-07-21T21:53:09.015875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"LightGBM Classification Model","metadata":{"cell_id":"eedee5f239e04c4080525a9ad371bb08","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"model = lgb.LGBMClassifier(\n n_estimators=400,\n num_leaves=20,\n min_data_in_leaf=60,\n learning_rate=0.01,\n boosting='gbdt',\n objective='binary',\n metric='auc',\n Is_training_metric=True,\n n_jobs=-1\n)","metadata":{"cell_id":"ed4f968a90ae4beb95899249b0444c24","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"c3f9c8ba","execution_start":1658392370476,"execution_millis":1,"deepnote_cell_type":"code","deepnote_cell_height":261,"execution":{"iopub.status.busy":"2022-07-21T21:53:13.259714Z","iopub.execute_input":"2022-07-21T21:53:13.260076Z","iopub.status.idle":"2022-07-21T21:53:13.266369Z","shell.execute_reply.started":"2022-07-21T21:53:13.260048Z","shell.execute_reply":"2022-07-21T21:53:13.265043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.fit(X_over,y_over)","metadata":{"cell_id":"9506c6f786204676b5103c3191394bad","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"626ba69e","execution_start":1658392386163,"execution_millis":100,"deepnote_cell_type":"code","deepnote_cell_height":286.28125,"deepnote_output_heights":[null,98.71875],"execution":{"iopub.status.busy":"2022-07-21T21:53:17.717396Z","iopub.execute_input":"2022-07-21T21:53:17.717800Z","iopub.status.idle":"2022-07-21T21:53:17.821050Z","shell.execute_reply.started":"2022-07-21T21:53:17.717765Z","shell.execute_reply":"2022-07-21T21:53:17.820110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred = model.predict(X_test)\npred_proba = model.predict_proba(X_test)[:,1]\nget_clf_eval(y_test, pred, pred_proba)","metadata":{"cell_id":"d60c5ab6033c44baadcd5536d5b5e0a0","tags":[],"deepnote_to_be_reexecuted":false,"source_hash":"de1d0dc5","execution_start":1658392412612,"execution_millis":56,"deepnote_cell_type":"code","deepnote_cell_height":289.5,"execution":{"iopub.status.busy":"2022-07-21T21:53:21.307094Z","iopub.execute_input":"2022-07-21T21:53:21.307474Z","iopub.status.idle":"2022-07-21T21:53:21.347519Z","shell.execute_reply.started":"2022-07-21T21:53:21.307444Z","shell.execute_reply":"2022-07-21T21:53:21.346290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"SGD Classification Model","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import SGDClassifier\n\nclf_SGD = SGDClassifier(loss=\"squared_loss\", penalty=\"l2\", max_iter=4500,tol=-1000, random_state=1)\nclf_SGD.fit(X_over,y_over)\npred = clf_SGD.predict(X_test)\npred_proba = lg_reg.predict_proba(X_test)[:,1]\nget_clf_eval(y_test, pred, pred_proba)","metadata":{"execution":{"iopub.status.busy":"2022-07-21T22:07:16.911997Z","iopub.execute_input":"2022-07-21T22:07:16.912387Z","iopub.status.idle":"2022-07-21T22:07:17.114232Z","shell.execute_reply.started":"2022-07-21T22:07:16.912357Z","shell.execute_reply":"2022-07-21T22:07:17.113049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 6: Submission","metadata":{"cell_id":"48b014632ae043dbb3efc48bc37f5ca0","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":82}},{"cell_type":"markdown","source":"Re-run the best model (Logistic Regression after SMOTE) to update the 'pred' variable","metadata":{"cell_id":"e6ccc423ea7b4e4cac51c754d4c74149","tags":[],"deepnote_cell_type":"markdown","deepnote_cell_height":52.390625}},{"cell_type":"code","source":"# Create a list of columns to be used for the predictions\ntarget_test_columns = X_train.columns\ntarget_test_columns","metadata":{"execution":{"iopub.status.busy":"2022-07-21T22:09:48.452648Z","iopub.execute_input":"2022-07-21T22:09:48.453109Z","iopub.status.idle":"2022-07-21T22:09:48.461719Z","shell.execute_reply.started":"2022-07-21T22:09:48.453076Z","shell.execute_reply":"2022-07-21T22:09:48.460576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train.columns","metadata":{"execution":{"iopub.status.busy":"2022-07-21T22:09:56.286049Z","iopub.execute_input":"2022-07-21T22:09:56.286483Z","iopub.status.idle":"2022-07-21T22:09:56.294810Z","shell.execute_reply.started":"2022-07-21T22:09:56.286446Z","shell.execute_reply":"2022-07-21T22:09:56.293638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a list of columns to be used for the predictions\ntarget_test_columns = X_train.columns\ntarget_test_columns","metadata":{"execution":{"iopub.status.busy":"2022-07-21T22:10:01.056796Z","iopub.execute_input":"2022-07-21T22:10:01.057217Z","iopub.status.idle":"2022-07-21T22:10:01.064244Z","shell.execute_reply.started":"2022-07-21T22:10:01.057186Z","shell.execute_reply":"2022-07-21T22:10:01.063402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"titanic_test.columns","metadata":{"execution":{"iopub.status.busy":"2022-07-21T22:10:04.533483Z","iopub.execute_input":"2022-07-21T22:10:04.533875Z","iopub.status.idle":"2022-07-21T22:10:04.541221Z","shell.execute_reply.started":"2022-07-21T22:10:04.533845Z","shell.execute_reply":"2022-07-21T22:10:04.540306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"titanic_test_for_pred = titanic_test[target_test_columns]","metadata":{"execution":{"iopub.status.busy":"2022-07-21T22:10:07.605767Z","iopub.execute_input":"2022-07-21T22:10:07.606164Z","iopub.status.idle":"2022-07-21T22:10:07.613038Z","shell.execute_reply.started":"2022-07-21T22:10:07.606132Z","shell.execute_reply":"2022-07-21T22:10:07.611715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check null values\n\npd.DataFrame(titanic_test_for_pred.isnull().sum(), columns=[\"Null Count\"]).style.background_gradient(cmap='Blues')","metadata":{"execution":{"iopub.status.busy":"2022-07-21T22:10:12.409090Z","iopub.execute_input":"2022-07-21T22:10:12.409482Z","iopub.status.idle":"2022-07-21T22:10:12.426102Z","shell.execute_reply.started":"2022-07-21T22:10:12.409451Z","shell.execute_reply":"2022-07-21T22:10:12.425337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Imputing Fare with mean value","metadata":{}},{"cell_type":"code","source":"titanic_test_for_pred.loc[titanic_test_for_pred.Fare.isna(), 'Fare'] = titanic_test_for_pred.Fare.mean()\nprint(\"Missing values in Fare:\",titanic_test_for_pred.Fare.isna().sum())","metadata":{"execution":{"iopub.status.busy":"2022-07-21T22:10:16.253714Z","iopub.execute_input":"2022-07-21T22:10:16.254120Z","iopub.status.idle":"2022-07-21T22:10:16.263317Z","shell.execute_reply.started":"2022-07-21T22:10:16.254084Z","shell.execute_reply":"2022-07-21T22:10:16.262084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred = clf_SGD.predict(titanic_test_for_pred)\n\npred[:20]","metadata":{"execution":{"iopub.status.busy":"2022-07-21T22:13:21.400060Z","iopub.execute_input":"2022-07-21T22:13:21.400482Z","iopub.status.idle":"2022-07-21T22:13:21.410379Z","shell.execute_reply.started":"2022-07-21T22:13:21.400445Z","shell.execute_reply":"2022-07-21T22:13:21.409556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a submisison dataframe and append the relevant columns\nsubmission = pd.DataFrame()\nsubmission['PassengerId'] = titanic_test['PassengerId']\nsubmission['Survived'] = pred # our model predictions on the test dataset\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T22:13:47.260541Z","iopub.execute_input":"2022-07-21T22:13:47.261419Z","iopub.status.idle":"2022-07-21T22:13:47.275293Z","shell.execute_reply.started":"2022-07-21T22:13:47.261375Z","shell.execute_reply":"2022-07-21T22:13:47.274197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checking for inconsistencies\n\nif len(submission) == len(titanic_test):\n print(\"Submission dataframe is the same length as test ({} rows).\".format(len(submission)))\nelse:\n print(\"Dataframes mismatched, won't be able to submit to Kaggle.\")","metadata":{"execution":{"iopub.status.busy":"2022-07-21T22:13:52.040211Z","iopub.execute_input":"2022-07-21T22:13:52.040596Z","iopub.status.idle":"2022-07-21T22:13:52.047445Z","shell.execute_reply.started":"2022-07-21T22:13:52.040566Z","shell.execute_reply":"2022-07-21T22:13:52.046322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('/kaggle/working/submission.csv', index=False)\nprint('Submission CSV is ready!')","metadata":{"execution":{"iopub.status.busy":"2022-07-21T22:21:18.282125Z","iopub.execute_input":"2022-07-21T22:21:18.282573Z","iopub.status.idle":"2022-07-21T22:21:18.291143Z","shell.execute_reply.started":"2022-07-21T22:21:18.282536Z","shell.execute_reply":"2022-07-21T22:21:18.290223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\nCreated in deepnote.com \nCreated in Deepnote","metadata":{"tags":[],"created_in_deepnote_cell":true,"deepnote_cell_type":"markdown"}}]}