{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Introduction

\n\nThis notebook attempts to perform **Exploratory Data Analysis** on the Titanic dataset and eventually train a Machine Learning model on it and fine-tune the model using Randomized Search.","metadata":{}},{"cell_type":"code","source":"# useful imports\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# for comparing classifiers\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.linear_model import SGDClassifier\nfrom sklearn.svm import SVC\n\n# for evaluating\nfrom sklearn.model_selection import cross_val_predict, cross_val_score\nfrom sklearn.metrics import RocCurveDisplay, PrecisionRecallDisplay\nfrom sklearn.metrics import classification_report","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-10T20:26:50.837063Z","iopub.execute_input":"2022-07-10T20:26:50.837517Z","iopub.status.idle":"2022-07-10T20:26:50.847415Z","shell.execute_reply.started":"2022-07-10T20:26:50.837479Z","shell.execute_reply":"2022-07-10T20:26:50.846072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-10T20:26:50.850538Z","iopub.execute_input":"2022-07-10T20:26:50.851478Z","iopub.status.idle":"2022-07-10T20:26:50.863511Z","shell.execute_reply.started":"2022-07-10T20:26:50.851428Z","shell.execute_reply":"2022-07-10T20:26:50.862248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Knowing the Dataset

\n\nLet's first know what information the dataset contains.\n\nThe data has columns:\n- **Survival**: Whether a passenger survived or not (0 or 1)\n- **Pclass**: The socio-ecomonic class\n - Upper: 1\n - Middle: 2\n - Lower: 3\n- **Sex**: Gender of the passenger (Male or Female)\n- **Age**: Age in years (Age is fractional if less than 1. If the age is estimated, it is in the form of xx.5)\n- **SibSp**: Number of siblings / spouses aboard the Titanic\n- **Parch**: Number of parents / children aboard the Titanic\n- **Ticket**: Ticket number\n- **Fare**: Passenger fare\n- **Cabin**: Cabin number\n- **Embarked**: Port of Embarkation\n - C: Cherbourg\n - Q: Queenstown\n - S: Southampton","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/titanic/train.csv\")\ntest = pd.read_csv(\"/kaggle/input/titanic/test.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:50.865483Z","iopub.execute_input":"2022-07-10T20:26:50.866722Z","iopub.status.idle":"2022-07-10T20:26:50.889077Z","shell.execute_reply.started":"2022-07-10T20:26:50.866645Z","shell.execute_reply":"2022-07-10T20:26:50.887658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:50.893545Z","iopub.execute_input":"2022-07-10T20:26:50.894211Z","iopub.status.idle":"2022-07-10T20:26:50.920093Z","shell.execute_reply.started":"2022-07-10T20:26:50.894116Z","shell.execute_reply":"2022-07-10T20:26:50.918763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To gain some more insight into the dataframe...","metadata":{}},{"cell_type":"code","source":"train.describe()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:50.921866Z","iopub.execute_input":"2022-07-10T20:26:50.922246Z","iopub.status.idle":"2022-07-10T20:26:50.965358Z","shell.execute_reply.started":"2022-07-10T20:26:50.922210Z","shell.execute_reply":"2022-07-10T20:26:50.964404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.info()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:50.966779Z","iopub.execute_input":"2022-07-10T20:26:50.967307Z","iopub.status.idle":"2022-07-10T20:26:50.985666Z","shell.execute_reply.started":"2022-07-10T20:26:50.967274Z","shell.execute_reply":"2022-07-10T20:26:50.984425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We observe here that some columns have missing data.\nThese columns are:\n- **Age**\n- **Cabin**\n- **Embarked**\n\n> Note: Rather than treating the features **Sex** and **Embarked** as `object` dtype. Let's convert them into categorical features to save some memory. Also the features **Cabin**, **Name** and **Ticket** won't be included in the final prepared training set so we can leave them as it is.","metadata":{}},{"cell_type":"code","source":"attribs = [\"Sex\", \"Embarked\"]\n\ndef convert_cat(df, attrs):\n for col in attrs:\n df[col] = df[col].astype('category')\n \n# Use the above function for both train and test sets\nconvert_cat(train, attribs)\nconvert_cat(test, attribs)","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:50.987646Z","iopub.execute_input":"2022-07-10T20:26:50.989014Z","iopub.status.idle":"2022-07-10T20:26:51.001739Z","shell.execute_reply.started":"2022-07-10T20:26:50.988969Z","shell.execute_reply":"2022-07-10T20:26:51.000680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, let us look at the count of missing values, some columns contain.","metadata":{}},{"cell_type":"code","source":"def count_na(df, col):\n print(f\"Null values in {col}: \", df[col].isna().sum())\n \ncount_na(train, \"Age\")\ncount_na(train, \"Cabin\")\ncount_na(train, \"Embarked\")","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:51.003655Z","iopub.execute_input":"2022-07-10T20:26:51.004069Z","iopub.status.idle":"2022-07-10T20:26:51.019379Z","shell.execute_reply.started":"2022-07-10T20:26:51.004035Z","shell.execute_reply":"2022-07-10T20:26:51.018174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Data Visualization

\n\nIn this section, we will start visualizing the features of the dataset one by one.\nFirstly, **Univariate** feature visualization will be done, then we will move onto **Multivariate** feature visualization.\n\n> To learn more about what **graphs** are useful for what **data-types**, check out this notebook here:\n[Statistical Data Types and Graphs (using Seaborn)](https://www.kaggle.com/code/maharshipandya/statistical-data-types-and-graphs-using-seaborn)","metadata":{}},{"cell_type":"code","source":"# Setting some styles\nsns.set_style(\"darkgrid\")\nsns.set_palette(\"viridis\")","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"execution":{"iopub.status.busy":"2022-07-10T20:26:51.021345Z","iopub.execute_input":"2022-07-10T20:26:51.022199Z","iopub.status.idle":"2022-07-10T20:26:51.029103Z","shell.execute_reply.started":"2022-07-10T20:26:51.022151Z","shell.execute_reply":"2022-07-10T20:26:51.027780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Univariate Analysis

","metadata":{}},{"cell_type":"markdown","source":"#### Analysis of Survived\n\nA Histogram and a Pie chart will be two useful plots to analyse the `Survived` column as it is a categorical feature. Usefulness in the sense, both the plots will allow us to observe the distribution of each category in the feature. ","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(20, 7))\n\nsns.histplot(data=train, x=\"Survived\", stat=\"percent\", bins=3, multiple=\"stack\", ax=ax[0])\ntrain[\"Survived\"].value_counts().plot.pie(explode=[0.1, 0], autopct=\"%1.1f%%\", shadow=True, ax=ax[1])\n\nplt.show()","metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"execution":{"iopub.status.busy":"2022-07-10T20:26:51.032166Z","iopub.execute_input":"2022-07-10T20:26:51.033195Z","iopub.status.idle":"2022-07-10T20:26:51.437159Z","shell.execute_reply.started":"2022-07-10T20:26:51.033148Z","shell.execute_reply":"2022-07-10T20:26:51.435723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We observe from the above plots that\n- **Only 38.4% of passengers survived the disaster**\n- **While 61.6% of passengers didn't survive!**","metadata":{}},{"cell_type":"markdown","source":"#### Analysis of Sex\n\nSimilar to `Survived`, a histogram and a pie plot will provide us with distributions of categories since `Sex` is also a categorical feature.","metadata":{}},{"cell_type":"code","source":"fig1, ax1 = plt.subplots(1, 2, figsize=(20, 7))\n\nsns.histplot(data=train, x=\"Sex\", ax=ax1[0])\ntrain[\"Sex\"].value_counts().plot.pie(shadow=True, autopct=\"%1.1f%%\", explode=[0.1, 0], ax=ax1[1])\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:51.439948Z","iopub.execute_input":"2022-07-10T20:26:51.441589Z","iopub.status.idle":"2022-07-10T20:26:51.765166Z","shell.execute_reply.started":"2022-07-10T20:26:51.441531Z","shell.execute_reply":"2022-07-10T20:26:51.763731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- There are approximately 65% of **Male** passengers\n- Only 35.2% of passengers are **Female**","metadata":{}},{"cell_type":"markdown","source":"#### Analysis of Pclass\n\n`Pclass` is a categorical feature which is **ordinal** in nature.\nFor this, **Bar charts** are useful plots.","metadata":{}},{"cell_type":"code","source":"fig2, ax2 = plt.subplots(1, 2, figsize=(20, 7))\n\nsns.countplot(data=train, x=\"Pclass\", ax=ax2[0])\ntrain[\"Pclass\"].value_counts().plot.pie(shadow=True, autopct=\"%1.1f%%\", ax=ax2[1])\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:51.766798Z","iopub.execute_input":"2022-07-10T20:26:51.767435Z","iopub.status.idle":"2022-07-10T20:26:52.097030Z","shell.execute_reply.started":"2022-07-10T20:26:51.767396Z","shell.execute_reply":"2022-07-10T20:26:52.095786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Analysis of Age\n\nAs observed, `Age` is a **Quantitative** feature. There are many plots to analyse these type of data. Histograms and Box plots are useful to know how the data is distributed.","metadata":{}},{"cell_type":"code","source":"fig3, ax3 = plt.subplots(1, 2, figsize=(20, 7))\n\nsns.histplot(data=train, x=\"Age\", ax=ax3[0], kde=True)\nsns.boxplot(data=train, x=\"Age\", ax=ax3[1])\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:52.100180Z","iopub.execute_input":"2022-07-10T20:26:52.101142Z","iopub.status.idle":"2022-07-10T20:26:52.935940Z","shell.execute_reply.started":"2022-07-10T20:26:52.101097Z","shell.execute_reply":"2022-07-10T20:26:52.934605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Analysis on Fare\n\nSimilar to `Age`, `Fare` is also a Quantitative feature.","metadata":{}},{"cell_type":"code","source":"fig4, ax4 = plt.subplots(1, 2, figsize=(20, 7))\n\nsns.histplot(data=train, x=\"Fare\", ax=ax4[0], kde=True)\nsns.boxplot(data=train, x=\"Fare\", ax=ax4[1])\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:52.937947Z","iopub.execute_input":"2022-07-10T20:26:52.938867Z","iopub.status.idle":"2022-07-10T20:26:53.548711Z","shell.execute_reply.started":"2022-07-10T20:26:52.938809Z","shell.execute_reply":"2022-07-10T20:26:53.547479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The histogram for `Fare` is quite skewed. Let us observe some facts about `Fare`.","metadata":{}},{"cell_type":"code","source":"max_fare, min_fare = train[\"Fare\"].max(), train[\"Fare\"].min()\n\nprint(f\"Number of passengers who paid ${min_fare}: \", train[train[\"Fare\"] == min_fare].shape[0])\nprint(f\"Number of passengers who paid ${max_fare}: \", train[train[\"Fare\"] == max_fare].shape[0])\nprint(f\"Fare given by maximum number of passengers: $\", list(dict(train[\"Fare\"].value_counts()).keys())[0])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-10T20:26:53.550316Z","iopub.execute_input":"2022-07-10T20:26:53.550660Z","iopub.status.idle":"2022-07-10T20:26:53.566858Z","shell.execute_reply.started":"2022-07-10T20:26:53.550628Z","shell.execute_reply":"2022-07-10T20:26:53.565730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We observe that:\n\n- **Only 3 people paid 512 dollars to be on Titanic**\n- **15 people paid no fare to be on Titanic. I wonder who they were?**\n- **Maximum people paid approximately 8 dollars**","metadata":{}},{"cell_type":"markdown","source":"People who paid no fare to be on titanic:","metadata":{}},{"cell_type":"code","source":"train[train[\"Fare\"] == min_fare]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-10T20:26:53.568537Z","iopub.execute_input":"2022-07-10T20:26:53.569283Z","iopub.status.idle":"2022-07-10T20:26:53.596117Z","shell.execute_reply.started":"2022-07-10T20:26:53.569241Z","shell.execute_reply":"2022-07-10T20:26:53.594468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Analysis on Embarked\n\nAgain like `Survived` and `Sex`, `Embarked` is also a categorical feature. So Bar plot and Pie chart is the way to go.","metadata":{}},{"cell_type":"code","source":"fig5, ax5 = plt.subplots(1, 2, figsize=(20, 7))\n\nsns.countplot(data=train, x=\"Embarked\", ax=ax5[0])\ntrain[\"Embarked\"].value_counts().plot.pie(ax=ax5[1], autopct=\"%1.1f%%\",\n explode=(0.1, 0, 0), shadow=True)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:53.598028Z","iopub.execute_input":"2022-07-10T20:26:53.598779Z","iopub.status.idle":"2022-07-10T20:26:53.910938Z","shell.execute_reply.started":"2022-07-10T20:26:53.598727Z","shell.execute_reply":"2022-07-10T20:26:53.909589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As seen from the above plots, most passengers, approximately **72.4%**, boarded the Titanic from Southampton.","metadata":{}},{"cell_type":"markdown","source":"#### Analysis on Parch and SibSp\n\nBoth of these features are Quantitative in nature but has discrete values. So Bar plots will be useful to gain insights about their structure.","metadata":{}},{"cell_type":"code","source":"fig6, ax6 = plt.subplots(2, 2, figsize=(20, 10))\n\n# SibSp\nsns.countplot(data=train, x=\"SibSp\", ax=ax6[0, 0]).set_title(\"Siblings and Spouses\")\ntrain[\"SibSp\"].value_counts().plot.pie(ax=ax6[0, 1], shadow=True, title=\"Distribution of SibSp\")\n\n# Parch\nsns.countplot(data=train, x=\"Parch\", ax=ax6[1, 0]).set_title(\"Parents and Children\")\ntrain[\"Parch\"].value_counts().plot.pie(ax=ax6[1, 1], shadow=True, title=\"Distribution of Parch\")\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:53.912740Z","iopub.execute_input":"2022-07-10T20:26:53.913246Z","iopub.status.idle":"2022-07-10T20:26:54.616759Z","shell.execute_reply.started":"2022-07-10T20:26:53.913194Z","shell.execute_reply":"2022-07-10T20:26:54.615767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Multivariate Analysis

\n\nIn this section we will visualize two or more features together which comes under **Multivariate Analysis**.","metadata":{}},{"cell_type":"markdown","source":"#### Analysis of Survived and Pclass\n\nUsing seaborn, we will plot a Histogram of `Pclass` having hue based on whether they survived or not.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20, 7))\n\nsns.histplot(data=train, x=\"Pclass\", hue=\"Survived\", multiple=\"stack\", ax=ax).set_title(\"Classes Survival Stat\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:54.622035Z","iopub.execute_input":"2022-07-10T20:26:54.623201Z","iopub.status.idle":"2022-07-10T20:26:55.023960Z","shell.execute_reply.started":"2022-07-10T20:26:54.623135Z","shell.execute_reply":"2022-07-10T20:26:55.022801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Hmm, people of the Upper class (class 1) survived more than Middle or Lower classes. Maybe Upper class people were favoured more (on the lifeboats?) over Lower and Middle classes.","metadata":{}},{"cell_type":"code","source":"sur_upper = train[(train[\"Survived\"] == 1) & (train[\"Pclass\"] == 1)].shape[0]\nsur_middle = train[(train[\"Survived\"] == 1) & (train[\"Pclass\"] == 2)].shape[0]\nsur_lower = train[(train[\"Survived\"] == 1) & (train[\"Pclass\"] == 3)].shape[0]\n\nprint(\"Upper class survival: \", sur_upper)\nprint(\"Middle class survival: \", sur_middle)\nprint(\"Lower class survival: \", sur_lower)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-10T20:26:55.026039Z","iopub.execute_input":"2022-07-10T20:26:55.026565Z","iopub.status.idle":"2022-07-10T20:26:55.041199Z","shell.execute_reply.started":"2022-07-10T20:26:55.026513Z","shell.execute_reply":"2022-07-10T20:26:55.039554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Analysis on Survived and Sex\n\nLet's plot a Pie chart to show the distribution of which sex/gender survived more.","metadata":{}},{"cell_type":"code","source":"df_m = train[(train[\"Survived\"] == 1) & (train[\"Sex\"] == \"male\")]\ndf_f = train[(train[\"Survived\"] == 1) & (train[\"Sex\"] == \"female\")]\n\ndf_sur = pd.concat([df_m, df_f])\ndf_sur[\"Sex\"].value_counts().plot.pie(explode=[0, 0.1], shadow=True, autopct=\"%1.1f%%\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:55.043293Z","iopub.execute_input":"2022-07-10T20:26:55.044275Z","iopub.status.idle":"2022-07-10T20:26:55.176280Z","shell.execute_reply.started":"2022-07-10T20:26:55.044217Z","shell.execute_reply":"2022-07-10T20:26:55.174466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the above Pie plot, we observe that women survived the Titanic disaster more than the men. Only 32% of men survived, while almost 68% women survived.","metadata":{}},{"cell_type":"markdown","source":"#### Analysis on Survived and Age\n\nSince `Age` is a continuous feature, we plot a histogram with hue based on `Survived`.","metadata":{}},{"cell_type":"code","source":"fig7, ax7 = plt.subplots(figsize=(20, 8))\n\nsns.histplot(data=train, x=\"Age\", hue=\"Survived\", multiple=\"stack\", kde=True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:55.178678Z","iopub.execute_input":"2022-07-10T20:26:55.179716Z","iopub.status.idle":"2022-07-10T20:26:55.685708Z","shell.execute_reply.started":"2022-07-10T20:26:55.179638Z","shell.execute_reply":"2022-07-10T20:26:55.684684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that in the age range 0-10, the passengers who survived are greater than non survivors. Assuming that the kids (or younger passengers) were favoured more for the lifeboats, first?","metadata":{"execution":{"iopub.status.busy":"2022-07-10T16:39:43.596373Z","iopub.execute_input":"2022-07-10T16:39:43.598031Z","iopub.status.idle":"2022-07-10T16:39:43.610124Z","shell.execute_reply.started":"2022-07-10T16:39:43.597954Z","shell.execute_reply":"2022-07-10T16:39:43.606776Z"}}},{"cell_type":"code","source":"print(\"Kids survived in age ranges: \")\nprint(\"Age 0-4:\", train[(train[\"Age\"] < 4) & (train[\"Survived\"] == 1)].shape[0])\nprint(\"Age 4-7:\", train[(train[\"Age\"] >= 4) & (train[\"Age\"] < 7) & (train[\"Survived\"] == 1)].shape[0])\nprint(\"Age 7-10:\", train[(train[\"Age\"] >= 7) & (train[\"Age\"] <= 10) & (train[\"Survived\"] == 1)].shape[0])\n\nprint(\"\\nKids NOT survived in age ranges: \")\nprint(\"Age 0-4:\", train[(train[\"Age\"] < 4) & (train[\"Survived\"] == 0)].shape[0])\nprint(\"Age 4-7:\", train[(train[\"Age\"] >= 4) & (train[\"Age\"] < 7) & (train[\"Survived\"] == 0)].shape[0])\nprint(\"Age 7-10:\", train[(train[\"Age\"] >= 7) & (train[\"Age\"] <= 10) & (train[\"Survived\"] == 0)].shape[0])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-10T20:26:55.687049Z","iopub.execute_input":"2022-07-10T20:26:55.688292Z","iopub.status.idle":"2022-07-10T20:26:55.707474Z","shell.execute_reply.started":"2022-07-10T20:26:55.688249Z","shell.execute_reply":"2022-07-10T20:26:55.706223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Analyzing Correlations

\n\nWe visualized the data to gain some insights on the data. Now its time to analyse correlations between every feature, using a correlation matrix.","metadata":{}},{"cell_type":"code","source":"# The 2D correlation matrix\ncorr = train.corr()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:55.708992Z","iopub.execute_input":"2022-07-10T20:26:55.709351Z","iopub.status.idle":"2022-07-10T20:26:55.716107Z","shell.execute_reply.started":"2022-07-10T20:26:55.709319Z","shell.execute_reply":"2022-07-10T20:26:55.714912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plotting the heatmap of corr\n\nfig, ax = plt.subplots(figsize=(20, 7))\ndataplot = sns.heatmap(data=corr, annot=True, ax=ax)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:55.718145Z","iopub.execute_input":"2022-07-10T20:26:55.719547Z","iopub.status.idle":"2022-07-10T20:26:56.267313Z","shell.execute_reply.started":"2022-07-10T20:26:55.719487Z","shell.execute_reply":"2022-07-10T20:26:56.266056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We observe here that\n\n- `Pclass` has considerable amount of negative correlation with `Survived`. This is true as seen above from visualizing the data i.e. Upper class people survived more than the Middle or Lower class people.\n- `Fare` has a considerable amount of positive correlation with `Survived`. As fare increases the chances of survival kinda increases.","metadata":{}},{"cell_type":"markdown","source":"

Preparing the Data

\n\nIts now time to prepare the data for machine learning algorithms to train on. There are a few things we need to do.\n\n- **Fill in the missing values (using an imputer)**\n- **Handle categorical and numerical variables (One hot encoding)**\n- **Use custom transformers (Using scikit-learn API)**\n\nFor this, we need to create different pipelines for numerical and categorical attributes.","metadata":{}},{"cell_type":"code","source":"df = train.drop(\"Survived\", axis=1)\nlabels = train[\"Survived\"].copy()\n\nprint(labels)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-10T20:26:56.269462Z","iopub.execute_input":"2022-07-10T20:26:56.270355Z","iopub.status.idle":"2022-07-10T20:26:56.281127Z","shell.execute_reply.started":"2022-07-10T20:26:56.270303Z","shell.execute_reply":"2022-07-10T20:26:56.279750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using scikit-learn API, we will create some custom transformers to ease our task when creating a pipeline.\n\nThere will be 2 custom transformers in this case:\n\n- **DataFrameSelector**: To select important features only\n- **AttribAdder**: A kind of feature engineering to create a new feature `famSize` from `Parch` and `SibSp`\n\n> Note: This notebook does not dwell into the topics of Feature Engineering to keep things a bit simple","metadata":{}},{"cell_type":"code","source":"# Custom transformer to sample only the useful attributes\nfrom sklearn.base import BaseEstimator, TransformerMixin\n\nclass DataFrameSelector(BaseEstimator, TransformerMixin):\n def __init__(self, attrs):\n self.attrs = attrs\n def fit(self, X, y=None):\n return self\n def transform(self, X, y=None):\n return X[self.attrs].values","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:56.282889Z","iopub.execute_input":"2022-07-10T20:26:56.283569Z","iopub.status.idle":"2022-07-10T20:26:56.293283Z","shell.execute_reply.started":"2022-07-10T20:26:56.283530Z","shell.execute_reply":"2022-07-10T20:26:56.292211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We need to drop the usless or missing features. Features like\n\n- **Ticket**\n- **PassengerId**\n- **Cabin**\n- **Name**\n\nfall under these.","metadata":{}},{"cell_type":"markdown","source":"We create two arrays for selecting important **numerical** and **categorical** attributes.","metadata":{}},{"cell_type":"code","source":"num_attrs = [\"Pclass\", \"Age\", \"SibSp\", \"Parch\", \"Fare\"]\ncat_attrs = [\"Sex\", \"Embarked\"]","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:56.295325Z","iopub.execute_input":"2022-07-10T20:26:56.295866Z","iopub.status.idle":"2022-07-10T20:26:56.305139Z","shell.execute_reply.started":"2022-07-10T20:26:56.295824Z","shell.execute_reply":"2022-07-10T20:26:56.304134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Custom transformer to add Parch and SibSp as FamSize\nsibsp_ix, parch_ix = 2, 3\n\nclass AttribAdder(BaseEstimator, TransformerMixin):\n def __init__(self):\n pass\n def fit(self, X, y=None):\n return self\n def transform(self, X, y=None):\n famSize = X[:, sibsp_ix] + X[:, parch_ix] + 1\n return np.c_[X, famSize]","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:56.306276Z","iopub.execute_input":"2022-07-10T20:26:56.307156Z","iopub.status.idle":"2022-07-10T20:26:56.318744Z","shell.execute_reply.started":"2022-07-10T20:26:56.307108Z","shell.execute_reply":"2022-07-10T20:26:56.317757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Numerical Pipeline\n\nWe will perform 3 tasks in this pipeline:\n\n- **Select numerical columns**\n- **Imputer for missing values**\n- **Attributes adder**","metadata":{}},{"cell_type":"code","source":"from sklearn.impute import SimpleImputer\nfrom sklearn.pipeline import Pipeline\n\nnum_pipeline = Pipeline([\n (\"selector\", DataFrameSelector(num_attrs)),\n (\"imputer\", SimpleImputer(strategy=\"median\")),\n (\"attrib_adder\", AttribAdder())\n])","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:56.320160Z","iopub.execute_input":"2022-07-10T20:26:56.320748Z","iopub.status.idle":"2022-07-10T20:26:56.331168Z","shell.execute_reply.started":"2022-07-10T20:26:56.320684Z","shell.execute_reply":"2022-07-10T20:26:56.330111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Categorical Pipeline\n\nWe will perform 3 tasks in this pipeline:\n\n- **Select categorical features**\n- **Impute missing values using `most frequent`**\n- **One hot encoding to convert the categories to one hot vectors**\n\n> Know more about [One Hot Encoding](https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/)","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import OneHotEncoder\n\ncat_pipeline = Pipeline([\n (\"selector\", DataFrameSelector(cat_attrs)),\n (\"imputer\", SimpleImputer(strategy=\"most_frequent\")),\n (\"label_binarize\", OneHotEncoder(sparse=False))\n])","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:56.332473Z","iopub.execute_input":"2022-07-10T20:26:56.333167Z","iopub.status.idle":"2022-07-10T20:26:56.345819Z","shell.execute_reply.started":"2022-07-10T20:26:56.333082Z","shell.execute_reply":"2022-07-10T20:26:56.344255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us combine these two pipelines, using **FeatureUnion** and fit it on the training dataframe.","metadata":{}},{"cell_type":"code","source":"from sklearn.pipeline import FeatureUnion\n\nfull_pipeline = FeatureUnion([\n (\"num\", num_pipeline),\n (\"cat\", cat_pipeline)\n])\n\n# Use fit transfrom on full pipeline\ntitanic_prepared = full_pipeline.fit_transform(df)\ntitanic_prepared_df = pd.DataFrame(titanic_prepared, columns=[\n \"Pclass\", \"Age\", \"SibSp\", \"Parch\", \"Fare\", \"FamSize\", \"Female\", \"Male\", \"C\", \"Q\", \"S\"\n])\ntitanic_prepared_df","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:56.347684Z","iopub.execute_input":"2022-07-10T20:26:56.348513Z","iopub.status.idle":"2022-07-10T20:26:56.402714Z","shell.execute_reply.started":"2022-07-10T20:26:56.348454Z","shell.execute_reply":"2022-07-10T20:26:56.401292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Classification

\n\nData preparation is done, now its time to run ML algorithms on the preprocessed data. We will write a function to compare few models, by a technique called **[Cross Validation](https://en.wikipedia.org/wiki/Cross-validation_(statistics))**.","metadata":{}},{"cell_type":"code","source":"estimators = [SVC(), RandomForestClassifier(), KNeighborsClassifier(), SGDClassifier()]\nfig, ax = plt.subplots(len(estimators), 2, figsize=(20, 20))\n\ndef run_compare(estis, cv=3):\n for esti_ix, esti in enumerate(estis):\n esti_preds = cross_val_predict(esti, titanic_prepared, labels, cv=cv)\n \n esti_pr_disp = PrecisionRecallDisplay.from_predictions(labels,\n esti_preds, ax=ax[esti_ix][0], name=esti)\n esti_roc_disp = RocCurveDisplay.from_predictions(labels,\n esti_preds, ax=ax[esti_ix][1], name=esti)\n \n print(f\"\\nClassification Report for {esti}:\")\n print(classification_report(labels, esti_preds))\n\n# Run\nrun_compare(estimators, cv=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:56.404748Z","iopub.execute_input":"2022-07-10T20:26:56.405236Z","iopub.status.idle":"2022-07-10T20:26:59.415495Z","shell.execute_reply.started":"2022-07-10T20:26:56.405188Z","shell.execute_reply":"2022-07-10T20:26:59.414154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We observe that **RandomForestClassifier** performs the best out of all. So we will go forward with that.","metadata":{}},{"cell_type":"markdown","source":"

Fine-tuning Random Forest Classifier

\n\nWe will use **[Randomized Searching](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html)** to find good hyper-parameter values for our classifier.","metadata":{}},{"cell_type":"code","source":"# Parameters of random forest classifier\nn_estimators = np.linspace(50, 300, int((300 - 50) / 20), dtype=int)\nmax_depth = [1, 5, 10, 50, 100, 200, 300]\nmin_samples_split = [2, 4, 6]\nmax_features = [\"sqrt\", \"log2\"]\nbootstrap = [True, False]\n\ndistributions = {\n \"n_estimators\": n_estimators,\n \"max_depth\": max_depth,\n \"min_samples_split\": min_samples_split,\n \"max_features\": max_features,\n \"bootstrap\": bootstrap\n}","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:59.419303Z","iopub.execute_input":"2022-07-10T20:26:59.420302Z","iopub.status.idle":"2022-07-10T20:26:59.433268Z","shell.execute_reply.started":"2022-07-10T20:26:59.420242Z","shell.execute_reply":"2022-07-10T20:26:59.431485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Randomised search cv\nfrom sklearn.model_selection import RandomizedSearchCV\n\nrfc = RandomForestClassifier()\nrandom_search_cv = RandomizedSearchCV(\n rfc,\n param_distributions=distributions,\n n_iter=30,\n cv=5,\n n_jobs=4\n)\n\nsearch = random_search_cv.fit(titanic_prepared, labels)","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:26:59.435823Z","iopub.execute_input":"2022-07-10T20:26:59.436364Z","iopub.status.idle":"2022-07-10T20:27:21.887430Z","shell.execute_reply.started":"2022-07-10T20:26:59.436313Z","shell.execute_reply":"2022-07-10T20:27:21.886102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The results of this Randomized Search are stored in a dictionary named `cv_results_`. Let us print these results just to get an idea of what parameters were tested by our Randomized Search.","metadata":{}},{"cell_type":"code","source":"cvres = search.cv_results_\n\nfor score, params, rank in zip(cvres[\"mean_test_score\"], cvres[\"params\"], cvres[\"rank_test_score\"]):\n print(score, params, rank)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-10T20:27:21.888958Z","iopub.execute_input":"2022-07-10T20:27:21.889335Z","iopub.status.idle":"2022-07-10T20:27:21.897455Z","shell.execute_reply.started":"2022-07-10T20:27:21.889299Z","shell.execute_reply":"2022-07-10T20:27:21.896225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Also, the best estimator out of these tested ones is stored in a variable called `best_estimator_`. We can use this estimator as our fine-tuned model.","metadata":{}},{"cell_type":"code","source":"rfc_finetuned = search.best_estimator_\nbest_preds = cross_val_predict(rfc_finetuned, titanic_prepared, labels, cv=5)\n\nfig, ax = plt.subplots(1, 2, figsize=(20, 10))\nPrecisionRecallDisplay.from_predictions(labels, best_preds, ax=ax[0])\nRocCurveDisplay.from_predictions(labels, best_preds, ax=ax[1])\n\nprint(classification_report(labels, best_preds, digits=5))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T20:27:21.903193Z","iopub.execute_input":"2022-07-10T20:27:21.903604Z","iopub.status.idle":"2022-07-10T20:27:24.712627Z","shell.execute_reply.started":"2022-07-10T20:27:21.903567Z","shell.execute_reply":"2022-07-10T20:27:24.711327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the fine-tuned Random Forest Classifier has:\n\n- **An average precision of 82%**\n- **An average recall of about 81%**\n\nwhich is totally not bad than the previous bare-bones classifier!","metadata":{}},{"cell_type":"markdown","source":"**If this notebook helped you a slightest bit, do consider upvoting it and leaving a comment below! Feel free to extend this notebook with more knowledge. Thank you! ❤️**","metadata":{}}]}