{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Space Ship Titanic\n\nIn this competition the task is to predict whether a passenger was transported to an alternate dimension during the Spaceship Titanic's collision with the spacetime anomaly. A set of personal records recovered from the ship's damaged computer system are given to help make these predictions.\n\nhttps://www.kaggle.com/competitions/spaceship-titanic","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"## EDA and Missing Values Handling\n\nIn this notebook, we'll conduct a brief data analysis of the spaceship titanic data-set, and cover how the missing values of the data can be handled. But first a quick overview of the data-set.","metadata":{}},{"cell_type":"markdown","source":"### File and Data Field Descriptions\n\n- `PassengerId` - A unique Id for each passenger. Each Id takes the form gggg_pp where gggg indicates a group the passenger is travelling with and pp is their number within the group. People in a group are often family members, but not always.\n- `HomePlanet` - The planet the passenger departed from, typically their planet of permanent residence.\n- `CryoSleep` - Indicates whether the passenger elected to be put into suspended animation for the duration of the voyage. Passengers in cryosleep are confined to their cabins.\n- `Cabin` - The cabin number where the passenger is staying. Takes the form deck/num/side, where side can be either P for Port or S for Starboard.\n- `Destination` - The planet the passenger will be debarking to.\n- `Age` - The age of the passenger.\n- `VIP` - Whether the passenger has paid for special VIP service during the voyage.\n- `RoomService`, `FoodCourt`, `ShoppingMall`, `Spa`, `VRDeck` - Amount the passenger has billed at each of the Spaceship Titanic's many luxury amenities.\n- `Name` - The first and last names of the passenger.\n- `Transported` - Whether the passenger was transported to another dimension. This is the target, the column you are trying to predict.","metadata":{}},{"cell_type":"code","source":"!pip install missingno","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:11:26.823423Z","iopub.execute_input":"2022-08-04T17:11:26.824040Z","iopub.status.idle":"2022-08-04T17:11:38.144664Z","shell.execute_reply.started":"2022-08-04T17:11:26.823909Z","shell.execute_reply":"2022-08-04T17:11:38.143040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport missingno as msno\n\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\n\nsns.set_style(\"whitegrid\")\n\n%matplotlib inline\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:11:38.146846Z","iopub.execute_input":"2022-08-04T17:11:38.147252Z","iopub.status.idle":"2022-08-04T17:11:38.739314Z","shell.execute_reply.started":"2022-08-04T17:11:38.147213Z","shell.execute_reply":"2022-08-04T17:11:38.738160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path_to_train_dataset = \"../input/spaceship-titanic/train.csv\"\npath_to_test_dataset = \"../input/spaceship-titanic/test.csv\"","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:11:38.741239Z","iopub.execute_input":"2022-08-04T17:11:38.742110Z","iopub.status.idle":"2022-08-04T17:11:38.747615Z","shell.execute_reply.started":"2022-08-04T17:11:38.742040Z","shell.execute_reply":"2022-08-04T17:11:38.746131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataset = pd.read_csv(path_to_train_dataset, index_col='PassengerId')\ntarget_column = 'Transported'\ntrain_dataset.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:11:38.751090Z","iopub.execute_input":"2022-08-04T17:11:38.751622Z","iopub.status.idle":"2022-08-04T17:11:38.814173Z","shell.execute_reply.started":"2022-08-04T17:11:38.751522Z","shell.execute_reply":"2022-08-04T17:11:38.812790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### EDA on spaceship titanic data-set\n\nAn **EDA** is an examination meant to uncover the underlying structure of the information contained within a data-set. It is important because it exposes trends, patterns, and relationships that are not readily apparent at first glance.\n\nThe purpose of an **EDA** is to allow data scientists to analyze the data before coming to any assumption. In this **EDA** of the spaceship titanic dataset, we'll be looking at the relationships between the feature variables.\n\nLet's perform an Exploratory Data Analysis on the spaceship titanic data-set to deduce relationships.","metadata":{}},{"cell_type":"code","source":"data = train_dataset.drop(target_column, axis=1)\ntarget = train_dataset[target_column]\ndata.info()","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:11:38.816400Z","iopub.execute_input":"2022-08-04T17:11:38.816910Z","iopub.status.idle":"2022-08-04T17:11:38.837948Z","shell.execute_reply.started":"2022-08-04T17:11:38.816860Z","shell.execute_reply":"2022-08-04T17:11:38.836653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll be dropping the `Name` feature column as it provides no useful information to our cause. The `Cabin` feature column can be said to comprise of 3 there feature columns, namely `Deck`, `Num`, and `Side`. Let's screen this values out.","metadata":{}},{"cell_type":"code","source":"deck_num_side = ['Deck', 'Num', 'Side']\n\nfor i, col in enumerate(deck_num_side):\n    data[col] = data[\n        'Cabin'\n    ].map(\n        lambda x: (int(x.split('/')[i]) if col == 'Num' else x.split('/')[i]) if x is not np.nan else x\n    )\n\ndata.drop(['Cabin', 'Name'], axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:11:38.839441Z","iopub.execute_input":"2022-08-04T17:11:38.840296Z","iopub.status.idle":"2022-08-04T17:11:38.873701Z","shell.execute_reply.started":"2022-08-04T17:11:38.840254Z","shell.execute_reply":"2022-08-04T17:11:38.872520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exploratory_set = data.dropna(subset=data.columns)\nexploratory_set.info()","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:11:38.875177Z","iopub.execute_input":"2022-08-04T17:11:38.876412Z","iopub.status.idle":"2022-08-04T17:11:38.902057Z","shell.execute_reply.started":"2022-08-04T17:11:38.876362Z","shell.execute_reply":"2022-08-04T17:11:38.900640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exploratory_set.describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:11:38.904251Z","iopub.execute_input":"2022-08-04T17:11:38.905540Z","iopub.status.idle":"2022-08-04T17:11:38.942355Z","shell.execute_reply.started":"2022-08-04T17:11:38.905494Z","shell.execute_reply":"2022-08-04T17:11:38.941351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are a couple of continuous feature as seen by the describe method (although `Age` not being a continuous feature as proved by cell 8), we'll compare to see the relationship between all other continuous features.","metadata":{}},{"cell_type":"code","source":"def is_categorical(feature, dataframe=None, categorical_thresh=.1):\n    \"\"\"\n    Function to detect if a feature variable in a given dataframe is categorical\n    \"\"\"\n\n    error_condition_1 = dataframe is None and type(feature) == str\n    error_condition_2 = type(dataframe) == pd.DataFrame and type(feature) == pd.Series\n\n    if error_condition_1 or error_condition_2:\n        messages = [\n            \"Feature must be a pd Series if dataframe isn't provided\",\n            \"if features is a pd Dataframe, then a string must be given as feature name\"\n        ]\n        raise (ValueError(log_message(*messages, error_message=True, sep='\\n')))\n    else:\n        series = feature if type(feature) == pd.Series else dataframe[feature]\n\n        return (len(series.unique()) / len(series)) <= categorical_thresh\n\n\ndef is_continuous(feature, dataframe=None, categorical_thresh=.1):\n    \"\"\"\n    Function to detect if a feature variable in a given dataframe is continuous\n    \"\"\"\n\n    return not is_categorical(feature, dataframe, categorical_thresh)","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:11:38.943803Z","iopub.execute_input":"2022-08-04T17:11:38.944880Z","iopub.status.idle":"2022-08-04T17:11:38.954854Z","shell.execute_reply.started":"2022-08-04T17:11:38.944838Z","shell.execute_reply":"2022-08-04T17:11:38.953356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in exploratory_set.describe().columns:\n    print(f\"{col}, is categorical = {is_categorical(col, exploratory_set)}\", end='\\n\\n')","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:11:38.960600Z","iopub.execute_input":"2022-08-04T17:11:38.961356Z","iopub.status.idle":"2022-08-04T17:11:38.995278Z","shell.execute_reply.started":"2022-08-04T17:11:38.961295Z","shell.execute_reply":"2022-08-04T17:11:38.993785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"continuous_cols = ['RoomService', 'FoodCourt', 'ShoppingMall', 'Spa', 'VRDeck', 'Num']","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:11:38.997335Z","iopub.execute_input":"2022-08-04T17:11:38.997856Z","iopub.status.idle":"2022-08-04T17:11:39.003868Z","shell.execute_reply.started":"2022-08-04T17:11:38.997803Z","shell.execute_reply":"2022-08-04T17:11:39.002460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"g = sns.PairGrid(exploratory_set[continuous_cols].sample(1000))\ng.map_diag(sns.histplot)\ng.map_offdiag(sns.scatterplot);","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:11:39.005949Z","iopub.execute_input":"2022-08-04T17:11:39.006479Z","iopub.status.idle":"2022-08-04T17:11:55.654302Z","shell.execute_reply.started":"2022-08-04T17:11:39.006430Z","shell.execute_reply":"2022-08-04T17:11:55.652852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There's no pure linear relationship between the continuous feature column, in fact, the only discovery here is both features majorly aligns at point zero (0) especially for the `ShoppingMall` feature column.\n\nWe can also see multiple outliers in the continuous feature columns, with few data straining from major gravity point. We can handle this by fixing the outliers.","metadata":{}},{"cell_type":"code","source":"exploratory_set[continuous_cols].head(8)","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:11:55.655717Z","iopub.execute_input":"2022-08-04T17:11:55.656139Z","iopub.status.idle":"2022-08-04T17:11:55.677333Z","shell.execute_reply.started":"2022-08-04T17:11:55.656078Z","shell.execute_reply":"2022-08-04T17:11:55.675884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install slik_wrangler","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:11:55.679578Z","iopub.execute_input":"2022-08-04T17:11:55.681479Z","iopub.status.idle":"2022-08-04T17:12:06.868065Z","shell.execute_reply.started":"2022-08-04T17:11:55.681388Z","shell.execute_reply":"2022-08-04T17:12:06.866450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from slik_wrangler import preprocessing as pp","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:06.869780Z","iopub.execute_input":"2022-08-04T17:12:06.870236Z","iopub.status.idle":"2022-08-04T17:12:07.061351Z","shell.execute_reply.started":"2022-08-04T17:12:06.870194Z","shell.execute_reply":"2022-08-04T17:12:07.060138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exploratory_set = pp.detect_fix_outliers(\n    dataframe=exploratory_set,\n    num_features=continuous_cols,\n    display_inline=False\n)","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:07.063183Z","iopub.execute_input":"2022-08-04T17:12:07.064182Z","iopub.status.idle":"2022-08-04T17:12:07.127655Z","shell.execute_reply.started":"2022-08-04T17:12:07.064141Z","shell.execute_reply":"2022-08-04T17:12:07.126324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"g = sns.PairGrid(exploratory_set[continuous_cols].sample(1000))\ng.map_diag(sns.histplot)\ng.map_offdiag(sns.scatterplot);","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:07.129036Z","iopub.execute_input":"2022-08-04T17:12:07.129433Z","iopub.status.idle":"2022-08-04T17:12:13.586704Z","shell.execute_reply.started":"2022-08-04T17:12:07.129396Z","shell.execute_reply":"2022-08-04T17:12:13.585317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def pivot_dataframe(df, cat_1, cat_2, cont):\n    \"\"\"\n    Creates a heatmap dataframe of the provided features\n    \"\"\"\n    \n    indexes = list(df[cat_1].unique())\n    columns = list(df[cat_2].unique())\n    \n    data = np.zeros((len(indexes), len(columns)))\n    \n    for index, column, val in zip(\n        list(df[cat_1]),\n        list(df[cat_2]),\n        list(df[cont]),\n    ):\n        data[indexes.index(index)][columns.index(column)] += val\n    \n    return pd.DataFrame(\n        data, \n        index=indexes, \n        columns=columns\n    )\n\n\ndef plot_comparison_heatmap(cat_1, cat_2):\n    \"\"\"\n    Given two categorical variables, heatmap across the several \n    continuous feature variables would be plotted.\n    \"\"\"\n    \n    fig, axes = plt.subplots(2, 3, figsize=(20, 10))\n\n    for ax, col in zip(axes.ravel(), continuous_cols):\n        ax.title.set_text(col)\n        ax.set(xlabel=cat_1, ylabel=cat_2)\n        \n        sns.heatmap(\n            pivot_dataframe(\n                exploratory_set, \n                cat_1, cat_2, col\n            ),\n            linewidths=.5, cmap=\"YlGnBu\", ax=ax\n        );","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:13.588179Z","iopub.execute_input":"2022-08-04T17:12:13.588793Z","iopub.status.idle":"2022-08-04T17:12:13.599607Z","shell.execute_reply.started":"2022-08-04T17:12:13.588755Z","shell.execute_reply":"2022-08-04T17:12:13.598343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_comparison_heatmap('HomePlanet', 'Destination')","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:13.601463Z","iopub.execute_input":"2022-08-04T17:12:13.602023Z","iopub.status.idle":"2022-08-04T17:12:15.726191Z","shell.execute_reply.started":"2022-08-04T17:12:13.601986Z","shell.execute_reply":"2022-08-04T17:12:15.724869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the heat-maps the first thing we can observe is majority of the people are traveller from the home-planet **Earth** to the destination **TRAPPIST-1e**. Aside that common general fact, there are alot more information these maps provides.\n\n1. Majority of the **RoomService** bills were made by those traveling from **Mars** to **TRAPPIST-1e**.\n2. Majority of the **FoodCourt** bills were made by those traveling from **Europa** to **TRAPPIST-1e**, then **Europa** to **55 Cancri e**.\n3. Majority of the **ShoppingMall** bills were made by those traveling from **Mars** to **TRAPPIST-1e**.\n4. Majority of the **Spa** bills were made by those traveling from **Europa** to **TRAPPIST-1e**, then **Europa** to **55 Cancri e**.\n5. Majority of the **VRDeck** bills were made by those traveling from **Europa** to **TRAPPIST-1e**, then **Europa** to **55 Cancri e**.\n6. Majority of the **Num** count were by those traveling from **Mars** to **TRAPPIST-1e**.","metadata":{}},{"cell_type":"code","source":"plot_comparison_heatmap('CryoSleep', 'VIP')","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:15.727781Z","iopub.execute_input":"2022-08-04T17:12:15.729298Z","iopub.status.idle":"2022-08-04T17:12:17.686134Z","shell.execute_reply.started":"2022-08-04T17:12:15.729242Z","shell.execute_reply":"2022-08-04T17:12:17.684928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The figure above like the first heat-map figure does something similar but with the features `CryoSleep` and `VIP`. The general conclusion from the map though is that the majority of the bills paid were from non `VIP`'s and those who didn't make use of the `CryoSleep`. The `Num` feature is entirely different as it states the number on the deck per traveler, and can safely be ignored.","metadata":{}},{"cell_type":"code","source":"plot_comparison_heatmap('Deck', 'Side')","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:17.687589Z","iopub.execute_input":"2022-08-04T17:12:17.687967Z","iopub.status.idle":"2022-08-04T17:12:19.759201Z","shell.execute_reply.started":"2022-08-04T17:12:17.687934Z","shell.execute_reply":"2022-08-04T17:12:19.758276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Another relationship we deduced in terms of bills payment is that the travelers in the F deck paid the most bills... of course this could also mean they were very much. But as can be seen, most bill payment were mostly made for the features `RoomService` and `ShoppingMall`, and the majority of these came from the Port `Side`.","metadata":{}},{"cell_type":"markdown","source":"## Handling missing data\n\nWith the EDA, we can now proceed to handling the missing values, and there are three (3) types of missing-data in any data-set and three (3) ways to identify them.\n\n![Missing Data Mechanism](../input/images/missing_data_mechanism.jpeg)\n\nIdentifying the missing-data helps narrow down the approach that can be used for treating missing data.\n\n1. **Missing Completely at Random (MCAR)**. In this scenario the missing values have no correlation (relationship) with other values in the dataset. There's no systematic process at work that could make some of the data more likely to be missing than others.\n2. **Missing at Random (MAR)**. Unlike **MCAR** there is a systematic relationship between the missing values and the observed data, but not the missing data. Although, for **MAR** there’s an underlying reason for missing-data that can’t be directly observed.\n3. **Missing Not at Random (MNAR)**. For these scenarion there's a correlation between the missing-data and its values. \n\nCheck the reference section to understand more about this subject. Here though we'll be discovering the kind of missing type contained in the [spaceship-titanic data-set](../../data/spaceship-titanic/train.csv) and fixing it.\n\nThe tool we'll be making use of for this evaluation is called [**missingno**](https://github.com/ResidentMario/missingno) which is a small tool-set of flexible and easy-to-use missing data visualizations with utilities that allows for a quick visual summary of the completeness of a data-set.\n\n**missingno** provides four (4) major functions for identifying the type of missing-data. These functions includes:\n\n1. **matrix**\n2. **bar**\n3. **heatmap**\n4. **dendrogram**","metadata":{"execution":{"iopub.status.busy":"2022-08-04T15:28:10.572343Z","iopub.execute_input":"2022-08-04T15:28:10.572813Z","iopub.status.idle":"2022-08-04T15:28:10.592890Z","shell.execute_reply.started":"2022-08-04T15:28:10.572776Z","shell.execute_reply":"2022-08-04T15:28:10.591499Z"}}},{"cell_type":"code","source":"color = (.35, .53, .76)","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:19.760447Z","iopub.execute_input":"2022-08-04T17:12:19.761300Z","iopub.status.idle":"2022-08-04T17:12:19.767165Z","shell.execute_reply.started":"2022-08-04T17:12:19.761254Z","shell.execute_reply":"2022-08-04T17:12:19.765523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Bar\n\nWith the log parameter set to `True` the bar chart shows the degree to which a feature column is missing. The lower the bar of a feature column, the higher the degree of missing values as compared to other feature column.\n\nAs we can see from the plot below, the `CrypSleep` has the majority of the missing values, followed by `ShoppingMall`, then `VIP`, etc. These is useful to visualize the degree of the missing values in each feature column.\n\nAnother thing we notice is that `Deck`, `Num`, and `Side` all have equally missing values, but this is understandable since the feature columns `Deck`, `Num`, and `Side` were all gotten from the `Cabin` feature column.","metadata":{}},{"cell_type":"code","source":"msno.bar(data, color=color, log=True);","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:19.771289Z","iopub.execute_input":"2022-08-04T17:12:19.771779Z","iopub.status.idle":"2022-08-04T17:12:21.042492Z","shell.execute_reply.started":"2022-08-04T17:12:19.771732Z","shell.execute_reply":"2022-08-04T17:12:21.041504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Heat Map\n\nThe Heat Map measures how strongly the presence or absence of one variable affects the presence of another. This is useful in detecting correlations between feature variable and is very essential. \n\nCorrelation ranges from `-1` (if one variable appears the other definitely does not) to `0` (variables appearing or not appearing have no effect on one another) to `1` (if one variable appears the other definitely also does). Entries marked `<1` or `>-1` have a correlation that is close to being exactingly negative or positive, but is still not quite perfectly so. This points to a small number of records in the dataset which are erroneous.\n\nAccording to the heatmap though, there are 3 correlations, which are between `Deck` & `Num`, `Deck` & `Side`, and `Num` & `Side`. These may not be of much use as it is understandable since the feature columns `Deck`, `Num`, and `Side` were all gotten from the `Cabin` feature column.\n\n**Note**: This map is very useful (along with the `missingno` matrix) to discovering the **MNAR**.m","metadata":{}},{"cell_type":"code","source":"msno.heatmap(data, cmap='YlGnBu');","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:21.043830Z","iopub.execute_input":"2022-08-04T17:12:21.044783Z","iopub.status.idle":"2022-08-04T17:12:21.560294Z","shell.execute_reply.started":"2022-08-04T17:12:21.044743Z","shell.execute_reply":"2022-08-04T17:12:21.559001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Dendrogram\n\nThe dendrogram allows you to more fully correlate variable completion, revealing trends deeper than the pairwise ones visible in the correlation heatmap.\n\nBut as we can see from the heat map, the only correlation we see is with the `Deck`, `Num`, and `Side`, which were all gotten from the `Cabin` feature column.\n\n**Note**: This graph is very useful (along with the `missingno` matrix) to discovering the **MAR**.","metadata":{}},{"cell_type":"code","source":"msno.dendrogram(data, figsize=(20, 15));","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:21.561930Z","iopub.execute_input":"2022-08-04T17:12:21.562453Z","iopub.status.idle":"2022-08-04T17:12:22.499810Z","shell.execute_reply.started":"2022-08-04T17:12:21.562411Z","shell.execute_reply":"2022-08-04T17:12:22.498669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Matrix\n\nThe matrix function might be the hardest to read as it makes it difficult to see patterns, but it best summarizes the relationship between the feature columns as can be seen in the graph below.\n\nThe matrix is a data-dense display which lets you quickly visually pick out patterns in data completion. At a glance, we can pick out the relationship of which the bar, heatmap, and dendrogram graphs has reviled to us.\n\n**Note**: The matrix is very useful in discovering the **MCAR**, **MAR**, and **MNAR**.","metadata":{}},{"cell_type":"code","source":"msno.matrix(data, figsize=(25, 20), color=color);","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:22.501693Z","iopub.execute_input":"2022-08-04T17:12:22.502541Z","iopub.status.idle":"2022-08-04T17:12:23.162104Z","shell.execute_reply.started":"2022-08-04T17:12:22.502487Z","shell.execute_reply":"2022-08-04T17:12:23.160906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In conclusion, the summary of the missing data we can deduce is that the `CrypSleep` has the majority of the missing values, followed by `ShoppingMall`, then `VIP`, then the `Deck`, `Num`, and `Side` all have equally missing values next.\n\nThe heatmap shows there are no effect between feature variables other than that of `Deck` & `Num`, `Deck` & `Side`, and `Num` & `Side`. Similarly, the only correlation we see using the dendrogram is with the `Deck`, `Num`, and `Side`, which were all gotten from the `Cabin` feature column.\n\nWe also picked all this out from the matrix function, which at the end concludes, the type of missing data we have is a **MCAR** missing type (excluding the feature columns `Deck`, `Num`, and `Side` which has **MNAR** missing data type all together).\n\nSince we already applied the **Deletion** means to address the missing values and create the `exploratory_set`, we might as well use that to validate it's effectiveness.","metadata":{}},{"cell_type":"code","source":"exploratory_target = target[exploratory_set.index]","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:23.163645Z","iopub.execute_input":"2022-08-04T17:12:23.164006Z","iopub.status.idle":"2022-08-04T17:12:23.173247Z","shell.execute_reply.started":"2022-08-04T17:12:23.163974Z","shell.execute_reply":"2022-08-04T17:12:23.171957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exploratory_set.shape","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:23.180006Z","iopub.execute_input":"2022-08-04T17:12:23.180662Z","iopub.status.idle":"2022-08-04T17:12:23.187460Z","shell.execute_reply.started":"2022-08-04T17:12:23.180624Z","shell.execute_reply":"2022-08-04T17:12:23.186310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.preprocessing import OrdinalEncoder","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:23.189314Z","iopub.execute_input":"2022-08-04T17:12:23.189688Z","iopub.status.idle":"2022-08-04T17:12:23.197245Z","shell.execute_reply.started":"2022-08-04T17:12:23.189646Z","shell.execute_reply":"2022-08-04T17:12:23.196055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"categorical_cols = [col for col in data.columns if col not in continuous_cols]\ncategorical_cols","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:23.199308Z","iopub.execute_input":"2022-08-04T17:12:23.199714Z","iopub.status.idle":"2022-08-04T17:12:23.212035Z","shell.execute_reply.started":"2022-08-04T17:12:23.199678Z","shell.execute_reply":"2022-08-04T17:12:23.210888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"categorical_encoder = OrdinalEncoder()\nexploratory_set[categorical_cols] = categorical_encoder.fit_transform(exploratory_set[categorical_cols])\nexploratory_set[categorical_cols].head()","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:23.213575Z","iopub.execute_input":"2022-08-04T17:12:23.213922Z","iopub.status.idle":"2022-08-04T17:12:23.256894Z","shell.execute_reply.started":"2022-08-04T17:12:23.213891Z","shell.execute_reply":"2022-08-04T17:12:23.255556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"simple_model = RandomForestClassifier(n_estimators=300, max_depth=5, random_state=42, n_jobs=-1)","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:23.258304Z","iopub.execute_input":"2022-08-04T17:12:23.258794Z","iopub.status.idle":"2022-08-04T17:12:23.264643Z","shell.execute_reply.started":"2022-08-04T17:12:23.258749Z","shell.execute_reply":"2022-08-04T17:12:23.263512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores = cross_val_score(simple_model, exploratory_set, exploratory_target, scoring='accuracy', n_jobs=-1, cv=5)\nscores","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:23.265728Z","iopub.execute_input":"2022-08-04T17:12:23.266087Z","iopub.status.idle":"2022-08-04T17:12:28.319830Z","shell.execute_reply.started":"2022-08-04T17:12:23.266055Z","shell.execute_reply":"2022-08-04T17:12:28.318217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores.mean()","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:28.324463Z","iopub.execute_input":"2022-08-04T17:12:28.324928Z","iopub.status.idle":"2022-08-04T17:12:28.333357Z","shell.execute_reply.started":"2022-08-04T17:12:28.324884Z","shell.execute_reply":"2022-08-04T17:12:28.332329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After applying the **Deletion** means, we get an average of (roughly) **75%** performance from our simple random forest model. This is without any form of fine tuning the model and reducing the data point by about (**22%**) of the data-set.","metadata":{}},{"cell_type":"markdown","source":"## Handling the Missing Values in the Data-Set\n\nGiven that the missing type is of the **MCAR** type, we could incorporate the **Deletion** means to addressing the missing values as we did. But wouldn't it be better if we could fill the missing data, directly from the data-set?","metadata":{}},{"cell_type":"code","source":"test_dataset = pd.read_csv(path_to_test_dataset, index_col='PassengerId')\ntest_dataset.info()","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:28.334525Z","iopub.execute_input":"2022-08-04T17:12:28.336255Z","iopub.status.idle":"2022-08-04T17:12:28.391444Z","shell.execute_reply.started":"2022-08-04T17:12:28.336212Z","shell.execute_reply":"2022-08-04T17:12:28.389992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can also see that there are multiple missing values in our test-set, and finding a means to address this missing values would be necessary.\n\nIn an article by [Bala Priya C](https://dev.to/balapriya) where she talked about handling missing values by making use of the `SimpleImputer -> add_indicator` parameter in sklearn to detect missing values and automatically fix them by stating their strategies. ","metadata":{}},{"cell_type":"code","source":"from sklearn.impute import SimpleImputer","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:28.393228Z","iopub.execute_input":"2022-08-04T17:12:28.394996Z","iopub.status.idle":"2022-08-04T17:12:28.400996Z","shell.execute_reply.started":"2022-08-04T17:12:28.394936Z","shell.execute_reply":"2022-08-04T17:12:28.399632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# For example\n\n## Creating the Imputer\ntest_imputer = SimpleImputer(strategy='most_frequent', add_indicator=True)\n\n## Creating and evaluating the imputer\n## on a sample series\nhome_planet = train_dataset['HomePlanet'].copy()\nhome_planet = test_imputer.fit_transform(home_planet.to_numpy().reshape(-1, 1))\n\n## Displaying result\nhome_planet[:10]","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:28.403546Z","iopub.execute_input":"2022-08-04T17:12:28.404356Z","iopub.status.idle":"2022-08-04T17:12:28.424516Z","shell.execute_reply.started":"2022-08-04T17:12:28.404302Z","shell.execute_reply":"2022-08-04T17:12:28.423173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Notice that there are now two features being returned, the first represents the `HomePlanet` feature column, while the other represents an indicator of whether this feature column was missing or not.\n\nIf we're to apply this for all feature columns, separating them into either a continuous or a categorical feature column and performing a simple imputation on them both while adding an indicator, we should get a complete data-set indicative of missing features.\n\n**Note**: We would also have to find a way to address the column names after transforming the data. We can write a short program to handle that.","metadata":{}},{"cell_type":"code","source":"def add_indicator_columns(columns, title='col_'):\n    \"\"\"\n    Creating new columns for data with indicators\n    \"\"\"\n    \n    return list(map(lambda x: title + str(x), range(len(columns) * 2)))","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:28.426277Z","iopub.execute_input":"2022-08-04T17:12:28.427465Z","iopub.status.idle":"2022-08-04T17:12:28.434849Z","shell.execute_reply.started":"2022-08-04T17:12:28.427411Z","shell.execute_reply":"2022-08-04T17:12:28.433488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.compose import ColumnTransformer","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:28.436808Z","iopub.execute_input":"2022-08-04T17:12:28.437629Z","iopub.status.idle":"2022-08-04T17:12:28.446269Z","shell.execute_reply.started":"2022-08-04T17:12:28.437564Z","shell.execute_reply":"2022-08-04T17:12:28.444858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"categorical_encoder = OrdinalEncoder()\ndata[categorical_cols] = categorical_encoder.fit_transform(data[categorical_cols])\ndata[categorical_cols].head()","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:28.448292Z","iopub.execute_input":"2022-08-04T17:12:28.449123Z","iopub.status.idle":"2022-08-04T17:12:28.502639Z","shell.execute_reply.started":"2022-08-04T17:12:28.449051Z","shell.execute_reply":"2022-08-04T17:12:28.501319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transformer = ColumnTransformer([\n    ('continuous_transformer', SimpleImputer(strategy='mean', add_indicator=True), continuous_cols),\n    ('categorical_transformer', SimpleImputer(strategy='most_frequent', add_indicator=True), categorical_cols)\n])","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:28.504249Z","iopub.execute_input":"2022-08-04T17:12:28.504803Z","iopub.status.idle":"2022-08-04T17:12:28.509968Z","shell.execute_reply.started":"2022-08-04T17:12:28.504768Z","shell.execute_reply":"2022-08-04T17:12:28.509072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = pd.DataFrame(\n    transformer.fit_transform(data), \n    columns=add_indicator_columns(continuous_cols + categorical_cols), index=data.index\n)\ndata.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:28.511115Z","iopub.execute_input":"2022-08-04T17:12:28.512569Z","iopub.status.idle":"2022-08-04T17:12:28.570435Z","shell.execute_reply.started":"2022-08-04T17:12:28.512515Z","shell.execute_reply":"2022-08-04T17:12:28.569114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from slik_wrangler.dqa import data_cleanness_assessment","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:28.572006Z","iopub.execute_input":"2022-08-04T17:12:28.572538Z","iopub.status.idle":"2022-08-04T17:12:28.577630Z","shell.execute_reply.started":"2022-08-04T17:12:28.572499Z","shell.execute_reply":"2022-08-04T17:12:28.576629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_cleanness_assessment(data)","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:28.578955Z","iopub.execute_input":"2022-08-04T17:12:28.579589Z","iopub.status.idle":"2022-08-04T17:12:30.462908Z","shell.execute_reply.started":"2022-08-04T17:12:28.579539Z","shell.execute_reply":"2022-08-04T17:12:30.461546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.shape","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:30.464786Z","iopub.execute_input":"2022-08-04T17:12:30.465207Z","iopub.status.idle":"2022-08-04T17:12:30.472244Z","shell.execute_reply.started":"2022-08-04T17:12:30.465169Z","shell.execute_reply":"2022-08-04T17:12:30.471130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for method in ['rows', 'columns']:\n    data = pp.drop_duplicate(data, data.columns, method=method)","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:30.474425Z","iopub.execute_input":"2022-08-04T17:12:30.475801Z","iopub.status.idle":"2022-08-04T17:12:31.806794Z","shell.execute_reply.started":"2022-08-04T17:12:30.475747Z","shell.execute_reply":"2022-08-04T17:12:31.805355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For the duplicate columns, the duplicates are obviously from the repetition in missing data from the columns `Deck`, `Num`, and `Side`. So two out of the 3 columns had to be dropped to eliminate this repetition. There was 18 duplicate rows in the dataset though, which constitutes (roughly) **0.2%** of the data, which is better in comparison to dropping **22%** of the data.","metadata":{}},{"cell_type":"code","source":"target = target[data.index]","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:31.808963Z","iopub.execute_input":"2022-08-04T17:12:31.809377Z","iopub.status.idle":"2022-08-04T17:12:31.818087Z","shell.execute_reply.started":"2022-08-04T17:12:31.809337Z","shell.execute_reply":"2022-08-04T17:12:31.816701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores = cross_val_score(simple_model, data, target, scoring='accuracy', n_jobs=-1, cv=5)\nscores","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:31.819881Z","iopub.execute_input":"2022-08-04T17:12:31.820319Z","iopub.status.idle":"2022-08-04T17:12:35.618620Z","shell.execute_reply.started":"2022-08-04T17:12:31.820279Z","shell.execute_reply":"2022-08-04T17:12:35.617570Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores.mean()","metadata":{"execution":{"iopub.status.busy":"2022-08-04T17:12:35.619635Z","iopub.execute_input":"2022-08-04T17:12:35.619986Z","iopub.status.idle":"2022-08-04T17:12:35.627999Z","shell.execute_reply.started":"2022-08-04T17:12:35.619953Z","shell.execute_reply":"2022-08-04T17:12:35.626685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Conclusion\n\nThough the difference between both is roughly **1%**, it is essential to know when to apply which. Let's refer to the first method as the **Deletion Method** and the latter as the **Indicator Method**.\n\nThe **Deletion Method** is recommended for much larger data-set where the proportion of missing values doesn't exceed **50%** the number of the data point. while the **Indicator Method** is best for a small data-set like the one we made use of as it manages the data-set and takes note of the missing data across the feature columns.\n\nThe **Indicator Method** might be a big problem with large data-set, as the method sorts to expand the data to address the missing data-point, thus, further complicating large data.","metadata":{}},{"cell_type":"markdown","source":"### Reference\n\n- [A framework for handling missing data](https://towardsdatascience.com/missing-data-cfd9dbfd11b7)\n- [The missingno package](https://github.com/ResidentMario/missingno)\n- [How to Handle Missing Data Better](https://dev.to/balapriya/having-trouble-with-missing-data-here-s-how-you-can-handle-them-better-42d6)","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}