{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"In this notebook, we establish a logisitc regression baseline for the Spaceship Titanic dataset. Specifically, we do the following:\n\n- Feature Engineering Based On EDA\n- Imputing Missing Values Based On Insights (and Kaggle Discussions!)\n- Logistic Regression Baselines, where we also judge the usability of 2 dense categorical features (extracted from `PassengerId` and `Cabin` respectively)\n\nBy the end of this notebook, we will carry out 7 (!) experiments and have a definitive accuracy score that a final model should try to beat.\n\nThis is Part 2 of a three part series.\n\n* Part 1: [Spaceship Titanic - Exploratory Data Analysis](https://www.kaggle.com/code/defcodeking/spaceship-titanic-exploratory-data-analysis)\n* Part 2: [Spaceship Titanic - Logistic Regression Baselines](https://www.kaggle.com/code/defcodeking/spaceship-titanic-logistic-regression-baselines) (you are here!)\n* Part 3: [Ensembling (And Optuna 😉) Is All You Need!](https://www.kaggle.com/code/defcodeking/ensembling-and-optuna-is-all-you-need)","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"from sklearn import linear_model, preprocessing, impute, model_selection, metrics\nfrom scipy.stats import boxcox\nimport pandas as pd\nimport numpy as np\nimport random\nimport os\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nsns.set_theme()\nsns.set_style(\"ticks\")\nsns.despine()\n\n%matplotlib inline","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-06-08T21:12:55.725242Z","iopub.execute_input":"2022-06-08T21:12:55.726255Z","iopub.status.idle":"2022-06-08T21:12:55.740036Z","shell.execute_reply.started":"2022-06-08T21:12:55.726208Z","shell.execute_reply":"2022-06-08T21:12:55.738863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def seed_everything(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    \nseed_everything()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:55.802275Z","iopub.execute_input":"2022-06-08T21:12:55.80283Z","iopub.status.idle":"2022-06-08T21:12:55.811502Z","shell.execute_reply.started":"2022-06-08T21:12:55.80279Z","shell.execute_reply":"2022-06-08T21:12:55.810326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Config","metadata":{}},{"cell_type":"code","source":"DATA_DIR = \"../input/spaceship-titanic\"\n\ndef filepath(filename):\n    return os.path.join(DATA_DIR, filename)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:55.868303Z","iopub.execute_input":"2022-06-08T21:12:55.869207Z","iopub.status.idle":"2022-06-08T21:12:55.87479Z","shell.execute_reply.started":"2022-06-08T21:12:55.869163Z","shell.execute_reply":"2022-06-08T21:12:55.87361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Dataset","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(filepath(\"train.csv\"), index_col=\"PassengerId\")\ntest_df = pd.read_csv(filepath(\"test.csv\"), index_col=\"PassengerId\")\n\n# Add PassengerId since we need it for feature engineering\ntrain_df[\"PassengerId\"] = train_df.index\ntest_df[\"PassengerId\"] = test_df.index\n\nlen(train_df), len(test_df)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:55.935904Z","iopub.execute_input":"2022-06-08T21:12:55.936527Z","iopub.status.idle":"2022-06-08T21:12:55.996243Z","shell.execute_reply.started":"2022-06-08T21:12:55.936491Z","shell.execute_reply":"2022-06-08T21:12:55.995334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Initial Feature Engineering\n\nSee [Spaceship Titanic - Exploratory Data Analysis](https://www.kaggle.com/code/defcodeking/spaceship-titanic-exploratory-data-analysis) for more details.\n\n> Note: All features extraced from `Cabin` will be engineered after missing values are imputed but the function is created here.","metadata":{}},{"cell_type":"code","source":"expenditure_columns = ['RoomService', 'FoodCourt', 'ShoppingMall', 'Spa', 'VRDeck']","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:56.035345Z","iopub.execute_input":"2022-06-08T21:12:56.036008Z","iopub.status.idle":"2022-06-08T21:12:56.040504Z","shell.execute_reply.started":"2022-06-08T21:12:56.03597Z","shell.execute_reply":"2022-06-08T21:12:56.039708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## `GroupId`, `GroupSize` and `Alone`\n\nThe `from_passengerId()` function extracts features out of the `PassengerId` feature.","metadata":{}},{"cell_type":"code","source":"def from_passengerId(df):\n    split_id = df[\"PassengerId\"].str.split(\"_\", expand=True)\n    df[\"GroupId\"] = split_id[0]\n    df[\"GroupSize\"] = df.groupby(\"GroupId\")[\"GroupId\"].transform(\"count\")\n    \n    # Indicates whether the passenger was traveling alone or not\n    df[\"Alone\"] = df[\"GroupSize\"] == 1\n    \n    return df\n\ntrain_df = from_passengerId(train_df)\ntest_df = from_passengerId(test_df)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:56.14751Z","iopub.execute_input":"2022-06-08T21:12:56.148005Z","iopub.status.idle":"2022-06-08T21:12:56.202537Z","shell.execute_reply.started":"2022-06-08T21:12:56.147968Z","shell.execute_reply":"2022-06-08T21:12:56.201344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:56.227248Z","iopub.execute_input":"2022-06-08T21:12:56.227788Z","iopub.status.idle":"2022-06-08T21:12:56.259719Z","shell.execute_reply.started":"2022-06-08T21:12:56.227725Z","shell.execute_reply":"2022-06-08T21:12:56.258341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Presence of Missing Values\n\nThe function `mssing_value_features()` takes a list of columns and adds a new column indicating whether there is a missing value present or not. It also adds an additional feature called `TotalExpense_missing` which indicates whether `TotalExpense` (the sum of all the expenditure columns) is missing when nulls are not ignored. This sort of \"summarises\" the missing values in the expenditure columns.","metadata":{}},{"cell_type":"code","source":"def missing_value_features(df, columns, expenditure_columns):\n    for column in columns:\n        df[f\"{column}_missing\"] = df[column].isna()\n    \n    # An additional feature which encodes whether TotalExpense is missing if NAs are not ignored\n    df[\"TotalExpense_missing\"] = df[expenditure_columns].sum(axis=1, skipna=False).isna()\n    return df\n\ncolumns = [\"RoomService\", \"FoodCourt\", \"ShoppingMall\", \"Cabin\", \"VIP\"]\ntrain_df = missing_value_features(train_df, columns, expenditure_columns)\ntest_df = missing_value_features(test_df, columns, expenditure_columns)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:56.281288Z","iopub.execute_input":"2022-06-08T21:12:56.281806Z","iopub.status.idle":"2022-06-08T21:12:56.30912Z","shell.execute_reply.started":"2022-06-08T21:12:56.28173Z","shell.execute_reply":"2022-06-08T21:12:56.30787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:56.358959Z","iopub.execute_input":"2022-06-08T21:12:56.359576Z","iopub.status.idle":"2022-06-08T21:12:56.392436Z","shell.execute_reply.started":"2022-06-08T21:12:56.359539Z","shell.execute_reply":"2022-06-08T21:12:56.391262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## `TotalExpense`\n\nThe function `from_expenditure_features()` extracts a feature from all the expenditure columns which is their sum (ignoring nulls).","metadata":{}},{"cell_type":"code","source":"def from_expenditure_features(df, expenditure_columns):\n    df[\"TotalExpense\"] = df[expenditure_columns].sum(axis=1)\n    return df\n\ntrain_df = from_expenditure_features(train_df, expenditure_columns)\ntest_df = from_expenditure_features(test_df, expenditure_columns)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:56.414357Z","iopub.execute_input":"2022-06-08T21:12:56.415388Z","iopub.status.idle":"2022-06-08T21:12:56.427602Z","shell.execute_reply.started":"2022-06-08T21:12:56.415338Z","shell.execute_reply":"2022-06-08T21:12:56.426218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:56.492601Z","iopub.execute_input":"2022-06-08T21:12:56.493437Z","iopub.status.idle":"2022-06-08T21:12:56.525474Z","shell.execute_reply.started":"2022-06-08T21:12:56.493391Z","shell.execute_reply":"2022-06-08T21:12:56.524316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## `CabinDeck`, `CabinNum` and `CabinSide`\n\nThe function `from_cabin()` splits the `Cabin` feature into its three constituents: `Deck`, `Num` and `Side`.","metadata":{}},{"cell_type":"code","source":"def from_cabin(df):\n    df[[\"CabinDeck\", \"CabinNum\", \"CabinSide\"]] = df[\"Cabin\"].str.split(\"/\", expand=True)\n    return df","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:56.547149Z","iopub.execute_input":"2022-06-08T21:12:56.54761Z","iopub.status.idle":"2022-06-08T21:12:56.553746Z","shell.execute_reply.started":"2022-06-08T21:12:56.547575Z","shell.execute_reply":"2022-06-08T21:12:56.552834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Categorical Missing Values\n\nMissing values in some categorical features will be filled by the mode. Specifically, `HomePlanet`, `CryoSleep` and `Destination` will use the feature-level mode, while `Cabin` will use group-level mode taken in two ways. First, the null values will be filled by the group-mode based on `GroupId`. This will leave passengers who were travelling alone or those who belong to a group which has null values in `Cabin` for every member. These will be filled by the group-mode based on `HomePlanet` and `Destination`.\n\n`VIP` is a feature that cannot be filled with the mode. Instead, we will use some heuristics discovered by other Kagglers and posted on the competitions discussion page. As it turns out, the heuristics will not be enough and so, the remaining values will be filled according to the probability distribution of `VIP`.","metadata":{}},{"cell_type":"markdown","source":"## `HomePlanet`, `CryoSleep`, `Destination`","metadata":{}},{"cell_type":"code","source":"def simple_mode_replacement(df, columns):\n    df[columns] = df[columns].fillna(df[columns].mode().iloc[0])\n    return df\n\ncolumns = [\"HomePlanet\", \"CryoSleep\", \"Destination\"]\ntrain_df = simple_mode_replacement(train_df, columns)\ntest_df = simple_mode_replacement(test_df, columns)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:56.602638Z","iopub.execute_input":"2022-06-08T21:12:56.603746Z","iopub.status.idle":"2022-06-08T21:12:56.639185Z","shell.execute_reply.started":"2022-06-08T21:12:56.603683Z","shell.execute_reply":"2022-06-08T21:12:56.638355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[columns].isna().any()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:56.662219Z","iopub.execute_input":"2022-06-08T21:12:56.662976Z","iopub.status.idle":"2022-06-08T21:12:56.676553Z","shell.execute_reply.started":"2022-06-08T21:12:56.662934Z","shell.execute_reply":"2022-06-08T21:12:56.675287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:56.737575Z","iopub.execute_input":"2022-06-08T21:12:56.738387Z","iopub.status.idle":"2022-06-08T21:12:56.769717Z","shell.execute_reply.started":"2022-06-08T21:12:56.738342Z","shell.execute_reply":"2022-06-08T21:12:56.768675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## `Cabin`","metadata":{}},{"cell_type":"code","source":"def group_mode_replacement(df, groupby, column):\n    # Find all passengers belonging to groups where at least one member has a non-null column value\n    temp = df.groupby(groupby).filter(lambda x: x[column].notna().any())\n    \n    # Replace by mode\n    func = lambda x: x.fillna(x.mode().iloc[0]) if x.isna().any() else x\n    temp[column] = temp.groupby(groupby)[column].transform(func)\n    \n    # Update the original dataframe\n    df.loc[temp.index, column] = temp[column]\n    \n    return df\n\ntrain_df = group_mode_replacement(train_df, groupby=\"GroupId\", column=\"Cabin\")\ntest_df = group_mode_replacement(test_df, groupby=\"GroupId\", column=\"Cabin\")","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:12:56.794831Z","iopub.execute_input":"2022-06-08T21:12:56.795307Z","iopub.status.idle":"2022-06-08T21:13:02.921535Z","shell.execute_reply.started":"2022-06-08T21:12:56.795272Z","shell.execute_reply":"2022-06-08T21:13:02.91998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:02.924024Z","iopub.execute_input":"2022-06-08T21:13:02.924743Z","iopub.status.idle":"2022-06-08T21:13:02.960993Z","shell.execute_reply.started":"2022-06-08T21:13:02.924671Z","shell.execute_reply":"2022-06-08T21:13:02.959765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are still 99 passengers with null `Cabin` values in the train set and 63 in the test set. These can be filled with the mode of groups by `HomePlanet` and `Destination`.","metadata":{}},{"cell_type":"code","source":"train_df[\"Cabin\"].isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:02.962733Z","iopub.execute_input":"2022-06-08T21:13:02.96357Z","iopub.status.idle":"2022-06-08T21:13:02.973412Z","shell.execute_reply.started":"2022-06-08T21:13:02.963514Z","shell.execute_reply":"2022-06-08T21:13:02.972064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df[\"Cabin\"].isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:02.976211Z","iopub.execute_input":"2022-06-08T21:13:02.976992Z","iopub.status.idle":"2022-06-08T21:13:02.992648Z","shell.execute_reply.started":"2022-06-08T21:13:02.97695Z","shell.execute_reply":"2022-06-08T21:13:02.991738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = group_mode_replacement(train_df, groupby=[\"HomePlanet\", \"Destination\"], column=\"Cabin\")\ntest_df = group_mode_replacement(test_df, groupby=[\"HomePlanet\", \"Destination\"], column=\"Cabin\")","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:02.99421Z","iopub.execute_input":"2022-06-08T21:13:02.995156Z","iopub.status.idle":"2022-06-08T21:13:03.077168Z","shell.execute_reply.started":"2022-06-08T21:13:02.995109Z","shell.execute_reply":"2022-06-08T21:13:03.076125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"Cabin\"].isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.079082Z","iopub.execute_input":"2022-06-08T21:13:03.079715Z","iopub.status.idle":"2022-06-08T21:13:03.088195Z","shell.execute_reply.started":"2022-06-08T21:13:03.079675Z","shell.execute_reply":"2022-06-08T21:13:03.087228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df[\"Cabin\"].isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.089538Z","iopub.execute_input":"2022-06-08T21:13:03.089979Z","iopub.status.idle":"2022-06-08T21:13:03.105998Z","shell.execute_reply.started":"2022-06-08T21:13:03.08994Z","shell.execute_reply":"2022-06-08T21:13:03.104987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that all the null values have been filled in for `Cabin`, we will extract the features from `Cabin`.","metadata":{}},{"cell_type":"code","source":"train_df = from_cabin(train_df)\ntest_df = from_cabin(test_df)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.107188Z","iopub.execute_input":"2022-06-08T21:13:03.107932Z","iopub.status.idle":"2022-06-08T21:13:03.150936Z","shell.execute_reply.started":"2022-06-08T21:13:03.107897Z","shell.execute_reply":"2022-06-08T21:13:03.149843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.152637Z","iopub.execute_input":"2022-06-08T21:13:03.153068Z","iopub.status.idle":"2022-06-08T21:13:03.186573Z","shell.execute_reply.started":"2022-06-08T21:13:03.153032Z","shell.execute_reply":"2022-06-08T21:13:03.185429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"columns = [\"Cabin\", \"CabinDeck\", \"CabinNum\", \"CabinSide\"]\ntrain_df[columns].isna().any()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.189885Z","iopub.execute_input":"2022-06-08T21:13:03.190702Z","iopub.status.idle":"2022-06-08T21:13:03.209169Z","shell.execute_reply.started":"2022-06-08T21:13:03.190659Z","shell.execute_reply":"2022-06-08T21:13:03.20822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df[columns].isna().any()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.210395Z","iopub.execute_input":"2022-06-08T21:13:03.211177Z","iopub.status.idle":"2022-06-08T21:13:03.226773Z","shell.execute_reply.started":"2022-06-08T21:13:03.21114Z","shell.execute_reply":"2022-06-08T21:13:03.225688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## `VIP`","metadata":{}},{"cell_type":"code","source":"train_df[\"VIP\"].isna().sum(), test_df[\"VIP\"].isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.228053Z","iopub.execute_input":"2022-06-08T21:13:03.228571Z","iopub.status.idle":"2022-06-08T21:13:03.240019Z","shell.execute_reply.started":"2022-06-08T21:13:03.228535Z","shell.execute_reply":"2022-06-08T21:13:03.238341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following heuristics will be followed (see [Some rules to fill NaNs](https://www.kaggle.com/competitions/spaceship-titanic/discussion/315987)):\n\n- Passengers who have zero spending and are not in cryo sleep are not VIPs.\n- Passengers who are below or at the age of 12 are not VIPs.\n- Passengers from Earth are not VIPs.\n- Mars VIPs have `Age` >= 18, no `CryoSleep` and never go to \"55 Cancri e\"","metadata":{}},{"cell_type":"code","source":"def impute_vip_for_no_spend(df):\n    df.loc[\n        (df[\"VIP\"].isna()) & (df[\"TotalExpense\"] == 0.0) & (~df[\"CryoSleep\"]), \"VIP\"\n    ] = False\n    return df\n\ndef impute_vip_for_children(df):\n    df.loc[(df[\"VIP\"].isna()) & (df[\"Age\"] <= 12), \"VIP\"] = False\n    return df\n\ndef impute_vip_for_earthlings(df):\n    df.loc[(df[\"VIP\"].isna()) & (df[\"HomePlanet\"] == \"Earth\"), \"VIP\"] = False\n    return df\n\ndef impute_vip_for_martians(df):\n    df.loc[\n        (df[\"VIP\"].isna())\n        & (df[\"Age\"] >= 18)\n        & (~df[\"CryoSleep\"])\n        & (df[\"Destination\"] != \"55 Cancri e\"),\n        \"VIP\",\n    ] = True\n    return df","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.241748Z","iopub.execute_input":"2022-06-08T21:13:03.243093Z","iopub.status.idle":"2022-06-08T21:13:03.25571Z","shell.execute_reply.started":"2022-06-08T21:13:03.243025Z","shell.execute_reply":"2022-06-08T21:13:03.254307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def impute_vip(df):\n    df = impute_vip_for_no_spend(df)\n    df = impute_vip_for_children(df)\n    df = impute_vip_for_earthlings(df)\n    df = impute_vip_for_martians(df)\n    return df\n\ntrain_df = impute_vip(train_df)\ntest_df = impute_vip(test_df)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.257199Z","iopub.execute_input":"2022-06-08T21:13:03.258175Z","iopub.status.idle":"2022-06-08T21:13:03.300456Z","shell.execute_reply.started":"2022-06-08T21:13:03.258112Z","shell.execute_reply":"2022-06-08T21:13:03.299336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"VIP\"].isna().sum(), test_df[\"VIP\"].isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.301906Z","iopub.execute_input":"2022-06-08T21:13:03.302432Z","iopub.status.idle":"2022-06-08T21:13:03.312972Z","shell.execute_reply.started":"2022-06-08T21:13:03.302388Z","shell.execute_reply":"2022-06-08T21:13:03.311859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The remaining values are filled using the proportion of VIPs and non-VIPs. ","metadata":{}},{"cell_type":"code","source":"def impute_vip_by_prob(df):\n    probs = df[\"VIP\"].value_counts() / df[\"VIP\"].notna().sum()\n    values = np.random.choice([False, True], size=df[\"VIP\"].isna().sum(), p=probs)\n    df.loc[df[\"VIP\"].isna(), \"VIP\"] = values\n    df[\"VIP\"] = df[\"VIP\"].astype(bool)\n    return df\n\ntrain_df = impute_vip_by_prob(train_df)\ntest_df = impute_vip_by_prob(test_df)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.314346Z","iopub.execute_input":"2022-06-08T21:13:03.314736Z","iopub.status.idle":"2022-06-08T21:13:03.34135Z","shell.execute_reply.started":"2022-06-08T21:13:03.314701Z","shell.execute_reply":"2022-06-08T21:13:03.340325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"VIP\"].isna().sum(), test_df[\"VIP\"].isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.34279Z","iopub.execute_input":"2022-06-08T21:13:03.343187Z","iopub.status.idle":"2022-06-08T21:13:03.351882Z","shell.execute_reply.started":"2022-06-08T21:13:03.343152Z","shell.execute_reply":"2022-06-08T21:13:03.350819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Drop Unnecessary Features\n\n`PassengerID`, `Cabin` and `Name` will be dropped from the dataset. It may be so that there are features that can be extracted from `Name` but most of the other Kagglers have reported decrease in accuracy when using any features from `Name`.","metadata":{}},{"cell_type":"code","source":"drop = [\"PassengerId\", \"Cabin\", \"Name\"]\ntrain_df = train_df.drop(drop, axis=1)\ntest_df = test_df.drop(drop, axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.353135Z","iopub.execute_input":"2022-06-08T21:13:03.353961Z","iopub.status.idle":"2022-06-08T21:13:03.371717Z","shell.execute_reply.started":"2022-06-08T21:13:03.353922Z","shell.execute_reply":"2022-06-08T21:13:03.370358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.373834Z","iopub.execute_input":"2022-06-08T21:13:03.374463Z","iopub.status.idle":"2022-06-08T21:13:03.413948Z","shell.execute_reply.started":"2022-06-08T21:13:03.374411Z","shell.execute_reply":"2022-06-08T21:13:03.413099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, all the null values are in the numerical columns.","metadata":{}},{"cell_type":"code","source":"train_df.isna().any()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.415368Z","iopub.execute_input":"2022-06-08T21:13:03.416273Z","iopub.status.idle":"2022-06-08T21:13:03.434839Z","shell.execute_reply.started":"2022-06-08T21:13:03.41623Z","shell.execute_reply":"2022-06-08T21:13:03.433839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Encoding Categorical Variables (Except `CabinNum` and `GroupId`)\n\nTo encode the features, we will combine the two datasets into one. The functions `concat_train_test()` and `split_train_test()` handle the combining and resplitting of the datasets respectively.\n\n`CabinNum` and `GroupSize` will not be not be encoded since they are part of the experiment.","metadata":{}},{"cell_type":"code","source":"# Take out labels from training data\ndef concat_train_test(train, test, has_labels=False):\n    transported = None\n    \n    # Since the test set doesn't have labels\n    # If there are labels in the train set\n    # They need to be dropped\n    if has_labels is True:\n        transported = train[\"Transported\"].copy()\n        train = train.drop(\"Transported\", axis=1)\n\n    # Store indices so that they can be used to\n    # Split the dataset again\n    train_index = train.index\n    test_index = test.index\n\n    # Concatenate the two datasets\n    df = pd.concat([train, test])\n\n    return df, train_index, test_index, transported\n\n\ndef split_train_test(df, train_index, test_index, transported=None):\n    # Get the training set in the df according to index\n    train_df = df.loc[train_index, :]\n    \n    # If transported is passed\n    # Add it to the dataframe\n    if transported is not None:\n        train_df[\"Transported\"] = transported\n        \n    # Get the test set in the df according to the index\n    test_df = df.loc[test_index, :]\n    \n    return train_df, test_df","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.436111Z","iopub.execute_input":"2022-06-08T21:13:03.437121Z","iopub.status.idle":"2022-06-08T21:13:03.449169Z","shell.execute_reply.started":"2022-06-08T21:13:03.437063Z","shell.execute_reply":"2022-06-08T21:13:03.447742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Combine the datasets\ndf, train_idx, test_idx, transported = concat_train_test(train_df, test_df, has_labels=True)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.451169Z","iopub.execute_input":"2022-06-08T21:13:03.452016Z","iopub.status.idle":"2022-06-08T21:13:03.504156Z","shell.execute_reply.started":"2022-06-08T21:13:03.45196Z","shell.execute_reply":"2022-06-08T21:13:03.502742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In case of logistic regression, all binary categorical features need to be encoded as `0` or `1`, while non-binary categorical features need to be one-hot encoded.\n\nWe will first convert all Boolean columns to `int` so that they are `0` or `1`.","metadata":{}},{"cell_type":"code","source":"def bool2int(df):\n    # Find all bool columns\n    columns = [column for column in df.columns if df[column].dtype.name == \"bool\"]\n    # Convert to integer\n    df[columns] = df[columns].astype(int)\n    \n    return df\n\n\ndf = bool2int(df)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.505984Z","iopub.execute_input":"2022-06-08T21:13:03.506376Z","iopub.status.idle":"2022-06-08T21:13:03.522178Z","shell.execute_reply.started":"2022-06-08T21:13:03.506343Z","shell.execute_reply":"2022-06-08T21:13:03.520625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.523967Z","iopub.execute_input":"2022-06-08T21:13:03.524518Z","iopub.status.idle":"2022-06-08T21:13:03.557952Z","shell.execute_reply.started":"2022-06-08T21:13:03.524467Z","shell.execute_reply":"2022-06-08T21:13:03.556745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, we will encode `CabinSide` as `0` or `1`.","metadata":{}},{"cell_type":"code","source":"df[\"CabinSide\"] = df[\"CabinSide\"].map({\"S\": 0, \"P\": 1})","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.559409Z","iopub.execute_input":"2022-06-08T21:13:03.560438Z","iopub.status.idle":"2022-06-08T21:13:03.569452Z","shell.execute_reply.started":"2022-06-08T21:13:03.560395Z","shell.execute_reply":"2022-06-08T21:13:03.568156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, we will use `pd.get_dummies()` to one-hot encode the non-binary features.","metadata":{}},{"cell_type":"code","source":"to_be_encoded = [\"HomePlanet\", \"Destination\", \"GroupSize\", \"CabinDeck\"]\ndf = pd.get_dummies(df, columns=to_be_encoded)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.570968Z","iopub.execute_input":"2022-06-08T21:13:03.571529Z","iopub.status.idle":"2022-06-08T21:13:03.607993Z","shell.execute_reply.started":"2022-06-08T21:13:03.571468Z","shell.execute_reply":"2022-06-08T21:13:03.606666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.609406Z","iopub.execute_input":"2022-06-08T21:13:03.609891Z","iopub.status.idle":"2022-06-08T21:13:03.639989Z","shell.execute_reply.started":"2022-06-08T21:13:03.60985Z","shell.execute_reply":"2022-06-08T21:13:03.638947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.columns","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.646177Z","iopub.execute_input":"2022-06-08T21:13:03.646638Z","iopub.status.idle":"2022-06-08T21:13:03.653119Z","shell.execute_reply.started":"2022-06-08T21:13:03.646601Z","shell.execute_reply":"2022-06-08T21:13:03.652318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we will split the dataset again before imputing numerical missing values. The splitting is important since while encoding is not influenced by the distribution of datapoints across the two datasets, imputation of missing values is. We want the two datasets to maintain the difference in their distributions and not be influenced by each other.","metadata":{}},{"cell_type":"code","source":"train_df, test_df = split_train_test(df, train_idx, test_idx, transported=transported)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.654427Z","iopub.execute_input":"2022-06-08T21:13:03.654918Z","iopub.status.idle":"2022-06-08T21:13:03.683478Z","shell.execute_reply.started":"2022-06-08T21:13:03.654882Z","shell.execute_reply":"2022-06-08T21:13:03.682186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.685181Z","iopub.execute_input":"2022-06-08T21:13:03.685681Z","iopub.status.idle":"2022-06-08T21:13:03.718098Z","shell.execute_reply.started":"2022-06-08T21:13:03.685634Z","shell.execute_reply":"2022-06-08T21:13:03.717049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.722038Z","iopub.execute_input":"2022-06-08T21:13:03.722461Z","iopub.status.idle":"2022-06-08T21:13:03.753723Z","shell.execute_reply.started":"2022-06-08T21:13:03.722427Z","shell.execute_reply":"2022-06-08T21:13:03.752675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Numerical Features Missing Values\n\nAll missing values in numerical features are imputed using KNN.\n\nFor now, we will drop `GroupId` and `CabinNum`. The goal is to train a total of 7 baselines:\n1. Without `CabinNum` and `GroupId`\n2. Only with `CabinNum` One-Hot Encoded.\n3. Only with `CabinNum` Label Encoded\n4. Only with `GroupId` One-Hot Encoded.\n5. Only with `GroupId` Label Encoded.\n6. With both `CabinNum` and `GroupId` One-Hot Encoded.\n7. With both `CabinNum` and `GroupId` Label Encoded.\n\nThis will tell us whether these features actually contribute to the model or not and the best encoding for them, allowing us to keep the best features.","metadata":{}},{"cell_type":"code","source":"def impute_missing_using_knn(df, numeric_cols, has_labels=False):\n    x = df\n    \n    # We should not use the labels for imputing\n    # So, if there are labels, drop them\n    if has_labels is True:\n        transported = df[\"Transported\"]\n        x = df.drop(\"Transported\", axis=1)\n        \n    # Standardize the numerical columns\n    scaler = preprocessing.StandardScaler()\n    x[numeric_cols] = scaler.fit_transform(x[numeric_cols])\n    \n    # Impute missing values\n    imputer = impute.KNNImputer(n_neighbors=5, weights=\"distance\")\n    # Note: x is now a NumPy array\n    x = imputer.fit_transform(x)\n    \n    # Add the labels again if they were dropped\n    if has_labels is True:\n        x = np.hstack((x, transported.values.reshape(-1, 1)))\n        \n    return pd.DataFrame(x, columns=df.columns, index=df.index)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.75512Z","iopub.execute_input":"2022-06-08T21:13:03.755496Z","iopub.status.idle":"2022-06-08T21:13:03.76612Z","shell.execute_reply.started":"2022-06-08T21:13:03.755462Z","shell.execute_reply":"2022-06-08T21:13:03.765053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Store CabinNum and GroupId for later use before dropping\ntrain_cabin_num = train_df[\"CabinNum\"]\ntrain_group_id = train_df[\"GroupId\"]\n\ntest_cabin_num = test_df[\"CabinNum\"]\ntest_group_id = test_df[\"GroupId\"]","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.76747Z","iopub.execute_input":"2022-06-08T21:13:03.768431Z","iopub.status.idle":"2022-06-08T21:13:03.78709Z","shell.execute_reply.started":"2022-06-08T21:13:03.768357Z","shell.execute_reply":"2022-06-08T21:13:03.786075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"to_drop = [\"GroupId\", \"CabinNum\"]\nnumeric_cols = [\"Age\", \"TotalExpense\"] + expenditure_columns\n\ntrain_df = impute_missing_using_knn(train_df.drop(to_drop, axis=1), numeric_cols, has_labels=True)\ntest_df = impute_missing_using_knn(test_df.drop(to_drop, axis=1), numeric_cols)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:03.789205Z","iopub.execute_input":"2022-06-08T21:13:03.790629Z","iopub.status.idle":"2022-06-08T21:13:05.775187Z","shell.execute_reply.started":"2022-06-08T21:13:03.790582Z","shell.execute_reply":"2022-06-08T21:13:05.77416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:05.77671Z","iopub.execute_input":"2022-06-08T21:13:05.777334Z","iopub.status.idle":"2022-06-08T21:13:05.810452Z","shell.execute_reply.started":"2022-06-08T21:13:05.777297Z","shell.execute_reply":"2022-06-08T21:13:05.809565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:05.811977Z","iopub.execute_input":"2022-06-08T21:13:05.812529Z","iopub.status.idle":"2022-06-08T21:13:05.85064Z","shell.execute_reply.started":"2022-06-08T21:13:05.81249Z","shell.execute_reply":"2022-06-08T21:13:05.849155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.isna().any()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:05.852359Z","iopub.execute_input":"2022-06-08T21:13:05.853008Z","iopub.status.idle":"2022-06-08T21:13:05.869019Z","shell.execute_reply.started":"2022-06-08T21:13:05.852964Z","shell.execute_reply":"2022-06-08T21:13:05.867957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:05.870625Z","iopub.execute_input":"2022-06-08T21:13:05.871747Z","iopub.status.idle":"2022-06-08T21:13:05.907137Z","shell.execute_reply.started":"2022-06-08T21:13:05.871698Z","shell.execute_reply":"2022-06-08T21:13:05.906191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Folds\n\nWe will create 5 folds. This leads to a 6954 samples in the training set and 1739 samples in the validation set for each fold.","metadata":{}},{"cell_type":"code","source":"# Reset index so that the output of kf.split() can be used directly\ntrain_df = train_df.reset_index()\n\n# Add column for fold index and initialize kf\ntrain_df[\"kfold\"] = -1\nkf = model_selection.KFold(n_splits=5, random_state=42, shuffle=True)\n\nfor idx, (_, val_idx) in enumerate(kf.split(train_df)):\n    train_df.loc[val_idx, \"kfold\"] = idx\n\n# Restore the index\ntrain_df = train_df.set_index(\"PassengerId\")\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:05.908298Z","iopub.execute_input":"2022-06-08T21:13:05.90927Z","iopub.status.idle":"2022-06-08T21:13:05.956944Z","shell.execute_reply.started":"2022-06-08T21:13:05.909229Z","shell.execute_reply":"2022-06-08T21:13:05.955768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(train_df[train_df[\"kfold\"] != 0])","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:05.958575Z","iopub.execute_input":"2022-06-08T21:13:05.959109Z","iopub.status.idle":"2022-06-08T21:13:05.970324Z","shell.execute_reply.started":"2022-06-08T21:13:05.959055Z","shell.execute_reply":"2022-06-08T21:13:05.969248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's save these two files so that we can reuse them in other notebooks.","metadata":{}},{"cell_type":"code","source":"train_df.to_csv(\"train_prepared.csv\", index=False)\ntest_df.to_csv(\"test_prepared.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:05.971838Z","iopub.execute_input":"2022-06-08T21:13:05.972573Z","iopub.status.idle":"2022-06-08T21:13:06.498168Z","shell.execute_reply.started":"2022-06-08T21:13:05.972525Z","shell.execute_reply":"2022-06-08T21:13:06.497031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Logistic Regression","metadata":{}},{"cell_type":"markdown","source":"We first define the training loop.","metadata":{}},{"cell_type":"code","source":"def train(df):\n    # Add prediction column\n    df[\"preds\"] = pd.NA\n    \n    # Need to drop target, predictions and kfold\n    # In each training iteration\n    drop = [\"Transported\", \"preds\", \"kfold\"]\n    \n    for fold in range(5):\n        train = df[df[\"kfold\"] != fold]\n        \n        # Get training features and labels\n        y_train = train[\"Transported\"].values\n        X_train = train.drop(drop, axis=1).values\n        \n        val = df[df[\"kfold\"] == fold]\n        \n        # Get validation features and labels\n        y_val = val[\"Transported\"].values\n        X_val = val.drop(drop, axis=1).values\n        \n        # The default max_iter is too small for this dataset\n        model = linear_model.LogisticRegression(max_iter=1000)\n        model.fit(X_train, y_train)\n        \n        # Predict on the validation set\n        preds = model.predict(X_val)\n        df.loc[val.index, \"preds\"] = preds\n        \n        # Calculate accuracy\n        acc = metrics.accuracy_score(y_val, preds)\n        print(f\"Fold {fold + 1} - Accuracy = {acc: .4f}\")\n    \n    # Convert target, prediction to integer\n    df[drop] = df[drop].astype(int)\n    \n    # Calculate overall accuracy\n    acc = metrics.accuracy_score(df[\"Transported\"].values, df[\"preds\"].values)\n    print(f\"Overall accuracy = {acc: .4f}\")\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:06.49946Z","iopub.execute_input":"2022-06-08T21:13:06.500375Z","iopub.status.idle":"2022-06-08T21:13:06.512212Z","shell.execute_reply.started":"2022-06-08T21:13:06.500337Z","shell.execute_reply":"2022-06-08T21:13:06.510863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1. Without `CabinNum` and `GroupId`","metadata":{}},{"cell_type":"code","source":"# Send copy so that original dataset remains unchanged\ntrain_df_exp1 = train(train_df.copy())","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:06.514214Z","iopub.execute_input":"2022-06-08T21:13:06.515605Z","iopub.status.idle":"2022-06-08T21:13:07.428798Z","shell.execute_reply.started":"2022-06-08T21:13:06.515547Z","shell.execute_reply":"2022-06-08T21:13:07.426996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Around 61% of the mistakes are for passengers traveling alone.","metadata":{}},{"cell_type":"code","source":"wrong = train_df_exp1[train_df_exp1[\"Transported\"] != train_df_exp1[\"preds\"]]\nwrong[\"GroupSize_1\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:07.431266Z","iopub.execute_input":"2022-06-08T21:13:07.432561Z","iopub.status.idle":"2022-06-08T21:13:07.457317Z","shell.execute_reply.started":"2022-06-08T21:13:07.432499Z","shell.execute_reply":"2022-06-08T21:13:07.456193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Only With `CabinNum` One-Hot Encoded\n\nHere we one-hot encode `CabinNum`. For this, we need to combine the train and test sets again and use `pd.get_dummies()`. We define a helper function which can be reused for `GroupId`. This function takes the training and test dataframes, the label that should be assigned to the column being encoded, and the training and test values for the column.","metadata":{}},{"cell_type":"code","source":"def add_onehot_column(train_df, test_df, column_label, train_values, test_values):\n    folds = train_df[\"kfold\"]\n    \n    # Create copies so that original dataframes remain unchanged\n    train_df_with_col = train_df.drop(\"kfold\", axis=1)\n    test_df_with_col = test_df.copy()\n\n    # Add the column to the dataframes\n    train_df_with_col[column_label] = train_values\n    test_df_with_col[column_label] = test_values\n\n    # Merge, one-hot encode and then split\n    df, train_idx, test_idx, transported = concat_train_test(\n        train=train_df_with_col,\n        test=test_df_with_col,\n        has_labels=True\n    )\n    \n    df = pd.get_dummies(df, columns=[column_label])\n    \n    train_df_with_col, test_df_with_col = split_train_test(\n        df=df,\n        train_index=train_idx,\n        test_index=test_idx,\n        transported=transported\n    )\n    \n    # Add the folds column\n    train_df_with_col[\"kfold\"] = folds\n    \n    return train_df_with_col, test_df_with_col","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:07.459078Z","iopub.execute_input":"2022-06-08T21:13:07.459805Z","iopub.status.idle":"2022-06-08T21:13:07.472338Z","shell.execute_reply.started":"2022-06-08T21:13:07.459734Z","shell.execute_reply":"2022-06-08T21:13:07.470994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get a new dataframe with one-hot encoded `CabinNum`\ntrain_df_cabinnum_oh, test_df_cabinnum_oh = add_onehot_column(\n    train_df=train_df,\n    test_df=test_df,\n    column_label=\"CabinNum\",\n    train_values=train_cabin_num,\n    test_values=test_cabin_num,\n)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:07.474148Z","iopub.execute_input":"2022-06-08T21:13:07.474887Z","iopub.status.idle":"2022-06-08T21:13:07.875287Z","shell.execute_reply.started":"2022-06-08T21:13:07.474836Z","shell.execute_reply":"2022-06-08T21:13:07.87335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_cabinnum_oh.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:07.877405Z","iopub.execute_input":"2022-06-08T21:13:07.877865Z","iopub.status.idle":"2022-06-08T21:13:07.914309Z","shell.execute_reply.started":"2022-06-08T21:13:07.87783Z","shell.execute_reply":"2022-06-08T21:13:07.912691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df_cabinnum_oh.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:07.915887Z","iopub.execute_input":"2022-06-08T21:13:07.916374Z","iopub.status.idle":"2022-06-08T21:13:07.949357Z","shell.execute_reply.started":"2022-06-08T21:13:07.916324Z","shell.execute_reply":"2022-06-08T21:13:07.947973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_exp2 = train(train_df_cabinnum_oh.copy())","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:07.951027Z","iopub.execute_input":"2022-06-08T21:13:07.951497Z","iopub.status.idle":"2022-06-08T21:13:22.897702Z","shell.execute_reply.started":"2022-06-08T21:13:07.951437Z","shell.execute_reply":"2022-06-08T21:13:22.896266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Around 60% of the mistakes are in case of passengers traveling alone.","metadata":{}},{"cell_type":"code","source":"wrong = train_df_exp2[train_df_exp2[\"Transported\"] != train_df_exp2[\"preds\"]]\nwrong[\"GroupSize_1\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:22.900359Z","iopub.execute_input":"2022-06-08T21:13:22.901518Z","iopub.status.idle":"2022-06-08T21:13:22.979004Z","shell.execute_reply.started":"2022-06-08T21:13:22.901449Z","shell.execute_reply":"2022-06-08T21:13:22.977092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Only With `CabinNum` Label Encoded\n\nFor this, we need to merge the two datasets, label encode `CabinNum` and then split them. Let's create a function which can be reused for `GroupId`. This function is similar to the one used for one-hot encoding and takes the same parameters.","metadata":{}},{"cell_type":"code","source":"def add_labelencoded_column(train_df, test_df, column_label, train_values, test_values):\n    folds = train_df[\"kfold\"]\n    \n    # Create copies so that original dataframes remain unchanged\n    train_df_with_col = train_df.drop(\"kfold\", axis=1)\n    test_df_with_col = test_df.copy()\n\n    # Add the column to the dataframes\n    train_df_with_col[column_label] = train_values\n    test_df_with_col[column_label] = test_values\n\n    # Merge, label encode and then split\n    df, train_idx, test_idx, transported = concat_train_test(\n        train=train_df_with_col,\n        test=test_df_with_col,\n        has_labels=True\n    )\n    \n    # To label encode, we first get the labels\n    # Then, we create a dictionary like {'a': 0, 'b': 1, ...}\n    # Finally, we use the .map() method to change the values\n    levels = df[column_label].value_counts().index\n    mapping = {level: idx for idx, level in enumerate(levels)}\n    df[column_label] = df[column_label].map(mapping)\n    \n    train_df_with_col, test_df_with_col = split_train_test(\n        df=df,\n        train_index=train_idx,\n        test_index=test_idx,\n        transported=transported)\n    \n    # Add the folds column\n    train_df_with_col[\"kfold\"] = folds\n    \n    return train_df_with_col, test_df_with_col","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:22.987852Z","iopub.execute_input":"2022-06-08T21:13:22.98907Z","iopub.status.idle":"2022-06-08T21:13:23.003288Z","shell.execute_reply.started":"2022-06-08T21:13:22.988994Z","shell.execute_reply":"2022-06-08T21:13:23.001254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Add CabinNum as a label encoded column\ntrain_df_cabinnum_le, test_df_cabinnum_le = add_labelencoded_column(\n    train_df=train_df,\n    test_df=test_df,\n    column_label=\"CabinNum\",\n    train_values=train_cabin_num,\n    test_values=test_cabin_num\n)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:23.005234Z","iopub.execute_input":"2022-06-08T21:13:23.006434Z","iopub.status.idle":"2022-06-08T21:13:23.058317Z","shell.execute_reply.started":"2022-06-08T21:13:23.006385Z","shell.execute_reply":"2022-06-08T21:13:23.057115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_cabinnum_le.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:23.060573Z","iopub.execute_input":"2022-06-08T21:13:23.061041Z","iopub.status.idle":"2022-06-08T21:13:23.095033Z","shell.execute_reply.started":"2022-06-08T21:13:23.060994Z","shell.execute_reply":"2022-06-08T21:13:23.094054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df_cabinnum_le.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:23.097377Z","iopub.execute_input":"2022-06-08T21:13:23.098222Z","iopub.status.idle":"2022-06-08T21:13:23.141237Z","shell.execute_reply.started":"2022-06-08T21:13:23.098175Z","shell.execute_reply":"2022-06-08T21:13:23.140183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_exp3 = train(train_df_cabinnum_le.copy())","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:23.142582Z","iopub.execute_input":"2022-06-08T21:13:23.143025Z","iopub.status.idle":"2022-06-08T21:13:25.849846Z","shell.execute_reply.started":"2022-06-08T21:13:23.142987Z","shell.execute_reply":"2022-06-08T21:13:25.848601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Around 60.2% of the mistakes are in case of passengers who were traveling alone.","metadata":{}},{"cell_type":"code","source":"wrong = train_df_exp3[train_df_exp3[\"Transported\"] != train_df_exp3[\"preds\"]]\nwrong[\"GroupSize_1\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:25.852286Z","iopub.execute_input":"2022-06-08T21:13:25.853405Z","iopub.status.idle":"2022-06-08T21:13:25.872039Z","shell.execute_reply.started":"2022-06-08T21:13:25.853339Z","shell.execute_reply":"2022-06-08T21:13:25.870228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4. Only with `GroupId` One-Hot Encoded\n\nThe process will be similar to `CabinNum`.","metadata":{}},{"cell_type":"code","source":"# Get a new dataframe with one-hot encoded GroupId\ntrain_df_groupid_oh, test_df_groupid_oh = add_onehot_column(\n    train_df=train_df.copy(),\n    test_df=test_df.copy(),\n    column_label=\"GroupId\",\n    train_values=train_group_id,\n    test_values=test_group_id,\n)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:25.874482Z","iopub.execute_input":"2022-06-08T21:13:25.87555Z","iopub.status.idle":"2022-06-08T21:13:27.60299Z","shell.execute_reply.started":"2022-06-08T21:13:25.875491Z","shell.execute_reply":"2022-06-08T21:13:27.601802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_groupid_oh.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:27.604425Z","iopub.execute_input":"2022-06-08T21:13:27.604909Z","iopub.status.idle":"2022-06-08T21:13:27.642301Z","shell.execute_reply.started":"2022-06-08T21:13:27.604803Z","shell.execute_reply":"2022-06-08T21:13:27.640878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df_groupid_oh.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:27.643949Z","iopub.execute_input":"2022-06-08T21:13:27.644348Z","iopub.status.idle":"2022-06-08T21:13:27.674876Z","shell.execute_reply.started":"2022-06-08T21:13:27.644315Z","shell.execute_reply":"2022-06-08T21:13:27.673346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_exp4 = train(train_df_groupid_oh.copy())","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:13:27.67635Z","iopub.execute_input":"2022-06-08T21:13:27.677324Z","iopub.status.idle":"2022-06-08T21:14:35.817121Z","shell.execute_reply.started":"2022-06-08T21:13:27.677281Z","shell.execute_reply":"2022-06-08T21:14:35.815842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Around 60.7% of the mistakes are in case of passengers who were traveling alone.","metadata":{}},{"cell_type":"code","source":"wrong = train_df_exp4[train_df_exp4[\"Transported\"] != train_df_exp4[\"preds\"]]\nwrong[\"GroupSize_1\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:14:35.819294Z","iopub.execute_input":"2022-06-08T21:14:35.820332Z","iopub.status.idle":"2022-06-08T21:14:35.972723Z","shell.execute_reply.started":"2022-06-08T21:14:35.820269Z","shell.execute_reply":"2022-06-08T21:14:35.971959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5. Only With `GroupId` Label Encoded\n\nThe process will be similar to `CabinNum`.","metadata":{}},{"cell_type":"code","source":"# Add GroupId as a label encoded column\ntrain_df_groupid_le, test_df_groupid_le = add_labelencoded_column(\n    train_df=train_df,\n    test_df=test_df,\n    column_label=\"GroupId\",\n    train_values=train_group_id,\n    test_values=test_group_id\n)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:14:35.974206Z","iopub.execute_input":"2022-06-08T21:14:35.974801Z","iopub.status.idle":"2022-06-08T21:14:36.020014Z","shell.execute_reply.started":"2022-06-08T21:14:35.974763Z","shell.execute_reply":"2022-06-08T21:14:36.01895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_groupid_le.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:14:36.02171Z","iopub.execute_input":"2022-06-08T21:14:36.022106Z","iopub.status.idle":"2022-06-08T21:14:36.057834Z","shell.execute_reply.started":"2022-06-08T21:14:36.022073Z","shell.execute_reply":"2022-06-08T21:14:36.056839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df_groupid_le.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:14:36.059432Z","iopub.execute_input":"2022-06-08T21:14:36.059823Z","iopub.status.idle":"2022-06-08T21:14:36.099807Z","shell.execute_reply.started":"2022-06-08T21:14:36.059788Z","shell.execute_reply":"2022-06-08T21:14:36.098626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_exp5 = train(train_df_groupid_le.copy())","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:14:36.101422Z","iopub.execute_input":"2022-06-08T21:14:36.102569Z","iopub.status.idle":"2022-06-08T21:14:37.463868Z","shell.execute_reply.started":"2022-06-08T21:14:36.102511Z","shell.execute_reply":"2022-06-08T21:14:37.462476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Around 60.2% of the mistakes are in case of passengers who were traveling alone.","metadata":{}},{"cell_type":"code","source":"wrong = train_df_exp5[train_df_exp5[\"Transported\"] != train_df_exp5[\"preds\"]]\nwrong[\"GroupSize_1\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:14:37.466442Z","iopub.execute_input":"2022-06-08T21:14:37.4676Z","iopub.status.idle":"2022-06-08T21:14:37.486546Z","shell.execute_reply.started":"2022-06-08T21:14:37.467532Z","shell.execute_reply":"2022-06-08T21:14:37.484509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6. With Both `CabinNum` and `GroupId` One-Hot Encoded","metadata":{}},{"cell_type":"code","source":"# Get new dataframe with one-hot encoded CabinNum\ntrain_df_both_oh, test_df_both_oh = add_onehot_column(\n    train_df=train_df,\n    test_df=test_df,\n    column_label=\"CabinNum\",\n    train_values=train_cabin_num,\n    test_values=test_cabin_num,\n)\n\n# Get final dataframe with one-hot encoded GroupId\ntrain_df_both_oh, test_df_both_oh = add_onehot_column(\n    train_df=train_df_both_oh,\n    test_df=test_df_both_oh,\n    column_label=\"GroupId\",\n    train_values=train_group_id,\n    test_values=test_group_id,\n)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:14:37.489305Z","iopub.execute_input":"2022-06-08T21:14:37.490536Z","iopub.status.idle":"2022-06-08T21:14:40.026076Z","shell.execute_reply.started":"2022-06-08T21:14:37.490464Z","shell.execute_reply":"2022-06-08T21:14:40.024867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_both_oh.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:14:40.027433Z","iopub.execute_input":"2022-06-08T21:14:40.027831Z","iopub.status.idle":"2022-06-08T21:14:40.06085Z","shell.execute_reply.started":"2022-06-08T21:14:40.02779Z","shell.execute_reply":"2022-06-08T21:14:40.059802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df_both_oh.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:14:40.062279Z","iopub.execute_input":"2022-06-08T21:14:40.062785Z","iopub.status.idle":"2022-06-08T21:14:40.094071Z","shell.execute_reply.started":"2022-06-08T21:14:40.062725Z","shell.execute_reply":"2022-06-08T21:14:40.09278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_exp6 = train(train_df_both_oh.copy())","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:14:40.095746Z","iopub.execute_input":"2022-06-08T21:14:40.096287Z","iopub.status.idle":"2022-06-08T21:16:50.222992Z","shell.execute_reply.started":"2022-06-08T21:14:40.096239Z","shell.execute_reply":"2022-06-08T21:16:50.22111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Around 60% of the mistakes are in case of passengers who were traveling alone.","metadata":{}},{"cell_type":"code","source":"wrong = train_df_exp6[train_df_exp6[\"Transported\"] != train_df_exp6[\"preds\"]]\nwrong[\"GroupSize_1\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:16:50.231345Z","iopub.execute_input":"2022-06-08T21:16:50.235892Z","iopub.status.idle":"2022-06-08T21:16:50.420499Z","shell.execute_reply.started":"2022-06-08T21:16:50.235782Z","shell.execute_reply":"2022-06-08T21:16:50.418889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 7. With Both `CabinNum` and `GroupId` Label Encoded","metadata":{}},{"cell_type":"code","source":"# Add CabinNum as a label encoded column\ntrain_df_both_le, test_df_both_le = add_labelencoded_column(\n    train_df=train_df,\n    test_df=test_df,\n    column_label=\"CabinNum\",\n    train_values=train_cabin_num,\n    test_values=test_cabin_num\n)\n\n# Add GroupId as a label encoded column\ntrain_df_both_le, test_df_both_le = add_labelencoded_column(\n    train_df=train_df_both_le,\n    test_df=test_df_both_le,\n    column_label=\"GroupId\",\n    train_values=train_group_id,\n    test_values=test_group_id\n)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:16:50.422227Z","iopub.execute_input":"2022-06-08T21:16:50.42263Z","iopub.status.idle":"2022-06-08T21:16:50.506622Z","shell.execute_reply.started":"2022-06-08T21:16:50.422596Z","shell.execute_reply":"2022-06-08T21:16:50.505395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_both_le.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:16:50.508321Z","iopub.execute_input":"2022-06-08T21:16:50.508914Z","iopub.status.idle":"2022-06-08T21:16:50.54413Z","shell.execute_reply.started":"2022-06-08T21:16:50.508861Z","shell.execute_reply":"2022-06-08T21:16:50.542996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df_both_le.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:16:50.545477Z","iopub.execute_input":"2022-06-08T21:16:50.546002Z","iopub.status.idle":"2022-06-08T21:16:50.582121Z","shell.execute_reply.started":"2022-06-08T21:16:50.545961Z","shell.execute_reply":"2022-06-08T21:16:50.580839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_exp7 = train(train_df_both_le.copy())","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:16:50.58398Z","iopub.execute_input":"2022-06-08T21:16:50.584496Z","iopub.status.idle":"2022-06-08T21:16:51.923285Z","shell.execute_reply.started":"2022-06-08T21:16:50.584454Z","shell.execute_reply":"2022-06-08T21:16:51.921984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Around 60% of the mistakes are in case of passengers who were traveling alone.","metadata":{}},{"cell_type":"code","source":"wrong = train_df_exp7[train_df_exp7[\"Transported\"] != train_df_exp7[\"preds\"]]\nwrong[\"GroupSize_1\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:16:51.925618Z","iopub.execute_input":"2022-06-08T21:16:51.926743Z","iopub.status.idle":"2022-06-08T21:16:51.951954Z","shell.execute_reply.started":"2022-06-08T21:16:51.926667Z","shell.execute_reply":"2022-06-08T21:16:51.949795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Additional Files\n\nThe following three experiments yield close results:\n\n- With `CabinNum` label encoded\n- With `GroupId` label encoded\n- With both `CabinNum` and `GroupId` label encoded.\n\nThus, we will also save the datasets used for these experiments so that they can be used in future notebooks.","metadata":{}},{"cell_type":"code","source":"# `CabinNum` label encoded\ntrain_df_cabinnum_le.to_csv(\"train_prepared_cabinnum_le.csv\", index=False)\ntest_df_cabinnum_le.to_csv(\"test_prepared_cabinnum_le.csv\")\n\n# `GroupId` label encoded\ntrain_df_groupid_le.to_csv(\"train_prepared_groupid_le.csv\", index=False)\ntest_df_groupid_le.to_csv(\"test_prepared_groupid_le.csv\")\n\n# Both label encoded\ntrain_df_both_le.to_csv(\"train_prepared_both_le.csv\", index=False)\ntest_df_both_le.to_csv(\"test_prepared_both_le.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-06-08T21:17:56.055908Z","iopub.execute_input":"2022-06-08T21:17:56.056524Z","iopub.status.idle":"2022-06-08T21:17:56.097256Z","shell.execute_reply.started":"2022-06-08T21:17:56.056478Z","shell.execute_reply":"2022-06-08T21:17:56.095838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion\n\nThe experiment suggests that we are better off not using `CabinNum` in the model and using `GroupId` with label encoding. Across the board, label encoding has beaten one-hot encoding. This is most likely due to the dense nature of these features.\n\nWe also find out that most of the mistakes are in case of passengers who were traveling alone. In each model, passengers traveling alone make up ~60% of the mistakes. The most likely culprit behind this is how the missing values are computed but this requires further investigation.\n\n\n|                   **Experiment**                   | **Fold 1** | **Fold 2** | **Fold 3** | **Fold 4** | **Fold 5** | **Overall** |\n|:--------------------------------------------------:|:----------:|:----------:|:----------:|:----------:|:----------:|:-----------:|\n|          Without `CabinNum` and `GroupId`          |   0.7867   |   0.7861   |   0.7936   |   0.7975   |   0.7969   |    0.7921   |\n|           With only `CabinNum` (One-Hot)           |   0.7832   |   0.7872   |   0.7941   |   0.7992   |   0.7831   |    0.7894   |\n|        With only `CabinNum` (Label Encoded)        |   0.7849   | **0.7913** |   0.7953   |   0.7952   | **0.8021** |    0.7937   |\n|            With only `GroupId` (One-Hot)           |   0.7821   |   0.7878   |   0.7959   |   0.7957   |   0.7940   |    0.7911   |\n|         With only `GroupId` (Label Encoded)        | **0.7913** |   0.7890   | **0.7999** |   0.7940   |   0.7975   |  **0.7943** |\n|    With both `CabinNum` and `GroupId` (One-Hot)    |   0.7832   |   0.7890   |   0.7878   | **0.8003** |   0.7854   |    0.7891   |\n| With both `CabinNum` and `GroupId` (Label Encoded) |   0.7815   |   0.7901   |   0.7993   |   0.7900   |   0.7992   |    0.7920   |\n\n\nThank you for reading!","metadata":{}}]}