{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# SS Titanic EDA\n\nby David Zambrano\n\n<img width=\"300\" src=\"https://cdn.drawception.com/drawings/KO3D1hhjOk.png\">\n\ncredits to the author of the image.\n\n## **Competition context:**\n\nWelcome to the year 2912, where your data science skills are needed to solve a cosmic mystery. We've received a transmission from four lightyears away and things aren't looking good.\n\nThe Spaceship Titanic was an interstellar passenger liner launched a month ago. With almost 13,000 passengers on board, the vessel set out on its maiden voyage transporting emigrants from our solar system to three newly habitable exoplanets orbiting nearby stars.\n\nWhile rounding Alpha Centauri en route to its first destination—the torrid 55 Cancri E—the unwary Spaceship Titanic collided with a spacetime anomaly hidden within a dust cloud. Sadly, it met a similar fate as its namesake from 1000 years before. Though the ship stayed intact, almost half of the passengers were transported to an alternate dimension!\n\nTo help rescue crews and retrieve the lost passengers, you are challenged to predict which passengers were transported by the anomaly using records recovered from the spaceship’s damaged computer system.\n\nFor more information about this competition click [here](https://www.kaggle.com/competitions/spaceship-titanic).","metadata":{"id":"7Lfb7QqqrWDN"}},{"cell_type":"markdown","source":"## Data Description:\n\n**File name:** train.csv - Contains personal records for about two-thirds (~8700) of the passengers, to be used as training data.\n\n\n> **PassengerId** - A unique Id for each passenger. Each Id takes the form gggg_pp where gggg indicates a group the passenger is travelling with and pp is their number within the group. People in a group are often family members, but not always.\n\n> **HomePlanet** - The planet the passenger departed from, typically their planet of permanent residence.\n\n> **CryoSleep** - Indicates whether the passenger elected to be put into suspended animation for the duration of the voyage. Passengers in cryosleep are confined to their cabins.\n\n> **Cabin** - The cabin number where the passenger is staying. Takes the form deck/num/side, where side can be either P for Port or S for Starboard.\n\n> **Destination** - The planet the passenger will be debarking to.\n\n> **Age** - The age of the passenger.\n\n> **VIP** - Whether the passenger has paid for special VIP service during the voyage.\n\n> **RoomService, FoodCourt, ShoppingMall, Spa, VRDeck** - Amount the passenger has billed at each of the Spaceship Titanic's many luxury amenities.\n\n> **Name** - The first and last names of the passenger.\n\n> **Transported** - Whether the passenger was transported to another dimension. This is the target, the column you are trying to predict.","metadata":{"id":"97VhY3vZ7lDw"}},{"cell_type":"markdown","source":"## Train Dataset EDA:","metadata":{"id":"OnEir4T2j9X_"}},{"cell_type":"code","source":"import pandas as pd\n\ntrain_dataset = pd.read_csv('../input/spaceship-titanic/train.csv')\ntrain_dataset.head()","metadata":{"id":"UMvIgEpl61jc","outputId":"0a76c834-ce5a-43a9-e8f7-03d22287008e","execution":{"iopub.status.busy":"2022-08-05T14:46:15.788464Z","iopub.execute_input":"2022-08-05T14:46:15.788972Z","iopub.status.idle":"2022-08-05T14:46:15.899125Z","shell.execute_reply.started":"2022-08-05T14:46:15.788866Z","shell.execute_reply":"2022-08-05T14:46:15.897813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Basic Statistics:","metadata":{"id":"v3nuuC_akDsC"}},{"cell_type":"code","source":"train_dataset.describe(include = \"all\")","metadata":{"id":"7E_ySm2LcGQS","outputId":"ed72c169-ae90-4a6b-8d8a-73591581ef30","execution":{"iopub.status.busy":"2022-08-05T14:46:15.901040Z","iopub.execute_input":"2022-08-05T14:46:15.901370Z","iopub.status.idle":"2022-08-05T14:46:15.981019Z","shell.execute_reply.started":"2022-08-05T14:46:15.901341Z","shell.execute_reply":"2022-08-05T14:46:15.979875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The former results indicate that no PassangerId is duplicated, despite the few cases of repeated names. There are 3 HomePlanet labels and 3 Destinations. ","metadata":{"id":"thryzlfv4NXW"}},{"cell_type":"markdown","source":"### Missing Values:","metadata":{"id":"U7xgugcEkGvo"}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport missingno as msnum \n\nmsnum.matrix(train_dataset)\nplt.show()","metadata":{"id":"wJ6ZCPCOKse5","outputId":"4de7a405-4140-4e9c-cd1c-e98505c203f4","execution":{"iopub.status.busy":"2022-08-05T14:46:15.982874Z","iopub.execute_input":"2022-08-05T14:46:15.983209Z","iopub.status.idle":"2022-08-05T14:46:17.309615Z","shell.execute_reply.started":"2022-08-05T14:46:15.983179Z","shell.execute_reply":"2022-08-05T14:46:17.308422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp_df = pd.DataFrame()\nfor var in train_dataset.columns:\n  temp_df = pd.concat([temp_df, pd.DataFrame(train_dataset[var].isnull().value_counts())], axis = 1)\n\ntemp_df = temp_df.T.fillna(0)\ntemp_df.columns = ['data', 'missing']\ntemp_df['missing_percent'] = (temp_df['missing'] / len(train_dataset)) * 100\ntemp_df","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:46:17.312001Z","iopub.execute_input":"2022-08-05T14:46:17.312348Z","iopub.status.idle":"2022-08-05T14:46:17.355457Z","shell.execute_reply.started":"2022-08-05T14:46:17.312317Z","shell.execute_reply":"2022-08-05T14:46:17.354566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"About 2.06% to 2.5% of data is missing for every variable.","metadata":{}},{"cell_type":"code","source":"round(1 - len(train_dataset.dropna()) / len(train_dataset), 5)","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:46:17.356906Z","iopub.execute_input":"2022-08-05T14:46:17.357269Z","iopub.status.idle":"2022-08-05T14:46:17.375123Z","shell.execute_reply.started":"2022-08-05T14:46:17.357236Z","shell.execute_reply":"2022-08-05T14:46:17.373636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If rows with missing data are removed, we'll lose about 24% of data.","metadata":{}},{"cell_type":"markdown","source":"### Transported","metadata":{}},{"cell_type":"code","source":"train_dataset[\"Transported\"].value_counts(dropna = False)","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:46:17.378200Z","iopub.execute_input":"2022-08-05T14:46:17.378567Z","iopub.status.idle":"2022-08-05T14:46:17.387843Z","shell.execute_reply.started":"2022-08-05T14:46:17.378537Z","shell.execute_reply":"2022-08-05T14:46:17.387003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The dataset is almost divided 50-50 between people transported and rejected to be transported.","metadata":{}},{"cell_type":"markdown","source":"### Cuantitative Variables Exploration:","metadata":{"id":"gF8E5ZN7kJaE"}},{"cell_type":"code","source":"import seaborn as sns\n\ncuantitative_variables = [\"Age\", \"RoomService\", \"FoodCourt\", \"ShoppingMall\", \n                         \"Spa\", \"VRDeck\", \"Transported\"]\n                         \nsns.pairplot(train_dataset[cuantitative_variables], \n             hue = \"Transported\" )\n\nplt.show()","metadata":{"id":"icNVCHEmEAWp","outputId":"259207dc-c743-44ac-ef9f-eee3c7acc6c9","execution":{"iopub.status.busy":"2022-08-05T14:48:19.137585Z","iopub.execute_input":"2022-08-05T14:48:19.139180Z","iopub.status.idle":"2022-08-05T14:48:41.471605Z","shell.execute_reply.started":"2022-08-05T14:48:19.139131Z","shell.execute_reply":"2022-08-05T14:48:41.470659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The previous pairplot indicates that a few variables can be explicative about the Transported status. For instance, some feature engineering applied to FoodCourt vs VRDeck and vs Spa, ShoppingMall vs VRDeck and vs Spa, as well as RoomService vs VRDeck and vs Spa can give relevant information for the desired prediction.","metadata":{"id":"NN1A3P9VXogc"}},{"cell_type":"code","source":"plt.subplots(figsize=(8, 6))\nsns.heatmap(train_dataset.corr(), annot = True, cmap = 'coolwarm')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:46:39.532632Z","iopub.execute_input":"2022-08-05T14:46:39.534771Z","iopub.status.idle":"2022-08-05T14:46:40.042380Z","shell.execute_reply.started":"2022-08-05T14:46:39.534658Z","shell.execute_reply":"2022-08-05T14:46:40.039595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"None of the variables seems to be strongly correlated with **Transported**. For instance, **FoodCourt**, the highest one, is about 0.047, while the lowest one is **RoomService** around -0.24.","metadata":{}},{"cell_type":"markdown","source":"### Cualitative Variable Exploration:","metadata":{}},{"cell_type":"code","source":"import numpy as np\n\n\ncualitative_variables = ['HomePlanet', 'CryoSleep', 'VIP', 'Destination', 'Transported']\n\ndef cualitative_analysis(cuali_var):\n\n  data = train_dataset.copy()\n\n  if cuali_var == \"CryoSleep\":\n    data[\"CryoSleep\"] = data[\"CryoSleep\"].astype(str)\n  elif cuali_var == \"VIP\":\n    data[\"VIP\"] = data[\"VIP\"].astype(str)\n\n  fig, axes = plt.subplots(1, 6, sharey=True, figsize=(30, 4))\n  axes_x = 0\n  for var in cuantitative_variables[:-1]:\n    sns.violinplot(data = data,\n                   y = cuali_var,\n                   x = np.log(data[var] + 1),\n                   hue = \"Transported\",\n                   ax = axes[axes_x],\n                   split=True, inner=\"quart\", linewidth=1)\n    axes_x += 1\n  plt.show()  ","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:46:40.044979Z","iopub.execute_input":"2022-08-05T14:46:40.047602Z","iopub.status.idle":"2022-08-05T14:46:40.058996Z","shell.execute_reply.started":"2022-08-05T14:46:40.047468Z","shell.execute_reply":"2022-08-05T14:46:40.057248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for cuali_var in cualitative_variables[:-1]:\n  cualitative_analysis(cuali_var)","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:48:59.913071Z","iopub.execute_input":"2022-08-05T14:48:59.913471Z","iopub.status.idle":"2022-08-05T14:49:05.131906Z","shell.execute_reply.started":"2022-08-05T14:48:59.913439Z","shell.execute_reply":"2022-08-05T14:49:05.130753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Variable Analysis:","metadata":{"id":"g_dviW8TkZmx"}},{"cell_type":"markdown","source":"#### FoodCourt:","metadata":{"id":"HPZjWpoUkiCD"}},{"cell_type":"code","source":"def kde_plot_compare(var1, var2, var3, xlim1, ylim1, xlim2, ylim2):\n  fig, axes = plt.subplots(1, 2, sharex=False, figsize=(12,6))\n  title = 'KDEplot: ' + var1 + ' vs ' + var2 + ' and vs ' + var3\n  fig.suptitle(title)\n  sns.kdeplot(data = train_dataset, \n              x = var1, \n              y = var2, \n              hue = \"Transported\", \n              ax = axes[0]).set(ylim=(0, ylim1), xlim=(0, xlim1))\n\n  sns.kdeplot(data = train_dataset, \n              x = var1, \n              y = var3, \n              hue = \"Transported\",\n              ax = axes[1]).set(ylim=(0, ylim2), xlim=(0, xlim2))\n\n  plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:49:31.634487Z","iopub.execute_input":"2022-08-05T14:49:31.634924Z","iopub.status.idle":"2022-08-05T14:49:31.643624Z","shell.execute_reply.started":"2022-08-05T14:49:31.634892Z","shell.execute_reply":"2022-08-05T14:49:31.642710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def expending_compare(var1, var2):\n  fig, axes = plt.subplots(2, 2, sharex=False, figsize=(12,10))\n  title = var1 + ' and ' + var2 + ' by Transported (yes/no)'\n  fig.suptitle(title)\n\n  g = sns.scatterplot(data = train_dataset, \n                  x = var1,\n                  y = var2, \n                  hue = \"Transported\",\n                  ax = axes[0,0])\n\n  x0, x1 = g.get_xlim()\n  y0, y1 = g.get_ylim()\n  lims = [max(x0, y0), min(x1, y1)]\n  g.plot(lims, lims, \"-r\")\n\n  temp_df = train_dataset[[\"Transported\", var1, var2]].copy()\n  dummy_name = 'expenses_' + var1 + '>' + var2\n  temp_df[dummy_name] = np.where(temp_df[var1] > temp_df[var2], True, False)\n\n  sns.heatmap(pd.crosstab(temp_df[dummy_name], \n                          temp_df['Transported']), \n              annot = True, \n              fmt = \".0f\",\n              cmap = \"coolwarm\",\n              ax=axes[0,1])\n\n  sns.histplot(data=train_dataset, \n               x = np.log(train_dataset[var1] + 1), \n               hue = \"Transported\", \n               bins = 20, multiple=\"stack\",\n               ax=axes[1,0])\n\n  sns.histplot(data=train_dataset, \n               x = np.log(train_dataset[var2] + 1), \n               hue = \"Transported\", \n               bins = 20, multiple=\"stack\",\n               ax=axes[1,1])\n\n  plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:49:31.656622Z","iopub.execute_input":"2022-08-05T14:49:31.657367Z","iopub.status.idle":"2022-08-05T14:49:31.670979Z","shell.execute_reply.started":"2022-08-05T14:49:31.657333Z","shell.execute_reply":"2022-08-05T14:49:31.669817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kde_plot_compare('FoodCourt', 'VRDeck', 'Spa', 10000, 5000, 10000, 5000)","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:49:31.680662Z","iopub.execute_input":"2022-08-05T14:49:31.681435Z","iopub.status.idle":"2022-08-05T14:49:47.186527Z","shell.execute_reply.started":"2022-08-05T14:49:31.681398Z","shell.execute_reply":"2022-08-05T14:49:47.184764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This KDE plot indicates that for extreme cases, people spending more on VRDeck and Spa than in FoodCourt are less likely to be transported.","metadata":{"id":"DY2ZZ_YzyLAC"}},{"cell_type":"code","source":"expending_compare('FoodCourt', 'VRDeck')","metadata":{"id":"jyh5Q2XZvYIO","outputId":"1ab4ec27-31ce-430e-82bb-74efee2379bf","execution":{"iopub.status.busy":"2022-08-05T14:49:47.189066Z","iopub.execute_input":"2022-08-05T14:49:47.189571Z","iopub.status.idle":"2022-08-05T14:49:48.770842Z","shell.execute_reply.started":"2022-08-05T14:49:47.189525Z","shell.execute_reply":"2022-08-05T14:49:48.769495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Comparing the expenses in FoodCourt vs VRDeck it is not that clear that spending more in FC than in VR will increase the probability to be transported (905 + 3116 TT and FF cases vs 3473 + 1199 TF or FT cases). Furthermore, according to the observed distribution of FC and VR they look very close.\n","metadata":{"id":"2IlNRTqpYHfE"}},{"cell_type":"code","source":"expending_compare('FoodCourt', 'Spa')","metadata":{"id":"woOMDL_C0U5s","outputId":"acbbe61a-3930-4f59-b3bb-19ba98f3bdb9","execution":{"iopub.status.busy":"2022-08-05T14:49:48.772316Z","iopub.execute_input":"2022-08-05T14:49:48.772682Z","iopub.status.idle":"2022-08-05T14:49:50.257425Z","shell.execute_reply.started":"2022-08-05T14:49:48.772652Z","shell.execute_reply":"2022-08-05T14:49:50.256268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Likewise the previous case, it is not clear that spending more on FoodCourt than in Spa improves the chances to be transported.","metadata":{"id":"AlCSFHdsfsFv"}},{"cell_type":"markdown","source":"#### ShoppingMall:","metadata":{"id":"vKwkIc4xk-CK"}},{"cell_type":"code","source":"kde_plot_compare('ShoppingMall', 'VRDeck', 'Spa', 3000, 5000, 3000, 5000)","metadata":{"id":"9CHwQm2i10Np","outputId":"5a4ba69e-45e3-485f-e96d-b6bc58f8aeed","execution":{"iopub.status.busy":"2022-08-05T14:49:50.259802Z","iopub.execute_input":"2022-08-05T14:49:50.261038Z","iopub.status.idle":"2022-08-05T14:50:04.092999Z","shell.execute_reply.started":"2022-08-05T14:49:50.260990Z","shell.execute_reply":"2022-08-05T14:50:04.091483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Likewise the previous finding, this KDE Plots suggests that spending more on VRDeck or Spa than in Shopping Mall will indicate that the person won't be transported.","metadata":{"id":"2ddT4CYwypPf"}},{"cell_type":"code","source":"expending_compare('ShoppingMall', 'VRDeck')","metadata":{"id":"b9sNp25O14zP","outputId":"47be7e8f-ca00-4b37-8c03-51d595a1c768","execution":{"iopub.status.busy":"2022-08-05T14:50:04.094592Z","iopub.execute_input":"2022-08-05T14:50:04.094983Z","iopub.status.idle":"2022-08-05T14:50:05.576318Z","shell.execute_reply.started":"2022-08-05T14:50:04.094948Z","shell.execute_reply":"2022-08-05T14:50:05.575186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this case the assumption that spending more on ShoppingMall than in VRDeck is not confirmed. For instance, results indicate the opposite.","metadata":{"id":"aIVqHpBDlDXO"}},{"cell_type":"code","source":"expending_compare('ShoppingMall', 'Spa')","metadata":{"id":"kgS-h8q61ycc","outputId":"1296a2de-72e2-445b-b187-6e5e4e532e90","execution":{"iopub.status.busy":"2022-08-05T14:50:05.577493Z","iopub.execute_input":"2022-08-05T14:50:05.577827Z","iopub.status.idle":"2022-08-05T14:50:07.084246Z","shell.execute_reply.started":"2022-08-05T14:50:05.577797Z","shell.execute_reply":"2022-08-05T14:50:07.083139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### RoomService:","metadata":{"id":"CKVBoZdAmd7W"}},{"cell_type":"code","source":"kde_plot_compare('RoomService', 'VRDeck', 'Spa', 2000, 2000, 2000, 2000)","metadata":{"id":"v_khvz5kgb6K","outputId":"70ee9ea3-2436-4068-8440-1f264025f730","execution":{"iopub.status.busy":"2022-08-05T14:50:07.086021Z","iopub.execute_input":"2022-08-05T14:50:07.086372Z","iopub.status.idle":"2022-08-05T14:50:22.356789Z","shell.execute_reply.started":"2022-08-05T14:50:07.086342Z","shell.execute_reply":"2022-08-05T14:50:22.355426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The former KDE plot suggests that the less they spend on VRDeck or RoomService, the more chances they'll have to be transported.","metadata":{"id":"9qpu9ffrzqz6"}},{"cell_type":"code","source":"expending_compare('RoomService', 'VRDeck')","metadata":{"id":"vmTFlp_gaWzg","outputId":"92c2a753-9b5b-42af-8465-d3bc55f1b914","execution":{"iopub.status.busy":"2022-08-05T14:50:22.358328Z","iopub.execute_input":"2022-08-05T14:50:22.358665Z","iopub.status.idle":"2022-08-05T14:50:23.869242Z","shell.execute_reply.started":"2022-08-05T14:50:22.358634Z","shell.execute_reply":"2022-08-05T14:50:23.867769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this case we found that actually people expending less in RoomService than in VRDeck were more likely to be transported.","metadata":{"id":"MhWZk029mjMc"}},{"cell_type":"code","source":"expending_compare('RoomService', 'Spa')","metadata":{"id":"5YZjg4Xa2_mY","outputId":"67cfe2d9-89fa-4a26-e513-8a778fb5d826","execution":{"iopub.status.busy":"2022-08-05T14:50:23.870890Z","iopub.execute_input":"2022-08-05T14:50:23.871615Z","iopub.status.idle":"2022-08-05T14:50:25.332428Z","shell.execute_reply.started":"2022-08-05T14:50:23.871573Z","shell.execute_reply":"2022-08-05T14:50:25.330991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Likewise the previous finding, spending more on Spa can be linked to the probability of being transported.","metadata":{"id":"YFa6VJzq3JfT"}},{"cell_type":"code","source":"kde_plot_compare('RoomService', 'FoodCourt', 'ShoppingMall', 5000, 10000, 5000, 3000)","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:50:25.336724Z","iopub.execute_input":"2022-08-05T14:50:25.337791Z","iopub.status.idle":"2022-08-05T14:50:39.814033Z","shell.execute_reply.started":"2022-08-05T14:50:25.337745Z","shell.execute_reply":"2022-08-05T14:50:39.812792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These plots indicate that spending less in RoomService than in FoodCourt or in ShopingMall might have a positive incidence about being transported.","metadata":{}},{"cell_type":"code","source":"expending_compare('RoomService', 'FoodCourt')","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:50:39.815824Z","iopub.execute_input":"2022-08-05T14:50:39.816171Z","iopub.status.idle":"2022-08-05T14:50:41.244983Z","shell.execute_reply.started":"2022-08-05T14:50:39.816140Z","shell.execute_reply":"2022-08-05T14:50:41.243771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"expending_compare('RoomService', 'ShoppingMall')","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:50:41.246580Z","iopub.execute_input":"2022-08-05T14:50:41.247810Z","iopub.status.idle":"2022-08-05T14:50:42.662282Z","shell.execute_reply.started":"2022-08-05T14:50:41.247769Z","shell.execute_reply":"2022-08-05T14:50:42.661449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The TT values are still too low to confirm what the kdeplots suggests in all cases. Further information should be taken from other variables and some feature engineering should be helpful to solve this.","metadata":{}},{"cell_type":"markdown","source":"#### HomePlanet:","metadata":{"id":"qR1wbkEam3XH"}},{"cell_type":"code","source":"def probabilities_by_category(var1):\n  fig, axes = plt.subplots(1, 2, sharex=False, sharey=True, figsize=(12,3))\n\n  sns.heatmap(pd.crosstab(train_dataset[var1], \n                          train_dataset['Transported']), \n              annot = True, \n              fmt = \".0f\",\n              cmap = \"coolwarm\",\n              ax=axes[0])\n\n  temp_df = pd.crosstab(train_dataset[var1], train_dataset['Transported']) \n  temp_df[\"sum\"] = temp_df.sum(axis = 1)\n  temp_df[\"False_prob\"] = temp_df[False] / temp_df[\"sum\"]\n  temp_df[\"True_prob\"] = temp_df[True] / temp_df[\"sum\"]\n  sns.heatmap(temp_df[[\"False_prob\", \"True_prob\"]],\n              annot = True,\n              cmap = \"coolwarm\",\n              ax = axes[1])\n\n  plt.show()  ","metadata":{"id":"zDMyAvLb3kF1","execution":{"iopub.status.busy":"2022-08-05T14:50:42.663427Z","iopub.execute_input":"2022-08-05T14:50:42.664541Z","iopub.status.idle":"2022-08-05T14:50:42.672327Z","shell.execute_reply.started":"2022-08-05T14:50:42.664504Z","shell.execute_reply":"2022-08-05T14:50:42.671470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cualitative_analysis(\"HomePlanet\")","metadata":{"id":"ZVo__JIf31AP","outputId":"9809cc06-4418-4233-9bc4-6ea103fc3e05","execution":{"iopub.status.busy":"2022-08-05T14:50:42.674150Z","iopub.execute_input":"2022-08-05T14:50:42.674674Z","iopub.status.idle":"2022-08-05T14:50:44.255706Z","shell.execute_reply.started":"2022-08-05T14:50:42.674631Z","shell.execute_reply":"2022-08-05T14:50:44.254595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Segregating data by HomePlanet, it is noticeable that people spending less on items like RoomService, Spa and VRDeck are more likely to be transported. On the other hand, some variables don't show a strong difference between the two groups (transported or not), like: **Age**, **FoodCourt** and **ShoppingMall**.","metadata":{"id":"oDfXqAoS6LKb"}},{"cell_type":"code","source":"probabilities_by_category('HomePlanet')","metadata":{"id":"oTVGSE5S4U5S","outputId":"137f1059-46ea-4ba5-8cae-3e1ee80b99c1","execution":{"iopub.status.busy":"2022-08-05T14:50:44.257324Z","iopub.execute_input":"2022-08-05T14:50:44.257654Z","iopub.status.idle":"2022-08-05T14:50:44.764588Z","shell.execute_reply.started":"2022-08-05T14:50:44.257623Z","shell.execute_reply":"2022-08-05T14:50:44.763464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The only noticeable difference here is that people from **Europa** have a little more chances to be transported than people from other Planets. ","metadata":{"id":"vbb2Zdcrhz9q"}},{"cell_type":"markdown","source":"#### CryoSleep:","metadata":{"id":"QUOI0h0GnI4s"}},{"cell_type":"code","source":"cualitative_analysis(\"CryoSleep\")","metadata":{"id":"tuv4cny14g3e","outputId":"459f013e-109e-439a-f7d8-7a8b047010a0","execution":{"iopub.status.busy":"2022-08-05T14:50:44.765770Z","iopub.execute_input":"2022-08-05T14:50:44.766099Z","iopub.status.idle":"2022-08-05T14:50:45.985058Z","shell.execute_reply.started":"2022-08-05T14:50:44.766070Z","shell.execute_reply":"2022-08-05T14:50:45.983952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Not much new information is obtained after segregating data by CryoSleep.","metadata":{"id":"Us17N-zz922V"}},{"cell_type":"code","source":"train_dataset['CryoSleep'].value_counts(dropna=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:50:45.986380Z","iopub.execute_input":"2022-08-05T14:50:45.986727Z","iopub.status.idle":"2022-08-05T14:50:45.996628Z","shell.execute_reply.started":"2022-08-05T14:50:45.986682Z","shell.execute_reply":"2022-08-05T14:50:45.995394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for cualitative_var in cuantitative_variables[1:-1]:\n  print(train_dataset[train_dataset[\"CryoSleep\"] == True][cualitative_var].value_counts())","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:50:45.998820Z","iopub.execute_input":"2022-08-05T14:50:45.999558Z","iopub.status.idle":"2022-08-05T14:50:46.027296Z","shell.execute_reply.started":"2022-08-05T14:50:45.999512Z","shell.execute_reply":"2022-08-05T14:50:46.026296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"According to this finding, those people traveling in CryoSleep are not spending in any of the amenities.","metadata":{}},{"cell_type":"code","source":"train_dataset[train_dataset[\"CryoSleep\"] == False]['RoomService'].value_counts()[:10].plot(kind='bar')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:50:46.028671Z","iopub.execute_input":"2022-08-05T14:50:46.029668Z","iopub.status.idle":"2022-08-05T14:50:46.279157Z","shell.execute_reply.started":"2022-08-05T14:50:46.029627Z","shell.execute_reply":"2022-08-05T14:50:46.277999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Nevertheless, there are (2464) people rejecting to travel on CryoSleep and deciding not to spend a quarter in any of the amenities.","metadata":{}},{"cell_type":"code","source":"probabilities_by_category('CryoSleep')","metadata":{"id":"9jjRIyT34sXF","outputId":"bb201860-fa68-40c1-c4cc-23a6cdf995e5","execution":{"iopub.status.busy":"2022-08-05T14:50:46.283074Z","iopub.execute_input":"2022-08-05T14:50:46.283453Z","iopub.status.idle":"2022-08-05T14:50:46.760942Z","shell.execute_reply.started":"2022-08-05T14:50:46.283419Z","shell.execute_reply":"2022-08-05T14:50:46.759613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**CryoSleep** seems to explain a lot of the chances of being transported according to the previous heatmaps. About 80% of people asking for CryoSleep are actually transported, while 67% of people who rejects this is not being transported.","metadata":{"id":"QFMFmWnsg5x4"}},{"cell_type":"markdown","source":"#### Destination:","metadata":{"id":"U7b2eWIgn6n7"}},{"cell_type":"code","source":"cualitative_analysis(\"Destination\")","metadata":{"id":"fjShPkaZ4_ZY","outputId":"c06f9168-bca5-4730-a4fa-0ed7adec9192","execution":{"iopub.status.busy":"2022-08-05T14:50:46.762896Z","iopub.execute_input":"2022-08-05T14:50:46.764107Z","iopub.status.idle":"2022-08-05T14:50:48.160804Z","shell.execute_reply.started":"2022-08-05T14:50:46.764049Z","shell.execute_reply":"2022-08-05T14:50:48.159571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Not much new information is obtained after segregating data by Destination.","metadata":{"id":"m8Vtto1xjgf0"}},{"cell_type":"code","source":"probabilities_by_category('Destination')","metadata":{"id":"HtxxqjP25Hu8","outputId":"c0fb02a2-27e4-44a0-89a6-47cb86064629","execution":{"iopub.status.busy":"2022-08-05T14:50:48.162805Z","iopub.execute_input":"2022-08-05T14:50:48.163168Z","iopub.status.idle":"2022-08-05T14:50:48.673920Z","shell.execute_reply.started":"2022-08-05T14:50:48.163135Z","shell.execute_reply":"2022-08-05T14:50:48.672914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"People who choose **55 Cancri e** as destination, are more likely to be transported, while the **TRAPPIST-1e** destination is the one which gives less chances to be transported.","metadata":{"id":"duvsU0YgiZGa"}},{"cell_type":"markdown","source":"### VIP:","metadata":{}},{"cell_type":"code","source":"cualitative_analysis(\"VIP\")","metadata":{"id":"SUnLNxAB5QOx","outputId":"d425716c-a2fc-4df5-b8a4-9417910e36bd","execution":{"iopub.status.busy":"2022-08-05T14:50:48.675532Z","iopub.execute_input":"2022-08-05T14:50:48.675936Z","iopub.status.idle":"2022-08-05T14:50:50.348157Z","shell.execute_reply.started":"2022-08-05T14:50:48.675901Z","shell.execute_reply":"2022-08-05T14:50:50.346606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataset['VIP'].value_counts(dropna=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:50:50.350181Z","iopub.execute_input":"2022-08-05T14:50:50.350725Z","iopub.status.idle":"2022-08-05T14:50:50.364623Z","shell.execute_reply.started":"2022-08-05T14:50:50.350657Z","shell.execute_reply":"2022-08-05T14:50:50.362766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is noticeable that the distribution of FoodCourt is different between transported and not transported when differentiating by VIP. Besides that finding not much new information is obtained.","metadata":{"id":"t40yVvtYjolW"}},{"cell_type":"code","source":"probabilities_by_category('VIP')","metadata":{"id":"SvEuWyeP54ww","outputId":"fb4058b1-eb2f-43b1-f160-61b83c5047ac","execution":{"iopub.status.busy":"2022-08-05T14:50:50.366376Z","iopub.execute_input":"2022-08-05T14:50:50.366907Z","iopub.status.idle":"2022-08-05T14:50:50.839881Z","shell.execute_reply.started":"2022-08-05T14:50:50.366851Z","shell.execute_reply":"2022-08-05T14:50:50.838565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Choosing to be **VIP** doesn't make a big difference about being transported or not.","metadata":{"id":"WSE-ic_DjDH4"}},{"cell_type":"markdown","source":"## Test Dataset EDA:","metadata":{"id":"O0LTBoRHoYs6"}},{"cell_type":"code","source":"test_dataset = pd.read_csv('../input/spaceship-titanic/test.csv')\ntest_dataset.head()","metadata":{"id":"tV2hcIS77Qgl","outputId":"01202f0b-03c0-4829-ca61-0e21a70eaacf","execution":{"iopub.status.busy":"2022-08-05T14:50:50.841503Z","iopub.execute_input":"2022-08-05T14:50:50.841896Z","iopub.status.idle":"2022-08-05T14:50:50.888672Z","shell.execute_reply.started":"2022-08-05T14:50:50.841862Z","shell.execute_reply":"2022-08-05T14:50:50.887747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Series(train_dataset.columns[:-1] == test_dataset.columns).value_counts()","metadata":{"id":"QOauY6mD8u38","outputId":"9f32d5b6-1a50-495a-b1d1-d9ae5e73ca54","execution":{"iopub.status.busy":"2022-08-05T14:50:50.890007Z","iopub.execute_input":"2022-08-05T14:50:50.890565Z","iopub.status.idle":"2022-08-05T14:50:50.899493Z","shell.execute_reply.started":"2022-08-05T14:50:50.890532Z","shell.execute_reply":"2022-08-05T14:50:50.898252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"test_dataset has the same columns as train_dataset except for the target \"Transported\"","metadata":{"id":"PTZL9XVE7Iz7"}},{"cell_type":"markdown","source":"### Basic Statistics:","metadata":{"id":"dHhsUYR7ooKo"}},{"cell_type":"code","source":"test_dataset.describe(include=\"all\")","metadata":{"id":"waa7IWrQoqP3","outputId":"53a127ba-ea17-4866-defa-885fc30a377f","execution":{"iopub.status.busy":"2022-08-05T14:50:50.906821Z","iopub.execute_input":"2022-08-05T14:50:50.907519Z","iopub.status.idle":"2022-08-05T14:50:50.967779Z","shell.execute_reply.started":"2022-08-05T14:50:50.907480Z","shell.execute_reply":"2022-08-05T14:50:50.966566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this case we have 4277 unique passangers, and the same number of HomePlanets and Destinations.","metadata":{"id":"XOkU5PHiovee"}},{"cell_type":"code","source":"train_ID = set(train_dataset['PassengerId'])\ntest_ID = set(test_dataset['PassengerId'])\nprint(len(train_ID) + len(test_ID), len(train_ID.union(test_ID)))","metadata":{"id":"m7yTnCe9GjyY","outputId":"2eb5f783-a48d-4912-bce0-fa73e3164981","execution":{"iopub.status.busy":"2022-08-05T14:50:50.969135Z","iopub.execute_input":"2022-08-05T14:50:50.969494Z","iopub.status.idle":"2022-08-05T14:50:50.980802Z","shell.execute_reply.started":"2022-08-05T14:50:50.969460Z","shell.execute_reply":"2022-08-05T14:50:50.979611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With the previous step it was identified that no passangers are repeated in both datasets.","metadata":{"id":"G2Wqwm0JHhsf"}},{"cell_type":"markdown","source":"### Missing Values:","metadata":{"id":"BnIG9zgDolX3"}},{"cell_type":"code","source":"msnum.matrix(test_dataset)\nplt.show()","metadata":{"id":"3h5Vk8lTMBm_","outputId":"34bcb4e6-36ab-42ce-cd6b-218bdf327a59","execution":{"iopub.status.busy":"2022-08-05T14:50:50.982466Z","iopub.execute_input":"2022-08-05T14:50:50.983137Z","iopub.status.idle":"2022-08-05T14:50:51.550883Z","shell.execute_reply.started":"2022-08-05T14:50:50.983101Z","shell.execute_reply":"2022-08-05T14:50:51.549993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp_df = pd.DataFrame()\nfor var in test_dataset.columns:\n  temp_df = pd.concat([temp_df, pd.DataFrame(test_dataset[var].isnull().value_counts())], axis = 1)\n\ntemp_df = temp_df.T.fillna(0)\ntemp_df.columns = ['data', 'missing']\ntemp_df['percent_missing'] = (temp_df['missing'] / len(train_dataset)) * 100\ntemp_df","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:50:51.552421Z","iopub.execute_input":"2022-08-05T14:50:51.552777Z","iopub.status.idle":"2022-08-05T14:50:51.593252Z","shell.execute_reply.started":"2022-08-05T14:50:51.552744Z","shell.execute_reply":"2022-08-05T14:50:51.592371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"round(1 - len(test_dataset.dropna()) / len(test_dataset), 5)","metadata":{"execution":{"iopub.status.busy":"2022-08-05T14:50:51.594537Z","iopub.execute_input":"2022-08-05T14:50:51.595126Z","iopub.status.idle":"2022-08-05T14:50:51.608111Z","shell.execute_reply.started":"2022-08-05T14:50:51.595091Z","shell.execute_reply":"2022-08-05T14:50:51.607054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Are train and test datasets balanced?","metadata":{"id":"Q4laHZKfpcfR"}},{"cell_type":"markdown","source":"#### FoodCourt:","metadata":{"id":"Vhdcd2s9qhsv"}},{"cell_type":"code","source":"from scipy.stats import ttest_ind\n\ndef density_comparison(var1):\n  stat, p = ttest_ind(test_dataset[var1].dropna(), train_dataset[var1].dropna())\n  print(\"p-value for identical distribution:\", p)\n  sns.kdeplot(np.log(test_dataset[var1] + 1), shade=True, color=\"r\")\n  sns.kdeplot(np.log(train_dataset[var1] + 1), shade=True, color=\"b\")\n  plt.show()","metadata":{"id":"D_H3uuKcVWBV","execution":{"iopub.status.busy":"2022-08-05T14:50:51.609799Z","iopub.execute_input":"2022-08-05T14:50:51.610430Z","iopub.status.idle":"2022-08-05T14:50:51.616977Z","shell.execute_reply.started":"2022-08-05T14:50:51.610397Z","shell.execute_reply":"2022-08-05T14:50:51.615865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"density_comparison('FoodCourt')","metadata":{"id":"2mvkIhKyVkWJ","outputId":"d0eb3ce2-e990-4781-8abf-3aee1c0ebb28","execution":{"iopub.status.busy":"2022-08-05T14:50:51.618439Z","iopub.execute_input":"2022-08-05T14:50:51.618885Z","iopub.status.idle":"2022-08-05T14:50:51.903750Z","shell.execute_reply.started":"2022-08-05T14:50:51.618853Z","shell.execute_reply":"2022-08-05T14:50:51.902538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### ShoppingMall:","metadata":{"id":"oI1z6-C-ql5x"}},{"cell_type":"code","source":"density_comparison('ShoppingMall')","metadata":{"id":"j02SnOYfW27l","outputId":"bb5322a8-c133-4853-b591-2bf7563fc8fa","execution":{"iopub.status.busy":"2022-08-05T14:50:51.905535Z","iopub.execute_input":"2022-08-05T14:50:51.905930Z","iopub.status.idle":"2022-08-05T14:50:52.160812Z","shell.execute_reply.started":"2022-08-05T14:50:51.905880Z","shell.execute_reply":"2022-08-05T14:50:52.159536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### RoomService:","metadata":{"id":"yhTb2mxoqoy1"}},{"cell_type":"code","source":"density_comparison('RoomService')","metadata":{"id":"sWj6h1MZW_7v","outputId":"5cd6f9ef-2479-4b5e-d5f3-60b8c727ed75","execution":{"iopub.status.busy":"2022-08-05T14:50:52.162150Z","iopub.execute_input":"2022-08-05T14:50:52.162540Z","iopub.status.idle":"2022-08-05T14:50:52.378961Z","shell.execute_reply.started":"2022-08-05T14:50:52.162499Z","shell.execute_reply":"2022-08-05T14:50:52.377723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Age:","metadata":{"id":"QrJjcM_hYfoH"}},{"cell_type":"code","source":"density_comparison('Age')","metadata":{"id":"ud-1vPE5XIY4","outputId":"f40d542e-2b40-4921-ace0-d9b81b85b263","execution":{"iopub.status.busy":"2022-08-05T14:50:52.380667Z","iopub.execute_input":"2022-08-05T14:50:52.381369Z","iopub.status.idle":"2022-08-05T14:50:52.598745Z","shell.execute_reply.started":"2022-08-05T14:50:52.381327Z","shell.execute_reply":"2022-08-05T14:50:52.596103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Spa:","metadata":{"id":"OCkYfL_bYrq1"}},{"cell_type":"code","source":"density_comparison('Spa')","metadata":{"id":"5uC6uft-YtMF","outputId":"e44ec806-5ec1-4b54-8877-e48540c820b9","execution":{"iopub.status.busy":"2022-08-05T14:50:52.600336Z","iopub.execute_input":"2022-08-05T14:50:52.600847Z","iopub.status.idle":"2022-08-05T14:50:52.816907Z","shell.execute_reply.started":"2022-08-05T14:50:52.600790Z","shell.execute_reply":"2022-08-05T14:50:52.815669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### VRDeck:","metadata":{"id":"9AcLuo55Ywg_"}},{"cell_type":"code","source":"density_comparison('VRDeck')","metadata":{"id":"6EWKGhHPYzjY","outputId":"dfa8f20b-0080-47b2-f70b-cd8f4afbb8f1","execution":{"iopub.status.busy":"2022-08-05T14:50:52.818553Z","iopub.execute_input":"2022-08-05T14:50:52.819327Z","iopub.status.idle":"2022-08-05T14:50:53.041256Z","shell.execute_reply.started":"2022-08-05T14:50:52.819279Z","shell.execute_reply":"2022-08-05T14:50:53.038939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### HomePlanet:","metadata":{"id":"tP8QQNcbqrvM"}},{"cell_type":"code","source":"def compare_categories(var1):\n  \n  temp_df = pd.concat([\n     train_dataset[[var1, \"PassengerId\"]].groupby(var1).count(),\n     test_dataset[[var1, \"PassengerId\"]].groupby(var1).count()],\n        axis = 1)\n  \n  temp_df.columns = [\"train_dataset\", \"test_dataset\"]\n  temp_df = temp_df.T\n  temp_df[\"total\"] = temp_df.sum(axis = 1)\n\n  new_labels = []\n  \n  for label in list(train_dataset[var1].dropna().unique()):\n    new_label = str(label) + \"_p\"\n    temp_df[new_label] = temp_df[label] / temp_df[\"total\"]\n    new_labels.append(new_label)\n\n  sns.heatmap(temp_df[new_labels],\n              annot = True,\n              cmap = \"coolwarm\")\n\n  plt.show()","metadata":{"id":"0tGKAU0oY91F","execution":{"iopub.status.busy":"2022-08-05T14:50:53.043105Z","iopub.execute_input":"2022-08-05T14:50:53.043913Z","iopub.status.idle":"2022-08-05T14:50:53.055355Z","shell.execute_reply.started":"2022-08-05T14:50:53.043831Z","shell.execute_reply":"2022-08-05T14:50:53.054472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"compare_categories(\"HomePlanet\")","metadata":{"id":"8RZHxgaZeJF-","outputId":"875a9f2b-7d90-4b5d-eb35-88be5deb5ca5","execution":{"iopub.status.busy":"2022-08-05T14:50:53.056515Z","iopub.execute_input":"2022-08-05T14:50:53.057613Z","iopub.status.idle":"2022-08-05T14:50:53.305537Z","shell.execute_reply.started":"2022-08-05T14:50:53.057577Z","shell.execute_reply":"2022-08-05T14:50:53.304321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### CryoSleep:","metadata":{"id":"13co7l1Dqvd8"}},{"cell_type":"code","source":"test_dataset[\"CryoSleep\"] = test_dataset[\"CryoSleep\"].astype(bool)\ncompare_categories(\"CryoSleep\")","metadata":{"id":"GhnG-OuzepR-","outputId":"6cf1b05c-d184-4e78-970d-60d3909f688c","execution":{"iopub.status.busy":"2022-08-05T14:50:53.306932Z","iopub.execute_input":"2022-08-05T14:50:53.307245Z","iopub.status.idle":"2022-08-05T14:50:53.506783Z","shell.execute_reply.started":"2022-08-05T14:50:53.307216Z","shell.execute_reply":"2022-08-05T14:50:53.505456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Destination:","metadata":{"id":"_a1W4N6QqyoX"}},{"cell_type":"code","source":"compare_categories(\"Destination\")","metadata":{"id":"bh43CaChfBXD","outputId":"4bac9432-1ccd-4aea-be05-021a01ee8070","execution":{"iopub.status.busy":"2022-08-05T14:50:53.508988Z","iopub.execute_input":"2022-08-05T14:50:53.509809Z","iopub.status.idle":"2022-08-05T14:50:53.722879Z","shell.execute_reply.started":"2022-08-05T14:50:53.509768Z","shell.execute_reply":"2022-08-05T14:50:53.721748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### VIP:","metadata":{"id":"UvAzz8yDq1dZ"}},{"cell_type":"code","source":"compare_categories(\"VIP\")","metadata":{"id":"joaTGGaWfLAg","outputId":"6bdd6464-386a-440e-9cf3-0f83f2c427cc","execution":{"iopub.status.busy":"2022-08-05T14:50:53.724382Z","iopub.execute_input":"2022-08-05T14:50:53.725044Z","iopub.status.idle":"2022-08-05T14:50:53.908259Z","shell.execute_reply.started":"2022-08-05T14:50:53.725009Z","shell.execute_reply":"2022-08-05T14:50:53.907273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can conclude that both sets train and test are balanced as their distributions across the variables are quite similar confirmed with the respective statistical tests. ","metadata":{}},{"cell_type":"markdown","source":"## Next Steps:\n\n> [Feature Engineering](https://www.kaggle.com/code/davidzambrano87/ss-titanic-fe-by-dz) \n\n> [Models and Error Analysis.](https://www.kaggle.com/code/davidzambrano87/ss-titanic-modeling-by-dz)","metadata":{"id":"fa-RikDvpk16"}}]}