{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "f9eea58c-6938-7b57-5f4a-9243dcf1309b" }, "source": [ "## Overview \n", "\n", "Now I am in the Top 4% of this task. \n", "My notebook is very simple and understandable, it doesn't consist from difficult formulas, plots and lots of words :)\n", "\n", "I want to improve my decision if you have any advice or comments I will be glad to know about this.\n", "\n", "I slightly changed parameters of functions. You can select the appropriate parameters if you want" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c9eb7bfa-48e0-4639-f322-293fb25ba437" }, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import scipy \n", "import seaborn as sns\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "pd.options.display.max_rows = 999\n", "verbose = False # param for debugging" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "afd782c6-e28c-ff81-09c5-04581cd33579" }, "outputs": [], "source": [ "# Load data\n", "train = pd.read_csv('../input/train.csv', header=0,sep=',')\n", "test = pd.read_csv('../input/test.csv', header=0)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "af64a38f-29c1-9c70-4ed5-6f883c8f89b3" }, "outputs": [], "source": [ "# Let's look at the general information about the training set\n", "train.info()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "92ceace4-2329-029b-0358-8ad81422fb7e" }, "source": [ "## 1 Processing of the dataset\n", "\n", "### 1.1 Data Dictionary\n", "\n", " **Variable**\t**Definition**\t **Key**\n", "\n", " survival\t Survival\t 0 = No, 1 = Yes\n", " pclass\t Ticket class\t 1 = 1st, 2 = 2nd, 3 = 3rd\n", " sex\t Sex\t\n", " Age\t Age in years\t\n", " sibsp\t # of siblings / spouses aboard the Titanic\t\n", " parch # of parents / children aboard the Titanic\t\n", " ticket\t Ticket number\t\n", " fare\t Passenger fare\t\n", " cabin\t Cabin number\t\n", " embarked\t Port of Embarkation\t C = Cherbourg, Q = Queenstown, S = Southampton\n", "\n", "### 1.2 NaN values" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e3c3b62b-2a28-9306-d2f2-1f74b6ae3b46" }, "outputs": [], "source": [ "# Create a function for processing NaN values\n", "\n", "def fillNaN(df=train,drop=True):\n", " if drop:\n", " df.dropna(subset=['Embarked','Fare'],inplace=True,axis=0) # Drop NaN for Embarked and Fare\n", " else:\n", " df['Fare']=df['Fare'].fillna(0.) # Fill NaN as 0 for Fare\n", " df['Embarked']=df['Embarked'].fillna('S') # Fill NaN as S for Embarked\n", " df['Age']=df['Age'].fillna(95.) # Fill NaN as 95 for Age. I chose 95 because max Age is 80\n", " df['Cabin']=df['Cabin'].fillna('-1') # Fill NaN as -1 for Cabin \n", "fillNaN()\n", "fillNaN(test,False)\n", "\n", "# Check empty values\n", "train.info()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a14f95dc-b9c4-5b72-ce43-94ff1cfdbba8" }, "source": [ "### 1.3 Mapping object (string) values to numeric type & creating groups from values\n", "\n", "1. As we saw above, we have 5 object columns. There are **Name, Sex, Ticket, Cabin, Embarked**. For future calculation we should map this values to numbers. Let's try to map string values to numbers for **Sex, Cabin, Embarked**\n", "2. Also we have 2 columns with large range of values. There are **Cabin and Age**. For future calculation we should group this values\n", "3. We have some columns useless for predictions, there are **Name and Ticket**\n", "\n", " **Cabin**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a96dab60-5d8e-c37e-3236-c46e96850796" }, "outputs": [], "source": [ "train.groupby('Cabin').count()\n", "# We will see 'T' as noise" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "50005f08-40b9-ed01-5f95-99abfed601e5" }, "outputs": [], "source": [ "# I want to get code for Cabine, \n", "# I suppose that the first letter is ship's deck ( depends on location and class of passenger)\n", "\n", "data_clear = train.copy() \n", "\n", "# T is noise in the field\n", "data_clear = data_clear[train.Cabin!= 'T'] \n", "\n", "\n", "cabin,cab_map = [],[]\n", "\n", "def modCabin(df):\n", " global cabin,cab_map\n", " df['cab'] = df['Cabin'] # Create a new column\n", " df['cab'] = [ x[0] for x in df.Cabin] # get the first letter from Cabin\n", " if len(cabin) == 0:\n", " cabin = df['cab'].unique() # get unique values\n", " cabin.sort(axis=0) # sort values\n", " print(cabin) # note NA = '-'\n", " cab_map =np.arange(len(cabin)) # List of numbers for mapping \n", " df['cab'].replace(cabin,cab_map,inplace=True) # Map letters to numbers \n", " df['cab'].astype(dtype='int64') # Convert data type to int64" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "18c32339-372a-268e-ec22-bfd2d9cb08a9" }, "outputs": [], "source": [ "modCabin(data_clear)\n", "modCabin(test)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "1f0c6e50-8d06-03dd-772a-e70bfe2716c3" }, "source": [ "**Sex & Embarked**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "99292be2-bcdb-59f2-7fbf-6aa20d52e89c" }, "outputs": [], "source": [ "# Create a function to mapping string to int \n", "def modSexEmb(df):\n", " df['Sex'].replace(['male','female'],[0,1],inplace=True) # Map male = 0, female = 1\n", " df['Sex'].astype(dtype='int64') # Convert data type to int64\n", "\n", " df['Embarked'].replace(['C','Q','S'],[0,1,2],inplace=True) # Map C = 0, Q = 1, S = 2, N = 3\n", " df['Embarked'].astype(dtype='int64') # Convert data type to int64\n", "modSexEmb(data_clear)\n", "modSexEmb(test)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "5e11ce73-32c4-f0b0-5897-ba3b3d1f88c0" }, "source": [ "**Drop useless fields**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "089d3801-2077-81c4-9d9c-074bf5c382be" }, "outputs": [], "source": [ "# Create a function for dropping useless fields \n", "def drop(df, delPas=True):\n", " df.drop('Cabin',axis=1,inplace=True) # Drop 'Cabin' field\n", " if delPas == True:\n", " df.drop('PassengerId',axis=1,inplace=True) # Drop 'PassengerId' field\n", " df.drop('Name',axis=1,inplace=True) # Drop 'Name' field\n", " df.drop('Ticket',axis=1,inplace=True) # Drop 'Ticket' field\n", " df.info() # Check \n", "drop(data_clear)\n", "drop(test,False)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4129dca4-8209-5bef-899a-d3fa7b5da800" }, "source": [ " **Grouping values**\n", "\n", "As we saw above, Age and Fare have a lot of values and they must be grouped" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "89fb2baf-237c-a2f7-02ce-e57984d43530" }, "outputs": [], "source": [ "# Create a function for groupping\n", "def split(col_range, df,col):\n", " n=0\n", " for i in col_range:\n", " \n", " if i == col_range[len(col_range)-1]:\n", " df[col].replace(df[(df[col]>=i)][col],i,inplace=True)\n", " else:\n", " alpha = col_range[n+1]-col_range[n]\n", " df[col].replace( df[(df[col]>=i) & (df[col]