{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd \nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\ntrain = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"37a7a1cd123438af1e8a34e38693e7e68ac99603"},"cell_type":"code","source":"train.head()\n#Survived is the target variable","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bbcf9214deeb10f1920b6efe468f76f0a72c6a73"},"cell_type":"markdown","source":"**Missing Value Analysis and Data Clean Up**\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"#dropping PassengerId as it does not contain information\ntrain.drop('PassengerId', axis=1,inplace=True)\n\n#Missing Value Analysis\ndef missing_check(column):\n    check = column.isna().value_counts()\n    try:\n        count =  check[True]\n    except KeyError:\n        print('{} does not have any missing'.format(column.name))\n    else:\n        percentage_missing = count/column.shape[0]\n        print('{} has {} of its data missing'.format(column.name, str(percentage_missing)))\n\nfor col in train:\n    missing_check(train[col])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f18c718b80d29930e6498f7c8bf11678426ec0b3"},"cell_type":"markdown","source":"You can see that Age, Cabin and Embarked have missing data. Age and Embarked have relatively insignificant number of missing data while Cabin has most of its data missing.\n\nThere are two ways to deal with missing data.\n1. Discard the feature entirely\n2. Fill in the missing data\n    \nDiscarding method is usually employed on features with majority of its data missing (e.g. Cabin in our case), because the resulting features will be uninformative  that are no better than randomly generated ones.\n\nFor now, we will fill the values only for Age as LGBM model is able to handle null values. I have chosen to fill with the value 100 in order make them outliers."},{"metadata":{"trusted":true,"_uuid":"64bb153e78ce2a0d1b6e6a4023a08db7badb4439"},"cell_type":"code","source":"train.Age.fillna(value = 100, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6c740c12aea3972ccd46fafbbff694f4a1d5eb44","scrolled":true},"cell_type":"code","source":"sns.countplot(x='Survived', data = train)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8831eb4d9eb1150c772ee478f9a8ac99f0b62fe3"},"cell_type":"markdown","source":" The data is imbalanced. Even though the imbalance is not significant, we will see that this imbalance will influence our final decision in making predictions. Dealing with imbalanced data is one of the hottest topic in machine learning and data mining at the moment. Using a naive model will run into Accuracy Paradox. There are different approaches to this. Sampling to artificially balance the data, different learning methods, calibrating probability outputs,  etc. Currently sampling method is the most efficient way to tackle this problem, but it is an expedient approach as the sampled data does not represent the original distribution.. In this kernel, I will try t\n\n\n**Analysis of features by graphs**\n\n**2.1 Distribution of Features**\n\nIt is important to get a good idea of each feature's distribution, which will be important for feature extraction and model selection. The very first thing one should do is to make histrogram plots for each feature. For example, linear regression works on the assumption that predictor variables are approximately normally distributed (which can be checked by Q-Q plots). \n\nWe will use count plots for categoricals and histograms for other ordinal or nominal variables"},{"metadata":{"trusted":true,"_uuid":"ac95b674c4e9f26e3b95aa1b3776ece0b7841ffe","scrolled":true},"cell_type":"code","source":"numerical = ['Age', 'Fare']\n\nfig, axs = plt.subplots(1,len(numerical), figsize=(30, 15))\nfor k, col in enumerate(numerical):\n        sns.distplot(train[col], kde=False, ax=axs[k])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ddfdf0d8825fdd65d5c07498055b08779c4b62d8","scrolled":true},"cell_type":"code","source":"fig, axs = plt.subplots(2,3, figsize=(30, 30))\naxs = axs.flatten()\nc = 0\nfor col in train.columns:\n    if col not in (numerical + ['Survived','Name', 'Ticket', 'Cabin']):\n        sns.countplot(x=col, data=train, ax=axs[c])\n        c += 1\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ab5c44d40f70a11f4a1e602a3d75c9bc9427b00b"},"cell_type":"markdown","source":"**2.2 Distribution of the variable with respect to the target variable**\n\nGraphing features with respect to the target variable will show each feature's general predictive power. Large separation between the disbritutions will suggest that it can be a powerful predictive tool whereas small separation will suggest the otherwise. "},{"metadata":{"trusted":true,"_uuid":"f0d5b2b807582d539742eb709c4c24dde6532f6c","scrolled":true},"cell_type":"code","source":"def v_plot(violin_d, train):\n    #standardize for plotting\n    violin_d = (violin_d - violin_d.mean())/ violin_d.std() \n    violin_d = pd.concat([violin_d, train.Survived], axis=1)\n    violin_d = pd.melt(violin_d, id_vars = 'Survived')\n\n    plt.figure(figsize=(10,10))\n    ax = sns.violinplot(x = 'variable', y='value', hue = 'Survived', data=violin_d\n                  ,palette=\"muted\", split=True)\n    plt.show()\n    plt.figure(figsize=(10,10))\n    ax1 = sns.swarmplot(x = 'variable', y='value', hue = 'Survived', data=violin_d\n                  ,palette=\"muted\", split=True) \n\n    plt.figure(figsize=(10,10))\n    ax2 = sns.boxplot(x = 'variable', y='value', hue = 'Survived', data=violin_d )\n    plt.show()\n    \nviolin_d = train[['Age', 'Fare']].copy()\nv_plot(violin_d, train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2647313571fef8775bd6357c5d7d5ffbe70b2b5b"},"cell_type":"markdown","source":"It seems like Age might not a be a very powerful feature, as the two distributions are very similar. For Fare, distributions quite differ. It is evident that people with more expensive tickets were more likely to survive."},{"metadata":{"trusted":true,"_uuid":"3954f64f12b6d197657c624776ab856dd36fe49a"},"cell_type":"code","source":"not_these = ['Name', 'Ticket', 'Cabin']\nfor col in train.columns:\n    if col not in (['Survived','Age', 'Fare'] + not_these ) :\n        sns.catplot(x= col, hue='Survived', kind=\"count\", data=train)\n        plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d07099d9b904b5293b82d0d1e6cba52dedbc3e11"},"cell_type":"markdown","source":"**2.3 Feature Importance Analysis**\n\nIf the given variables were real not nominal, these graphs can also be summarized numerically by calculating each feature's correlation with the target variable. A predictor's Pearson correlation coefficient being close to 0 will suggest that it is indiscriminate. \n\nWith ordinal or nominal variables,, one can make use of the Chi-Squared test of independence. However, it also assumes indepedence. \n\nThe reason why the above graphs, the Pearson's R, or the Chi-Square  only suggest one feature's GENERAL predictive power is because of its independence assumptions which is made by any univariate methods. **Individually irrelevant features maybe become relevant in the context of others. Also, relevant features maybe not be useful due to possible redundancies. **\n\nThus, along with the above univariate methods, we will also make use of a multi-variate method to find the relative importance of each feature. In other words, we will look at the importance of a single feature in the context of others. The method that we will employ is called Random Forrest. It is a greedy method that adds one feature at a time to the subset of features measuring the improvment of a performance index. We will use information gain (entropy) as our performance index."},{"metadata":{"trusted":true,"_uuid":"5b23b37f985e8f847e75aa691ff8f9faf3b1c68a"},"cell_type":"code","source":"from sklearn import model_selection\nfrom sklearn.metrics import roc_curve, auc\nimport lightgbm as lgb\n\ntrain[['Sex', 'Embarked']] = train[['Sex', 'Embarked']].astype('category')\n\nY = train.Survived\nX = train.drop(['Survived', 'Name', 'Cabin', 'Ticket'], axis = 1)\n\n#this choice of parameters is rather arbitrary (standard choice)\n\n\nkfold = model_selection.StratifiedKFold(n_splits = 6, random_state=50)\n\n\ndef rf_v_importance(X, Y):\n    vimp = pd.DataFrame()\n    vimp['feature'] = X.columns\n    AUC_t = []\n    AUC_v = []\n    \n    params = { 'boosting_type': 'rf',\n            'num_leaves' : 50,\n            'colsample_bytree' : 0.6,\n            'n_estimators' : 300,\n            'min_child_weight' : 5,\n            'min_child_samples' : 10,\n            'subsample' : 0.632, \n            'subsample_freq' : 1,\n            'metric' : 'auc'}\n    \n    for i, (t_idx, v_idx) in enumerate(kfold.split(X.values, Y.values)):\n\n        print(\"Cross validation {0:d}-------------\".format(i+1))\n        lgb_t = lgb.Dataset(X.iloc[t_idx], Y.iloc[t_idx])\n        lgb_v = lgb.Dataset(X.iloc[v_idx], Y.iloc[v_idx])\n        t_ = lgb.train(params, lgb_t, valid_sets=lgb_v, verbose_eval=False, early_stopping_rounds = 15)\n\n        fpr, tpr, _ = roc_curve(Y.iloc[t_idx], t_.predict(X.iloc[t_idx]))\n        AUC_t.append(auc(fpr, tpr))\n        print(\"Training AUC score: {0:.3f}\".format(AUC_t[i]))\n\n        fpr, tpr, _ = roc_curve(Y.iloc[v_idx], t_.predict(X.iloc[v_idx]))\n        AUC_v.append(auc(fpr, tpr))\n        print(\"Validation AUC score: {0:.3f}\".format(AUC_v[i]))\n\n        vimp[\"CV_\"+ str(i)] = t_.feature_importance(importance_type='gain') \n\n    print(\"Final RF training AUC score {} | Final RF validation AUC score {}\".format(\n        sum(AUC_t)/len(AUC_t) , sum(AUC_v)/len(AUC_v)))\n\n    vimp['S'] = pd.Series(vimp.iloc[:,1:-1].mean(axis=1).values)\n    plt.figure(figsize = (20,10))\n    sns.barplot(x='S',y='feature',data=vimp)\n    plt.xlabel('Score')\n    plt.ylabel('Feature Names')\n    plt.title(\"Feature Importance by Information Gain\")\n    plt.show()\n\nrf_v_importance(X, Y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6aeb56da793a71f5af5caec1d33ac249a50c2cf1"},"cell_type":"markdown","source":"**Baseline Model Performance**\n\nWe will use a LGBM gradient bossting machine model to set a baseline of our untouched dataset. (Random forrest model's baseline has already been determined above). Parameters used for training was quite arbitrary, mostly default setting give by LightGBM.. "},{"metadata":{"trusted":true,"_uuid":"aa4f672d850148671344ece53fee833a8fb7bd9f","scrolled":true},"cell_type":"code","source":"gbm_cv = lgb.Dataset(X,Y)\n\nparams = {\n        'objective': 'binary',\n        'boosting': 'gbdt',\n        'learning_rate': 0.01,\n        'n_estimators' : 300,\n        'metric' : 'auc'\n    }\n\nprint(\"Cross Validation Score\")\nCV_baseline = lgb.cv(params, train_set=gbm_cv)\nprint(sum(CV_baseline['auc-mean'])/len(CV_baseline['auc-mean']))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e80b706a6b160dbdf1a9a3ed2c5cab27323a96cb"},"cell_type":"markdown","source":"**ROC-AUC as the choice of a performance index**\n\nIt can be noted that ROC-AUC was used to measure the above model's performance rather than accuracy. The following section will illustrate the downside of using accruacy or other static metric to measure a model's performance.\n\n\n"},{"metadata":{"trusted":true,"_uuid":"cff11e0daad1d26761edced7f3390f10a9bcd97d","scrolled":false},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix,accuracy_score, f1_score, recall_score, average_precision_score, precision_score, precision_recall_curve\ndef auc_plot(true, pred):\n    fpr, tpr, threshold = roc_curve(true, pred)\n    roc_auc = auc(fpr, tpr)\n    plt.plot(fpr, tpr, label='area = %0.2f' % roc_auc)\n    plt.plot([0,1], [0,1])\n    plt.title(\"Receiver Operating Characteristic\")\n    plt.ylabel('True Positive Rate')\n    plt.xlabel('False Positive Rate')\n    plt.xlim([0.0, 1.0])\n    plt.ylim([0.0, 1.05])\n    plt.legend(loc=\"lower right\")\n    plt.show()\n\ndef precision_recall_plot(true, pred):\n    precision, recall, threshold = precision_recall_curve(true, pred)\n    AP = average_precision_score(true, pred)\n    plt.plot(recall, precision, label='AP = %0.2f' % AP)\n    plt.title(\"Precision-Recall Curve\")\n    plt.ylabel('Precision')\n    plt.xlabel('Recall')\n    plt.xlim([0.0, 1.0])\n    plt.ylim([0.0, 1.05])\n    plt.legend(loc=\"lower right\")\n    plt.show()\n\nX_train, X_val, Y_train, Y_val,= model_selection.train_test_split(X, Y, test_size = 0.1, random_state = 20, shuffle =True)\ngbm_train = lgb.Dataset(X_train, Y_train)\n\nparams = {\n        'objective': 'binary',\n        'boosting': 'gbdt',\n        'learning_rate': 0.1,\n        'n_estimators' : 100,\n        #'num_leaves': 31,\n        'max_depth': 3,\n        #'min_child_samples': 2,\n    }\n\nmodel = lgb.train(params, gbm_train) #model for illustration\nprediction = model.predict(X_val)\nauc_plot(Y_val, prediction)\nprecision_recall_plot(Y_val, prediction)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"e4ece96031403aa8bf54a54dbf1608a6a2eff19e"},"cell_type":"code","source":"def plot_confusion_matrix(true, pred, classifier, classes):\n    decision = pred.copy()\n    decision[decision <=  classifier] = 0 #make decisions\n    decision[decision != 0] = 1\n    \n    cm = confusion_matrix(true, decision)\n    \n    accuracy = accuracy_score(true, decision)\n    fscore = f1_score(true, decision)\n    R = recall_score(true, decision)\n    P = precision_score(true, decision)\n    print(\"With {} decision threshold, the model's accuracy is {}\".format(str(classifier),str(accuracy)))\n    print(\"With {} decision threshold, the model's f1-score is {}\".format(str(classifier),str(fscore)))\n    print(\"With {} decision threshold, the model's recall is {}\".format(str(classifier),str(R)))\n    print(\"With {} decision threshold, the model's precision is {}\".format(str(classifier),str(P)))\n    plt.imshow(cm)\n    plt.title('Confusion Matrix')\n    for i in range(len(cm)):\n        for j in range(len(cm)):\n            plt.text(i,j, cm[i,j])\n\n    plt.xticks(range(len(classes)), classes)\n    plt.yticks(range(len(classes)), classes)\n    plt.ylabel\n    ('True label')\n    plt.xlabel('Predicted label')\n    plt.show()\n\nclasses = ['Died', 'Survived']\nplot_confusion_matrix(Y_val, prediction, 0.5, classes)\nplot_confusion_matrix(Y_val, prediction, 0.4, classes)\nplot_confusion_matrix(Y_val, prediction, 0.6, classes)\nplot_confusion_matrix(Y_val, prediction, 0.3838, classes) #prior probability: p(Survived = 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"81da029e81ea0ef2e008a6a39d09e3ed4c94a0d2"},"cell_type":"markdown","source":"Using different decision thresholds led to different accuracy scores. Using 0.5 as the decision threshold resulted in lower accuracy than using the prior probability of the positive class. \n\nThere are two problems in using 0.5 as the decision threshold to make predictions (thus using it to measure performance);\n\n1. Distribution of classes are imbalanced\n2. Decision Tree classifier's resulting probabilities are not proper posterior probabilities. (LGBM does a good job of \"calibrating\" probabilities as it does leaf-wise splitting)\n\nROC-AUC performance measure is indifferent to class-distribution (in the scope of binary classification). In this case, the imbalance is not great and carefully training the model can render using 0.5 as the threshold appropriate. (It is usually positive to negative ratio of 1:10 or worse that will require special treatment)\n\nThe important thing to take away from this is that once a model has ouputted baseline probabilities, statistical analysis has ended. Choosing the right decision threshold will depend on one's the objective. For example, in the context of cancer prediction, reducing the rate of false negative is far more of importance than the false positive rate (as telling cancer baring patient that he/she is okay would be disasterous). Thus, one would take this into account to make decisions.\n"},{"metadata":{"_uuid":"8f64d5c2c240e1069d867dd296d2c6641f83b4a1"},"cell_type":"markdown","source":"**3. Feature Selection and Feature Engineering**\n\n**When you are adding new features, you are providing more informaiton to the model but also you are adding noise to the model**. Thus, the act of adding more features will not necessarily improve the model's effectiveness. \n\nAlso, decision trees work by making horizontal or vertical cuts on a domain, which means altaring an existing non-categorical feature into a ordinal or nominal feature will not be effective (i.e. Altering Age features into an ordinal feature by age category.)\n\nFirst we will look at yet to be used features and see if we can extract valuable features from them.\n\n\n**3.1 Cabin**\n\nWe will extract Deck and Cabin Number"},{"metadata":{"_uuid":"37c63a068a1370942105a48a6a62f98c5c31a73a","trusted":true,"scrolled":true},"cell_type":"code","source":"#Cabin\ntrain.Cabin.fillna(value = 'Non', inplace = True)\ndef get_deck(x):\n    if x == \"Non\":\n        return \"N\"\n    else:\n        return x[0]\n       \ndef get_cnumber(x):\n    if x == 'Non':\n        return 0\n    else:\n        c = x.split(\" \")\n        n = []\n        for i in c:\n            if i[1:] != '':\n                n.append(int(i[1:]))\n            else:\n                n.append(200)\n        \n        return sum(n)/len(n)\n    \ntrain['Deck'] = train.Cabin.apply(lambda x: get_deck(x))\ntrain.Deck = train.Deck.astype('category')\ntrain['CNumber'] = train.Cabin.apply(lambda x: get_cnumber(x))    \n\nY = train.Survived\nX = train.drop(['Survived', 'Name', 'Cabin', 'Ticket'], axis = 1)\n\nsns.countplot(x='Deck', hue='Survived', data=train)\nv_plot(train['CNumber'].copy(), train)\nrf_v_importance(X, Y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f274ebfec6307b269a335dfd97639fc9bbf219b"},"cell_type":"markdown","source":"Adding Cabin features added more noise than information. This makes sense as 77 percent of its data was missing from the column.\n\n**3.2 Ticket**\n\nWe will extract the number only."},{"metadata":{"trusted":true,"_uuid":"8800b7d679254e7283a6402d554f9bf0eeb40481"},"cell_type":"code","source":"def clean_ticket(x):\n    t = x.split(' ')\n    if len(t) != 1:\n        return int(t[-1])\n    else:\n        if t[0].isnumeric():\n            return int(t[0])\n        else:\n            return 0\ntrain['ticket'] = train.Ticket.apply(lambda x: clean_ticket(x))\ntrain.ticket = train.ticket.astype('int32')\n\nY = train.Survived\nX = train.drop(['Survived', 'Name', 'Cabin', 'Ticket', 'Deck', 'CNumber'], axis = 1)\n\nv_plot(train['ticket'].copy(), train)\nrf_v_importance(X, Y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f122d7c1196dc3bfeff77e94bb3327dd760d2cb"},"cell_type":"markdown","source":"Adding Cabin features added more noise than information. "},{"metadata":{"trusted":true,"_uuid":"557b3eafeb3a74527848fd50437004e5d1338b96"},"cell_type":"code","source":"a = ['Pclass', 'Age','Fare', 'CNumber', 'ticket', 'Deck']\nfor i in range(len(a)):\n    for j in ['Parch','SibSp']:\n         sns.catplot(x=j, y= a[i], hue='Survived', data=train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e1d1c53c2d528ec63bfff2be35a2b18bad6be13c"},"cell_type":"markdown","source":"**3.3 Title**\n\nWe will estract title from each name."},{"metadata":{"trusted":true,"_uuid":"003efb7b7eea07ab78ba22dc06f3d9a3ec821ad0","scrolled":true},"cell_type":"code","source":"import re\ndef get_title(x):\n    title = re.search('\\s(.*)\\.', x).group(1)\n    t = title.split(' ')\n    if len(t) != 1:\n        return t[-1]\n    else:\n        return t[0]\n\ntrain['Title'] = train.Name.apply(lambda x: get_title(x))\ntrain.Title = train.Title.astype('category')\n\nY = train.Survived\nX = train.drop(['Survived', 'Name', 'Cabin', 'Ticket', 'Deck', 'CNumber', 'ticket'], axis = 1)\n\nplt.figure(figsize=(20,10))\nsns.countplot(x='Title', hue='Survived', data=train)\nplt.show()\nrf_v_importance(X, Y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"771a1acea1364a68bdad1f023df6cffeea51ccbe"},"cell_type":"markdown","source":"This feature seems to be useful!"},{"metadata":{"_uuid":"c35aba243152f8a101e24ffb16d29fbc8b5d55be"},"cell_type":"markdown","source":"**3.4 Has Other Name**\nObserving \"Name\", it can be seen that some people will have an additional name attached to it. We will make a feature out of this."},{"metadata":{"trusted":true,"_uuid":"d418f809fb1057f22a93b2cbff326a149430ff30","scrolled":true},"cell_type":"code","source":"def has_other_name(x):\n    try:\n        re.search('\\(', x).group(0)\n    except AttributeError:\n        return 0\n    else:\n        return 1\ntrain['Other_Name'] = train.Name.apply(lambda x: has_other_name(x))\ntrain.Other_Name = train.Other_Name.astype('uint8')\n\nY = train.Survived\nX = train.drop(['Survived', 'Name', 'Cabin', 'Ticket', 'Deck', 'CNumber', 'ticket', 'Title'], axis = 1)\n\nsns.countplot(x='Other_Name', hue='Survived', data=train)\nplt.show()\nrf_v_importance(X, Y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2cd3b60081e97d7228e3b1dbe9cff3255182f533"},"cell_type":"markdown","source":"Now we will look at all the features together and compare it with the previous subset of features.."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"6141efec47f4e2a79824bc5134b13e7e183d24e1"},"cell_type":"code","source":"Y = train.Survived\nX = train.drop(['Survived', 'Name', 'Cabin', 'Ticket', 'Deck'], axis = 1)\nrf_v_importance(X, Y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c4d091c09feec9cd0f2f6866003270774cac4d2"},"cell_type":"markdown","source":"This result asserts the point made above: \n**Individually irrelevant features maybe become relevant in the context of others.** \n\nAlso, note that \"Deck\" has little to no importance  to the model. Leaving out this feature results in a slight decrease in our validation score (0.01), thus we will keep the column. However. this is the case only because we are not working with a big dataset with hundreds of columns and a lot of records where having one more columns can be much more costly to train and maintain in the future, This tradeoff between cost and accuracy is an important one."},{"metadata":{"trusted":true,"_uuid":"821396439468080a632a463deee831cd3209bc0b"},"cell_type":"markdown","source":"**4. Hyperparameter Tuning**\n\nInstead of using grid/random search we will make use of hyperopt for hyperparameter tuning. "},{"metadata":{"trusted":true,"_uuid":"1c831019278f84f4b6691373faa48e2f92471fa0","scrolled":true},"cell_type":"code","source":"from hyperopt import hp, tpe\nfrom hyperopt.fmin import fmin\nX_cv = lgb.Dataset(X, Y)\ndef objective(params):\n    params = {\n        'num_leaves': int(params['num_leaves']),\n        'colsample_bytree': '{:.3f}'.format(params['colsample_bytree']),\n        'n_estimators': int(params['n_estimators']),\n        'learning_rate': '{:.5f}'.format(params['learning_rate'])\n    }\n    \n    clf = lgb.LGBMClassifier(params)\n\n    \n    model_cv = lgb.cv(params, X_cv, nfold = 10,metrics = 'auc',seed = 50)\n    score = 1 - max(model_cv['auc-mean'])\n    print(\"AUC {:.6f} params {}\".format(1 - score, params))\n    return score\n\nspace = {\n    'num_leaves': hp.quniform('num_leaves', 8, 128, 2),\n    'colsample_bytree': hp.uniform('colsample_bytree', 0.3, 1.0),\n    'n_estimators': hp.quniform('n_estimator',50, 500, 50),\n    'learning_rate': hp.loguniform('learning_rate', np.log(0.005), np.log(0.5))\n}\n\nbest = fmin(fn=objective,\n            space=space,\n            algo=tpe.suggest,\n            max_evals=30)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fde26991f2553328282377b15d0a5de382468f65"},"cell_type":"code","source":"best","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a4b25be5b748ac4ca41d9ce591c7bef75023bc6f"},"cell_type":"code","source":"best['boosting'] = 'gbdt'\nbest['metric'] = 'auc'\nbest['n_estimator'] = int(best['n_estimator'])\nbest['num_leaves'] = int(best['num_leaves'] )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"10516c9d20a4348dd8a2abd403694836e1f1cc09","scrolled":true},"cell_type":"code","source":"Y = train.Survived\nX = train.drop(['Survived', 'Name', 'Cabin', 'Ticket'], axis = 1)\n\nX_train, X_val, Y_train, Y_val = model_selection.train_test_split(X, Y, test_size = 0.1, random_state = 50, shuffle = True)\ngbm_train = lgb.Dataset(X_train, Y_train)\ngbm_val = lgb.Dataset(X_val, Y_val)\n\nfinal_model = lgb.train(best, gbm_train, valid_sets = gbm_val, early_stopping_rounds = 15)\n\ndef predict(t, final_model, threshold):\n    t.Age.fillna(value = 200, inplace = True)\n    t.Cabin.fillna(value = 'Non', inplace = True)\n    t.Embarked.fillna(value = 'Non', inplace = True)\n    t['Deck'] = t.Cabin.apply(lambda x: get_deck(x))\n    t['Title'] = t.Name.apply(lambda x: get_title(x))\n    t['ticket'] = train.Ticket.apply(lambda x: clean_ticket(x))\n    t['Other_Name'] = t.Name.apply(lambda x: has_other_name(x))\n    t['CNumber'] = train.Cabin.apply(lambda x: get_cnumber(x))   \n    t[['Sex', 'Embarked', 'Title', 'Deck']] = t[['Sex', 'Embarked','Title', 'Deck']].astype('category')\n    t.drop(['PassengerId','Name', 'Cabin', 'Ticket'], axis = 1, inplace = True)\n    \n    prediction = final_model.predict(t)\n    prediction[prediction <= threshold] = 0\n    prediction[prediction != 0] = 1\n    \n    return prediction\n\nprediction = predict(test.copy(),final_model, 0.5)\nsubmission = pd.DataFrame({'PassengerId':test['PassengerId'], 'Survived':prediction})\nsubmission.to_csv('Submission',index=False)\n    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fbf5f7adae6bb3bbb922b1af1905481dd8707fde"},"cell_type":"markdown","source":"**5.Ensembling**"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}