{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nPorto Seguro Auto\n\nThis notebook starts by giving an introduction in the data of Porto Seguro competition. Then follows with preparing and running few predictive models using cross-validation and stacking and prepares a submission.\n\nThe notebook is using elements from the following kernels:\n\n- [Data Preparation and Exploration](https://www.kaggle.com/bertcarremans/data-preparation-exploration) by Bert Carremans.\n- [Steering Whell of Fortune](https://www.kaggle.com/headsortails/steering-wheel-of-fortune-porto-seguro-eda) - Porto Seguro EDA by Heads or Tails\n- [Interactive Porto Insights](https://www.kaggle.com/arthurtok/interactive-porto-insights-a-plot-ly-tutorial) - A Plot.ly Tutorial by Anisotropic\n- [Simple Stacker](https://www.kaggle.com/yekenot/simple-stacker-lb-0-284) by Vladimir Demidov","metadata":{}},{"cell_type":"markdown","source":"#  Analysis packages","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nfrom sklearn.utils import shuffle\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.preprocessing import PolynomialFeatures\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.feature_selection import VarianceThreshold\nfrom sklearn.feature_selection import SelectFromModel\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import cross_val_score\n\nfrom lightgbm import LGBMClassifier\nfrom xgboost import XGBClassifier\nfrom sklearn.linear_model import LogisticRegression\n\npd.set_option('display.max_columns', 100)\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:00.403089Z","iopub.execute_input":"2022-07-12T04:46:00.403565Z","iopub.status.idle":"2022-07-12T04:46:02.593018Z","shell.execute_reply.started":"2022-07-12T04:46:00.403446Z","shell.execute_reply":"2022-07-12T04:46:02.591823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load the data","metadata":{}},{"cell_type":"code","source":"trainset = pd.read_csv('../input/porto-seguro-safe-driver-prediction/train.csv')\ntestset = pd.read_csv('../input/porto-seguro-safe-driver-prediction/test.csv')","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:02.595373Z","iopub.execute_input":"2022-07-12T04:46:02.595777Z","iopub.status.idle":"2022-07-12T04:46:12.828575Z","shell.execute_reply.started":"2022-07-12T04:46:02.595743Z","shell.execute_reply":"2022-07-12T04:46:12.827657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Few quick observations","metadata":{}},{"cell_type":"markdown","source":"We can make few observation based on the data description in the competition:\n\n- Few **groups** are defined and features that belongs to these groups include patterns in the name (ind, reg, car, calc). The **ind** indicates most probably **individual**, **reg** is probably **registration**, **car** is self_explanatory, **calc** suggests a **calculated** field;\n- The postfix **bin** is uesd for binary features;\n- The postfix **cat** to is used for categorical features;\n- Features without the **bin** or **cat** indications are real number (continous value) of integers (ordinal values);\n- A missing value is indicated by -1;\n- The value that is subject of prediction is in the **target** column. This one indicates whether or not a claim was filed for that insured person;\n- **id** is a data input ordinal number.\n\nLet's glimpse the data to see if these interpretations are confirmed.","metadata":{}},{"cell_type":"code","source":"trainset.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:12.830186Z","iopub.execute_input":"2022-07-12T04:46:12.831274Z","iopub.status.idle":"2022-07-12T04:46:12.874260Z","shell.execute_reply.started":"2022-07-12T04:46:12.831233Z","shell.execute_reply":"2022-07-12T04:46:12.873186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Indeed, we can observed the **cat** values are **categorical**, integer values ranging from **0** to **n**, **bin** values are **binary** (either 0 or 1)","metadata":{}},{"cell_type":"markdown","source":"Let's see how many rows and columns are in the data","metadata":{}},{"cell_type":"code","source":"print('Train dataset (rows, cols):', trainset.shape, \n      '\\nTest dataset (rows, cols):', testset.shape)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:12.877576Z","iopub.execute_input":"2022-07-12T04:46:12.878049Z","iopub.status.idle":"2022-07-12T04:46:12.884824Z","shell.execute_reply.started":"2022-07-12T04:46:12.878004Z","shell.execute_reply":"2022-07-12T04:46:12.883414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 59 columns in the training dataset and only 58 in the testing dataset. Since from this dataset should have been extracted the **target**, this seems fine. Let's check the difference between the columns set in the two datasets, to make sure everything is fine.","metadata":{}},{"cell_type":"code","source":"print('Columns in train and not in test dataset:', set(trainset.columns)- set(testset.columns))","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:12.886970Z","iopub.execute_input":"2022-07-12T04:46:12.887879Z","iopub.status.idle":"2022-07-12T04:46:12.896009Z","shell.execute_reply.started":"2022-07-12T04:46:12.887838Z","shell.execute_reply":"2022-07-12T04:46:12.894862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Introduction of metadata\n\nTo make easier the manipulation of data, we will associate few meta-information to the variables in the trainset. This will facilitate the selection of various types of features for analysis, inspection or modeling. We are using as well a **category** field for the **car, ind, reg,** and **calc** types of features.\n\nWhat metadata will be used:\n\n- **use**: input, ID, target\n- **type**: nominal, interval, ordinal, binary\n- **preserve**: True or False\n- **dataType**: int, float, char\n- **category**: ind, reg, car, calc","metadata":{}},{"cell_type":"code","source":"# uses code from https://www.kaggle.com/bertcarremans/data-preparation-exploration (see references)\ndata = []\nfor feature in trainset.columns:\n    # Defining the role\n    if feature == 'target':\n        use = 'target'\n    elif feature == 'id':\n        use = 'id'\n    else:\n        use = 'input'\n         \n    # Defining the type\n    if 'bin' in feature or feature == 'target':\n        type = 'binary'\n    elif 'cat' in feature or feature == 'id':\n        type = 'categorical'\n    elif trainset[feature].dtype == float or isinstance(trainset[feature].dtype, float):\n        type = 'real'\n    elif trainset[feature].dtype == int:\n        type = 'integer'\n        \n    # Initialize preserve to True for all variables except for id\n    preserve = True\n    if feature == 'id':\n        preserve = False\n    \n    # Defining the data type \n    dtype = trainset[feature].dtype\n    \n    category = 'none'\n    # Defining the category\n    if 'ind' in feature:\n        category = 'individual'\n    elif 'reg' in feature:\n        category = 'registration'\n    elif 'car' in feature:\n        category = 'car'\n    elif 'calc' in feature:\n        category = 'calculated'\n    \n    \n    # Creating a Dict that contains all the metadata for the variable\n    feature_dictionary = {\n        'varname': feature,\n        'use': use,\n        'type': type,\n        'preserve': preserve,\n        'dtype': dtype,\n        'category' : category\n    }\n    data.append(feature_dictionary)\n    \nmetadata = pd.DataFrame(data, columns=['varname', 'use', 'type', 'preserve', 'dtype', 'category'])\nmetadata.set_index('varname', inplace=True)\nmetadata","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:12.897530Z","iopub.execute_input":"2022-07-12T04:46:12.897837Z","iopub.status.idle":"2022-07-12T04:46:12.942380Z","shell.execute_reply.started":"2022-07-12T04:46:12.897809Z","shell.execute_reply":"2022-07-12T04:46:12.941505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can extract, for example, all categorical valuse:","metadata":{}},{"cell_type":"code","source":"metadata[(metadata.type == 'categorical') & (metadata.preserve)].index","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:12.943858Z","iopub.execute_input":"2022-07-12T04:46:12.944410Z","iopub.status.idle":"2022-07-12T04:46:12.961506Z","shell.execute_reply.started":"2022-07-12T04:46:12.944376Z","shell.execute_reply":"2022-07-12T04:46:12.960332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's inspect all features, to see how many category distinct values do we have:","metadata":{}},{"cell_type":"code","source":"pd.DataFrame({'count': metadata.groupby(['category'])['category'].size()}).reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:12.963232Z","iopub.execute_input":"2022-07-12T04:46:12.963660Z","iopub.status.idle":"2022-07-12T04:46:12.983227Z","shell.execute_reply.started":"2022-07-12T04:46:12.963626Z","shell.execute_reply":"2022-07-12T04:46:12.982393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have 20 calculated features, 16 car, 18 individual and 3 registration.\n\nLet's inspect now all features, to see how many use and type distinct values do we have:","metadata":{}},{"cell_type":"code","source":"pd.DataFrame({'count': metadata.groupby(['use', 'type'])['use'].size()}).reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:12.985383Z","iopub.execute_input":"2022-07-12T04:46:12.985707Z","iopub.status.idle":"2022-07-12T04:46:12.999915Z","shell.execute_reply.started":"2022-07-12T04:46:12.985678Z","shell.execute_reply":"2022-07-12T04:46:12.999005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are one nominal feature (the **id**), 20 binary valuse, 21 real (or float numbers), 16 categorical features - all these being as wll **input** values and one **target** value, which is as well **binary**, the **target**","metadata":{}},{"cell_type":"markdown","source":"# Data analysis and statistics","metadata":{}},{"cell_type":"markdown","source":"### Target variable","metadata":{}},{"cell_type":"code","source":"plt.figure()\nfig, ax = plt.subplots(figsize=(6,6))\nx = trainset['target'].value_counts().index.values\ny = trainset[\"target\"].value_counts().values\n# Bar plot\n# Order the bars descending on target mean\nsns.barplot(ax=ax, x=x, y=y)\nplt.ylabel('Number of values', fontsize=12)\nplt.xlabel('Target value', fontsize=12)\nplt.tick_params(axis='both', which='major', labelsize=12)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:13.004157Z","iopub.execute_input":"2022-07-12T04:46:13.004688Z","iopub.status.idle":"2022-07-12T04:46:13.177700Z","shell.execute_reply.started":"2022-07-12T04:46:13.004655Z","shell.execute_reply":"2022-07-12T04:46:13.176738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Only 3.64% of the target data have 1 value. This means that the training dataset is highly imbalanced. We can either undersample the records with target = 0 or oversample records with target = 1; because is a large dataset, we will do undersampling of records with target = 0. ","metadata":{}},{"cell_type":"markdown","source":"### Real features","metadata":{}},{"cell_type":"code","source":"variable = metadata[(metadata.type == 'real') & (metadata.preserve)].index\ntrainset[variable].describe()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:13.179733Z","iopub.execute_input":"2022-07-12T04:46:13.180392Z","iopub.status.idle":"2022-07-12T04:46:13.508085Z","shell.execute_reply.started":"2022-07-12T04:46:13.180340Z","shell.execute_reply":"2022-07-12T04:46:13.506968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(pow(trainset['ps_car_12']*10,2)).head(10)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:13.509512Z","iopub.execute_input":"2022-07-12T04:46:13.509873Z","iopub.status.idle":"2022-07-12T04:46:13.521559Z","shell.execute_reply.started":"2022-07-12T04:46:13.509841Z","shell.execute_reply":"2022-07-12T04:46:13.520489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(pow(trainset['ps_car_15'],2)).head(10)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:13.523603Z","iopub.execute_input":"2022-07-12T04:46:13.524032Z","iopub.status.idle":"2022-07-12T04:46:13.534551Z","shell.execute_reply.started":"2022-07-12T04:46:13.523989Z","shell.execute_reply":"2022-07-12T04:46:13.533583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Features with missing values\n**ps_reg_03, ps_car_12, ps_car_14** have missing values (their minimum value is -1)\n\n\n#### Registration features\n**ps_reg_01** and **ps_reg_02** are fractions with denominator 10 (values of 0.1, 0.2, 0.3)\n\n\n#### Car features\n**ps_car_12** are (with some approximations) square roots (divided by 10) of natural numbers whilst\n\n**ps_car_15** are square roots of natural numbers. Let's represent the values using *pairplot*","metadata":{}},{"cell_type":"code","source":"sample = trainset.sample(frac=0.05)\nvar = ['ps_car_12', 'ps_car_15', 'target']\nsample = sample[var]\nsns.pairplot(sample,  hue='target', palette = 'Set1', diag_kind='kde')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:13.535681Z","iopub.execute_input":"2022-07-12T04:46:13.536123Z","iopub.status.idle":"2022-07-12T04:46:18.314350Z","shell.execute_reply.started":"2022-07-12T04:46:13.536076Z","shell.execute_reply":"2022-07-12T04:46:18.312979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Calculated features\n\nThe features **ps_calc_01, ps_calc_02, ps_calc_03** have very similar distributions and could be some kind of ratio, since the maximum value if for al three 0.9. The other calculated values have maximum value an integer value (5, 6, 7, 10, 12)","metadata":{}},{"cell_type":"code","source":"var = metadata[(metadata.type == 'real') & (metadata.preserve)].index\ni = 0\nt1 = trainset.loc[trainset['target'] != 0]\nt0 = trainset.loc[trainset['target'] == 0]\n\nsns.set_style('whitegrid')\nplt.figure()\nfig, ax = plt.subplots(3,4,figsize=(16,12))\n\nfor feature in var:\n    i += 1\n    plt.subplot(3,4,i)\n    sns.kdeplot(t1[feature], bw=0.5,label=\"target = 1\")\n    sns.kdeplot(t0[feature], bw=0.5,label=\"target = 0\")\n    plt.ylabel('Density plot', fontsize=12)\n    plt.xlabel(feature, fontsize=12)\n    locs, labels = plt.xticks()\n    plt.tick_params(axis='both', which='major', labelsize=12)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:18.316080Z","iopub.execute_input":"2022-07-12T04:46:18.316439Z","iopub.status.idle":"2022-07-12T04:46:39.858277Z","shell.execute_reply.started":"2022-07-12T04:46:18.316406Z","shell.execute_reply":"2022-07-12T04:46:39.856851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**ps_reg_02, ps_car_13, ps_car_15** shows the most different distributions between sets of values associated with target=0 and target=1.\n","metadata":{}},{"cell_type":"markdown","source":"Let's visualize the correlation between the real features","metadata":{}},{"cell_type":"code","source":"def corr_heatmap(var):\n    correlations = trainset[var].corr()\n    \n    # Create color map ranging between two colors\n    cmap = sns.diverging_palette(50, 10, as_cmap= True)\n    \n    fig, ax= plt.subplots(figsize= (10, 10))\n    sns.heatmap(correlations, cmap= cmap, vmax= 1.0, center= 0, fmt= '.2f',\n               square= True, linewidths= .5, annot= True, cbar_kws= {'shrink': .75})\n    plt.show();\n    \nvar = metadata[(metadata.type == 'real') & (metadata.preserve)].index\ncorr_heatmap(var)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:39.859871Z","iopub.execute_input":"2022-07-12T04:46:39.860236Z","iopub.status.idle":"2022-07-12T04:46:40.801005Z","shell.execute_reply.started":"2022-07-12T04:46:39.860202Z","shell.execute_reply":"2022-07-12T04:46:40.800059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's visualize the plots of the variables with strong correlations. These are:\n\n- ps_reg_01 with ps_reg_02 (0.47);\n- ps_reg_01 with ps_reg_03 (0.64);\n- ps_reg_02 with ps_reg_03 (0.52);\n- ps_car_12 with ps_car_13 (0.67);\n- ps_car_13 with ps_car_15 (0.53);\n\nTo show the pairs of values that are correlated we use pairplot. Before representing the pairs, we subsample the data, using only 2% in the sample.","metadata":{}},{"cell_type":"code","source":"sample = trainset.sample(frac= 0.05)\nvar = ['ps_reg_01', 'ps_reg_02', 'ps_reg_03', 'ps_car_12', 'ps_car_13', 'ps_car_15', 'target']\nsample= sample[var]\nsns.pairplot(sample, hue= 'target', palette= 'Set1', diag_kind= 'kde')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:46:40.802431Z","iopub.execute_input":"2022-07-12T04:46:40.803021Z","iopub.status.idle":"2022-07-12T04:47:40.457504Z","shell.execute_reply.started":"2022-07-12T04:46:40.802986Z","shell.execute_reply":"2022-07-12T04:47:40.456593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Binary features","metadata":{}},{"cell_type":"code","source":"v = metadata[(metadata.type == 'binary') & (metadata.preserve)].index\ntrainset[v].describe()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:47:40.458816Z","iopub.execute_input":"2022-07-12T04:47:40.459309Z","iopub.status.idle":"2022-07-12T04:47:40.833624Z","shell.execute_reply.started":"2022-07-12T04:47:40.459277Z","shell.execute_reply":"2022-07-12T04:47:40.832490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's plot the distribution of the binary data in the training dataset. With blue we represent the percent of 0 and with red the percent of 1.","metadata":{}},{"cell_type":"code","source":"bin_col = [col for col in trainset.columns if '_bin' in col]\nzero_list = []\none_list = []\nfor col in bin_col:\n    zero_list.append((trainset[col]== 0).sum() / trainset.shape[0]*100)\n    one_list.append((trainset[col]== 1).sum() / trainset.shape[0]*100)\n\nplt.figure()\nfig, ax= plt.subplots(figsize= (6, 6))\n# Bar plot\np1 = sns.barplot(ax= ax, x= bin_col, y= zero_list, color= 'blue')\np2 = sns.barplot(ax= ax, x= bin_col, y= one_list, bottom= zero_list, color= 'red')\nplt.ylabel('Percent of zero/one [%]', fontsize= 12)\nplt.xlabel('Binary features', fontsize= 12)\nlocs, labels = plt.xticks()\nplt.setp(labels, rotation= 90)\nplt.tick_params(axis= 'both', which= 'major', labelsize= 12)\nplt.legend((p1, p2), ('Zero', 'One'))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:47:40.834978Z","iopub.execute_input":"2022-07-12T04:47:40.835314Z","iopub.status.idle":"2022-07-12T04:47:41.342281Z","shell.execute_reply.started":"2022-07-12T04:47:40.835283Z","shell.execute_reply":"2022-07-12T04:47:41.341415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**ps_ind_10_bin, ps_ind_11_bin, ps_ind_12_bin, ps_ind_13_bin** have very small number of values 1 (less than 0.5%) whilst the number of value 1 is very large for **ps_in_16_bin** and **ps_cals_16_bin** (more than 60%)\n\nLet's see now the distribution of binary data and the corresponding valuse of **target** variable.","metadata":{}},{"cell_type":"code","source":"var = metadata[(metadata.type== 'binary') & (metadata.preserve)].index\nvar = [col for col in trainset.columns if '_bin' in col]\ni = 0\nt1 = trainset.loc[trainset['target'] != 0]\nt0 = trainset.loc[trainset['target'] == 0]\n\nsns.set_style('whitegrid')\nplt.figure()\nfig, ax= plt.subplots(6, 3, figsize= (12, 24))\n\nfor feature in var:\n    i += 1\n    plt.subplot(6, 3, i)\n    sns.kdeplot(t1[feature], bw= 0.5, label= 'target = 1')\n    sns.kdeplot(t0[feature], bw= 0.5, label= 'target = 0')\n    plt.ylabel('Density plot', fontsize= 12)\n    plt.xlabel(feature, fontsize= 12)\n    locs, labels = plt.xticks()\n    plt.tick_params(axis= 'both', which= 'major', labelsize= 12)\n    \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:47:41.343501Z","iopub.execute_input":"2022-07-12T04:47:41.344266Z","iopub.status.idle":"2022-07-12T04:48:16.961873Z","shell.execute_reply.started":"2022-07-12T04:47:41.344231Z","shell.execute_reply":"2022-07-12T04:48:16.960431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**ps_ind_06_bin, ps_ind_07_bin, ps_ind_16_bin, ps_ind_17_bin** shows high inbalance between distribution of values of 1 and 0 for values of target equals with 1 and 0, **ps_ind_08_bin** shows a small inbalance while the other features are well balanced, having similar density plots.","metadata":{}},{"cell_type":"markdown","source":"### Categorical feature","metadata":{}},{"cell_type":"markdown","source":"We will represent the distribution on categorical data in two ways. First we calculate the percentage of target = 1 per category value and represent these percentages using bar plots","metadata":{}},{"cell_type":"code","source":"var = metadata[(metadata.type== 'categorical') & (metadata.preserve)].index\n\nfor feature in var:\n    fig, ax = plt.subplots(figsize= (6, 6))\n    # Calculate the percentage of target = 1 per category value\n    cat_perc = trainset[[feature, 'target']].groupby([feature], as_index= False).mean()\n    cat_perc.sort_values(by= 'target', ascending= False, inplace = True)\n    # Barplot\n    # Order the bars descending on target mean\n    sns.barplot(ax= ax, x= feature, y= 'target', data= cat_perc, order= cat_perc[feature])\n    plt.ylabel('Percent of target with value 1 [%]', fontsize= 12)\n    plt.xlabel(feature, fontsize= 12)\n    plt.tick_params(axis= 'both', which= 'major', labelsize= 12)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:48:16.963929Z","iopub.execute_input":"2022-07-12T04:48:16.964452Z","iopub.status.idle":"2022-07-12T04:48:22.527446Z","shell.execute_reply.started":"2022-07-12T04:48:16.964401Z","shell.execute_reply":"2022-07-12T04:48:22.526186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Alternativelly we represent the categorical features using density plot. We select values with target = 0 and target = 1 and represent both density plots in the same graphic.","metadata":{}},{"cell_type":"code","source":"var = metadata[(metadata.type == 'categorical') & (metadata.preserve)].index\ni = 0\nt1 = trainset.loc[trainset['target'] != 0]\nt0 = trainset.loc[trainset['target'] == 0]\n\nsns.set_style('whitegrid')\nplt.figure()\nfig, ax= plt.subplots(4, 4, figsize= (16,16))\n\nfor feature in var:\n    i += 1\n    plt.subplot(4, 4, i)\n    sns.kdeplot(t1[feature], bw= 0.5, label= 'target = 1')\n    sns.kdeplot(t0[feature], bw= 0.5, label= 'target = 0')\n    plt.ylabel('Density plot', fontsize= 12)\n    plt.xlabel(feature, fontsize= 12)\n    locs, labels = plt.xticks()\n    plt.tick_params(axis= 'both', which= 'major', labelsize= 12)\n    \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:48:22.529269Z","iopub.execute_input":"2022-07-12T04:48:22.530350Z","iopub.status.idle":"2022-07-12T04:48:50.443120Z","shell.execute_reply.started":"2022-07-12T04:48:22.530298Z","shell.execute_reply":"2022-07-12T04:48:50.441902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**ps_car_03_cat, ps_car_05_cat** shows the most different density plot between values associated with target=0 and target=1.","metadata":{}},{"cell_type":"markdown","source":"### Data unblance between train and test data","metadata":{}},{"cell_type":"markdown","source":"Let's compare the distribution of the features in the train and test dataset.\n\nWe start with the reg or registration features.","metadata":{}},{"cell_type":"code","source":"var = metadata[(metadata.category == 'registration') & (metadata.preserve)].index\n\n# Bar plot\nsns.set_style('whitegrid')\n\nplt.figure()\nfig, ax = plt.subplots(1,3,figsize=(12,4))\ni = 0\nfor feature in var:\n    i = i + 1\n    plt.subplot(1,3,i)\n    sns.kdeplot(trainset[feature], bw=0.5, label=\"train\")\n    sns.kdeplot(testset[feature], bw=0.5, label=\"test\")\n    plt.ylabel('Distribution', fontsize=12)\n    plt.xlabel(feature, fontsize=12)\n    locs, labels = plt.xticks()\n    plt.tick_params(axis='both', which='major', labelsize=12)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:48:50.444718Z","iopub.execute_input":"2022-07-12T04:48:50.445409Z","iopub.status.idle":"2022-07-12T04:49:03.950554Z","shell.execute_reply.started":"2022-07-12T04:48:50.445366Z","shell.execute_reply":"2022-07-12T04:49:03.949183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All reg features shows well balanced train test sets.\n\nLet's continue with car features.","metadata":{}},{"cell_type":"code","source":"var = metadata[(metadata.category == 'car') & (metadata.preserve)].index\n\n# Bar plot\nsns.set_style('whitegrid')\n\nplt.figure()\nfig, ax= plt.subplots(4, 4, figsize= (20, 16))\n\ni = 0\nfor feature in var:\n    i += 1\n    plt.subplot(4, 4, i)\n    sns.kdeplot(trainset[feature], bw= 0.5, label = 'train')\n    sns.kdeplot(testset[feature], bw= 0.5, label = 'test')\n    plt.ylabel('Distribution', fontsize= 12)\n    plt.xlabel(feature, fontsize= 12)\n    locs, labels = plt.xticks()\n    plt.tick_params(axis= 'both', which= 'major', labelsize= 12)\n    \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:49:03.952490Z","iopub.execute_input":"2022-07-12T04:49:03.952997Z","iopub.status.idle":"2022-07-12T04:50:15.892252Z","shell.execute_reply.started":"2022-07-12T04:49:03.952952Z","shell.execute_reply":"2022-07-12T04:50:15.890936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the car features, all variables looks well balanced between train and test set.\n\nLet's look now to the ind (individual) values.","metadata":{}},{"cell_type":"code","source":"var = metadata[(metadata.category== 'individual') & (metadata.preserve)].index\n\n# Barplot\nsns.set_style('whitegrid')\n\nplt.figure()\nfig, ax= plt.subplots(5, 4, figsize= (20, 16))\n\ni = 0\nfor feature in var:\n    i += 1\n    plt.subplot(5, 4, i)\n    sns.kdeplot(trainset[feature], bw= 0.5, label = 'train')\n    sns.kdeplot(testset[feature], bw= 0.5, label = 'test')\n    plt.ylabel('Distribution', fontsize= 12)\n    plt.xlabel(feature, fontsize= 12)\n    locs, labels = plt.xticks()\n    plt.tick_params(axis= 'both', which= 'major', labelsize= 12)\n    \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:50:15.893863Z","iopub.execute_input":"2022-07-12T04:50:15.894298Z","iopub.status.idle":"2022-07-12T04:51:43.615391Z","shell.execute_reply.started":"2022-07-12T04:50:15.894266Z","shell.execute_reply":"2022-07-12T04:51:43.613790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"var = metadata[(metadata.category== 'calculated') & (metadata.preserve)].index\n\n# Barplot\nsns.set_style('whitegrid')\n\nplt.figure()\nfig, ax= plt.subplots(5, 4, figsize= (20, 16))\n\ni = 0\nfor feature in var:\n    i += 1\n    plt.subplot(5, 4, i)\n    sns.kdeplot(trainset[feature], bw= 0.5, label = 'train')\n    sns.kdeplot(testset[feature], bw= 0.5, label = 'test')\n    plt.ylabel('Distribution', fontsize= 12)\n    plt.xlabel(feature, fontsize= 12)\n    locs, labels = plt.xticks()\n    plt.tick_params(axis= 'both', which= 'major', labelsize= 12)\n    \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:51:43.617504Z","iopub.execute_input":"2022-07-12T04:51:43.618787Z","iopub.status.idle":"2022-07-12T04:53:12.853056Z","shell.execute_reply.started":"2022-07-12T04:51:43.618748Z","shell.execute_reply":"2022-07-12T04:53:12.851855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Check data quality","metadata":{}},{"cell_type":"markdown","source":"Let's inspect the features with missing values:","metadata":{}},{"cell_type":"code","source":"vars_with_missing = []\n\nfor feature in trainset.columns:\n    missings = trainset[trainset[feature] == -1][feature].count()\n    if missings > 0:\n        vars_with_missing.append(feature)\n        missings_perc = missings/trainset.shape[0]\n        \n        print('Variable {} has {} records ({:.2%}) with missing values'.format(feature, missings, missings_perc))\n        \nprint('In total, there are {} variables with mising vlaues'.format(len(vars_with_missing)))","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:53:12.854781Z","iopub.execute_input":"2022-07-12T04:53:12.855223Z","iopub.status.idle":"2022-07-12T04:53:13.193756Z","shell.execute_reply.started":"2022-07-12T04:53:12.855186Z","shell.execute_reply":"2022-07-12T04:53:13.192321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare the data for model","metadata":{}},{"cell_type":"markdown","source":"> #### Drop **clac** columns\n\nWe also drop the **calc** columns, as recommended in [5]. These seems to be all engineered and, according  to Dmitry Altukhov, he was able to improve his CV score while succeesivelly removing all of them.","metadata":{}},{"cell_type":"code","source":"col_to_drop = trainset.columns[trainset.columns.str.startswith('ps_calc_')]\ntrainset = trainset.drop(col_to_drop, axis=1)  \ntestset = testset.drop(col_to_drop, axis=1) ","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:53:13.199553Z","iopub.execute_input":"2022-07-12T04:53:13.199918Z","iopub.status.idle":"2022-07-12T04:53:13.381478Z","shell.execute_reply.started":"2022-07-12T04:53:13.199889Z","shell.execute_reply":"2022-07-12T04:53:13.380410Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> #### Drop variables with too many missing values\n\nWe select from the variables with missing values two, **ps_car_03_cat** and **ps_car_05_cat** to drop.","metadata":{}},{"cell_type":"code","source":"# Dropping the variables with too many missing values\nvars_to_drop = ['ps_car_03_cat', 'ps_car_05_cat']\ntrainset.drop(vars_to_drop, inplace=True, axis=1)\ntestset.drop(vars_to_drop, inplace=True, axis=1)\nmetadata.loc[(vars_to_drop),'keep'] = False  # Updating the meta","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:53:13.382929Z","iopub.execute_input":"2022-07-12T04:53:13.383370Z","iopub.status.idle":"2022-07-12T04:53:13.544512Z","shell.execute_reply.started":"2022-07-12T04:53:13.383336Z","shell.execute_reply":"2022-07-12T04:53:13.543328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Script by https://www.kaggle.com/ogrellier\n# Code: https://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features\ndef add_noise(series, noise_level):\n    return series * (1 + noise_level * np.random.randn(len(series)))\n\n    \"\"\"\n    Smoothing is computed like in the following paper by Daniele Micci-Barreca\n    https://kaggle2.blob.core.windows.net/forum-message-attachments/225952/7441/high%20cardinality%20categoricals.pdf\n    trn_series : training categorical feature as a pd.Series\n    tst_series : test categorical feature as a pd.Series\n    target : target data as a pd.Series\n    min_samples_leaf (int) : minimum samples to take category average into account\n    smoothing (int) : smoothing effect to balance categorical average vs prior  \n    \"\"\" \ndef target_encode(trn_series=None, \n                  tst_series=None, \n                  target=None, \n                  min_samples_leaf=1, \n                  smoothing=1,\n                  noise_level=0):\n    assert len(trn_series) == len(target)\n    assert trn_series.name == tst_series.name\n    temp = pd.concat([trn_series, target], axis=1)\n    \n    # Compute target mean \n    averages = temp.groupby(by=trn_series.name)[target.name].agg([\"mean\", \"count\"])\n    \n    # Compute smoothing\n    smoothing = 1 / (1 + np.exp(-(averages[\"count\"] - min_samples_leaf) / smoothing))\n    \n    # Apply average function to all target data\n    prior = target.mean()\n    \n    # The bigger the count the less full_avg is taken into account\n    averages[target.name] = prior * (1 - smoothing) + averages[\"mean\"] * smoothing\n    averages.drop([\"mean\", \"count\"], axis=1, inplace=True)\n    \n    # Apply averages to trn and tst series\n    ft_trn_series = pd.merge(\n        trn_series.to_frame(trn_series.name),\n        averages.reset_index().rename(columns={'index': target.name, target.name: 'average'}),\n        on=trn_series.name,\n        how='left')['average'].rename(trn_series.name + '_mean').fillna(prior)\n    # pd.merge does not keep the index so restore it\n    ft_trn_series.index = trn_series.index \n    \n    ft_tst_series = pd.merge(\n        tst_series.to_frame(tst_series.name),\n        averages.reset_index().rename(columns={'index': target.name, target.name: 'average'}),\n        on=tst_series.name,\n        how='left')['average'].rename(trn_series.name + '_mean').fillna(prior)\n    # pd.merge does not keep the index so restore it\n    ft_tst_series.index = tst_series.index\n    \n    return add_noise(ft_trn_series, noise_level), add_noise(ft_tst_series, noise_level)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:53:13.546051Z","iopub.execute_input":"2022-07-12T04:53:13.546627Z","iopub.status.idle":"2022-07-12T04:53:13.562608Z","shell.execute_reply.started":"2022-07-12T04:53:13.546591Z","shell.execute_reply":"2022-07-12T04:53:13.561040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Replace ps_car_11_cat with encoded value\n\nUsing the **target_encode** function, we replace the **ps_car_11_cat** with an encoded value in both **train** and **test** datasets.","metadata":{}},{"cell_type":"code","source":"train_encoded, test_encoded = target_encode(trainset[\"ps_car_11_cat\"], \n                             testset[\"ps_car_11_cat\"], \n                             target=trainset.target, \n                             min_samples_leaf=100,\n                             smoothing=10,\n                             noise_level=0.01)\n    \ntrainset['ps_car_11_cat_te'] = train_encoded\ntrainset.drop('ps_car_11_cat', axis=1, inplace=True)\nmetadata.loc['ps_car_11_cat','keep'] = False  # Updating the metadata\ntestset['ps_car_11_cat_te'] = test_encoded\ntestset.drop('ps_car_11_cat', axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:53:13.564146Z","iopub.execute_input":"2022-07-12T04:53:13.564860Z","iopub.status.idle":"2022-07-12T04:53:14.050748Z","shell.execute_reply.started":"2022-07-12T04:53:13.564824Z","shell.execute_reply":"2022-07-12T04:53:14.049478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Balance target variable\n\nThe target variable is highly unbalanced. This cna be imporved by either undersampling values with **target = 0** or oversampling values with **target = 1**.\n\nBecause there is a rather large training set, we opt for the **undersampling.**","metadata":{}},{"cell_type":"code","source":"desired_apriori=0.10\n\n# Get the indices per target value\nidx_0 = trainset[trainset.target == 0].index\nidx_1 = trainset[trainset.target == 1].index\n\n# Get original number of records per target value\nnb_0 = len(trainset.loc[idx_0])\nnb_1 = len(trainset.loc[idx_1])\n\n# Calculate the undersampling rate and resulting number of records with target=0\nundersampling_rate = ((1-desired_apriori)*nb_1)/(nb_0*desired_apriori)\nundersampled_nb_0 = int(undersampling_rate*nb_0)\nprint('Rate to undersample records with target=0: {}'.format(undersampling_rate))\nprint('Number of records with target=0 after undersampling: {}'.format(undersampled_nb_0))\n\n# Randomly select records with target=0 to get at the desired a priori\nundersampled_idx = shuffle(idx_0, random_state=314, n_samples=undersampled_nb_0)\n\n# Construct list with remaining indices\nidx_list = list(undersampled_idx) + list(idx_1)\n\n# Return undersample data frame\ntrainset = trainset.loc[idx_list].reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:53:14.052617Z","iopub.execute_input":"2022-07-12T04:53:14.053404Z","iopub.status.idle":"2022-07-12T04:53:14.483556Z","shell.execute_reply.started":"2022-07-12T04:53:14.053354Z","shell.execute_reply":"2022-07-12T04:53:14.481948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Replace -1 values with NaN\n\nMost of the classifiers we would use have preety good stratgies to manage missing (or NaN) values.","metadata":{}},{"cell_type":"code","source":"trainset = trainset.replace(-1, np.nan)\ntestset = testset.replace(-1, np.nan)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:53:14.485187Z","iopub.execute_input":"2022-07-12T04:53:14.485624Z","iopub.status.idle":"2022-07-12T04:53:14.679327Z","shell.execute_reply.started":"2022-07-12T04:53:14.485587Z","shell.execute_reply":"2022-07-12T04:53:14.677789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Dummify **cat** values\n\nWe wiil create dummy variables for the **categorical (cat)** features","metadata":{}},{"cell_type":"code","source":"cat_features = [a for a in trainset.columns if a.endswith('cat')]\n\nfor column in cat_features:\n    temp = pd.get_dummies(pd.Series(trainset[column]))\n    trainset = pd.concat([trainset,temp],axis=1)\n    trainset = trainset.drop([column],axis=1)\n    \nfor column in cat_features:\n    temp = pd.get_dummies(pd.Series(testset[column]))\n    testset = pd.concat([testset,temp],axis=1)\n    testset = testset.drop([column],axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:53:14.681383Z","iopub.execute_input":"2022-07-12T04:53:14.681817Z","iopub.status.idle":"2022-07-12T04:53:20.371855Z","shell.execute_reply.started":"2022-07-12T04:53:14.681782Z","shell.execute_reply":"2022-07-12T04:53:20.370564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Drop unused and **target** columns\n\nWe separate the **id** and **target** (drop these columns)","metadata":{}},{"cell_type":"code","source":"id_test = testset['id'].values\ntarget_train = trainset['target'].values\n\ntrainset = trainset.drop(['target', 'id'], axis = 1)\ntestset = testset.drop(['id'], axis= 1)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:58:33.964993Z","iopub.execute_input":"2022-07-12T04:58:33.965401Z","iopub.status.idle":"2022-07-12T04:58:34.213872Z","shell.execute_reply.started":"2022-07-12T04:58:33.965366Z","shell.execute_reply":"2022-07-12T04:58:34.212622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's inspect the training and test sets:","metadata":{}},{"cell_type":"code","source":"print('Train dataset (rows, cols):', trainset.values.shape,\n     '\\nTest dataset (rows, cols):', testset.values.shape)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:58:36.792046Z","iopub.execute_input":"2022-07-12T04:58:36.792494Z","iopub.status.idle":"2022-07-12T04:58:37.091554Z","shell.execute_reply.started":"2022-07-12T04:58:36.792463Z","shell.execute_reply":"2022-07-12T04:58:37.090307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare the model\n\n#### Ensemble class for cross validation and ensemble\n\nPrepare an **Ensemble** calss to split the data in KFolds, train the models and ensemble the results.\n\nThe class has an **init** method (called when an Ensemble object is created) that accepts 4 parameters:\n\n- **self** - the object to be initialized\n- **n_splits** - the number of cross-validation splits to be used\n- **stacker** - the model used for stacking the prediction results from the trained base models\n- **base_models** - the list of base models used in training\n\nA second methond, **fit_predict** has for functions:\n\n- split the training data in **n_splits** folds;\n- run the **base models** for each fold;\n- perform prediction using each model;\n- ensemble the results using the **stacker**;","metadata":{}},{"cell_type":"code","source":"class Ensemble(object):\n    def __init__(self, n_splits, stacker, base_models):\n        self.n_splits = n_splits\n        self.stacker = stacker\n        self.base_models = base_models\n\n    def fit_predict(self, X, y, T):\n        X = np.array(X)\n        y = np.array(y)\n        T = np.array(T)\n\n        folds = list(StratifiedKFold(n_splits=self.n_splits, shuffle=True, random_state=314).split(X, y))\n\n        S_train = np.zeros((X.shape[0], len(self.base_models)))\n        S_test = np.zeros((T.shape[0], len(self.base_models)))\n        for i, clf in enumerate(self.base_models):\n\n            S_test_i = np.zeros((T.shape[0], self.n_splits))\n\n            for j, (train_idx, test_idx) in enumerate(folds):\n                X_train = X[train_idx]\n                y_train = y[train_idx]\n                X_holdout = X[test_idx]\n\n\n                print (\"Base model %d: fit %s model | fold %d\" % (i+1, str(clf).split('(')[0], j+1))\n                clf.fit(X_train, y_train)\n                cross_score = cross_val_score(clf, X_train, y_train, cv=3, scoring='roc_auc')\n                print(\"cross_score [roc-auc]: %.5f [gini]: %.5f\" % (cross_score.mean(), 2*cross_score.mean()-1))\n                y_pred = clf.predict_proba(X_holdout)[:,1]                \n\n                S_train[test_idx, i] = y_pred\n                S_test_i[:, j] = clf.predict_proba(T)[:,1]\n            S_test[:, i] = S_test_i.mean(axis=1)\n\n        results = cross_val_score(self.stacker, S_train, y, cv=3, scoring='roc_auc')\n        # Calculate gini factor as 2 * AUC - 1\n        print(\"Stacker score [gini]: %.5f\" % (2 * results.mean() - 1))\n\n        self.stacker.fit(S_train, y)\n        res = self.stacker.predict_proba(S_test)[:,1]\n        return res","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:58:40.466085Z","iopub.execute_input":"2022-07-12T04:58:40.466504Z","iopub.status.idle":"2022-07-12T04:58:40.484300Z","shell.execute_reply.started":"2022-07-12T04:58:40.466470Z","shell.execute_reply":"2022-07-12T04:58:40.483251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Parameters for the base models\n\nFor the base models, we prepare three different LightGBM models and one XGB model.\n\nEach model is used to train the data (using as well cross-validation, with 3 folds)","metadata":{}},{"cell_type":"code","source":"# LightGBM params\n# lgb_1\nlgb_params1 = {}\nlgb_params1['learning_rate'] = 0.02\nlgb_params1['n_estimators'] = 650\nlgb_params1['max_bin'] = 10\nlgb_params1['subsample'] = 0.8\nlgb_params1['subsample_freq'] = 10\nlgb_params1['colsample_bytree'] = 0.8\nlgb_params1['min_child_samples'] = 500\nlgb_params1['seed'] = 314\nlgb_params1['num_threads'] = 4\n\n#lgb_2\nlgb_params2 = {}\nlgb_params2['n_estimators'] = 1090\nlgb_params2['learning_rate'] = 0.02\nlgb_params2['colsample_bytree']= 0.3\nlgb_params2['subsample'] = 0.7\nlgb_params2['subsample_freq'] = 2\nlgb_params2['num_leaves'] = 16\nlgb_params2['seed'] = 314\nlgb_params2['num_threads'] = 4\n\n# lgb_3\nlgb_params3 = {}\nlgb_params3['n_estimators'] = 1100\nlgb_params3['max_depth'] = 4\nlgb_params3['learning_rate'] = 0.02\nlgb_params3['seed'] = 314\nlgb_params3['num_threads'] = 4\n\n# XGBoost params\nxgb_params = {}\nxgb_params['objective'] = 'binary:logistic'\nxgb_params['learning_rate'] = 0.04\nxgb_params['n_estimators'] = 490\nxgb_params['max_depth'] = 4\nxgb_params['subsample'] = 0.9\nxgb_params['colsample_bytree'] = 0.9\nxgb_params['min_child_weight'] = 10\nxgb_params['num_threads'] = 4","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:58:43.516399Z","iopub.execute_input":"2022-07-12T04:58:43.516794Z","iopub.status.idle":"2022-07-12T04:58:43.528039Z","shell.execute_reply.started":"2022-07-12T04:58:43.516753Z","shell.execute_reply":"2022-07-12T04:58:43.527179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Initialize the models with the parameters\n\nWe inin the 3 base models and the stacking model. For the base models we are using the predefined parameters initialized above.","metadata":{}},{"cell_type":"code","source":"# Base model\nlgb_model1 = LGBMClassifier(**lgb_params1)\n\nlgb_model2 = LGBMClassifier(**lgb_params2)\n\nlgb_model3 = LGBMClassifier(**lgb_params3)\n\nxgb_model = XGBClassifier(**xgb_params)\n\n# Stacking model\nlog_model = LogisticRegression()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:58:46.997542Z","iopub.execute_input":"2022-07-12T04:58:46.998635Z","iopub.status.idle":"2022-07-12T04:58:47.004506Z","shell.execute_reply.started":"2022-07-12T04:58:46.998587Z","shell.execute_reply":"2022-07-12T04:58:47.003039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Initialize the ensembleing object\n\nUsing Ensemble.init we init the stacking object","metadata":{}},{"cell_type":"code","source":"stack = Ensemble(n_splits = 3,\n                stacker = log_model,\n                base_models = (lgb_model1, lgb_model2, lgb_model3, xgb_model))","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:58:48.703040Z","iopub.execute_input":"2022-07-12T04:58:48.703751Z","iopub.status.idle":"2022-07-12T04:58:48.708457Z","shell.execute_reply.started":"2022-07-12T04:58:48.703702Z","shell.execute_reply":"2022-07-12T04:58:48.707446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Run the predictive models\n\nCalling the **fit_predict** method of **stack** object, we run training of the base models, predict the **target** with each model, ensemble the results using the **stacker** model and output the stacked result.","metadata":{}},{"cell_type":"code","source":"y_prediction = stack.fit_predict(trainset, target_train, testset)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T04:58:55.625439Z","iopub.execute_input":"2022-07-12T04:58:55.625870Z","iopub.status.idle":"2022-07-12T05:22:52.766366Z","shell.execute_reply.started":"2022-07-12T04:58:55.625836Z","shell.execute_reply":"2022-07-12T05:22:52.765091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare the submission","metadata":{}},{"cell_type":"code","source":"submission = pd.DataFrame()\nsubmission['id'] = id_test\nsubmission['target'] = y_prediction\nsubmission.to_csv('./stacked.csv', index= False)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T05:22:52.768788Z","iopub.execute_input":"2022-07-12T05:22:52.769961Z","iopub.status.idle":"2022-07-12T05:22:56.026354Z","shell.execute_reply.started":"2022-07-12T05:22:52.769909Z","shell.execute_reply":"2022-07-12T05:22:56.025343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# References","metadata":{}},{"cell_type":"markdown","source":"[1] Porto Seguro Safe Driver Prediction, Kaggle Competition,\nhttps://www.kaggle.com/c/porto-seguro-safe-driver-prediction\n\n[2] Bert Carremans, Data Preparation and Exploration, Kaggle Kernel,\nhttps://www.kaggle.com/bertcarremans/data-preparation-exploration\n\n[3] Head or Tails, Steering Whell of Fortune - Porto Seguro EDA, Kaggle Kernel, \nhttps://www.kaggle.com/headsortails/steering-wheel-of-fortune-porto-seguro-eda\n\n[4] Anisotropic, Interactive Porto Insights - A Plot.ly Tutorial, Kaggle Kernel, \nhttps://www.kaggle.com/arthurtok/interactive-porto-insights-a-plot-ly-tutorial\n\n[5] Dmitry Altukhov, Kaggle Porto Seguro's Safe Driver Prediction (3rd place solution), \nhttps://www.youtube.com/watch?v=mbxZ_zqHV9c\n\n[6] Vladimir Demidov, Simple Staker LB 0.284, \nhttps://www.kaggle.com/yekenot/simple-stacker-lb-0-284\n\n[7] Anisotropic, Introduction to Ensembling/Stacking in Python, \nhttps://www.kaggle.com/arthurtok/introduction-to-ensembling-stacking-in-python","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}