{"cells":[{"metadata":{"_uuid":"85fa197e0238c1326ec9b0fb821b4e7e72547363"},"cell_type":"markdown","source":"**HWR Berlin**     \n**Data Science Group Project**     \n**Elo Merchant Category Recommendation**       \n**30.01.2019**\n\nFabian Petsche, Matr. Nr: 463511\n\nOksana Hrytsiv, Matr. Nr: 741766\n\nFabian Asal, Matr. Nr: 734883  "},{"metadata":{"_uuid":"049d17267ee26452b69211ad424094f522e9ea80"},"cell_type":"markdown","source":"## Notebook Content:\n\n  1. [Introduction](#1)\n \n  1.1 [Elo](#1.1)\n  \n  1.2 [Problem Statement](#1.2)\n  \n  \n 2. [EDA](#2)\n \n    2.1 [Data Load](#2.1)\n   \n    2.2 [Visualization](#2.2)\n    \n    2.3 [Feature Engineering](#2.3)\n    \n    2.4 [Pre-Processing](#2.4)\n    \n       2.4.1     [Missing Values](#2.4.1)\n       \n       2.4.2     [Scaling](#2.4.2)\n       \n       2.4.3     [Outliers](#2.4.3)\n    \n\n 3. [Model Training](#3)\n \n     3.1 [Linear Models](#3.1)\n     \n     3.2 [Sklearn Tree's](#3.2)\n       \n       3.2.1     [GridSearch](#3.2.1)\n      \n     3.3 [LightGBM Tree's](#3.3)\n       \n       3.3.1     [Bayesian Optimization](#3.3.1)\n       \n       3.3.2     [Feature Importance](#3.3.2)\n     \n     3.4 [Models without Outliers](#3.4)\n   \n 4. [Conclusion](#4)"},{"metadata":{"_uuid":"278f1b54742de728cfa2de2f8d6c39611ecc0427"},"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n# 1. Introduction\n<a id=\"1.1\"></a> <br>\n## 1.1 Elo\n\n### - Competition description: \n\n> Imagine being hungry in an unfamiliar part of town and getting restaurant recommendations served up, based on your personal preferences, at just the right moment. The **recommendation** comes with an attached discount from your credit card provider for a local place around the corner!\n> \n> Right now, Elo, one of the largest payment brands in Brazil, has built partnerships with merchants in order to **offer promotions or discounts to cardholders**. But do these promotions work for either the consumer or the merchant? Do customers enjoy their experience? Do merchants see repeat business? Personalization is key.\n> \n> Elo has built machine learning models to understand the most important aspects and preferences in their customers’ lifecycle, from food to shopping. But so far none of them is **specifically tailored** for an individual or profile. This is where you come in.\n> \n> In this competition, Kagglers will **develop algorithms to identify and serve the most relevant opportunities to individuals, by uncovering signal in customer loyalty**. Your input will improve customers’ lives and help Elo reduce unwanted campaigns, to create the right experience for customers.\n\n<a id=\"1.2\"></a> <br>\n## 1.2 Problem Statement\n\n\n\n### - What is Loyalty?:\n\n According to the 'Data_Dictionary':  \n\n- \"*Loyalty is a **numerical score calculated 2 months after historical and evaluation period***\".\n\n'Valeria Elo' (Topic Author)_\n\n- *\"This competition is not a traditional collaborative filtering/recommendation system challenge since we don’t have a direct definition of what is great merchant or experience. Despite that, we want to **link historical information of merchants visits and purchases to a loyalty score**, which is basically a **business metric** that **considered both future spending and retention** as main components. We expect you to link this information in order to find **what triggers higher loyalty scores** and you can use any kind of modeling, including recommendation system, but not exclusively. You can be as creative as you want!\"*\n\n- By looking at 'historical_transactions.csv' and 'new_merchant_transactions.csv,' we can see that the historical transactions are those occurred **before the \"reference date\"** and new merchant transactions the ones that occurred **after the reference date** (\"month lag to reference date\").\n\n### - Our understanding of the situation:\n\n - Based on the data in historical_transactions.csv, Elo picked new merchants to recommend for each card holder.\n - The date when Elo began providing recommentations is called the 'reference date'.\n - The recommended merchant data is not provided (so we can not figure out the recommendation algorithm Elo uses).\n - After the reference date, for each card ID Elo gathered transaction history for all new merchants that appeared on the card.\n - By comparing each card's new merchant activity and the secret list of the merchants recommended by Elo, the loyalty score was calculated.\n \n \n### - Goal:\n Evaluate Elo's recommendation algorithm by **predicting the loyalty score** and finding out which features are correlated to the loyalty score.\n"},{"metadata":{"_uuid":"a605e76aac535217e42d3465b0fc5afdf0b203a6"},"cell_type":"markdown","source":"### - Procedure: \nWe were choosing the following procedure in order to accomplish our goal. The illustration does not show the selection of the right parameters for each model (Hyperparameter Tuning), which turned out to be crucial for the success of our models. We focus on this in chapters 3.2.1 and 3.3.1.\n\n![](https://i.imgur.com/aNshtKq.png)"},{"metadata":{"_uuid":"c88712a18a33b52f5b11539f8b877b9097725356"},"cell_type":"markdown","source":"<a id=\"2\"></a> <br>\n## 2. EDA"},{"metadata":{"_uuid":"a7174d68d8df9d53837177dcf62e54b49bbce47f"},"cell_type":"markdown","source":"<a id=\"2.1\"></a> <br>\n## 2.1 Data Load\n\nLoading different libraries and packeges into the notebook. Some modules are imported in the relevant cells to make it more understandable and readable. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"# Importing libraries and packages\nimport numpy as np ; import pandas as pd \nimport matplotlib.pyplot as plt ; import seaborn as sns; import graphviz\nfrom matplotlib.pyplot import figure;\nimport matplotlib.ticker as ticker\nimport warnings ;import time ;import sys ;import datetime;import gc; warnings.simplefilter(action='ignore')\nimport plotly.offline as py;py.init_notebook_mode(connected=True);import plotly.graph_objs as go; import plotly.tools as tls\n\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.linear_model import BayesianRidge\nfrom sklearn import model_selection, preprocessing, metrics\nfrom sklearn.model_selection import train_test_split, KFold, StratifiedKFold, learning_curve, validation_curve\nfrom sklearn import tree\nfrom sklearn.tree import DecisionTreeRegressor\nimport lightgbm as lgb ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fb8a48aa8bab4d973e4e927d1c5767f449549f66"},"cell_type":"markdown","source":"On of the first necessary steps was it, to challenge the **memory problem**. We reduced the memory usage by this function from Ashish Gupta.  https://www.kaggle.com/roydatascience/elo-stack-interactions-on-categorical-variables\n\nIt basicially changes the data types and helped us to reduce our memory usage by up to 56%."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"# Copied function to reduce memory usage\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1ae7626c90602b0ff9b410239bd4d270c35978b0"},"cell_type":"markdown","source":"Loading the data into the Kernel:"},{"metadata":{"trusted":true,"_uuid":"fd32ef726bcac7a0020599518e51311fcd83a7bf","scrolled":true},"cell_type":"code","source":"%%time\nnew_transactions = pd.read_csv('../input/new_merchant_transactions.csv', parse_dates=['purchase_date'])\nhistorical_transactions = pd.read_csv('../input/historical_transactions.csv', parse_dates=['purchase_date'])\n\norg_train = pd.read_csv('../input/train.csv')\norg_test = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61b9e5a07eac41b66b684cca0782d6e758a402eb"},"cell_type":"markdown","source":"Reducing the memory usage with the predefined funtion."},{"metadata":{"trusted":true,"_uuid":"52e1a72288098af842eef6b75757613ca4ae2c27"},"cell_type":"code","source":"# Reduce memory usage\nhistorical_transactions = reduce_mem_usage(historical_transactions)\nnew_transactions = reduce_mem_usage(new_transactions)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"db1e4d269e612f7019b9c714be8fe47aaeb0348c"},"cell_type":"markdown","source":"<a id=\"2.2\"></a> <br>\n### 2.2 Visualization"},{"metadata":{"trusted":true,"_uuid":"0f59820fe1cd36bc4cfc66681218dfe5a02c9c43"},"cell_type":"code","source":"print('historical_transactions contains ' + str(len(historical_transactions)) + ' transactions ' + 'for ' + str(len(historical_transactions.card_id.unique())) + ' card_ids')\nprint('new_marchants_transactions contains ' + str(len(new_transactions)) + ' transactions ' + 'for ' + str(len(new_transactions.card_id.unique())) + ' card_ids')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4f7c376821caa6e73d26ec3f3311931f981a6f01"},"cell_type":"markdown","source":"Just looking at org_train does not shape an understanding of data, so the next visualizations bring important aspects of that data into focus for further analysis.\n\nFirstly, let's take a look on the distribution of feature_1, feature_2 and feature_3. We can make the conclussion that feature_3 has to be binarized and the best approach in dealing with feature_1 and feature_2 is an implementation of dummies."},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":true,"_uuid":"309b4737f586995e6c421c77de4fb27243723ceb"},"cell_type":"code","source":"fig, ax = plt.subplots(1, 3, figsize = (14, 4));\norg_train['feature_1'].value_counts().sort_index().plot(kind='bar', ax=ax[0], color='#099EE8', title='feature_1', fontsize = 5);\norg_train['feature_2'].value_counts().sort_index().plot(kind='bar', ax=ax[1], color='red', title='feature_2', fontsize = 5);\norg_train['feature_3'].value_counts().sort_index().plot(kind='pie', ax=ax[2], colors = ['#099EE8', '#FFE600'], title='feature_3', explode=[0,0.1], autopct='%1.1f%%',shadow=False)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"0c5407a8d3083e257c75c8c1be15de1d15797de2"},"cell_type":"markdown","source":"What we can also do is to directly plot a correlation matrix. The correlation between different variables in a dataframe is very low, meaning that we are lacking the data in the org_train for accurate predictions."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"219e236591a70b2d2758607a11843ce3380c6eac"},"cell_type":"code","source":"train_corr_matrix = org_train.corr()\n# Generate mask for the upper triangle\nmask = np.zeros_like(train_corr_matrix, dtype=np.bool)\nmask[np.triu_indices_from(mask)] = True\n\nf, ax = plt.subplots(figsize=(10, 5))\ncmap = sns.diverging_palette(10, 255)\nsns.heatmap(train_corr_matrix, mask=mask, cmap=cmap, vmax=.3, center=0,\n            square=True, linewidths=.5, cbar_kws={\"shrink\": .5})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"537cace8e5e065f8fd8a70d94de4058998e6a5a0"},"cell_type":"markdown","source":"Now, lets have a look at both org_train and org_test. The column first_active_month shows the month of the first purchase for a card_id. Basically, it makes sense to compare the number of first purchases in a month for train and test data.\n\nFrom the following graph the next information is discovered:\n\n- Test dataset is highly correlated with the train date having the same distribution\n- Test dataset has approximately twice less of observations"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"3e41c651d627052494e9a05ca8bdde8b263f1e68"},"cell_type":"code","source":"d1 = org_train['first_active_month'].value_counts().sort_index()\nd2 = org_test['first_active_month'].value_counts().sort_index()\ndata = [go.Scatter(x=d1.index, y=d1.values, name='train', mode = 'lines+markers', marker={'color': 'red'}), go.Scatter(x=d2.index, y=d2.values, name='test', mode = 'lines+markers', marker={'color': '#099EE8'})]\nlayout = go.Layout(dict(title = \"Counts of first active\",\n                  xaxis = dict(title = 'Month'),\n                  yaxis = dict(title = 'Count'),\n                  ),legend=dict(\n                orientation=\"v\"))\npy.iplot(dict(data=data, layout=layout))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c53fbbce63075cc5ef50dc546a038b12a574cdf8"},"cell_type":"markdown","source":"Loyalty numerical score is calculated two months after historical and evaluation period. \nLet's plot a histogram of loyalty score to understand what are we dealing with. From the histogram and descriptive statistics summary we observe that the **most of the output variables are flactuating around 0**, however the minimum loyalty score observed in the train dataset is -33.21 and the maximum reached 17.96. In comparison to 2nd quantile (-0.02) that is more robust to outliers, the mean is -0.39."},{"metadata":{"trusted":true,"_uuid":"026186098e5e667aae335fb60252afc7c5c54d89"},"cell_type":"code","source":"# Descriptive statistics summary\nprint(org_train['target'].describe())\n# Histogram\nplt.figure(figsize=(12,8))\nsns.distplot(org_train.target.values, bins=50, color=\"red\", vertical = False, kde=True, kde_kws={\"color\": \"#FFE600\"})\nplt.title(\"Histogram of Loyalty score\", fontsize = 18)\nplt.ylabel('Loyalty score', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5c670e92574a028478d07a42462733f3fd566ff"},"cell_type":"markdown","source":"<a id=\"2.3\"></a> <br>\n### 2.3 Feature Engineering\n\n*“Feature engineering is the process of **transforming raw data into features** that better represent the underlying problem to the predictive models, resulting in improved model accuracy on unseen data.”\n— Dr. Jason Brownlee*\n\nGenrally speaking,  data cleaning can be seen as a process of subtraction and **feature engineering as a process of addition**. Often the feature engineering is  one of the most valuable tasks in order to improve model performance, for three reasons:\n\n- Isolate and highlight key information, which helps the algorithms \"focus\" on what’s important\n- Use domain expertise\n- Understand the \"vocabulary\" of feature engineering, you can bring in other people’s domain expertise!"},{"metadata":{"_uuid":"a528a860e4dd50fbb0a2cd55be0339cd9a63ac7d"},"cell_type":"markdown","source":"The first step of dealing with columns containing dates is to change the date format. There is a method(pandas.to_datetime) in pandas library that converts argument to datetime.\n"},{"metadata":{"trusted":true,"_uuid":"6613c7036006e6efd5837e71ad1a8c3b458e53d2"},"cell_type":"code","source":"# Read files, change date format and calculate 'elapsed_time'\ndef read_data(input_file):\n    df = pd.read_csv(input_file)\n    df['first_active_month'] = pd.to_datetime(df['first_active_month'])\n    df['elapsed_time'] = (datetime.date(2018, 2, 1) - df['first_active_month'].dt.date).dt.days\n    return df\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6290346ddc453ac26e372fc3833176d8132e6db6"},"cell_type":"markdown","source":"\nTaking the first look into data, there are some categorical columns and hence, they have to be binarized. The function below is the first step of dealing with such columns."},{"metadata":{"trusted":true,"_uuid":"98bb7aac81ca432add23404286f4f308ed33236e"},"cell_type":"code","source":"# Binarize categorical values\ndef binarize(df):\n    for col in ['authorized_flag', 'category_1']:\n        df[col] = df[col].map({'Y':1, 'N':0})\n    return df\n\norg_train = read_data('../input/train.csv')\norg_test = read_data('../input/test.csv')\n\nhistorical_transactions = binarize(historical_transactions)\nnew_transactions = binarize(new_transactions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4e02915c4479457e626eca181aba674c40044d1d"},"cell_type":"code","source":"# Dummy encoding for catogories\nhistorical_transactions = pd.get_dummies(historical_transactions, columns=['category_2', 'category_3'])\nnew_transactions = pd.get_dummies(new_transactions, columns=['category_2', 'category_3'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5e6a01e6ed3067788b1c23d986873fb090beb4a7"},"cell_type":"code","source":"# Purchase_month format to month\nhistorical_transactions['purchase_month'] = historical_transactions['purchase_date'].dt.month\nnew_transactions['purchase_month'] = new_transactions['purchase_date'].dt.month","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"316970db8921bb81069d4eeefd31ef21b75ebcee"},"cell_type":"markdown","source":"The aggregation of the different transcations for every 'card_id' turned out to be crucial to the success of the project. As seen in the procedure in the beginning of our project, we changed the aggregation of the different rows many times after we evaluated the model. \n\nAs you will see in the correlation analysis of our features, tryining to find features which are really **correlating to the target**, appeared almost as an unsolvable task for us. What really helped us to find valuable features, was the given 'feature_importance' by the LightGBM trees at the end of the modelling chapter."},{"metadata":{"trusted":true,"_uuid":"7140ade520ddfc8e14e73611f025005b9fe7a49d"},"cell_type":"code","source":"#'nunique' -> Returns number of unique elements in the group\ndef aggregate_data(transactions):\n    transactions.loc[:, 'purchase_date'] = pd.DatetimeIndex(transactions['purchase_date']).\\\n                                      astype(np.int64) * 1e-9\n    \n    # Aggregate columns by:\n    aggregations = {\n        'category_1': ['sum', 'mean'],'category_2_1.0': ['mean'],'category_2_2.0': ['mean'],'category_2_3.0': ['mean'],\n        'category_2_4.0': ['mean'],'category_2_5.0': ['mean'],'category_3_A': ['mean'],'category_3_B': ['mean'],'category_3_C': ['mean'],\n        'merchant_id': ['nunique'],'merchant_category_id': ['nunique'],'state_id': ['nunique'],'city_id': ['nunique'],'subsector_id': ['nunique'],\n        'purchase_amount': ['sum', 'mean', 'max', 'min', 'std'], 'installments': ['sum', 'mean', 'max', 'min', 'std'],\n        'purchase_month': ['mean', 'max', 'min', 'std'], 'purchase_date': [np.ptp, 'min', 'max'],\n        'month_lag': ['min', 'max']\n        }\n    \n    # Group rows by 'card_id'\n    agg_transactions = transactions.groupby(['card_id']).agg(aggregations)\n    agg_transactions.columns = ['_'.join(col).strip() for col in agg_transactions.columns.values]\n    agg_transactions.reset_index(inplace=True)\n    \n    #\n    data = (transactions.groupby('card_id').size().reset_index(name='transactions_count'))\n    agg_transactions = pd.merge(data, agg_transactions, on='card_id', how='left')\n    return agg_transactions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"53a268dafea97f4d9d3c1781fae87266ee496f86"},"cell_type":"code","source":"# Aggregate historical_transactions\nhist_agg = aggregate_data(historical_transactions)\nhist_agg.columns = ['hist_' + c if c != 'card_id' else c for c in hist_agg.columns]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"57c9fbdb3f34eebe91aa040313fd90bc807208df"},"cell_type":"code","source":"# Aggregate new_transactions\nnew_agg = aggregate_data(new_transactions)\nnew_agg.columns = ['hist_' + c if c != 'card_id' else c for c in new_agg.columns]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"834a97bbf99fae20ba7c794cabea4331365e2a1e"},"cell_type":"markdown","source":"Finally we choosed a 'left-join' to merge every unique 'card_id' with its aggregated historical and new transactions. As we can see, that reduced the number of rows in our final_train set down to 0.01%."},{"metadata":{"trusted":true,"_uuid":"dc22345d42d6bf3d4731ea851c1e07c213ce9b25","scrolled":false},"cell_type":"code","source":"# Join with train- and test-dataset\nfinal_train = pd.merge(org_train, hist_agg, on='card_id', how='left')\nfinal_test = pd.merge(org_test, hist_agg, on='card_id', how='left')\n\nfinal_train = pd.merge(final_train, new_agg, on='card_id', how='left')\nfinal_test = pd.merge(final_test, new_agg, on='card_id', how='left')\n\nprint(\"Train original:\", org_train.shape); print(\"Test original:\", org_test.shape)\nprint(\"Historical Transactions:\", historical_transactions.shape); print(\"New Transactions:\", new_transactions.shape); \nprint(\"Test merged:\", final_train.shape); print(\"Train merged:\", final_test.shape)\nprint(\"Number of Rows through Aggregation  {:0.2f}%\".format(final_train.shape[0]/(historical_transactions.shape[0] + new_transactions.shape[0])))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"47de9a6267f10bf9a0ca35acd210cc33eba24619"},"cell_type":"markdown","source":"That leads us to our final data sets to work with. We created one complete data set ( 'Full load' ) and a sample data set ( 'Data to work with') to reduce computation times and make working more convinient."},{"metadata":{"trusted":true,"_uuid":"86c2a29b3f81c8d0c068f1caf5469674ecaff3f5"},"cell_type":"code","source":"# Get features\nfeatures = [c for c in final_train.columns if c not in ['card_id', 'first_active_month','target']]\ncategorical_feats = [c for c in features if 'feature_' in c]\n\n# Full load\ntarget= final_train.target\ntrain = final_train[features]\ntest = final_test[features]\n\n# Data to work with \nwork_target=final_train.target\nwork_data=final_train.sample(n=10000)\nwork_train=final_train[features]\n\n# Training and test-dataset\nX_train,X_test,y_train,y_test=train_test_split(work_train,work_target,test_size=0.33,random_state=42)\nprint(\"X_train:\",X_train.shape,\" y_train:\",y_train.shape,\" X_test:\",X_test.shape,\" y_test:\",y_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4fac2ffd7ed4bd78ab16630517b02c1fc07453fd"},"cell_type":"markdown","source":"Time to have a look an the correlation of those new features:"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"b70d254a3b635973fec72122be30e63edd99336e"},"cell_type":"code","source":"# Compute correlation matrix\ntrain_corr = final_train.corr()\n\n# Generate mask for the upper triangle\nmask = np.zeros_like(train_corr, dtype=np.bool)\nmask[np.triu_indices_from(mask)] = True\n\nf, ax = plt.subplots(figsize=(11, 9))\ncmap = sns.diverging_palette(220, 10, as_cmap=True)\nsns.heatmap(train_corr, mask=mask, cmap=cmap, vmax=.3, center=0,\n            square=True, linewidths=.5, cbar_kws={\"shrink\": .5})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7cfa8b774742ce7b1d90ad0d820760b9d7b78fd1"},"cell_type":"markdown","source":"To make it a little more clear, we choosed the top 10 featueres with the highest correlation to the target. Unfortunately the observed correlations are super low. "},{"metadata":{"trusted":true,"_uuid":"516474996f501298807c2d3204c590f1da8cf741"},"cell_type":"code","source":"\n\n# Get correlation of features to target\ncorr_matr=final_train.corr().abs().sort_values(by=['target'],ascending=False)['target'];\ncorr_matr.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eefd7dc78e3816cd678f079567a0ac406fd711cb"},"cell_type":"markdown","source":"<a id=\"2.4\"></a> <br>\n### 2.4 Pre-Processing"},{"metadata":{"_uuid":"f42640fd5d628e7036c47dd2139283f81ed554c9"},"cell_type":"markdown","source":"To train and validate our models in the following chapters we had to do some preprocessing. We choosed three different approaches and created various of our training data for each method:\n- Filling NaN's with the median value - > Sklearn trees do not except missing values\n- Scaling the data -> turned out to be useless, as the theory suggested \n- Removing the rows with outliers ( target value < -31 (around 10 times the standard deviation of the target))\n     -> Gave us the most exciting insights "},{"metadata":{"_uuid":"3ba798815ee61a8867d9da59e4b194f7e6d1edcb"},"cell_type":"markdown","source":"<a id=\"2.4.1\"></a> <br>\n### 2.4.1 Missing Values"},{"metadata":{"trusted":true,"_uuid":"964a5940923e38360f5d436d611ade82ce5a78c4"},"cell_type":"code","source":"# checkin data set for NaN's\nX_train.isna().sum().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ec7cb2033eb4fb1c40ecccda03ed80dfefa08845","scrolled":false},"cell_type":"code","source":"# Fill NaN's with median value\nfrom sklearn.impute import SimpleImputer\nimputer=SimpleImputer(strategy='median')\nimputer.fit(X_train)\n\n# For training data\nX_train_numpy=imputer.transform(X_train)\nX_train_filled=pd.DataFrame(X_train_numpy)\n\n# For test data\nX_test_numpy=imputer.transform(X_test)\nX_test_filled=pd.DataFrame(X_test_numpy)\n\n# Checkin data set for NaN's\nX_train_filled.isna().sum().sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"26585fc194d44d0ed225ff221f0a05df143405f0"},"cell_type":"markdown","source":"<a id=\"2.4.2\"></a> <br>\n### 2.4.2 Scaling"},{"metadata":{"trusted":true,"_uuid":"f43da07e65585bf098c947233f2cf6ab3ac73c2e"},"cell_type":"code","source":"# Creating scaled data sets\nfrom sklearn.preprocessing import MinMaxScaler\nscaler = MinMaxScaler(); train_scale=X_train; test_scale=X_test\nX_train_scaled = scaler.fit_transform(train_scale)\nX_train_scaled = pd.DataFrame(X_train_scaled,columns=features)\nX_test_scaled = scaler.fit_transform(test_scale)\nX_test_scaled = pd.DataFrame(X_test_scaled, columns=features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"c1f33fdf894d666c8d13efd51a72d4e0898a1e59"},"cell_type":"code","source":"# Some models required a multiclass label instead of a continous one\nfrom sklearn import utils\nlab_enc = preprocessing.LabelEncoder()\ny_train_encoded = lab_enc.fit_transform(y_train)\ny_test_encoded = lab_enc.fit_transform(y_test)\nprint(utils.multiclass.type_of_target(y_train))\nprint(utils.multiclass.type_of_target(y_train_encoded))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6807ea4f67c962dad2630f493169b05df5ea7ed4"},"cell_type":"markdown","source":"<a id=\"2.4.3\"></a> <br>\n### 2.4.3 Outliers"},{"metadata":{"_uuid":"fdf38b7a013b7ad510648db0d318beb4b1712ca7"},"cell_type":"markdown","source":"Based on the visualisation of Loyalty score, **1% of outliers** was identified. Let's try now to train the model on data without outliers and compare the results."},{"metadata":{"trusted":true,"_uuid":"805320ab1b5ddc7f0bd729f58155ca6f7fe941a7"},"cell_type":"code","source":"org_train = pd.read_csv('../input/train.csv')\norg_test = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b0c263e63ac6ce3373208918094eea4d3a22dd1c"},"cell_type":"code","source":"# Adding a column to classify outliers\nidentify_outlier_train = org_train.assign(outliers = org_train.target)\nidentify_outlier_train.shape\n\nidentify_outlier_train.loc[identify_outlier_train.target < -31, 'outliers'] = 1\nidentify_outlier_train.loc[identify_outlier_train.target > -31, 'outliers'] = 0\n\n# Only outliers\ndirty_train = identify_outlier_train[identify_outlier_train.outliers == 1]\n# Without outliers\nclean_train = identify_outlier_train[identify_outlier_train.outliers == 0]\n\n# Training and test data without outliers\nclean_train = clean_train.drop(['first_active_month','card_id'],axis=1)\nclean_target = clean_train['target']\nX_train_clean,X_test_clean,y_train_clean,y_test_clean=train_test_split(clean_train,clean_target,test_size=0.33,random_state=42)\n\nprint('Number of outliers in train data is: ' + str(len(dirty_train)))\nprint('Number of non-outliers in train data is: ' + str(len(clean_train)))\nprint('Proportion of outliers in train dataset is {:0.2%}'.format((len(dirty_train)/len(identify_outlier_train))))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6ce11a6fe0cfa5d02e24c4d96f9272334498c8ba"},"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n## 3. Model Training"},{"metadata":{"_uuid":"db9f8e91a1bf0e43fb2f5d4c7c627fb1783fe576"},"cell_type":"markdown","source":"#### Evaluation of Models\n\nIn order to measure the quality of fit  and to evaluate the models in the following chapter, we defined the root-mean-square-error (**RMSE**). \nAccording to the ISLR the (R)MSE is small if the predicted response is close to the true responses and large if there is a substantial difference between the predicted and the true response.\n\nTo compare our results we created a dummy-model, which is predicting the average of the giving targets by the training set. With an RMSE of 3.875 the is only 6.5% worse than the leading score on the competitions leaderbord (Team Name: senkin 13, Score: 3.637, Date 28.01.2019)."},{"metadata":{"trusted":true,"_uuid":"20ff7d181778b4b0c1623a2ee4fb37e5f2276de6"},"cell_type":"code","source":"# define function to calculate RMSE\ndef rmse(y_test,prediction):\n    rmse=np.sqrt(mean_squared_error(y_test,prediction))\n    return rmse;","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"26a674821a7448666251933e1b79140a61eded07"},"cell_type":"code","source":"# create a dummy prediciton with the average, in order to compare the model to the most simplest one\ny_dummy=pd.Series(data=np.full((y_test.shape),np.average(target)))\nrmse_dummy=rmse(y_test,y_dummy)\nprint('Dummy-Model RMSE: {:0.2f}'.format(rmse_dummy))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ca159351a5855754c359fdf6a639fe326c363238"},"cell_type":"code","source":"# define function in order to calculate RMSE and Improvement of Model\ndef evaluation(model,X_train=X_train,y_train=y_train,X_test=X_test,y_test=y_test):\n    \"\"\"\n   This function evaluates the model, calculates the RMSE and measures its imporvement in comparison to the dummy-prediction.\n   If no specific Training and Test-set is given, it defaults to X_train,y_train,X_test and y_test.\n   \"\"\"\n    model.fit(X_train,y_train)\n    model_pred=model.predict(X_test)\n    rmse_new=rmse(y_test,model_pred)\n    #print(\"Number of features used:\",model.coef_!=0 )\n    print(\"Model:\",model.__class__.__name__,\";\",'RMSE: {:0.2f}'.format(rmse_new),\";\",'Improvement of: {:0.2f}%'.format(100 * (rmse_dummy- rmse_new) / rmse_new))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"33d19590965821a1c6c2aea07b0cae4bc98306be"},"cell_type":"code","source":"# define RMSE for sklearn-models\nfrom sklearn.metrics import make_scorer\nrmse_scorer = make_scorer(rmse, greater_is_better=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c57ac96dae06e598ae4394b6cb970338e90c798"},"cell_type":"markdown","source":"<a id=\"3.1\"></a> <br>\n## 3.1 Linear Models"},{"metadata":{"_uuid":"eb291713f2dad273d2a5358283d8b416f7248ffd"},"cell_type":"markdown","source":"As a first approach to predict the loyalty score for the elo customers, we choosed Linear Models. Sklearn provides a wide range of linear models, which are easy to implement. In comparison to the usual LinearRegression, we tested the Lasso, Ridge and ElasticNet Regressor. Since we have a wide range of features, our motivation was it to use L1/L2 regularization in order select the optimal features. In the following the main charaktertics of the models are summed up:\n\n**Lasso (L1):**\n- L1 regularization adds a penalty equal to the sum of the absolute value of the coefficients\n- Shrinks some parameters to zero. Hence some variables will not play any role in the model, L1 regression can be seen as a way to select features in a model. \n- The optimization objective for Lasso is: *(1 / (2 * n_samples)) * ||y - Xw||^2_2 + alpha * ||w||_1*\n- alpha : float, optional \"Constant that multiplies the L1 term. Defaults to 1.0. alpha = 0 is equivalent to an ordinary least square, solved by the LinearRegression object.\" (Source: sklean)\n\n**Ridge (L2):**\n- L2 regularization adds a penalty equal to the sum of the squared value of the coefficients\n- Forces the parameters to be relatively small, the bigger the penalization, the smaller (and the more robust) the coefficients are.\n- Minimizes the objective function: *||y - Xw||^2_2 + alpha * ||w||^2_2*\n- alpha : {float, array-like}, shape (n_targets): Regularization strength; must be a positive float. Regularization improves the conditioning of the problem and reduces the variance of the estimates. (Source: sklearn)\n\n**Elastic Net (L1/L2):**\n- Linear regression with combined L1 and L2 priors as regularizer.\n- Minimizes the objective function: *1 / (2 * n_samples) * ||y - Xw||^2_2 + alpha * l1_ratio * ||w||_1 + 0.5 * alpha * (1 - l1_ratio) * ||w||^2_2*\n- alpha = a + b and l1_ratio = a / (a + b). Specifically, l1_ratio = 1 is the lasso penalty. Currently, l1_ratio <= 0.01 is not reliable, unless you supply your own sequence of alpha (Source: sklearn)"},{"metadata":{"trusted":true,"_uuid":"19ceb9ea5a14f8e851be1ba2134acc0dbd44d6f5"},"cell_type":"code","source":"# Testing Linear Models on Dataset\n# https://towardsdatascience.com/ridge-and-lasso-regression-a-complete-guide-with-python-scikit-learn-e20e34bcbf0b\nfrom sklearn.linear_model import LinearRegression, LogisticRegression, Lasso, Ridge, ElasticNet\n\nlinReg=LinearRegression()\nlogReg=LogisticRegression()\nlasso=Lasso(alpha=0.9)\nridge=Ridge(alpha=0.9) # higher the alpha value, more restriction on the coefficients; \n                       # low alpha > more generalization, coefficients are barely\nelasNet=ElasticNet(0.5)\n\nfor reg in (linReg,lasso,ridge,elasNet):\n    evaluation(reg,X_train=X_train_filled,X_test=X_test_filled);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"56d3d99a6066f0d9d575d150fc6b9beab28f4ab5"},"cell_type":"code","source":"# add significane test\nimport statsmodels.api as sm\nimport statsmodels.formula.api as smf\n\nest = smf_linReg = smf.ols('target ~ feature_1 + feature_2 + feature_3', final_train).fit()\nest.summary().tables[1]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a91ce4971634ac89bee3e2d0c0ba01f3e0564694"},"cell_type":"markdown","source":"As seen in the results, the linear models did not really work on the given data set. We did not expect something else, because we do not have any real correlation between the target and the features."},{"metadata":{"_uuid":"0d05bc707cc8982d27b07ec84a32b9b0b2eea14f"},"cell_type":"markdown","source":"<a id=\"3.2\"></a> <br>\n## 3.2 Decision Trees"},{"metadata":{"_uuid":"665ad76165a0ff564d7d0868633348fb2b799376"},"cell_type":"markdown","source":"After the linear models were quite unsuccesfull we decided to go for **decision Trees**. Firsty we trained our data sets on the sklearn trees, later on we tried our luck with the **LighGBM**. Decision Tree's in general bring the following advantages with them:\n\n- Decision trees implicitly perform variable screening or **feature selection**. When we fit a decision tree to a training dataset, the top few nodes on which the tree is split are essentially the most important variables within the dataset and feature selection is completed automatically.\n\n- Decision trees require relatively little effort from users for data preparation. The Decision trees structure remains the same, with or without transformation. Also missing values will not prevent splitting the data for building trees. Decision trees are also **not sensitive to outliers** since the splitting happens based on proportion of samples within the split ranges and not on absolute values.\n\n- **Nonlinear relationships** between parameters do not affect tree performance. As mentioned earlier, highly nonlinear relationships between variables will result in failing checks for simple regression models and thus make such models invalid. Decision trees do not require any assumptions of linearity in the data. \n\n- And after all: Decision trees are **easy to interpret** and explain to executives!\n\n\nSource: http://www.simafore.com/blog/bid/62333/4-key-advantages-of-using-decision-trees-for-predictive-analytics"},{"metadata":{"_uuid":"32ad15f8a3c98fca6234bb3d9594e0f595a08f96"},"cell_type":"markdown","source":"#### Decision Tree"},{"metadata":{"trusted":true,"_uuid":"fb091e4f34f6fb39aeb0b2afbeaa152e78dcb9ba","scrolled":true},"cell_type":"code","source":"# Training Decision Tree and first Evaluation of Performance\ntree1 = DecisionTreeRegressor(max_depth=3)\nevaluation(tree1, X_train = X_train_filled, X_test = X_test_filled)\n\n# Visualize simple decision tree\ndot_tree = tree.export_graphviz(tree1, out_file=None)\ntree1_graph = graphviz.Source(dot_tree); tree1_graph","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a360e3cb4fa1da2999965cc7b558c717434d6a2"},"cell_type":"markdown","source":"#### Decision Tree Validation Curve\n\nIn the following we plotted a **validation curve**, which is showing the performance of a decision tree on the training- and on the test datas et.\nWhile the training score keeps improving, the test score decreases after a while. This is due to **overfitting**. A highly complex tree (deep tree) is not as able anymore to **generalize** on unknown Test data."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"f9f31f23ab892f046d17e70238b6d9f30887baeb"},"cell_type":"code","source":"# Evaluation Training- and Test Error\n# Create CV training and test scores for various training set sizes\ndepth_range = np.linspace(1, 10, 10) # Range of depth for Decison Tree's\n\ntrain_scores,test_scores = validation_curve(DecisionTreeRegressor(),X_train_filled, y_train,\n                                            param_name =\"max_depth\", param_range = depth_range, cv=3 ,\n                                            scoring=rmse_scorer, n_jobs=-1)\n\n# Create means and standard deviations of training set scores\ntrain_scores_mean = np.mean(train_scores, axis=1); test_scores_mean = np.mean(test_scores, axis=1)\n\n# Calculating optimal regularization parameter\ni_depth_range_optim = np.argmax(test_scores_mean)\ndepth_range_optim = depth_range[i_depth_range_optim]\nprint(\"Optimal regularization parameter : %s\" % depth_range_optim )\ntree2=DecisionTreeRegressor(max_depth=depth_range_optim); evaluation(tree2,X_train=X_train_filled,X_test=X_test_filled)\n\n# Plotting the Validation Curve\nfigure(num=None, figsize=(12, 8), dpi=80, facecolor='w', edgecolor='k')\nplt.title(\"Validation Curve with DecisionTreeRegressor - Complexity\");plt.xlabel(\"Complexity/max_depth\");plt.ylabel(\"RMSE\")\nplt.plot(depth_range,train_scores_mean,  label=\"Training score\",linewidth=4,color='blue') ;\nplt.plot(depth_range,test_scores_mean, label=\"Cross-validation score\",linewidth=4,color='red')\nplt.vlines(depth_range_optim, plt.ylim()[0], np.max(test_scores_mean), color='k',linewidth=2,linestyles='--',label='Optimum on test')\nplt.legend(loc=\"best\"); plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc9519c82361677b49396f9cf2762f6c5539d9dc"},"cell_type":"markdown","source":"** Decision Tree Learning Curve**\n\nAfter the evaluation of the optimum depth for tree, we tried to find out how much data the tree actually needs to perform. As seen below, the performance of the DecisionTreeRegressor does not really improve with an increasing trainind data size. "},{"metadata":{"trusted":true,"_uuid":"ba2c63dda64164d2430e5370566672130df97130"},"cell_type":"code","source":"# Evaluation Training Size\n# Create CV training and test scores for various training set sizes\ntrain_sizes, train_scores, test_scores = learning_curve(DecisionTreeRegressor(max_depth=3), \n                                                        X_train_filled, y_train,cv=3,scoring=rmse_scorer, n_jobs=-1, \n                                                        # 20 different sizes of the training set\n                                                        train_sizes=np.linspace(0.01, 1.0, 20))\n\n# Create means and standard deviations of training set scores\ntrain_mean = np.mean(train_scores, axis=1)\ntrain_std = np.std(train_scores, axis=1)\n# Create means and standard deviations of test set scores\ntest_mean = np.mean(test_scores, axis=1)\ntest_std = np.std(test_scores, axis=1)\n\n# Draw lines\nfigure(num=None, figsize=(12, 8), dpi=80, facecolor='w', edgecolor='k')\nplt.plot(train_sizes, train_mean, '--', color='blue',  label=\"Training score\")\nplt.plot(train_sizes, test_mean, color= 'red', label=\"Cross-validation score\")\n# Draw bands\nplt.fill_between(train_sizes, train_mean - train_std, train_mean + train_std, color=\"#DDDDDD\")\nplt.fill_between(train_sizes, test_mean - test_std, test_mean + test_std, color=\"#DDDDDD\")\n# Create plot -> change y-label\nplt.title(\"Learning Curve with DecisionTreeRegressor - Training Size\")\n#y_labels = ax.get_yticks()\n#ax.yaxis.set_major_formatter(ticker.)\nplt.xlabel(\"Training Set Size\"), plt.ylabel(\"RMSE\"), plt.legend(loc=\"best\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b00bc4fa4fce8b5ca1f0e1acaf956ed42f81ead2"},"cell_type":"markdown","source":"### Radom Forrest\nAfter modelling single tree's, the next logical step is the **RandomForrestRegressor**. This one builds an **esemble** of decision trees, which are training with the **bagging** method. The idea of bagging is, that a combination of many so called **\"weak learners\"** increases the final result.  \nFurthermore a Random Forest adds  randomness to the model, while growing the trees. Instead of searching for the most important feature while splitting a node, it searches for the best feature among a **random subset of features**. This results in a wide diversity that generally results in a better model.\n\nSource: https://towardsdatascience.com/the-random-forest-algorithm-d457d499ffcd"},{"metadata":{"trusted":true,"_uuid":"89d18fefa39989adaad2cfc75c09fcaceabb8e28","scrolled":true},"cell_type":"code","source":"# Create first random forrest \nfrom sklearn.ensemble import RandomForestRegressor\n\nranFor1 = RandomForestRegressor(max_depth=3, random_state=0,\n                            n_estimators=20)\nevaluation(ranFor1,X_train=X_train_filled,X_test=X_test_filled)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"48fd47f34c4b0d109e3a5d4ad214fabf05922720"},"cell_type":"markdown","source":"**RandomForrest Validation Curve**\n\nSame like we evaluated the depth for single tree, we analyzed the number of trees for the RandomForrest."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"3cbc1bd2dfc55f73e3c045f0e367b68bcc6c8194"},"cell_type":"code","source":"# Evaluation of n_estimators = n_trees\n# Create CV training and test scores for various training set sizes\nestimator_range = np.arange(2, 40, 2)\ntrain_scores,test_scores = validation_curve(RandomForestRegressor(max_depth=3),X_train_filled, y_train,\n                                                          param_name=\"n_estimators\", param_range=estimator_range,\n                                                          cv=3,scoring=rmse_scorer, n_jobs=-1)\n\n# Create means and standard deviations of training set scores\ntrain_scores_mean = np.mean(train_scores, axis=1)\ntrain_scores_std = np.std(train_scores, axis=1)\ntest_scores_mean = np.mean(test_scores, axis=1)\ntest_scores_std = np.std(test_scores, axis=1)\n\ni_estimator_range_optim = np.argmax(test_scores_mean)\nestimator_range_optim = estimator_range[i_estimator_range_optim]\nprint(\"Optimal regularization parameter :\",estimator_range_optim )\nranFor2=RandomForestRegressor(n_estimators=estimator_range_optim)\nevaluation(ranFor2,X_train=X_train_filled,X_test=X_test_filled)\n\nfigure(num=None, figsize=(8, 6), dpi=80, facecolor='w', edgecolor='k')\nplt.title(\"Validation Curve with RandomForestRegressor\");plt.xlabel(\"Number of Trees\");plt.ylabel(\"RMSE\")\nplt.plot(estimator_range,train_scores_mean,  label=\"Training score\",linewidth=4,color='blue') ;\nplt.plot(estimator_range,test_scores_mean, label=\"Cross-validation score\",linewidth=4,color='red')\nplt.vlines(depth_range_optim, plt.ylim()[0], np.max(test_scores_mean), color='k',linewidth=2,linestyles='--',label='Optimum on test')\nplt.legend(loc=\"best\"); plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"74e1d5c4df0785378e82052827dc17de08d9182d"},"cell_type":"markdown","source":"<a id=\"3.2.1\"></a> <br>\n### 3.2.1 GridSearch"},{"metadata":{"_uuid":"72d1637ec1d54e4e864700df35ac3359f0f61cce"},"cell_type":"markdown","source":"### HyperParameter Tuning\n\nIn the following we optimized the **hyperparamter** of the RandomForrest with **GridSearch**. The best way to think about hyperparameters is like the settings of an algorithm that can be adjusted to optimize performance. While model parameters are learned during training , such as the slope and intercept in a linear regression ,  hyperparameters must be **set before training**. In the case of a random forest the following ones are critical:\n\n- **n_estimators** = number of trees in the foreset\n- **max_features** = max number of features considered for splitting a node\n- **max_depth** = max number of levels in each decision tree\n- **min_samples_split** = min number of data points placed in a node before the node is split\n- **min_samples_leaf** = min number of data points allowed in a leaf node\n- **bootstrap** = method for sampling data points (with or without replacement)\n\nWe used GridSearch to find the optimum parameters for RandomForrest."},{"metadata":{"trusted":true,"_uuid":"3801020eea119de93253684111546f4299bc3261","scrolled":false},"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\nranFor3=RandomForestRegressor()\n\n# Create the parameter grid based on the results of random search \nparam_grid_ranFor = {\n    'bootstrap': [True], 'max_depth': [2,8], 'max_features': [2,10],\n    'min_samples_leaf': [2,20], 'min_samples_split': [4,20], 'n_estimators': [10,20]\n}\n# Instantiate the grid search model\ngrid_search_ranFor = GridSearchCV(estimator = ranFor3, param_grid = param_grid_ranFor, \n                                  cv = 3, n_jobs = -1, verbose = 2, return_train_score=True)\n\n# Fit the grid search to the data\ngrid_search_ranFor.fit(X_train_filled,y_train);\n\n# Best Parameters of GridSearch\nbest_param_ranFor=grid_search_ranFor.best_params_ ; print(best_param_ranFor)\n\n# Evaluation of RandomForrest with GirdSearch Parameters\nranFor3 = RandomForestRegressor(**best_param_ranFor) ; evaluation(ranFor3,X_train=X_train_filled,X_test=X_test_filled)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd3c7cd6cd87cfb67bdfad67a84e7ed7811aaa4b"},"cell_type":"markdown","source":"Unfortunately the even an simple GridSearch took several, even for only a small part of the data set and for a small number of parameters. \nFor that reason we decided to do the GridSearch ones for demonstration (with only a small grid of parameters)  and then focus on the LightGBM Models which turned out to be way for efficient."},{"metadata":{"_uuid":"2db0c6d27454a117784709a1fd3a79e0b1912ff2"},"cell_type":"markdown","source":"<a id=\"3.3\"></a> <br>\n## 3.3 LightGBM"},{"metadata":{"_uuid":"5d2fe547f87d8b5eb18407fdb10fde0f5b3942c4"},"cell_type":"markdown","source":"What is Light GBM?\n- According to LightGBM'S documentation: \"LightGBM is a **gradient boosting framework** that uses tree based learning algorithms. It is designed to be distributed and efficient with the following advantages:\n- Faster training speed and higher efficiency.\n- Lower memory usage.\n- Better accuracy.\n- Support of parallel and GPU learning.\n- Capable of handling large-scale data.\"\n\nHow it differs from other tree based algorithm?\n- Light GBM **grows tree vertically/ leaf-wise** while other algorithm grows trees **horizontally/ leaf-wise** . It will choose the leaf with max delta loss to grow. When growing the same leaf, Leaf-wise algorithm can reduce more loss than a level-wise algorithm. \n\n\n![](https://i.imgur.com/kLlL7YF.png)\n\n( Source: http://mlexplained.com/2018/01/05/lightgbm-and-xgboost-explained/)\n\nLevel-wise training can be seen as a form of regularized training since leaf-wise training can construct any tree that level-wise training can. Therefore, leaf-wise training is more **prone to overfitting but is more flexible**. This makes it a better choice for large and wide datasets.\n\nhttps://medium.com/@pushkarmandot/https-medium-com-pushkarmandot-what-is-lightgbm-how-to-implement-it-how-to-fine-tune-the-parameters-60347819b7fc"},{"metadata":{"_uuid":"b5cd60c92e6847481a41278bac367cb6e71e8ed9"},"cell_type":"markdown","source":"### Parameters LightGBM\nLightGBM's documentation differs between learning control and core paremters:\n\n**Learning Control Parameters**\n- **max_depth**: Describes the maximum depth of tree. Used to handle model overfitting. \n- **min_data_in_leaf**: Minimum number of the records a leaf may have. The default value is 20, optimum value. It is also used to deal overfitting.\n- **feature_fraction**: % of parameters selected randomly in each iteration for building trees.\n- **bagging_fraction**: Specifies the fraction of data to be used for each iteration and is generally used to speed up the training and avoid overfitting.\n- **early_stopping_round**: Stops training if one metric of one validation data doesn’t improve in last - early_stopping_round rounds. \n- **lambda**: Specifies regularization. Typical value ranges from 0 to 1.\n- **min_gain_to_split**: Describe the minimum gain to make a split. It can used to control number of useful splits in tree."},{"metadata":{"_uuid":"67778c3b916112887b21f35eff8f5cade89ed3da"},"cell_type":"markdown","source":"**Core Parameters**\n- **task**: Specifies the task you want to perform on data. It may be either train or predict.\n- **application**: Specifies the application of your model, whether it is a regression problem or classification problem. \n- **boosting**: Defines the type of algorithm you want to run, default=gdbt\n  - gbdt: traditional Gradient Boosting Decision Tree\n  - rf: random forest\n - **num_boost_round**: Number of boosting iterations, typically 100+\n- **learning_rate**: Determines the impact of each tree on the final outcome. GBM works by starting with an initial estimate which is updated using the output of each tree. The learning parameter controls the magnitude of this change in the estimates. Typical values: 0.1, 0.001, 0.003…\n- **num_leaves**: number of leaves in full tree, default: 31\n- **device**: default: cpu, can also pass gpu"},{"metadata":{"trusted":true,"_uuid":"61c7faed44aaa10681e0c8890c0f602e913cd095"},"cell_type":"code","source":"# Default parameters for LightGBM\nparam1 = {'application': \"regression\", \"boosting\": \"gbdt\", \"metric\": 'rmse', 'max_depth': 3,  \n          'learning_rate': 0.1, 'num_leaves':31, 'min_data_in_leaf': 20, \"random_state\": 2019,\n          'min_gain_to_split':0.5,      # default =0\n          'feature_fraction': 0.5,      # default =1 \n          'bagging_fraction': 0.5,      # default =1 \n         }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7bce38bf775632006216c9592c0a7b9ebbb7407e"},"cell_type":"code","source":"# define function in order to calculate RMSE and Improvement of Model\ndef lgb_evaluation(model,X_train=X_train,y_train=y_train,X_test=X_test,y_test=y_test):\n    pred_lgb_model=model.predict(X_test)\n    rmse_new=rmse(y_test,pred_lgb_model)\n    print(\"Model: LGB;\",'RMSE: {:0.2f}'.format(rmse_new),\";\",'Improvement of: {:0.2f}%'.format(100 * (rmse_dummy- rmse_new) / rmse_new))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6834173faf0323847c29e8e67a6739cbcf983018","scrolled":true},"cell_type":"code","source":"#Setting up the first LGBM Model\nrounds = 10\ntraining_data = lgb.Dataset(data = X_train, label = y_train, params = param1, \n                          categorical_feature = categorical_feats, free_raw_data = False)\n#Training the Model\n%time lgb1 = lgb.train(train_set = training_data, params = param1, num_boost_round = rounds)\n\n#Making Predictions\npred_lgb1=lgb1.predict(X_test)\n\n#Evaluation of first LGB\nlgb_evaluation(model=lgb1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"875d657690de40210d87dd9d5cb329cdb84a3ec7"},"cell_type":"markdown","source":"The result of the first LGBM tree shows, that without specific parameters and intense training, there is not much difference to a usual decision tree. Let's see if we can find the optimum parameters more elegant than in Sklearn and finally get a good result."},{"metadata":{"_uuid":"070eb5dd277b92446cb6a050c8b8b81995367c7e"},"cell_type":"markdown","source":"<a id=\"3.3.1\"></a> <br>\n### 3.3.1 Bayesian Optimization"},{"metadata":{"_uuid":"f9722c585b107057189567f041d64b18771d715b"},"cell_type":"markdown","source":"\nAccording to TowardsDataScience, the challenge with hyperparameter optimization is that **evaluating the objective function is extremely expensive**. For each different set of parameters, we need to train the model, make predictions and calculate the validation set. For complex models and large numbers of parameters that can take up to days.\nMethods like GridSearch or RandomSearch are relatively inefficient, because they do not choose the next hyperparameters to evaluate based on previous results. They are **uninformed by past evaluations**, and as a result, often spend a significant amount of time evaluating “bad” hyperparameters.\n\nBayesian optimization finds the value that **minimizes an objective function** by building a surrogate function (probability model) **based on past evaluation** results of the objective. \nThe surrogate is cheaper to optimize than the objective, so the next input values to evaluate are selected by applying a criterion to the surrogate (often Expected Improvement). \n\nThe concept is: limit expensive evaluations of the objective function by choosing the next input values based on those that have done well in the past. Basically there are four parts of optimization problem:\n- **Objective Function**: what we want to minimize, in this case the validation error of a machine learning model with respect to the hyperparameters\n- **Domain Space**: hyperparameter values to search over\n- **Optimization algorithm**: method for constructing the surrogate model and choosing the next hyperparameter values to evaluate\n- **Result history**: stored outcomes from evaluations of the objective function consisting of the hyperparameters and validation loss\n\n(Source: https://towardsdatascience.com/a-conceptual-explanation-of-bayesian-model-based-hyperparameter-optimization-for-machine-learning-b8172278050f)"},{"metadata":{"_uuid":"379f8cec541af9c4c7902e852a26198f8f0d1691"},"cell_type":"markdown","source":"Bayesian Optimization to find optimum Parameters:"},{"metadata":{"trusted":true,"_uuid":"ee191f892245893c5ac88e2eaaa99a01f3a0db23"},"cell_type":"code","source":"from bayes_opt import BayesianOptimization\n\n# Define Bayes Optimization function for LGBM\ndef bayes_parameter_opt_lgb(training, testing, init_round=5, opt_round=10, n_folds=4, random_seed=2019, \n                            n_estimators=100, learning_rate=0.05, output_process=False):\n    \n    # Training data\n    training_data = lgb.Dataset(data= train, label= target, categorical_feature = categorical_feats, free_raw_data=False)\n    \n    # Parameters to optimize\n    def lgb_evaluation(learning_rate, num_leaves, feature_fraction, bagging_fraction, max_depth, lambda_l1,min_data_in_leaf,min_split_gain):\n        \n        params = {'application':'regression','num_iterations': n_estimators,  \n                  'early_stopping_round':100, 'metric':'rmse'}\n        # Bayes opt's outputs are always float\n        params['learning_rate'] = max(min(feature_fraction, 1), 0)\n        params[\"num_leaves\"] = int(round(num_leaves)) # Rounding of parameters did not work\n        params['bagging_fraction'] = max(min(bagging_fraction, 1), 0)\n        params['max_depth'] = int(round(max_depth))\n        params['min_data_in_leaf'] = int(round(min_data_in_leaf))\n        params['lambda_l1'] = max(lambda_l1, 0)\n        params['min_split_gain'] = min_split_gain\n        cv_result = lgb.cv(params, training_data, nfold=n_folds, seed=random_seed, stratified=False, \n                           verbose_eval =200, metrics=['rmse'])\n        return max(cv_result['rmse-mean'])\n   \n    # Parameter Range\n    lgbBO = BayesianOptimization(lgb_evaluation, {'learning_rate': (0.0001, 0.1),\n                                            'num_leaves': (150, 300),\n                                            'feature_fraction': (0.01, 0.2),\n                                            'bagging_fraction': (0.2, 1),\n                                            'max_depth': (1, 3),\n                                            'lambda_l1': (0, 2),\n                                            'min_data_in_leaf': (200,400),\n                                            'min_split_gain': (0.001, 0.2)})\n    # Optimize\n    lgbBO.maximize(init_points=init_round, n_iter=opt_round)\n    \n    # Output optimization process to CSV\n    if output_process==True: lgbBO.points_to_csv(\"bayes_opt_result1.csv\")\n    \n    # return best parameters\n    return lgbBO.max","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9506f93a7708d90a9e24088f3f2ff9fd8103bad7","scrolled":false},"cell_type":"code","source":"# Using optimization function on training data\n%time \nopt_params_lgb = bayes_parameter_opt_lgb(train, target)\nparam1=opt_params_lgb['params'];param1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"659c1daf848f12a1881f3c29d3c624832041335c"},"cell_type":"code","source":"# Results of different rounds of Bayesian Optimization\nparam1 = {\"boosting\": \"gbdt\", 'objective':'regression',\"metric\": 'rmse', \"random_state\": 2019, \"verbosity\": -1,\n          'bagging_fractions':0.7768, 'feature_fraction':0.3249, 'lambda_l1':0.0460, 'learning_rate':0.067,\n          'max_depth':3, 'min_data_in_leaf':176, 'min_split_gain':0.0297, 'num_leaves':77}\nparam2 = {'num_leaves': 111, 'min_data_in_leaf': 149,'objective':'regression','max_depth': 9,\n         'learning_rate': 0.005, \"boosting\": \"gbdt\", \"feature_fraction\": 0.7522, \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.7083 , \"bagging_seed\": 11, \"metric\": 'rmse', \"lambda_l1\": 0.2634,\n         \"random_state\": 133, \"verbosity\": -1}\nparam3 = {\"boosting\": \"gbdt\", 'objective':'regression',\"metric\": 'rmse', \"random_state\": 2019, \"verbosity\": -1,\n          'bagging_fractions':1, 'feature_fraction':0.1, 'lambda_l1':1.0, 'learning_rate':0.001,\n          'max_depth':2, 'min_data_in_leaf':250, 'min_split_gain':0.001, 'num_leaves':160}\nparam4 = {\"boosting\": \"gbdt\", 'objective':'regression',\"metric\": 'rmse', \"random_state\": 2019, \"verbosity\": -1,\n          'bagging_fractions':0.7, 'feature_fraction':0.02, 'lambda_l1':1.75, 'learning_rate':0.05,\n          'max_depth':3, 'min_data_in_leaf':305, 'min_split_gain':0.01, 'num_leaves':300}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fdfbda674b0f162361970b3e39ac397a6ee8cc5b"},"cell_type":"markdown","source":"After finding (optimal) hyperparameters in an incedible **short amount of time** (in comparison to GridSearch) we defined a function to run the LGBM.\n\nThe function uses KFold to create indizes for **cross-validation**. Each fold is used once for validation and k - 1 times for training. After that the LGBM is trained on each fold for a given number of rounds, whereas every round can be seen as one tree. The training stops after the validation did not improve for a given number of 'early_stopping_rounds."},{"metadata":{"trusted":true,"_uuid":"ba0f3c405103b070318df81105fb92f366f4aa16","scrolled":true},"cell_type":"code","source":"#train/test indices to split data in train/test sets -> each fold is then used once as a validation while the k - 1 remaining folds form the training set\nfolds             = KFold(n_splits=4,shuffle=True,random_state=15)\nX                 = train\ny                 = target\nparam             = param3\nnum_boost_rounds  = 10000\nearly_stopping    = 100\nout_of_folds      = np.zeros(len(X))\npredictions       = np.zeros(len(y))\nfeature_importance= pd.DataFrame()\nevals_result      = {}\n\n# for-loop to split data in n training and validation sets and perfom LGBM on every fold\nfor fold, (train_ind,val_ind) in enumerate(folds.split(X,y)):\n    print(\"fold n°{}\".format(fold))\n    \n    #training&validation sets\n    training=lgb.Dataset(X.iloc[train_ind],label=y.iloc[train_ind])\n    validation=lgb.Dataset(X.iloc[val_ind],label=y.iloc[val_ind])\n    \n    #training for each fold:\n    lgb2=lgb.train(param1,training,valid_sets = [training,validation], verbose_eval=10,num_boost_round=num_boost_rounds,\n                   early_stopping_rounds = early_stopping  , evals_result = evals_result)\n    \n    #predictions on validation fold -> OOF:\"Out-of-fold\" -> using k-fold validation in which the predictions from each set of folds are grouped together into one group of 1000 predictions\n    out_of_folds[val_ind] = lgb2.predict(X.iloc[val_ind], num_iteration=lgb2.best_iteration)\n    \n    #storing in feature importance\n    fold_importance= pd.DataFrame()\n    fold_importance[\"feature\"] = features\n    fold_importance[\"importance\"] = lgb2.feature_importance()\n    fold_importance[\"fold\"] = fold + 1\n    feature_importance = pd.concat([feature_importance, fold_importance], axis=0)\n    \nlgb_evaluation(lgb2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a72d35fcc6e0b01c41b15987190a216badf81af6","scrolled":true},"cell_type":"code","source":"lgb2.best_score","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"143e4d17a3caad2e95fb748de4c1f4828b73a2b6"},"cell_type":"markdown","source":"We can nicely see how the training score keeps improving while overfitting the train and the validation score stagnates."},{"metadata":{"trusted":true,"_uuid":"57cd9bda6ee72745c31441ecb5a9b7b9abc0ed1f","scrolled":true},"cell_type":"code","source":"#figure(num=None, figsize=(8, 6), dpi=80, facecolor='w', edgecolor='k')\n#plt.title(\"Validation Curve with RandomForestRegressor\");plt.xlabel(\"Number of Trees\");plt.ylabel(\"RMSE\")\n#plt.legend(loc=\"best\"); plt.show()\nax = lgb.plot_metric(evals_result, metric='rmse')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e913fac11e6e10f3dac914aa49996b024dbefa0c"},"cell_type":"markdown","source":"<a id=\"3.3.2\"></a> <br>\n### 3.3.2 Feature Importance"},{"metadata":{"_uuid":"4391990511349084491441d39418fe31de2e3a09"},"cell_type":"markdown","source":"LGBM provides us the importance of the features, like every other tree, without extra effort. We can clearly see that the LGBM **gained to most** when he used features which have been created by the **purchase amount** of an customer."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"e5c48756d62fbc2b6074992ee0d9f730484e2f1d"},"cell_type":"code","source":"feature_importance.sort_values(by=['importance'],ascending=False).head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4cfaf37877347eb968a6775829f076ffb5692e6e","scrolled":true},"cell_type":"code","source":"# Group by feature\nagg_features = (feature_importance[[\"feature\",\"importance\"]].groupby(\"feature\").mean().sort_values(by=\"importance\",ascending=False)[:1000].index)\nbest_features = feature_importance.loc[feature_importance.feature.isin(agg_features)]\n\n# Plot feature importance\nplt.figure(figsize=(12,20))\nsns.barplot(x=\"importance\",y=\"feature\",data=best_features.sort_values(by=\"importance\",ascending=False))\nplt.title('LightGBM Features (avg over folds)');plt.tight_layout()\n#plt.savefig('lgbm_importances.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ced50e2fce0d4943b731078ffd2ccbcdd575227b"},"cell_type":"code","source":"predictions = lgb2.predict(test)\nelo_sub_XX = pd.DataFrame({\"card_id\":org_test[\"card_id\"].values})\nelo_sub_XX[\"target\"] = predictions\nelo_sub_XX.to_csv(\"elo_subX.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"227b95c1e1bb8105b87b7cef4474c59aea77f2db"},"cell_type":"markdown","source":"<a id=\"3.4\"></a> <br>\n### 3.4 Models without Outliers"},{"metadata":{"_uuid":"8f851897480cfa816db38183d9cbd76fccaa1927"},"cell_type":"markdown","source":"Finally we train our models on the data without outliers. The results have been quite imressive, apperently the only reason why our models did not perform have been the outliers."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"5aec727c7aa4a81895d7987d031e6d4ac851ecce"},"cell_type":"code","source":"#train/test indices to split data in train/test sets -> each fold is then used once as a validation while the k - 1 remaining folds form the training set\nfolds             = KFold(n_splits=5,shuffle=True,random_state=15)\nX                 = clean_train\ny                 = clean_target\nparam             = param3\nnum_boost_rounds  = 100\nearly_stopping    = 10\nout_of_folds      = np.zeros(len(X))\npredictions       = np.zeros(len(y))\nfeature_importance= pd.DataFrame()\nevals_result      = {}\n\n# for-loop to split data in n training and validation sets and perfom LGBM on every fold\nfor fold, (train_ind,val_ind) in enumerate(folds.split(X,y)):\n    print(\"fold n°{}\".format(fold))\n    \n    #training&validation sets\n    training=lgb.Dataset(X.iloc[train_ind],label=y.iloc[train_ind])\n    validation=lgb.Dataset(X.iloc[val_ind],label=y.iloc[val_ind])\n    \n    #training for each fold:\n    lgb_clean=lgb.train(param1,training,valid_sets = [training,validation], verbose_eval=10,num_boost_round=num_boost_rounds,\n                   early_stopping_rounds = early_stopping  , evals_result = evals_result)\n    \n    #predictions on validation fold -> OOF:\"Out-of-fold\" -> using k-fold validation in which the predictions from each set of folds are grouped together into one group of 1000 predictions\n    out_of_folds[val_ind] = lgb_clean.predict(X.iloc[val_ind], num_iteration=lgb_clean.best_iteration)\n\n    \nlgb_evaluation(lgb_clean ,X_train = X_train_clean, y_train = y_train_clean,\n           X_test = X_test_clean, y_test = y_test_clean );","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"85956f67f0df6e18e604bcb05e9f251dba89681e"},"cell_type":"code","source":"lgb_clean.best_score","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"713920aa8b0e05a3ba018e8d22af68f2da50d026"},"cell_type":"markdown","source":"Even the Linear Models are providing really good results now. "},{"metadata":{"trusted":true,"_uuid":"67b19dce09e7f8519ee280368e906de161b35da3"},"cell_type":"code","source":"linReg=LinearRegression()\nlogReg=LogisticRegression()\nlasso=Lasso(alpha=0.9)\nridge=Ridge(alpha=0.9) # higher the alpha value, more restriction on the coefficients; \n                       # low alpha > more generalization, coefficients are barely\nelasNet=ElasticNet(0.5)\n\nfor reg in (linReg,lasso,ridge,elasNet):\n    evaluation(reg ,X_train = X_train_clean, y_train = y_train_clean,\n               X_test = X_test_clean, y_test = y_test_clean );","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"90b4db11ef1f181108776af2bff2016ca59a3c54"},"cell_type":"code","source":"#predictions = lgb_clean.predict(test)\n#elo_sub_clean = pd.DataFrame({\"card_id\":org_test[\"card_id\"].values})\n#elo_sub_clean[\"target\"] = predictions\n#elo_sub_clean.to_csv(\"elo_sub_clean.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"76cea630265ea9dbd14a6f3ecb52f765774674dc"},"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n# 4. Conclusion\n\n\n### Findings:\n\n- The real challenge of the competetion is to **predict the outliers**. Whoever will manage it to include an outlier prediction into his model, will probably win the competition\n- Althoug it is only around 1 %, the  **extreme outliers** (around 10 standard deviations -> -31 value) **blow out the RMSE's** of all competitors so far\n- Looking at the Leaderboard ( best RMSE around 3.6) no one has accomplished that so far\n- Training an data set **without outliers** gives us **RMSE's down to 0.37**, but of course only test data without outliers as well\n- Looking at **the feature importance**, the amount one customer spent turned out to be the most important indicator for loyalty\n- The features which stand out the most are the ones that represent **how much a customer paid and when they committed those transactions**\n\n### Lessons learned:\n \n -  Handle the **memory usage** of your data. Chaning the format of the data as well as using samples of the data turned out to be really helpful to move the data through the pipeline\n - **Aggregate** rows from the historical data set by using min, mean, max etc. \n - Save lots of time and use **Bayesian Optimization** to find the right parameters, instead of GridSearch\n -  Drop the columns that do not have a similar distribution of values between the train and test files. This will cause your predictions on test to be well prepared by the train data set.\n - Evaluate the **data input** for your model. It turned out over and over again, that the **data quality** is way more important for a model to learn than the algorithm itself.\n - Let the model know, which features are **categorical**\n - While working on one data project for a long time, it can be really helpful to **define functions and pipelines** to automate certain repeating steps\n \n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}