{"cells":[{"metadata":{"_uuid":"05301e318a0c341f6f2e9a007ecc608354a846c1"},"cell_type":"markdown","source":"## Notebook  Content\n1. [Inspirational Kernels](#1)\n2. [EDA](#2)\n3. [Loading the data](#3)\n4. [Feature engineering](#4)\n5. [Train](#5)\n6. [Ridge and Lasso](#6)\n7. [Regression Tree](#7)\n8. [Random Forest](#8)\n9. [Boosting](#9)\n10. [Lightgbm](#10)\n11. [ Model Performance Comparison and Conclusion](#11)"},{"metadata":{"_uuid":"81e2dd5984e205bfbf414069a7fa89a56cd7e5a8"},"cell_type":"markdown","source":"**1. Description summary**\n\nElo is a leading Brazilian payment provider. It cooperates with different merchants to offer customers various discounts and promotions. However, Elo is not sure how these promotions affect both merchants and customers. For this reason the company wants to look into a more personalised approach. \n\nElo is looking to develop algorithms to tailor its discounts and promotions to each individual based on their loyalty.\n\n\n**2. Our understanding of the problem**\n\nThe goal of this competition is to understand the relationship between the provided and engineered features and the target value - users' individual loyalty score. For this Elo needs a model that will predict people’s reaction to discounts and promotions based on their past behaviour and personal details."},{"metadata":{"_uuid":"47c3d394acd48f79042a06244489823c5f45852b"},"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n## 1. Inspirational Kernels\nThese are the kernels that helped us \n* [Simple Data Exploration with Python [LB : 3.764]](https://www.kaggle.com/chocozzz/simple-data-exploration-with-python-lb-3-764) \n* [Making Sense of Elo Data (EDA)](https://www.kaggle.com/batalov/making-sense-of-elo-data-eda)\n* [Ridge + LightGBM + feature Engineering + Bayesian](https://www.kaggle.com/ashishpatel26/ridge-lightgbm-feature-engineering-bayesian)\n* [Elo World](https://www.kaggle.com/fabiendaniel/elo-world)\n* [Simple Exploration Notebook - Elo](https://www.kaggle.com/sudalairajkumar/simple-exploration-notebook-elo)\n* [Elo EDA and models](https://www.kaggle.com/artgor/elo-eda-and-models)\n* [Simple LightGBM without blending](https://www.kaggle.com/mfjwr1/simple-lightgbm-without-blending)\n* [Data Science Classes 8 & 10](http://)"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","scrolled":true,"trusted":true},"cell_type":"code","source":"# importing all needed libraries \n\nimport time\nimport datetime\nimport numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\n# display the output of plotting commands inline within frontends, directly below the code cell that produced it.\n%matplotlib inline\nplt.style.use('ggplot')\n\n\nfrom plotly import tools\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\n\nfrom sklearn import model_selection, preprocessing, metrics\nimport lightgbm as lgb\n\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.externals.six import StringIO  \nfrom sklearn.tree import DecisionTreeRegressor, DecisionTreeClassifier, export_graphviz\nfrom sklearn.ensemble import BaggingClassifier, RandomForestClassifier, RandomForestRegressor, GradientBoostingRegressor\nfrom sklearn.metrics import mean_squared_error,confusion_matrix, classification_report\nimport pydot\nfrom IPython.display import Image\n\nfrom sklearn.linear_model import LinearRegression, Ridge, RidgeCV, Lasso, LassoCV\nfrom sklearn.model_selection import GridSearchCV\nimport sklearn.linear_model as skl_lm\nfrom sklearn.preprocessing import scale \n\n\n# print all files available in the data folder\nimport os\nprint(os.listdir(\"../input/elo-merchant-category-recommendation/\"))\n\n# ignore warnings\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"108d0f00d052d738b62e6887a50da99d8625123a","trusted":true},"cell_type":"code","source":"# Set matplotlib figure sizes to 10 and 6, font size to 12.\nfrom matplotlib import rcParams\nrcParams['figure.figsize'] = (10, 6)\nrcParams['font.size'] = 12","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"},"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n## 3. Loading the data"},{"metadata":{"_uuid":"13725c77eefb1cd6351957ff73d7e557ec7a682a","trusted":true},"cell_type":"code","source":"# it takes more than 50s to read, because there are 29 million lines in historical_transactions\ntrain = pd.read_csv('../input/elo-merchant-category-recommendation/train.csv', parse_dates=['first_active_month'])\ntest = pd.read_csv('../input/elo-merchant-category-recommendation/test.csv', parse_dates=['first_active_month'])\nhistorical_transactions = pd.read_csv('../input/elo-merchant-category-recommendation/historical_transactions.csv', parse_dates=['purchase_date'])\nnew_merchant_transactions = pd.read_csv('../input/elo-merchant-category-recommendation/new_merchant_transactions.csv', parse_dates=['purchase_date'])\nmerchants = pd.read_csv('../input/elo-merchant-category-recommendation/merchants.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e4e516dccc3c97f2f015f76a095e37bdc9dcea84"},"cell_type":"markdown","source":"Historical transactions has more than 29 Million rows, we're going to sample the data so that we can join it faster together while working and so that our kernel does not die while running. New merchant transactions has about 2 Million rows.\n\nIn the case of multiple commits, we applied a downsampling technique through taking a random sample of the 5 Million from historical transactions and of 500 thousand from new merchant transactions. In the final version the notebook was ran without any downsampling. "},{"metadata":{"_uuid":"d343f47694c4d6e7547854fc4615a67ce03374b1","trusted":true},"cell_type":"code","source":"historical_transactions = historical_transactions.sample(n=5000000, random_state=1111)  # random_state is the seed \nnew_merchant_transactions = new_merchant_transactions.sample(n=500000, random_state=1111)  # random_state is the seed ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61f1705c855320e1edec6d55e1eed8ee952f9888"},"cell_type":"markdown","source":"<a id=\"2\"></a> <br>\n## 2. EDA"},{"metadata":{"_uuid":"09a65028f44759e39f7157e5171266309465cf72"},"cell_type":"markdown","source":"The first step of the project was to get familiar with the available data. Our method was to look into each fileand explore its shape, features, size, values etc. "},{"metadata":{"_uuid":"264a47a60e38cdcf3ca576ebd0f929e5c81794f6"},"cell_type":"markdown","source":"**1) Description of the columns of all the data. **"},{"metadata":{"_uuid":"4e2027f468e8b224bb8006c4adb3b68a0568dea1","trusted":true},"cell_type":"code","source":"train_data = pd.read_excel('../input/elo-merchant-category-recommendation/Data_Dictionary.xlsx', sheet_name='train')\nhistory_data = pd.read_excel('../input/elo-merchant-category-recommendation/Data_Dictionary.xlsx', sheet_name='history')\nnew_merchant_period = pd.read_excel('../input/elo-merchant-category-recommendation/Data_Dictionary.xlsx', sheet_name='new_merchant_period')\nmerchant = pd.read_excel('../input/elo-merchant-category-recommendation/Data_Dictionary.xlsx', sheet_name='merchant')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f55f312b6cc313e3efd80458af07deeb62e5bd2","trusted":true},"cell_type":"code","source":"# description of the train data\ntrain_data","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a0e27ba71306ad1849e4a48ccbe5eff6c664a76","trusted":true},"cell_type":"code","source":"history_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bd3863b27ed68e51b0519fa4a30e28581b8b2aa4"},"cell_type":"code","source":"new_merchant_period","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d78bc5fac42fd13427f91e269d67b1f6790f78d7"},"cell_type":"code","source":"merchant","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7e21894ce3eff82a99b528cc69dd489bb3e3926a"},"cell_type":"markdown","source":"**2) Shapes of the different dataframes**"},{"metadata":{"_uuid":"766e8b21c061172cdf268f861847ad93d886d260","trusted":true},"cell_type":"code","source":"train.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"59ad62a4cca8e9aee1d1ef6c3197400859677948","trusted":true},"cell_type":"code","source":"historical_transactions.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9fd59f2be82062db4caa202d7380cf86a0862b75","trusted":true},"cell_type":"code","source":"new_merchant_transactions.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6b7d14fa50ce6aaddc411a9c4e027f84a892d88f"},"cell_type":"code","source":"merchants.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8d7c95d70c567f39880cc3a29e206531fab50e0d"},"cell_type":"markdown","source":"**3) Function to print out null values**"},{"metadata":{"trusted":true,"_uuid":"add8ff5b609cbe6c4bdc3d54658e2670d7db6cf0"},"cell_type":"code","source":"'''The function prints out the number and persentage of null values a dataframe column has.'''\ndef print_null(df):\n    for col in df:\n        if df[col].isnull().any():\n            print('%s has %.0f null values: %.3f%%'%(col, df[col].isnull().sum(), df[col].isnull().sum()/df[col].count()*100))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ef79766bad4e2f5588511637619f13982be51c10"},"cell_type":"markdown","source":"**4) Exploration of the merchants dataframe**"},{"metadata":{"trusted":true,"_uuid":"733cc05eb0620fe831dc6ca0eb89fd010d6d2f28"},"cell_type":"code","source":"# Checking the types of the column values\nprint(merchants.dtypes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1dfaafdcbed9e5fc6740a14ce9d0a261c2c1e30d","scrolled":false},"cell_type":"code","source":"# Checking for missing data\nprint_null(merchants)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3bbfee23b5ac9c8c42c326ff288e9aa53d0e8ae6"},"cell_type":"code","source":"#Now, let's look at column histograms:\n\ncat_cols = ['active_months_lag6','active_months_lag3','most_recent_sales_range', 'most_recent_purchases_range','category_1','active_months_lag12','category_4', 'category_2']\nnum_cols = ['numerical_1', 'numerical_2','merchant_group_id','merchant_category_id','avg_sales_lag3', 'avg_purchases_lag3', 'subsector_id', 'avg_sales_lag6', 'avg_purchases_lag6', 'avg_sales_lag12', 'avg_purchases_lag12']\n\n# Removing infinite values and replacing them with NAN\nmerchants.replace([-np.inf, np.inf], np.nan, inplace=True)\n\nplt.figure(figsize=[15, 15])\nplt.suptitle('Merchants table histograms', y=1.02, fontsize=20)\nncols = 4\nnrows = int(np.ceil((len(cat_cols) + len(num_cols))/4))\nlast_ind = 0\nfor col in sorted(list(merchants.columns)):\n    #print('processing column ' + col)\n    if col in cat_cols:\n        last_ind += 1\n        plt.subplot(nrows, ncols, last_ind)\n        vc = merchants[col].value_counts()\n        x = np.array(vc.index)\n        y = vc.values\n        inds = np.argsort(x)\n        x = x[inds].astype(str)\n        y = y[inds]\n        plt.bar(x, y, color=(0, 0, 0, 0.7))\n        plt.title(col, fontsize=15)\n    if col in num_cols:\n        last_ind += 1\n        plt.subplot(nrows, ncols, last_ind)\n        merchants[col].hist(bins = 50, color=(0, 0, 0, 0.7))\n        plt.title(col, fontsize=15)\n    plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"868a24f424b20f069a03d6450845fad72883c7b8"},"cell_type":"markdown","source":" numerical_1 and numerical_2 seem not to be numerical but on the contrary - categorical \n"},{"metadata":{"trusted":true,"_uuid":"204e3c08c8e93ce1f64a471def5ff3e89c5081b4"},"cell_type":"code","source":"#Now, let's look at correlations between columns in merchants.csv:\n\ncorrs = np.abs(merchants.corr())\nordered_cols = (corrs).sum().sort_values().index\nnp.fill_diagonal(corrs.values, 0)\nplt.figure(figsize=[10,10])\nplt.imshow(corrs.loc[ordered_cols, ordered_cols], cmap='plasma', vmin=0, vmax=1)\nplt.colorbar(shrink=0.7)\nplt.xticks(range(corrs.shape[0]), list(ordered_cols), rotation=90)\nplt.yticks(range(corrs.shape[0]), list(ordered_cols))\nplt.title('Heat map of coefficients of correlation between merchant\\'s features', fontsize=17)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"745f308e11902a3d10122eeca36a63ae6fd6a376"},"cell_type":"markdown","source":"* numerical_1 and numerical_2 are highly correlated\n* avg_sales and avg_purchases within the last 3, 6, and 12 months are highly correlated\n* mechant_group_id is loosely correlated with numerical_1, 2, city_id, and sales statistics\n* merchant_category_id shows little correlation with merchant_group_id, city_id, or really anything else.\n* category_1 is slightly correlated with the merchant's location (city_id and state_id)"},{"metadata":{"trusted":true,"_uuid":"a2d4ee61262873ae4a0f468a699b81c4e42a2154"},"cell_type":"code","source":"x = np.array([12, 6, 3]).astype(str)\nsales_rates = merchants[['avg_sales_lag3', 'avg_sales_lag6', 'avg_sales_lag12']].mean().values\npurchase_rates = merchants[['avg_purchases_lag3', 'avg_purchases_lag6', 'avg_purchases_lag12']].mean().values\nplt.bar(x, sales_rates, width=0.3, align='edge', label='average sales', edgecolor=[0.2]*3)\nplt.bar(x, purchase_rates, width=-0.3, align='edge', label='average purchases', edgecolor=[0.2]*3)\nplt.legend()\nplt.title('Avergage sales and number of purchases\\nover the last 12, 6, and 3 months', fontsize=17)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3d499f3f1bb35c06700fb836f5d4cb0e518eb2ae"},"cell_type":"markdown","source":"It looks like the sales are steadily growing over time.\n\nSince we saw in the discussion board, in quite a few other kernels and in a few of our commits that the merchants data does not improve the model, we decided not to use it in the next steps of the project. "},{"metadata":{"trusted":true,"_uuid":"b55ce46ab95aab7da4b51772fa16f916aeebe63b"},"cell_type":"markdown","source":"**5) Exploration of the train and test dataframes**"},{"metadata":{"trusted":true,"_uuid":"e790ade4937411d61f7dfbee56ca0317b1b8ced0"},"cell_type":"code","source":"# Target distribution in the train dataframe\nplt.hist(train['target'], bins= 50)\nplt.title('Loyalty score')\nplt.xlabel('Loyalty score')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6e17dd9b649401546e4c64cee7ca919a037758a3"},"cell_type":"code","source":"((train['target']<-30).sum() / train['target'].count()) * 100 # percentage of outliers","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eae2f84277763313ced04424c9ea5bef76db0ad4"},"cell_type":"markdown","source":"We see there are 1,09 % of outliers in the data. We did try to run the models with and without the outliers and noticed that it did not improve our submission score. Thus, we are not dropping them in the feature engineering.."},{"metadata":{"_uuid":"97d0b73b762b2c89f9973987854e86bdefef9e7c"},"cell_type":"markdown","source":"Is there a difference between the train and the test data sets when it comes to the first active month?. We see that in the last month the counts for the data decline.The test dataframe has less values for first_active_month. This probably because it also has less records."},{"metadata":{"trusted":true,"_uuid":"24b5cc7656146d271a82c8d1ad9d5f924e72a150"},"cell_type":"code","source":"print(max(train['first_active_month']))\nprint(max(test['first_active_month']))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"853fe6d113e8a1658e784d52405de39db56a130d"},"cell_type":"code","source":"d1 = train['first_active_month'].value_counts().sort_index()\nd2 = test['first_active_month'].value_counts().sort_index()\ndata = [go.Scatter(x=d1.index, y=d1.values, name='train'), go.Scatter(x=d2.index, y=d2.values, name='test')]\nlayout = go.Layout(dict(title = \"Counts of first active\",\n                  xaxis = dict(title = 'Month'),\n                  yaxis = dict(title = 'Count'),\n                  ),legend=dict(\n                orientation=\"v\"))\npy.iplot(dict(data=data, layout=layout))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"35a98f4b98cb7ece6c3a67e1cfabc044663586a7"},"cell_type":"markdown","source":"**Historical Transactions**"},{"metadata":{"trusted":true,"_uuid":"ca46763f57c81421e01d57bff3d0e72a212bcc45"},"cell_type":"code","source":"# binarize authorized_flag, replace Y with 1 and N with 0\nhistorical_transactions['authorized_flag'] = historical_transactions['authorized_flag'].map({'Y':1, 'N':0})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f063d614585a2f3a3e7b5f45d178a3b573289bf9"},"cell_type":"code","source":"# authorized_flag distribution in historical transactions\n(\"At average \" + str(historical_transactions['authorized_flag'].mean() * 100) + \"% transactions are authorized\")\nhistorical_transactions['authorized_flag'].value_counts().plot(kind='bar', title='authorized_flag value counts');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e7a6216666902697967d7afae85d2f1f63a2884c"},"cell_type":"code","source":"historical_transactions['installments'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"86a4d0f28dc0d482447b9f6c0106c296df856f87"},"cell_type":"markdown","source":"Most common values for installments are 0 and 1."},{"metadata":{"trusted":true,"_uuid":"c7f5388487fb41078c942279c4ce2b3eddc79cdf"},"cell_type":"code","source":"historical_transactions.groupby(['installments'])['authorized_flag'].mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5659e3160034ccb9a1ab36478c3a66c1647577e9"},"cell_type":"markdown","source":"It seems as if installments with 999 as value have not been authorized. "},{"metadata":{"trusted":true,"_uuid":"a7a62293ed2903b719002767a45b02636d08db00"},"cell_type":"code","source":"# We know from the Dictionary that Purchase Amount is normalized\nfor i in [-1, 0]:\n    n = historical_transactions.loc[historical_transactions['purchase_amount'] < i].shape[0]\n    print(\"There are \" + str(n) + \" transactions with purchase_amount less than \" + str(i) + \".\")\nfor i in [0, 10, 100]:\n    n = historical_transactions.loc[historical_transactions['purchase_amount'] > i].shape[0]\n    print(\"There are \" + str(n) + \" transactions with purchase_amount more than \" + str(i) + \".\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e66894904956d48973c9b50c00b605dda5945173"},"cell_type":"code","source":"max(historical_transactions['purchase_amount'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a541dd5e935917e33a8c65ff3cadd6ddca9db861"},"cell_type":"markdown","source":"There are many  transactions with a purchase_amount between -1 and 0. Maybe this suggests that the purchase has been standardized (scaled). Nonetheless, the highest purchase amount in downsampled data is over 6 Million."},{"metadata":{"trusted":true,"_uuid":"de52bec4a5c56852c326c6d4170e133805bcfee1"},"cell_type":"code","source":"# Unique values in historical transactions\nfor col in ['city_id', 'merchant_category_id', 'merchant_id', 'state_id', 'subsector_id']:\n    print(\"There are \" + str(historical_transactions[col].nunique()) + \" unique values in \" + str(col) + \".\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"50d12fe7d681a07fc3af08795f64c5b0dd793d81"},"cell_type":"markdown","source":"**New merchant transactions**"},{"metadata":{"trusted":true,"_uuid":"44b8e7de34839652c12cd9aa6bd3aa554636e60f"},"cell_type":"code","source":"# binarize authorized_flag, replace Y with 1 and N with 0\nnew_merchant_transactions['authorized_flag'] = new_merchant_transactions['authorized_flag'].map({'Y':1, 'N':0})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"635dd59bdae018763af3244cc8b4266e7c175dcb"},"cell_type":"code","source":"# authorized_flag distribution in new merchant transactions\nprint(\"At average \" + str(new_merchant_transactions['authorized_flag'].mean() * 100) + \"% transactions are authorized\")\nnew_merchant_transactions['authorized_flag'].value_counts().plot(kind='bar', title='authorized_flag value counts');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e95456ac1eecb1719d12c174d0e83351a9bc3528"},"cell_type":"markdown","source":"In the new merchant transactions dataframe all transactions are authorized."},{"metadata":{"trusted":true,"_uuid":"3ac2284560c9d350b1a00c2ac174711f9342f68a"},"cell_type":"code","source":"new_merchant_transactions['installments'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9815c9cd4ebaeea249b405299125fd5b7041edef"},"cell_type":"markdown","source":"In the new merchant transactions as well most of the installments have the value 0 oor 1."},{"metadata":{"trusted":true,"_uuid":"d6f09d782d9b7f62e65e744ed4def22f55fa67ce"},"cell_type":"code","source":"# We know from the Dictionary that Purchase Amount is normalized\nfor i in [-1, 0]:\n    n = new_merchant_transactions.loc[new_merchant_transactions['purchase_amount'] < i].shape[0]\n    print(\"There are \" + str(n) + \" transactions with purchase_amount less than \" + str(i) + \".\")\nfor i in [0, 10, 100]:\n    n = new_merchant_transactions.loc[new_merchant_transactions['purchase_amount'] > i].shape[0]\n    print(\"There are \" + str(n) + \" transactions with purchase_amount more than \" + str(i) + \".\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"40df59f5a883d245f7029eb434d9212c270d38e6"},"cell_type":"markdown","source":"There are many transactions with a purchase_amount between -1 and 0 (scaled value?)."},{"metadata":{"trusted":true,"_uuid":"7813ffeb98a71132ab862040e879a2ed8ba59436"},"cell_type":"code","source":"# Unique values in new merchant transactions\nfor col in ['city_id', 'merchant_category_id', 'merchant_id', 'state_id', 'subsector_id']:\n    print(\"There are \" + str(new_merchant_transactions[col].nunique()) + \" unique values in \" + str(col) + \".\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d7f22653c46c6c8dd1bf4ecd1cbcd8d8c2a75b36"},"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n## 4. Feature engineering"},{"metadata":{"_uuid":"0c0bac984ee66a2ba2f15863d1c4c1fabd5be943"},"cell_type":"markdown","source":"1. Function to reduce memory usage from kernel: Elo World."},{"metadata":{"_uuid":"2bd078696d7d8612a2ac38959c98d6c788a22591","trusted":true},"cell_type":"code","source":"def reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a446106cd367ee4bb6235b2a5dc2502c647138ac"},"cell_type":"markdown","source":"Function to fill discrete missing values in with random sampling"},{"metadata":{"_uuid":"41d8bc082449d4acf8788b8bbff7f88072e0ec13","trusted":true},"cell_type":"code","source":"def impute_na(X_train, df, variable):\n    # make temporary df copy\n    temp = df.copy()\n    \n    # extract random from train set to fill the na\n    # temp[variable].isnull().sum() is the size of our sample\n    random_sample = X_train[variable].dropna().sample(temp[variable].isnull().sum(), random_state=1111, replace=True)\n    \n    # pandas needs to have the same index in order to merge datasets\n    random_sample.index = temp[temp[variable].isnull()].index\n    temp.loc[temp[variable].isnull(), variable] = random_sample\n    return temp[variable]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7fa718ffb6ed769780003b87b16f36a612ab88f7","trusted":false},"cell_type":"code","source":"# It was noticed that clipping the outliers does not improve the model. \n# Maybe because the tree based models that were used are robust to outliers anyway.\n'''Function to clip outliers\ndef clipping_outliers(X_train, df, var):\n    # Calculate the IQR\n    IQR = X_train[var].quantile(0.75) - X_train[var].quantile(0.25)\n    # Get the data that is located in the lower bound\n    lower_bound = X_train[var].quantile(0.25) - 6 * IQR\n    # Get the data that is located in the upper bound\n    upper_bound = X_train[var].quantile(0.75) + 6 * IQR\n    # Extract the data out of the dataframe that is located between the bounds\n    no_outliers = len(df[df[var]>upper_bound]) + len(df[df[var]<lower_bound])\n    print('There are %i outliers in %s: %.3f%%' %(no_outliers, var, no_outliers/len(df)))\n    df[var] = df[var].clip(lower_bound, upper_bound)\n    return df\n'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b5e19453bd28e80ecae754da25e5c57f08621dda"},"cell_type":"markdown","source":"**Merchants' preprocessing**"},{"metadata":{"_uuid":"33b7b5855e03e3de4a0f2c3eab08f9f3ab54dd22","scrolled":true,"trusted":false},"cell_type":"code","source":"'''# WE'RE NOT USING MERCHANTS ANYMORE\n# Merchants null\nmerchants = merchants.replace([np.inf,-np.inf], np.nan)  # How does this change the values?\nprint('Merchants null')\nprint_null(merchants)\n\n# We fill null values in the merchants data with the mean value of the column.\nnull_cols = ['avg_purchases_lag3','avg_sales_lag3', 'avg_purchases_lag6','avg_sales_lag6','avg_purchases_lag12','avg_sales_lag12']\nfor col in null_cols:\n    merchants[col] = merchants[col].fillna(merchants[col].mean())\n\n# Fill category_2 with random sampling from available data\nmerchants['category_2'] = impute_na(merchants, merchants, 'category_2')\n'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3d70f215f5c72054f43f2fdce77c8da3404f8134","trusted":true},"cell_type":"code","source":"'''# WE'RE NOT USING MERCHANTS ANYMORE\nmerchants['category_1'] = merchants['category_1'].map({'Y':1, 'N':0})\nmerchants['category_4'] = merchants['category_4'].map({'Y':1, 'N':0})\n\nmap_cols = ['most_recent_purchases_range', 'most_recent_sales_range']\nfor col in map_cols:\n    merchants[col] = merchants[col].map({'A':5,'B':4,'C':3,'D':2,'E':1})\n\nnumeric_cols = ['numerical_1','numerical_2'] + null_cols + map_cols\n\ncolormap = plt.cm.RdBu\nplt.figure(figsize=(12,12))\nsns.heatmap(merchants[numeric_cols].astype(float).corr(), linewidths=0.1, vmax=1.0, vmin=-1., square=True, cmap=colormap, linecolor='white', annot=True)\nplt.title('Pair-wise correlation')\n\nmerchants.head()\n'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ee2900542a5dbc1f61798b44a04f5caa07a790ec"},"cell_type":"markdown","source":"**Transaction data**"},{"metadata":{"_uuid":"9cf7ee3144b3337e1e9de79bc47687c3bd74ce3d"},"cell_type":"markdown","source":"Here we add new features to our transaction data. "},{"metadata":{"trusted":true,"_uuid":"2d3489d2b39806d6c0eb0e3044dc143fe597d525"},"cell_type":"code","source":"max(new_merchant_transactions['purchase_date']) # when did the last transaction happen?","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1d029fc6b714dff363cbaba66415e3359955f9b3","trusted":true},"cell_type":"code","source":"# The last date to calculate time lags from \nREF_DATE = datetime.datetime.strptime('2018-12-31', '%Y-%m-%d')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"48d5803f1a7bee6d88f3041ce669cdb97a0b05ea","trusted":true},"cell_type":"code","source":"# Create columns that calculate the number of days from the transaction day to the reference day (2018-12-31)\nhistorical_transactions['days_to_date'] = ((REF_DATE - historical_transactions['purchase_date']).dt.days) \n#historical_transactions['days_to_date'] = historical_transactions['days_to_date'] #+ df_hist_trans['month_lag']*30\nnew_merchant_transactions['days_to_date'] = ((REF_DATE - new_merchant_transactions['purchase_date']).dt.days)#//30\n\n### Here we're concatinatig historical transactions with new transactions, since they both have the same columns and form\n### and therefore do not need to be joined together. ### \ntransactions = pd.concat([historical_transactions, new_merchant_transactions])  \n\n# Create column months_ro_date: this is the number of months from transaction date to reference date (2018-12-31)\ntransactions['months_to_date'] = transactions['days_to_date']//30\ntransactions = transactions.drop(columns=['days_to_date'])\n\n# Reduce memory usage\ntransactions = reduce_mem_usage(transactions)\n\ntransactions.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"302989b01b24ca7224ecb9c6dad1f8563b5dd271","trusted":true},"cell_type":"code","source":"# We do not need the 2 dataframes anymore, beccause we have all the data needed in transactions.\ndel historical_transactions\ndel new_merchant_transactions","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d926598d394f53ecdfafebcc2d706e92911f4e1","trusted":false},"cell_type":"code","source":"'''# WE'RE NOT USING MERCHANTS ANYMORE\n# Merge trasactions with merchant data\ntransactions = pd.merge(transactions, merchants, how='left', left_on='merchant_id', right_on='merchant_id')\ntransactions.head()\n'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7e685c19168b02426a16b31c8bb9990c8a931607","trusted":true},"cell_type":"code","source":"'''# WE'RE NOT USING MERCHANTS ANYMORE\n# Take the 2 last characters out from the column names of the transactions data frame.\nt = list(transactions)\ntrans_cols = []\nfor e in t:\n    trans_cols.append(e[:-2])\n'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e216db8e0096831fa97d35ad170dd69baefeb93c","trusted":true},"cell_type":"code","source":"'''# WE'RE NOT USING MERCHANTS ANYMORE\nseen = {}\ndupes = []\n\nfor x in trans_cols:\n    if x not in seen:\n        seen[x] = 1\n    else:\n        if seen[x] == 1:\n            dupes.append(x)\n        seen[x] += 1\ndupes  # there are duplicate columns in transactions, which end in _x and _y\n'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1ef868c2384390d33cb30d9519f1d43a316c4cf0","trusted":false},"cell_type":"code","source":"'''# WE'RE NOT USING MERCHANTS ANYMORE\ntransactions = transactions.drop(columns=['category_1_y', 'category_2_y', 'city_id_y', 'state_id_y', 'merchant_category_id_y',\n                                        'merchant_category_id_y', 'subsector_id_y'])\n\ntransactions.rename(columns={'category_1_x': 'category_1', \n                            'category_2_x': 'category_2',\n                            'city_id_x': 'city_id',\n                            'state_id_x': 'state_id',\n                            'merchant_category_id_x': 'merchant_category_id',\n                            'merchant_category_id_x': 'merchant_category_id',\n                            'subsector_id_x': 'subsector_id'}, inplace=True)\n'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d46f8d6426b0928f2757686064fca28cab430dd2","trusted":true},"cell_type":"code","source":"# Null ratio\nprint('Null ratio')\nprint_null(transactions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e3c493afc59368fc5819ca5abaa1eb11081bf741"},"cell_type":"code","source":"# The function prints out the most common values of one column\ndef most_frequent(x):\n    return x.value_counts().index[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e4d8a29a3a78ce9aa27d99dcf5e78b60af408920"},"cell_type":"code","source":"print(\"merchant_id\", most_frequent(transactions['merchant_id'])) ##A:'M_ID_00a6ca8a8a'\n# print(\"category_4\", most_frequent(merchants['category_4'])) ##A:'0.0'\n# print(\"most_recent_sales_range\", most_frequent(merchants['most_recent_sales_range'])) ##A:'1.0'\n# print(\"most_recent_purchases_range: \", most_frequent(merchants['most_recent_purchases_range'])) ##A:'1.0'\nprint(\"category_2: \", most_frequent(transactions['category_2']))\nprint(\"category_3: \", most_frequent(transactions['category_3']))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d8e1900666af86de124b91017d99a1f348acc6a2","trusted":true},"cell_type":"code","source":"# Fill null by most frequent data\ntransactions['category_2'].fillna(1.0,inplace=True)\ntransactions['category_3'].fillna('A',inplace=True)\ntransactions['merchant_id'].fillna('M_ID_00a6ca8a8a',inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72f404b6e93306a377d3d3c4c0294c5e953d0dca","trusted":false},"cell_type":"code","source":"'''# WE'RE NOT USING MERCHANTS ANYMORE\n# Fill the merchant columns that have null values with random values.\n\n# nan_cols = transacations.columns[transactions.isna().any()].tolist()\nnan_cols = ['active_months_lag3','active_months_lag6','active_months_lag12','avg_purchases_lag3','avg_sales_lag3', 'avg_purchases_lag6','avg_sales_lag6','avg_purchases_lag12','avg_sales_lag12']\nfor col in nan_cols:\n    transactions[col] = impute_na(transactions, transactions, col)\n\n# merchants['category_4'].fillna(0.0,inplace=True)\n# merchants['most_recent_sales_range'].fillna(1.0,inplace=True)\n'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7193fa36e0cfb5dd97298af1fc47bedd3866d44f","trusted":true},"cell_type":"code","source":"print('Null ratio')\nprint_null(transactions) # There are no more null values in the transactions dataframe for the moment","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f00542c6fa90cc8b6ac92b29ecb9f752ddb0fd4d","trusted":true},"cell_type":"code","source":"# Encoding (Mapping/ Dummy vars)\n# Binarizing Y to 1 and N to 0\n# transactions['authorized_flag'] = transactions['authorized_flag'].map({'Y':1,'N':0})  # already done for authoried_flag\n# Category 1 has only 2 distinct values\ntransactions['category_1'] = transactions['category_1'].map({'Y':1,'N':0})\n\n\n# pd.get_dummies when applied to a column of categories where we have one category per observation \n# will produce a new column (variable) for each unique categorical value. \n# It will place a one in the column corresponding to the categorical value present for that observation.\ndummies = pd.get_dummies(transactions[['category_2', 'category_3']], prefix = ['cat_2','cat_3'], columns=['category_2','category_3'])\ntransactions = pd.concat([transactions, dummies], axis=1) # axis=1 joins all the columns\n \ntransactions.head()\ntransactions = reduce_mem_usage(transactions)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0c7aa1e840a5be46cbd1f8f1652c074d8ab97e9f"},"cell_type":"markdown","source":"\nCreate knowledge-based features:\n\n* Weekend or not\n* Hour of the day: categorize into Morning (5 to 12), Afternoon (12 to 17), Evening (17 to 22) and Night (22 to 5)\n* Day of month: categorize into Early (<10), Middle (>10 and <20) and Late (>20)\n* Time to christmas 2017, Black Friday 2017 and time to many other national holidays. Purchase amounts could increase significantly around these times."},{"metadata":{"_uuid":"652f4c3cfa534731859d5d1a203c94b45c60fe73","trusted":true},"cell_type":"code","source":"transactions['weekend'] = (transactions['purchase_date'].dt.weekday >=5).astype(int)\ntransactions['hour'] = transactions['purchase_date'].dt.hour\ntransactions['day'] = transactions['purchase_date'].dt.day\n\n# Calculate the weeks left till Christmas (2017-12-25)\ntransactions['weeks_to_Xmas_2017'] = ((pd.to_datetime('2017-12-25') - transactions['purchase_date']).dt.days//7).apply(lambda x: x if x>=0 and x<=60 else 0)\n# Calculate the weeks left till Black Friday (2017-11-25)\ntransactions['weeks_to_BFriday'] = ((pd.to_datetime('2017-11-25') - transactions['purchase_date']).dt.days//7).apply(lambda x: x if x>=0 and x<=60 else 0)\n#Mothers Day: May 14 2017 and 2018\ntransactions['Mothers_Day_2017']=(pd.to_datetime('2017-06-04')-transactions['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 60 else 0)\ntransactions['Mothers_Day_2018']=(pd.to_datetime('2018-05-13')-transactions['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 60 else 0)\n#fathers day: August 13 2017\ntransactions['Fathers_day_2017']=(pd.to_datetime('2017-08-13')-transactions['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 60 else 0)\n#Childrens day: October 12 2017\ntransactions['Children_day_2017']=(pd.to_datetime('2017-10-12')-transactions['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 60 else 0)\n#Valentine's Day : 12th June, 2017\ntransactions['Valentine_Day_2017']=(pd.to_datetime('2017-06-12')-transactions['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 60 else 0)\n# Carnival in Brasil 27.02.2017 - 28.02.2017\ntransactions['Carnival_2017']=(pd.to_datetime('2017-02-27')-transactions['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 60 else 0)\n# Carnival in Brasil 09.02.2018\ntransactions['Carnival_2018']=(pd.to_datetime('2018-02-09')-transactions['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 60 else 0)\n# Easter: 14.04.2017 - Restaurants\ntransactions['Easter_2017']=(pd.to_datetime('2017-04-14')-transactions['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 60 else 0)\n# Tiradentes: 21.04.2017 - Restaurants\ntransactions['Tiradentes_2017']=(pd.to_datetime('2017-04-21')-transactions['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 60 else 0)\n# Labour day: 01.05.2017 - Restaurants\ntransactions['Labour_day_2017']=(pd.to_datetime('2017-05-01')-transactions['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 60 else 0)\n# Independence day 01.09.2017 - Restaurants\ntransactions['Independence_day_2017']=(pd.to_datetime('2017-09-01')-transactions['purchase_date']).dt.days.apply(lambda x: x if x > 0 and x < 60 else 0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e8133812eb0d3860a15020e928c4868e32cdc264","trusted":true},"cell_type":"code","source":"# Categorize time in 4 categories: (0) from 5 to 11, (1) from 12 to 16, (2) from 17 to 20 and (3) from 21 to 4\n#Hypothesis: when do people go out to restaurants?\ndef get_session(hour):\n    hour = int(hour)\n    if hour > 4 and hour < 12:\n        return 0\n    elif hour >= 12 and hour < 17:\n        return 1\n    elif hour >= 17 and hour < 21:\n        return 2\n    else:\n        return 3\n    \ntransactions['hour'] = transactions['hour'].apply(lambda x: get_session(x))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"416a83717623d7a9f728dcc6f5192becdfa47752","trusted":true},"cell_type":"code","source":"# Categorize day in 3 categories: (0) 0 to 10, (1) 11 to 20  and (2) over 20\n# Hypothesis: People have more money or less money in the beginning or end of the month.\ndef get_day(day):\n    if day <= 10:\n        return 0\n    elif day <=20:\n        return 1\n    else:\n        return 2\n\ntransactions['day'] = transactions['day'].apply(lambda x: get_day(x))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"14b71deecd06f366c4d852153f5b903933553437","trusted":true},"cell_type":"code","source":"transactions.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e1d98f483983f259551bc5be902f06c236c6c7f2"},"cell_type":"markdown","source":"**Aggregating features**"},{"metadata":{"_uuid":"e3464e97d2d3e6975b9faa66e912e5c93fd6df06"},"cell_type":"markdown","source":"Aggregate other columns of the transactions dataframe grouping by the card_id."},{"metadata":{"_uuid":"1a0b4e93c24fab3258dfe123b5c71b40c6e0ae45","trusted":true},"cell_type":"code","source":"def aggregate_trans(df):\n    agg_func = {\n        'authorized_flag': ['mean', 'std'],\n        'category_1': ['mean'],\n        'cat_2_1.0': ['mean'],\n        'cat_2_2.0': ['mean'],\n        'cat_2_3.0': ['mean'],\n        'cat_2_4.0': ['mean'],\n        'cat_2_5.0': ['mean'],\n        'cat_3_A': ['mean'],\n        'cat_3_B': ['mean'],\n        'cat_3_C': ['mean'],\n        ###'numerical_1':['nunique','mean','std'], # merchants\n        #'most_recent_sales_range': ['mean','std'], # merchants\n        #'most_recent_purchases_range': ['mean','std'], # merchants\n        ###'avg_sales_lag12':['mean','std'], # merchants\n        ###'avg_purchases_lag12':['mean','std'], # merchants\n        ###'active_months_lag12':['nunique'], # merchants\n        'merchant_id': ['nunique'],\n        'merchant_category_id': ['nunique'],  # counts unique values of id for the rows that were groupped by card_id\n        ###'state_id': ['nunique'], #\n        'city_id': ['nunique'],\n        ###'subsector_id': ['nunique'], # merchants\n        ###'merchant_group_id': ['nunique'], # merchants\n        'installments': ['sum','mean', 'max', 'min', 'std'],\n        'purchase_amount': ['sum', 'mean', 'max', 'min', 'std'],\n        'weekend': ['mean', 'std'],\n        'hour': ['mean', 'std'],\n        'day': ['mean', 'std'],\n        'weeks_to_Xmas_2017': ['mean', 'sum'],\n        'weeks_to_BFriday': ['mean', 'sum'],\n        'purchase_date': ['count'],\n        'months_to_date': ['mean', 'max', 'min', 'std'],\n        'Mothers_Day_2017': ['mean', 'sum'],\n        'Fathers_day_2017': ['mean', 'sum'],\n        'Children_day_2017': ['mean', 'sum'],\n        'Valentine_Day_2017': ['mean', 'sum'],\n        'Mothers_Day_2018': ['mean', 'sum'],\n        'Labour_day_2017': ['mean', 'sum'],\n        'Independence_day_2017': ['mean', 'sum'],\n        'Easter_2017': ['mean', 'sum'],\n        'Tiradentes_2017': ['mean', 'sum'],\n        'Carnival_2017': ['mean', 'sum'],\n        'Carnival_2018': ['mean', 'sum']\n    }\n    #'mer_category_4': ['mean'],\n    #'mer_avg_sales_lag6':['nunique', 'mean','std'],\n    #'mer_avg_purchases_lag6':['nunique', 'mean','std'],\n    #'months_to_date': ['mean', 'max', 'min', 'std'],\n    agg_df = df.groupby(['card_id']).agg(agg_func)\n    agg_df.columns = ['_'.join(col)for col in agg_df.columns.values]\n    agg_df.reset_index(inplace=True)\n    return agg_df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eae53ad599f5d89a673fe1316a7ab567cbaa08fe"},"cell_type":"markdown","source":"Aggregate transactions by card_id and month_lag."},{"metadata":{"trusted":true,"_uuid":"cf67bdf8b92cc4b95e03a4d7429d2928c3595166"},"cell_type":"code","source":"def aggregate_per_month(history):\n    \n    # Group the dataframe by card_id and month_lag\n    grouped = history.groupby(['card_id', 'month_lag'])\n    # Convert the data type of the column installments to integer\n    history['installments'] = history['installments'].astype(int)\n    # Add aggregate functions count, sum, mean, min, max, std to the dataframe\n    # agg_func is a dictionary that assigns the aggregate functions to the columns they will be applied on\n    agg_func = {\n            'purchase_amount': ['count', 'sum', 'mean', 'min', 'max', 'std'],\n            'installments': ['count', 'sum', 'mean', 'min', 'max', 'std'],\n            }\n\n    #Aggregate using The above mentioned functions over the dictionary keys (purchase_amount, installments).\n    intermediate_group = grouped.agg(agg_func)\n    # Rename the columns add '-' between column name and aggregate function name\n    intermediate_group.columns = ['_'.join(col).strip() for col in intermediate_group.columns.values]\n    # Reset the index of the dataframe after aggregating\n    intermediate_group.reset_index(inplace=True)\n\n    # Group by card_id and add functions mean and std\n    end_df = intermediate_group.groupby('card_id').agg(['mean', 'std'])\n    end_df.columns = ['_'.join(col).strip() for col in end_df.columns.values]\n    end_df.reset_index(inplace=True)\n    \n    return end_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d33ff6f54328a2dfe4e76ce902feac908258e34d"},"cell_type":"code","source":"# Aggregate transactions\nagg_transactions = aggregate_trans(transactions)\nagg_transactions_permonth = aggregate_per_month(transactions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"626c1cb6652b70e7aefdd400f75ab4ce95834a1a"},"cell_type":"code","source":"# Merge aggregated transactions\nagg_trans = pd.merge(agg_transactions, agg_transactions_permonth, how='left', on='card_id')\nagg_trans = reduce_mem_usage(agg_trans)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9617623f790cddd169dcb66b2e5da53320529166"},"cell_type":"code","source":"agg_trans.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8b2186497dbdb0a28aececf42e4ec8f7d4705009"},"cell_type":"code","source":"# Delete agg_transactions and agg_transactions_per_month because they were already merged together\ndel agg_transactions\ndel agg_transactions_permonth","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d556a29742ceb996e472602a6fefb867a8381be6"},"cell_type":"code","source":"# Columns that have na values in them\nnan_cols = agg_trans.columns[agg_trans.isna().any()].tolist()\nnan_cols","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"743fdc5a6746cc6c3383f06a739ad6a847e2fce5"},"cell_type":"code","source":"## Replace infinite values with their\nagg_trans = agg_trans.replace([np.inf,-np.inf], np.nan)\n#agg_trans = agg_trans.fillna(value=0)  # Take the mean?\n\n# agg_trans.mean(): calculates the mean of every column of the dataframe\n#agg_trans = agg_trans.fillna(value=agg_trans.mean())\n# It works faster when definined the columns containing nan values\nagg_trans[nan_cols] = agg_trans[nan_cols].fillna(value=agg_trans[nan_cols].mean())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"81dd03f18de4e4672788b72f92f56eb5725d1a96"},"cell_type":"markdown","source":"<a id=\"5\"></a> <br>\n## 5. Train and Test"},{"metadata":{"_uuid":"ecce5c3ced643238b8ee22fd18259312f4834f6b","trusted":false},"cell_type":"code","source":"# Droping outliers worsened the performance\n# What if we drop outliers from target variable\n#train['outliers'] = 0\n#train.loc[train['target'] < -30, 'outliers'] = 1\n#train['outliers'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"25577e2710e9785f3e24de0cd44751885e6ed9f1","trusted":false},"cell_type":"code","source":"#train = train[train.outliers != 1]\n#train = train.drop(columns=['outliers'])\n#train.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f219ed6e52e9143aa934b99679517db567654b5d","trusted":true},"cell_type":"code","source":"# Add random values to the null values of the column\ntest['first_active_month'] = impute_na(test, train, 'first_active_month')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4b7ca6616503057c4af20c0898abc8bf0e08e1bd"},"cell_type":"code","source":"# Merge the train and test dataframes with the agg_trans dataframe\ntrain = pd.merge(train, agg_trans, on='card_id', how='left')\ntest = pd.merge(test, agg_trans, on='card_id', how='left')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3131cd446eb38c3b10326777729b501b26ca14a7","trusted":true},"cell_type":"code","source":"# Create columns year and month out of the column first_active_month \ntrain[\"year\"] = train[\"first_active_month\"].dt.year\ntest[\"year\"] = test[\"first_active_month\"].dt.year\ntrain[\"month\"] = train[\"first_active_month\"].dt.month\ntest[\"month\"] = test[\"first_active_month\"].dt.month","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a2cc8f305212b82f8aa64693b29f9424cac775a","trusted":true},"cell_type":"code","source":"# Get numerical features\nnumerical = [var for var in train.columns if train[var].dtype!='O']\nprint('There are {} numerical variables'.format(len(numerical)))\n\n# Get discrete features\ndiscrete = []\nfor var in numerical:\n    if len(train[var].unique())<8:\n        discrete.append(var)\n        \nprint('There are {} discrete variables'.format(len(discrete)))\n\n# Get continuous features\ncontinuous = [var for var in numerical if var not in discrete and var not in ['card_id', 'first_active_month','target']]\nprint('There are {} continuous variables'.format(len(continuous)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"21d92ac3a025e27e327ca3c5ec26e55a1b58b634","trusted":true},"cell_type":"code","source":"# Detect all null columns\ntrain_null = train.columns[train.isnull().any()].tolist()\ntest_null = test.columns[test.isnull().any()].tolist()\n\n# Get a set out of the null columns. The set only contains unique values.\nin_first = set(train_null)\nin_second = set(test_null)\n\n# Get columns that are in the test dataframe but not in the train dataframe\nin_second_but_not_in_first = in_second - in_first\n\n# Create list of null columns\nnull_cols = train_null + list(in_second_but_not_in_first)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"368c32ae9400d74da7d950c341ed82af3342d330","trusted":true},"cell_type":"code","source":"# Filling null\nfor col in null_cols:\n    if col in continuous:\n        # if it is a continuous column, fill with 0-s\n        train[col] = train[col].fillna(0)#df_train[col].astype(float).mean())\n        test[col] = test[col].fillna(0)#df_train[col].astype(float).mean())\n    if col in discrete:\n        # if it is a descrete columns fill with random values\n        train[col] = impute_na(train, train, col)\n        test[col] = impute_na(test, train, col)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c642f27aa9df0994cee7aa57cb1f668bed4c0d2","trusted":true},"cell_type":"code","source":"print('Final null')\n# There are no more null values in the dataframes\nprint_null(train)\nprint_null(test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"de8bbc3c0c14566a865a3ae18aa2a229daefbafd","trusted":true},"cell_type":"code","source":"# Take card_id, first_active_month and target out of the list of the columns to use for training models, so that no errors are caused.\ncols_to_use = list(train)\ncols_to_use.remove('card_id')\ncols_to_use.remove('first_active_month')\ncols_to_use.remove('target')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a4e3473d29e904d4790cb40d908d2de72f77b737"},"cell_type":"markdown","source":"<a id=\"6\"></a> <br>\n## 6. Ridge and Lasso"},{"metadata":{"_uuid":"112e0860d8606f16db882e5633a9c767103405f7"},"cell_type":"markdown","source":"**RidgeCV**"},{"metadata":{"_uuid":"45311dc10619a31aa71b8b1473a5b0ce03074bd3","trusted":true},"cell_type":"code","source":"# Take card_id, first_active_month and target out of the list of the columns to use for training models, so that no errors are caused.\nnames = list(train)\nnames.remove('card_id')\nnames.remove('first_active_month')\nnames.remove('target')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d688ac141cb33615778dda5bf13f152fb095337","trusted":true},"cell_type":"code","source":"# Define a function that returns the cross-validation rmse error \ndef rmse_cv(model):\n    rmse= np.sqrt(-cross_val_score(model, train[names], train['target'], scoring=\"neg_mean_squared_error\", cv = 5))\n    return(rmse)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e0a58f9ec0ed51ee7d44c54d31d4e3c2a268b2e4","trusted":true},"cell_type":"code","source":"alphas = [0.05, 0.1, 0.3, 1, 3, 5, 10, 15, 30, 50, 75]\ncv_ridge = [rmse_cv(Ridge(alpha = alpha)).mean() \n            for alpha in alphas]\n\ncv_ridge = pd.Series(cv_ridge, index = alphas)\ncv_ridge.plot(title = \"Validation - Just Do It\")\nplt.xlabel(\"alpha\")\nplt.ylabel(\"rmse\")\n#cv_ridge.min()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9607b6198e1c8ed844527ca8fef9b4f5f8e8435f","trusted":true},"cell_type":"code","source":"# Fit the training data to RidgeCV model\nridgeCV = RidgeCV(alphas = [0.05, 0.1, 0.3, 1, 3, 5, 10, 15, 30, 50, 75]).fit(train[names], train['target'])\nrmse_cv(ridgeCV).mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a0664e22d5456aaf5a392790f747b43e9961fa90","trusted":true},"cell_type":"code","source":"# Predict the loyalty score of the test data\nridgeCV_pred = ridgeCV.predict(test[names])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"19053212409521452af8238804eae1e14297d244","trusted":true},"cell_type":"code","source":"#Submitting the prediction of the ridgecv regression.\nsubmit = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\nsubmit[\"target\"] = ridgeCV_pred\nsubmit.to_csv(\"elo_submission_ridgeCV.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fe55899a1f4f1d4bc2822350c89e259961fd2fb0"},"cell_type":"markdown","source":"**LassoCV**"},{"metadata":{"_uuid":"8f4d1f95cbb1d61ce2b43608cb65aa8b00159ec8","trusted":true},"cell_type":"code","source":"# Fit the training data to LassoCV model\nlassoCV = LassoCV(alphas = [1, 0.1, 0.001, 0.0005]).fit(train[names], train['target'])\nrmse_cv(lassoCV).mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dfee9c74e1af9546ac5cfe27ca5ed4eb4ec3ef0d","trusted":true},"cell_type":"code","source":"# Predict the loyalty score of the test data\nlassoCV_pred = lassoCV.predict(test[names])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa7329ca9f9937de76bec063bc58284f5becc7a5","trusted":true},"cell_type":"code","source":"# Submitting the prediction of the lasso regression.\nsubmit = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\nsubmit[\"target\"] = lassoCV_pred\nsubmit.to_csv(\"elo_submission_lasso.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7aad682c2d859712d4e99b71742bec0bad823be6","trusted":true},"cell_type":"code","source":"# Take a look at the coefficients\ncoef = pd.Series(lassoCV.coef_, index = train[names].columns)\n\nprint(\"Lasso picked \" + str(sum(coef != 0)) + \" variables and eliminated the other \" +  str(sum(coef == 0)) + \" variables\")\n\n# We are taking the first (highest) and last (lowest) 10 values of the coeffiecients and then we plot them.\nimp_coef = pd.concat([coef.sort_values().head(10),coef.sort_values().tail(10)])\n\nplt.rcParams['figure.figsize'] = (8.0, 10.0)\nimp_coef.plot(kind = \"barh\")\nplt.title(\"Coefficients in the Lasso Model\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"786dba87af4f0a5fbf7e698e5619f7ab58d0d308"},"cell_type":"markdown","source":"<a id=\"7\"></a> <br>\n## 7. Regression Tree"},{"metadata":{"_uuid":"78d9c4e52acfa47c81c897e844ef0f9e0cd82c26"},"cell_type":"markdown","source":"Predict with a decesion tree regression.\n\nThe decision tree is used to fit the data with addition noisy observation. As a result, it learns local linear regressions approximating the fitted curve.\nParameter max_depth: The maximum depth of the tree. If none, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples. If the maximum depth of the tree is set too high, then the decision trees learn too fine details of the training data, so they learn from the noise and thus overfit."},{"metadata":{"_uuid":"34623e4247b0b3df25059f923f5fe4b3421d9352","trusted":true},"cell_type":"code","source":"# Choosing max depth 3\n# Fit the training data to the Regression Tree model\nregr2 = DecisionTreeRegressor(max_depth=3)\nregr2.fit(train[names], train['target'])\n\n# Predict the loyalty score of the test data\npred_tree = regr2.predict(test[names])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"12473ab095d328832822fe84d2be19fe9414f600","trusted":true},"cell_type":"code","source":"# Submitting the prediction of a regression decision tree.\nsubmit = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\nsubmit[\"target\"] = pred_tree\nsubmit.to_csv(\"elo_submission_regression_tree.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9b0902fbe1da8d54a870f601dd9141904589201d","trusted":true},"cell_type":"code","source":"# This function creates images of tree models using pydot\ndef print_tree(estimator, features, class_names=None, filled=True):\n    tree = estimator\n    names = features\n    color = filled\n    classn = class_names\n    \n    dot_data = StringIO()\n    export_graphviz(estimator, out_file=dot_data, feature_names=features, class_names=classn, filled=filled)\n    graph = pydot.graph_from_dot_data(dot_data.getvalue())\n    return(graph)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d13bec54eff1f9a45175facf832f74b244913205","trusted":true},"cell_type":"code","source":"# Print tree\ngraph, = print_tree(regr2, features=cols_to_use)\nImage(graph.create_png())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e064ee50809b4714c1f193150ac6b2d4817c2599"},"cell_type":"markdown","source":"<a id=\"8\"></a> <br>\n## 8. Random Forest"},{"metadata":{"_uuid":"d1269c635cb508c11c16ef0735b9719522683fa1"},"cell_type":"markdown","source":"\n> Random forest is an ensemble method. An ensemble method consists of aggregating multiple outputs made by a diverse set of predictors to obtain better results. The purpose of these methods is to average out the outcome of individual predictions by diversifying the set of predictors, thus lowering the variance, to arrive at a powerful prediction model that reduces overfitting the training set.\n\n> The random forest is an ensemble of Decision Trees (weak learners). They are trained via the bagging method. Bagging or Bootstrap Aggregating, consists of randomly sampling subsets of the training data, fitting a model to these smaller data sets, and aggregating the predictions. This method allows several instances to be used repeatedly for the training stage given that we are sampling with replacement. Tree bagging consists of sampling subsets of the training set, fitting a Decision Tree to each, and aggregating their result.\nRandom forest introduces more randomness by applying the bagging method to the feature space. Instead of searching greedily for the best predictors to create branches, it randomly samples the elements of the predictor space, thus adding more diversity and reducing the variance of the trees at the cost of equal or higher bias (The Variance-Bias Trade-off). This process is known as \"feature bagging\". [Random Forest Source](http://www.kdnuggets.com/2017/10/random-forests-explained.html)"},{"metadata":{"_uuid":"c7ff8a359bcc907f17fee8a500f4445a6fd25b5c","trusted":true},"cell_type":"code","source":"# Fit the training data to Random Forest model\nregr1 = RandomForestRegressor(max_features=13, random_state=1)\nregr1.fit(train[names], train['target'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b5390883bbf98972060089607eca115853f125e","trusted":true},"cell_type":"code","source":"# Predict the loyalty score of the test data\npred_forest = regr1.predict(test[names])\n\n# Submitting the prediction of a random forest now.\nsubmit = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\nsubmit[\"target\"] = pred_forest\nsubmit.to_csv(\"elo_submission_random_forest_13.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"06e0646d3aeb35f05ca386ced13ee61eb07d10ef","trusted":true},"cell_type":"code","source":"# Print feature importance\nfig, ax = plt.subplots(figsize=(12,27))\nImportance = pd.DataFrame({'Importance':regr1.feature_importances_*100}, index=train[names].columns)\nImportance.sort_values('Importance', axis=0, ascending=True).plot(kind='barh', color='g', ax=ax )\nplt.xlabel('Variable Importance')\nplt.gca().legend_ = None","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5a5cf440ccb788b467c7745d554faad1f5f94758"},"cell_type":"markdown","source":"<a id=\"9\"></a> <br>\n## 9. Boosting"},{"metadata":{"_uuid":"5a36d2a038920505e08205ac38e766abc1d50250"},"cell_type":"markdown","source":"> - Unlike fitting a single large decision tree to the data, which amounts to fitting the data hard and potentially overfitting, the boosting approach instead learns slowly.\n> - Given the current model, we fit a decision tree to the residuals from the model.  We then add this new decision tree into the fitted function in order to update the residuals.\n> - Each of these trees can be rather small, with just a few terminal nodes, determined by the parameter d in the algorithm.\n> - By fitting small trees to the residuals, we slowly improve ˆf in areas where it does not perform well.  The shrinkage parameter λ slows the process down even further, allowing more and different shaped trees to attack the residuals.\n[Source](https://lagunita.stanford.edu/c4x/HumanitiesScience/StatLearning/asset/trees.pdf)\n"},{"metadata":{"_uuid":"56618b31a63b29fb76114394e142f64060adf790","trusted":true},"cell_type":"code","source":"# Fit the training data to Boosting model\n# n_estimators: The number of boosting stages to perform.\nregr_b = GradientBoostingRegressor(n_estimators=600, learning_rate=0.005, random_state=1)\nregr_b.fit(train[names], train['target'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0d5f1585a883d7d9369050262bdb30c3b93c898e","trusted":true},"cell_type":"code","source":"# Print feature importance\nfig, ax = plt.subplots(figsize=(12,27))\nfeature_importance = regr_b.feature_importances_*100\nrel_imp = pd.Series(feature_importance, index=train[names].columns).sort_values(inplace=False)\nprint(rel_imp)\nrel_imp.T.plot(kind='barh', color='r', ax=ax)\nplt.xlabel('Variable Importance')\nplt.gca().legend_ = None","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"089e09a23573b1af7813ddf203108949241e1bed","trusted":true},"cell_type":"code","source":"# Predict the loyalty score of the test data\npred_b = regr_b.predict(test[names])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5ad199e2b93ca1211b5ba286db395063aa7f3bc0","trusted":true},"cell_type":"code","source":"# Submitting the prediction of a boosting method.\nsubmit = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\nsubmit[\"target\"] = pred_b","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c4b6c403186b538fd01cf9492ce69f6e190d757b"},"cell_type":"markdown","source":"<a id=\"10\"></a> <br>\n## 10. Lightgbm"},{"metadata":{"_uuid":"0370efb37ad6cc0d196e0e18cd4a0023161b7c2a"},"cell_type":"markdown","source":">Light GBM is a gradient boosting framework that uses tree-based learning algorithm. [Light GBM source](https://medium.com/@pushkarmandot/https-medium-com-pushkarmandot-what-is-lightgbm-how-to-implement-it-how-to-fine-tune-the-parameters-60347819b7fc)\n\n>Light GBM grows the trees vertically (leaf-wise), while other tree-algorithms grow horizontally (level-wise). It will choose the leaf with max delta loss to grow. The algorithm can handle large amounts of data and takes lower memory to run. It focuses on accuracy of results. Light GBM is sensitive to overfitting and thus it is not advisable to use it with small amounts of data. [Light GBM source](https://medium.com/@pushkarmandot/https-medium-com-pushkarmandot-what-is-lightgbm-how-to-implement-it-how-to-fine-tune-the-parameters-60347819b7fc)\n\n>It covers more than 100 parameters and we couldn't possibly look at them all. Here are the main ones:\n\n> * num_leaves: number of leaves in full tree (default: 31)\n> * min_data_in_leaf: the minimum number (default value: 20) of records a leaf could have. It also could help with overfitting.\n> * max_depth: describes the maximum depth of the tree. It handles model overfitting. When the model is overfitted, lowering the max_depth could help.\n> * learning_rate: determines the impact of each tree on the final outcome. It slows down the algorithm to learn more. It controls the magnitude of the change each new tree brings to the estimate. \n> * boosting: defines the type of algorithm to run. Default: gdbt (gradient boosting decision tree). There also is rf (random forest), dart (dropouts meet multiple additive regression trees), gross (gradient-based one-side sampling)\n> * feature_fraction: used when the boosting is random forest. 0.8 feature fraction means LightGBM will select 80% of parameters randomly in each iteration for building trees.\n> * bagging_fraction: specifies the fraction of the data to be used for each iteration and is generally used to speed up training and avoid overfitting.\n> * bagging_freq: is used for faster speed\n> * metric: specifies loss for model building\n> * lambda_l1: specifies regularization. Typical value ranges from 0 to 1.\n"},{"metadata":{"_uuid":"43a821099289f3e33d0c2c8b0ec6bdf6bf1b71f5","trusted":true},"cell_type":"code","source":"def run_lgb(train_X, train_y, val_X, val_y, test_X):\n    \n    # Define the model parameters    \n    params = {'num_leaves': 111,\n             'min_data_in_leaf': 150,  # was 149 \n             'objective':'regression',\n             'max_depth': 9,\n             'learning_rate': 0.005,\n             \"boosting\": \"gbdt\",\n             \"feature_fraction\": 0.75,\n             \"bagging_freq\": 1,\n             \"bagging_fraction\": 0.70,\n             \"bagging_seed\": 11,\n             \"metric\": 'rmse',\n             \"lambda_l1\": 0.25,  # was 0.26\n             \"random_state\": 1111,\n             \"verbosity\": -1}\n\n    # Convert train dataframe to a dataset\n    lgtrain = lgb.Dataset(train_X, label=train_y)\n    lgval = lgb.Dataset(val_X, label=val_y)\n    evals_result = {}\n    # Fit the training data to the lgb model\n    model = lgb.train(params, lgtrain, 10000, valid_sets=[lgval], early_stopping_rounds=100, verbose_eval=100, evals_result=evals_result)\n    # Predict the loyalty score of the test data\n    pred_test_y = model.predict(test_X, num_iteration=model.best_iteration)\n    return pred_test_y, model, evals_result\n\ntrain_X = train[names]\ntest_X = test[names]\ntrain_y = train.target.values\n\npred_test = 0  # Initialize pred_test\n\n# Running a k-fold cross validation\nkf = model_selection.KFold(n_splits=5, random_state=1111, shuffle=True)\nfor dev_index, val_index in kf.split(train):\n    dev_X, val_X = train_X.loc[dev_index,:], train_X.loc[val_index,:]\n    dev_y, val_y = train_y[dev_index], train_y[val_index]\n    \n    pred_test_tmp, model, evals_result = run_lgb(dev_X, dev_y, val_X, val_y, test_X)\n    pred_test += pred_test_tmp\npred_test /= 5. ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3452c9d8be459663e3fda29d6f820dda87df14ce","trusted":true},"cell_type":"code","source":"# Plotting the first 50 features with highest importance\nfig, ax = plt.subplots(figsize=(12,10))\nlgb.plot_importance(model, max_num_features=50, height=0.8, ax=ax)\nax.grid(False)\nplt.title(\"LightGBM - Feature Importance\", fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"27d6c3bbe5fb03dfbf8a7532eac8f49681fa81ec","trusted":true},"cell_type":"code","source":"#Submitting the prediction of the lighgbm model.\nsubmit = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\nsubmit[\"target\"] = pred_test\nsubmit.to_csv(\"elo_submission_lightgbm.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bfc9cef6406df3c919efe3a487a1efc3b6fdd63c"},"cell_type":"markdown","source":"<a id=\"11\"></a> <br>\n## 11. Model Performance Comparison and Conclusion"},{"metadata":{"_uuid":"e435499a59e3b6d036e91c87c637e2c262919e1e","trusted":true},"cell_type":"code","source":"model_results = pd.read_excel('../input/model-results/submissions_table.xlsx')\nmodel_results","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a6ac7be63b28377c6a5019c5b9be1571894cecc7"},"cell_type":"markdown","source":"In order to predict the loyalty score of Elo customers we conducted EDA, followed by pre-processing, then engineered additional features and lastly we fitted the data to different models.\n\nAfter the testing of these various models, the LightGBM gave us the best RMSE, thus we used exatlly this model for our final submission.\n\nWe saw that a few features have higher importnace than the others and namely, the features that handle time. Even more particualrly months_lag and all its engineered aggregate features scored high importance.\n\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}