{"cells":[{"metadata":{"_uuid":"784e598604c977d9c1f55e6f564148b7da7723a2"},"cell_type":"markdown","source":"\n\n---\n<h1><center> ELO Merchant Category Recommendation </center></h1>\n\n---\n\n<h2><center> Comparing Baseline Model Performance to Stacked Models  </center></h2>\n   <h2><center> LGBM, XGBoost, RF and Stacking (LB & CV) </center><h2>\n\n--- \n\nThe purpose of this notebook is to provide an example of the improvements that can be made to a base LightGBM Model by stacking with other models. To summarize, the base LGBM model scores around a 3.713 on the public LB while the XGBoost and RF Model score 3.718 and 3.752 respectively. Despite these models performing worse than the LightGBM stacking them together provides a small improvement to both CV and LB reaching a result of around 3.708."},{"metadata":{"trusted":false,"_uuid":"6a6376be983e3c1770a0b32d2e01dadd96720f14"},"cell_type":"code","source":"# Basic Packages\nimport pandas as pd\nimport numpy as np\nimport datetime\nimport time\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# plotting packages\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n\n# machine learning packages\nimport xgboost as xgb\nimport lightgbm as lgb\nfrom sklearn import model_selection, preprocessing, metrics\nfrom sklearn.preprocessing import Imputer\npd.set_option('display.max_columns',None)\nfrom sklearn.linear_model import RidgeCV\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.ensemble import GradientBoostingRegressor\nfrom sklearn import linear_model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"335f6d7414e415418c9f0f55afa9287215f075cf"},"cell_type":"markdown","source":"# Loading Data\n\n## Train and Test Sets:"},{"metadata":{"trusted":false,"_uuid":"16cc77d7e9985463960851615523fb3bb8b26bb4"},"cell_type":"code","source":"dtypes = {'feature_1':'int16',\n          'feature_2':'int16',\n          'feature_3':'int16'\n         }\n\ntrain_df = pd.read_csv(\"../input/train.csv\",dtype=dtypes,parse_dates=['first_active_month'])\ntest_df = pd.read_csv(\"../input/test.csv\",dtype=dtypes,parse_dates=['first_active_month'])\n\nprint(f\"Training Set Shape:{train_df.shape}\")\nprint(f\"Test Set Shape:{test_df.shape}\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b3e60eae3aaa03256cb9f8a39f360f0132931d63"},"cell_type":"markdown","source":"## Loading Historical and New Transactions"},{"metadata":{"trusted":false,"_uuid":"5cc129235ea9c60f8d712d473275b3d883ded3b9"},"cell_type":"code","source":"# Using smaller data types reduces the memory usage by ~50%\ndata_types = {'authorized_flag':'str',\n                   'card_id':'str',\n                   'city_id':'int16',\n                   'category_1':'str',\n                   'installments':'int16',\n                   'category_3':'str',\n                   'merchant_category_id':'int16',\n                   'merchant_id':'str',\n                   'purchase_amount':'float',\n                   'category_2':'str',\n                   'state_id':'int16',\n                   'subsector_id':'int16'}\n\nhist_df = pd.read_csv(\"../input/historical_transactions.csv\",dtype=data_types,parse_dates=True)\nnew_trans_df = pd.read_csv('../input/new_merchant_transactions.csv',dtype=data_types,parse_dates=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e09d71497ac94f39a4788ac989240d7d00f8fe02"},"cell_type":"markdown","source":"# Preprocessing data"},{"metadata":{"trusted":false,"_uuid":"d12de8ca152912ed1adc67442a373b8b55ee450a"},"cell_type":"code","source":"last_hist_date = datetime.datetime(2018,2,28)\nfor df in [new_trans_df,hist_df]:\n    print(f'Preprocessing DataFrame...')\n    df['authorized_flag'] = df['authorized_flag'].map({'Y':1,'N':0}).astype('bool')\n    df['category_1'] = df['category_1'].map({'Y':1,'N':0}).astype('bool')\n    df['purchase_date'] = pd.to_datetime(df['purchase_date'])\n    df['time_since_purchase_date'] = (last_hist_date-df['purchase_date']).dt.days\n    #df.loc[:,'purchase_date'] = pd.DatetimeIndex(df['purchase_date']).astype(np.int64)*1**(-9)\n    df['installments'] = df['installments'].replace(999,-1)\n    null_cols = ['city_id','state_id','subsector_id','installments']\n    nan_cols = ['city_id','state_id','subsector_id','installments','merchant_id','category_3','category_2']\n    \n    # Identify -1 values as nans\n    for col in null_cols:\n        df[col] = df[col].replace(-1,np.nan)\n    \n    # Fill categorical nan values with mode\n    for column in nan_cols:\n        fill = df.loc[:,column].mode().values[0]\n        df[column].fillna(fill,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d4b56125f11fb543f9bced586ad660b2d438d55a"},"cell_type":"markdown","source":"# Feature Engineering\n\nChau Huynh's \"My First Kernel\" was a starting point of this competition for me so I borrow a lot from him in this section.\nhttps://www.kaggle.com/chauhuynh/my-first-kernel-3-699/\n\n\n## Encoding Date Times"},{"metadata":{"trusted":false,"_uuid":"1c43dfbc258a607a5eb1ca5ea3823cafc3e9d1c7"},"cell_type":"code","source":"print('Encoding Date Times...')\nfor df in [hist_df,new_trans_df]:\n    print('...')\n    df['year'] = df['purchase_date'].dt.year.astype('int16')\n    df['weekofyear'] = df['purchase_date'].dt.month.astype('int16')\n    df['weekend'] = (df.purchase_date.dt.weekday >=5).astype('bool')\n    df['hour'] = df['purchase_date'].dt.hour.astype('int16')\n\n    #https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/73244\n    df['month_diff'] = (((datetime.datetime.today()-df['purchase_date']).dt.days)//30).astype('int16')\n    df['month_diff'] += df['month_lag']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a4a1a5c07f666dc01733dfb452260b3541fae5ad"},"cell_type":"markdown","source":"# Adding Aggregate Features"},{"metadata":{"_uuid":"7378d597c55d2aa7c1819e7ecac3bb1aba5897fa"},"cell_type":"markdown","source":"## Aggregating Historical Transactions"},{"metadata":{"trusted":false,"_uuid":"94142e923942cfd3a57628f9dd34b86eba83f349"},"cell_type":"code","source":"# Aggregating Historical Transactions DataFrame by card_id\n\nprint('Aggregating Historical Transactions...')\n\n# Create dictionary of column names and aggregation functions to use\nagg_func = {'authorized_flag' : ['mean'],\n            'city_id' : ['nunique'],\n            'category_1' : ['sum','mean'],\n            'installments': ['sum','min','max','var','mean'],\n            'category_3' : ['nunique'],\n            'merchant_category_id':['nunique'],\n            'merchant_id':['nunique'],\n            'purchase_amount':['sum','mean','max','min','var'],\n            'purchase_date':['max','min'],\n            'time_since_purchase_date':['min','max','mean'],\n            'category_2':['nunique'],\n            'weekend':['sum','mean'],\n            'month_lag':['min','max','mean','var'],\n            'month_diff':['mean','var']\n           }\n\n# Aggregate columns based on dictionary passed to agg function\nghist_df = hist_df.groupby(['card_id']).agg(agg_func)\n\n# Rename columns before joining train/test set\nghist_df.columns = ['hist_'+'_'.join(col).strip() for col in ghist_df.columns.values]\nghist_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a69d0f5c319350364fb8a293602d543d45f64c25"},"cell_type":"markdown","source":"## Aggregating New Transactions Dataframe"},{"metadata":{"trusted":false,"_uuid":"ceb6dbafc9218e5b71756b448f342ad65eddbbf4"},"cell_type":"code","source":"# Aggregate Columns based on Dictionary for new_trans_df\nprint('Aggregating New Transactions DataFrame...')\n\ngnew_trans_df = new_trans_df.groupby(['card_id']).agg(agg_func)\n\n# Rename columns before joining train / test set\ngnew_trans_df.columns = ['new_'+'_'.join(col).strip() for col in gnew_trans_df.columns.values]\ngnew_trans_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"2ac14ed4335d7e9919d473fcddd555ab8be1becc"},"cell_type":"code","source":"# Merge with train and test set\nprint('Merging with training set...')\ntrain = pd.merge(train_df,ghist_df,on='card_id',how='left')\ntrain = pd.merge(train,gnew_trans_df,on='card_id',how='left')\n\nprint('Merging with testing set...')\ntest = pd.merge(test_df,ghist_df,on='card_id',how='left')\ntest = pd.merge(test,gnew_trans_df,on='card_id',how='left')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d0b8cfefa4d3191ae0e9fd03a20001634d4ce0da"},"cell_type":"markdown","source":"## Feature Engineering from Aggregate Features"},{"metadata":{"trusted":false,"_uuid":"a73e75d1864b6e41a6082dba61e19a3284d926ed"},"cell_type":"code","source":"for df in [train,test]:\n    df['hist_purchase_date_uptonow'] = (datetime.datetime.today() - \n                                      df['hist_purchase_date_max']).dt.days\n    df['new_purchase_date_uptonow'] = (datetime.datetime.today() - \n                                      df['new_purchase_date_max']).dt.days\n    \n    dt_features = ['hist_purchase_date_max','hist_purchase_date_min',\n               'new_purchase_date_max','new_purchase_date_min']\n    \n    # Models cannot use datetime features so they are encoded here as int64s\n    for feature in dt_features:\n        df[feature] = df[feature].astype(np.int64)*1e-9","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"42a96fd88a28f0f3dedf40032e2fd1c285647c88"},"cell_type":"code","source":"# Final Train and Test Sets\ndisplay(train.head())\ndisplay(test.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"64b9706799f348579ff8019f1cf6b93611b200c2"},"cell_type":"code","source":"# Encoding Date times for first_active_month\nfor df in [train,test]:\n    df['first_month'] = df['first_active_month'].dt.month\n    df['first_year'] = df['first_active_month'].dt.year\n    df.drop(columns = ['first_active_month'],inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"7e438cdfecbc058b046092c4be08631fee1a24ba"},"cell_type":"code","source":"# Dealing with outliers\ntrain['outliers'] = 0\ntrain.loc[train['target'] < -30,'outliers'] = 1\ntrain['outliers'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"1bcc1de8d1c9055ea82c14ea5d347dcddad6ec3f"},"cell_type":"code","source":"target_col = train['target']\n\nfeatures = [name for name in train.columns if name not in ['target','card_id','new_authorized_flag_mean','outliers']]\n\ntarget = train['target']\ndel train['target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"168a1c3eb962d95e58d0a1581f103397bbc80646"},"cell_type":"code","source":"# Fill Nan Columns\nfiller = Imputer()\ntrain.loc[:,features] = filler.fit_transform(train[features].values)\ntest.loc[:,features] = filler.transform(test[features].values)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"484c51b06a8c5d31be01d1a1ec1ad199951e7540"},"cell_type":"markdown","source":"# LGBM Model with Outlier Stratified KFold Validation"},{"metadata":{"trusted":false,"_uuid":"6f479f755bbbfb01e86ca4b262945f34c70c0128"},"cell_type":"code","source":"# Set lgbm model params\nparam = {'num_leaves': 31,\n         'min_data_in_leaf': 30,\n         'objective':'regression',\n         'max_depth': -1,\n         'learning_rate': 0.005,\n         \"boosting\": \"gbdt\",\n         \"min_child_samples\":20,\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9 ,\n         \"bagging_seed\": 11,\n         \"metric\": 'rmse',\n         \"lambda_l1\": 0.1,\n         \"random_state\": 133,\n         \"nthread\":4,\n         \"verbosity\": -1}\n\n# Validating on a stratified outlier data set to give more consistent cv across the folds\n# https://www.kaggle.com/chauhuynh/my-first-kernel-3-699\n\nfolds = model_selection.StratifiedKFold(n_splits=5,shuffle=True,random_state=15)\nlgbm_oof = np.zeros(len(train))\nlgbm_pred = np.zeros(len(test))\n\nfor fold_, (train_index,valid_index) in enumerate(folds.split(train,train['outliers'].values)):\n    print(f\"fold number: {fold_ + 1}\")\n    \n    train_data = lgb.Dataset(train.iloc[train_index][features],label=target.iloc[train_index])\n    val_data = lgb.Dataset(train.iloc[valid_index][features],label=target.iloc[valid_index])\n    num_rounds = 10000\n    clf = lgb.train(param,\n                    train_data,\n                    num_rounds,\n                    valid_sets=[train_data,val_data],\n                    verbose_eval=100,\n                    early_stopping_rounds=200)\n    lgbm_oof[valid_index] = clf.predict(train.iloc[valid_index][features],num_iteration=clf.best_iteration)\n    lgbm_pred += clf.predict(test[features],num_iteration=clf.best_iteration)/folds.n_splits","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f0ac1d1df69100f53121730cfbd3dc0f7d59d2e7"},"cell_type":"markdown","source":"### LGBM CV SCORE"},{"metadata":{"trusted":false,"_uuid":"7c342aac2665ffd14a8580339785226995abeaf8"},"cell_type":"code","source":"np.sqrt(mean_squared_error(lgbm_oof, target))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3d1105b275eb15d85c44236a5365681a023fedca"},"cell_type":"markdown","source":"## XGBoost Model"},{"metadata":{"trusted":false,"_uuid":"e2a0d8e8d11e927cc98b7cdd5af0603195d7951b"},"cell_type":"code","source":"##xgb model\nxgb_params = {\n    'booster': 'gbtree',\n    'objective': 'reg:linear',\n    'gamma': 0.1,\n    'max_depth': 6,\n    'eval_metric':'rmse',\n    'lambda': 2,\n    'subsample': 0.7,\n    'colsample_bytree': 0.7,\n    'min_child_weight': 3,\n    'silent': 1,\n    'eta': 0.1,\n    'seed': 1000,\n    'nthread': 4,\n}\n\nfolds= model_selection.KFold(n_splits=5, shuffle=True, random_state=15)\nxgb_oof = np.zeros(len(train))\nxgb_pred = np.zeros(len(test))\n\nfor fold_,(train_index,valid_index) in enumerate(folds.split(train.values, train['outliers'].values)):\n    print(\"fold: {}/5\".format(fold_+1))\n    start = time.time()\n    train_data = xgb.DMatrix(train.iloc[train_index][features],\n                           label=target.iloc[train_index])\n    valid_data = xgb.DMatrix(train.iloc[valid_index][features],\n                           label=target.iloc[valid_index])\n    \n    xgb_evals = [(train_data, 'train'), (valid_data, 'valid')]\n    num_rounds = 2000\n    xgb_model = xgb.train(xgb_params, train_data, num_rounds, xgb_evals, early_stopping_rounds=50, verbose_eval=1000)\n    xgb_oof[valid_index] = xgb_model.predict(xgb.DMatrix(train.iloc[valid_index][features]), ntree_limit=xgb_model.best_ntree_limit+50)\n    xgb_pred += xgb_model.predict(xgb.DMatrix(test[features]), ntree_limit=xgb_model.best_ntree_limit+50) / folds.n_splits\n    print(f\"fold n°{fold_+1}/5 completed after: {time.time()-start:.2f} seconds\",'\\n')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"576fc12a87bc473194d5c88e703d7a9e68da5542"},"cell_type":"markdown","source":"### XGBoost CV Score:"},{"metadata":{"trusted":false,"_uuid":"28117002ab89ed896cd22cac44edf8c22bf4bc66"},"cell_type":"code","source":"print(np.sqrt(mean_squared_error(xgb_oof, target)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"695c675fd9cb176b174ac8cc9b0b50466f5facff"},"cell_type":"markdown","source":"## Sklearn Random Forest Model"},{"metadata":{"trusted":false,"_uuid":"8e24263404d90c72fab7912e3e44518c50735715"},"cell_type":"code","source":"# model trainer for sklearn pipeline\ndef sk_trainer(model):\n    folds = model_selection.StratifiedKFold(n_splits=5,shuffle=True,random_state=15)\n    oof = np.zeros(len(train))\n    predictions = np.zeros(len(test))\n\n    for fold_, (train_index,valid_index) in enumerate(folds.split(train,train['outliers'].values)):\n        print(f\"fold: {fold_ + 1}/5...\")\n        start = time.time()\n        train_x = train.iloc[train_index][features]\n        train_y=target.iloc[train_index]\n        val_x = train.iloc[valid_index][features]\n        val_y = target.iloc[valid_index]\n        model.fit(train_x,train_y)\n        oof[valid_index] = model.predict(val_x)\n        predictions += model.predict(test[features])/folds.n_splits\n        print(f\"~~~fold {fold_ +1} completed after: {time.time()-start:.2f} seconds~~~\")\n    return oof, predictions, model","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"40520fac8d6a541c5fe931069cfd65e2cfd2d8ed"},"cell_type":"code","source":"rf_model = RandomForestRegressor(n_estimators=50,max_depth=10)\nrf_oof, rf_pred, rf_model = sk_trainer(rf_model)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a64ae9bc444fc24ecfcc6f74f80a62b1ff15a473"},"cell_type":"markdown","source":"### Random Forest CV Score:"},{"metadata":{"trusted":false,"_uuid":"ceedad222a5d4eeeb169318fd5cbe1ddd9eb6ced"},"cell_type":"code","source":"print('CV Score:',np.sqrt(mean_squared_error(rf_oof, target)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7ba8126a4b18ba71eb652d3dea8b53c79b98f678"},"cell_type":"markdown","source":"### Averaged CV Score"},{"metadata":{"trusted":false,"_uuid":"98664a0ecfeb2a0c1e1d9c042dea45a50173ae5a"},"cell_type":"code","source":"averaged_oof = (rf_oof+lgbm_oof+xgb_oof)/3\naveraged_pred = (rf_pred+lgbm_pred+xgb_pred)/3\nprint('CV Score:',np.sqrt(mean_squared_error(averaged_oof, target)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"961547bb9326267baba0435832edb2f5b7fe449f"},"cell_type":"markdown","source":"### Stacked Linear Model"},{"metadata":{"trusted":false,"_uuid":"9d838d38fa2c026ccd29dc2864f5e181c481815b"},"cell_type":"code","source":"x = pd.DataFrame()\nx['lgbm'] = lgbm_oof\nx['rf'] = rf_oof\nx['xgb'] = xgb_oof\n\ntest_pred = pd.DataFrame()\ntest_pred['lgbm'] = lgbm_pred\ntest_pred['rf'] = rf_pred\ntest_pred['xgb'] = xgb_pred","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"58aa8ebabb58cd90649f0615f7499b0f84d068c5"},"cell_type":"code","source":"def level_2_trainer(model):\n    folds = model_selection.StratifiedKFold(n_splits=5,shuffle=True,random_state=10)\n    oof_normal = np.zeros(len(train))\n    predictions_normal = np.zeros(len(test))\n\n    for fold_, (train_index,valid_index) in enumerate(folds.split(train,train['outliers'].values)):\n        print(f\"fold number: {fold_ + 1}...\")\n        start = time.time()\n        train_x = x.iloc[train_index]\n        train_y=target.iloc[train_index]\n        val_x = x.iloc[valid_index]\n        val_y = target.iloc[valid_index]\n        model.fit(train_x,train_y)\n        oof_normal[valid_index] = model.predict(val_x)\n        predictions_normal += model.predict(test_pred)/folds.n_splits\n        print(f\"fold{fold_ +1} completed after {time.time()-start}seconds\")\n    return oof_normal, predictions_normal, model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d4f6d6f3ca2d791c7f6927e747805ce53bdaccac"},"cell_type":"markdown","source":"### Stacked Bayesian Ridge Model:"},{"metadata":{"trusted":false,"_uuid":"551ea648904d99aad5ce597353cdfa7fb68be462"},"cell_type":"code","source":"bay_ridge = linear_model.BayesianRidge()\nbay_oof,bay_pred, bay_model = level_2_trainer(bay_ridge)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"46b060fc32a8665bed737f74779729c54686f5a7"},"cell_type":"markdown","source":"### Stacked Bayesian Ridge CV:"},{"metadata":{"trusted":false,"_uuid":"9b784f1bba6015fc3d079f2f41ee506038577346"},"cell_type":"code","source":"print('CV Score:',np.sqrt(mean_squared_error(bay_oof, target)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a7c6815509ae89c25717dc44b7c79349ab7e680d"},"cell_type":"markdown","source":"## Second Level Linear Model Weights"},{"metadata":{"trusted":false,"_uuid":"02f556b94b53b000148ee94d8b24bb009f3d4d30"},"cell_type":"code","source":"models = ['LGBM','RF','XGB']\nfor model,weight in zip(models,bay_model.coef_):\n    print(f\"{model} Model Weights:{weight:0.3f}\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c98296f2bc3ede2301077a94b5c78fc31adc8385"},"cell_type":"markdown","source":"## Summary of Predictions (LB and CV)"},{"metadata":{"trusted":false,"_uuid":"fa35a92fd7241b926ae1221e2e6f650e56d3df8f"},"cell_type":"code","source":"pred_df = pd.DataFrame({\"card_id\":test[\"card_id\"].values})\npred_df['lgbm_target'] = lgbm_pred\npred_df['rf_target'] = rf_pred\npred_df['xgb_target'] = xgb_pred\npred_df['avg_target'] = averaged_pred\npred_df['bayridge_target'] = bay_pred\npred_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3263bb0fa04801aa0688baf12c6bb34fee711831"},"cell_type":"markdown","source":"#### LGBM Score: \n\n**CV:** 3.666\n\n**Public LB:** 3.713\n\n\n**Difference:** +0.047"},{"metadata":{"trusted":false,"_uuid":"9e8ebdc98dd5fec868cf78c9732ce5d252883b33"},"cell_type":"code","source":"sub_df = pd.DataFrame()\nsub_df['card_id'] = pred_df['card_id']\nsub_df['target'] = pred_df['lgbm_target']\n#sub_df.to_csv(\"ELOsubmission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a61a4684cac7faf54767a4be9447fc79eecf9c9b"},"cell_type":"markdown","source":"#### Random Forest Score: \n\n**CV:** 3.6988\n\n**Public LB:** 3.752\n\n\n**Difference:** +0.0542"},{"metadata":{"trusted":false,"_uuid":"3cf739f961a96ec6346e3c334911d89eab9d6600"},"cell_type":"code","source":"# Test Random Forest Score\nsub_df['target'] = pred_df['rf_target']\n#sub_df.to_csv(\"ELOsubmission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"616613b1a5aa885703a3ab50eb58f96fd503bf69"},"cell_type":"markdown","source":"#### XGBoost Score:\n\n**CV:** 3.68\n\n**Public LB:** 3.718\n\n**Difference:** +0.038"},{"metadata":{"trusted":false,"_uuid":"550cd7d60afc1e53e574065eb499a2b778db97ae"},"cell_type":"code","source":"sub_df['target'] = pred_df['xgb_target']\n#sub_df.to_csv(\"ELOsubmission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f27bf27ce8c8bd13beec97d8dad07f18f1688e4"},"cell_type":"markdown","source":"#### Simple Average Score:\n**CV:** 3.668 \n\n**Public LB:** 3.718\n\n**Difference:** 0.05"},{"metadata":{"trusted":false,"_uuid":"eb5221e95c4be6438a15dee8b92dade1da85506b"},"cell_type":"code","source":"sub_df['target'] = pred_df['avg_target']\n#sub_df.to_csv(\"ELOsubmission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e2fbe9cf65224ce66559c09d7b7939a0fc15e6ac"},"cell_type":"markdown","source":"#### Stacked Bayesian Ridge Model Score:\n\n**CV:** 3.664\n\n**Public LB:** 3.707\n\n**Difference:** 0.043"},{"metadata":{"trusted":false,"_uuid":"0932ce0e6b36baffcfd6ff4cee7f3fc1bc474098"},"cell_type":"code","source":"sub_df['target'] = pred_df['bayridge_target']\nsub_df.to_csv(\"ELOsubmission.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"toc":{"base_numbering":1,"nav_menu":{},"number_sections":true,"sideBar":true,"skip_h1_title":false,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":false,"toc_position":{},"toc_section_display":true,"toc_window_display":false}},"nbformat":4,"nbformat_minor":1}