{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"cell_type":"markdown","source":"# Feature Engineering & Importance Testing\nGoal of this notebook is to get an overview of all the features found around different kernels in this competition, as well as add a few new ones, and test their importance for predicting is_attributed using xgBoost. \n\n## Current best score with these features\nCurrent I've got a public LB score of **0.9769** using V2 of these features, and the xgBoost parameters found by [Bayesian Optimization in this notebook](https://www.kaggle.com/nanomathias/bayesian-tuning-of-xgboost-lightgbm-lb-0-9769), training on the entire dataset.\n\n## Inspirational Notebooks\nInspiration for features from (let me know if I have forgotten to give credit to anyone)\n* https://www.kaggle.com/nuhsikander/lgbm-new-features-corrected\n* https://www.kaggle.com/rteja1113/lightgbm-with-count-features\n* https://www.kaggle.com/aharless/swetha-s-xgboost-revised\n* https://www.kaggle.com/bk0000/non-blending-lightgbm-model-lb-0-977\n\n\n# 1. Loading data\nI'll just load a small subset of the data for more speedy testing"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import gc\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\n# Load subset of the training data\nX_train = pd.read_csv('../input/train.csv', nrows=1000000, parse_dates=['click_time'])\n\n# Show the head of the table\nX_train.head()","execution_count":4,"outputs":[]},{"metadata":{"_cell_guid":"475ba02d-4512-42ce-993c-cebdd238ee65","_uuid":"f62aefd5ab8c508cdc6b4ab5086f2c2de918e835"},"cell_type":"markdown","source":"# 2. Creating Features\n## 2.1 Extracting time information\nFirst extract day, minute, hour, second from the click_time. "},{"metadata":{"_cell_guid":"0201e666-e9c8-4f47-93e6-f10c6403c485","_uuid":"d4182f8bc9ffcb1f74986daa20160f677bd5f7c5","trusted":true},"cell_type":"code","source":"X_train['day'] = X_train['click_time'].dt.day.astype('uint8')\nX_train['hour'] = X_train['click_time'].dt.hour.astype('uint8')\nX_train['minute'] = X_train['click_time'].dt.minute.astype('uint8')\nX_train['second'] = X_train['click_time'].dt.second.astype('uint8')\nX_train.head()","execution_count":5,"outputs":[]},{"metadata":{"_cell_guid":"482455d5-6ad9-4451-8958-9c9311c5cae1","_uuid":"e8d07f262debbb02a2c2c65b1d2f4ae40cc81f9a"},"cell_type":"markdown","source":"## 2.2. Confidence Rates for is_attributed\nMy thought is that some ips, apps, devices, etc. might have higher frequencies of is_attributed, and I wish to add that information, i.e. I'm calculating the following \"attributed rates\":\n\n\\begin{equation}\n\\text{P}\\,\\text{(is_attributed}\\,\\,|\\,\\,\\text{category)}\n\\end{equation}\n\nor in some cases two- or multiple-paired combinations:\n\n\\begin{equation}\n\\text{P}\\,\\text{(is_attributed}\\,\\,|\\,\\,\\text{category_1, category_2)}\n\\end{equation}\n\nThe danger of this is that if a given category-combination has very few clicks, then the statistical significance of above equations cannot be trusted. Therefore I'll be weighing the rates by the following confidence rates:\n\n\\begin{equation}\n    \\text{conf}_{\\text{is_attributed}} = \\frac{\\log(\\text{views}_{\\text{category_1}})}{\\log(100000)}\n\\end{equation}\n\nwhere the value 100000 has been chosen arbitrarily to such that if a given category has 1000 views, then it gets a confidence weight of 60%, if it has 100 views then onfly a confidence weight of 40% etc."},{"metadata":{"_cell_guid":"abaf5be7-37f1-4fb2-8961-5124f6a91540","_uuid":"8e8a17ae2959795f797caa30344e44e257373430","trusted":true},"cell_type":"code","source":"ATTRIBUTION_CATEGORIES = [        \n    # V1 Features #\n    ###############\n    ['ip'], ['app'], ['device'], ['os'], ['channel'],\n    \n    # V2 Features #\n    ###############\n    ['app', 'channel'],\n    ['app', 'os'],\n    ['app', 'device'],\n    \n    # V3 Features #\n    ###############\n    ['channel', 'os'],\n    ['channel', 'device'],\n    ['os', 'device']\n]\n\n\n# Find frequency of is_attributed for each unique value in column\nfreqs = {}\nfor cols in ATTRIBUTION_CATEGORIES:\n    \n    # New feature name\n    new_feature = '_'.join(cols)+'_confRate'    \n    \n    # Perform the groupby\n    group_object = X_train.groupby(cols)\n    \n    # Group sizes    \n    group_sizes = group_object.size()\n    log_group = np.log(100000) # 1000 views -> 60% confidence, 100 views -> 40% confidence \n    print(\">> Calculating confidence-weighted rate for: {}.\\n   Saving to: {}. Group Max /Mean / Median / Min: {} / {} / {} / {}\".format(\n        cols, new_feature, \n        group_sizes.max(), \n        np.round(group_sizes.mean(), 2),\n        np.round(group_sizes.median(), 2),\n        group_sizes.min()\n    ))\n    \n    # Aggregation function\n    def rate_calculation(x):\n        \"\"\"Calculate the attributed rate. Scale by confidence\"\"\"\n        rate = x.sum() / float(x.count())\n        conf = np.min([1, np.log(x.count()) / log_group])\n        return rate * conf\n    \n    # Perform the merge\n    X_train = X_train.merge(\n        group_object['is_attributed']. \\\n            apply(rate_calculation). \\\n            reset_index(). \\\n            rename( \n                index=str,\n                columns={'is_attributed': new_feature}\n            )[cols + [new_feature]],\n        on=cols, how='left'\n    )\n    \nX_train.head()","execution_count":6,"outputs":[]},{"metadata":{"_cell_guid":"d9a71e26-2aec-4fbc-89da-cf4a902e193e","_uuid":"0b9430e2003bf219ab413e8e04a21480fad72e0c"},"cell_type":"markdown","source":"Locally I computed 10-fold cross-validation scores with 10million samples using xgBoost to see how each feature improved the model or not. The result was as follows:\n    \n<img src='http://i66.tinypic.com/120h01w.png' alt=\"Local Frequency CV tests\" width='50%'/>\n\nOn the public LB the score was improved from **0.955 -> 0.9624** by just including these features"},{"metadata":{"_cell_guid":"e6a2afd4-9457-4b11-a881-1e3b836d7ed2","_uuid":"5d76202d193849034a12a165674f2d722cdaaaba"},"cell_type":"markdown","source":"## 2.3. Group-By-Aggregation\nThere are a lot of groupby -> count()/var()/mean() etc. feature engineering in the kernels I've checked out, so of course those have to be added as well :)"},{"metadata":{"_cell_guid":"348678d3-d94a-4522-8584-b9a3846f451c","_uuid":"8c33e48891e76acc9ca7150baf554b34721c83cd","trusted":true},"cell_type":"code","source":"# Define all the groupby transformations\nGROUPBY_AGGREGATIONS = [\n    \n    # V1 - GroupBy Features #\n    #########################    \n    # Variance in day, for ip-app-channel\n    {'groupby': ['ip','app','channel'], 'select': 'day', 'agg': 'var'},\n    # Variance in hour, for ip-app-os\n    {'groupby': ['ip','app','os'], 'select': 'hour', 'agg': 'var'},\n    # Variance in hour, for ip-day-channel\n    {'groupby': ['ip','day','channel'], 'select': 'hour', 'agg': 'var'},\n    # Count, for ip-day-hour\n    {'groupby': ['ip','day','hour'], 'select': 'channel', 'agg': 'count'},\n    # Count, for ip-app\n    {'groupby': ['ip', 'app'], 'select': 'channel', 'agg': 'count'},        \n    # Count, for ip-app-os\n    {'groupby': ['ip', 'app', 'os'], 'select': 'channel', 'agg': 'count'},\n    # Count, for ip-app-day-hour\n    {'groupby': ['ip','app','day','hour'], 'select': 'channel', 'agg': 'count'},\n    # Mean hour, for ip-app-channel\n    {'groupby': ['ip','app','channel'], 'select': 'hour', 'agg': 'mean'}, \n    \n    # V2 - GroupBy Features #\n    #########################\n    # Average clicks on app by distinct users; is it an app they return to?\n    {'groupby': ['app'], \n     'select': 'ip', \n     'agg': lambda x: float(len(x)) / len(x.unique()), \n     'agg_name': 'AvgViewPerDistinct'\n    },\n    # How popular is the app or channel?\n    {'groupby': ['app'], 'select': 'channel', 'agg': 'count'},\n    {'groupby': ['channel'], 'select': 'app', 'agg': 'count'},\n    \n    # V3 - GroupBy Features                                              #\n    # https://www.kaggle.com/bk0000/non-blending-lightgbm-model-lb-0-977 #\n    ###################################################################### \n    {'groupby': ['ip'], 'select': 'channel', 'agg': 'nunique'}, \n    {'groupby': ['ip'], 'select': 'app', 'agg': 'nunique'}, \n    {'groupby': ['ip','day'], 'select': 'hour', 'agg': 'nunique'}, \n    {'groupby': ['ip','app'], 'select': 'os', 'agg': 'nunique'}, \n    {'groupby': ['ip'], 'select': 'device', 'agg': 'nunique'}, \n    {'groupby': ['app'], 'select': 'channel', 'agg': 'nunique'}, \n    {'groupby': ['ip', 'device', 'os'], 'select': 'app', 'agg': 'nunique'}, \n    {'groupby': ['ip','device','os'], 'select': 'app', 'agg': 'cumcount'}, \n    {'groupby': ['ip'], 'select': 'app', 'agg': 'cumcount'}, \n    {'groupby': ['ip'], 'select': 'os', 'agg': 'cumcount'}, \n    {'groupby': ['ip','day','channel'], 'select': 'hour', 'agg': 'var'}    \n]\n\n# Apply all the groupby transformations\nfor spec in GROUPBY_AGGREGATIONS:\n    \n    # Name of the aggregation we're applying\n    agg_name = spec['agg_name'] if 'agg_name' in spec else spec['agg']\n    \n    # Name of new feature\n    new_feature = '{}_{}_{}'.format('_'.join(spec['groupby']), agg_name, spec['select'])\n    \n    # Info\n    print(\"Grouping by {}, and aggregating {} with {}\".format(\n        spec['groupby'], spec['select'], agg_name\n    ))\n    \n    # Unique list of features to select\n    all_features = list(set(spec['groupby'] + [spec['select']]))\n    \n    # Perform the groupby\n    gp = X_train[all_features]. \\\n        groupby(spec['groupby'])[spec['select']]. \\\n        agg(spec['agg']). \\\n        reset_index(). \\\n        rename(index=str, columns={spec['select']: new_feature})\n        \n    # Merge back to X_total\n    if 'cumcount' == spec['agg']:\n        X_train[new_feature] = gp[0].values\n    else:\n        X_train = X_train.merge(gp, on=spec['groupby'], how='left')\n        \n     # Clear memory\n    del gp\n    gc.collect()\n\nX_train.head()","execution_count":8,"outputs":[]},{"metadata":{"_cell_guid":"c6c89b69-f790-499d-8e47-3ddf2701f4cc","_uuid":"97c4f3f0b22b7e92d67bad0a27f8e7c7223f35ff"},"cell_type":"markdown","source":"1. Locally I computed 10-fold cross-validation scores with 10million samples using xgBoost to see how each feature improved the model or not. The result was as follows:\n    \n<img src='http://i68.tinypic.com/33xkis8.png' alt=\"Local groupby CV tests\" width='50%'/>\n\nOn the public leaderboard the score was improved from **0.955 -> 0.9584** by just including these features"},{"metadata":{"_cell_guid":"2f564e60-73db-4bdd-9dcf-810663b825b2","_uuid":"2fb40a82c20b602bf63a8577249a98b63962130a"},"cell_type":"markdown","source":"# 2.4. Time till next click\nIt might be interesting to know e.g. how long it takes for a given ip-app-channel before they perform the next click. So I'll create some features for these as well. ****"},{"metadata":{"_cell_guid":"4b9d12f4-58de-47ba-b747-1bf1c3fb0ff8","_uuid":"5aea1d66c89c4677e21f7bf64f837216b5c2b242","trusted":true},"cell_type":"code","source":"GROUP_BY_NEXT_CLICKS = [\n    \n    # V1\n    {'groupby': ['ip']},\n    {'groupby': ['ip', 'app']},\n    {'groupby': ['ip', 'channel']},\n    {'groupby': ['ip', 'os']},\n    \n    # V3\n    {'groupby': ['ip', 'app', 'device', 'os', 'channel']},\n    {'groupby': ['ip', 'os', 'device']},\n    {'groupby': ['ip', 'os', 'device', 'app']}\n]\n\n# Calculate the time to next click for each group\nfor spec in GROUP_BY_NEXT_CLICKS:\n    \n    # Name of new feature\n    new_feature = '{}_nextClick'.format('_'.join(spec['groupby']))    \n    \n    # Unique list of features to select\n    all_features = spec['groupby'] + ['click_time']\n    \n    # Run calculation\n    print(f\">> Grouping by {spec['groupby']}, and saving time to next click in: {new_feature}\")\n    X_train[new_feature] = X_train[all_features].groupby(spec['groupby']).click_time.transform(lambda x: x.diff().shift(-1)).dt.seconds\n    \nX_train.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2b97a6f5-6e0c-4229-830e-0dcad092d0ed","_uuid":"8a4d93f63ad5d32842a5644f705c4ff8a6c876be"},"cell_type":"markdown","source":"Locally I computed 10-fold cross-validation scores with 10million samples using xgBoost to see how each feature improved the model or not. The result was as follows:\n    \n<img src='http://i67.tinypic.com/2dt7y2p.png' alt=\"Local nextClick CV tests\" width='50%'/>\n\nOn the public leaderboard the score was improved from **0.955 -> 0.9608** by just including these features"},{"metadata":{"_cell_guid":"96798009-4e63-4562-8b17-56040957afc0","_uuid":"502791e0009fb80c319e0440f91eb8ec812713b9"},"cell_type":"markdown","source":"## 2.5. Clicks on app ad before & after\nHas the user previously or subsequently clicked the exact same app-device-os-channel? I thought that might be an interesting feature to test out as well."},{"metadata":{"_cell_guid":"01301983-6588-4c93-86f2-524d00d3849d","_uuid":"d039ac65ce0fb010baf93939245d55dd40b55761","trusted":true,"collapsed":true},"cell_type":"code","source":"HISTORY_CLICKS = {\n    'identical_clicks': ['ip', 'app', 'device', 'os', 'channel'],\n    'app_clicks': ['ip', 'app']\n}\n\n# Go through different group-by combinations\nfor fname, fset in HISTORY_CLICKS.items():\n    \n    # Clicks in the past\n    X_train['prev_'+fname] = X_train. \\\n        groupby(fset). \\\n        cumcount(). \\\n        rename('prev_'+fname)\n        \n    # Clicks in the future\n    X_train['future_'+fname] = X_train.iloc[::-1]. \\\n        groupby(fset). \\\n        cumcount(). \\\n        rename('future_'+fname).iloc[::-1]\n\n# Count cumulative subsequent clicks\nX_train.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"876f5332-8602-4254-a57f-2bd3998ee433","_uuid":"667d7a8a438c8d0091bc241516ed387ab7c7a9a4"},"cell_type":"markdown","source":"Locally I computed 10-fold cross-validation scores with 10million samples using xgBoost to see how each feature improved the model or not. The result was as follows:\n    \n<img src='http://i64.tinypic.com/2j675fr.png' alt=\"Local nextClick CV tests\" width='50%'/>\n\nPossibly a very small improvement. This small improvement was also seen on public LB, where the score went from **0.955 -> 0.9568** by just including these features"},{"metadata":{"_cell_guid":"86a119dd-8c69-443e-b64f-22c54fa64b8c","_uuid":"f6dbcbc687ffed55ef994e13a0db9d9f1ef505c7"},"cell_type":"markdown","source":"# 3. Evaluating Feature Importance\nHaving created heaps of features, I'll fit xgBoost to the data, and evaluate the feature importances. First split into X and y"},{"metadata":{"_cell_guid":"fc901735-9c9c-4251-a678-b1ea1d2d5bfa","_uuid":"a295f37e16ff0b6b1b197fd92ab0e8ff381305d2","trusted":true,"collapsed":true},"cell_type":"code","source":"import xgboost as xgb\n\n# Split into X and y\ny = X_train['is_attributed']\nX = X_train.drop('is_attributed', axis=1).select_dtypes(include=[np.number])\n\n# Create a model\n# Params from: https://www.kaggle.com/aharless/swetha-s-xgboost-revised\nclf_xgBoost = xgb.XGBClassifier(\n    max_depth = 4,\n    subsample = 0.8,\n    colsample_bytree = 0.7,\n    colsample_bylevel = 0.7,\n    scale_pos_weight = 9,\n    min_child_weight = 0,\n    reg_alpha = 4,\n    n_jobs = 4, \n    objective = 'binary:logistic'\n)\n# Fit the models\nclf_xgBoost.fit(X, y)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"47527a6b-710e-40ca-902f-cd9358f1a09c","_uuid":"8f88190cfe9161f84d211b2387832a6b9d692ff2"},"cell_type":"markdown","source":"The feature importances are MinMax scaled, put into a DataFrame, and finally plotted ordered by the mean feature importance."},{"metadata":{"_cell_guid":"bd5df2f2-6351-4b11-8589-05e93b4a64c5","_uuid":"509683962f22f399bb8b437316d23ce959629da6","trusted":true,"collapsed":true},"cell_type":"code","source":"from sklearn import preprocessing\n\n# Get xgBoost importances\nimportance_dict = {}\nfor import_type in ['weight', 'gain', 'cover']:\n    importance_dict['xgBoost-'+import_type] = clf_xgBoost.get_booster().get_score(importance_type=import_type)\n    \n# MinMax scale all importances\nimportance_df = pd.DataFrame(importance_dict).fillna(0)\nimportance_df = pd.DataFrame(\n    preprocessing.MinMaxScaler().fit_transform(importance_df),\n    columns=importance_df.columns,\n    index=importance_df.index\n)\n\n# Create mean column\nimportance_df['mean'] = importance_df.mean(axis=1)\n\n# Plot the feature importances\nimportance_df.sort_values('mean').plot(kind='bar', figsize=(20, 7))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"7f58afb0-12ce-4f1c-a653-37e25bcb5c0f","collapsed":true,"_uuid":"55b0b7bde1ce0dec6394045483000585e3f0fd61"},"cell_type":"markdown","source":"# Evaluation on Public Leaderboard\nI've evaluated the impact of all the above features on the public leaderboard - the results are as follows for the simple xgBoost model used in this notebook applyed to the entire training set, without any additional tuning etc.\n\n<img src='http://i63.tinypic.com/jjtpow.png' width='50%' />"},{"metadata":{"_cell_guid":"9b19f5b7-126d-45bd-ad07-ea84a8542ffb","collapsed":true,"_uuid":"3df1907a137ecde7d4160ce7ae77428fa3e853e1","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}