{"cells":[{"metadata":{},"cell_type":"markdown","source":"# TalkingData AdTracking Fraud Detection Challenge"},{"metadata":{},"cell_type":"markdown","source":"### Description\n\nFraud risk is everywhere, but for companies that advertise online, click fraud can happen at an overwhelming volume, resulting in misleading click data and wasted money. Ad channels can drive up costs by simply clicking on the ad at a large scale. With over 1 billion smart mobile devices in active use every month, China is the largest\nmobile market in the world and therefore suffers from huge volumes of fradulent traffic.\n\nTalkingData, China’s largest independent big data service platform, covers over 70% of active mobile devices nationwide. They handle 3 billion clicks per day, of which 90% are potentially fraudulent. Their current approach to prevent click fraud for app developers is to measure the journey of a user’s click across their portfolio, and flag IP addresses who produce lots of clicks, but never end up installing apps. With this information, they've built an IP blacklist and device blacklist.\n\nWhile successful, they want to always be one step ahead of fraudsters and have turned to the Kaggle community for help in further developing their solution. In their 2nd competition with Kaggle, you’re challenged to build an algorithm that predicts whether a user will download an app after clicking a mobile app ad. To support your modeling, they have provided a generous dataset covering approximately 200 million clicks over 4 days!\n\n### Evaluation\nSubmissions are evaluated on <b>area under the ROC curve </b> between the predicted probability and the observed target.\n\n### Data fields\nEach row of the training data contains a click record, with the following features.\n\n<b>ip:</b> ip address of click.<br>\n<b>app:</b> app id for marketing.<br>\n<b>device:</b> device type id of user mobile phone (e.g., iphone 6 plus, iphone 7, huawei mate 7, etc.)<br>\n<b>os:</b> os version id of user mobile phone<br>\n<b>channel:</b> channel id of mobile ad publisher<br>\n<b>click_time:</b> timestamp of click (UTC)<br>\n<b>attributed_time:</b> if user download the app for after clicking an ad, this is the time of the app download<br>\n<b>is_attributed:</b> the target that is to be predicted, indicating the app was downloaded<br>\n\nNote that ip, app, device, os, and channel are encoded.\n\nThe test data is similar, with the following differences:\n\n<b>click_id:</b> reference for making predictions<br>\n<b>is_attributed:</b> not included"},{"metadata":{},"cell_type":"markdown","source":"## Import Packages"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport time\nimport numpy as np\nimport gc\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport lightgbm as lgb\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix,accuracy_score, roc_curve, auc, classification_report\nfrom sklearn.model_selection import GridSearchCV","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_file_name = '03-sub_lgb_balanced99.csv'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#exp_path = '../../experiments/exp_3/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Import Data"},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"dtypes = {\n        'ip'            : 'uint32',\n        'app'           : 'uint16',\n        'device'        : 'uint16',\n        'os'            : 'uint16',\n        'channel'       : 'uint16',\n        'is_attributed' : 'uint8',\n        'click_id'      : 'uint32'\n        }\n\nprint('loading train data...')\ntrain_df = pd.read_csv('../input/talkingdata-adtracking-fraud-detection/train.csv', skiprows=range(1,144903891), nrows=40000000, dtype=dtypes, usecols=['ip','app','device','os', 'channel', 'click_time', 'is_attributed'])\n\nprint('loading test data...')\ntest_df = pd.read_csv('../input/talkingdata-adtracking-fraud-detection/test.csv', dtype=dtypes, usecols=['ip','app','device','os', 'channel', 'click_time', 'click_id'])\n\nlen_train = len(train_df)\n\n# Join the datasets to apply the transformations only one time\ntrain_df=train_df.append(test_df)\n\ntrain_df.shape, test_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Train Data:')\ndisplay(train_df.head())\ndisplay(train_df.tail())\n\nprint('Test Data:')\ndisplay(test_df.head())\ndisplay(test_df.tail())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del test_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature Engineering"},{"metadata":{},"cell_type":"markdown","source":"#### Extracting time information from click_time feature"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['hour'] = pd.to_datetime(train_df.click_time).dt.hour.astype('uint8')\ntrain_df['day'] = pd.to_datetime(train_df.click_time).dt.day.astype('uint8')\ntrain_df['minute'] = pd.to_datetime(train_df.click_time).dt.minute.astype('uint8')\ntrain_df['second'] = pd.to_datetime(train_df.click_time).dt.second.astype('uint8')\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"####  Group-By-Aggregation"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define all the groupby transformations\nGROUPBY_AGGREGATIONS = [\n    # V1 - GroupBy Features #\n    # Variance in day, for ip-app-channel\n    {'groupby': ['ip','app','channel'], 'select': 'day', 'agg': 'var'},\n    # Variance in hour, for ip-app-os\n    {'groupby': ['ip', 'app', 'os'], 'select': 'hour', 'agg': 'var'},\n    # Variance in hour, for ip-day-channel\n    {'groupby': ['ip','day','channel'], 'select': 'hour', 'agg': 'var'},\n    # Count, for ip-day-hour\n    {'groupby': ['ip','day','hour'], 'select': 'channel', 'agg': 'count'},\n    # Count, for ip-app\n    {'groupby': ['ip', 'app'], 'select': 'channel', 'agg': 'count'},        \n    # Count, for ip-app-os\n    {'groupby': ['ip', 'app', 'os'], 'select': 'channel', 'agg': 'count'},\n    # Count, for ip-app-day-hour\n    {'groupby': ['ip','app','day','hour'], 'select': 'channel', 'agg': 'count'},\n    # Mean hour, for ip-app-channel\n    {'groupby': ['ip','app','channel'], 'select': 'hour', 'agg': 'mean'}, \n    # V2 - GroupBy Features #\n    # Average clicks on app by distinct users; is it an app they return to?\n    {'groupby': ['app'], \n     'select': 'ip', \n     'agg': lambda x: float(len(x)) / len(x.unique()), \n     'agg_name': 'AvgViewPerDistinct'\n    },\n    # How popular is the app or channel?\n    {'groupby': ['app'], 'select': 'channel', 'agg': 'count'},\n    {'groupby': ['channel'], 'select': 'app', 'agg': 'count'},\n#     # V3 - GroupBy Features                                              #\n#     # https://www.kaggle.com/bk0000/non-blending-lightgbm-model-lb-0-977 #\n    {'groupby': ['ip'], 'select': 'channel', 'agg': 'nunique'}, \n    {'groupby': ['ip'], 'select': 'app', 'agg': 'nunique'}, \n    {'groupby': ['ip','day'], 'select': 'hour', 'agg': 'nunique'}, \n    {'groupby': ['ip','app'], 'select': 'os', 'agg': 'nunique'}, \n#     {'groupby': ['ip'], 'select': 'device', 'agg': 'nunique'}, \n#     {'groupby': ['app'], 'select': 'channel', 'agg': 'nunique'}, \n#     {'groupby': ['ip', 'device', 'os'], 'select': 'app', 'agg': 'nunique'}, \n#     {'groupby': ['ip','device','os'], 'select': 'app', 'agg': 'cumcount'}, \n#     {'groupby': ['ip'], 'select': 'app', 'agg': 'cumcount'}, \n#     {'groupby': ['ip'], 'select': 'os', 'agg': 'cumcount'}, \n#     {'groupby': ['ip','day','channel'], 'select': 'hour', 'agg': 'var'}    \n]\n\n# Apply all the groupby transformations\nfor spec in GROUPBY_AGGREGATIONS:\n    # Name of the aggregation we're applying\n    agg_name = spec['agg_name'] if 'agg_name' in spec else spec['agg']\n    # Name of new feature\n    new_feature = '{}_{}_{}'.format('_'.join(spec['groupby']), agg_name, spec['select'])\n    # Info\n    print(\"Grouping by {}, and aggregating {} with {}\".format(\n        spec['groupby'], spec['select'], agg_name))\n    # Unique list of features to select\n    all_features = list(set(spec['groupby'] + [spec['select']]))\n    # Perform the groupby\n    gp = train_df[all_features]. \\\n        groupby(spec['groupby'])[spec['select']]. \\\n        agg(spec['agg']). \\\n        reset_index(). \\\n        rename(index=str, columns={spec['select']: new_feature})\n    # Merge back to X_total\n    if 'cumcount' == spec['agg']:\n        train_df[new_feature] = gp[0].values\n    else:\n        train_df = train_df.merge(gp, on=spec['groupby'], how='left')\n        \n    # Clear memory\n    del gp\n    gc.collect()\n\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"####  Clicks on app ad before & after "},{"metadata":{"trusted":true},"cell_type":"code","source":"HISTORY_CLICKS = {\n    'identical_clicks': ['ip', 'app', 'device', 'os', 'channel'],\n    'app_clicks': ['ip', 'app']\n}\n\n# Go through different group-by combinations\nfor fname, fset in HISTORY_CLICKS.items():\n    # Clicks in the past\n    train_df['prev_'+fname] = train_df.groupby(fset).cumcount().rename('prev_'+fname)\n#     # Clicks in the future\n#     train_df['future_'+fname] = train_df.iloc[::-1].groupby(fset).cumcount().rename('future_'+fname).iloc[::-1]\n\n# Count cumulative subsequent clicks\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Save FE Process"},{"metadata":{},"cell_type":"raw","source":"train_df.to_csv(exp_path+'train_processed.csv', index=False)"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# only train data\ndisplay(train_df[:len_train].isna().sum())\n\n# only test data\ndisplay(train_df[len_train:].isna().sum())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Separate the Datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"len_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = train_df[len_train:]\nval_df = train_df[(len_train-2500000):len_train]\ntrain_df = train_df[:(len_train-2500000)]\n\ntrain_df.shape, test_df.shape, val_df.shape","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"print('Train Data:')\ndisplay(train_df.head())\ndisplay(train_df.tail())\n\nprint('Val Data:')\ndisplay(val_df.head())\ndisplay(val_df.tail())\n\nprint('Test Data:')\ndisplay(test_df.head())\ndisplay(test_df.tail())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Modeling"},{"metadata":{"trusted":true},"cell_type":"code","source":"# function\ndef lgb_modelfit_nocv(params, dtrain, dvalid, predictors, target='target', objective='binary', metrics='auc',\n                       feval=None, early_stopping_rounds=20, num_boost_round=3000, verbose_eval=10, \n                      categorical_features=None):\n    lgb_params = {\n        'boosting_type': 'gbdt',\n        'objective': objective,\n        'metric':metrics,\n        'learning_rate': 0.01,\n        #'is_unbalance': 'true',  #because training data is unbalance (replaced with scale_pos_weight)\n        'num_leaves': 31,  # we should let it be smaller than 2^(max_depth)\n        'max_depth': -1,  # -1 means no limit\n        'min_child_samples': 20,  # Minimum number of data need in a child(min_data_in_leaf)\n        'max_bin': 255,  # Number of bucketed bin for feature values\n        'subsample': 0.6,  # Subsample ratio of the training instance.\n        'subsample_freq': 0,  # frequence of subsample, <=0 means no enable\n        'colsample_bytree': 0.3,  # Subsample ratio of columns when constructing each tree.\n        'min_child_weight': 5,  # Minimum sum of instance weight(hessian) needed in a child(leaf)\n        'subsample_for_bin': 200000,  # Number of samples for constructing bin\n        'min_split_gain': 0,  # lambda_l1, lambda_l2 and min_gain_to_split to regularization\n        'reg_alpha': 0,  # L1 regularization term on weights\n        'reg_lambda': 0,  # L2 regularization term on weights\n        'nthread': 4,\n        'verbose': 0,\n        'metric':metrics\n    }\n\n    lgb_params.update(params)\n    \n    print(\"preparing validation datasets\")\n    xgtrain = lgb.Dataset(dtrain[predictors].values, label=dtrain[target].values,\n                          feature_name=predictors,\n                          categorical_feature=categorical_features)\n    xgvalid = lgb.Dataset(dvalid[predictors].values, label=dvalid[target].values,\n                          feature_name=predictors,\n                          categorical_feature=categorical_features)\n    evals_results = {}\n    bst1 = lgb.train(lgb_params, \n                     xgtrain, \n                     valid_sets=[xgtrain, xgvalid], \n                     valid_names=['train','valid'], \n                     evals_result=evals_results, \n                     num_boost_round=num_boost_round,\n                     early_stopping_rounds=early_stopping_rounds,\n                     verbose_eval=10, \n                     feval=feval)\n    n_estimators = bst1.best_iteration\n    \n    print(\"\\nModel Report\")\n    print(\"n_estimators : \", n_estimators)\n    print(metrics+\":\", evals_results['valid'][metrics][n_estimators-1])\n    \n    print('\\nPlot - Feature Importance')\n    lgb.plot_importance(bst1)\n    plt.show()\n    \n    return bst1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# defininf the features and target\n\ntarget = 'is_attributed'\n\npredictors = ['app','device','os', 'channel', 'hour', 'day', \n              'ip_day_hour_count_channel', 'ip_day_channel_var_hour', \n              'ip_app_count_channel',\n              'ip_app_os_count_channel', 'ip_app_os_var_hour',\n              'ip_app_channel_var_day','ip_app_channel_mean_hour',\n             'ip_app_day_hour_count_channel']\n\nprint('Total predictors: {}'.format(len(predictors)))\n\n\ncategorical = ['app', 'device', 'os', 'channel', 'hour', 'day']\n\nsub = pd.DataFrame()\nsub['click_id'] = test_df['click_id'].astype('int')\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true},"cell_type":"code","source":"print(\"Training...\")\nstart_time = time.time()\n\nparams = {\n    'learning_rate': 0.15,\n    #'is_unbalance': 'true', # replaced with scale_pos_weight argument\n    'num_leaves': 7,  # 2^max_depth - 1\n    'max_depth': 3,  # -1 means no limit\n    'min_child_samples': 100,  # Minimum number of data need in a child(min_data_in_leaf)\n    'max_bin': 100,  # Number of bucketed bin for feature values\n    'subsample': 0.7,  # Subsample ratio of the training instance.\n    'subsample_freq': 1,  # frequence of subsample, <=0 means no enable\n    'colsample_bytree': 0.9,  # Subsample ratio of columns when constructing each tree.\n    'min_child_weight': 0,  # Minimum sum of instance weight(hessian) needed in a child(leaf)\n    'scale_pos_weight':99 # because training data is extremely unbalanced \n}\nbst = lgb_modelfit_nocv(params, \n                        train_df, \n                        val_df, \n                        predictors, \n                        target, \n                        objective='binary', \n                        metrics='auc',\n                        early_stopping_rounds=30, \n                        verbose_eval=True, \n                        num_boost_round=500, \n                        categorical_features=categorical)\n\nprint('[{}]: model training time'.format(time.time() - start_time))\n\n\n# del train_df\n# del val_df\n# gc.collect()\n\nprint(\"Predicting...\")\nsub['is_attributed'] = bst.predict(test_df[predictors])\n\nprint(\"writing to file...\")\nsub.to_csv(sub_file_name,index=False)\n\nprint(\"done...\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluate Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Predicting in validation dataset...\")\n\npredictions_lgbm_valdf_prob = bst.predict(val_df[predictors])\n\npredictions_lgbm_valdf = np.where(predictions_lgbm_valdf_prob > 0.5, 1, 0) #Turn probability to 0-1 binary output","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Print accuracy\nacc_lgbm = accuracy_score(val_df['is_attributed'], predictions_lgbm_valdf)\nprint('Overall accuracy of Light GBM model:', acc_lgbm)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Print Area Under Curve\nplt.figure()\nfalse_positive_rate, recall, thresholds = roc_curve(val_df['is_attributed'], predictions_lgbm_valdf)\n\nroc_auc = auc(false_positive_rate, recall)\n\nplt.title('Receiver Operating Characteristic (ROC)')\nplt.plot(false_positive_rate, recall, 'b', label = 'AUC = %0.3f' %roc_auc)\nplt.legend(loc='lower right')\nplt.plot([0,1], [0,1], 'r--')\nplt.xlim([0.0,1.0])\nplt.ylim([0.0,1.0])\nplt.ylabel('Recall')\nplt.xlabel('Fall-out (1-Specificity)')\nplt.show()\n\nprint('AUC score:', roc_auc)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Print Confusion Matrix\n\nplt.figure()\ncm = confusion_matrix(val_df['is_attributed'], predictions_lgbm_valdf)\n\nlabels = ['App Not Downloaded', 'App Downloaded']\nplt.figure(figsize=(8,6))\nsns.heatmap(cm, annot = True, fmt='d',vmin = 0.2);\nplt.title('Confusion Matrix')\nplt.ylabel('True Class')\nplt.xlabel('Predicted Class')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# classification report\n\nreport = classification_report(val_df['is_attributed'], predictions_lgbm_valdf)\n\nprint(report)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"kaggle-talkingdata","language":"python","name":"kaggle-talkingdata"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.8.2"}},"nbformat":4,"nbformat_minor":4}