{"cells":[{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true},"cell_type":"markdown","source":"![#](https://www.techworm.net/wp-content/uploads/2015/02/Untitledbd.png)\n> *\"Everybody has the right to like what they want\"* \n\n> Source: https://www.techworm.net/wp-content/uploads/2015/02/Untitledbd.png\n\n# Click Farm Features\nIt's been a while since we re-examined the features we use to capture fraudulent clicks and other essential assumptions we made about them.  So let's do that.\n\nLet's create a few features based on some asmptions of how click farms may operate and reason about thm.\n\nThis kernel is based on [this script by baris][1].\n\n[1]: https://www.kaggle.com/bk0000/non-blending-lightgbm-model-lb-0-977?scriptVersionId=3224614"},{"metadata":{"_uuid":"de8a691bb33e92cb826a3023c814287b6ebc6150","_cell_guid":"6f2ce76a-f660-4697-9d5d-63b218d9ed49","collapsed":true,"trusted":false},"cell_type":"code","source":"import os\nimport sys\nimport time\nimport gc\nimport logging\n\nimport pandas as pd\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nimport lightgbm as lgb\nimport matplotlib.pyplot as plt\n\n%matplotlib inline\n\n\n\"\"\"This code is based on this script by baris:\nhttps://www.kaggle.com/bk0000/non-blending-lightgbm-model-lb-0-977?scriptVersionId=3224614\n\"\"\"\n\nDEBUG = 0\n\nlogger = logging.getLogger()\nlogger.handlers = [logging.StreamHandler(sys.stdout)]\nlogger.setLevel(20 - DEBUG * 10)\n\npredictors=[]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ac8039fab3a4613c4d93f6ba08ccf4fc94ec62fb","_cell_guid":"aa5c0dbe-7425-4eea-a61a-e68fd414e37e"},"cell_type":"markdown","source":"## What would a click farmer do?\nTo run a profitable business, click farmers have a few restrictions that limit their behavior.   The *next click* based features measure the number of seconds between clicks, which is a proxy to click rate, when we group the data on a few feature sets.  Here is a list of some of some of these sets and the reason why we should measure their effectiveness:\n\n- [ `device` ] :  Click farmers use a limited number of hardware devices to generate clicks and that pumps up their click rate.\n- [ `device`, `channel` ] : Professional clickers, like the lady in the photo above, typically work on multiple devices and channels simultaneously.  The click rate on one device (regardless of the channel) may not be high. \n- [ `app`, `device`, `channel` ] : Advertising channels contract with app publishers, and the farmers will use the apps to increase their channel's clicks/revenue.  This triplet of features could zoom down on fraudesters.\n- [ `device`, `hour` ] : A professional clicker may have fixed work hours.\n\n## .. and not do?\nMoreover, we should not be using the following *next click* and *previous click* features:\n\n- [ `ip` ] : IP addresses are cheap to release and replace with new ones, and click generators do that as much as they could. \n- [ `channel` ] : Click farmers mostly target specific channels to increase their revenue.  However, YouTube reported that the fraudesters have been occasionally using other channels to camouflage their behavior.  Since the best performing model in this competition is at around 0.9827, we have excelent models alread and using only channel would hurt more than help. \n- [ `app` ] : Similar rational as for the `channel` feature, but amplified giving that a significant number of most apps are not click farmers.\n- [ `device`, `os` ] : Most devices run one operating system for their entire life, and click farmers are not more likely to use any combination of them than any other user.\n\n## No *previous click*\nWe are not using the *previous click* based features in this model since the *next click* based ones will capture the click rate.  There is no reason to believe that the first or last click in a group of features will affect such rate."},{"metadata":{"_uuid":"81ee1a7fdf8a49ff11ff0046f1ff9fa922ca8a42","_cell_guid":"577973ef-8e46-4b9c-88a3-72dc3c3500d7","collapsed":true,"trusted":false},"cell_type":"code","source":"def do_next_Click(df, agg_suffix='nextClick', agg_type='float32'):\n    \"\"\"Extracting next click feature.\n    Taken help from https://www.kaggle.com/nanomathias/feature-engineering-importance-testing  \n    \"\"\"\n    logger.info(\"Extracting {} time calculation features...\".format(agg_suffix))\n    \n    GROUP_BY_NEXT_CLICKS = [\n        {'groupby': ['ip', 'os', 'device', 'app']},\n        {'groupby': ['ip', 'os', 'device', 'app', 'channel']},\n        {'groupby': ['app', 'device', 'channel']},\n        {'groupby': ['ip', 'os', 'device']},\n        {'groupby': ['device', 'hour']},\n        \n       # {'groupby': ['device']},\n        \n        {'groupby': ['ip', 'app']},\n        {'groupby': ['ip', 'channel']},\n        {'groupby': ['device', 'channel']}\n    ]\n\n    # Calculate the time to next click for each group\n    for spec in GROUP_BY_NEXT_CLICKS:\n    \n       # Name of new feature\n        new_feature = '{}_{}'.format('_'.join(spec['groupby']),agg_suffix)    \n    \n        # Unique list of features to select\n        all_features = spec['groupby'] + ['click_time']\n\n        # Run calculation\n        logger.info(\">> Grouping by {}\".format(spec['groupby']))\n        df[new_feature] = (df[all_features]\n                           .groupby(spec['groupby'])\n                           .click_time.shift(-1) - df.click_time).dt.seconds.astype(agg_type)\n        predictors.append(new_feature)\n        gc.collect()\n    return df\n\n\ndef do_prev_Click(df, agg_suffix='prevClick', agg_type='float32'):\n    \"\"\"Extracting previous click feature.\n    Taken help from https://www.kaggle.com/nanomathias/feature-engineering-importance-testing  \n    \"\"\"\n    logger.info(\">> Extracting {} time calculation features...\".format(agg_suffix))\n    \n    GROUP_BY_NEXT_CLICKS = [\n        {'groupby': ['ip', 'channel']},\n        {'groupby': ['ip', 'os']}\n    ]\n\n    # Calculate the time to next click for each group\n    for spec in GROUP_BY_NEXT_CLICKS:\n    \n       # Name of new feature\n        new_feature = '{}_{}'.format('_'.join(spec['groupby']),agg_suffix)    \n    \n        # Unique list of features to select\n        all_features = spec['groupby'] + ['click_time']\n\n        # Run calculation\n        logger.info(\">> Grouping by {}\".format(spec['groupby']))\n        df[new_feature] = (df.click_time - \n                           df[all_features]\n                           .groupby(spec['groupby'])\n                           .click_time.shift(+1)).dt.seconds.astype(agg_type)\n        \n        predictors.append(new_feature)\n        gc.collect()\n    return df\n\n\ndef do_count(df, group_cols, agg_type='uint32', show_max=False, show_agg=True):\n    \"\"\"Add a new column with the count of another one after \n    grouping on a set of columns.\n    \"\"\"\n    agg_name='{}_count'.format('_'.join(group_cols))\n    gp = df[group_cols][group_cols].groupby(group_cols).size().rename(agg_name).to_frame().reset_index()\n    df = df.merge(gp, on=group_cols, how='left')\n    del gp\n    logger.info(\"{} max value = {}\".format(agg_name, df[agg_name].max()))\n    df[agg_name] = df[agg_name].astype(agg_type)\n    predictors.append(agg_name)\n    gc.collect()\n    return df\n\n\ndef do_countuniq(df, group_cols, counted, agg_type='uint32', show_max=False, show_agg=True):\n    \"\"\"Add a new column with the unique count of another one after \n    grouping on a set of columns.\n    \"\"\"\n    agg_name= '{}_by_{}_countuniq'.format(('_'.join(group_cols)),(counted))  \n    gp = df[group_cols+[counted]].groupby(group_cols)[counted].nunique().reset_index().rename(columns={counted:agg_name})\n    df = df.merge(gp, on=group_cols, how='left')\n    del gp\n    logger.info(\"{} max value = {}\".format(agg_name, df[agg_name].max()))\n    df[agg_name] = df[agg_name].astype(agg_type)\n    predictors.append(agg_name)\n    gc.collect()\n    return df\n\n\ndef do_cumcount(df, group_cols, counted,agg_type='uint32', show_max=False, show_agg=True):\n    \"\"\"Add a new column with the cumulative count of another one after \n    grouping on a set of columns.\n    \"\"\"\n    agg_name = '{}_by_{}_cumcount'.format(('_'.join(group_cols)),(counted)) \n    gp = df[group_cols+[counted]].groupby(group_cols)[counted].cumcount()\n    df[agg_name] = gp.values\n    del gp\n    logger.info(\"{} max value = {}.\".format(agg_name, df[agg_name].max()))\n    df[agg_name] = df[agg_name].astype(agg_type)\n    predictors.append(agg_name)\n    gc.collect()\n    return df\n\n\ndef do_mean(df, group_cols, counted, agg_type='float32', show_max=False, show_agg=True):\n    \"\"\"Add a new column with the mean value of a another one after \n    grouping on a set of columns.\n    \"\"\"\n    agg_name= '{}_by_{}_mean'.format(('_'.join(group_cols)),(counted))  \n    gp = df[group_cols+[counted]].groupby(group_cols)[counted].mean().reset_index().rename(columns={counted:agg_name})\n    df = df.merge(gp, on=group_cols, how='left')\n    del gp\n    logger.info(\"{} max value = {}\".format(agg_name, df[agg_name].max()))\n    df[agg_name] = df[agg_name].astype(agg_type)\n    predictors.append(agg_name)\n    gc.collect()\n    return df\n\n\ndef do_var(df, group_cols, counted, agg_type='float32', show_max=False, show_agg=True):\n    \"\"\"Add a new column with the variance value of another one after\n    grouping on a set of columns.\n    \"\"\"\n    agg_name= '{}_by_{}_var'.format(('_'.join(group_cols)),(counted)) \n    gp = df[group_cols+[counted]].groupby(group_cols)[counted].var().reset_index().rename(columns={counted:agg_name})\n    df = df.merge(gp, on=group_cols, how='left')\n    del gp\n    logger.info(\"{} max value = {}\".format(agg_name, df[agg_name].max()))\n    df[agg_name] = df[agg_name].astype(agg_type)\n    predictors.append(agg_name)\n    gc.collect()\n    return df\n\n\ndef lgb_modelfit_nocv(params, dtrain, dvalid, predictors, target='target',\n                      objective='binary', metrics='auc', feval=None,\n                      early_stopping_rounds=50, num_boost_round=3000,\n                      verbose_eval=10, categorical_features=None):\n    lgb_params = {\n        'boosting_type': 'gbdt',\n        'objective': objective,\n        'metric':metrics,\n        'learning_rate': 0.05,\n        #'is_unbalance': 'true',  #because training data is unbalance (replaced with scale_pos_weight)\n        'num_leaves': 31,  # we should let it be smaller than 2^(max_depth)\n        'max_depth': -1,  # -1 means no limit\n        'min_child_samples': 20,  # Minimum number of data need in a child(min_data_in_leaf)\n        'max_bin': 255,  # Number of bucketed bin for feature values\n        'subsample': 0.6,  # Subsample ratio of the training instance.\n        'subsample_freq': 0,  # frequence of subsample, <=0 means no enable\n        'colsample_bytree': 0.3,  # Subsample ratio of columns when constructing each tree.\n        'min_child_weight': 5,  # Minimum sum of instance weight(hessian) needed in a child(leaf)\n        'subsample_for_bin': 200000,  # Number of samples for constructing bin\n        'min_split_gain': 0,  # lambda_l1, lambda_l2 and min_gain_to_split to regularization\n        'reg_alpha': 0,  # L1 regularization term on weights\n        'reg_lambda': 0,  # L2 regularization term on weights\n        'nthread': 8,\n        'verbose': 0,\n    }\n\n    lgb_params.update(params)\n\n    xgtrain = lgb.Dataset(dtrain[predictors].values,\n                          label=dtrain[target].values,\n                          feature_name=predictors)\n    \n    xgvalid = lgb.Dataset(dvalid[predictors].values,\n                          label=dvalid[target].values,\n                          feature_name=predictors)\n    \n    del dtrain\n    del dvalid\n    gc.collect()\n    \n    evals_results = {}\n\n    bst1 = lgb.train(lgb_params, \n                     xgtrain, \n                     valid_sets=[xgvalid], \n                     valid_names=['valid'], \n                     evals_result=evals_results, \n                     num_boost_round=num_boost_round,\n                     early_stopping_rounds=early_stopping_rounds,\n                     verbose_eval=10, \n                     feval=feval)\n\n    logger.info(\"Model Report\")\n    logger.info(\"bst1.best_iteration: {}\".format(bst1.best_iteration))\n    logger.info(\"{}:{}\".format(metrics, evals_results['valid'][metrics][bst1.best_iteration-1]))\n    return bst1, bst1.best_iteration\n\n\ndef sample_positive(df: pd.DataFrame, positive_ratio: float = 0.1) -> pd.DataFrame:\n    \"\"\"Over sample positive events.\n    :param positive_ratio: The ratio of positive events to maintain.\n    :return: Over sampled `DataFrame`.\n    \"\"\"\n    positive = df[df.is_attributed == 1]  # Select positive events\n    negative = df[df.is_attributed == 0]  # And negative events\n    \n    negative_sampled = negative.sample(len(positive) * 99)  # Sample negative events with negative 1 : 9 positive ratio\n    logger.info('Sampled data: {:,} positive, {:,} => {:,} negative.'\n                .format(positive.shape[0], negative.shape[0], negative_sampled.shape[0]))\n    return (positive\n            .append(negative_sampled)\n            .sort_values(by='click_time')\n            .reset_index(drop=True))  # Combine negative and positive ","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"_uuid":"bc3c795db75153e1b6cd5c8481693706f67aabdc","_cell_guid":"51488c8d-2a37-4526-a7c7-480dc4965ea2","collapsed":true,"trusted":false},"cell_type":"code","source":"def main():\n    dtypes = {\n        'ip'            : 'uint32',\n        'app'           : 'uint16',\n        'device'        : 'uint8',\n        'os'            : 'uint16',\n        'channel'       : 'uint16',\n        'is_attributed' : 'uint8',\n        'click_id'      : 'uint32'\n    }\n\n    logger.debug('*** Running in DEBUG mode. ***')\n    nrows = 100000 if logger.getEffectiveLevel() == logging.DEBUG else None\n    \n    logger.info(\"Loading training data...\")\n    train = pd.read_csv('../input/train.csv',\n                        parse_dates=['click_time'],\n                        nrows=nrows,\n                        dtype=dtypes,\n                        usecols=['ip','app','device','os', 'channel', 'click_time', 'is_attributed'])\n    \n    train_df = sample_positive(train) # Oversampling positive events\n    del(train)\n        \n    logger.info('Loading test data...')\n    test_df = pd.read_csv(\"../input/test.csv\",\n                          nrows=nrows,\n                          parse_dates=['click_time'],\n                          dtype=dtypes,\n                          usecols=['ip','app','device','os', 'channel', 'click_time', 'click_id'])\n\n    train_size = len(train_df)\n    val_size = int(train_size * 0.4)\n\n    all_df = train_df.append(test_df).reset_index(drop=True)\n    del test_df\n\n    gc.collect()\n    all_df['hour'] = pd.to_datetime(all_df.click_time).dt.hour.astype('int8')\n    all_df['day'] = pd.to_datetime(all_df.click_time).dt.day.astype('int8') \n    \n    all_df = do_countuniq(all_df, ['ip'], 'app'); gc.collect()\n    all_df = do_var(all_df, ['channel', 'day'], 'hour'); gc.collect()\n    all_df = do_mean(all_df, ['ip', 'app', 'channel'], 'hour'); gc.collect()\n    all_df = do_countuniq(all_df, ['ip'], 'device'); gc.collect()\n    all_df = do_next_Click(all_df, agg_suffix='nextClick', agg_type='float32'); gc.collect()\n    all_df = do_count(all_df, ['ip', 'day', 'hour']); gc.collect()\n    \n    all_df = do_countuniq(all_df, ['ip'], 'channel'); gc.collect()\n    all_df = do_count(all_df, ['ip'], 'app'); gc.collect()\n    all_df = do_countuniq(all_df, ['ip', 'device', 'os'], 'app'); gc.collect()\n    all_df = do_cumcount(all_df, ['ip', 'device', 'os'], 'app'); gc.collect()\n    \n    # all_df = do_countuniq(all_df, ['device'], 'day'); gc.collect()\n    # all_df = do_var(all_df, ['device', 'day'], 'hour'); gc.collect()\n    # all_df = do_count(all_df, ['app']); gc.collect()\n    # all_df = do_count(all_df, ['channel']); gc.collect()\n    \n    \n    # all_df = do_var(all_df, ['device'], 'day'); gc.collect()\n    # all_df = do_countuniq(all_df, ['ip', 'channel'], 'app'); gc.collect()\n    # all_df = do_countuniq(all_df, ['channel', 'day'], 'hour'); gc.collect()\n    # all_df = do_countuniq(all_df, ['device', 'day'], 'hour'); gc.collect()\n    \n    del all_df['day']\n    gc.collect()\n    \n    logger.info('Before appending predictors...{}'.format(sorted(predictors)))\n    target = 'is_attributed'\n    word = ['app','device','os', 'channel', 'hour']\n    for feature in word:\n        if feature not in predictors:\n            predictors.append(feature)\n    categorical = ['app', 'device', 'os', 'channel', 'hour']\n    logger.info('After appending predictors...{}'.format(sorted(predictors)))\n\n    train_df = all_df.iloc[:(train_size - val_size)]\n    val_df = all_df.iloc[(train_size - val_size) : train_size]\n    test_df = all_df.iloc[train_size:]\n\n    logger.info(\"Training size: {}\".format(len(train_df)))\n    logger.info(\"Validation size: {}\".format(len(val_df)))\n    logger.info(\"Test size : {}\".format(len(test_df)))\n\n    sub = pd.DataFrame()\n    sub['click_id'] = test_df['click_id'].astype('int')\n\n    gc.collect()\n    start_time = time.time()\n\n    params = {\n        'learning_rate': 0.10,\n        #'is_unbalance': 'true', # replaced with scale_pos_weight argument\n        'num_leaves': 7,  # 2^max_depth - 1\n        'max_depth': 3,  # -1 means no limit\n        'min_child_samples': 100,  # Minimum number of data need in a child(min_data_in_leaf)\n        'max_bin': 100,  # Number of bucketed bin for feature values\n        'subsample': 0.7,  # Subsample ratio of the training instance.\n        'subsample_freq': 1,  # frequence of subsample, <=0 means no enable\n        'colsample_bytree': 0.9,  # Subsample ratio of columns when constructing each tree.\n        'min_child_weight': 0,  # Minimum sum of instance weight(hessian) needed in a child(leaf)\n        'scale_pos_weight':200 # because training data is extremely unbalanced \n    }\n    \n    bst, best_iteration = lgb_modelfit_nocv(params,\n                                            train_df,\n                                            val_df,\n                                            predictors,\n                                            target,\n                                            objective='binary',\n                                            metrics='auc',\n                                            early_stopping_rounds=30,\n                                            verbose_eval=True,\n                                            num_boost_round=1000,\n                                            categorical_features=categorical)\n\n    logger.info('[{}]: model training time'.format(time.time() - start_time))\n    del train_df\n    del val_df\n    gc.collect()\n\n    ax = lgb.plot_importance(bst, max_num_features=300)\n    plt.show()\n\n    logger.info(\"Predicting...\")\n    sub['is_attributed'] = bst.predict(test_df[predictors], num_iteration=best_iteration)\n    sub.to_csv('sub_{}.csv'.format(str(int(time.time()))), index=False, float_format='%.9f')\n    logger.info(\"Done...\")\n    return sub\n\n\nif __name__ == '__main__':\n    main()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d1bb5a372675950dfdab27ff91d9ece318ca2731","_cell_guid":"adcd2ec6-50ab-411e-aa62-38715fff947e"},"cell_type":"markdown","source":"## Conclusion\nThis model attempts to understand click farm behavior by reasoning about their motivations and constraints.  This is how model building should proceed, not by throwing as many features as the machine would allow us and see what sticks.  Still, this approach could be made better by measuring the [mutual information][mi] between each of the selected features (or group of features) and the click event.  This is typically followed by using the [Fast Correlation Based Feature Selection][fcbf] to filter out spurious correlation between the predictor and predicted event.  This should be added here soon.\n\n\n## References\n- Lei Yu and Huan Liu. 2003. *[Feature selection for high-dimensional data: a fast correlation-based filter solution][fcbf]*. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning (ICML'03).\n\n- Charles Arthur. *[How low-paid workers at 'click farms' create appearance of online popularity?][cf]*, The Guardian, Aug 2, 2013\n\n-  Doug Bock Clark. *[The Bot Bubble][cf2]*, The New Republic, April 20, 2015.\n\n- Joey Lee. *[What exactly is click-farming and how can it affect you or your business][cf3]*, Asia One, Jun 16, 2017\n\n\n[mi]: https://en.wikipedia.org/wiki/Mutual_information\n[fcbf]: http://www.public.asu.edu/~huanliu/papers/icml03.pdf\n[cf]: https://www.theguardian.com/technology/2013/aug/02/click-farms-appearance-online-popularity\n[cf2]: https://newrepublic.com/article/121551/bot-bubble-click-farms-have-inflated-social-media-currency\n[cf3]: http://www.asiaone.com/digital/what-exactly-click-farming-and-how-can-it-affect-you-or-your-business"}],"metadata":{"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}