{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Please select an option before submitting results to the competition"},{"metadata":{"trusted":true},"cell_type":"code","source":"submit_flag = True #False #True\nprint(submit_flag)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"available_encodings = ['No Encoding', 'Count Encoding', 'Target Encoding', 'Target Encoding No IP', 'CatBoost Encoding']\nmy_encoding = available_encodings[4]\nprint(my_encoding)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TalkingData AdTracking Fraud Detection Challenge\n# Can you detect fraudulent click traffic for mobile app ads?\n# https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection"},{"metadata":{},"cell_type":"markdown","source":"**This notebook is inspired by an exercise in the [Feature Engineering](https://www.kaggle.com/learn/feature-engineering) course.**  \n**You can reference the tutorial at [this link](https://www.kaggle.com/matleonard/categorical-encodings)**  \n**You can reference my notebook at [this link](http://www.kaggle.com/georgezoto/feature-engineering-categorical-encodings)**  \n\n---\n"},{"metadata":{},"cell_type":"markdown","source":"<center><a href=\"https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection\"><img src=\"https://i.imgur.com/srKxEkD.png\" width=600px></a></center>"},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nIn this exercise you'll apply more advanced encodings to encode the categorical variables ito improve your classifier model. The encodings you will implement are:\n\n- Count Encoding\n- Target Encoding\n- CatBoost Encoding\n\nYou'll refit the classifier after each encoding to check its performance on hold-out data. \n\nBegin by running the next code cell to set up the notebook."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom sklearn import preprocessing, metrics\nimport lightgbm as lgb\n\nclicks = pd.read_parquet('../input/feature-engineering-data/baseline_data.pqt')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clicks.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clicks.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clicks['is_attributed'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clicks['is_attributed'].value_counts(normalize=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Competition data"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Read only first limit rows\nlimit = 20_000_000\n\n#Read only these columns - skip attributed_time \nusecols = ['ip', 'app', 'device', 'os', 'channel', 'click_time', 'is_attributed']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_data = pd.read_csv('../input/talkingdata-adtracking-fraud-detection/train.csv', nrows=limit, usecols=usecols, parse_dates=['click_time'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_data['is_attributed'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_data['is_attributed'].value_counts(normalize=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_test_data = pd.read_csv('../input/talkingdata-adtracking-fraud-detection/test.csv', parse_dates=['click_time'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Add new columns for timestamp features day, hour, minute, and second\ncompetition_test_data = competition_test_data.copy()\ncompetition_test_data['day'] = competition_test_data['click_time'].dt.day.astype('uint8')\n# Fill in the rest\ncompetition_test_data['hour'] = competition_test_data['click_time'].dt.hour.astype('uint8')\ncompetition_test_data['minute'] = competition_test_data['click_time'].dt.minute.astype('uint8')\ncompetition_test_data['second'] = competition_test_data['click_time'].dt.second.astype('uint8')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_test_data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_test_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we define a couple functions that you'll use to test the encodings that you implement in this exercise."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_data_splits(dataframe, valid_fraction=0.1):\n    \"\"\"Splits a dataframe into train, validation, and test sets.\n\n    First, orders by the column 'click_time'. Set the size of the \n    validation and test sets with the valid_fraction keyword argument.\n    \"\"\"\n\n    dataframe = dataframe.sort_values('click_time')\n    valid_rows = int(len(dataframe) * valid_fraction)\n    train = dataframe[:-valid_rows * 2]\n    # valid size == test size, last two sections of the data\n    valid = dataframe[-valid_rows * 2:-valid_rows]\n    test = dataframe[-valid_rows:]\n    \n    return train, valid, test\n\ndef train_model(train, valid, test=None, feature_cols=None):\n    \n    if feature_cols is None:\n        feature_cols = train.columns.drop(['click_time', 'attributed_time',\n                                           'is_attributed'])\n    dtrain = lgb.Dataset(train[feature_cols], label=train['is_attributed'])\n    dvalid = lgb.Dataset(valid[feature_cols], label=valid['is_attributed'])\n    \n    param = {'num_leaves': 64, 'objective': 'binary', \n             'metric': 'auc', 'seed': 7}\n    \n    num_round = 1000\n    \n    #Record eval results for plotting\n    validation_metrics = {} \n    \n    bst = lgb.train(param, dtrain, num_round, valid_sets=[dvalid], \n                    early_stopping_rounds=20, evals_result=validation_metrics, verbose_eval=True)\n    \n    valid_pred = bst.predict(valid[feature_cols])\n    valid_score = metrics.roc_auc_score(valid['is_attributed'], valid_pred)\n    print(f\"Validation AUC score: {valid_score}\")\n    \n    if test is not None: \n        test_pred = bst.predict(test[feature_cols])\n        test_score = metrics.roc_auc_score(test['is_attributed'], test_pred)\n        return bst, valid_score, test_score, validation_metrics\n    else:\n        return bst, valid_score, validation_metrics","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run this cell to get a baseline score. "},{"metadata":{"trusted":true},"cell_type":"code","source":"my_own_metrics = {}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'No Encoding':\n    print('No Encoding model') \n    train, valid, test = get_data_splits(clicks)\n    bst, valid_score, validation_metrics = train_model(train, valid)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'No Encoding':\n    my_own_metrics['No Encoding'] = valid_score\n    my_own_metrics","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clicks.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1) Categorical encodings and leakage\n\nThese encodings are all based on statistics calculated from the dataset like counts and means. \n\nConsidering this, what data should you be using to calculate the encodings?  Specifically, can you use the validation data?  Can you use the test data?"},{"metadata":{},"cell_type":"markdown","source":"### 2) Count encodings\n\nBegin by running the next code cell to get started."},{"metadata":{},"cell_type":"markdown","source":"### Count Encoding\nCount encoding replaces each categorical value with the number of times it appears in the dataset. For example, if the value \"GB\" occured 10 times in the country feature, then each \"GB\" would be replaced with the number 10.\n\n### Question: How about unseen values in the valid and test sets ???"},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding != 'No Encoding':\n    import category_encoders as ce\n    cat_features = ['ip', 'app', 'device', 'os', 'channel']\n    train, valid, test = get_data_splits(clicks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, encode the categorical features `['ip', 'app', 'device', 'os', 'channel']` using the count of each value in the data set. \n- Using `CountEncoder` from the `category_encoders` library, fit the encoding using the categorical feature columns defined in `cat_features`. \n- Then apply the encodings to the train and validation sets, adding them as new columns with names suffixed `\"_count\"`."},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'Count Encoding':  \n    # Create the count encoder\n    count_enc = ce.CountEncoder(cols=cat_features)\n\n    # Learn encoding from the training set\n    count_enc.fit(train[cat_features])\n\n    # Apply encoding to the train and validation sets as new columns\n    # Make sure to add `_count` as a suffix to the new columns\n    train_encoded = train.join(count_enc.transform(train[cat_features]).add_suffix(\"_count\")) \n    valid_encoded = valid.join(count_enc.transform(valid[cat_features]).add_suffix(\"_count\")) \n    \n    # Apply encoding to the competition test dataset\n    competition_test_data = competition_test_data.join(count_enc.transform(competition_test_data[cat_features]).add_suffix(\"_count\")) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'Count Encoding':\n    print('train_encoded.head()\\n')\n    print(train_encoded.head())\n    print('\\ncompetition_encoded.head()')\n    print(competition_test_data.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run the next code cell to see how count encoding changes the results."},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'Count Encoding':\n    # Train the model on the encoded datasets\n    # This can take around 30 seconds to complete\n    bst, valid_score, validation_metrics = train_model(train_encoded, valid_encoded)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'Count Encoding':\n    my_own_metrics['Count Encoding'] = valid_score\n    print(my_own_metrics)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Count encoding improved our model's score!"},{"metadata":{},"cell_type":"markdown","source":"### 3) Why is count encoding effective?\nAt first glance, it could be surprising that count encoding helps make accurate models. \nWhy do you think count encoding is a good idea, or how does it improve the model score?"},{"metadata":{},"cell_type":"markdown","source":"### 4) Target encoding\n\nHere you'll try some supervised encodings that use the labels (the targets) to transform categorical features. The first one is target encoding. \n- Create the target encoder from the `category_encoders` library. \n- Then, learn the encodings from the training dataset, apply the encodings to all the datasets, and retrain the model."},{"metadata":{},"cell_type":"markdown","source":"### Target encoding replaces a categorical value with the average value of the target for that value of the feature. For example, given the country value \"CA\", you'd calculate the average outcome for all the rows with country == 'CA', around 0.28. \n\n### This is often blended with the target probability over the entire dataset to reduce the variance of values with few occurences.\n\n### Data leakage ??? blended with the target probability over the entire dataset ???"},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'Target Encoding':    \n    # Create the target encoder. You can find this easily by using tab completion.\n    # Start typing ce. the press Tab to bring up a list of classes and functions.\n    target_enc = ce.TargetEncoder(cols=cat_features)\n\n    # Learn encoding from the training set. Use the 'is_attributed' column as the target.\n    target_enc.fit(train[cat_features], train['is_attributed'])\n\n    # Apply encoding to the train and validation sets as new columns\n    # Make sure to add `_target` as a suffix to the new columns\n    train_encoded = train.join(target_enc.transform(train[cat_features]).add_suffix(\"_target\"))\n    valid_encoded = valid.join(target_enc.transform(valid[cat_features]).add_suffix(\"_target\"))\n    \n    # Apply encoding to the competition test dataset\n    competition_test_data = competition_test_data.join(target_enc.transform(competition_test_data[cat_features]).add_suffix(\"_target\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'Target Encoding': \n    print('train_encoded.head()\\n')\n    print(train_encoded.head())\n    print('\\ncompetition_encoded.head()')\n    print(competition_test_data.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run the next cell to see how target encoding affects your results."},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'Target Encoding': \n    bst, valid_score, validation_metrics = train_model(train_encoded, valid_encoded)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'Target Encoding': \n    my_own_metrics['Target Encoding'] = valid_score\n    print(my_own_metrics)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5) Try removing IP encoding\n\nIf you leave `ip` out of the encoded features and retrain the model with target encoding, you should find that the score increases and is above the baseline score! Why do you think the score is below baseline when we encode the IP address but above baseline when we don't?"},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'Target Encoding No IP': \n    cat_features_no_ip = ['app', 'device', 'os', 'channel']\n    \n    target_enc = ce.TargetEncoder(cols=cat_features_no_ip)\n\n    target_enc.fit(train[cat_features_no_ip], train['is_attributed'])\n\n    train_encoded = train.join(target_enc.transform(train[cat_features_no_ip]).add_suffix(\"_target\"))\n    valid_encoded = valid.join(target_enc.transform(valid[cat_features_no_ip]).add_suffix(\"_target\"))\n    \n    # Apply encoding to the competition test dataset\n    competition_test_data = competition_test_data.join(target_enc.transform(competition_test_data[cat_features_no_ip]).add_suffix(\"_target\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'Target Encoding No IP': \n    print('train_encoded.head()\\n')\n    print(train_encoded.head())\n    print('\\ncompetition_encoded.head()')\n    print(competition_test_data.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'Target Encoding No IP': \n    bst, valid_score, validation_metrics = train_model(train_encoded, valid_encoded)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'Target Encoding No IP': \n    my_own_metrics['Target Encoding No IP'] = valid_score\n    print(my_own_metrics)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 6) CatBoost Encoding\n\nThe CatBoost encoder is supposed to work well with the LightGBM model. Encode the categorical features with `CatBoostEncoder` and train the model on the encoded data again."},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'CatBoost Encoding': \n    # Remove IP from the encoded features\n    cat_features = ['app', 'device', 'os', 'channel']\n\n    # Create the CatBoost encoder\n    cb_enc = ce.CatBoostEncoder(cols=cat_features, random_state=7)\n\n    # Learn encoding from the training set\n    cb_enc.fit(train[cat_features], train['is_attributed'])\n\n    # Apply encoding to the train and validation sets as new columns\n    # Make sure to add `_cb` as a suffix to the new columns\n    train_encoded = train.join(cb_enc.transform(train[cat_features]).add_suffix(\"_cb\"))\n    valid_encoded = valid.join(cb_enc.transform(valid[cat_features]).add_suffix(\"_cb\"))\n    \n    # Apply encoding to the competition test dataset\n    competition_test_data = competition_test_data.join(cb_enc.transform(competition_test_data[cat_features]).add_suffix(\"_cb\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'CatBoost Encoding':\n    print('train_encoded.head()\\n')\n    print(train_encoded.head())\n    print('\\ncompetition_encoded.head()')\n    print(competition_test_data.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run the next code cell to see how the CatBoost encoder changes your results."},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'CatBoost Encoding':\n    bst, valid_score, validation_metrics = train_model(train_encoded, valid_encoded)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if my_encoding == 'CatBoost Encoding':\n    my_own_metrics['CatBoost Encoding'] = valid_score\n    print(my_own_metrics)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Keep Going\n\nNow you are ready to **[generate completely new features](https://www.kaggle.com/matleonard/feature-generation)** from the data."},{"metadata":{},"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161443) to chat with other Learners.*"},{"metadata":{},"cell_type":"markdown","source":"## Model information"},{"metadata":{"trusted":true},"cell_type":"code","source":"bst.num_trees()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plot model performance"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.rcParams[\"figure.figsize\"] = [16,9]\n\nax = lgb.plot_metric(validation_metrics, metric='auc');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.rcParams[\"figure.figsize\"] = (16,9)\n\ndef plot_my_own_metrics(my_own_metrics):\n    x=list(my_own_metrics.keys())\n    y=list(my_own_metrics.values())\n    plt.barh(x, y);\n\n    for index, value in enumerate(y):\n        plt.text(value, index, str(value))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_my_own_metrics(my_own_metrics)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ML Explainability and taking a closer look at feature importance, individual trees\nInspired by: https://github.com/Microsoft/LightGBM/blob/2e93cdab9eee02d4d7f5cb3b6b31128dec94e25e/examples/python-guide/plot_example.py"},{"metadata":{"trusted":true},"cell_type":"code","source":"bst.num_trees()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tree_index = 0\nprint('Plot '+str(tree_index)+'th tree...')  # one tree use categorical feature to split\nax = lgb.plot_tree(bst, tree_index=tree_index, figsize=(64, 36), show_info=['split_gain'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Plot feature importances...')\nax = lgb.plot_importance(bst, max_num_features=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submit test predictions to TalkingData AdTracking Fraud Detection Challenge competition using the limited train.csv records from this notebook"},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_test_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_cols = competition_test_data.columns.drop(['click_id', 'click_time'])\nfeature_cols","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_predictions = bst.predict(competition_test_data[feature_cols])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_predictions_df = pd.DataFrame(competition_predictions, columns=['is_attributed'])\ncompetition_predictions_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_predictions_df['click_id'] = competition_test_data['click_id']\ncompetition_predictions_df = competition_predictions_df[['click_id', 'is_attributed']]\ncompetition_predictions_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.cut(competition_predictions_df['is_attributed'], bins=10).value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.cut(competition_predictions_df['is_attributed'], bins=10).value_counts().plot(kind='bar', rot=45);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#available_encodings = ['No Encoding', 'Count Encoding', 'Target Encoding', 'Target Encoding No IP', 'CatBoost Encoding']\n\nif my_encoding == 'No Encoding':\n    my_own_metrics['private score'] = 0.88343\n    my_own_metrics['public score'] = 0.89154\n    \nelif my_encoding == 'Count Encoding':\n    my_own_metrics['private score'] = 0.77323\n    my_own_metrics['public score'] = 0.77341    \n    \nelif my_encoding == 'Target Encoding':\n    my_own_metrics['private score'] = 0.55412\n    my_own_metrics['public score'] = 0.55253    \n    \nelif my_encoding == 'Target Encoding No IP':\n    my_own_metrics['private score'] = 0.74309\n    my_own_metrics['public score'] = 0.76522    \n\nelif my_encoding == 'CatBoost Encoding':\n    my_own_metrics['private score'] = 0.79337\n    my_own_metrics['public score'] = 0.81397 \n    \nprint(my_own_metrics)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if submit_flag == True:\n    competition_predictions_df.to_csv('submission.csv', index=False)\n    print('submission.csv generated successfully :)')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submit csv to competition"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}