{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Please select an option before submitting results to the competition"},{"metadata":{"trusted":true},"cell_type":"code","source":"submit_flag = True #False #True\nprint(submit_flag)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TalkingData AdTracking Fraud Detection Challenge\n# Can you detect fraudulent click traffic for mobile app ads?\n# https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection"},{"metadata":{},"cell_type":"markdown","source":"**This notebook is inspired by an exercise in the [Feature Engineering](https://www.kaggle.com/learn/feature-engineering) course**  \n**You can reference the tutorial at [this link](https://www.kaggle.com/matleonard/feature-selection)**  \n**You can reference my notebook at [this link](https://www.kaggle.com/georgezoto/feature-engineering-feature-selection/)**  \n\n---\n"},{"metadata":{},"cell_type":"markdown","source":"<center><a href=\"https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection\"><img src=\"https://i.imgur.com/srKxEkD.png\" width=600px></a></center>"},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nIn this exercise you'll use some feature selection algorithms to improve your model. Some methods take a while to run, so you'll write functions and verify they work on small samples.\n\nTo begin, run the code cell below to set up the exercise."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom sklearn import preprocessing, metrics\nimport lightgbm as lgb\n\nimport os\n\nclicks = pd.read_parquet('../input/feature-engineering-data/baseline_data.pqt')\ndata_files = ['count_encodings.pqt',\n              'catboost_encodings.pqt',\n              'interactions.pqt',\n              'past_6hr_events.pqt',\n              'downloads.pqt',\n              'time_deltas.pqt',\n              'svd_encodings.pqt']\ndata_root = '../input/feature-engineering-data'\nfor file in data_files:\n    features = pd.read_parquet(os.path.join(data_root, file))\n    clicks = clicks.join(features)\n\ndef get_data_splits(dataframe, valid_fraction=0.1):\n\n    dataframe = dataframe.sort_values('click_time')\n    valid_rows = int(len(dataframe) * valid_fraction)\n    train = dataframe[:-valid_rows * 2]\n    # valid size == test size, last two sections of the data\n    valid = dataframe[-valid_rows * 2:-valid_rows]\n    test = dataframe[-valid_rows:]\n    \n    return train, valid, test\n\ndef train_model(train, valid, test=None, feature_cols=None, valid_name_model='Baseline Model'):\n    if feature_cols is None:\n        feature_cols = train.columns.drop(['click_time', 'attributed_time',\n                                           'is_attributed'])\n    dtrain = lgb.Dataset(train[feature_cols], label=train['is_attributed'])\n    dvalid = lgb.Dataset(valid[feature_cols], label=valid['is_attributed'])\n    \n    param = {'num_leaves': 64, 'objective': 'binary', \n             'metric': 'auc', 'seed': 7}\n    num_round = 1000\n    \n    #Record eval results for plotting\n    validation_metrics = {} \n    \n    print(\"Training model!\")\n    bst = lgb.train(param, dtrain, num_round, valid_sets=[dvalid], valid_names=valid_name_model,\n                    early_stopping_rounds=20, evals_result=validation_metrics, verbose_eval=False)\n    \n    valid_pred = bst.predict(valid[feature_cols])\n    valid_score = metrics.roc_auc_score(valid['is_attributed'], valid_pred)\n    print(f\"Validation AUC score: {valid_score}\")\n    \n    if test is not None: \n        test_pred = bst.predict(test[feature_cols])\n        test_score = metrics.roc_auc_score(test['is_attributed'], test_pred)\n        return bst, valid_score, test_score, validation_metrics\n    else:\n        return bst, valid_score, validation_metrics","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def my_own_train_plot_model(clicks, valid_name_model, my_own_metrics):\n    #valid_name_model='V11 FI Numerical ip_past_6hr_counts Model'\n    print(valid_name_model+' score')\n\n    train, valid, test = get_data_splits(clicks)\n    bst, valid_score, validation_metrics = train_model(train, valid, valid_name_model=valid_name_model)\n\n    my_own_metrics[valid_name_model] = valid_score\n    print(my_own_metrics)\n    plot_model_information(bst, validation_metrics, my_own_metrics)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def my_own_train_plot_model_v2(train, valid, valid_name_model, my_own_metrics):\n    #valid_name_model='V11 FI Numerical ip_past_6hr_counts Model'\n    print(valid_name_model+' score')\n\n    bst, valid_score, validation_metrics = train_model(train, valid, valid_name_model=valid_name_model)\n\n    my_own_metrics[valid_name_model] = valid_score\n    print(my_own_metrics)\n    plot_model_information(bst, validation_metrics, my_own_metrics)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model information"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.rcParams[\"figure.figsize\"] = (16,9)\n\ndef plot_model_information(bst, validation_metrics, my_own_metrics):\n    print('Number of trees:', bst.num_trees())\n    \n    print('Plot model performance')\n    ax = lgb.plot_metric(validation_metrics, metric='auc');\n    plt.show()\n    \n    print('Plot feature importances...')\n    ax = lgb.plot_importance(bst, max_num_features=15)\n    plt.show()\n    \n    def plot_my_own_metrics(my_own_metrics):\n        x=list(my_own_metrics.keys())\n        y=list(my_own_metrics.values())\n        plt.barh(x, y);\n\n        for index, value in enumerate(y):\n            plt.text(value, index, str(value))\n\n    print('plot_my_own_metrics')    \n    plot_my_own_metrics(my_own_metrics)\n    plt.show()\n    \n    tree_index = 0\n    print('Plot '+str(tree_index)+'th tree...')  # one tree use categorical feature to split\n    ax = lgb.plot_tree(bst, tree_index=tree_index, figsize=(64, 36), show_info=['split_gain'])\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clicks.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clicks.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clicks['is_attributed'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clicks['is_attributed'].value_counts(normalize=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Competition data"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Read only first limit rows\nlimit = 20_000_000\n\n#Read only these columns - skip attributed_time \nusecols = ['ip', 'app', 'device', 'os', 'channel', 'click_time', 'is_attributed']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_data = pd.read_csv('../input/talkingdata-adtracking-fraud-detection/train.csv', \n                               nrows=limit, \n                               usecols=usecols, \n                               parse_dates=['click_time'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_data['is_attributed'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_data['is_attributed'].value_counts(normalize=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_test_data = pd.read_csv('../input/talkingdata-adtracking-fraud-detection/test.csv', \n                                    parse_dates=['click_time'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Add new columns for timestamp features day, hour, minute, and second\ncompetition_test_data = competition_test_data.copy()\ncompetition_test_data['day'] = competition_test_data['click_time'].dt.day.astype('uint8')\n# Fill in the rest\ncompetition_test_data['hour'] = competition_test_data['click_time'].dt.hour.astype('uint8')\ncompetition_test_data['minute'] = competition_test_data['click_time'].dt.minute.astype('uint8')\ncompetition_test_data['second'] = competition_test_data['click_time'].dt.second.astype('uint8')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_test_data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_test_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Baseline Score\n\nLet's look at the baseline score for all the features we've made so far."},{"metadata":{"trusted":true},"cell_type":"code","source":"my_own_metrics = {}\nvalid_name_model='Baseline LightGBM Model'\nbst = my_own_train_plot_model(clicks, valid_name_model, my_own_metrics)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1) Which data to use for feature selection?\n\nSince many feature selection methods require calculating statistics from the dataset, should you use all the data for feature selection?"},{"metadata":{},"cell_type":"markdown","source":"Now we have 91 features we're using for predictions. With all these features, there is a good chance the model is overfitting the data. We might be able to reduce the overfitting by removing some features. Of course, the model's performance might decrease. But at least we'd be making the model smaller and faster without losing much performance."},{"metadata":{},"cell_type":"markdown","source":"### 2) Univariate Feature Selection\n\nBelow, use `SelectKBest` with the `f_classif` scoring function to choose 40 features from the 91 features in the data. "},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.feature_selection import SelectKBest, f_classif\nfeature_cols = clicks.columns.drop(['click_time', 'attributed_time', 'is_attributed'])\ntrain, valid, test = get_data_splits(clicks)\n\n# Create the selector, keeping 40 features\nselector = SelectKBest(f_classif, k=40)\n\n# Use the selector to retrieve the best features\nX_new = selector.fit_transform(train[feature_cols], train['is_attributed'])\n\n# Get back the kept features as a DataFrame with dropped columns as all 0s\nselected_features = pd.DataFrame(selector.inverse_transform(X_new),\n                                index=train.index,\n                                columns=feature_cols)\n\n# Find the columns that were dropped\ndropped_columns = selected_features.columns[selected_features.var() == 0]\n#print(dropped_columns)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dropped_columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_name_model='Top40 f_classif Univariate Feature Selection'\nbst = my_own_train_plot_model_v2(train.drop(dropped_columns, axis=1), \n                                 valid.drop(dropped_columns, axis=1),\n                                 valid_name_model, \n                                 my_own_metrics)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3) The best value of K\n\nWith this method we can choose the best K features, but we still have to choose K ourselves. How would you find the \"best\" value of K? That is, you want it to be small so you're keeping the best features, but not so small that it's degrading the model's performance."},{"metadata":{},"cell_type":"markdown","source":"### 4) Use L1 regularization for feature selection\n\nNow try a more powerful approach using L1 regularization. Implement a function `select_features_l1` that returns a list of features to keep.\n\nUse a `LogisticRegression` classifier model with an L1 penalty to select the features. For the model, set:\n- the random state to 7,\n- the regularization parameter to 0.1,\n- and the solver to `'liblinear'`.\n\nFit the model then use `SelectFromModel` to return a model with the selected features.\n\nThe checking code will run your function on a sample from the dataset to provide more immediate feedback."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\nfrom sklearn.feature_selection import SelectFromModel\n\ndef select_features_l1(X, y):\n    \"\"\"Return selected features using logistic regression with an L1 penalty.\"\"\"\n\n    logistic = LogisticRegression(C=0.1, penalty='l1', solver='liblinear', random_state=7).fit(X,y)\n    model = SelectFromModel(logistic, prefit=True)\n\n    X_new = model.transform(X)\n    \n    # Get back the kept features as a DataFrame with dropped columns as all 0s\n    selected_features = pd.DataFrame(model.inverse_transform(X_new), \n                                 index=X.index,\n                                 columns=X.columns)\n\n    # Dropped columns have values of all 0s, keep other columns \n    selected_columns = selected_features.columns[selected_features.var() != 0]\n    \n    return selected_columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_samples = 10_000\nX, y = train[feature_cols][:n_samples], train['is_attributed'][:n_samples]\nselected = select_features_l1(X, y)\n\ndropped_columns = feature_cols.drop(selected)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dropped_columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_name_model='L1 regularization LogisticRegression 0.1'\nbst = my_own_train_plot_model_v2(train.drop(dropped_columns, axis=1), \n                                 valid.drop(dropped_columns, axis=1),\n                                 valid_name_model, \n                                 my_own_metrics)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5) Feature Selection with Trees\n\nSince we're using a tree-based model, using another tree-based model for feature selection might produce better results. What would you do different to select the features using a trees classifier?"},{"metadata":{},"cell_type":"markdown","source":"### 6) Top K features with L1 regularization\n\nHere you've set the regularization parameter `C=0.1` which led to some number of features being dropped. However, by setting `C` you aren't able to choose a certain number of features to keep. What would you do to keep the top K important features using L1 regularization?"},{"metadata":{},"cell_type":"markdown","source":"Congratulations on finishing this course! To keep learning, check out the rest of [our courses](https://www.kaggle.com/learn/overview). The machine learning explainability and deep learning courses are great next skills to learn!"},{"metadata":{},"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161443) to chat with other Learners.*"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}