{"cells":[{"metadata":{},"cell_type":"markdown","source":"**This notebook is an exercise in the [Feature Engineering](https://www.kaggle.com/learn/feature-engineering) course.  You can reference the tutorial at [this link](https://www.kaggle.com/matleonard/baseline-model).**\n\n---\n"},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nIn the exercise, you will work with data from the TalkingData AdTracking competition.  The goal of the competition is to predict if a user will download an app after clicking through an ad. \n\n<center><a href=\"https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection\"><img src=\"https://i.imgur.com/srKxEkD.png\" width=600px></a></center>\n\nFor this course you will use a small sample of the data, dropping 99% of negative records (where the app wasn't downloaded) to make the target more balanced.\n\nAfter building a baseline model, you'll be able to see how your feature engineering and selection efforts improve the model's performance.\n\n## Setup\n\nBegin by running the code cell below to set up the exercise."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set up code checking\nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.feature_engineering.ex1 import *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Baseline Model\n\nThe first thing you'll do is construct a baseline model. We'll begin by looking at the data."},{"metadata":{},"cell_type":"markdown","source":"Data fields  \nEach row of the training data contains a click record, with the following features.  \n\n- ip: ip address of click.\n- app: app id for marketing.\n- device: device type id of user mobile phone (e.g., iphone 6 plus, iphone 7, huawei mate 7, etc.)\n- os: os version id of user mobile phone\n- channel: channel id of mobile ad publisher\n- click_time: timestamp of click (UTC)\n- attributed_time: if user download the app for after clicking an ad, this is the time of the app download\n- is_attributed: the target that is to be predicted, indicating the app was downloaded  \n\nNote that ip, app, device, os, and channel are encoded.\n\nThe test data is similar, with the following differences:\n- click_id: reference for making predictions\n- is_attributed: not included"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\n\nclick_data = pd.read_csv('../input/feature-engineering-data/train_sample.csv',\n                         parse_dates=['click_time'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(click_data.shape)\nclick_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Competition submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_test_data = pd.read_csv('../input/talkingdata-adtracking-fraud-detection/test.csv',\n                         parse_dates=['click_time'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(competition_test_data.shape)\ncompetition_test_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1) Construct features from timestamps\n\nNotice that the `click_data` DataFrame has a `'click_time'` column with timestamp data.\n\nUse this column to create features for the coresponding day, hour, minute and second. \n\nStore these as new integer columns `day`, `hour`, `minute`, and `second` in a new DataFrame `clicks`."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Add new columns for timestamp features day, hour, minute, and second\nclicks = click_data.copy()\nclicks['day'] = clicks['click_time'].dt.day.astype('uint8')\n# Fill in the rest\nclicks['hour'] = clicks['click_time'].dt.hour.astype('uint8')\nclicks['minute'] = clicks['click_time'].dt.minute.astype('uint8')\nclicks['second'] = clicks['click_time'].dt.second.astype('uint8')\n\n# Check your answer\nq_1.check()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clicks.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Uncomment these if you need guidance\n#q_1.hint()\n#q_1.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Competition submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Add new columns for timestamp features day, hour, minute, and second\ncompetition_test_data = competition_test_data.copy()\ncompetition_test_data['day'] = competition_test_data['click_time'].dt.day.astype('uint8')\n# Fill in the rest\ncompetition_test_data['hour'] = competition_test_data['click_time'].dt.hour.astype('uint8')\ncompetition_test_data['minute'] = competition_test_data['click_time'].dt.minute.astype('uint8')\ncompetition_test_data['second'] = competition_test_data['click_time'].dt.second.astype('uint8')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_test_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2) Label Encoding\nFor each of the categorical features `['ip', 'app', 'device', 'os', 'channel']`, use scikit-learn's `LabelEncoder` to create new features in the `clicks` DataFrame. The new column names should be the original column name with `'_labels'` appended, like `ip_labels`."},{"metadata":{"trusted":true},"cell_type":"code","source":"type(clicks['app'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"type(clicks['app'].values.reshape(-1, 1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x = clicks['app'].values.reshape(1, -1)#.tolist()\nx","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Question ??? \n\nclass sklearn.preprocessing.LabelEncoder[source]\nEncode target labels with value between 0 and n_classes-1.\n\nThis transformer should be used to encode **target values**, i.e. y, and not the input X."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import preprocessing\n\ncat_features = ['ip', 'app', 'device', 'os', 'channel']\n\n#encoder = preprocessing.LabelEncoder() - Incorrect, we need a label encoder for each feature\n# Create new columns in clicks using preprocessing.LabelEncoder()\n\nfor feature in cat_features:\n    encoder = preprocessing.LabelEncoder()\n    encoded = encoder.fit_transform(clicks[feature])\n    clicks[feature+'_labels'] = encoded\n    \n    #encoded[feature+'_labels'] = clicks[feature].apply(encoder.fit_transform) - Incorrect \n    #ValueError: y should be a 1d array, got an array of shape () instead.\n    \n    #Competition submission\n    #competition_enencoded = encoder.transform(competition_test_data[feature]) \n    #ValueError: y contains previously unseen labels: [0, 2, 3, 4, 5,\n    #competition_test_data[feature+'_labels'] = competition_enencoded\n\n\n# Check your answer\nq_2.check()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clicks.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### How different is preprocessing.LabelEncoder() from the original dataset encoding ?"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Original dataset size', clicks.shape)\nfor feature in cat_features:\n    print(feature, '{:.2%} different'.format(sum(clicks[feature] != clicks[feature+'_labels'])/clicks.shape[0]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Uncomment these if you need guidance\n#q_2.hint()\n#q_2.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run the next code cell to view your new DataFrame."},{"metadata":{"trusted":true},"cell_type":"code","source":"clicks.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3) One-hot Encoding\n\nIn the code cell above, you used label encoded features.  Would it have also made sense to instead use one-hot encoding for the categorical variables `'ip'`, `'app'`, `'device'`, `'os'`, or `'channel'`?\n\n**Note**: If you're not familiar with one-hot encoding, please check out **[this lesson](https://www.kaggle.com/alexisbcook/categorical-variables)** from the Intermediate Machine Learning course.\n\nRun the following line after you've decided your answer."},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Original dataset size', clicks.shape)\nfor feature in cat_features:\n    print(feature, len(clicks[feature].unique()), len(clicks[feature+'_labels'].unique()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_3.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Typo ??? The ip column has 58,000 values"},{"metadata":{},"cell_type":"markdown","source":"## Train, validation, and test sets\nWith our baseline features ready, we need to split our data into training and validation sets. We should also hold out a test set to measure the final accuracy of the model.\n\n### 4) Train/test splits with time series data\nThis is time series data. Are there any special considerations when creating train/test splits for time series? If so, what are they?\n\nUncomment the following line after you've decided your answer."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_4.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Create train/validation/test splits\n\nHere we'll create training, validation, and test splits. First, `clicks` DataFrame is sorted in order of increasing time. The first 80% of the rows are the train set, the next 10% are the validation set, and the last 10% are the test set."},{"metadata":{"trusted":true},"cell_type":"code","source":"clicks.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_cols = ['day', 'hour', 'minute', 'second', \n                'ip_labels', 'app_labels', 'device_labels',\n                'os_labels', 'channel_labels']\n\nvalid_fraction = 0.1\nclicks_srt = clicks.sort_values('click_time')\nvalid_rows = int(len(clicks_srt) * valid_fraction)\ntrain = clicks_srt[:-valid_rows * 2]\n# valid size == test size, last two sections of the data\nvalid = clicks_srt[-valid_rows * 2:-valid_rows]\ntest = clicks_srt[-valid_rows:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(clicks.shape,'\\n',train.shape,'\\n',valid.shape,'\\n',test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train with LightGBM\n\nNow we can create LightGBM dataset objects for each of the smaller datasets and train the baseline model."},{"metadata":{"trusted":true},"cell_type":"code","source":"import lightgbm as lgb\n\ndtrain = lgb.Dataset(train[feature_cols], label=train['is_attributed'])\ndvalid = lgb.Dataset(valid[feature_cols], label=valid['is_attributed'])\ndtest = lgb.Dataset(test[feature_cols], label=test['is_attributed'])\n\nparam = {'num_leaves': 64, 'objective': 'binary'}\nparam['metric'] = 'auc'\nnum_round = 1000\nbst = lgb.train(param, dtrain, num_round, valid_sets=[dvalid], early_stopping_rounds=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"type(bst)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### ??? TypeError: booster must be dict or LGBMModel\n\n- booster (dict or LGBMModel) – Dictionary returned from lightgbm.train() or LGBMModel instance.\nhttps://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.plot_metric.html","attachments":{}},{"metadata":{"trusted":true},"cell_type":"code","source":"#lgb.plot_metric(bst, metric=metrics.roc_auc_score, dataset_names=[dtrain, dvalid, dtest]) #??? TypeError: booster must be dict or LGBMModel\n#, ax=None, xlim=None, ylim=None, title='Metric during training', xlabel='Iterations', ylabel='auto', figsize=None, dpi=None, grid=True)[source]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- evals_result (dict or None, optional (default=None)) –\n\nThis dictionary used to store all evaluation results of all the items in valid_sets.\n\nhttps://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.train.html","attachments":{}},{"metadata":{},"cell_type":"markdown","source":"- validation_metrics inspired by:\nhttps://github.com/Microsoft/LightGBM/blob/2e93cdab9eee02d4d7f5cb3b6b31128dec94e25e/examples/python-guide/plot_example.py"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Record eval results for plotting\nvalidation_metrics = {}  \n\nbst = lgb.train(param, \n                dtrain, \n                num_round, \n                valid_sets=[dvalid], \n                early_stopping_rounds=10,\n                evals_result=validation_metrics,\n                verbose_eval=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"validation_metrics","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Plot validation AUC during training"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.rcParams[\"figure.figsize\"] = [16,9]\n\nax = lgb.plot_metric(validation_metrics, metric='auc');\n#plt.show();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ML Explainability and taking a closer look at feature importance, individual trees\nInspired by: https://github.com/Microsoft/LightGBM/blob/2e93cdab9eee02d4d7f5cb3b6b31128dec94e25e/examples/python-guide/plot_example.py\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Plot feature importances...')\nax = lgb.plot_importance(bst, max_num_features=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Plot 84th tree...')  # one tree use categorical feature to split\nax = lgb.plot_tree(bst, tree_index=83, figsize=(64, 36), show_info=['split_gain'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Plot 84th tree with graphviz...')\ngraph = lgb.create_tree_digraph(bst, tree_index=83, name='Tree84')\ngraph.render(view=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Download 'Tree84.gv.pdf' from the working directory --->","attachments":{}},{"metadata":{},"cell_type":"markdown","source":"## Evaluate the model\nFinally, with the model trained, we evaluate its performance on the test set. "},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import metrics\n\nypred = bst.predict(test[feature_cols])\nscore = metrics.roc_auc_score(test['is_attributed'], ypred)\nprint(f\"Test score: {score}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This will be our baseline score for the model. When we transform features, add new ones, or perform feature selection, we should be improving on this score. However, since this is the test set, we only want to look at it at the end of all our manipulations. At the very end of this course you'll look at the test score again to see if you improved on the baseline model.\n\n# Keep Going\nNow that you have a baseline model, you are ready to **[use categorical encoding techniques](https://www.kaggle.com/matleonard/categorical-encodings)** to improve it."},{"metadata":{},"cell_type":"markdown","source":"# Submit test predictions to TalkingData AdTracking Fraud Detection Challenge competition using the train_sample 2.3M records from this notebook\n\n# Note that the official competition has its own train and test datasets which should be used instead of the train_sample in this notebook!\n\n# Issue with encoding values so we need to use a new notebook utilizing the competion - much larger - dataset\nhttps://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/"},{"metadata":{},"cell_type":"markdown","source":"## Read official competition data"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data = pd.read_csv('../input/talkingdata-adtracking-fraud-detection/test.csv',\n                         parse_dates=['click_time'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Apply the same Feauture Engineering we applied to the training and validation data"},{"metadata":{},"cell_type":"markdown","source":"# See notebook:\n# TalkingData AdTracking Competition- Baseline Model\n# https://www.kaggle.com/georgezoto/talkingdata-adtracking-competition-baseline-model\n"},{"metadata":{},"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161443) to chat with other Learners.*"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}