{"cells":[{"metadata":{"_uuid":"80f731850042d5be24eb6c28273cc25f678ed0c8"},"cell_type":"markdown","source":"# TalkingData AdTracking Fraud Detection Challenge\n\nTalkingData is back with another competition: This time, our task is to predict where a click on some advertising is fraudlent given a few basic attributes about the device that made the click. What sets this competition apart is the sheer scale of the dataset: with 240 million rows it might be the biggest one I've seen on Kaggle so far.\n\nThere are some similarities with the last competition TalkingData launched: https://www.kaggle.com/c/talkingdata-mobile-user-demographics - that competition was about predicting the demographics of a user given their activity, and you can view this as a similar problem (predicting whether a user is real or not given their activity). However, that competition was plagued by a [leak](https://www.kaggle.com/wiki/Leakage) where the dataset wasn't sorted properly and certain portions of the dataset had different demographic distribtions. This meant that by adding the row ID as a feature you could get a huge boost in performance. Let's hope TalkingData have learnt their lesson this time around. 😉\n\nLooking at the evaluation page, we can see that the evaluation metric used is** ROC-AUC** (the area under a curve on a Receiver Operator Characteristic graph).\nIn english, this means a few important things:\n* This competition is a **binary classification** problem - i.e. our target variable is a binary attribute (Is the user making the click fraudlent or not?) and our goal is to classify users into \"fraudlent\" or \"not fraudlent\" as well as possible\n* Unlike metrics such as [LogLoss](http://www.exegetic.biz/blog/2015/12/making-sense-logarithmic-loss/), the AUC score only depends on **how well you well you can separate the two classes**. In practice, this means that only the order of your predictions matter,\n    * As a result of this, any rescaling done to your model's output probabilities will have no effect on your score. In some other competitions, adding a constant or multiplier to your predictions to rescale it to the distribution can help but that doesn't apply here.\n  \nIf you want a more intuitive explanation of how AUC works, I recommend [this post](https://stats.stackexchange.com/questions/132777/what-does-auc-stand-for-and-what-is-it).\n\nLet's dive right in by looking at the data we're given:"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport mlcrate as mlc\nimport os\nimport gc\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\npal = sns.color_palette()\n\nprint('# File sizes')\nfor f in os.listdir('../input'):\n    if 'zip' not in f:\n        print(f.ljust(30) + str(round(os.path.getsize('../input/' + f) / 1000000, 2)) + 'MB')","execution_count":18,"outputs":[]},{"metadata":{"_uuid":"33c5eec493e287aae60a9226d58fa61545d3a77a"},"cell_type":"markdown","source":"Wow, that is some really big data. Unfortunately we don't have enough kernel memory to load the full dataset into memory; however we can get a glimpse at some of the statistics:"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import subprocess\nprint('# Line count:')\nfor file in ['train.csv', 'test.csv', 'train_sample.csv']:\n    lines = subprocess.run(['wc', '-l', '../input/{}'.format(file)], stdout=subprocess.PIPE).stdout.decode('utf-8')\n    print(lines, end='', flush=True)","execution_count":19,"outputs":[]},{"metadata":{"_uuid":"b1ec574cb675150940b453bb9fbd648a5285390f"},"cell_type":"markdown","source":"That makes **185 million rows** in the training set and ** 19 million** in the test set. Handily the organisers have provided a `train_sample.csv` which contains 100K rows in case you don't want to download the full data\n\nFor this analysis, I'm going to use the first 1M rows of the training and test datasets.\n\n## Data overview"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"67335c75ba85c06463285be0cf78ccdc68e38609"},"cell_type":"code","source":"df_train = pd.read_csv('../input/train.csv', nrows=1000000)\ndf_test = pd.read_csv('../input/test.csv', nrows=1000000)","execution_count":20,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e44fbc483dc19be1b44f2aba7d4356ac183534e9"},"cell_type":"code","source":"print('Training set:')\ndf_train.head()","execution_count":24,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"65a54cab2e0492a5f984d761f7efbc7ed5377aee"},"cell_type":"code","source":"print('Test set:')\ndf_test.head()","execution_count":26,"outputs":[]},{"metadata":{"_uuid":"ee6a77f214c58bb3f1a03d903108a563ece0ad21"},"cell_type":"markdown","source":"### Looking at the columns\n\nAccording to the data page, our data contains:\n\n* `ip`: ip address of click\n* `app`: app id for marketing\n* `device`: device type id of user mobile phone (e.g., iphone 6 plus, iphone 7, huawei mate 7, etc.)\n* `os`: os version id of user mobile phone\n* `channel`: channel id of mobile ad publisher\n* `click_time`: timestamp of click (UTC)\n* `attributed_time`: if user download the app for after clicking an ad, this is the time of the app download\n* `is_attributed`: the target that is to be predicted, indicating the app was downloaded\n\n**A few things of note:**\n* If you look at the data samples above, you'll notice that all these variables are encoded - meaning we don't know what the actual value corresponds to - each value has instead been assigned an ID which we're given. This has likely been done because data such as IP addresses are sensitive, although it does unfortunately reduce the amount of feature engineering we can do on these.\n* The `attributed_time` variable is only available in the training set - it's not immediately useful for classification but it could be used for some interesting analysis (for example, one could fill in the variable in the test set by building a model to predict it).\n\nFor each of our encoded values, let's look at the number of unique values:"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_uuid":"8eb35b5d9e773650c20b43484f598a3dea5ecbb0"},"cell_type":"code","source":"plt.figure(figsize=(15, 8))\ncols = ['ip', 'app', 'device', 'os', 'channel']\nuniques = [len(df_train[col].unique()) for col in cols]\nsns.set(font_scale=1.2)\nax = sns.barplot(cols, uniques, palette=pal, log=True)\nax.set(xlabel='Feature', ylabel='log(unique count)', title='Number of unique values per feature')\nfor p, uniq in zip(ax.patches, uniques):\n    height = p.get_height()\n    ax.text(p.get_x()+p.get_width()/2.,\n            height + 10,\n            uniq,\n            ha=\"center\") \n# for col, uniq in zip(cols, uniques):\n#     ax.text(col, uniq, uniq, color='black', ha=\"center\")","execution_count":53,"outputs":[]},{"metadata":{"_uuid":"922f91e941b7b268c1fcd0aedb777ad4d24b7fe1"},"cell_type":"markdown","source":"##  Encoded variables statistics\n\nAlthough the actual values of these variables aren't helpful for us, it can still be useful to know what their distributions are. Note these statistics are computed on 1M samples, and so will be higher for the full dataset."},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"8d3bbdefa393b9365b2b58aeab8d2677c817a971"},"cell_type":"code","source":"for col, uniq in zip(cols, uniques):\n    counts = df_train[col].value_counts()\n\n    sorted_counts = np.sort(counts.values)\n    fig = plt.figure()\n    ax = fig.add_subplot(1, 1, 1)\n    line, = ax.plot(sorted_counts, color='red')\n    ax.set_yscale('log')\n    plt.title(\"Distribution of value counts for {}\".format(col))\n    plt.ylabel('log(Occurence count)')\n    plt.xlabel('Index')\n    plt.show()\n    \n    fig = plt.figure()\n    ax = fig.add_subplot(1, 1, 1)\n    plt.hist(sorted_counts, bins=50)\n    ax.set_yscale('log', nonposy='clip')\n    plt.title(\"Histogram of value counts for {}\".format(col))\n    plt.ylabel('Number of IDs')\n    plt.xlabel('Occurences of value for ID')\n    plt.show()\n    \n    max_count = np.max(counts)\n    min_count = np.min(counts)\n    gt = [10, 100, 1000]\n    prop_gt = []\n    for value in gt:\n        prop_gt.append(round((counts > value).mean()*100, 2))\n    print(\"Variable '{}': | Unique values: {} | Count of most common: {} | Count of least common: {} | count>10: {}% | count>100: {}% | count>1000: {}%\".format(col, uniq, max_count, min_count, *prop_gt))\n    ","execution_count":79,"outputs":[]},{"metadata":{"_uuid":"f46b1c067e51ba664e1c8a22434919456d7f4774"},"cell_type":"markdown","source":"## What we're trying to predict"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_uuid":"efe6fde36b993a865e7a3f592e01061b1b49caa2"},"cell_type":"code","source":"plt.figure(figsize=(8, 8))\nsns.set(font_scale=1.2)\nmean = (df_train.is_attributed.values == 1).mean()\nax = sns.barplot(['Fraudulent (1)', 'Not Fradulent (0)'], [mean, 1-mean], palette=pal)\nax.set(xlabel='Target Value', ylabel='Probability', title='Target value distribution')\nfor p, uniq in zip(ax.patches, [mean, 1-mean]):\n    height = p.get_height()\n    ax.text(p.get_x()+p.get_width()/2.,\n            height+0.01,\n            '{}%'.format(round(uniq * 100, 2)),\n            ha=\"center\") ","execution_count":63,"outputs":[]},{"metadata":{"_uuid":"219351441d94b83e4e0ede20dcd7ac01973b7251"},"cell_type":"markdown","source":"Wow, that's a really unbalanced dataset. Only 0.2% of the dataset is made up of fradulent clicks. This means that any models we run on the data will either need to be robust against class imbalance or will require some data resampling."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}