{"cells":[{"metadata":{"_uuid":"80f731850042d5be24eb6c28273cc25f678ed0c8"},"cell_type":"markdown","source":"# TalkingData AdTracking Fraud Detection Challenge\n![](https://i.ytimg.com/vi/0V8CSoO23tw/maxresdefault.jpg)\n\nTalkingData is back with another competition: This time, our task is to predict where a click on some advertising is fraudlent given a few basic attributes about the device that made the click. What sets this competition apart is the sheer scale of the dataset: with 240 million rows it might be the biggest one I've seen on Kaggle so far.\n\nThere are some similarities with the last competition TalkingData launched: https://www.kaggle.com/c/talkingdata-mobile-user-demographics - that competition was about predicting the demographics of a user given their activity, and you can view this as a similar problem (predicting whether a user is real or not given their activity). However, that competition was plagued by a [leak](https://www.kaggle.com/wiki/Leakage) where the dataset wasn't sorted properly and certain portions of the dataset had different demographic distribtions. This meant that by adding the row ID as a feature you could get a huge boost in performance. Let's hope TalkingData have learnt their lesson this time around. 😉\n\nLooking at the evaluation page, we can see that the evaluation metric used is** ROC-AUC** (the area under a curve on a Receiver Operator Characteristic graph).\nIn english, this means a few important things:\n* This competition is a **binary classification** problem - i.e. our target variable is a binary attribute (Is the user making the click fraudlent or not?) and our goal is to classify users into \"fraudlent\" or \"not fraudlent\" as well as possible\n* Unlike metrics such as [LogLoss](http://www.exegetic.biz/blog/2015/12/making-sense-logarithmic-loss/), the AUC score only depends on **how well you well you can separate the two classes**. In practice, this means that only the order of your predictions matter,\n    * As a result of this, any rescaling done to your model's output probabilities will have no effect on your score. In some other competitions, adding a constant or multiplier to your predictions to rescale it to the distribution can help but that doesn't apply here.\n  "},{"metadata":{},"cell_type":"markdown","source":"### Looking at the columns\n\nAccording to the data page, our data contains:\n\n* `ip`: ip address of click\n* `app`: app id for marketing\n* `device`: device type id of user mobile phone (e.g., iphone 6 plus, iphone 7, huawei mate 7, etc.)\n* `os`: os version id of user mobile phone\n* `channel`: channel id of mobile ad publisher\n* `click_time`: timestamp of click (UTC)\n* `attributed_time`: if user download the app for after clicking an ad, this is the time of the app download\n* `is_attributed`: the target that is to be predicted, indicating the app was downloaded\n\n**A few things of note:**\n* If you look at the data samples above, you'll notice that all these variables are encoded - meaning we don't know what the actual value corresponds to - each value has instead been assigned an ID which we're given. This has likely been done because data such as IP addresses are sensitive, although it does unfortunately reduce the amount of feature engineering we can do on these.\n* The `attributed_time` variable is only available in the training set - it's not immediately useful for classification but it could be used for some interesting analysis (for example, one could fill in the variable in the test set by building a model to predict it).\n\nFor each of our encoded values, let's look at the number of unique values:"},{"metadata":{},"cell_type":"markdown","source":"# Imports\nWe are using a typical data science stack: ``numpy``, ``pandas``, ``sklearn``, ``matplotlib``."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport mlcrate as mlc\nimport os\nimport gc\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\npal = sns.color_palette()\n\nprint('# File sizes')\nfor f in os.listdir('../input'):\n    if 'zip' not in f:\n        print(f.ljust(30) + str(round(os.path.getsize('../input/' + f) / 1000000, 2)) + 'MB')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"33c5eec493e287aae60a9226d58fa61545d3a77a"},"cell_type":"markdown","source":"Wow, that is some really big data. Unfortunately we don't have enough kernel memory to load the full dataset into memory; however we can get a glimpse at some of the statistics:"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import subprocess\nprint('# Line count:')\nfor file in ['train.csv', 'test.csv', 'train_sample.csv']:\n    lines = subprocess.run(['wc', '-l', '../input/{}'.format(file)], stdout=subprocess.PIPE).stdout.decode('utf-8')\n    print(lines, end='', flush=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b1ec574cb675150940b453bb9fbd648a5285390f"},"cell_type":"markdown","source":"That makes **185 million rows** in the training set and ** 19 million** in the test set. Handily the organisers have provided a `train_sample.csv` which contains 100K rows in case you don't want to download the full data\n\nFor this analysis, I'm going to use the first 1M rows of the training and test datasets.\n\n## Glimpse of Data"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"67335c75ba85c06463285be0cf78ccdc68e38609"},"cell_type":"code","source":"df_train = pd.read_csv('../input/train.csv', nrows=1000000)\ndf_test = pd.read_csv('../input/test.csv', nrows=1000000)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Training set"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e44fbc483dc19be1b44f2aba7d4356ac183534e9"},"cell_type":"code","source":"print('Training set:')\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Test set"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"65a54cab2e0492a5f984d761f7efbc7ed5377aee"},"cell_type":"code","source":"print('Test set:')\ndf_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"features = df_train.columns.values[0:30]\nunique_max_train = []\nunique_max_test = []\nfor feature in features:\n    values = df_train[feature].value_counts()\n    perc = values.max() / ( df_train.shape[0]*100 )\n    unique_max_train.append([feature, values.max(), values.idxmax(), perc])\n    \nnp.transpose((pd.DataFrame(unique_max_train, columns=['Feature', 'Max duplicados', 'Valor', 'Percentage'])).\\\n            sort_values(by = 'Max duplicados', ascending=False).head(15))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Target Distribution"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_uuid":"8eb35b5d9e773650c20b43484f598a3dea5ecbb0"},"cell_type":"code","source":"plt.figure(figsize=(15, 8))\ncols = ['ip', 'app', 'device', 'os', 'channel']\nuniques = [len(df_train[col].unique()) for col in cols]\nsns.set(font_scale=1.2)\nax = sns.barplot(cols, uniques, palette=pal, log=True)\nax.set(xlabel='Feature', ylabel='log(unique count)', title='Number of unique values per feature')\nfor p, uniq in zip(ax.patches, uniques):\n    height = p.get_height()\n    ax.text(p.get_x()+p.get_width()/2.,\n            height + 10,\n            uniq,\n            ha=\"center\") \n# for col, uniq in zip(cols, uniques):\n#     ax.text(col, uniq, uniq, color='black', ha=\"center\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Examine Missing Value\nNext we can look at the number and percentage of missing values in each column"},{"metadata":{"trusted":true},"cell_type":"code","source":"# checking missing data\ntotal = df_train.isnull().sum().sort_values(ascending = False)\npercent = (df_train.isnull().sum()/df_train.isnull().count()*100).sort_values(ascending = False)\nmissing__train_data  = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing__train_data.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Package called missingno (https://github.com/ResidentMario/missingno) ``!pip install quilt``"},{"metadata":{"trusted":true},"cell_type":"code","source":"# !pip install quilt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Nullity Matrix\nThe msno.matrix nullity matrix is a data-dense display which lets you quickly visually analyse data completion\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# import missingno as msno\n# msno.matrix(df_train.head(20000))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Heatmap\nThe missingno correlation heatmap measures nullity correlation: how strongly the presence or absence of one variable affects the presence of another:"},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"# msno.heatmap(df_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"anually dealing with missing values will often improve model performance\n\n"},{"metadata":{"_uuid":"922f91e941b7b268c1fcd0aedb777ad4d24b7fe1"},"cell_type":"markdown","source":"##  Encoded variables statistics\n\nAlthough the actual values of these variables aren't helpful for us, it can still be useful to know what their distributions are. Note these statistics are computed on 1M samples, and so will be higher for the full dataset."},{"metadata":{"_kg_hide-input":true,"trusted":true,"scrolled":false,"_uuid":"8d3bbdefa393b9365b2b58aeab8d2677c817a971"},"cell_type":"code","source":"for col, uniq in zip(cols, uniques):\n    counts = df_train[col].value_counts()\n\n    sorted_counts = np.sort(counts.values)\n    fig = plt.figure()\n    ax = fig.add_subplot(1, 1, 1)\n    line, = ax.plot(sorted_counts, color='red')\n    ax.set_yscale('log')\n    plt.title(\"Distribution of value counts for {}\".format(col))\n    plt.ylabel('log(Occurence count)')\n    plt.xlabel('Index')\n    plt.show()\n    \n    fig = plt.figure()\n    ax = fig.add_subplot(1, 1, 1)\n    plt.hist(sorted_counts, bins=50)\n    ax.set_yscale('log', nonposy='clip')\n    plt.title(\"Histogram of value counts for {}\".format(col))\n    plt.ylabel('Number of IDs')\n    plt.xlabel('Occurences of value for ID')\n    plt.show()\n    \n    max_count = np.max(counts)\n    min_count = np.min(counts)\n    gt = [10, 100, 1000]\n    prop_gt = []\n    for value in gt:\n        prop_gt.append(round((counts > value).mean()*100, 2))\n    print(\"Variable '{}': | Unique values: {} | Count of most common: {} | Count of least common: {} | count>10: {}% | count>100: {}% | count>1000: {}%\".format(col, uniq, max_count, min_count, *prop_gt))\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Imputation strategies\n- Test- Time series imputation\n\n- Non-time-series specific method\n\n- Mean imputation\n\n- Median imputation\n\n- Mode imputation\n\n- Calculate the appropriate measure and replace NAs with the values.\n\n- Appropriate for stationary time series, for example, white noise data\n\n- Random sample imputation replace missing values with observations randomly selected from the remaining (either of it or just some section of it)\n\n### Time-Series specific method\n- Last observation carried forward (LOCF)\n- Next observation carried backward (NOCB)\n\nA good article is Data Cleaning: A guide to dealing with NA values: https://www.linkedin.com/pulse/data-cleaning-guide-dealing-na-values-karan-rajwanshi/ Talk about types of missing data:\n\n- Missing Completely at Random (MCAR)\n- Missing at Random (MAR)\n- Missing Not at Random (MNAR)"},{"metadata":{},"cell_type":"markdown","source":"# Exploratory Data Analysis\nExploratory Data Analysis (EDA) is an open-ended process where we calculate statistics and make figures to find trends, anomalies, patterns, or relationships within the data"},{"metadata":{"_uuid":"f46b1c067e51ba664e1c8a22434919456d7f4774"},"cell_type":"markdown","source":"## What we're trying to predict"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_uuid":"efe6fde36b993a865e7a3f592e01061b1b49caa2"},"cell_type":"code","source":"plt.figure(figsize=(8, 8))\nsns.set(font_scale=1.2)\nmean = (df_train.is_attributed.values == 1).mean()\nax = sns.barplot(['Fraudulent (1)', 'Not Fradulent (0)'], [mean, 1-mean], palette=pal)\nax.set(xlabel='Target Value', ylabel='Probability', title='Target value distribution')\nfor p, uniq in zip(ax.patches, [mean, 1-mean]):\n    height = p.get_height()\n    ax.text(p.get_x()+p.get_width()/2.,\n            height+0.01,\n            '{}%'.format(round(uniq * 100, 2)),\n            ha=\"center\") ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"219351441d94b83e4e0ede20dcd7ac01973b7251"},"cell_type":"markdown","source":"Wow, that's a really unbalanced dataset. Only 0.2% of the dataset is made up of fradulent clicks. This means that any models we run on the data will either need to be robust against class imbalance or will require some data resampling."},{"metadata":{},"cell_type":"markdown","source":"# Correlation\nNow that we have dealt with the categorical variables and the outliers, let's continue with the EDA. One way to try and understand the data is by looking for correlations between the features and the target. We can calculate the Pearson correlation coefficient between every variable and the target using the .corr dataframe method.\n\nThe correlation coefficient is not the greatest method to represent \"relevance\" of a feature, but it does give us an idea of possible relationships within the data. [Some general interpretations](http://www.statstutor.ac.uk/resources/uploaded/pearsons.pdf) of the absolute value of the correlation coefficent are:\n\n- .00-.19 “very weak”\n- .20-.39 “weak”\n- .40-.59 “moderate”\n- .60-.79 “strong”\n- .80-1.0 “very strong”"},{"metadata":{"trusted":true},"cell_type":"code","source":"corrs = df_train.corr()\ncorrs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (20, 8))\n\n# Heatmap of correlations\nsns.heatmap(corrs, cmap = plt.cm.RdYlBu_r, vmin = -0.25, annot = True, vmax = 0.6)\nplt.title('Correlation Heatmap');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Reference: \n\nTalkingData AdTracking EDA by @anokas\nhttps://www.kaggle.com/anokas/talkingdata-adtracking-eda"},{"metadata":{},"cell_type":"markdown","source":"# The end"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}