{"cells":[{"metadata":{"_cell_guid":"565b7d66-eba6-41de-b246-0475f2052616","_uuid":"d3151b4c619dfb6ea34cb267d419ff6661f9c61c"},"cell_type":"markdown","source":"# TalkingData AdTracking Fraud Detection Challenge\n\nThis notebook presents some plots that simply show click frequency over time, for one value, e.g. all records with app==3.\n\nEach plot is 600 pixels wide, one pixel is one second, so the width represents 10 minutes. The pixel value represents the click count for that second (with log scaling to make it clearer), so one hour is 6 rows of pixels, and one day is 144 rows. (The [color map](http://matplotlib.org/users/colormaps.html) is 'afmhot', black means 0 clicks.)\n\nThe training set spans 264460 seconds (3.06 days), and the test set 39600 seconds (0.46 days), though nearly half of the test set is blank (no records).\n\nBeyond those facts, you're on your own, these plots are good for a general view but not specific details :)\n\nI've also added a [script to generate all the plots](http://www.kaggle.com/jtrotman/generate-temporal-click-count-data-plots) for the top N most common values in each column, this notebook is for selected highlights.\n\nI generated these visualisations expecting to just see fairly uniform/noisy variation over time.\n\nInstead, many of the plots show extreme regularity, mechanistic clicking which obviously cannot be natural human behaviour... (The plots are the sum of lots of overlapping behaviour, even so, regular patterns presumably from heavy click automation are highly visible.)\n\nThe target (*is_attributed*) is not used to generate these plots, but I'd guess that feature engineering that describes the (conditional) density of clicks at a point in time will be needed to get the best performance...\n\nIn other words, good, predictive features will encode \"is the record part of one of these mechanistic patterns?\""},{"metadata":{"_cell_guid":"d322ad32-45e2-4a5f-ba09-8634e2557f80","collapsed":true,"_uuid":"b0c7954089f46f850646b8c0f7fc46c586c7e151","trusted":false},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport os, sys, time\n\ndtypes = {\n        'ip'            : 'uint32',\n        'app'           : 'uint16',\n        'device'        : 'uint16',\n        'os'            : 'uint16',\n        'channel'       : 'uint16'\n        }\n\n# read a subset, runs out of memory otherwise ('os' seems least interesting)\nfields = [ 'ip', 'app', 'device', 'channel' ]\nto_read = fields + [ 'click_time' ]\n\ntrain_df  = pd.read_csv('../input/train.csv', usecols=to_read, parse_dates=['click_time'], dtype=dtypes) #, nrows=72000000)\ntest_df   = pd.read_csv('../input/test.csv', usecols=to_read, parse_dates=['click_time'], dtype=dtypes)\nprint('Loaded', train_df.shape, test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e9a72175-790c-4d69-810e-e018c165a245","collapsed":true,"_uuid":"fa7a35994123c3e60dd69666c1cdac18270ee5c7","trusted":false},"cell_type":"code","source":"def datetime_to_deltas(series, delta=np.timedelta64(1, 's')):\n    t0 = series.min()\n    return ((series-t0)/delta).astype(np.int32)\n\ntrain_df['sec'] = datetime_to_deltas(train_df.click_time)\ntest_df['sec'] = datetime_to_deltas(test_df.click_time)\nprint('Added seconds')\n\ntrain_df.drop('click_time', axis=1, inplace=True)\ntest_df.drop('click_time', axis=1, inplace=True)\nprint('Dropped click_time')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","collapsed":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":false},"cell_type":"code","source":"from matplotlib.colors import LogNorm\n\n# e.g. train_df.loc[train_df.ip==234]\ndef generate_plot(df):\n    w = 600\n    n = df.sec.max()+1\n    l = int(np.ceil(n/float(w))*w)\n    c = np.zeros(l, dtype=np.float32)\n    np.add.at(c, df.sec.values, 1)\n    print(f'total clicks {c.sum():.0f} \\t max clicks {c.max():.0f} \\t mean click rate {c.mean():.02f} ')\n    return c.reshape((-1,w))\n\ndef show(pix, title):\n    pix += 1 # matplotlib restriction\n    ysize = 5 if pix.shape[0] < 400 else 8\n    fig, ax0 = plt.subplots(figsize=(18, ysize))\n    ax0.invert_yaxis()\n    ax0.set_yticks(np.arange(0, pix.shape[0], 144), False)\n    ax0.set_yticks(np.arange(0, pix.shape[0], 6), True)\n    ax0.set_xticks(np.arange(0, pix.shape[1], 60), False)\n    ax0.set_xticks(np.arange(0, pix.shape[1], 10), True)\n    c = ax0.pcolormesh(pix, norm=LogNorm(vmin=1, vmax=pix.max()), cmap='afmhot')\n    ax0.set_title(title)\n    return fig.colorbar(c, ax=ax0)\n\ndef gen_show(df, col, value):\n    idx = df[col]==value\n    if idx.sum()<1:\n        print('Not found!')\n    else:\n        pix = generate_plot(df.loc[idx])\n        show(pix, f'{col} {value}')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"3ff1b87c-f472-44ae-b09a-9d1a52cfef22","_uuid":"7fb4e5a8a0d1ac33c46c1a81d4697e69488a676f"},"cell_type":"markdown","source":"# Training Set\n\n## Apps"},{"metadata":{"_cell_guid":"129f18ba-55ce-478a-b239-79da85c5c0d3","_uuid":"839de885ba76b8e3a37498a487ff5a240e02c531"},"cell_type":"markdown","source":"App 3, most common in train set. Not much striking here, it serves as a guide to the general form of the plots, e.g. the darker bands indicate night time. The black pixels at the top indicate 0 clicks, the data starts off slowly... The x-axis is marked with seconds 0..600 and the y-axis counts 10-minute rows, 0..144 for one day.\n\nAlso the plotting function lists some stats: total clicks for the plot, the max clicks (in one second), and the mean clicks per second.\n"},{"metadata":{"_cell_guid":"889a29b2-8106-4228-b233-6fae5b29ce28","collapsed":true,"_uuid":"5dbaf4247215ab0ec7a4e7e7dc724d24d3414666","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'app', 3)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"abb15334-8c1d-4da4-997d-5d3b58b8a27b","_uuid":"049c335ad13b162cbea7350f33cc21b6f0c79d5d"},"cell_type":"markdown","source":"App 1, strong periodic behaviour every minute, and a comb-like pattern on the right (some hourly regularity?)\n"},{"metadata":{"_cell_guid":"25140e68-8376-4e00-b08a-0f13f3b1b1bf","collapsed":true,"_uuid":"affd527077e2af685c474778fe1f8484240a25b3","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'app', 1)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"1c0faad5-9f81-4e12-9d8f-3955fe24972f","_uuid":"86b970d5bab39135a92c090131c4b816f3fc9c71"},"cell_type":"markdown","source":"App 7, heavy but short spikes.\n"},{"metadata":{"_cell_guid":"fb402c96-ca1f-4bf0-a3cb-977f0f410183","collapsed":true,"_uuid":"fb953e241fdaa072a390b8849c2743d25b8436ed","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'app', 7)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a9d9ff83-f157-45dc-a29a-87ea68b2538e","_uuid":"d1d5b57059dd89d5b858de0bd6b3a2d4f3a8cadb"},"cell_type":"markdown","source":"App 20, very striking mechanistic behaviour, busy one minute, quiet the next!\n"},{"metadata":{"_cell_guid":"ad2a8dbb-af54-495e-8b10-776271502545","collapsed":true,"_uuid":"66e508576ce48d928307fccc7f0a55215aa33cca","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'app', 20)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ccaf88d8-2ac4-4a99-926a-d4038038ebea","_uuid":"314ff3855e6ec266801acbde802fe1e19af1f6fb"},"cell_type":"markdown","source":"App 46, too regular to be human (clicks every ~5 seconds), plus some streaks that last 20-30 seconds.\n"},{"metadata":{"_cell_guid":"a6570c77-cb83-45b8-9913-9965043ca9ed","collapsed":true,"_uuid":"95e73c7b6c98b4a98dcfc3c7bf3b483dc8072c69","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'app', 46)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ea01eb62-b9ab-405b-95f6-5ae2defe6d05","_uuid":"03333b879dba3cd15b1a860e29ea57f7b8272b1f"},"cell_type":"markdown","source":"App 151, this is the most clicked app in the training set that has no *is_attributed==1* records at all (a target rate of 0). Zooming in on the plot it's clearly a highly regular mechanistic pattern.\n"},{"metadata":{"_cell_guid":"4e2d69b4-221a-4878-aa3c-a870d297bca2","collapsed":true,"_uuid":"b8ba9cd9980576bd3f25f70ac2139f0530087385","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'app', 151)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ead6c8b0-b6b6-46f4-b32b-ba3c7f31b0eb","_uuid":"dcdc82d64e7b1cacd64acd8c4c6814a7bb68a40d"},"cell_type":"markdown","source":"App 56 is the second most clicked app with no positive targets (3rd is 183, 4th is 93), these ar similar to app 151 above but with a very heavy spike in day 2.\n"},{"metadata":{"_cell_guid":"e98e2162-281f-48d6-8218-803c1c531ad3","collapsed":true,"_uuid":"732cc5daf90378fb83c664727cc868b293aadf24","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'app', 56)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2f160c6d-6d50-4c66-b1e0-5481e32662a4","_uuid":"49c115f43b0b50067efe0a3d254935e4f31e71d4"},"cell_type":"markdown","source":"## Channels"},{"metadata":{"_cell_guid":"b9c24004-cb06-4c7a-a74b-b3585a9f96bb","_uuid":"abe3483ba266ab578d0b9b8dbed61beb6b916cb1"},"cell_type":"markdown","source":"Channel 236 and 237, more periodic behaviour, phase shifting over time, in different ways. (This is a real whoa! moment... anyone have any ideas on the cause of this?)\n"},{"metadata":{"_cell_guid":"ca54ceb3-8f11-44bd-a203-e7bbf9fe4788","collapsed":true,"_uuid":"a638d16adb1b9b7bb37e910a2f247f8f17e52663","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'channel', 236)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"19b90901-15cf-478a-99ff-284a12dfa1b5","_uuid":"8a2c21a939faa9e6b9edc75dd0d3cb81f4fae439"},"cell_type":"markdown","source":"Channel 105 - more minutely switching, the exact same point in every minute, no phase drift...\n"},{"metadata":{"_cell_guid":"eff0a1be-b14c-4572-9129-eb4785d48af0","collapsed":true,"_uuid":"d158583221916e764c4b1508c0ad592ee3bb2181","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'channel', 105)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"27108de9-98db-455d-990a-bb29bf5efbc9","_uuid":"bb60a591bdc3aeb1c74d307a5b949dafa4d2ba56"},"cell_type":"markdown","source":"Channel 244 - some fine horizontal banding particularly visible in darker night-time periods\n."},{"metadata":{"_cell_guid":"3d4fa6db-4b5e-416c-bbf3-13e85ad48e04","collapsed":true,"_uuid":"0f63e4a420d7b5d35ecd62d5a3ea8c5dd3dd10e4","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'channel', 244)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ac1872e5-810d-41da-a8e2-42066cc05510","_uuid":"d13f855fb027b4737504cc0f2092a7d6d2ce28a1"},"cell_type":"markdown","source":"Channel 419 is an example of what I expected to see: occasional clicks, some smooth day/night variation, no odd patterns. There are 10,371 clicks and for 6,675 *is_attributed==1*, a download rate of 64%, very high: apparently channel 419 is not a lucrative target for click fraud."},{"metadata":{"_cell_guid":"0a51df42-0d2d-499c-bd08-f55c7f2aeb92","collapsed":true,"_uuid":"02d9d896d1716164a8d52fe454f571e79d84e4c6","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'channel', 419)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ccbe26ebe10762e53c4fb63de422b60988ed88bd"},"cell_type":"markdown","source":"Channel 272 is low volume, but with no *is_attributed==1*, the regularity of the presumably bot-generated clicks is easier to see, with 5-6 second gaps. (Unfortunately the test set only has 79 records for channel 272... but are there other channels in the test set with click patterns with easy-to-detect regularity like this? ;)"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"15a435c581369df61e8455339c2497c75cbc9bc5"},"cell_type":"code","source":"gen_show(train_df, 'channel', 272)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4434a2b524f11d3621114491bdb1a9b4aea92c06"},"cell_type":"markdown","source":"Channel 114 is also low volume: 677 clicks but 629 downloads, a ~93% *is_attributed* rate, kind of abnormally high, perhaps a special offer? Also very few nighttime clicks, could it be a *channel* targeting young children or pensioners? (Just my speculation.)"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"286b58b6bfd8a842c843e562fce172edbc94f26a"},"cell_type":"code","source":"gen_show(train_df, 'channel', 114)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a6c74b52-08ac-4e35-9853-076fe5c0267a","_uuid":"9ef4b67cad3cb7f7c8156cf7a8eddd88a89e64f5"},"cell_type":"markdown","source":"## Devices"},{"metadata":{"_cell_guid":"ef5b05f3-4693-4e6c-ae6f-ea8e703b46da","_uuid":"3b21547bacdc5ce1d355e0c392011a9b1bb14e2b"},"cell_type":"markdown","source":"Device 2 (2nd most common) some faint traces of patterns.\n"},{"metadata":{"_cell_guid":"b9cea422-ce60-4626-bb0b-df79ec193b14","collapsed":true,"_uuid":"57acb0947862db4d4e8dd97e2e3e5d1bfc1f9425","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'device', 2)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"238e6d22-bf7f-4414-8711-edf39092415f","_uuid":"10ba20061aad6b795dd348dcc6a885bbcb36ebc1"},"cell_type":"markdown","source":"Devices 3032, 3543, 3866, total cut outs in time. (Seems to cut off bottom of image in the notebook...)\n"},{"metadata":{"_cell_guid":"3bff4786-8dc7-46eb-ac1b-ba5443328eb1","collapsed":true,"_uuid":"22bd43cbefcabf9691356ff0e7df1f2d45c2dab3","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'device', 3543)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0b10c80c-2a42-48bd-bd06-a59f918f495d","_uuid":"7163c2b93e29dd75cc31a23f8002c1a467269124"},"cell_type":"markdown","source":"Device 154, normal looking sporadic clicks, but with one streak on day 1.\n"},{"metadata":{"_cell_guid":"f7c93275-8f98-4fab-856c-fccc9966db19","collapsed":true,"_uuid":"d6577fe6f1fd751015c9622570b0d716c778c30c","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'device', 154)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a819d3e7-e663-4ea5-8e49-b9e5b4c405ab","_uuid":"4f49195c2748ee8c9a79d81cef6a66ddae74eacf"},"cell_type":"markdown","source":"## IPs"},{"metadata":{"_cell_guid":"67b0d694-600c-40ac-a60e-ade2f440d612","_uuid":"f52604cc9c3ea8759f174863d74540da39fc4264"},"cell_type":"markdown","source":"IP 5314 and 5348 - the most active IPs in train - featuring an hour or so of abnormally low activity on day 2.\n"},{"metadata":{"_cell_guid":"7ecd7898-ea23-482d-8bab-fb7802cf25e2","collapsed":true,"_uuid":"70490bcce11761a933dae189921ff8ec05ea1b51","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'ip', 5314)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"42a04122-1d81-44fd-9825-9b6ec444ca7e","collapsed":true,"_uuid":"392682783a40269e28b0f10a1cf1ec5717b035ab","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'ip', 5348) ","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8c687b8d-3d50-418e-a682-f2c1509f6c97","_uuid":"16192abc90666fe9ac158a31916fbf6a2b7f830d"},"cell_type":"markdown","source":"IP 86767 - the same hour? This takes over from the other two IPs? (This may well have nothing to do with click fraud and instead be a network architecture effect.)\n"},{"metadata":{"_cell_guid":"22baf75e-f799-40b9-b2c0-e3d012d55552","collapsed":true,"_uuid":"cf522d1de5fef12ab04dca089125f9836ffd5f7b","trusted":false},"cell_type":"code","source":"gen_show(train_df, 'ip', 86767)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ee6f295f-035c-4e3b-8ddb-7d80bc9d371d","_uuid":"3da44356f647d74bcf6584acaa1e6c997bb31cb6"},"cell_type":"markdown","source":"# Test Set\n\nNow what is in the test set? Note that the [test set has some time periods missing](http://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/51877)...\n\n> **Aaron Yin wrote**:\n\n>  We choose three ranges of time, as we believed they would play an more significant role in Chinese daily life.\nThe ranges of time are: 12pm-14pm, 17pm-19pm, 21pm-23pm\nNote: the data's timestamp is in UTC+0, and Chinese timezone is UTC+8\n\nThe [original test set](http://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/51506) has uniform coverage of the whole test set time period and so would present a much better picture than we have here...\n\n## Apps\n\nTest App 24, same checkerboard pattern as seen in train set. Seems to be generally ramping up in intensity, especially in the interval between band 2 and 3..."},{"metadata":{"_cell_guid":"56bf0319-0855-4367-b4be-af5254e68677","collapsed":true,"_uuid":"73c477d7804be3c319c871cb37f6f7f12d1e0ae9","trusted":false},"cell_type":"code","source":"gen_show(test_df, 'app', 24)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"4e212c83-904a-4514-808f-2670b1457654","_uuid":"6b2fa42b30ad6bbe1cd72807ed40519fc0aaa18c"},"cell_type":"markdown","source":"App 183, if you zoom in, the pattern is very regular..."},{"metadata":{"_cell_guid":"8a7d9a77-f993-418c-b6c6-5fabb7741c4a","collapsed":true,"_uuid":"523e9a4128e68c5a5904dc4a945f556d4e632e69","trusted":false},"cell_type":"code","source":"gen_show(test_df, 'app', 183)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0269b505-6a32-4e4a-b8d0-52b06fd986a3","_uuid":"0596e24f9bd7b6cbac0a06c993ea546d1e14175f"},"cell_type":"markdown","source":"## Channels\n\nTest Channel 125, 145, same 1 minute schedule switching pattern as above."},{"metadata":{"_cell_guid":"5bb4f3f3-b728-4bfb-a572-0f8eb6da79d1","collapsed":true,"_uuid":"3c07a29073124b52126d417eb0a360a53bdbddd5","scrolled":true,"trusted":false},"cell_type":"code","source":"gen_show(test_df, 'channel', 125)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"84599573-ba61-46dd-9ede-cc87b9876b60","_uuid":"4ecb1081d71cb0663b2ef776f4b3f6d6bbac0fe7"},"cell_type":"markdown","source":"Channel 236, some smoother 1-minute peaks (10 per row), again drifting in phase a little..."},{"metadata":{"_cell_guid":"dd1544b8-abc3-41c0-845a-e39de62f774e","collapsed":true,"_uuid":"6cc48540e775a290b2ab15283e24cfa0d0724e0d","trusted":false},"cell_type":"code","source":"gen_show(test_df, 'channel', 236)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e3913311-ada6-4bd7-9231-4fcb45420e80","collapsed":true,"_uuid":"1705b578cbf393ce448e2929e4f8e9cff4b53371"},"cell_type":"markdown","source":"That's it for now, happy feature engineering!"}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}