{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","collapsed":true,"trusted":false},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport itertools\n\n%matplotlib inline\nmpl.style.use('seaborn')\nsns.set(rc={'figure.figsize':(12,5)});\ngfig = plt.figure(figsize=(12,5));","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"trusted":false},"cell_type":"code","source":"df = pd.read_csv('../input/train.csv', parse_dates=['click_time', 'attributed_time'], nrows=1000000)\ncategorical = ['ip', 'app', 'device', 'os', 'channel', 'is_attributed']\nfor c in categorical:\n    df[c] = df[c].astype('category')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"18e3c83a-e007-4dd3-a6eb-a12080fb72a9","_uuid":"a7919acf0883293472c8d6ffe05ed73e3e31336c"},"cell_type":"markdown","source":"## Basic Overview"},{"metadata":{"_cell_guid":"b57a4220-d4ad-45df-8b15-549024eeca0e","_uuid":"a70a1a6ef9f1a9cd37610ada444e7ffa58b9b2ba"},"cell_type":"markdown","source":"We can start with a basic overview of the data."},{"metadata":{"_cell_guid":"f3bb9522-5bc5-4981-935e-342d302ed4fe","_uuid":"428c3a38305e906ed6230a143489b951cf6bef70","collapsed":true,"scrolled":true,"trusted":false},"cell_type":"code","source":"df.sample(10)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"6a6fe19c-a9ba-441e-b8eb-247f7bddbe74","_uuid":"f88c7c5ca5607fc6b2d6fcf38bcc5f20e70b206e","collapsed":true,"scrolled":true,"trusted":false},"cell_type":"code","source":"df.describe()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f1c9a904-0074-4df1-a931-f49cef22778a","_uuid":"9d14a06eb36615dd2f7188bccee46ebf3c937e09"},"cell_type":"markdown","source":"The data contains very few `attributed_time` entries, creating an extreme class imbalance for prediction:"},{"metadata":{"_cell_guid":"cbedb203-011b-4de7-9723-91e6f816c0d5","_uuid":"22325a8a1b3441a4d2d4c0d52c2d6ccf3ec513d3","collapsed":true,"scrolled":true,"trusted":false},"cell_type":"code","source":"df['is_attributed'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"d2e43a55-0ca8-44ea-9bde-4cbbc2d85c67","_uuid":"3b1e2accf9276098c4965f4a61636a8045c46b3e"},"cell_type":"markdown","source":"We can calculate potential class weights for the classes as follows:"},{"metadata":{"_cell_guid":"dfd8b485-8304-485e-8534-3153c449b30a","_uuid":"c7d27416181214ac17684e5b6a61e1285af59958","collapsed":true,"trusted":false},"cell_type":"code","source":"from sklearn.utils import class_weight\nclass_weights = class_weight.compute_class_weight('balanced', df['is_attributed'].unique(), df['is_attributed'].values)\nprint(\"is_attributed==0 weight: {},\\nis_attributed==1 weight: {}\".format(class_weights[0], class_weights[1]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"867e6af3eb2649320f6717a1500dde5adc82c8d4"},"cell_type":"markdown","source":"As seen in the class weighting, the class imbalance is truly massive. The class weights might be useful when looking into a deep learning approach."},{"metadata":{"_cell_guid":"50dfcd4d-ae56-4d69-a078-962182a6ad84","_uuid":"b96f2ac3efee50c6f53f650946459d8acbe2fd1a"},"cell_type":"markdown","source":"### Top 20 by IP, Device, OS, App, Channel: attributed vs non-attributed"},{"metadata":{"_cell_guid":"ca110cf2-bf93-43ed-bf89-e0771bd89f5d","_uuid":"1292f2bda9dc48844021fe3ed1598f8552064f79"},"cell_type":"markdown","source":"Next we explore the top features comparing attributed vs non-attributed entries."},{"metadata":{"_cell_guid":"9c21db1e-9231-408b-84aa-2ad0fed58327","_uuid":"b5e7842bd874a7dddfa37d239984cf56d607e046","collapsed":true,"scrolled":false,"trusted":false},"cell_type":"code","source":"fig, axes = plt.subplots(2, 5)\nfig.set_figheight(8)\nfig.set_figwidth(20)\nfig.tight_layout()\nattributed = [0, 1]\nattributes = ['ip', 'device', 'os', 'app', 'channel']\nfor attributed in attributed:\n    for idx, attr in enumerate(attributes):\n        values = df[df.is_attributed == attributed][attr].value_counts().head(20)\n        ax = values.plot.bar(ax=axes[attributed][idx])\n        ax.set_title(attr)\n        if idx == 0:\n            if attributed == 0:\n                h = ax.set_ylabel('not-attributed', rotation='vertical', size='large')\n            else:\n                h= ax.set_ylabel('attributed', rotation='vertical', size='large')\nplt.subplots_adjust(hspace=0.3)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"aeee058a-c337-4f18-ab0d-2b8d109b712d","_uuid":"67d4552306e925fc3af6028cdb69da2c178e8b6c"},"cell_type":"markdown","source":"There are a number of notable things:\n* there are a small number of IPs that have many more attributtions than other IPs.\n* attributed clicks are spread across more devices\n* similarly, attributed clicks are more evenly spread across OSs\n* there a a small number of apps that have many more attributions than other apps.\n* specific channels seem to have more attributions than others"},{"metadata":{"_cell_guid":"72d5d380-f7bd-47f8-9dd3-006de9dc2dfb","_uuid":"a626d814225adf0d062c073806a84796a5775efd","collapsed":true},"cell_type":"markdown","source":"## Frequency of clicked time vs attributed time"},{"metadata":{"_cell_guid":"5f261be9-f946-48cf-8e8f-6824b88b93d5","_uuid":"76634d5e385fe314214cbea42dd009b7950baf5b"},"cell_type":"markdown","source":"An interesting pattern to consider is to look at both `click_time` and `attributed_time` time of day (by hour) and see whether there are any noticable patterns."},{"metadata":{"_cell_guid":"6cf8b372-a1dc-4de0-a665-771181132095","_uuid":"1fe802b8cbf629fa358a96543bb7cf7f2e3137a5","collapsed":true,"trusted":false},"cell_type":"code","source":"df['click_h'] = df['click_time'].dt.hour + df['click_time'].dt.minute / 60\ndf['attributed_h'] = df['attributed_time'].dt.hour + df['attributed_time'].dt.minute / 60","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"67c954d4-db3b-4561-9997-8550d665d999","_uuid":"b0a0d7540f090a09f01a0f516ce13505c1881598","collapsed":true,"scrolled":false,"trusted":false},"cell_type":"code","source":"fig, axes = plt.subplots(1, 2)\nfig.set_figwidth(20)\nax = df['click_h'].plot.hist(bins=24, ax=axes[0])\nxl = ax.set_xlabel('hour')\ntitle = ax.set_title('click_time')\n\nax = df['attributed_h'].plot.hist(bins=24, ax=axes[1])\nxl = ax.set_xlabel('hour')\ntitle = ax.set_title('attributed_time')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"4594ba6d-e10b-4242-9930-e9727f958507","_uuid":"dfaa5e74ae0ec118d74d7d891f00921f6e2fc85d"},"cell_type":"markdown","source":"The amount of data dwarves the click_time frequences, but it is clear that many more clicks take place after business hours. Attributed time is spread more evenly, but is also skewed toward the evening."},{"metadata":{"_cell_guid":"205a232a-aeda-4612-8438-618dde2f500c","_uuid":"0647d62d9c72cc7c0beefc74d97a8d8d5c9f63fd"},"cell_type":"markdown","source":"\n### Clicks aggregated by time"},{"metadata":{"_cell_guid":"ead4887c-746f-4f5e-8bae-cfd24a985c7e","_uuid":"237cd68116556216209d06483004d0478e854b78"},"cell_type":"markdown","source":"We can also look at clicks aggregate by time: clicks per month, day and hour and correlate these with IP and other features."},{"metadata":{"_cell_guid":"c5f602fd-aa53-484f-8a50-499f6a18ae2e","_uuid":"86e59e5f69e7cae28224cffa6856fd5dd1ad4e12"},"cell_type":"markdown","source":"#### Time based aggegation"},{"metadata":{"_cell_guid":"89f235cd-8f19-41d2-adbc-e3aabaf2a9e9","_uuid":"be5917c87f3a138c0b710df31d2bc6fd16eadda0","collapsed":true,"scrolled":true,"trusted":false},"cell_type":"code","source":"df['click_month'] = df['click_time'].dt.month\ndf['click_day'] = df['click_time'].dt.day\ndf['click_hour'] = df['click_time'].dt.hour","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"1235fb17-6d0e-4738-b4a6-0d6d8e0cba36","_uuid":"235cf34a3b30a931b40283da7182b600c6654d73","collapsed":true,"trusted":false},"cell_type":"code","source":"# convert back to object to avoid an issue with merging with categorical data: https://github.com/pandas-dev/pandas/issues/18646\nfor c in categorical:\n    df[c] = df[c].astype('object')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"26daf807-e02e-4d3b-90cc-7201c819b5af","_uuid":"2014d26817239f7d31c1d3f3a7c89ec11e699d69","collapsed":true,"trusted":false},"cell_type":"code","source":"def create_click_aggregate(frame, name, idxs):\n    aggregate = frame.groupby(by=idxs, as_index=False).click_time.count()\n    aggregate = aggregate.rename(columns={'click_time': name})\n    return frame.merge(aggregate, on=idxs)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"862c5b58-2efd-4e63-910f-4898bc4cf8a0","_uuid":"0e006ad90914a044271f6f79698b4418e974940d","collapsed":true,"trusted":false},"cell_type":"code","source":"def create_attributed_aggregate(frame, name, idxs):\n    aggregate = frame[frame['is_attributed'] == 1].groupby(by=idxs, as_index=False).is_attributed.count()\n    aggregate = aggregate.rename(columns={'is_attributed': name})\n    return frame.merge(aggregate, on=idxs)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"fb293d04-63f9-4660-aa47-a5500137e005","_uuid":"a299b530b84ab006c8e929d1114d80df1039e368","collapsed":true,"trusted":false},"cell_type":"code","source":"df = create_click_aggregate(df, 'total_clicks', ['ip'])\ndf = create_click_aggregate(df, 'clicks_in_day', ['ip', 'click_month', 'click_day'])\ndf = create_click_aggregate(df, 'clicks_in_hour', ['ip', 'click_month', 'click_day', 'click_hour'])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"4b7ccc5c-af04-4ddc-b90c-1d8047316d36","_uuid":"693a34abc1b4cff6005f2cf96884e65e1c1f7e1f","collapsed":true,"scrolled":true,"trusted":false},"cell_type":"code","source":"df = create_attributed_aggregate(df, 'total_attributions', ['ip'])\ndf = create_attributed_aggregate(df, 'attributed_in_day', ['ip', 'click_month', 'click_day'])\ndf = create_attributed_aggregate(df, 'attributed_in_hour', ['ip', 'click_month', 'click_day', 'click_hour'])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0477f6a8-5d3a-4418-a5ca-11735c0daa9b","_uuid":"627fc59188c761975158ddf726caa4651b7a23fd","collapsed":true,"scrolled":false,"trusted":false},"cell_type":"code","source":"fig, axes = plt.subplots(3, 1)\nfig.set_figheight(8)\nfig.set_figwidth(20)\nfig.tight_layout()\n\ntime_aggregates = [('total_clicks', 'total_attributions'), ('clicks_in_day', 'attributed_in_day'), ('clicks_in_hour', 'attributed_in_hour')]\nrow = 0\nfor time_aggregate in time_aggregates:\n    ax = df[['ip', time_aggregate[0], time_aggregate[1]]].drop_duplicates().sort_values(time_aggregate[0], ascending=False).head(20).set_index('ip').plot.bar(ax=axes[row], secondary_y=time_aggregate[1])\n    if row == 0:\n        ax.set_title('Non-attributed')\n    row+=1","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"7f31ada1-bb47-4d3b-b46a-013346945c9c","_uuid":"d64c7b0431fffd00db770ef885e8aa4163c87721"},"cell_type":"markdown","source":"Notable patterns we can see here:\n* total_attributions increase with total_clicks, but is somewhat noisy.\n* attributions per day increases with clicks per day, but is also noisy.\n* there appears to be a slight, negative correlation between clicks in an hour and the attributions in the hour"},{"metadata":{"_cell_guid":"fa238eec-0a55-4bc0-8ef6-2fe197d9f862","_uuid":"441ab90f39e038c1bb632a7665b01c52b45147f3"},"cell_type":"markdown","source":"## Unique feature value correlation by IP"},{"metadata":{"_cell_guid":"e8c96185-e981-432d-9740-e96a6f0285c9","_uuid":"046fb4ada1dd48db3921ecf5b23092d27336f53f"},"cell_type":"markdown","source":"Finally look at potential feature correlations, calculating unique values by IP."},{"metadata":{"_cell_guid":"6460120b-3063-4f2b-8e62-9e5aab83958b","_uuid":"c7cad6c692c53d724ca6b90c4271732938114c56","collapsed":true,"trusted":false},"cell_type":"code","source":"def unique_values_by_ip(frame, value):\n    n_values_by_ip = frame.groupby(by='ip')[value].nunique()\n    frame.set_index('ip', inplace=True)\n    frame['n_' + value] = n_values_by_ip\n    frame.reset_index(inplace=True)\n    return frame","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"daf3a747-2583-448f-bc8c-9200f4e443dd","_uuid":"438d43dbd362eb63f5bb0c73db184362eb06c714","collapsed":true,"trusted":false},"cell_type":"code","source":"df = unique_values_by_ip(df, 'os')\ndf = unique_values_by_ip(df, 'app')\ndf = unique_values_by_ip(df, 'device')\ndf = unique_values_by_ip(df, 'channel')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0ea5f074-4e4d-4e8d-b26a-947bd4a4fa4a","_uuid":"8795471421ee91e7e1549f51429f2edfe3aacb17","collapsed":true,"trusted":false},"cell_type":"code","source":"facets = ['n_os', 'n_app', 'n_channel', 'n_device', 'total_clicks', 'total_attributions']\n\ncombinations = [c for c in itertools.combinations(facets, 2)]\nrows = 5\ncols = int(len(combinations) / rows)\n\nfig, axes = plt.subplots(rows, cols)\nfig.set_figheight(20)\nfig.set_figwidth(20)\nfig.tight_layout()\n\nidx = 0\nfor row in range(0, rows):\n    for col in range(0, cols):\n        combo = combinations[idx]\n        ax = df.plot.hexbin(combo[0], combo[1], ax=axes[row, col], gridsize=22)\n        idx+=1\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"3b43c0b7-de46-4d19-bae3-eace8252e4c8","_uuid":"b9163f2abd970650726e9ae668fb301538221161","collapsed":true},"cell_type":"markdown","source":"There are a few interesting obsevations here:\n* exponential increase in number of channels relative to total clicks by IP\n* a similar increase in number of channels relative to total attributions by IP\n* a number of expected linear correlations. E.g. n_os vs n_app, n_app vs n_device"},{"metadata":{"_cell_guid":"684262c8-9a7b-482b-8f05-bb963ed4a4fc","_uuid":"5f14e25cc09e0038b3da92fb4890f9b388934721","collapsed":true},"cell_type":"markdown","source":"## [Update] References"},{"metadata":{"_uuid":"96aaacac48001fb130d90a26159212dede6a000b"},"cell_type":"markdown","source":"Other great EDAs that explore similar aspects of the data:\n* https://www.kaggle.com/kailex/talkingdata-eda-and-class-imbalance\n* https://www.kaggle.com/yuliagm/talkingdata-eda-plus-time-patterns"},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"b0f7de09266d2f314dff5453d43afe073c52d1bb"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}