{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"b81d4eec-8375-2a6a-9a99-ba60480354dd"},"source":"### *Use this one simple trick to get to the top of the leaderboard - Grandmasters hate him!*\n\nHere's quite a high-level EDA - since the data is so huge, we want to get a better understanding of what we actually have.\n\nIf this EDA helps you, make sure to leave an upvote to motivate me to make more! :)\n\nFirst off: What files do we have?"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"afc556c1-42ab-466b-e56a-d12ff2546681"},"outputs":[],"source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport gc # We're gonna be clearing memory a lot\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\np = sns.color_palette()\n\nprint('# File sizes')\nfor f in os.listdir('../input'):\n    if 'zip' not in f:\n        print(f.ljust(30) + str(round(os.path.getsize('../input/' + f) / 1000000, 2)) + 'MB')"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8a7d4876-7733-720c-628e-a38ac3398997"},"outputs":[],"source":"ls"},{"cell_type":"markdown","metadata":{"_cell_guid":"817adf24-04e9-66c5-d125-47cce86462aa"},"source":"Wow, that's a lot of data! Let's start off by looking at the clicks_train.csv and clicks_test.csv files, as these contain the main things we need.\n\nEach display has a certain number of adverts. Let's look at the distribution of these advert counts, and see if they are consistent between train and test."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"35ec6f17-362f-6da9-877b-17ac05db3d67"},"outputs":[],"source":"df_train = pd.read_csv('../input/clicks_train.csv')\ndf_test = pd.read_csv('../input/clicks_test.csv')"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"dfde6d85-3bf5-1220-a7b3-16bce8c2de22"},"outputs":[],"source":"sizes_train = df_train.groupby('display_id')['ad_id'].count().value_counts()\nsizes_test = df_test.groupby('display_id')['ad_id'].count().value_counts()\nsizes_train = sizes_train / np.sum(sizes_train)\nsizes_test = sizes_test / np.sum(sizes_test)\n\nplt.figure(figsize=(12,4))\nsns.barplot(sizes_train.index, sizes_train.values, alpha=0.8, color=p[0], label='train')\nsns.barplot(sizes_test.index, sizes_test.values, alpha=0.6, color=p[1], label='test')\nplt.legend()\nplt.xlabel('Number of Ads in display', fontsize=12)\nplt.ylabel('Proportion of set', fontsize=12)"},{"cell_type":"markdown","metadata":{"_cell_guid":"e72ae2a7-7df4-681f-623c-a602640aa862"},"source":"This looks like a perfect split to me! So we can assume the distribution to be the same within the sets - no weird trickery going on here.\n\nWhat about adverts? How many adverts are there that are very often used, and how many are rare?"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"084fed1a-30d1-8aba-5c60-0413ecf01654"},"outputs":[],"source":"ad_usage_train = df_train.groupby('ad_id')['ad_id'].count()\n\nfor i in [2, 10, 50, 100, 1000]:\n    print('Ads that appear less than {} times: {}%'.format(i, round((ad_usage_train < i).mean() * 100, 2)))\n\nplt.figure(figsize=(12, 6))\nplt.hist(ad_usage_train.values, bins=50, log=True)\nplt.xlabel('Number of times ad appeared', fontsize=12)\nplt.ylabel('log(Count of displays with ad)', fontsize=12)\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"2ac4310e-082f-0622-aace-76e21078c911"},"source":"Here we can see that a **huge** number of ads appear just a few times in the training set (so much that we have to use a log graph to show it), with two-thirds having less than 10 appearances. This shows us that we have to be able to predict whether someone will click on an ad not just based on the past data of that specific ad, but also by linking it to other adverts.\n\nI think this is the underlying challenge that Outbrain has for us.\n\nBefore we move on, let's check how many ads in the test set are not in the training set."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"17cd4e14-8553-f482-627f-38d23a928cc8"},"outputs":[],"source":"ad_prop = len(set(df_test.ad_id.unique()).intersection(df_train.ad_id.unique())) / len(df_test.ad_id.unique())\nprint('Proportion of test ads in test that are in training: {}%'.format(round(ad_prop * 100, 2)))"},{"cell_type":"markdown","metadata":{"_cell_guid":"170da032-c536-3860-0c6d-7c6017427801"},"source":"This number is a little more reasonable, with 88% of ads appearing in both sets.\n\n## Events\n\nLet's move on to the events file. I'm not going to cover the [timestamp](https://www.kaggle.com/joconnor/outbrain-click-prediction/date-exploration-and-train-test-split) or the [location](https://www.kaggle.com/andreyg/outbrain-click-prediction/explore-user-base-by-geo), since these have already been beautifully explored - you can click the links to view those EDAs."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"d04c8b83-1cbd-7056-f87c-bfdc58f26448"},"outputs":[],"source":"try:del df_train,df_test # Being nice to Azure\nexcept:pass;gc.collect()\n\nevents = pd.read_csv('../input/events.csv')\nprint('Shape:', events.shape)\nprint('Columns', events.columns.tolist())\nevents.head()"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"dad492be-4d95-122f-b418-d99d07ae7a7b"},"outputs":[],"source":"plat = events.platform.value_counts()\n\nprint(plat)\nprint('\\nUnique values of platform:', events.platform.unique())"},{"cell_type":"markdown","metadata":{"_cell_guid":"63cd32b2-f4ab-75ae-d786-b61547af06aa"},"source":"This is very interesting, notice how 1, 2 and 3 are repeated twice in the platform, once as floats, and once as strings.\n\nThis might become useful in the future (leak? :P) - but for now, I'm just going to treat them as the same thing."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"abf57bbc-bed4-5652-5606-fca1e6f25fa5"},"outputs":[],"source":"events.platform = events.platform.astype(str)\nplat = events.platform.value_counts()\n\nplt.figure(figsize=(12,4))\nsns.barplot(plat.index, plat.values, alpha=0.8, color=p[2])\nplt.xlabel('Platform', fontsize=12)\nplt.ylabel('Occurence count', fontsize=12)"},{"cell_type":"markdown","metadata":{"_cell_guid":"ad44eef5-f813-8b54-ea7c-49fc4c551467"},"source":"It's still unclear what the platform means, but it's possible that it's things like computers, phones, tablets etc. \n\nThe `\\\\N`s and string numbers may have come from the way that a file was parsed while creating the dataset, but it's still a mystery. If anyone has any other ideas, I'd love to hear them!\n\nLet's do some quick analysis on the UUID next."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"8bc2b579-b44d-8f90-9359-f63b1de00fbc"},"outputs":[],"source":"uuid_counts = events.groupby('uuid')['uuid'].count().sort_values()\n\nprint(uuid_counts.tail())\n\nfor i in [2, 5, 10]:\n    print('Users that appear less than {} times: {}%'.format(i, round((uuid_counts < i).mean() * 100, 2)))\n    \nplt.figure(figsize=(12, 4))\nplt.hist(uuid_counts.values, bins=50, log=True)\nplt.xlabel('Number of times user appeared in set', fontsize=12)\nplt.ylabel('log(Count of users)', fontsize=12)\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"30681980-a433-1fa2-2e12-acaa8e18bc3f"},"source":"Here we see a distribution much like the ad ids, with 88% of users being unique - there will be little scope of building user-based recommendation profiles here.\n\nI'd love to look at things like whether the same user ever clicks on the same ad twice, or whether a user transverses the training & testing set, but we're limited by the Kernels memory limit here :(\n\n## Categories\n\nOutbrain has some content classification algorithms, and they have provided us with the output from these classifications. Let's take a look at some of the most popular classifications."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"25d20056-2311-cc15-9a2f-4f82e9189a14"},"outputs":[],"source":"try:del events\nexcept:pass;gc.collect()\n\ntopics = pd.read_csv('../input/documents_topics.csv')\nprint('Columns:',topics.columns.tolist())\nprint('Number of unique topics:', len(topics.topic_id.unique()))\n\ntopics.head()"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"0e598b3d-a93d-8b4c-5867-ffaf4b8d3fe2"},"outputs":[],"source":"topic_ids = topics.groupby('topic_id')['confidence_level'].count().sort_values()\n\nfor i in [10000, 50000, 100000, 200000]:\n    print('Number of topics that appear more than {} times: {}'\n          .format(i, (topic_ids > i).sum()))\n\nplt.figure(figsize=(12, 4))\nsns.barplot(topic_ids.index, topic_ids.values, order=topic_ids.index, alpha=1, color=p[5])\nplt.xlabel('Document Topics', fontsize=12)\nplt.ylabel('Total occurences', fontsize=12)\nplt.show()"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"7a03ce36-4e3b-63c1-eeb2-b5927c8aeb6a"},"outputs":[],"source":"cat = pd.read_csv('../input/documents_categories.csv')\nprint('Columns:', cat.columns.tolist())\nprint('Number of unique categories:', len(cat.category_id.unique()))\n\ncat_ids = cat.groupby('category_id')['confidence_level'].count().sort_values()\n\nfor i in [1000, 10000, 50000, 100000]:\n    print('Number of categories that appear more than {} times: {}'\n          .format(i, (cat_ids > i).sum()))\n\nplt.figure(figsize=(12, 4))\nsns.barplot(cat_ids.index, cat_ids.values, order=cat_ids.index, alpha=1, color=p[3])\nplt.xlabel('Document Categories', fontsize=12)\nplt.ylabel('Total occurences', fontsize=12)\nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"02811160-6bb2-596b-49a5-f27cb1c77dc7"},"source":"That's it for today, folks!\n\nHopefully this helps you think of some ideas - I'll continue updating this as I do more exploration over the next few days.\n\nOnce again, please upvote if this was useful :P"}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":0}