{"cells":[{"metadata":{"_uuid":"62102e350fc5ad2c68bf599c6673fdfa760fa569","_cell_guid":"a9ef97fe-b475-4bbe-a674-913ce00def93"},"cell_type":"markdown","source":"This kernel tries to understand the key differences between the training and test data, which potentially have an impact for the final leaderboard compared to the public leaderboard. I want to provide some pointers to data issues and maybe spur a discussion on how to cope with them.\n\nFrom my point of view, two major challenges of this dataset are\n1. Categorical data with unique values >> 100\n2. Strong imbalance between signal (target) and background 1:1000\n\nA potential trap for algorithms in this scenario is that esp. decision trees are prone to select individual downloads rather than generalizing download patterns/behavior.  Even though boosting is of help here, the performance on the full test set will depend on how comparable the feature distributions in train/test sets are."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n# Thanks to yuliagm: https://www.kaggle.com/yuliagm/talkingdata-eda-plus-time-patterns\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"a109643c4141a6c9ce1532b6b4d2b78e0df988e7","_cell_guid":"6936d646-51e0-4a2a-9029-bfafe4a78d98"},"cell_type":"markdown","source":"For memory efficiency the full training set is read in chunks of size of the test set.\n\nOverall this comparison will focus on the categorical features IP, APP, DEVICE, OS and CHANNEL "},{"metadata":{"collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"#Read training dataset via chunks\ntrain_reader = pd.read_csv('../input/train.csv', chunksize=18790470)\ntest = pd.read_csv('../input/test.csv')\n\n#test['click_time'] = pd.to_datetime(test['click_time'])\n#Analysis focuses on 5 variables\nvariables = ['ip', 'app', 'device', 'os', 'channel']\n\n#Creating the sets that will contain the unique values in training_only, testing_only and that are shared by both training and testset\n#In training and for shared values I differentiate between unique values that are \"attributed\" and those that aren't.\ntrain_only_sets00 = [set([]) for _ in variables]\ntrain_only_sets01 = [set([]) for _ in variables]\ntrain_only_sets11 = [set([]) for _ in variables]\n\ntest_only_sets = [set([]) for _ in variables]\n\nshared_sets00 = [set([]) for _ in variables]\nshared_sets01 = [set([]) for _ in variables]\nshared_sets11 = [set([]) for _ in variables]","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"8fbc2f9c0d8960ea72c99dbe83275cf1203573c8","_cell_guid":"10c65c1b-f107-4d5e-9d8d-f8a6452c751c"},"cell_type":"markdown","source":"The analysis will investigate if the unique values of the categorical features are comparably distributed in both test and train set. More specifically I focus on\n1. Unique values that are shared by training and test set\n2. Unique values that occur in the training set only\n3. Unique values that occur in the test set only\nOf course, 1 and 2 can be broken down into two sub sets (1a, 1b and 2a, 2b) for training item where a download has occured (\"is_attributed\" == 1).\n\nCalculation of the sets might take some time (10 - 30s)"},{"metadata":{"_uuid":"2afd48ccd85c4f47db9ace2600ec7fb4c80f4b2e","_cell_guid":"4fd5e30c-ba17-4c95-9ff7-9502d5dae1f2","_kg_hide-input":false,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"#Creating the sets for all variables chunk by chunk\nloaded_only = [set([]) for _ in variables]\nnotloaded_only = [set([]) for _ in variables]\nboth_states = [set([]) for _ in variables]\n\nfor cnum, chunk in enumerate(train_reader):\n    print(\"Reading chunk %i\" % cnum)\n    app_loaded = chunk[\"is_attributed\"] == 1\n    app_notloaded = chunk[\"is_attributed\"] == 0\n    for vnum, v in enumerate(variables):\n        train_notloaded = set(chunk[v][app_notloaded])\n        train_loaded = set(chunk[v][app_loaded])\n        \n        loaded_diff = loaded_only[vnum] | (train_loaded - train_notloaded)\n        notloaded_diff = notloaded_only[vnum]| (train_notloaded - train_loaded)\n        \n        chunk_carry = loaded_diff & notloaded_diff\n        \n        both_states[vnum] = both_states[vnum]| (train_notloaded & train_loaded) | chunk_carry\n        loaded_only[vnum] = loaded_diff - both_states[vnum]\n        notloaded_only[vnum] = notloaded_diff - both_states[vnum]\nprint(\"Finalizing sets\")\n\nfor vnum, v in enumerate(variables):\n    test_set = set(test[v])\n    \n    #Unique values of variable that are shared between training and test set\n    shared_sets00[vnum] = test_set & notloaded_only[vnum]\n    shared_sets01[vnum] = test_set & both_states[vnum]\n    shared_sets11[vnum] = test_set & loaded_only[vnum]\n\n    #Unique values of variable that are only in the test set\n    test_only_sets[vnum] = test_set - shared_sets00[vnum] - shared_sets01[vnum] - shared_sets11[vnum]\n    \n    #Unique values of variable that are only in the training set\n    train_only_sets00[vnum] = notloaded_only[vnum] - test_set\n    train_only_sets01[vnum] = both_states[vnum] - test_set\n    train_only_sets11[vnum] = loaded_only[vnum] - test_set\n\nprint(\"Done creating sets\")","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"1b448c182f6ab24b4f8946ec9543d02f916726ef","_cell_guid":"5c50e892-1553-40a9-b573-6d502adf429b"},"cell_type":"markdown","source":"After the sets of unique values have been calculated. The dataframe for analysis is prepared."},{"metadata":{"_uuid":"78059431e36090e139dbfa83082cd374eb07da15","_cell_guid":"75d6c52b-f973-4544-adc9-6baa7fb77d12","trusted":true},"cell_type":"code","source":"#DataFrame for analysis will consist of 5 columns, each for the subsets 1a, 1b, 2a, 2b, 3\n#The rows of the dataframe are the number of unique items in the subset - one row per variable (ip, app...)\n\nana_data = {\"shared_onlyloaded\": list(map(len, shared_sets11)),\n            \"shared_both\": list(map(len, shared_sets01)),\n            \"shared_noneloaded\": list(map(len, shared_sets00)),\n            \"train_only_onlyloaded\": list(map(len, train_only_sets11)),\n            \"train_only_both\": list(map(len, train_only_sets01)),\n            \"train_only_noneloaded\": list(map(len, train_only_sets00)),\n            \"test_only\": list(map(len, test_only_sets))\n           }\n\nana_frame = pd.DataFrame(data=ana_data)\n#DataFrame is normalized towards total number of unique items\ntotal = ana_frame.sum(axis=1)\n\nfor col in list(ana_frame):\n    ana_frame[col] = ana_frame[col].divide(total)","execution_count":4,"outputs":[]},{"metadata":{"_uuid":"b7cd4e2cf37ad36de46f0e28c2a950cf3e59e8e3","_cell_guid":"3470b215-fa47-405a-ac79-004194d936db"},"cell_type":"markdown","source":"To analyze these sets I want to understand the size of these sets and how their distribution overlaps to get an indication of the comparability of training and test set."},{"metadata":{"collapsed":true,"_uuid":"f98cab3de104494cd55ba797dc8acbf3a1721192","_cell_guid":"49cf0685-a972-4002-878a-dca965d708ac","trusted":true},"cell_type":"code","source":"from matplotlib import pyplot\n#General function for analysis of the categorical variables\n\ndef analyze_column(col_num, col_name, bins, ignore_overview=False):\n    #Is there an easier way to drop all but a particular row?\n    inds = np.arange(5)\n    inds_mask = np.ones(5, dtype=bool)\n    inds_mask[col_num] = False\n    ip_frame = ana_frame.drop(ana_frame.index[inds[inds_mask]])\n    \n    if not ignore_overview:\n        #Plotting the initial overview\n        ax = ip_frame.plot.bar()\n        ax.set_xticklabels(variables)\n        ax.set_title(col_name + \"-Overview on items that are shared or differ between training and test set\")\n\n        for pnum, p in enumerate(ax.patches):\n            ax.annotate(str(round(p.get_height()*total[col_num])), (p.get_x() * 0.99, p.get_height() * (1.01)))\n\n    #plotting distribution\n    pyplot.figure()\n    pyplot.hist([list(shared_sets01[col_num]),list(shared_sets00[col_num]),list(shared_sets11[col_num]),\n                list(test_only_sets[col_num]),list(train_only_sets01[col_num]), list(train_only_sets00[col_num]),\n                list(train_only_sets11[col_num])], bins, alpha=1.0, \n                label=['shared_both', 'shared_noneloaded', 'shared_onlyloaded', 'test_only',\n                       'train_only_both', 'train_only_noneloaded', 'train_only_onlyloaded'],\n                stacked=True\n               )\n    pyplot.legend()\n    pyplot.title(col_name + \"-Analysis: Distribution of shared and differing unique values in training and test set\")\n    pyplot.show()","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"1594f1f6afa75c4bc0d3a16db59fb4de2416fa18","_cell_guid":"40e8d5df-b1f4-4bab-9e4d-06044b0b711b","trusted":true},"cell_type":"code","source":"analyze_column(0, \"IP\", 50)","execution_count":6,"outputs":[]},{"metadata":{"_uuid":"8277fff52caba9731d00082a7949a35af1ee63b5","_cell_guid":"9483f797-d34c-4eaf-a24b-3e3e83aef12a"},"cell_type":"markdown","source":"**Overview:**One can clearly see that the IP range in the test and the train set are differing significantly. The majority of unique values (50% - train_only_both) is only available in the training_set and w/o a clear signal (i.e. has attributed==0 and attributed==1). Of the 10% with a strong signal (i.e. exclusively attributed=1; train_only_onlyloaded+shared_onlyloaded) only 0.3% exist in both the training and testing set.  Overall the test set introduces more new ip addresses (~18% - dark red) than it shares with the training set (~12% - blue, orange)\n\n**Distribution**: The right hand side around IP>125000 is dominated by IP addresses that only occur in the training set. Let's take a closer look."},{"metadata":{"_uuid":"7af7f819a5cd1b9ac5ca60669bb771471e523add","_cell_guid":"98075438-fc04-40f1-a841-f2c8a41cb728","trusted":true},"cell_type":"code","source":"analyze_column(0, \"IP\", np.arange(125000, 128000, 50), True)","execution_count":7,"outputs":[]},{"metadata":{"_uuid":"d0268e4694ff7b38792d5d65162a1f0417c40c0b","_cell_guid":"ff7d4230-540c-44d3-ba06-0f2ec42176c3"},"cell_type":"markdown","source":"With a closer look one can see:\n\n1. The cut-off seems as an intentional mix designed by the competitions hosts:IP < 126500 is a regular mix of IPs that are shared with the training set and are attributed); IP >=126500 consist of only training IP addresses. Still there is a large number of exclusive training IP addreses that are attribute, which might derail algorithms. Furthermore there is a large number of IP addresse that are only occuring in the test set. Hence, the decision tree branches are hopefully well generalized to cope with these new items.\n\n2. The unique values are not evenly distributed and accumulate around certain IP addresses. Interestingly there are areas where test_only dominates (dark red) and others where the shared values dominate (blue, orange). Another hint to the test set being designed to ensure that the models are not a \"glorified IP blacklist\".\n\n3. The unique values for train_only (IP >=126500) have no holes, i.e., all IP addresses are covered - compared to the testing piece which seems incomplete. That that for the relevant IP area for testing our model we lack the full range of IP address behavior.\n\nAs pointed out by the competition hosts, IP addresses are encoded. Identifying this encoding could help reduce cardinality of IP addresses and improve generalization."},{"metadata":{"_uuid":"54a7628340e610a8602dab5241c5b254fb16269a","_cell_guid":"80e2c8b9-a531-4173-93c9-b18a577ed0e0","trusted":true},"cell_type":"code","source":"analyze_column(0, \"IP\", np.arange(123128, 123257, 2), True)","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"d59f900690b514f92e97c7b22dc9edc2227679dc","_cell_guid":"0d3d352a-80ca-460f-b756-43dd3e688a00"},"cell_type":"markdown","source":"Looking at the most granular level one can see that areas with no training information (dark red), weak information (blue) or information on non-download (orange) IP addresess are alternating and are quite narrow. Defining proper IP bands via decision trees might be quite challenging."},{"metadata":{"_uuid":"8ebbaacc2fa649807e215098b6432957287469aa","_cell_guid":"31a7e0af-687d-45c8-9d6b-db4af5fb94d0","trusted":true},"cell_type":"code","source":"analyze_column(1, \"APP\", 20)","execution_count":9,"outputs":[]},{"metadata":{"_uuid":"66cdd917b1d92e302425213b0cc97598651cf0e9"},"cell_type":"markdown","source":"The graphs for APP show a very different picture compared to IP.  Roughly 40% of apps are in the training set only (dark brown) and have no downloads.  Moreover, only a fraction of <5% of apps is newly introduced via the test set. Also, distracting downloads (i.e. that are only attributable to this app and occur only in the training set) are limited to <1%. \n\nIn summary, it seems like the app number is also an indicator for the popularity of the app. Lower numbers correlate with higher blue color, i.e. for the majority of apps at the lower end, both data on downloading/not-downloading the app is available and the app is available in both train and test set). \n\nApps number > 550 could be good for training time-series models on how users typically click apps they don't end up downloading.\nApps number < 400 could be used in an over-sampled fashion to train specifically for app download patterns."},{"metadata":{"_uuid":"d4a0b5a8af2caf769600940e00f52496f63bace9","_cell_guid":"ca5d804a-a8b1-40d3-8e22-3956f526fa8c","trusted":true},"cell_type":"code","source":"analyze_column(2, \"Device\", 50)","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"d9292f27644c2e036cd34123ec7864fc8c74ac20","_cell_guid":"f9c83b09-71b5-49fa-92de-efc8335af041","trusted":true},"cell_type":"code","source":"analyze_column(3, \"OS\", 50)","execution_count":14,"outputs":[]},{"metadata":{"_uuid":"31f81d2b51b928ea0ef42e7b64709e914ad28a7c","_cell_guid":"579bc9d5-987b-40aa-8369-69cd46822e1d","trusted":true},"cell_type":"code","source":"analyze_column(4, \"channel\", 30)","execution_count":15,"outputs":[]},{"metadata":{"_uuid":"06d74ec35442cbc2f585b8b688d318ec01882090"},"cell_type":"markdown","source":"Overall channel data shows no issues with channel partners only available in test set, but 10% of the channel partners in the training set are not used in the training set. Looking at the distribution here, one could infer that channel partners are organized in tiers (e.g., < 100, top-tier highly-trustworthy or with high volumes). The number of training_only channels at the higher end of the spectrum supports this perspective"},{"metadata":{"_uuid":"5c8c61bc3d97fae97c29233bc84d3f25857d1aaf"},"cell_type":"markdown","source":"This was an initial introduction on the unique values and challenges in test and training data. Hopefully that was help and insightful. I will add some more analysis on click-behavior and also on the actual distributions beyond unique values.\n\nKindly let me know, where you agree, where you disagree or what your thoughts on the data are."},{"metadata":{"_uuid":"3330189e4b7e980cbbd3e631b978b15cbf70e1a8"},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}