{"cells":[{"metadata":{"_uuid":"77fc049c3126736a55b3886b50a190f579c500da"},"cell_type":"markdown","source":"Based on the discussion here: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/52374\nI did some analysis on IP assignment.\n\n**It looks very much like the organizers used different patterns in assigning dummy IPs in TEST and TRAIN sets.******\n\nIn the Train set, there is a strong correlation to assign IPs with higher frequencies to lower numbers.  The pattern **DOES NOT hold in Test**.\n\nTo copy my comment from discussion:\n\n\" Based on train_sample.csv and my own subsamples (i haven't been able to run a test on full train set), it appears that lower number IPs are strongly associated with higher number of clicks. i.e. numbers in the range 1 through 125000 have substantially more clicks than IPs in the range 300000 and up. It's almost as if when the ip values were generated to mask real ips, the data was pre-sorted by the number of clicks per IP, in a few major chunks. (So they gathered most frequent group, and assigned numbers from 1 to 125000, than next group and another bulk of numbers, than next, etc). I see about 4 bands of frequencies.\n\nHowever, based on a few test subsamples I ran, the pattern does not repeat in test data. The ips over test seem to be mapped truly randomly, and if anything have consistent click density.\n\nAlso, (and somebody please check these calculations!) there appear to be the following distribution of IPs:\n\n**Overall the number of IPs (test OR train): 333168 <br>\nNumber of IPs that are in both (test AND train): 38164 <br>\nNumber of IPs that are in Train and NOT in Test: 239232<br> Number of IPs that are in Test and NOT in Train: 55772**\n\nThat means that way over half of IPs in test do not follow the mapping rules of train data.\n\nHense I think need to be careful in validation. The pattern of IP assignment in Train does not mimic the one in test. If you go by using IP value as a signal in train, your final test results will be off substantially.\"\n\n** see below for visualization based on Train subsample and full Test data**\n"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","collapsed":true,"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport datetime\nimport gc\n%matplotlib inline","execution_count":26,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"trusted":true},"cell_type":"code","source":"input_path = '../input/'","execution_count":27,"outputs":[]},{"metadata":{"_uuid":"f743633446805db34141cf953fb45ee0a33e0fa7"},"cell_type":"markdown","source":"### TRAIN SAMPLE"},{"metadata":{"trusted":true,"_uuid":"8a99812fc16cd958cd85097d2dbac993b00e35e2"},"cell_type":"code","source":"dtypes = {\n        'ip'            : 'uint32',\n        'app'           : 'uint16',\n        'device'        : 'uint16',\n        'os'            : 'uint16',\n        'channel'       : 'uint16',\n        'is_attributed' : 'uint8',\n        }\n\ntrain = pd.read_csv(input_path+'train_sample.csv', dtype=dtypes)\ntrain.head()","execution_count":3,"outputs":[]},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"9d11a9a9f008c9567ae9fdb9acc0a288cc6dcad9"},"cell_type":"code","source":"#convert to date/time\ntrain['click_time'] = pd.to_datetime(train['click_time'])\ntrain['attributed_time'] = pd.to_datetime(train['attributed_time'])\n\n#extract hour as a feature\ntrain['click_hour']=train['click_time'].dt.hour","execution_count":12,"outputs":[]},{"metadata":{"_kg_hide-input":true,"collapsed":true,"trusted":true,"_uuid":"ed08ca7eb6c1e4c040dd31f3497dc48dc045007c"},"cell_type":"code","source":"def plotStrip(x, y, hue, figsize = (14, 9)):\n    \n    fig = plt.figure(figsize = figsize)\n    colours = plt.cm.tab10(np.linspace(0, 1, 9))\n    with sns.axes_style('ticks'):\n        ax = sns.stripplot(x, y, \\\n             hue = hue, jitter = 0.4, marker = '.', \\\n             size = 4, palette = colours)\n        ax.set_xlabel('')\n        for axis in ['top','bottom','left','right']:\n            ax.spines[axis].set_linewidth(2)\n\n        handles, labels = ax.get_legend_handles_labels()\n        plt.legend(handles, ['col1', 'col2'], bbox_to_anchor=(1, 1), \\\n               loc=2, borderaxespad=0, fontsize = 16);\n    return ax","execution_count":28,"outputs":[]},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"d270701ab22ad68fb568ce87b7582ed132f6cf7e"},"cell_type":"code","source":"X = train\nY = X['is_attributed']","execution_count":18,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"a01ac3992b39e0123dc960f83056c0bcd7edd156"},"cell_type":"code","source":"ax = plotStrip(X.click_hour, X.ip, Y)\nax.set_ylabel('ip', size = 16)\nax.set_title('IP (vertical), by HOUR(horizontal), split by converted or not(color)', size = 20);","execution_count":19,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"653b12e4f9e6a31dace0b78fec971e1b806a0b18"},"cell_type":"code","source":"del train\ngc.collect()","execution_count":42,"outputs":[]},{"metadata":{"_uuid":"f019c64ec37b91eabeb11608bf1449e65a6ec9b4"},"cell_type":"markdown","source":"### TEST SUBSAMPLE"},{"metadata":{"trusted":true,"_uuid":"1a2cb2db4e24ecd65c3695460bda95923ef4caed"},"cell_type":"code","source":"total_rows = 18790470\nsample_size = total_rows//120\n\ndef get_skiprows(total_rows, sample_size):\n    inc = total_rows // sample_size\n    return [row for row in range(1, total_rows) if row % inc != 0]\n\ndtypes = {\n        'ip'            : 'uint32',\n        'app'           : 'uint16',\n        'device'        : 'uint16',\n        'os'            : 'uint16',\n        'channel'       : 'uint16',\n        }\n\ntest = pd.read_csv(input_path+'test.csv',\n                 skiprows=get_skiprows(total_rows,sample_size), dtype=dtypes)\ntest.head()","execution_count":44,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"a2970bb9387bce83717d086ad467dac4f3aa7f0d"},"cell_type":"code","source":"#convert to date/time\ntest['click_time'] = pd.to_datetime(test['click_time'])\n\n#extract hour as a feature\ntest['click_hour']=test['click_time'].dt.hour","execution_count":45,"outputs":[]},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"e2e3f8e2a49cba7e776cccc08fe17f942724fdc3"},"cell_type":"code","source":"#dummy variable for hour color bands in test\ntest['band'] = np.where(test['click_hour']<=6, 0, \\\n                        np.where(test['click_hour']<=11, 1, \\\n                                np.where(test['click_hour']<=15, 2, 3)))","execution_count":46,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"47e63a1de527a6b343f6b6841ff39bae8e6df68d"},"cell_type":"code","source":"print(len(test))\nX = test\nY = test['band']","execution_count":47,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"97d541c712d04f5ac11774d7f50fb7137092d659"},"cell_type":"code","source":"ax = plotStrip(X.click_hour, X.ip, Y)\nax.set_ylabel('ip', size = 16)\nax.set_title('IP (vertical), by HOUR(horizontal), split by hour band', size = 20);","execution_count":48,"outputs":[]},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"33557a9d01ac8e6e209f5c9de584deb56c045d8f"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}