{"cells":[{"metadata":{"_cell_guid":"5a6a94f6-012d-412b-9be4-be67382954a6","_uuid":"8a347be499abb85740fb2cd6485af630fddf54e9"},"cell_type":"markdown","source":"I  found an interesting relationship between IP count and download."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline\nnrows=None\ndtypes = {\n        'ip'            : 'uint32',\n        'is_attributed' : 'uint8',\n        }\ntrain_df = pd.read_csv('../input/train.csv',dtype=dtypes,nrows=nrows,usecols=['ip','is_attributed'])\nip_grp = train_df[['ip','is_attributed']].groupby(['ip']).agg(['count', 'sum']).is_attributed.sort_values(by='count').reset_index()\nip_grp.columns = ['ip', 'occurrences', 'download_count']\ndf = ip_grp[['download_count','occurrences']].groupby('occurrences').agg(['count', 'sum'])['download_count']\ndf.columns = ['num_IPs', 'num_downloads']\ndf['cvr_x_occurrences'] = df.num_downloads /df.num_IPs\ndf = df.reset_index()\ndf.head(10)","execution_count":1,"outputs":[]},{"metadata":{"_cell_guid":"cec1606b-546e-420e-9736-38e100135488","_uuid":"a84e7fabf7155cb1c7008c0fd6bce7a24ae70d4f"},"cell_type":"markdown","source":"Regardless of the number of occurrences, download count from the IP is one.\n\nThis trend applies to IPs with an occurrence count of 400 or less."},{"metadata":{"scrolled":false,"_cell_guid":"e61fdea7-5b40-4f62-8572-6bdd3729e82c","_uuid":"62d76018454062086298dfbfdc5ba5b9d3f28e02","trusted":true},"cell_type":"code","source":"thre = 400\n_df = df[df.occurrences<thre].copy()\nplt.plot(_df.occurrences, _df.cvr_x_occurrences)","execution_count":2,"outputs":[]},{"metadata":{"_cell_guid":"32c747ab-eb01-48d4-9358-3e10521d30b5","_uuid":"06f0ecf99c36b2ee88cfbac2c30fe07299d247c7"},"cell_type":"markdown","source":"For example, even if he clicks 1 times or 3 times, he usually downloads once and only once."},{"metadata":{"_cell_guid":"26127cda-6753-40a9-a049-243e72ae8ba7","_uuid":"91a2a10dabfffda5200dd6d4fc9f24d3b50ad7c1","trusted":true},"cell_type":"code","source":"train_df[train_df.ip.isin(ip_grp[ip_grp.occurrences==1].head(10).ip)].sort_values(['ip','is_attributed'])","execution_count":3,"outputs":[]},{"metadata":{"_cell_guid":"5cabaec8-6cfd-4058-b4ed-19309e649810","_uuid":"e6a7d88d0d6cdf56367829e3b61a78533ce68b05","trusted":true},"cell_type":"code","source":"train_df[train_df.ip.isin(ip_grp[ip_grp.occurrences==3].head(10).ip)].sort_values(['ip','is_attributed'])","execution_count":4,"outputs":[]},{"metadata":{"_cell_guid":"b86889b4-99d7-45a8-b257-8a0a0b6f4efd","_uuid":"b6c688ff7c10dc03bb2e0fb60a8a9e62ee0b7aa3"},"cell_type":"markdown","source":"This tendency disappears when the number of appearances becomes 400 or more."},{"metadata":{"scrolled":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"thre = 3000\n_df = df[df.occurrences<thre].copy()\n_df['roll'] = _df.cvr_x_occurrences.rolling(window=int(10)).mean()\nplt.plot(_df.occurrences, _df.roll)","execution_count":5,"outputs":[]},{"metadata":{"_cell_guid":"56a0db8c-a6a4-4a25-898b-489075bcf93e","_uuid":"d2a9a68281b9d58a07325d0e55e5fd114391fc3a"},"cell_type":"markdown","source":"There is not much proportion of IP with occurrence count of 400 or less, \nbut the number of downloads is large.\n\nAs you can see below,  the percentage of clicks from the IP addresses with 50 clicks or less \nis only 1%, but the percentage of download accounts for 40%. This is significant for AUC."},{"metadata":{"_cell_guid":"e2f29581-ec5a-424e-8f15-475b9785272f","_uuid":"32b72bcb9284ef8c394414236bba8c59717a194f","trusted":true},"cell_type":"code","source":"sum_download = sum(df.num_downloads)\ndf['cum_download'] = df.num_downloads.cumsum()\ndf['cum_download_ratio'] = df['cum_download']/sum_download\ndf['pv'] = df.num_IPs*df.occurrences\nsum_pv = sum(df.pv)\ndf['cum_pv'] = df.pv.cumsum()\ndf['cum_pv_ratio'] = df.cum_pv / sum_pv\ndf['cvr'] = df.num_downloads / df.pv\ndf[['occurrences','cum_download_ratio','cum_pv_ratio']].head(50)","execution_count":6,"outputs":[]},{"metadata":{"_cell_guid":"bbd0bad5-9b26-432f-bb35-fc5bb032cace","_uuid":"857631ef7d09de433c8d854e3062654d0886c21b"},"cell_type":"markdown","source":"Probably this is the effect of fraud access.\n\nOr I may possibly have some misunderstanding."},{"metadata":{"_cell_guid":"83f3794b-6a8d-4667-9c66-8262f6cef2ed","_uuid":"b7c3f21cbf3d95ffce17c07f172c739a8b873b11","collapsed":true,"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}