{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"},"cell_type":"markdown","source":"Welcome to my Kernel to try understand the TalkingData AdTrack. \n\n*English is not my first language, so sorry for any error\n\n"},{"metadata":{"_uuid":"78e90bd4c34012424a273e39d50e97962f68443f","_cell_guid":"edaf3d1c-e30a-470e-aa4b-faa7ce19610c"},"cell_type":"markdown","source":"I will try understand and reply some questions that I formulate.
\n\nFor this competition, our objective is to predict whether a user will download an app after clicking a mobile app advertisement. \n\n

Introducing to dataset:

\nEach row of the training data contains a click record, with the following features.\n\nThis is our features on train dataset:
\nip: ip address of click.
\napp: app id for marketing.
\ndevice: device type id of user mobile phone (e.g., iphone 6 plus, iphone 7, huawei mate 7, etc.)
\nos: os version id of user mobile phone
\nchannel: channel id of mobile ad publisher
\nclick_time: timestamp of click (UTC)
\nattributed_time: if user download the app for after clicking an ad, this is the time of the app download
\nis_attributed: the target that is to be predicted, indicating the app was downloaded
\nNote that ip, app, device, os, and channel are encoded.
\n\n\n

So, I will try answer this questions:

\n- Are the downloads and clicks balanced? \n- Are the IP's with the same distribuition? \n- Are all devices on our dataset with the same distribuition?\n- What is the most commom app ?\n- What is the most commom channel ?\n- Have any hour or minute that the download rate iw? \n- What's the distribuition of time? \n- We have some patterns that might can explain the downloads?"},{"metadata":{"_uuid":"8dfb5010803f01e2b099e20d36fcf9bc86fcc749","_cell_guid":"20423512-7f23-45a4-88d6-f0a2b4185f17"},"cell_type":"markdown","source":"# Couting the lines of full dataset\n"},{"metadata":{"_kg_hide-input":true,"_uuid":"ac429bd8ce065daf4406fb5e9548a8c243886117","_cell_guid":"eb2f2f80-4f70-4f4f-9158-50b0ef4475c0","trusted":true},"cell_type":"code","source":"import subprocess\nprint('# Line count:')\nfor file in ['train.csv', 'test.csv', 'train_sample.csv']:\n lines = subprocess.run(['wc', '-l', '../input/{}'.format(file)], stdout=subprocess.PIPE).stdout.decode('utf-8')\n print(lines, end='', flush=True)","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"d07f676ec3936855fc4f5f9a2c08487859e66997","_cell_guid":"e73954b2-4964-44b5-a559-b6e496a620e6"},"cell_type":"markdown","source":"That makes 185 million rows in the training set and 19 million in the test set. "},{"metadata":{"_uuid":"c282a67229de3ddd7573543097ede5b905270258","_cell_guid":"eadb3500-eb52-49c6-9518-5f3288e7d324"},"cell_type":"markdown","source":"# **Importing the librarys and datasets **"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nimport numpy as np\nimport pandas as pd","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"5e609254e503a0282fb435ee49835c6b652628d3","_cell_guid":"94c1eea9-1907-4870-88ae-49f221a11553"},"cell_type":"markdown","source":"I will import 10 millions of rows to do this analysis"},{"metadata":{"_uuid":"31a325a4950044a63ece8a43448cb1dce2dc392b","_cell_guid":"c28cf992-741b-481a-ad30-0f8f3b1327ef","collapsed":true,"trusted":true},"cell_type":"code","source":"df_talk = pd.read_csv(\"../input/train.csv\", nrows=2500000,parse_dates=['click_time'])","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"7e6a1d57af7214cd582d0e683c870d9f19370751","_cell_guid":"2ea5159d-78c4-4a47-8d7e-910c7de0d6a2"},"cell_type":"markdown","source":"

Looking data types and if we have any null values

"},{"metadata":{"_uuid":"f3b66f7aaff1be83a595737bc4be893503089f67","_cell_guid":"82071b87-0686-4378-88bf-ae588e8954e5","trusted":true},"cell_type":"code","source":"df_talk.info()","execution_count":4,"outputs":[]},{"metadata":{"_uuid":"46d005996d78cecd29721060fa00319b8ee299c5","_cell_guid":"0b409298-f185-42ed-83e4-c1e253823fb1"},"cell_type":"markdown","source":"The feature attributed_time have a high number of null values"},{"metadata":{"_uuid":"7363f8f08edc40b655142f0e0b67edf57fd9e385","_cell_guid":"ce337278-4a66-4539-9c1d-b651c4bd404f"},"cell_type":"markdown","source":"

Unique values of this sample of 1000000

"},{"metadata":{"_uuid":"62dc7c83432b3ba19851e9586acdf5637cd0648b","_cell_guid":"bf8b066e-4d91-41e4-958b-4a2a476cf6f8","trusted":true},"cell_type":"code","source":"df_talk.nunique()","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"64eb25bccfeeb83c3f6754b588802f8846f09e6c","_cell_guid":"e7372224-e85a-4d0b-98c3-147408d093ac"},"cell_type":"markdown","source":"We can see that we have 39611 different ip's "},{"metadata":{"_uuid":"8be077c5ae67846f642298c4529bb4584d9d4368","_cell_guid":"bd8bb74e-d658-4188-8dfb-5febfa6c1948"},"cell_type":"markdown","source":"

Looking the data

"},{"metadata":{"_uuid":"696d9c110ae2b6a398731f2a65e0cb6b72caae9b","_cell_guid":"d99fd1cc-10cd-4fee-964d-d82ac8e4fbc9","trusted":true},"cell_type":"code","source":"df_talk.head()","execution_count":6,"outputs":[]},{"metadata":{"_uuid":"f2b95cbc080a140e8a018ab5f8b726833ece4f5f","_cell_guid":"f921e94d-f258-46d5-a0d9-0ea8f889ca3a"},"cell_type":"markdown","source":"

Starting Feature Engineering in datetime column

"},{"metadata":{"_uuid":"f3b777893f5ccab6dd4553cf21172fd271582c07","_cell_guid":"a5c5c57a-41c6-4192-b788-7e69f83a0888"},"cell_type":"markdown","source":"I will do some feature engineering in datetime column to we have more feature that might can help explain the download\n"},{"metadata":{"_uuid":"6a49cf3c2fc48fd033176193262f70ccd2619abb","_cell_guid":"3daf8a81-2487-493b-8be2-8077560dddaa","collapsed":true,"trusted":true},"cell_type":"code","source":"def datetime_to_deltas(series, delta=np.timedelta64(1, 's')):\n t0 = series.min()\n return ((series-t0)/delta).astype(np.int32)\n\ndf_talk['sec'] = datetime_to_deltas(df_talk.click_time)","execution_count":7,"outputs":[]},{"metadata":{"_uuid":"e5a0e676bb1d49618e785b527fb704d1f9ffbaaf","_cell_guid":"880263a1-3bdc-454e-a8d2-a946ed7e8a93"},"cell_type":"markdown","source":"

Extracting datetime values

"},{"metadata":{"_uuid":"5f737f1d828a5c37b0527d4584c98f4a8e6fdd30","_cell_guid":"a43566e1-15ce-4a2c-b3f3-6125ee0257b8","trusted":true},"cell_type":"code","source":"df_talk['day'] = df_talk['click_time'].dt.day.astype('uint8')\ndf_talk['hour'] = df_talk['click_time'].dt.hour.astype('uint8')\ndf_talk['minute'] = df_talk['click_time'].dt.minute.astype('uint8')\ndf_talk['second'] = df_talk['click_time'].dt.second.astype('uint8')\ndf_talk['week'] = df_talk['click_time'].dt.dayofweek.astype('uint8')\n\ndf_talk.head()","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"5bfc0f21c32c37417a68da52c7e01be31224405f","_cell_guid":"3d9d2e85-9981-4e87-891f-243615a2d1e2"},"cell_type":"markdown","source":"Did it, let's start ploting some graphs to try understand the distribuitions. '"},{"metadata":{"_uuid":"50b60aebe394852d8bf48a8539aad1bee12972ce","_cell_guid":"212f6775-b20e-4316-a0a9-f52921a6301a"},"cell_type":"markdown","source":"

Starting by % of downloaded over the rest of data

"},{"metadata":{"_kg_hide-input":true,"_uuid":"84587f0e05368dfb10928ee6dfbf592520fec585","_cell_guid":"0dac404a-aaa3-4997-9d82-48fb798f4346","trusted":true},"cell_type":"code","source":"print(\"The proportion of downloaded over just click: \")\nprint(round((df_talk.is_attributed.value_counts() / len(df_talk.is_attributed) * 100),2))\nprint(\" \")\nprint(\"Downloaded over just clicks description: \")\nprint(df_talk.is_attributed.value_counts())\n\nplt.figure(figsize=(8, 5))\nsns.set(font_scale=1.2)\nmean = (df_talk.is_attributed.values == 1).mean()\n\nax = sns.barplot(['Fraudulent (1)', 'Not Fradulent (0)'], [mean, 1-mean])\nax.set_xlabel('Target Value', fontsize=15) \nax.set_ylabel('Probability', fontsize=15)\nax.set_title('Target value distribution', fontsize=20)\n\nfor p, uniq in zip(ax.patches, [mean, 1-mean]):\n height = p.get_height()\n ax.text(p.get_x()+p.get_width()/2.,\n height+0.01,\n '{}%'.format(round(uniq * 100, 2)),\n ha=\"center\") ","execution_count":9,"outputs":[]},{"metadata":{"_uuid":"a401f0cf79f74f23641fe60eb55aceb3e4df93fe","_cell_guid":"21347b7a-848f-41cc-bd8c-ffaed2734b60"},"cell_type":"markdown","source":"We can see a very unbalanced dataset. Sample of 1 million and we have just 1693 or .17% of target to train the model. But, it's very normal when we are working with fraud datasets.
\n\nLet's try take a best understand of this distribuition using another features. "},{"metadata":{"_uuid":"5c252f3235e0b35f1b297ba6240d69716981eb81","_cell_guid":"fd5ae496-13a6-4230-a4a9-db7dc8609f2d"},"cell_type":"markdown","source":"

Most Frequent IPs on dataset

"},{"metadata":{"_kg_hide-input":true,"_uuid":"e9fc50611766d794bc3b801df294039d782434da","_cell_guid":"5e146e32-a2d2-4ec9-885d-dc270786c2c9","trusted":true},"cell_type":"code","source":"ip_frequency_downloaded = df_talk[df_talk['is_attributed'] == 1]['ip'].value_counts()[:20]\nip_frequency_click = df_talk[df_talk['is_attributed'] == 0]['ip'].value_counts()[:20]\n\nplt.figure(figsize=(16,10))\nplt.subplot(2,1,1)\ng = sns.barplot(ip_frequency_downloaded.index, ip_frequency_downloaded.values, color='blue')\ng.set_title(\"TOP 20 IP's where the click come from was downloaded\",fontsize=20)\ng.set_xlabel('Most frequents IPs',fontsize=16)\ng.set_ylabel('Count',fontsize=16)\n\nplt.subplot(2,1,2)\ng1 = sns.barplot(ip_frequency_click.index, ip_frequency_click.values, color='blue')\ng1.set_title(\"TOP 20 IP's where the click come from was NOT downloaded\",fontsize=20)\ng1.set_xlabel('Most frequents IPs',fontsize=16)\ng1.set_ylabel('Count',fontsize=16)\n\nplt.subplots_adjust(wspace = 0.1, hspace = 0.4,top = 0.9)\n\nplt.show()","execution_count":10,"outputs":[]},{"metadata":{"_uuid":"bbae4851e4fca0de317aa9b3095d62ea4929cc13","_cell_guid":"7901b4cc-8168-4084-b4e3-cc1773d383f4"},"cell_type":"markdown","source":"We can see that 2 ip's have a almost 2 times the others ip's clicks, but it isn't very significant in the download rate, with just 9 downloads in total"},{"metadata":{"_uuid":"1ebfbf88b584bf84fd81c99c5351f54310e5ab90","_cell_guid":"0b685068-d56b-4e1b-9f5f-0a763d5bfa19"},"cell_type":"markdown","source":"

Taking a look on App feature

"},{"metadata":{"_kg_hide-input":true,"_uuid":"a64656b2b868abb22535e83661d59229c5ae5360","_cell_guid":"cd8309f3-4688-4417-a053-e7d4eb44e448","trusted":true},"cell_type":"code","source":"app_frequency_downloaded = df_talk[df_talk['is_attributed'] == 1]['app'].value_counts()[:20]\napp_frequency_click = df_talk[df_talk['is_attributed'] == 0]['app'].value_counts()[:20]\n\nplt.figure(figsize=(16,10))\nplt.subplot(2,1,1)\ng = sns.barplot(app_frequency_downloaded.index, app_frequency_downloaded.values,\n palette='husl')\ng.set_title(\"TOP 20 APP where the click come from and downloaded\",fontsize=20)\ng.set_xlabel('Most frequents APP ID',fontsize=16)\ng.set_ylabel('Count',fontsize=16)\n\nplt.subplot(2,1,2)\ng1 = sns.barplot(app_frequency_click.index, app_frequency_click.values,\n palette='husl')\ng1.set_title(\"TOP 20 APP where the click come from NOT downloaded\",fontsize=20)\ng1.set_xlabel('Most frequents APP ID',fontsize=16)\ng1.set_ylabel('Count',fontsize=16)\n\nplt.subplots_adjust(wspace = 0.1, hspace = 0.4,top = 0.9)\n\nplt.show()","execution_count":11,"outputs":[]},{"metadata":{"_uuid":"f3092a98b9e7cd45e1e1ffff10fda4fb9263919e","_cell_guid":"e18ffb9f-426d-443c-9f22-e32b59060fa7"},"cell_type":"markdown","source":"It's very intereresting note that the app 19, 9 and 35 have the 3 highest numbers of downloads but no one appear's on the most frequent APP's."},{"metadata":{"_uuid":"346857953129138ea4c7e794d56abfae9008eaef","_cell_guid":"fb9a14fc-84c3-4ae7-aec2-86736ee71b6a"},"cell_type":"markdown","source":"

Percentual Distribuition of App's

"},{"metadata":{"_kg_hide-input":true,"_uuid":"63893596f827071646b5b37f2b61cfc5744a69d2","_cell_guid":"9a7721fa-2832-404b-93e5-4c3915e67748","trusted":true},"cell_type":"code","source":"print(\"App percentual distribuition description: \")\nprint(round(df_talk[df_talk['is_attributed'] == 1]['app'].value_counts()[:5] \\\n / len(df_talk[df_talk['is_attributed'] == 1]) * 100),2)","execution_count":12,"outputs":[]},{"metadata":{"_uuid":"4e057fb8288c13a85c27f28205f029db2e4c208e","_cell_guid":"9a146db7-c396-4a42-8a84-313bd129caa2"},"cell_type":"markdown","source":"\n\nWith first 5 highest values we have 64% of downloads total. "},{"metadata":{"_uuid":"bdcc489f56d974c82879510e63421f652fc43dd4","_cell_guid":"a89f02d7-b16f-4b27-a95d-12b7785d99a1","collapsed":true},"cell_type":"markdown","source":"

Channel feature

\n- channel is of id of mobile ad publisher"},{"metadata":{"_kg_hide-input":true,"_uuid":"516a4ce8bfbfd1c073db41a33d98069f90874000","_cell_guid":"447f056e-7145-4a31-a187-600eb148a412","trusted":true},"cell_type":"code","source":"channel_frequency_downloaded = df_talk[df_talk['is_attributed'] == 1]['channel'].value_counts()[:20]\nchannel_frequency_click = df_talk[df_talk['is_attributed'] == 0]['channel'].value_counts()[:20]\n\nplt.figure(figsize=(16,10))\n\nplt.subplot(2,1,1)\ng = sns.barplot(channel_frequency_downloaded.index, channel_frequency_downloaded.values, \\\n palette='husl')\ng.set_title(\"TOP 20 channels with download Count\",fontsize=20)\ng.set_xlabel('Most frequents Channels ID',fontsize=16)\ng.set_ylabel('Count',fontsize=16)\n\nplt.subplot(2,1,2)\ng1 = sns.barplot(channel_frequency_click.index, channel_frequency_click.values,\\\n palette='husl')\ng1.set_title(\"TOP 20 channels clicks Count\",fontsize=20)\ng1.set_xlabel('Most frequents Channels ID',fontsize=16)\ng1.set_ylabel('Count',fontsize=16)\n\nplt.subplots_adjust(wspace = 0.1, hspace = 0.4,top = 0.9)\n\nplt.show()","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"7470053a0fd29868644cdbb64ca5b7419eda3c06","_cell_guid":"bfda1d4c-130d-4a8a-a1ec-269233b2add6"},"cell_type":"markdown","source":"Let's take a look at channel proportion distribuition"},{"metadata":{"_uuid":"67872ec005be474ffaf54929403c98624151a14d","_cell_guid":"b1e94929-aa02-4bca-b832-cad03eb48bca","trusted":true},"cell_type":"code","source":"print(\"Channel percentual distribuition description: \")\nprint(round(df_talk[df_talk['is_attributed'] == 1]['channel'].value_counts()[:5] \\\n / len(df_talk[df_talk['is_attributed'] == 1]) * 100),2)","execution_count":14,"outputs":[]},{"metadata":{"_uuid":"bb24fceec8caaef7fa53ea45f360cb29e3cd5f3f","_cell_guid":"dd55573a-6cdc-498f-b23a-a4dea002f115"},"cell_type":"markdown","source":"The top five highest channels corresponds to 59% of total downloads registereds in this sample. "},{"metadata":{"_uuid":"63fe0a7cad1413dc2da50f2b6b7cad6a54f7997b","_cell_guid":"dfd6bb72-af63-4c69-9c55-a4e1ad3c7295"},"cell_type":"markdown","source":"

Device Feature

"},{"metadata":{"_kg_hide-input":true,"_uuid":"694845b47ea5f7cdfd54e21ac99265a68788b362","_cell_guid":"10bf583d-5a7b-441a-99ca-88fb0c7a41db","_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"device_frequency_downloaded = df_talk[df_talk['is_attributed'] == 1]['device'].value_counts()[:20]\ndevice_frequency_click = df_talk[df_talk['is_attributed'] == 0]['device'].value_counts()[:20]\n\nplt.figure(figsize=(16,10))\nplt.subplot(2,1,1)\ng = sns.barplot(device_frequency_downloaded.index, device_frequency_downloaded.values,\n palette='husl')\ng.set_title(\"TOP 20 devices with download - Count\",fontsize=20)\ng.set_xlabel('Most frequents Devices ID',fontsize=16)\ng.set_ylabel('Count',fontsize=16)\n\nplt.subplot(2,1,2)\ng1 = sns.barplot(device_frequency_click.index, device_frequency_click.values,\n palette='husl')\ng1.set_title(\"TOP 20 devices with download - Count\",fontsize=20)\ng1.set_xlabel('Most frequents Devices ID',fontsize=16)\ng1.set_ylabel('Count',fontsize=16)\n\nplt.subplots_adjust(wspace = 0.1, hspace = 0.4,top = 0.9)\n\nplt.show()","execution_count":15,"outputs":[]},{"metadata":{"_uuid":"06c1a3ba0f4d4d9488fc66a1a47b552f5ec491df","_cell_guid":"10c9e378-a06c-410a-a581-d58d3f41d6b1"},"cell_type":"markdown","source":"We can see a clear difference in the data. Almost all data is from the same device type. "},{"metadata":{"_uuid":"82d37c2651a3e832765569d554502fa9dfbb2b79","_cell_guid":"069bd61e-f1b1-4c5c-99b0-6cedf5e1691e","trusted":true},"cell_type":"code","source":"print(\"Device percentual distribuition: \")\nprint(round(df_talk[df_talk['is_attributed'] == 1]['device'].value_counts()[:5] \\\n / len(df_talk[df_talk['is_attributed'] == 1]) * 100),2)","execution_count":16,"outputs":[]},{"metadata":{"_uuid":"e97308cddfe8e95a0dc03c79769c6aa620054c5d","_cell_guid":"19de4a53-5d57-48ed-af1f-b4e76be2d15c"},"cell_type":"markdown","source":"The top 5 corresponds to 89% of our sample, but with significant values in just two variable... What corresponds to this values? I'm am very interested to understand. Why just two? "},{"metadata":{"_uuid":"54b5777d9afb8fd0b42e7288eda58716bc8dc80b","_cell_guid":"074aea28-38a4-4462-84ed-96df31d2e44f"},"cell_type":"markdown","source":"

Operational System version (os) Feature

"},{"metadata":{"_kg_hide-input":true,"_uuid":"f039df5c564c468b5d772e8901746a92fd763948","_cell_guid":"c6b9ba91-f87e-4f2c-80e1-838ed64ee9ca","trusted":true},"cell_type":"code","source":"os_frequency_downloaded = df_talk[df_talk['is_attributed'] == 1]['os'].value_counts()[:20]\nos_frequency_click = df_talk[df_talk['is_attributed'] == 0]['os'].value_counts()[:20]\n\nplt.figure(figsize=(16,10))\nplt.subplot(2,1,1)\ng = sns.barplot(os_frequency_downloaded.index, os_frequency_downloaded.values,\n palette='husl')\ng.set_title(\"TOP 20 OS with download - Count\",fontsize=20)\ng.set_xlabel(\"Most frequents OS's ID\",fontsize=16)\ng.set_ylabel('Count',fontsize=16)\n\nplt.subplot(2,1,2)\ng1 = sns.barplot(os_frequency_downloaded.index, os_frequency_downloaded.values,\n palette='husl')\ng1.set_title(\"TOP 20 OS with download - Count\",fontsize=20)\ng1.set_xlabel(\"Most frequents OS's ID\",fontsize=16)\ng1.set_ylabel('Count',fontsize=16)\n\nplt.subplots_adjust(wspace = 0.1, hspace = 0.4,top = 0.9)\n\nplt.show()","execution_count":17,"outputs":[]},{"metadata":{"_uuid":"cbfc6e549e50dd57f310435c7bf9c511ed1995e5","_cell_guid":"16103ff2-8b1e-4cb7-ae52-683b64af51e7","trusted":true},"cell_type":"code","source":"print(\"Device percentual distribuition: \")\nprint(round(df_talk[df_talk['is_attributed'] == 1]['os'].value_counts()[:5] \\\n / len(df_talk[df_talk['is_attributed'] == 1]) * 100),2)","execution_count":18,"outputs":[]},{"metadata":{"_uuid":"c54eae1330cae69ea4c20c136125774669e9b959","_cell_guid":"9271555d-66a8-4474-a808-7554ecb31573"},"cell_type":"markdown","source":"The first 5 highest values in this sample represents 55% of total downloads"},{"metadata":{"_uuid":"0eefaa6a82423636f4c45fc029d1d4ebc0ac3652","_cell_guid":"06cae285-c3bb-4d02-a1cf-514c340b185c"},"cell_type":"markdown","source":"

Let's take a look at our new features extracteds by time

"},{"metadata":{"_uuid":"c11e1920db1f279ef57763f55835f4ef63d56323","_cell_guid":"d1d2169e-d064-47f5-a64a-f24b6c0a7fb3"},"cell_type":"markdown","source":"Visualizing the value's in hour column"},{"metadata":{"_kg_hide-input":true,"_uuid":"113b7cac7f26c78362c099c3a95327382a576d90","_cell_guid":"678cc897-5cc0-4794-bf37-5ded8e49650f","trusted":true},"cell_type":"code","source":"hour_frequency_downloaded = df_talk[df_talk['is_attributed'] == 1]['hour'].value_counts()\nhour_frequency_click = df_talk[df_talk['is_attributed'] == 0]['hour'].value_counts()\n\nplt.figure(figsize=(16,10))\nplt.subplot(2,1,1)\ng = sns.barplot(hour_frequency_downloaded.index, hour_frequency_downloaded.values,\n palette='husl')\ng.set_title(\"Downloads Count by Hour\",fontsize=20)\ng.set_xlabel(\"Hour Download distribuition\",fontsize=16)\ng.set_ylabel('Count',fontsize=16)\n\nplt.subplot(2,1,2)\ng1 = sns.barplot(hour_frequency_click.index, hour_frequency_click.values,\n palette='husl')\ng1.set_title(\"Clicks Count by Hour\",fontsize=20)\ng1.set_xlabel(\"Hour Click distribuition\",fontsize=16)\ng1.set_ylabel('Count',fontsize=16)\n\nplt.subplots_adjust(wspace = 0.1, hspace = 0.4,top = 0.9)\n\nplt.show()","execution_count":19,"outputs":[]},{"metadata":{"_uuid":"ad796e298bf7d2e5c1f04d23fdb7c4b96dac2326","_cell_guid":"df684578-918f-4c8e-a85d-f9d1b457cd7d"},"cell_type":"markdown","source":"We can see a clear difference in distribuition of hours, but we need see with the full dataset to a betters understand. "},{"metadata":{"_uuid":"55a36e303e8ce2afd4fb1f87de38b5b1b8ea64d4","_cell_guid":"9b46ba96-28ad-415e-9045-0b1d5ee2061d"},"cell_type":"markdown","source":"## Calculating new features using click_time\n\n- First I will transform the click_time in nanosecs"},{"metadata":{"_uuid":"6924adc18166c8d24d5832641b7614eadf21a454","_cell_guid":"b5464d4e-b93b-4037-82a9-de84f0ba269f","collapsed":true,"trusted":true},"cell_type":"code","source":"df_talk['click_nanosecs'] = (df_talk['click_time'].astype(np.int64) // 10 ** 9).astype(np.int32)","execution_count":20,"outputs":[]},{"metadata":{"_uuid":"1b70f8602b93fb11035f387b68271783bc1edaa1","_cell_guid":"7c5506d8-165e-42dd-a4b5-44917c14c23e","trusted":true,"collapsed":true},"cell_type":"code","source":"df_talk['next_click'] = (df_talk.groupby(['ip', 'app', 'device', 'os']).click_nanosecs.shift(-1) - df_talk.click_nanosecs).astype(np.float32)","execution_count":21,"outputs":[]},{"metadata":{"_uuid":"9ac118ad1130cf4b5eefcc0c8ecd7cabcef4f444","_cell_guid":"29c0892a-8448-4811-839e-3c268e64d476","collapsed":true,"trusted":true},"cell_type":"code","source":"df_talk['next_click'].fillna((df_talk['next_click'].mean()), inplace=True)","execution_count":22,"outputs":[]},{"metadata":{"_uuid":"cd7e826bdf29d6fcf21c93a374b48dd2a6ff924e","_cell_guid":"e7682b19-1784-4c8d-a93d-94d064498702","collapsed":true,"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2ffa93dca0c999c52effba0f9656bc7f25fc87e6","_cell_guid":"14eacc95-2397-4954-8207-7155ce6a39fb"},"cell_type":"markdown","source":"

Visualizing the minute column

"},{"metadata":{"_kg_hide-input":true,"_uuid":"4c7f2dd314b17146656556da5af91d2a4e03e64e","_cell_guid":"7ebd59c4-f006-496d-9988-1478d6d51317","trusted":true},"cell_type":"code","source":"minute_frequency_downloaded = df_talk[df_talk['is_attributed'] == 1]['minute'].value_counts()\nminute_frequency_click = df_talk[df_talk['is_attributed'] == 0]['minute'].value_counts()\n\nplt.figure(figsize=(16,10))\nplt.subplot(2,1,1)\ng = sns.barplot(minute_frequency_downloaded.index, minute_frequency_downloaded.values,\n palette='husl')\ng.set_title(\"Downloads Count by Minute\",fontsize=20)\ng.set_xlabel(\"Minute Download distribuition\",fontsize=16)\ng.set_ylabel('Count',fontsize=16)\n\nplt.subplot(2,1,2)\ng1 = sns.barplot(minute_frequency_click.index, minute_frequency_click.values,\n palette='husl')\ng1.set_title(\"Clicks Count by Minute\",fontsize=20)\ng1.set_xlabel(\"Minute Click distribuition\",fontsize=16)\ng1.set_ylabel('Count',fontsize=16)\n\nplt.subplots_adjust(wspace = 0.1, hspace = 0.4,top = 0.9)\n\nplt.show()","execution_count":23,"outputs":[]},{"metadata":{"_uuid":"e33d58c85298a13d59218ef29406d2e744f1cf11","_cell_guid":"3432e9f7-60c6-4bfb-9019-1e97cb00ca4e"},"cell_type":"markdown","source":"Interesting that in the first minutes of an hour the rate of downloads is higher. We can see this in the both filters\n"},{"metadata":{"_uuid":"47a37253d97a72c2d20ae6c383a978784e739904","_cell_guid":"fa887828-9c54-4038-9ab5-ec29adad7d39"},"cell_type":"markdown","source":"

Visualizing the Second distribuition.

"},{"metadata":{"_kg_hide-input":true,"_uuid":"bf1bf7da69a6fd229fcb4f478e297a13f6e3facc","_cell_guid":"a9e219a7-9049-4dc6-9f91-dde6e96c88f7","trusted":true},"cell_type":"code","source":"second_frequency_downloaded = df_talk[df_talk['is_attributed'] == 1]['second'].value_counts()\nsecond_frequency_click = df_talk[df_talk['is_attributed'] == 0]['second'].value_counts()\n\nplt.figure(figsize=(16,10))\nplt.subplot(2,1,1)\ng = sns.barplot(second_frequency_downloaded.index, second_frequency_downloaded.values,\n palette='husl')\ng.set_title(\"Downloads Count by Hour\",fontsize=20)\ng.set_xlabel(\"Second Download distribuition\",fontsize=16)\ng.set_ylabel('Count',fontsize=16)\n\nplt.subplot(2,1,2)\ng1 = sns.barplot(second_frequency_click.index, second_frequency_click.values,\n palette='husl')\ng1.set_title(\"Clicks Count by Hour\",fontsize=20)\ng1.set_xlabel(\"Second Click distribuition\",fontsize=16)\ng1.set_ylabel('Count',fontsize=16)\n\nplt.subplots_adjust(wspace = 0.1, hspace = 0.4,top = 0.9)\n\nplt.show()","execution_count":24,"outputs":[]},{"metadata":{"_uuid":"a27b2a1aa98ae7a640f19bece83aec416d9ce5ff","_cell_guid":"ee5b1f39-e072-42aa-9fbf-4acdec6652af","collapsed":true},"cell_type":"markdown","source":"In seconds we see a little difference to a secnnd to a nother"},{"metadata":{"_uuid":"53c69481f5b3d75c4f2577184bcbf020c3eb7bd5","_cell_guid":"abb7c8ca-2ec6-4393-a3d4-e4ffd0e79481"},"cell_type":"markdown","source":"

Feature Engineering in the categorical's features and IP

\n- This is a copy of the brilliannnt kernel of user NanoMathias that you can see the kernel here, that is a lecture of feature engineering"},{"metadata":{"_kg_hide-input":true,"_uuid":"4ae381f07654971afff3215612f55726a2070f17","_cell_guid":"e3eb7ed9-7648-477d-9363-180709543a05","trusted":true},"cell_type":"code","source":"import gc\n#Define all the groupby transformations\nGROUPBY_AGGREGATIONS = [\n \n # V1 - GroupBy Features #\n ######################### \n # Variance in day, for ip-app-channel\n {'groupby': ['ip','app','channel'], 'select': 'day', 'agg': 'var'},\n # Variance in hour, for ip-app-os\n {'groupby': ['ip','app','os'], 'select': 'hour', 'agg': 'var'},\n # Variance in hour, for ip-day-channel\n {'groupby': ['ip','day','channel'], 'select': 'hour', 'agg': 'var'},\n # Count, for ip-day-hour\n {'groupby': ['ip','day','hour'], 'select': 'channel', 'agg': 'count'},\n # Count, for ip-app\n {'groupby': ['ip', 'app'], 'select': 'channel', 'agg': 'count'}, \n # Count, for ip-app-os\n {'groupby': ['ip', 'app', 'os'], 'select': 'channel', 'agg': 'count'},\n # Count, for ip-app-day-hour\n {'groupby': ['ip','app','day','hour'], 'select': 'channel', 'agg': 'count'},\n # Mean hour, for ip-app-channel\n {'groupby': ['ip','app','channel'], 'select': 'hour', 'agg': 'mean'}, \n \n # V2 - GroupBy Features #\n #########################\n # Average clicks on app by distinct users; is it an app they return to?\n {'groupby': ['app'], \n 'select': 'ip', \n 'agg': lambda x: float(len(x)) / len(x.unique()), \n 'agg_name': 'AvgViewPerDistinct'\n },\n # How popular is the app or channel?\n {'groupby': ['app'], 'select': 'channel', 'agg': 'count'},\n {'groupby': ['channel'], 'select': 'app', 'agg': 'count'},\n \n # V3 - GroupBy Features #\n # https://www.kaggle.com/bk0000/non-blending-lightgbm-model-lb-0-977 #\n ###################################################################### \n {'groupby': ['ip'], 'select': 'channel', 'agg': 'nunique'}, \n {'groupby': ['ip'], 'select': 'app', 'agg': 'nunique'}, \n {'groupby': ['ip','day'], 'select': 'hour', 'agg': 'nunique'}, \n {'groupby': ['ip','app'], 'select': 'os', 'agg': 'nunique'}, \n {'groupby': ['ip'], 'select': 'device', 'agg': 'nunique'}, \n {'groupby': ['app'], 'select': 'channel', 'agg': 'nunique'}, \n {'groupby': ['ip', 'device', 'os'], 'select': 'app', 'agg': 'nunique'}, \n {'groupby': ['ip','device','os'], 'select': 'app', 'agg': 'cumcount'}, \n {'groupby': ['ip'], 'select': 'app', 'agg': 'cumcount'}, \n {'groupby': ['ip'], 'select': 'os', 'agg': 'cumcount'}, \n {'groupby': ['ip','day','channel'], 'select': 'hour', 'agg': 'var'} \n]\n\n# Apply all the groupby transformations\nfor spec in GROUPBY_AGGREGATIONS:\n \n # Name of the aggregation we're applying\n agg_name = spec['agg_name'] if 'agg_name' in spec else spec['agg']\n \n # Name of new feature\n new_feature = '{}_{}_{}'.format('_'.join(spec['groupby']), agg_name, spec['select'])\n \n # Info\n print(\"Grouping by {}, and aggregating {} with {}\".format(\n spec['groupby'], spec['select'], agg_name\n ))\n \n # Unique list of features to select\n all_features = list(set(spec['groupby'] + [spec['select']]))\n \n # Perform the groupby\n gp = df_talk[all_features]. \\\n groupby(spec['groupby'])[spec['select']]. \\\n agg(spec['agg']). \\\n reset_index(). \\\n rename(index=str, columns={spec['select']: new_feature})\n \n # Merge back to X_total\n if 'cumcount' == spec['agg']:\n df_talk[new_feature] = gp[0].values\n else:\n df_talk = df_talk.merge(gp, on=spec['groupby'], how='left')\n \n # Clear memory\n del gp\n gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b84bbf60527947a13753e94741288a015cdd3ba5","_cell_guid":"b3c2352b-6698-41ea-afe0-2b83a2fd5e38"},"cell_type":"markdown","source":"Did it, let's see if the new features affects our results"},{"metadata":{"_uuid":"104cb17f5f060fcaede7e2eb5ab59662bc9f22d6","_cell_guid":"b532a166-fda8-4200-a7ff-7ee66bb0e9c2"},"cell_type":"markdown","source":"# Evaluating Feature Importance\n - I'll fit xgBoost to the data, and evaluate the feature importances. First split into X and y"},{"metadata":{"_kg_hide-input":true,"_uuid":"4e7661a6102ce1c375ffae1991945563e5cc859d","_cell_guid":"8e60d65b-4b9c-47ef-a47f-f9ae3134a4e6","trusted":true,"collapsed":true},"cell_type":"code","source":"import xgboost as xgb\n\n# Split into X and y\ny = df_talk['is_attributed']\nX = df_talk.drop('is_attributed', axis=1).select_dtypes(include=[np.number])\n\n# Create a model\n# Params from: https://www.kaggle.com/aharless/swetha-s-xgboost-revised\nclf_xgBoost = xgb.XGBClassifier(\n max_depth = 4,\n subsample = 0.8,\n colsample_bytree = 0.7,\n colsample_bylevel = 0.7,\n scale_pos_weight = 9,\n min_child_weight = 0,\n reg_alpha = 4,\n n_jobs = 4, \n objective = 'binary:logistic'\n)\n# Fit the models\nclf_xgBoost.fit(X, y)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"5d5bbcdb20c1328cce4ae9ceb64d963e5e161790","_cell_guid":"f12fd12d-850c-436c-8a33-97bf1264085d","collapsed":true},"cell_type":"markdown","source":"## Verifying feature importances"},{"metadata":{"_kg_hide-input":true,"_uuid":"913408e61798553a4d27dce92e784007d4fb93f9","_cell_guid":"492057f9-5616-406a-a798-87035e5037c1","trusted":true,"collapsed":true},"cell_type":"code","source":"from sklearn import preprocessing\n\n# Get xgBoost importances\nfeature_importance = {}\nfor import_type in ['weight', 'gain', 'cover']:\n feature_importance['xgBoost-'+import_type] = clf_xgBoost.get_booster().get_score(importance_type=import_type)\n \n# MinMax scale all importances\nfeatures = pd.DataFrame(feature_importance).fillna(0)\nfeatures = pd.DataFrame(\n preprocessing.MinMaxScaler().fit_transform(features),\n columns=features.columns,\n index=features.index\n)\n\n# Create mean column\nfeatures['mean'] = features.mean(axis=1)\n\n# Plot the feature importances\nfeatures.sort_values('mean').plot(kind='bar', figsize=(16, 6))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0a37f9e5aa494980f7dd20977c39d13ff2413dc9","_cell_guid":"b80df836-9397-420a-9ebe-1504965ff287","_kg_hide-output":true,"collapsed":true},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"68b0b762cd465688da3577e565a5e21c27b799ac","_cell_guid":"470f6b67-e2e5-4c32-8762-9827e5d06eaf","collapsed":true},"cell_type":"markdown","source":"I will continue with the conclusion on this kernel"},{"metadata":{"_uuid":"8131d2d261919e44ba324dd62df2404c10361554","_cell_guid":"5b6615bb-a1fd-4303-8d8d-c0c08f1f2df4"},"cell_type":"markdown","source":"

Fonts:

\nI have implemented some some techniques from another kernels. Some of this:
\nhttps://www.kaggle.com/nanomathias/feature-engineering-importance-testing
\nhttps://www.kaggle.com/anokas/talkingdata-adtracking-edahttps://www.kaggle.com/anokas/talkingdata-adtracking-eda
\nhttps://www.kaggle.com/jtrotman/eda-talkingdata-temporal-click-count-plots
\nhttps://www.kaggle.com/chubing/feature-engineering-and-xgboost
\nand much others that I will also tag. \n \n \n"},{"metadata":{"_uuid":"36253a37287cce5b5bed012221a9fbd3fa4552c5","_cell_guid":"f6265477-5f83-4a3a-9d32-e69ca60ee02b","collapsed":true,"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61791bd1dcd3e44d267ab577e61844bbe4548fdf","_cell_guid":"0d97996c-2340-4501-a519-06c9616202f8","collapsed":true,"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dd95a3f738c7b7f6204aef9a2212a17ccd786e86","_cell_guid":"b28cea16-1785-4a01-8d10-f9314f1e1f50","collapsed":true,"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}