{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","collapsed":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"cell_type":"markdown","source":"# EDA - Fraud Detection\n---------------------------------------\n# Problem Statement\n\nTalkingData, China’s largest independent big data service platform, covers over 70% of active mobile devices nationwide. They handle 3 billion clicks per day, of which 90% are potentially fraudulent. Their current approach to prevent click fraud for app developers is to measure the journey of a user’s click across their portfolio, and flag IP addresses who produce lots of clicks, but never end up installing apps. With this information, they've built an IP blacklist and device blacklist.\n\nWhile successful, they want to always be one step ahead of fraudsters and have turned to the Kaggle community for help in further developing their solution. In their 2nd competition with Kaggle, we’re challenged to build an algorithm that predicts whether a user will download an app after clicking a mobile app ad. To support our modeling, they have provided a generous dataset covering approximately 200 million clicks over 4 days!\n\nThe clicks that lead to download are considered Non-Fraudulent. So I may use the terms App Downloaded and Non-Fraudulent Clicks interchangeably.\n\nThe training dataset is quite huge - 185 million\n\n# Talking Data is trying to accomplish something on the following lines:\n\n# Basic Prescriptive Action\n\n**Hypothesis** - IPs having % of Fradulent downloads = 100 % are possibly fradulent.\n**Prescriptive Action** - Block the IPs\n\n**Hypothesis** - IPs having % of Fradulent downloads > 80 % are possibly fradulent.\n**Prescriptive Action** - Temporarily hold the payment for the ad click and not charge the APP and investigate further.\nWhat not to do yet - Block the IP\n\n# Advanced Prescriptive Action\n\n**Hypothesis** - Predict the likelihood of a click from an app being fradulent using the **best predictive model** developed in Kaggle. If Likelyhood > Threshold 1 \n**Prescriptive Action** - Block the IPs\n\n**Hypothesis** - Predict the likelihood of a click from an app being fradulent using the **best predictive model** developed in Kaggle. If Likelyhood > Threshold 2\n**Prescriptive Action** - Temporarily hold the payment for the ad click and not charge the APP and investigate further\n**What not to do yet** - Block the IP\n\n\n# List of Vizulizations"},{"metadata":{"_cell_guid":"79a98b1c-e99c-428c-a6a4-b646cbaf52e1","_uuid":"9cb3ea0e9e61472d3afce8a271d77e690df91dd9"},"cell_type":"markdown","source":"\n- 0. Train Vs Test Vs Test Supplement \n- 1. Total Number of Fraudulent Vs Non-Fraudulent Clicks in numbers and percentage\n- 2. Count of Distinct Categorical variables\n- 3. Snapshot of Ips that have 100% Fraudulent Clicks\n- 4. Cluster Model for IPs\n- 5. Snapshot of OSs that have 100% Fraudulent Clicks\n- 6. Cluster Model for OSs\n- 7. Snapshot of Devices that have 100% Fradulent Clicks\n- 8. Cluster Model for Devices\n- 9. Snapshot of Apps that have 100% Fraudulent Clicks\n- 10. Cluster Model for Apps\n- 11. Snapshot of Channels that have 100% Fraudulent Clicks\n- 12. Cluster Model for Channels\n- 13. Number of Fraudulent Clicks by Device\n- 14. Number of Non-Fraudulent Clicks by Device\n- 15. Total Number of Clicks per Hour\n- 16. Total Number of Clicks per Day\n- 17. Percentage of Non-Fraudulent Clicks Day wise\n- 18. Percentage of Non-Fraudulent Clicks - Hour of the day\n- 19. Clustering Channels\n- 20. Clustering Apps\n- 21. Clustering Devices\n- 22. Clustering OS\n- 23. Clustering IPs\n- 24. Clustering Apps, Devices, Channels,OSs\n- 25. Scatter Plot of Fraudulent Vs Non-Fraudulent Clicks for all Channels\n- 26. Scatter Plot of Clicks(No Download) Vs Clicks (Download) for Hour of the Day\n- 27. Scatter Plot of Fraudulent Vs Non-Fraudulent Clicks for all IPs\n- 28. Scatter Plot of Fraudulent Vs Non-Fraudulent Clicks for all Devices\n- 29. Scatter Plot of Fraudulent Vs Non-Fraudulent Clicks for all Apps\n- 30. Scatter Plot of Fraudulent Vs Non-Fraudulent Clicks for all OSs\n- 31. Some Fun Visualizations\n\n\n \n \n "},{"metadata":{"_cell_guid":"4e12ef13-f15b-4bf4-b4ce-48d703b8f3f2","_uuid":"048b9d6ada99ee1d1ab22d2531c26fc246ed6834","trusted":true,"collapsed":true},"cell_type":"code","source":"# 0. Train Vs Test Vs Test Supplement\n---------------------------------------","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"295a3688-f86a-484c-9047-9648d50070d2","_uuid":"35511b2d713d2cd39facc552c6d5c071ed0473ec"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"f022a60f-b178-4fd1-a1dd-2a66acda6cd7","_uuid":"1bb866073b61b0cdc74706c53e014b6a92459709","trusted":true,"collapsed":true},"cell_type":"code","source":"So the train data has partial data for 6th and 9th of November and complete data for 7th and 8th of November. 6th,7th,8th,9th are Mon,Tue, Wed, Thurs respectively\n\nWe can say that from Monday to Thursday there is no big variation in the total number of clicks.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8d9d07d6-20f6-494c-b718-006867caa801","_uuid":"d8a7c5af589e25d0974df7202681d2c0768a7662","trusted":true,"collapsed":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"9fdfc875-3700-433c-974b-cb4bcef568fc","_uuid":"772c3c12781428c1f154239c5bd715bb027b3d9d"},"cell_type":"markdown","source":"The test data set has only data of 10th Nov which happens to be a Friday and data has been provided only for the hours - 4, 5, 9, 10, 13, 14. "},{"metadata":{"_cell_guid":"08679d1e-2df8-458c-a082-330b43070729","_uuid":"347b97880e3e202501954631e65f161f9115eb15"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"f1934ab1-e67b-4565-bd75-c074f27b3a7a","_uuid":"d43ea9082f03f3af52212b1d9c8b844b38e984cc"},"cell_type":"markdown","source":"This dataset was leaked unintentionally by Talkingdata during initial days of the compitition. This is only for reference. Evaluation will be done on the test set only."},{"metadata":{"_cell_guid":"34928b95-8f04-4cf0-ab00-9937857441e8","_uuid":"c72521542575b02f8378f6fd87d3e9dc0c8993ca"},"cell_type":"markdown","source":"\n# 1. Total Number of Fraudulent Vs Non-Fraudulent Clicks in numbers and percentage\n---------------------------------------\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"},"cell_type":"markdown","source":"\n"},{"metadata":{"_cell_guid":"fa854956-bd40-4e6e-a93b-d0d774638107","_uuid":"f9feef4adf1ac684dbd953dbac17d4cb8845bb7d"},"cell_type":"markdown","source":"So downloading the app is a Rare Event . It is a highly unbalanced datset with only 0.25% of clicks leading to Downloads. We may have to figure out a good sampling technique.\nNow that some fraud on staggering scale."},{"metadata":{"_cell_guid":"e5f4b8c9-a4b6-4cd9-aecc-af9505c23408","_uuid":"7dfb8ed21acaa92455d424d7df9c6e95458355db"},"cell_type":"markdown","source":"**IP, App, Device, OS and Channel are the categorical variables. Lets look at the distinct count of these categorical variables.**"},{"metadata":{"_cell_guid":"dc1bc1ee-b5b0-4c33-8745-992190d533b9","_uuid":"a85bd7734557d239e0b9b21a49048860d3f02a98"},"cell_type":"markdown","source":"# 2. Count of Distinct Categorical variables\n---------------------------------------\n"},{"metadata":{"_cell_guid":"11fb6407-55b8-4e72-9fbe-fef3425dd3f7","_uuid":"ee91271f570ec94bb120a8806b2e30c81b94bee2"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"809d25b6-4087-44ed-8aab-422383c56fd2","_uuid":"6c72355303f86ec484965220ac0282bc843a8fac"},"cell_type":"markdown","source":"**Organised Bot Driven Fraud**"},{"metadata":{"_cell_guid":"0bb82b61-9883-446e-9eb7-d020208a355e","_uuid":"2ab0229a96966b6c4f2e2822ef8d06c661e76173"},"cell_type":"markdown","source":"**Lets Blackist these IPs**"},{"metadata":{"_cell_guid":"9d55f3d5-532a-4405-83d3-db512a0a9817","_uuid":"fc1fa20ace7ffd6f3ec5abbe90d85e56005382d1"},"cell_type":"markdown","source":"# 3. Snapshot of Ips that have 100% Fraudulent Clicks\n---------------------------------------"},{"metadata":{"_cell_guid":"80389e83-063a-49fc-a572-478be30af9c6","_uuid":"fde6fef1d9f4d3e6c007ffb91f2811fda680325f"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"b368c7c5-9d46-4c3e-b7f1-103fd4497c4c","_uuid":"42368b9075c079f78f7c46488ca111999bfe8535"},"cell_type":"markdown","source":"This is just a snapshot. That's a insane nuber of clikcs coming from IPs that dont convert to a download in a period of 4 days! \n\n**Bold Statement : These IPs should become part of TalkingData's IP blaclist. Please comment if you guys think otherwise.**"},{"metadata":{"_cell_guid":"a737f431-2ea4-44ef-b8c4-461630808d57","_uuid":"2307b43c86c0f630ea306a1686017efde19ebbb9"},"cell_type":"markdown","source":"**Cluster Model for IPs**"},{"metadata":{"_cell_guid":"c4af8cca-1b03-4208-ac00-61b4a27c8a1f","_uuid":"2cb65a71536612095ad7562bae46a1e75d67ba23"},"cell_type":"markdown","source":"# 4. Cluster Model for IPs\n---------------------------------------"},{"metadata":{"_cell_guid":"50b586db-4798-4aa0-9979-f0fb1055a0dd","_uuid":"fd01d9f4256f6b2d5197342eb31f01a50999243d"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"293bd0d2-969d-480e-ab14-231e7b6f9f87","_uuid":"747bbf344cbf5ac69a373152a7a62509721d18dd"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"8b78000b-9a0d-4c0e-aa67-5945c3e8f7bd","_uuid":"8f695fb9168b72e05d1cc8b146e5c4452ea28f02"},"cell_type":"markdown","source":"The Cluster Model gives a fair idea of the distribution of IPs that are Fradulent and Non- Fradulent. Majority of IPs are centered around 1% of Non - Fradulent Clicks. While the nex majority of IPs are centered around 99% of Non- fradulent Clicks."},{"metadata":{"_cell_guid":"a956d51e-80f5-48e6-a4c9-148e53258731","_uuid":"0184febc5a1d051d98a93606ea8ef3534e246a6d"},"cell_type":"markdown","source":"# 5. Snapshot of OSs that have 100% Fraudulent Clicks\n---------------------------------------"},{"metadata":{"_cell_guid":"8fcf4e19-8ec2-4f11-ad03-7416de10aa98","_uuid":"b213668e1a2d4f954387a004ce81f7ffd8c60443"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"22795073-8009-4a16-8084-b27a6f163bad","_uuid":"f07f65f5f5a2ba2432a77d9f25ee2c93e3c7a398"},"cell_type":"markdown","source":"**Cluster Model for OS**"},{"metadata":{"_cell_guid":"75290c3d-c132-4492-bb63-c624728199db","_uuid":"18a3850e1e14e54aa5b9d5714873e834fc991f0d"},"cell_type":"markdown","source":"# 6. Cluster Model for OSs\n---------------------------------------"},{"metadata":{"_cell_guid":"ad4114c7-8cb1-4dd0-af36-69dbc70ad240","_uuid":"ff8e560c46123206ffccf908ed46c3cebb99958b"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"4b38dfc7-af4c-4401-b957-e92d8a0ebb06","_uuid":"472311c77a8a1be600120473fd822e9562568f43"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"fe750141-d4c1-44b5-b904-689e0bac8ab3","_uuid":"e65ccf12ae887b65bd9427db1340d97e0dad2da7"},"cell_type":"markdown","source":" Majority of OSs are centered around 0.06% of Non - Fradulent Clicks. Only few OSs are authentic."},{"metadata":{"_cell_guid":"ebdea9af-340a-45b6-acd6-f77539717faa","_uuid":"601102f895f01d09626c2f1c048914b1473f5e8f"},"cell_type":"markdown","source":"# 7. Snapshot of Devices that have 100% Fradulent Clicks\n---------------------------------------"},{"metadata":{"_cell_guid":"f4693400-b927-4556-adf5-ae400779a41d","_uuid":"c6eb52cf7ec53cd4163a592f7cbab5674d087053"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"d7babb62-e75d-47a9-bd3a-d703adfb3438","_uuid":"34b116f3750435e9a367085443678b7db53d6395"},"cell_type":"markdown","source":"**TalkingData should add Device 5, Device 182, Device 1728 to their Device Blacklist**"},{"metadata":{"_cell_guid":"d0840d8e-95b4-4b1a-8df0-54f5dc3d84ed","_uuid":"c866986953e538b1c864043fb34bf84fcd565657"},"cell_type":"markdown","source":"# 8. Cluster Model for Devices\n---------------------------------------"},{"metadata":{"_cell_guid":"a4ae6ae7-68f9-413a-a519-633eecc3e071","_uuid":"e5e0b8f85f45fc0bdccd41869e1d3d6b97eab42a"},"cell_type":"markdown","source":"**Cluster Model for Devices**"},{"metadata":{"_cell_guid":"cf7af69c-6bd1-4c2a-a5cd-8b98edaa2ec7","_uuid":"2e5610e00d4139a4fa30dcd44b3b72de79042e1d"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"1c429096-83ae-4cf3-bb2e-1ad16ba10df7","_uuid":"204c12ab0d74c26c2b845a80b21a92cad8fdd3d1"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"b6c7c13d-80d6-48ef-bde2-60fa6eb2ac67","_uuid":"606018fddc7478ccac7c9903fb94280e8ee97fec"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"70125cfd-467e-42f4-84a2-c2b30e892141","_uuid":"d291b3673801f73a1f76bafabe8c219fc3d9a6ee"},"cell_type":"markdown","source":"** Majority of Devices are centered around 0.36% of Non - Fradulent Clicks. We should note that 156 devices have 100 Non-Fradulent Clicks. This is really interesting.**"},{"metadata":{"_cell_guid":"718e0e5a-4ce6-4c57-b382-718fa7579f89","_uuid":"9e63268087f071ddb0f89649af9e3718b669cc03"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"04719ad1-b2e4-4b50-9651-96c4266ead6d","_uuid":"2f3b0cc634e0c40a959b935ef5f4604fd022a084"},"cell_type":"markdown","source":"# 9. Snapshot of Apps that have 100% Fraudulent Clicks\n---------------------------------------"},{"metadata":{"_cell_guid":"bc689de7-8aa4-4099-ac74-bd0934eab438","_uuid":"e68f6979839376cf4496ed22f5f1b83e7983e74b"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"8abe1cc3-e53c-4b61-8564-5b96fedac001","_uuid":"02d8dd7c4a44fff648453056ce0e0476c1281721"},"cell_type":"markdown","source":"**Cluster Model for Apps**"},{"metadata":{"_cell_guid":"6110e076-29d4-42d1-8ceb-45ec30d30d67","_uuid":"334593ae7560735ef7c5f2f3e9a3d3e19b59cf85"},"cell_type":"markdown","source":"# 10. Cluster Model for Apps\n---------------------------------------"},{"metadata":{"_cell_guid":"3232cba2-cfe9-4e0b-91ce-68f5e0c3de07","_uuid":"40951974d122465fce3d8f13ab68e8288099a095"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"36ceccc1-a655-46f7-b548-592662642c28","_uuid":"289658516f937d7e89a313053156d4610a99d3ee"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"0dc58dbf-612d-44b2-b4cf-c6bb83bd7abe","_uuid":"42066f2c153aab25386d5424e1e8af8a1929331d"},"cell_type":"markdown","source":" Majority of Apps are centered around 0.23% of Non - Fradulent Clicks. While rest of the Apps have mixed Clicks. Only 8 of them have 97% Non Fraduelent Clicks. Does this mean the Apps play a role in the fraud? SInce they are the benefactors of the clicks."},{"metadata":{"_cell_guid":"36fd2d39-1a2e-497d-99dd-417942bc27a9","_uuid":"33a1c5c100bd4b99ec96d317a9b30c8796f639cf"},"cell_type":"markdown","source":"# 11. Snapshot of Channels that have 100% Fraudulent Clicks\n---------------------------------------"},{"metadata":{"_cell_guid":"0616b238-1769-481b-90cd-fec1668b3d44","_uuid":"8cf2247139d5058f448a2016429ec28ff103e674"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"0ea9acec-029c-451e-864a-29389c2483ad","_uuid":"30340a712ad7cc8c6e1719e93ceda1d3869a26db"},"cell_type":"markdown","source":"**Cluster Model for Channels**"},{"metadata":{"_cell_guid":"4e2c4fd4-4e38-4cd0-8635-08fa94963e19","_uuid":"86f3470c1bed32f1b5bd62c27e828de3dabb6f26"},"cell_type":"markdown","source":"# 12. Cluster Model for Channels\n---------------------------------------"},{"metadata":{"_cell_guid":"7f57162a-e321-488c-9c5f-5befbd4365c7","_uuid":"0b5a82086d5ddeb947220565defe2a069dcc2538"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"84f9f229-fa42-43a2-853f-6f17c9b3d619","_uuid":"199ce3302d846edb7f13321c75e547e9559fe28e"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"4d99e29a-885e-4d77-b825-d4eec081b53b","_uuid":"64a34bfa5b3e7662624fc8cc68d1e243153dde93"},"cell_type":"markdown","source":" Majority of Channels are centered around 2% of Non - Fradulent Clicks. Wonder about the authenticity of the Channels."},{"metadata":{"_cell_guid":"6a8754cc-abf3-4af5-8ed1-bdd6f16cff3d","_uuid":"7b5dae90bd6720c1b41a3c25c842a8264b982618"},"cell_type":"markdown","source":"# 13. Number of Fraudulent Clicks by Device\n---------------------------------------"},{"metadata":{"_cell_guid":"3f1273e9-ad7a-4dc9-8d09-91ee48043e21","_uuid":"84a18a012d1d2ed5d5e87f91461891897bbf75fa"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"b979fbc0-193f-491f-8ec0-7901990139a5","_uuid":"5e071fdfd269e2832b9d1b30a1c2a79d2776c9df"},"cell_type":"markdown","source":"Majority of the Fradulent Clicks are from Device 1 and Device 2"},{"metadata":{"_cell_guid":"3f3ec7b0-75c1-4533-9355-65cdfa2fcbd6","_uuid":"bb30106d6a5ede478248d0078aab981c1a408b4e"},"cell_type":"markdown","source":"# 14. Number of Non-Fraudulent Clicks by Device\n---------------------------------------"},{"metadata":{"_cell_guid":"8d26986a-6402-4a19-9b65-1bc42d518a9f","_uuid":"747fc83e9cb8a4804d3e6fe3f69a87fc3d6c18df"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"8c51d351-3f99-47bc-b961-42a66d099378","_uuid":"2350913e962a14e0a1aecb60bf039cbd94abbf91"},"cell_type":"markdown","source":"Majority of Non-Fradulent Clicks are from Device 1 ans Device 0\n"},{"metadata":{"_cell_guid":"587b5a26-49a5-451c-9edb-707fc8a673dc","_uuid":"d540b48f79ee4bbd4ba0d549470a41c17953556c"},"cell_type":"markdown","source":"**Effect of Click Time**"},{"metadata":{"_cell_guid":"a83daf85-ba58-41b3-8fc6-18df6e5f819b","_uuid":"06832ad28f5b6eb1cef422649ad81a9c13db1862"},"cell_type":"markdown","source":"# 15. Total Number of Clicks per Hour\n---------------------------------------"},{"metadata":{"_cell_guid":"d9484157-c6d3-40af-9b1b-722fa1a668c7","_uuid":"e76fba9973c3be504228e2edc553cfd820920acf"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"d8e064e0-3953-455c-b4f8-8a0df8ea8f80","_uuid":"9e5176a4aaae1f375432b3a1ed3a6b7daed3797b"},"cell_type":"markdown","source":"# 16. Total Number of Clicks per Day\n---------------------------------------"},{"metadata":{"_cell_guid":"62ec0d14-e9d8-4a9f-b765-abc87fa0c9b1","_uuid":"f9cbc53c6334a5f753a62096b92ec67600ac9bfc"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"b89bf6ec-e825-46bf-a7f7-3278b9b25f26","_uuid":"cdd876dedea35b0bd25cb1860e7311e194f78352"},"cell_type":"markdown","source":"6th and 9th are partial data and 7th and 8th are full data. We can say that from Monday to Thursday there is no big variation in the total number of clicks."},{"metadata":{"_cell_guid":"4ae7465c-1b09-4096-9078-5ed3a9c1b65b","_uuid":"0fa2009a51574867c1a12f9674acf6bd34e25013"},"cell_type":"markdown","source":"***Day***"},{"metadata":{"_cell_guid":"0e7f5ce2-1fba-4a9a-88ae-6d7219b9a5f4","_uuid":"ab749dee94b76c47edbe4bb49a56c18720037e40"},"cell_type":"markdown","source":"# 17. Percentage of Non-Fraudulent Clicks Day wise\n---------------------------------------"},{"metadata":{"_cell_guid":"28ffc8d7-9bf3-482a-9e60-31574bb4f647","_uuid":"8d73845d7b8befaafc863ba7eaa594c1db3ded9f"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"deb2faaf-b540-4da0-9fc8-bd517a6461de","_uuid":"1418cc2a9ab23e8a7939d732e4bbd3a9b4f9c0ed"},"cell_type":"markdown","source":"***Hour of the Day***"},{"metadata":{"_cell_guid":"d6e87168-84e7-42cf-a5e7-2415fe8127cb","_uuid":"726f0077bda13221479c61ce1d6e2c2f26ee9105"},"cell_type":"markdown","source":"# 18. Percentage of Non-Fraudulent Clicks - Hour of the day\n---------------------------------------"},{"metadata":{"_cell_guid":"feee7bdb-f7e2-4a0f-9c18-927bd07b5d2e","_uuid":"43d5384dfa7bbe54c69bd824fc57f53859ea9bd8"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"72c2effb-64af-4089-be41-4b02316808cb","_uuid":"915bc14d44636ba545e6bea9091fddd0e07050cc"},"cell_type":"markdown","source":"Looks like we can tease out some information by extrating the Day and Hour of the Click Time. So on an average the Non- Fradulent clicks peak at 1 AM and bottoms at 3 PM "},{"metadata":{"_cell_guid":"c370a11a-3254-49f9-8529-5b7061324e12","_uuid":"5c62d092c9db889c27ae33426b85ac5443e9f487"},"cell_type":"markdown","source":"# The following Clusters should help TalkingData to prioritize App/Device/OS/Channel/IP level investigation based on Fradulent Clicks % thresholds\n---------------------------------------"},{"metadata":{"_cell_guid":"830eea50-b1d2-42db-ae47-65664b7804ce","_uuid":"a88469ed5f3c5971eaad58192fcb9aa90ec4f498"},"cell_type":"markdown","source":"# 19. Clustering Channels\n---------------------------------------"},{"metadata":{"_cell_guid":"2befcebf-7d1d-4b4d-9f07-06f6f8d5d6a1","_uuid":"118b8d7c801dce4ff09a68c2d7e2076022d0c687"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"5ca19f43-b497-464c-b7e4-3d292a9a08be","_uuid":"36965d9c5bff3f52046e056c5a0d19eed9d5c227"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"1054da35-3ea4-4d22-8a14-115c6ed30586","_uuid":"3e2c322dc62b915611f7af42ee1df5acf332d412"},"cell_type":"markdown","source":"# 20. Clustering Apps\n---------------------------------------"},{"metadata":{"_cell_guid":"7ff7e4ee-b72d-4814-becc-3734bca9df1e","_uuid":"826d121fb74913e79fb58339e363ae3bdb992448"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"6555f242-b301-4e24-902c-f5b6f2184fca","_uuid":"ae51763dd3480b53ff5c12779592803cf2457ee4"},"cell_type":"markdown","source":"# 21. Clustering Devices\n---------------------------------------"},{"metadata":{"_cell_guid":"f585c13d-6edf-49e5-a0bc-67c084625e83","_uuid":"1d34aa8ec1c660fd19c20ac096e6e78ee94a6c1f"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"b7395601-dc55-4890-a892-16c86cc3fea4","_uuid":"590279d39c96e8e2ffe88afe14b6ae5d5d7bd3f6"},"cell_type":"markdown","source":"# 22. Clustering OS\n---------------------------------------"},{"metadata":{"_cell_guid":"4dd2c459-da24-4dee-a0af-ddd0031c9cd3","_uuid":"1f4dd2b71d3551310c38801f46ed535762f2ed91"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"f6612c13-8099-4050-887b-d82397db5c1b","_uuid":"01c4c18c6161af53a87a0bdf8a90f0c549884e1c"},"cell_type":"markdown","source":"# 23. Clustering IPs\n---------------------------------------"},{"metadata":{"_cell_guid":"f9c50c0d-16e8-4d3c-9c61-9f7593d7b565","_uuid":"c023c64beb988674fd2d563f26676cfecc5ce110"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"b0d7b9af-e86e-4f55-a723-280f2d09c6dd","_uuid":"7db6824ee22c2f3ba1b41e8d26c8e7b82865271c"},"cell_type":"markdown","source":"# 24. Clustering Apps, Devices, Channel, OS\n---------------------------------------"},{"metadata":{"_cell_guid":"04c35e49-6912-4056-8d0b-17ccb257dae6","_uuid":"f5b8bfcfa14f52a0a0179277f076afb60556aaa5"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"f6f8dc4a-8918-43f3-92fe-71edcf6cd541","_uuid":"bd93d2b6cb9c31dacdef33d1ff3d610a5fcdfcc0"},"cell_type":"markdown","source":"**Scatter Plots for Fradulent Vs Non-Fradulent Clicks**"},{"metadata":{"_cell_guid":"b8978605-f8e6-4e46-94fe-b5ba24dc4627","_uuid":"06283660b044b2d15e9f1000f5dc603609cef09c"},"cell_type":"markdown","source":"**Channel**"},{"metadata":{"_cell_guid":"9e703a07-0ad7-4aca-acff-41b6947883e6","_uuid":"2cd1b0032c2f47a8c815030cb8c4adf689463e86"},"cell_type":"markdown","source":"# 25. Scatter Plot of Fraudulent Vs Non-Fraudulent Clicks for all Channels\n---------------------------------------"},{"metadata":{"_cell_guid":"2aee0105-cc3a-4869-8d8b-a1bad5fd1da9","_uuid":"56600b755806677bda913daf326da1c92ba5f5dc"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"d8637627-9918-45c4-a6c6-2330ac8aa6cb","_uuid":"15f0d54f91c53fced240c2d52f98ff3f71644b1f"},"cell_type":"markdown","source":"R - Aquared = 0.001\n\np - value =0.54\n"},{"metadata":{"_cell_guid":"9f3d7c3c-e7a1-455c-9619-3682589d76cc","_uuid":"1d351e7ac39950529f9c8f894223228413368f69"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"1e52c74f-6292-4529-b99a-4ac4890cb1f9","_uuid":"fdd1bef9d71e89737e2b92f7f187d0bd7a1177f2"},"cell_type":"markdown","source":"# 26. Scatter Plot of Clicks(No Download) Vs Clicks (Download) for Hour of the Day\n---------------------------------------"},{"metadata":{"_cell_guid":"232bf204-2b43-4e7a-b57e-4d4ed1709e99","_uuid":"caf7e07aec4de57efa5b556435bf0d3ece080e11"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"ce6d2496-9a61-4616-b760-02b9158dec8e","_uuid":"61021086b91318f87357d46c9e370173df9e4518"},"cell_type":"markdown","source":"R - Aquared = 0.86\n\np - value = < 0.0001"},{"metadata":{"_cell_guid":"cb15349f-c316-4407-8241-dbe5156a1d80","_uuid":"b4bd0e24339650f475986791bcdffc8e69b4e709"},"cell_type":"markdown","source":"# 27. Scatter Plot of Fraudulent Vs Non-Fraudulent Clicks for all IPs\n---------------------------------------"},{"metadata":{"_cell_guid":"117e74aa-3fa7-4035-9d5c-465ea6975f26","_uuid":"b940b486c7605bb8fd51864508523cecc9a1238a"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"c27b1f9e-92c1-4646-9e51-972be0377357","_uuid":"6b40b10bb09295eb8328b1599413c3581ecdece7"},"cell_type":"markdown","source":"R - Aquared = 0.79\n\np - value = < 0.0001"},{"metadata":{"_cell_guid":"6f75f802-662e-47c8-909f-75b94f361501","_uuid":"365a1310c952409fd46497691018c34f858deea5"},"cell_type":"markdown","source":"# 28. Scatter Plot of Fraudulent Vs Non-Fraudulent Clicks for all Devices\n---------------------------------------"},{"metadata":{"_cell_guid":"9b8d96e2-fe18-4f84-aff7-89b03827f039","_uuid":"e2d62202be8b78923a36399e7882fc51b9671280"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"0c29420a-48ad-411a-8488-cc269986906b","_uuid":"1992db9fe1d8cfb626e2a9bb04591e6af7079d25"},"cell_type":"markdown","source":"R - Aquared = 0.90\n\np - value = < 0.0001"},{"metadata":{"_cell_guid":"be9a6667-17db-43d7-af3a-d22bf7b1082e","_uuid":"e0f43c1c6ed9053fc30da48ef3105683e4071315"},"cell_type":"markdown","source":"# 29. Scatter Plot of Fraudulent Vs Non-Fraudulent Clicks for all Apps\n---------------------------------------"},{"metadata":{"_cell_guid":"5b0a85fd-b7bb-4a28-aa98-7bd1a0df8d00","_uuid":"d3f216ee0fa9868fd5fc9f64060a2d5f29dc1463"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"fe468b68-04a4-416d-89f2-467588509f70","_uuid":"c3d591fd31ddb62dd03f547530af3e58a3670d85"},"cell_type":"markdown","source":"R - Aquared = 0.01\n\np - value = < =0.00006\n\n**Many Apps have unusually high number of Fradulent Clicks than Non-Fradulent Clicks**"},{"metadata":{"_cell_guid":"bed81a76-ba30-4864-9a3a-dbec1c0c91ec","_uuid":"c7c5ec850033edc4cda6d389f680e2520a4e1a96"},"cell_type":"markdown","source":"# 30. Scatter Plot of Fraudulent Vs Non-Fraudulent Clicks for all OSs\n---------------------------------------"},{"metadata":{"_cell_guid":"463d18ee-0b39-41c0-8d2b-d0a0acc976ae","_uuid":"8988ac85ae7dca23743f92751a03c8467b12f74c"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"765789e1-815e-451e-bf1f-f85af70ad24f","_uuid":"a6be50e19fb608bcc9665b6d5fe4e12138837666"},"cell_type":"markdown","source":"R - Aquared = 0.68\n\np - value = < 0.0001"},{"metadata":{"_cell_guid":"9242a049-f603-4b4a-ac60-a9e08daea4d2","_uuid":"ee586dc4f72673773e368059dccfce11ba02df07"},"cell_type":"markdown","source":"**The below one is for fun...**"},{"metadata":{"_cell_guid":"60678789-1d07-4ef7-82b1-6517c76a316d","_uuid":"6497817f9831a7acbe7a7152774bbaeef25bbf9e"},"cell_type":"markdown","source":"# 31. Some Fun Visualizations\n---------------------------------------"},{"metadata":{"_cell_guid":"2dc5499a-aa55-4d44-9c6e-b53c1207622b","_uuid":"1dec4b9a4a6076deacfc5674dbfb91addffdb7ca"},"cell_type":"markdown","source":"\n"},{"metadata":{"_cell_guid":"ed29e038-d1a6-4deb-866f-4acedf3fba3f","_uuid":"8292442b8f3662affd16864e1c498c126bf372f6"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"7c96d066-7a4f-4505-a5d9-0fdd710374c0","_uuid":"34942c30f008d454ad33d864c522502e0aa28477"},"cell_type":"markdown","source":"**To be continued ...........**"},{"metadata":{"_cell_guid":"99738ee1-a561-4cf1-ad4b-cab94c0eca1a","_uuid":"617ee86e041f8d7d0704b3094a9ec10d3720a80f"},"cell_type":"markdown","source":""},{"metadata":{"_cell_guid":"f8ec00b9-1426-49a4-9f8b-8f4308c56913","_uuid":"edc8f7f30f5b826ab257d0f4cf97de9d712f3771"},"cell_type":"markdown","source":""}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}