{"cells":[{"metadata":{"_uuid":"08f831d106e6a90c603cae8c267ec8bb2f93ec93"},"cell_type":"markdown","source":"### Problem Statement"},{"metadata":{"_uuid":"3fb2ab3e58562d340045738d5798da4b6b2dbf60"},"cell_type":"markdown","source":"TalkingData, China’s largest independent big data service platform, covers over 70% of active mobile devices nationwide. They handle 3 billion clicks per day, of which 90% are potentially fraudulent. Their current approach to prevent click fraud for app developers is to measure the journey of a user’s click across their portfolio, and flag IP addresses who produce lots of clicks, but never end up installing apps. With this information, they've built an IP blacklist and device blacklist.\n\nWhile successful, they want to always be one step ahead of fraudsters and have turned to the Kaggle community for help in further developing their solution. In their 2nd competition with Kaggle, you’re challenged to build an algorithm that predicts whether a user will download an app after clicking a mobile app ad. To support your modeling, they have provided a generous dataset covering approximately 200 million clicks over 4 days!"},{"metadata":{"_uuid":"54f5ff93ba00fc11446f4e41b88aa75b69956914"},"cell_type":"markdown","source":"### Importing libraries"},{"metadata":{"trusted":false,"_uuid":"14f4a48738aca2107082aed8970eb540aef04aff"},"cell_type":"code","source":"import gc\nimport numpy as np #For numerical computations\nimport pandas as pd #For data wrangling\nimport matplotlib.pyplot as plt #For basic plotting\n%matplotlib inline  ","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"7b716454af4bd948d159e427d9b8b3bc049a3169"},"cell_type":"markdown","source":"### Reading data"},{"metadata":{"trusted":false,"_uuid":"68378f2e062b8c14c3a3bcd067402250e3ed4a3d"},"cell_type":"code","source":"import os\ninputpath = os.listdir(\"../input\")\n#It is important to define the new datatypes becuase of the space each of the default ones occupy\ndefine_dtypes = {\n    'ip':'uint32',\n    'app':'uint16',\n    'device':'uint16',\n    'os':'uint16',\n    'channel':'uint16',\n    'is_attributed':'uint16',\n    'click_id':'uint32'\n}","execution_count":2,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"c0ac868fd69ca866295faeb119124719c93fd919"},"cell_type":"code","source":"train_data = pd.read_csv(inputpath+'train.csv', nrows = 1000000,usecols = ['ip','app','device','os','channel','click_time','is_attributed'], dtype = define_dtypes)\ntest_data = pd.read_csv(inputpath+'test.csv',   usecols = ['ip','app','device','os','channel','click_time','click_id'], dtype = define_dtypes)","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"2c02d6b72e6a8ab98a5fc9a10cfd789f38e182b8"},"cell_type":"markdown","source":"### Exploring data"},{"metadata":{"trusted":false,"_uuid":"4bd6ed025a18a46b92462b9d19395ffa69878bc8"},"cell_type":"code","source":"print(train_data.isnull().sum(axis =0))\nprint(test_data.isnull().sum(axis =0))","execution_count":4,"outputs":[]},{"metadata":{"_uuid":"bd28f3a0a2586564c8b39abbbfddf6427695ce95"},"cell_type":"markdown","source":"Interestingly, there is no missing data in train or test sets. "},{"metadata":{"trusted":false,"_uuid":"7b18f531d50c85d6da58aae351c27c991a2f6971"},"cell_type":"code","source":"print ('The max and the min days for which the data is collected in train dataset is %s %s' %(train_data['click_time'].min(), train_data['click_time'].max()))\nprint ('The max and the min days for which the data is collected in test dataset is %s %s' %(test_data['click_time'].min(), test_data['click_time'].max()))","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"aefa768830ea276fec357774c7c97b1a96038dd4"},"cell_type":"markdown","source":"So, we can see that the train data was collected for 2 days.We can see that the test data is collected for one day. So, looking at the train and test data we can assume that both came from the same distribution."},{"metadata":{"trusted":false,"_uuid":"7511b53bb7746929cd6e9aedfbd53b31d37faaef"},"cell_type":"code","source":"nunique = train_data.nunique(dropna = False)","execution_count":6,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"4b76a72ead524359b06085b28bfda3881a7f1a9e"},"cell_type":"code","source":"train_data['click_day'] = pd.to_datetime(train_data['click_time']).dt.day.astype('uint8')\ntrain_data['click_hour'] = pd.to_datetime(train_data['click_time']).dt.hour.astype('uint8')\ntrain_data['click_minute'] = pd.to_datetime(train_data['click_time']).dt.minute.astype('uint8')\ntrain_data = train_data.drop(['click_time'], axis = 1)","execution_count":7,"outputs":[]},{"metadata":{"_uuid":"3479e24672a27b4da1afa45ec056845eac67fd63"},"cell_type":"markdown","source":"Looking for the distribution of target variable in train dataset:"},{"metadata":{"trusted":false,"_uuid":"0e437cba239be74a3c4b20f06dc2d45345c00c86"},"cell_type":"code","source":"print(train_data['is_attributed'].value_counts())\nplt.hist(train_data['is_attributed']);\nplt.title(\"Histogram of Target variable\")\nplt.xlabel = \"Target variable\"\nplt.ylabel = \"Frequency percentage\"\nplt.show()","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"3c3a95f0e3f8ed33eaa887e56e0175046bef8cbc"},"cell_type":"markdown","source":"We can see that we are dealing with the highly imbalanced data set with 99.81% values falling under non attributed values and 0.18% falling under attributed. I would prefer to use LGBM based on the references below:\n    https://www.kaggle.com/pranav84/lightgbm-fixing-unbalanced-data-lb-0-9680/code \n    Handling imbalanced datasets on : https://www.analyticsvidhya.com/blog/2017/03/imbalanced-classification-problem/    "},{"metadata":{"_uuid":"2cdade0a56cb031a7bc90e7f345ef263b961b288"},"cell_type":"markdown","source":"#### Why LightGBM?"},{"metadata":{"_uuid":"af6f2d6c0d770b88ff3c003c7226481654d01629"},"cell_type":"markdown","source":"LightGBM algorithm provides special support for categorical features. We can simply use categorical_feature parameter to specify the categorical features.\nIt good accuracy when using native categorical features instead of one-hot coding by finding the optimal split of categorical features. Such an optimal split can provide the much better accuracy than one-hot coding solution."},{"metadata":{"_uuid":"a1b87196f43269cbef21651905ffd11a83468b99"},"cell_type":"markdown","source":"In order to find the best features, the plan is to take sample data from train environment, implement lightgbm \nand then use the features that have highest gain. Currently 2 million records from the train dataset beause of the memory limitations."},{"metadata":{"_uuid":"40cf9e3594cc2f4bf4541f3c5f3d56305408ceef"},"cell_type":"markdown","source":"Defining the function to train the lightGBM model with the given parameters:"},{"metadata":{"code_folding":[],"trusted":false,"_uuid":"0f0fbf5954235cb02acd2997f4869e0dad68302d"},"cell_type":"code","source":"def lgb_modelfit_nocv(params, dtrain, dvalid, predictors, target='target', objective='binary', metrics='auc',\n                 feval=None, early_stopping_rounds=20, num_boost_round=3000, verbose_eval=10, categorical_features=None):\n    lgb_params = {\n        'boosting_type': 'gbdt',\n        'objective': objective,\n        'metric':metrics,\n        'learning_rate': 0.01,\n        #'is_unbalance': 'true',  #because training data is unbalance (replaced with scale_pos_weight)\n        'num_leaves': 31,  # we should let it be smaller than 2^(max_depth)\n        'max_depth': -1,  # -1 means no limit\n        'min_child_samples': 20,  # Minimum number of data need in a child(min_data_in_leaf)\n        'max_bin': 255,  # Number of bucketed bin for feature values\n        'subsample': 0.6,  # Subsample ratio of the training instance.\n        'subsample_freq': 0,  # frequence of subsample, <=0 means no enable\n        'colsample_bytree': 0.3,  # Subsample ratio of columns when constructing each tree.\n        'min_child_weight': 5,  # Minimum sum of instance weight(hessian) needed in a child(leaf)\n        'subsample_for_bin': 200000,  # Number of samples for constructing bin\n        'min_split_gain': 0,  # lambda_l1, lambda_l2 and min_gain_to_split to regularization\n        'reg_alpha': 0,  # L1 regularization term on weights\n        'reg_lambda': 0,  # L2 regularization term on weights\n        'nthread': 4,\n        'verbose': 0,\n        'metric':metrics\n    }\n\n    lgb_params.update(params)\n\n    #print(\"preparing validation datasets\")\n\n    xgtrain = lgb.Dataset(dtrain[predictors].values, label=dtrain[target].values,\n                          feature_name=predictors,\n                          categorical_feature=categorical_features\n                          )\n    xgvalid = lgb.Dataset(dvalid[predictors].values, label=dvalid[target].values,\n                          feature_name=predictors,\n                          categorical_feature=categorical_features\n                          )\n\n    evals_results = {}\n\n    model = lgb.train(lgb_params, \n                     xgtrain, \n                     valid_sets=[xgtrain, xgvalid], \n                     valid_names=['train','valid'], \n                     evals_result=evals_results, \n                     num_boost_round=num_boost_round,\n                     early_stopping_rounds=early_stopping_rounds,\n                     verbose_eval=10, \n                     feval=feval)\n\n    n_estimators = model.best_iteration\n    print(\"\\nModel Report\")\n    print(\"n_estimators : \", n_estimators)\n    print(metrics+\":\", evals_results['valid'][metrics][n_estimators-1])\n    return model ","execution_count":9,"outputs":[]},{"metadata":{"_uuid":"f2738b0709153e4ba61ed9cbbfb47d8a33a7923d"},"cell_type":"markdown","source":"### Feature Engineering and Modeling"},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"8f7501450d37faaea265ff6237590469bd8bfca0"},"cell_type":"code","source":"#Importing lightgbm\nimport lightgbm as lgb","execution_count":10,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"dd51ee7215dc8691c89759eba2d62e358f6547e0"},"cell_type":"code","source":"import sklearn as sk #For shuffling data\nsk.utils.shuffle(train_data) #Shuffling train data to split into train and validation sets","execution_count":11,"outputs":[]},{"metadata":{"_uuid":"ee6e25fd7b3c5fc169d32934dc203dc69053bbcf"},"cell_type":"markdown","source":"In order to find the important features for our final model, the model is first evaluated using training and validation set (5% of the training set) by shuffling the training set. "},{"metadata":{"trusted":false,"_uuid":"d6af0bfb2959463415265b8de093c98b395b41c5"},"cell_type":"code","source":"#Fitting the  lgbm model to train dataset in order to find the important features\npredictors = ['ip','app','device','os','channel','click_day','click_hour']\ntarget = 'is_attributed'\ntrain_df, val_df = np.split(train_data, [int(.95*(len(train_data)))]) \nparams = {\n        'learning_rate': 0.15,\n        #'is_unbalance': 'true', # replaced with scale_pos_weight argument\n        'num_leaves': 7,  # 2^max_depth - 1\n        'max_depth': 3,  # -1 means no limit\n        'min_child_samples': 100,  # Minimum number of data need in a child(min_data_in_leaf)\n        'max_bin': 100,  # Number of bucketed bin for feature values\n        'subsample': 0.7,  # Subsample ratio of the training instance.\n        'subsample_freq': 1,  # frequence of subsample, <=0 means no enable\n        'colsample_bytree': 0.9,  # Subsample ratio of columns when constructing each tree.\n        'min_child_weight': 0,  # Minimum sum of instance weight(hessian) needed in a child(leaf)\n        'scale_pos_weight':99 # because training data is extremely unbalanced \n    }\ncheck_model = lgb_modelfit_nocv(params, \n                        train_df, \n                        val_df, \n                        predictors, \n                        target, \n                        objective='binary', \n                        metrics='auc',\n                        early_stopping_rounds=30\n                        #verbose_eval=True, \n                        #num_boost_round=500 \n                        #categorical_features=categorical\n                        )","execution_count":12,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"9f28f3f7ac56b59cadbe9da1f656a79748f0cbb7"},"cell_type":"code","source":"del train_data\ngc.collect()","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"87e2b1b30df837624c488ac08630f5691ff81ab8"},"cell_type":"markdown","source":"Calculating the importance of features in order to find out the interactions between them for final model\n(Ref: https://www.kaggle.com/alijs1/lightgbm-starter-0-33342/code)"},{"metadata":{"trusted":false,"_uuid":"c7d578f74fb82cc40eea038ce00a7eddc0ff08f2"},"cell_type":"code","source":"print(\"Feature gain/importance...\")\ngain = check_model.feature_importance('gain')\nft = pd.DataFrame({'feature':check_model.feature_name(), \n                   'split':check_model.feature_importance('split'), \n                   'gain':100 * gain / gain.sum()}).sort_values('gain', ascending=False)\nprint(ft.head(25))\n\nplt.figure()\nft[['feature','split']].head(25).plot(kind='bar', x='feature', y='split', legend=False, figsize=(10, 20))\n#plt.gcf().savefig('feature_importance.png')\nplt.show()","execution_count":14,"outputs":[]},{"metadata":{"collapsed":true,"_uuid":"7cf62b93df0d8fcd569f8e7636cbc8ad15dcc2e9"},"cell_type":"markdown","source":"Based on the above numbers, we see that the ip, app, channel are the important features. Let's now draw new feature interaciton sbased on these for the final model. The final model uses train and test data. \n\nWe see that the ip, app and channel are the featrues with maximum importance. Eventough ip stands on the top, we do not take ip\ninto consideration. Please note that I have dropped IP address, attributed time and click time columns after utilizing frequency counts from IP addresses and extracting weekday and hour from click time. IP addresses seems to be dynamic or possibly fake so I decided to not to use for model training directly.\nThis is with reference to: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/52374"},{"metadata":{"_uuid":"364fc1241bb0cc8e2025c8458b48aa0f6ca75cb4"},"cell_type":"markdown","source":"Loading more data for modeling:"},{"metadata":{"trusted":false,"_uuid":"fdc7a7d1edca57e710281194adf7467ac7c5742d"},"cell_type":"code","source":"train_data = pd.read_csv(inputpath+'train.csv', nrows = 20000000,usecols = ['ip','app','device','os','channel','click_time','is_attributed'], dtype = define_dtypes)\ntest_data = pd.read_csv(inputpath+'test.csv',   usecols = ['ip','app','device','os','channel','click_time','click_id'], dtype = define_dtypes)","execution_count":15,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"9b85d58e51547bcac8de986d638647c6cd289fb8"},"cell_type":"code","source":"Combined_data = train_data.append(test_data)\ndel train_data\ndel test_data\ngc.collect()\nCombined_data.head()","execution_count":16,"outputs":[]},{"metadata":{"_uuid":"bcc22c0283ebe8f767a582d404b8f7bc6550dd41"},"cell_type":"markdown","source":"Looking thoroughly at the click_time, we notice that the entire data is limited to only a few days and it does not make sense to include it in the analysis.\nInstead we can only focus on the time of the clicks and get rid of the date."},{"metadata":{"trusted":false,"_uuid":"a42016cfd8ee1e97d17524b8f7038befc3cd2acc"},"cell_type":"code","source":"Combined_data['hour'] = pd.to_datetime(Combined_data.click_time).dt.hour.astype('uint8')\nCombined_data['day'] = pd.to_datetime(Combined_data.click_time).dt.day.astype('uint8')\nCombined_data = Combined_data.drop(['click_time'], axis = 1)\ngc.collect()","execution_count":17,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"7647f3cd80936180aece24848f5163dd0083b46e"},"cell_type":"code","source":"print('Grouping Combined data by ip-day-hour combination...')\ngp = Combined_data[['ip','day','hour','channel']].groupby(by=['ip','day','hour'])[['channel']].count().reset_index().rename(index=str, columns={'channel': 'ip_tcount'})\nCombined_data = Combined_data.merge(gp, on=['ip','day','hour'], how='left')\ndel gp\ngc.collect()","execution_count":18,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"4ea722803cdd1c0489fb85e3550bb2e875673030"},"cell_type":"code","source":"print('Grouping Combined data by ip-app combination...')\ngp =Combined_data[['ip', 'app', 'channel']].groupby(by=['ip', 'app'])[['channel']].count().reset_index().rename(index=str, columns={'channel': 'ip_app_count'})\nCombined_data = Combined_data.merge(gp, on=['ip','app'], how='left')\ndel gp\ngc.collect()","execution_count":19,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"12a1eb8aec5c42b90c04d26b4e234d7e2ce5c73e"},"cell_type":"code","source":"print('Grouping Combined data by ip-app-os combination')\ngp =  Combined_data[['ip','app', 'os', 'channel']].groupby(by=['ip', 'app', 'os'])[['channel']].count().reset_index().rename(index=str, columns={'channel': 'ip_app_os_count'})\nCombined_data = Combined_data.merge(gp, on=['ip','app', 'os'], how='left')\ndel gp\ngc.collect()","execution_count":20,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"e0e378163e0e826fc8df2c1185f6de141aa651b8"},"cell_type":"code","source":"#print(\"vars and data type: \")\nCombined_data.info()\nCombined_data['ip_tcount'] = Combined_data['ip_tcount'].astype('uint16')\nCombined_data['ip_app_count'] = Combined_data['ip_app_count'].astype('uint16')\nCombined_data['ip_app_os_count'] = Combined_data['ip_app_os_count'].astype('uint16')","execution_count":21,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"0ae81f4ae3485e5d269addc38efd02c3255ba023"},"cell_type":"code","source":"gc.collect()","execution_count":22,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"5840839a8637f4dc82f6b3a1c351901c7f691aa0"},"cell_type":"code","source":"test_df = Combined_data[Combined_data['click_id'].isnull() ==False] \ntrain_df, val_df = np.split(Combined_data[Combined_data['click_id'].isnull() ==True], [int(.95*(len(Combined_data[Combined_data['click_id'].isnull() ==True])))]) ","execution_count":24,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"1bb2fb86e2d4ecf9a8f88ba653615166c7858de8"},"cell_type":"code","source":"#Checking the counts to make sure the train, test and validaiton splits are right\nprint(Combined_data.shape)\nprint(train_df.shape)\nprint(test_df.shape)\nprint(val_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"55e2593307628a6fb4f4feb09d27fc1b1865226f"},"cell_type":"code","source":"#Fitting the  lgbm final model to train and validation datasets\noutput = 'is_attributed'\npredictors = ['app','device','os', 'channel', 'hour', 'day', 'ip_app_os_count','ip_app_count','ip_tcount']\n#train_df, val_df = np.split(train_data, [int(.95*(len(train_data)))]) #Splitting into train and validation datasets np.split(train_data, [int(.95*(len(train_data)))]) #Splitting into train and validation datasets\ntrain_df, val_df = np.split(Combined_data[Combined_data['click_id'].isnull() ==True], [int(.95*(len(Combined_data[Combined_data['click_id'].isnull() ==True])))]) \n#train_lgb(predictors,output)\nparams = {\n        'learning_rate': 0.15,\n        #'is_unbalance': 'true', # replaced with scale_pos_weight argument\n        'num_leaves': 7,  # 2^max_depth - 1\n        'max_depth': 3,  # -1 means no limit\n        'min_child_samples': 100,  # Minimum number of data need in a child(min_data_in_leaf)\n        'max_bin': 100,  # Number of bucketed bin for feature values\n        'subsample': 0.7,  # Subsample ratio of the training instance.\n        'subsample_freq': 1,  # frequence of subsample, <=0 means no enable\n        'colsample_bytree': 0.9,  # Subsample ratio of columns when constructing each tree.\n        'min_child_weight': 0,  # Minimum sum of instance weight(hessian) needed in a child(leaf)\n        'scale_pos_weight':99 # because training data is extremely unbalanced \n    }\ncheck_model = lgb_modelfit_nocv(params, \n                        train_df, \n                        val_df, \n                        predictors, \n                        target, \n                        objective='binary', \n                        metrics='auc',\n                        early_stopping_rounds=30\n                        #verbose_eval=True, \n                        #num_boost_round=500 \n                        #categorical_features=categorical\n                        )","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"ceaa06330b033e517eff627df19582698e705844"},"cell_type":"code","source":"Submission = pd.DataFrame()\n\n#Submission['click_id'] = test_df['click_id'].astype('int')\nSubmission['is_attributed'] = check_model.predict(test_df[predictors])\n#Submission.head()\nSubmission.to_csv('submission.csv',index=False)\n#print(\"done...\")","execution_count":48,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.0"}},"nbformat":4,"nbformat_minor":1}