{"cells":[{"metadata":{"_cell_guid":"2eaae793-a307-4baa-94bd-f9b7a7fe6494","_uuid":"481326de8bf3e89ba912fd1f01f7ae9c46dba768"},"cell_type":"markdown","source":"# Conversion Prediction #\n\nGoal of this analysis is to predict conversion - a user downloading an app after clicking a mobile app ad. \n\nEven though this challenge is titled \"Fraud Detection\", the data does not identify \"gold standard\" fraudulent clicks to support a true fraud analysis. This distinction - whether we are doing conversion prediction or fraud detection - could be of significance during feature engineering. For a conversion prediction, what intuitively makes a good predictor is app popularity (expect a long tail). For a fraud detection, that would be something indicating abnormal behaviors, such as the number of clicks within a short time period from the same type of device with the same IP address. Both types of features will be Incorporated in this analysis. Having planned that, I still believe it is desirable business acumen to attempt only one goal with one analysis. Ad channel implementation then combines findings from multiple analyses with various purposes. \n\nConversion is expected to be low even if there were not any fraud. That 90% of TalkingData's daily clicks are potentially fraudulent makes the event even more rare. Potential approaches include resampling and ensemble techniques. I will first look at how rare the event is. In case of extreme unbalancy, over-sampling and under-sampling would not seem appropriate. I will build simple random forest and gradient boosting models to quickly look at feature importance. "},{"metadata":{"_cell_guid":"4d45745c-ff38-45e1-96f0-a9b30bdd503b","_uuid":"26c1cdd0747674416e062ce030156b7995b56fab"},"cell_type":"markdown","source":"## 1. Load data ##"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":false,"collapsed":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport gc # Garbage Collector","execution_count":1,"outputs":[]},{"metadata":{"_cell_guid":"a36c8806-61a4-4e6b-b64b-535682babfe3","_uuid":"17fcac1ce72dc28d78153bdb5ee20ae595e7e922"},"cell_type":"markdown","source":"Load the provided subset of training data. Loading the entire training dataset here would lead to kernel death!"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","scrolled":true,"trusted":false,"collapsed":true},"cell_type":"code","source":"df_train = pd.read_csv(\"../input/train_sample.csv\", parse_dates=['click_time', 'attributed_time'])\nprint('Data frame column types:')\nprint(df_train.dtypes)\nprint(\"\\n\")\nprint('Glimpse:')\nprint(df_train.head())","execution_count":2,"outputs":[]},{"metadata":{"_cell_guid":"a230cb94-f96f-4cde-a690-7df8206bb474","_uuid":"5ec074b4c6759509cc1a97cb2b53e81bde6e0b41","trusted":false,"collapsed":true},"cell_type":"code","source":"df_test = pd.read_csv(\"../input/test.csv\", parse_dates=['click_time'])","execution_count":4,"outputs":[]},{"metadata":{"_cell_guid":"5dd05ab6-997d-453a-8661-2c5f185d8766","_uuid":"4d2db365752b67e91fb9f9cd6350a352b1174ca1"},"cell_type":"markdown","source":"## 2. Look at data and create features ##"},{"metadata":{"_cell_guid":"5d4c008a-cac0-4831-bbfd-697d7551adc7","_uuid":"4bd31c9bb1de43bb438e297ae41ab4bb1d3efde8"},"cell_type":"markdown","source":"### 2.1 Check missing ###"},{"metadata":{"_cell_guid":"f9d20fc5-de8d-42de-8535-ee36cf35d2a5","_uuid":"ad9a696ebc12de1684f734cb92e74e67be0763d2","trusted":false},"cell_type":"code","source":"print(f'Data has no missing value? {df_train.isnull().values.any()}')","execution_count":6,"outputs":[]},{"metadata":{"_cell_guid":"b1915ec4-145c-40a9-ba27-9c80cb3c15ab","_uuid":"09fca3bcdbd99a25838f87b738e80f627f1d98c5"},"cell_type":"markdown","source":"### 2.2 Conversion rate ###"},{"metadata":{"_cell_guid":"8d15426e-f985-4765-bf05-36268e1cca87","_uuid":"2701e55c8ed20789d7fd36ed79fb2cd697e6c5aa"},"cell_type":"markdown","source":"How rare is the event?"},{"metadata":{"_cell_guid":"6b0c955c-9b43-4aed-9995-501ffda03f44","_uuid":"354163b33944da55b09cfac55799776fed292bfc","trusted":false},"cell_type":"code","source":"print('app download frequency (0 - no, 1 - yes):')\nprint(df_train['is_attributed'].value_counts())\nprint('percentage:')\nprint(df_train['is_attributed'].value_counts(normalize=True))","execution_count":7,"outputs":[]},{"metadata":{"_cell_guid":"3564af3c-094a-4f37-9568-8f2088c402dc","_uuid":"15dd4a84f0d9600fd9e47cc592819ff296fb6142"},"cell_type":"markdown","source":"Conversion rate is extremely low!"},{"metadata":{"_cell_guid":"f1ad675f-2676-4808-a3c9-5f2de8f7fbd3","_uuid":"24653a890059043d0da18c212b716dbcea69848d"},"cell_type":"markdown","source":"### 2.3 Build time features ###"},{"metadata":{"_cell_guid":"b8524a76-dd3b-4b93-879b-fe8f415b8ebe","_uuid":"64b691440d706e5c78adbac138eba27de9a8e5b5"},"cell_type":"markdown","source":"The overview states that the clicks are over 4 days. Extract day, hour, minute, and second and discard year/month. "},{"metadata":{"_cell_guid":"9100f7b1-4a99-4d01-9963-cba002590ead","_uuid":"dcaf03cbb6de51e3aa3ac3f10c07968372fca413","trusted":false,"collapsed":true},"cell_type":"code","source":"def handle_time(df_name):\n    df_name['click_day'] = df_name['click_time'].dt.day\n    df_name['click_hour'] = df_name['click_time'].dt.hour\n#     df_train['click_minute'] = df_train['click_time'].dt.minute\n#     df_train['click_second'] = df_train['click_time'].dt.second\n    return df_name\n\ndf_train = handle_time(df_train)\ndf_test = handle_time(df_test)","execution_count":8,"outputs":[]},{"metadata":{"_cell_guid":"9617e802-fed0-4835-9fab-c4abc264d74e","_uuid":"e40a9f1422cd90aeee5af65e3a0cf6f2b27c76b9"},"cell_type":"markdown","source":"### 2.4 Split training and testing datasets ###"},{"metadata":{"_cell_guid":"49b5af00-7298-4c5c-8fec-8897db2e9d3b","_uuid":"c554f072d824cf26f17c6765ccec8a956fca7957","trusted":false},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX_train, X_validation, y_train, y_validation = train_test_split(df_train.drop(['attributed_time', 'click_time'], axis=1), \\\n                                                                df_train['is_attributed'], test_size=0.33, random_state=0)\n\ndel df_train\ngc.collect()","execution_count":9,"outputs":[]},{"metadata":{"_cell_guid":"a8d1fccd-08c8-4493-a0d1-dfc5cdd453ca","_uuid":"cc90ef0e717c82185eb6aedde235b77cb6ac58e5"},"cell_type":"markdown","source":"### 2.5 Conversion-oriented features ###"},{"metadata":{"_cell_guid":"baa8389d-59f8-47ff-9219-ddd2268e4f24","_uuid":"cdff5bf9a9d3d036d45445dce145ab3756289172"},"cell_type":"markdown","source":"My hypothesis is that, assuming no fraud, some apps on some oses/devices are more popular than others and naturally will get more downloads. Some marketing channels are more successful than others. Your IP address alone probably does not contribute much to your download decision. From the platform's and advertising company's perspective, channels may perform better if they are targeted, e.g., show an educational app when detecting an IP from an education institute. Time has been observed to have a relatively high impact on conversion. However, do platforms want to rule out the time effect when testing the effectiveness of channels? \n\nApp/os/device/etc. popularity can be represented as \"conversion rate\": \n$$\\frac{\\text{is_attributed count}}{\\text{click count in that category (or multiple-category combination)}}$$"},{"metadata":{"_cell_guid":"5b88f6bc-873e-4590-b664-4976c916fdb8","_uuid":"58c9d94d16d559ede806dc486a0b567a82ced655","trusted":false},"cell_type":"code","source":"feature_combinations = [        \n    ['app', 'channel'],\n    ['app', 'os'],\n    ['app', 'device'],\n    ['app', 'device', 'os']\n]\n\nfor cols in feature_combinations:\n    calc_df = X_train.groupby(cols)['is_attributed'].apply(lambda x: x.sum() / float(x.count())) \n    calc_df = calc_df.to_frame()\n    calc_df.rename(columns={'is_attributed': '_'.join(cols)+'_conv_rate'}, inplace=True)\n\n    X_train = X_train.join(calc_df, on=cols, how='left', sort=False)\n    X_validation = X_validation.join(calc_df, on=cols, how='left', sort=False)\n    X_validation.fillna(0, inplace=True)\n\n    df_test = df_test.join(calc_df, on=cols, how='left', sort=False)\n    df_test.fillna(0, inplace=True)\n\n    del calc_df\n\ngc.collect()","execution_count":10,"outputs":[]},{"metadata":{"_cell_guid":"60323080-d059-47b6-af8e-fc5cf71e9598","_uuid":"a17b4e0d6df8c8b15f8282bee3365ffd7fcbf5e8"},"cell_type":"markdown","source":"My training dataset is small so I have many app-channel-etc combinations that exist in the validation and test datasets respectively that are absent from the training data. "},{"metadata":{"_cell_guid":"e4875dcf-4d8e-4223-9f59-a7b9b3d08754","_uuid":"ff86a8602f56b983f473deec3e324d0bb1ee9427"},"cell_type":"markdown","source":"[NanoMathias's kernel](https://www.kaggle.com/nanomathias/feature-engineering-importance-testing) defines a confidence level to the rate above based on the number of views. I should make such adjustment too. "},{"metadata":{"_cell_guid":"3b48ac48-98db-44ac-8312-47548ae7f6a1","_uuid":"44446fb92c337aab50b6d62bcde0678cacabffe6"},"cell_type":"markdown","source":"### 2.6 Fraud-detection-oriented features ###"},{"metadata":{"_cell_guid":"1158cc4e-202c-4f5c-b5a8-1eaca28b35a8","_uuid":"41e1f3a09b42dbe595c20e75ea5169c1bfa7982b"},"cell_type":"markdown","source":"A large total number of clicks from the same IP could potentially be a good indicator for fraud. Frequent clicks within a short time sounds suspicious too but the total clicks may be lower to avoid detection. \nFor fraudsters IP address is easy to change. I assume device and OS are difficult to change. Let's count the number of clicks within the same ip-device-os-etc combinations during a time window. "},{"metadata":{"_cell_guid":"63196d89-0916-420e-b1ba-1b574480525f","_uuid":"dbc4b45940421c59fd90b807226bfbc66b443e0e","trusted":false},"cell_type":"code","source":"# https://www.kaggle.com/nanomathias/feature-engineering-importance-testing. Simplified. Only keep some combinations. \n# Define all the groupby transformations\nclick_aggregations = [\n    # Variance in hour, for ip-app-os\n    # {'groupby': ['ip','app','os'], 'select': 'click_hour', 'agg': 'var'},\n    # Count, for ip-app-os\n    {'groupby': ['ip', 'app', 'os'], 'select': 'click_hour', 'agg': 'count'},\n    # Count, for ip-app-day-hour\n    {'groupby': ['ip','app','click_day','click_hour'], 'select': 'channel', 'agg': 'count'},    \n    # How popular is the app or channel?\n    {'groupby': ['app'], 'select': 'channel', 'agg': 'count'},\n    {'groupby': ['channel'], 'select': 'app', 'agg': 'count'}    \n]\n\n\n# Apply all the groupby transformations\nfor spec in click_aggregations:\n\n    # Name pattern of new feature\n    new_feature = '{}_{}_{}'.format('_'.join(spec['groupby']), spec['agg'], spec['select'])\n\n    # Info\n#     print(\"Grouping by {}, and aggregating {} with {}\".format(\n#         spec['groupby'], spec['select'], spec['agg']\n#     ))\n\n    # Unique list of features to select\n    all_features = list(set(spec['groupby'] + [spec['select']]))\n\n    # Perform the groupby\n    gp = X_train[all_features]. \\\n        groupby(spec['groupby'])[spec['select']]. \\\n        agg(spec['agg']). \\\n        reset_index(). \\\n        rename(columns={spec['select']: new_feature})\n\n    # Merge back to total data frame\n    if 'cumcount' == spec['agg']:\n        X_train[new_feature] = gp[0].values\n        X_validation[new_feature] = gp[0].values\n        X_validation.fillna(0, inplace=True)\n        \n        df_test[new_feature] = gp[0].values\n        df_test.fillna(0, inplace=True)\n    else:\n        X_train = X_train.merge(gp, on=spec['groupby'], how='left')\n        X_validation = X_validation.merge(gp, on=spec['groupby'], how='left')\n        X_validation.fillna(0, inplace=True)\n        \n        df_test = df_test.merge(gp, on=spec['groupby'], how='left')\n        df_test.fillna(0, inplace=True)\n     # Clear memory\n    del gp\n    gc.collect()\n\ngc.collect()","execution_count":11,"outputs":[]},{"metadata":{"_cell_guid":"42db6aaf-b2e8-40b6-9bd1-b9e8d2061074","_uuid":"9e2b2f34a35b4e0246899a42135ea876119b6f8b"},"cell_type":"markdown","source":"### 2.5 Categorical features ###"},{"metadata":{"_cell_guid":"024c6342-2c29-4a65-8062-10a495dd9e10","_uuid":"be8b0c6795452485f5cbc1ac2f2d785897737a53","trusted":false},"cell_type":"code","source":"for col in ['os', 'app', 'device', 'channel']:\n    print(f'Number of unique {col} in training data: {X_train[col].nunique()}')","execution_count":12,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"48a25b4acf4f73e3ab04ede4d88d5b1cdadea76a"},"cell_type":"code","source":"for col in ['os', 'app', 'device', 'channel']:\n    print(f'Number of unique {col} in testing data: {df_test[col].nunique()}')","execution_count":13,"outputs":[]},{"metadata":{"_cell_guid":"2a91874f-c0cd-40fb-8eaa-cd0e5a6288a1","_uuid":"faca1238d26b522fefb93092eb55a745d6f37264"},"cell_type":"markdown","source":"One-hot encoding would be too sparse. Let's apply the hashing trick to these high-cardinality features."},{"metadata":{"_cell_guid":"6f7feef7-ff47-4ec0-8e0f-edd6cb395fa0","_uuid":"4acbde7844db20f2e15d025efd321b53b43c74a6","trusted":false},"cell_type":"code","source":"from sklearn.feature_extraction import FeatureHasher \n\nFH = FeatureHasher(n_features=6, input_type='string') # device will have hash collision\nfor col in ['os', 'app', 'device', 'channel']:\n    newcolnm = col+'_FH'\n    newcolnm = pd.DataFrame(FH.transform(X_train[col].astype('str')).toarray()).add_prefix(col)\n    X_train = X_train.join(newcolnm)\n    X_validation = X_validation.join(newcolnm)\n    X_validation.fillna(0, inplace=True)\n    \n    df_test = df_test.join(newcolnm)\n    df_test.fillna(0, inplace=True)\n    del newcolnm\n    gc.collect()\ndel FH\n    \ngc.collect()\n","execution_count":14,"outputs":[]},{"metadata":{"_cell_guid":"5c44df8a-c976-486f-ab19-2c5db3b74eb5","_uuid":"364275e56be8e00fb04e8451cce1ef39e5b21c51"},"cell_type":"markdown","source":"## 3. Fit Model ##"},{"metadata":{"_cell_guid":"4a3b1bc5-9ff8-4d8a-a35a-9176ba62f44a","_uuid":"57d687ea561065aedc27a9123c3ca81bf8563c75","trusted":false,"collapsed":true},"cell_type":"code","source":"# from h2o.estimators import H2ORandomForestEstimator\nfrom sklearn.ensemble import AdaBoostClassifier, RandomForestClassifier\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score","execution_count":15,"outputs":[]},{"metadata":{"_cell_guid":"09266f47-ae3e-4568-ac6b-78478b6d76de","_uuid":"92539d10b07b11dfdd4bbf7e166d2546a6d8ccbb","trusted":false,"collapsed":true},"cell_type":"code","source":"X_train.drop(['ip', 'app', 'device', 'os', 'channel', 'is_attributed'], axis=1, inplace=True)\nX_validation.drop(['ip', 'app', 'device', 'os', 'channel', 'is_attributed'], axis=1, inplace=True)\n\ndf_test.drop(['ip', 'app', 'device', 'os', 'channel', 'click_time'], axis=1, inplace=True)","execution_count":16,"outputs":[]},{"metadata":{"_cell_guid":"9f8b1187-054e-4159-8d5b-e1502f62eb23","_uuid":"fc006906b3f14e13628045695dca84da408d4f55"},"cell_type":"markdown","source":"### 3.2 Random Forest ###"},{"metadata":{"_cell_guid":"bca1501d-9b40-4ac7-a632-bc73c398c8f4","_uuid":"09884600588c4e370026c8a035a15fa7942bd6aa","trusted":false,"collapsed":true},"cell_type":"code","source":"# # Define model\n# clf_rf = RandomForestClassifier(random_state=0)\n# # Train model\n# clf_rf.fit(X_train, y_train)","execution_count":17,"outputs":[]},{"metadata":{"_cell_guid":"dce5e88c-518c-45f8-bcd5-6a15d403c8f6","_uuid":"16a54cd792e630f886f5535dacb1f31f498f9057"},"cell_type":"markdown","source":"### 3.3 Adaptive Boosting ###"},{"metadata":{"_cell_guid":"3e714ead-fa96-470d-8f4c-bb159201504e","_uuid":"01b233d1b8724b6ccff97f714d45215398e2da7d","trusted":false},"cell_type":"code","source":"clf_adab = AdaBoostClassifier(n_estimators=200, random_state=0)\nclf_adab.fit(X_train, y_train)","execution_count":18,"outputs":[]},{"metadata":{"_cell_guid":"bfd9c3c2-da75-490c-bf4a-0c47c703a91a","_uuid":"40919365242edbc4e0f8a814ad692610096928f1"},"cell_type":"markdown","source":"## 4. Performance ##"},{"metadata":{"_cell_guid":"5fc9c470-6673-4692-bbf3-b67320ad49fe","_uuid":"e9867ef9da12e212748591dd0f0bd48f579f0c83"},"cell_type":"markdown","source":"### 4.1 Random forest ###"},{"metadata":{"_cell_guid":"af989da8-16da-4fee-b5c3-29e048a91db4","_uuid":"aeb576d827e51e76cf266274d49463152c28984d","trusted":false,"collapsed":true},"cell_type":"code","source":"# importances = clf_rf.feature_importances_\n \n# print (\"Random Forest Sorted Feature Importance:\")\n# sorted_feature_importance = sorted(zip(importances, list(X_train)), reverse=True)\n# print (sorted_feature_importance)\n# print ('\\n')\n# print(f'Random Forest AUC: {roc_auc_score(y_validation, clf_rf.predict_proba(X_validation)[:, 1])}')\n\n# del importances","execution_count":19,"outputs":[]},{"metadata":{"_uuid":"1fb0e6d3ddfa04e00dbb0263bab7ff27cab387e6"},"cell_type":"markdown","source":"### 4.2 AdaBoost ###"},{"metadata":{"_uuid":"cb63b90a48f5d2357ced386e74e2cb02ead57a3f"},"cell_type":"markdown","source":"#### 4.2.1 Feature importance ####"},{"metadata":{"_cell_guid":"4a01d3e7-3c18-4ad7-8cd4-b215d338306c","_uuid":"c92365bd6561e5c517c9a7ff8c919e651fb64e5d","trusted":false},"cell_type":"code","source":"importances = clf_adab.feature_importances_\n\nprint (\"AdaBoost Sorted Feature Importance:\")\nsorted_feature_importance = sorted(zip(importances, list(X_train)), reverse=True)\nprint (sorted_feature_importance)","execution_count":20,"outputs":[]},{"metadata":{"_uuid":"b05f0637b5b4450eb33dfca4cdf28599f82cd8ef"},"cell_type":"markdown","source":"#### 4.2.2 AUC ####"},{"metadata":{"trusted":false,"_uuid":"fda0af30db62dcfd476a3de80296289621eddf07"},"cell_type":"code","source":"print(f'AdaBoost AUC: {roc_auc_score(y_validation, clf_adab.predict_proba(X_validation)[:, 1])}')\ndel importances","execution_count":21,"outputs":[]},{"metadata":{"_cell_guid":"cce6de85-4c33-4b53-9443-892c6eeff53f","_uuid":"d7fb451265f8c97f0ecf8a436f554e3465453e7a"},"cell_type":"markdown","source":"## 5. Prediction ##"},{"metadata":{"_cell_guid":"fe482828-eedc-4db4-aeb9-e01b11395098","_uuid":"137e811a0b15d7f7b5a512b9e51b60d940f9c8df","trusted":false},"cell_type":"code","source":"del X_train, y_train, X_validation, y_validation\ngc.collect()","execution_count":22,"outputs":[]},{"metadata":{"trusted":false,"collapsed":true,"_uuid":"875e5d3cfc33c501a50506574b5f27a33e8f4726"},"cell_type":"code","source":"split_size = 20\ntest_df_list = np.array_split(df_test, split_size, axis=0)","execution_count":23,"outputs":[]},{"metadata":{"trusted":false,"collapsed":true,"_uuid":"2d8d52307879cf56c97197d064a324650a7febfd"},"cell_type":"code","source":"submission_df_list = []\nfor i, test_df_chunk in reversed(list(enumerate(test_df_list))):\n    test_df_chunk['is_attributed'] = clf_adab.predict_proba(test_df_chunk.drop('click_id', axis=1))[:, 1]\n    submission_df_list.append(test_df_chunk[['click_id', 'is_attributed']])\n    del test_df_list[i]\n    gc.collect()","execution_count":24,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"4bfa00332689334e96257b9768a6b29b97be3b2b"},"cell_type":"code","source":"del df_test\ngc.collect()","execution_count":26,"outputs":[]},{"metadata":{"trusted":false,"collapsed":true,"_uuid":"79a9bc398247b5a51c7d6c55fb2cbdcdea974c69"},"cell_type":"code","source":"result = pd.concat(submission_df_list)\ndel submission_df_list\ngc.collect()\n\nresult.sort_values(by='click_id', inplace=True)\nresult.to_csv('adaboost_submission.csv', header=True, index=False)","execution_count":27,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.3"}},"nbformat":4,"nbformat_minor":1}