Thank you my briliant team, all teams who competed with us, and all people in the competition.
\nand Congrats Danijel, who became grandmaster in this competition!
\nIt's my first Kaggle competition and I'm really happy to get 5th in this competition.
\nMy model is based on simple LightGBM, with lotta feature engineering. I made 2 models, one with 561 features, the other with 695 features. but increasing the number of features from 561 to 695 gave very small score up, so I'll explain only about 561 features model.
\nI posted the '9836_features.txt', which shows the feature name and the importance(num_split).
actually I made more than 30 types of features. Here I show the 6 types which are effective.
\ntime_diff_1_for means the click time difference with next click,
\ntime_diff_1_back means the click time difference with previous click.
\ntime_diff_2_for means the click time difference with next next click.
\nwhen k = 1, 2, it is very effective.
For example, if specific ip has 100 samples and the distribution of the device is
\n2(30 samples), 3(25 samples), 5(20 samples), 7(15 samples), 8(10 samples),
\nnunique_counts is 5, nunique_counts_ratio is 0.02, top_counts is 30, top_counts_ratio is 0.3,
\ntop2_counts is 25, top2_counts_ratio is 0.25, top3_counts is 15, top3_counts_ratio is 0.15.
\ntop_device is 2, top2_device is 3, top3_device is 5, top4_device is 7, top5_device is 8.
sometimes, there are ip or (ip, os, device) that has always similar timedeltas. for example, if specific ip's click log is like this,
\n00:00:00
\n00:00:05
\n00:00:10
\n00:00:15
\nthey always have 5 sec timedeltas.
\nso, I got top_counts, top2_counts, top3_counts from timedeltas.
lagged target encoding.
\nin day1, this features are missing.
\nfor day2, I used target mean in day1.
\nfor day3, I used target mean in day2.
\nfor day4, I used target mean in day3.
I tried many patterns and select features using feature_importances, which were really time consuming. I made more than 7000 features, but most of the experiments failed.
As https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56319 says, Michael's blend NN improved the score and finally got 0.9840 :)
\nI tried very hard to deal with duplicate problem, but finally couldn't find the leak.
\nI was very sad to see this. We could have won this competition if we had noticed it…
\nhttps://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56268
\nAnyway I really enjoyed my first kaggle competition, I'll continue kaggle and want to win the next competition.
Thank you my briliant team, all teams who competed with us, and all people in the competition.
\nand Congrats Danijel, who became grandmaster in this competition!
\nIt's my first Kaggle competition and I'm really happy to get 5th in this competition.
\nMy model is based on simple LightGBM, with lotta feature engineering. I made 2 models, one with 561 features, the other with 695 features. but increasing the number of features from 561 to 695 gave very small score up, so I'll explain only about 561 features model.
\nI posted the '9836_features.txt', which shows the feature name and the importance(num_split).
actually I made more than 30 types of features. Here I show the 6 types which are effective.
\ntime_diff_1_for means the click time difference with next click,
\ntime_diff_1_back means the click time difference with previous click.
\ntime_diff_2_for means the click time difference with next next click.
\nwhen k = 1, 2, it is very effective.
For example, if specific ip has 100 samples and the distribution of the device is
\n2(30 samples), 3(25 samples), 5(20 samples), 7(15 samples), 8(10 samples),
\nnunique_counts is 5, nunique_counts_ratio is 0.02, top_counts is 30, top_counts_ratio is 0.3,
\ntop2_counts is 25, top2_counts_ratio is 0.25, top3_counts is 15, top3_counts_ratio is 0.15.
\ntop_device is 2, top2_device is 3, top3_device is 5, top4_device is 7, top5_device is 8.
sometimes, there are ip or (ip, os, device) that has always similar timedeltas. for example, if specific ip's click log is like this,
\n00:00:00
\n00:00:05
\n00:00:10
\n00:00:15
\nthey always have 5 sec timedeltas.
\nso, I got top_counts, top2_counts, top3_counts from timedeltas.
lagged target encoding.
\nin day1, this features are missing.
\nfor day2, I used target mean in day1.
\nfor day3, I used target mean in day2.
\nfor day4, I used target mean in day3.
I tried many patterns and select features using feature_importances, which were really time consuming. I made more than 7000 features, but most of the experiments failed.
As https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56319 says, Michael's blend NN improved the score and finally got 0.9840 :)
\nI tried very hard to deal with duplicate problem, but finally couldn't find the leak.
\nI was very sad to see this. We could have won this competition if we had noticed it…
\nhttps://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56268
\nAnyway I really enjoyed my first kaggle competition, I'll continue kaggle and want to win the next competition.
If we were to rank people on how hard they tried to win this competition, I am sure you will be ranked #1
\nYou worked day and night. I never saw you going to bed....
\nReally great teammate, hope you will win the next competition!
Thanks pocket! I'm very happy to team-up with you, I learned a lot from you :) I really hope you become grandmaster :)
", "rawMarkdown": "Thanks pocket! I'm very happy to team-up with you, I learned a lot from you :) I really hope you become grandmaster :)", "votes": null }, { "id": "326248", "postDate": "05/09/2018 13:33:59", "content": "wonderful. can you explain what is the meaning of \"tdf_top_counts_ratio\"??
", "rawMarkdown": "wonderful. can you explain what is the meaning of \"tdf_top_counts_ratio\"??", "votes": null }, { "id": "326250", "postDate": "05/09/2018 13:37:04", "content": "Experiments on over 7000 features!!! Impressive! I assume you have a very strong and stable pipeline to work with these amount of features, right?
", "rawMarkdown": "Experiments on over 7000 features!!! Impressive! I assume you have a very strong and stable pipeline to work with these amount of features, right?", "votes": null }, { "id": "326251", "postDate": "05/09/2018 13:37:16", "content": "for example, if specific ip's click log is like this,
\n00:00:00
\n00:00:05
\n00:00:10
\n00:00:15
\n00:00:23
\n00:00:30
\n00:00:35
\nthey have 5 sec timedeltas 4 times. so, tdf_top_counts_ratio can be caluclated by this.
\n4/(7-1) = 0.67
\nThis is tdf_top_counts_ratio.
Thank you for sharing your solution. Just curious,
\n\n\n\n\nI just tried all patterns
\n
means all combination of features? If we have 10 features here, you tried 10C1 + 10C2 + 10C3 ... 10C10? So you made 1022 models? Stepwise worked for me.
", "rawMarkdown": "Thank you for sharing your solution. Just curious, \n\n> I just tried all patterns\n\nmeans all combination of features? If we have 10 features here, you tried 10C1 + 10C2 + 10C3 ... 10C10? So you made 1022 models? Stepwise worked for me.", "votes": null }, { "id": "326256", "postDate": "05/09/2018 13:41:20", "content": "I can only confirm what pocket said. Incredible how much time and energy you dedicated to this competition. You literally tried all the features. You'd have deserved to win. Too bad about the leak, I'm also really upset about this. But still, without you we wouldn't have finished #5. Thank you so much for that! Looking forward to meet you in another competition. Hopefully in the same a team because it will be really hard to beat you.
", "rawMarkdown": "I can only confirm what [pocket][1] said. Incredible how much time and energy you dedicated to this competition. You literally tried all the features. You'd have deserved to win. Too bad about the leak, I'm also really upset about this. But still, without you we wouldn't have finished #5. Thank you so much for that! Looking forward to meet you in another competition. Hopefully in the same a team because it will be really hard to beat you.\n\n\n [1]: https://www.kaggle.com/pocketsuteado", "votes": null }, { "id": "326257", "postDate": "05/09/2018 13:42:06", "content": "I used n1-megamem-96 instance, which have 1.4TB RAM. It's not so expensive because I used preemptive instance.
", "rawMarkdown": "I used n1-megamem-96 instance, which have 1.4TB RAM. It's not so expensive because I used preemptive instance.", "votes": null }, { "id": "326258", "postDate": "05/09/2018 13:43:16", "content": "Thanks for sharing and congrats on the result. You have some interesting features I did not try.
", "rawMarkdown": "Thanks for sharing and congrats on the result. You have some interesting features I did not try.", "votes": null }, { "id": "326259", "postDate": "05/09/2018 13:44:12", "content": "I get! 4 mean：the 5 timedelta have 4 times. 7 - 1 mean all deltas count...
", "rawMarkdown": "I get! 4 mean：the 5 timedelta have 4 times. 7 - 1 mean all deltas count...", "votes": null }, { "id": "326261", "postDate": "05/09/2018 13:45:44", "content": "Thanks for reply! I mean the feature engineering and feature selection pipeline (framework/code). If you can share something about this.
", "rawMarkdown": "Thanks for reply! I mean the feature engineering and feature selection pipeline (framework/code). If you can share something about this.", "votes": null }, { "id": "326262", "postDate": "05/09/2018 13:46:25", "content": "No, It's impossible :( It means, For example, time_diff_k_for has 2^5 -1 = 31 patterns because there are (ip, os, device, app, channel). and nunique_counts_ratio has 180 patterns. I added all of them and select by feature_importances. I didn't do accurate feature selection, did very rough one.
", "rawMarkdown": "No, It's impossible :( It means, For example, time_diff_k_for has 2^5 -1 = 31 patterns because there are (ip, os, device, app, channel). and nunique_counts_ratio has 180 patterns. I added all of them and select by feature_importances. I didn't do accurate feature selection, did very rough one.", "votes": null }, { "id": "326263", "postDate": "05/09/2018 13:47:37", "content": "what is the means of \"back_10_mean_level_1_ip_os_device\"??
", "rawMarkdown": "what is the means of \"back_10_mean_level_1_ip_os_device\"??", "votes": null }, { "id": "326265", "postDate": "05/09/2018 13:47:59", "content": "I did nothing special, just used .npy file to store features and load it and concat them, which took much time. but I didn't want to use DataFrame to avoid memory explodes...
", "rawMarkdown": "I did nothing special, just used .npy file to store features and load it and concat them, which took much time. but I didn't want to use DataFrame to avoid memory explodes...", "votes": null }, { "id": "326267", "postDate": "05/09/2018 13:50:15", "content": "Ah I completely forgot this. this is the rolling mean of timedeltas of 10 samples before the click.
", "rawMarkdown": "Ah I completely forgot this. this is the rolling mean of timedeltas of 10 samples before the click.", "votes": null }, { "id": "326269", "postDate": "05/09/2018 13:52:39", "content": "Thanks Danijel, now you are grandmaster. :) I think I can't improve my model to 0.9836 without the discussion with you. I'm really looking forward to see you in another competition, and hopefully I want to see you in the real world :)
", "rawMarkdown": "Thanks Danijel, now you are grandmaster. :) I think I can't improve my model to 0.9836 without the discussion with you. I'm really looking forward to see you in another competition, and hopefully I want to see you in the real world :)", "votes": null }, { "id": "326271", "postDate": "05/09/2018 13:54:02", "content": "thanks
", "rawMarkdown": "thanks", "votes": null }, { "id": "326273", "postDate": "05/09/2018 13:55:25", "content": "Thanks CPMP, your information in the discussion/kernel were really valuable and I was always impressed with you :)
", "rawMarkdown": "Thanks CPMP, your information in the discussion/kernel were really valuable and I was always impressed with you :)", "votes": null }, { "id": "326284", "postDate": "05/09/2018 14:05:46", "content": "Thank you mamasingks for these feedbacks and hints. For noob like me it is highly valuable to read feedbacks from top kagglers. All the best to you
", "rawMarkdown": "Thank you mamasingks for these feedbacks and hints. For noob like me it is highly valuable to read feedbacks from top kagglers. All the best to you", "votes": null }, { "id": "326285", "postDate": "05/09/2018 14:06:58", "content": "Agree with you that CPMP comments and posts are very valuable.
", "rawMarkdown": "Agree with you that CPMP comments and posts are very valuable.", "votes": null }, { "id": "326556", "postDate": "05/09/2018 22:55:47", "content": "I believe some of features make your AUC worse. We can not use feature importance for selecting features at these kind of competition. Because datetime is different.
", "rawMarkdown": "I believe some of features make your AUC worse. We can not use feature importance for selecting features at these kind of competition. Because datetime is different.", "votes": null }, { "id": "326627", "postDate": "05/10/2018 04:38:44", "content": "I used feature importances to get the effective subset of the specific features group. For example, there are 180 patterns of \"nunique_counts_ratio\", but I didn't use all of them because if I use all of them training time becomes very long but the gain is small.
\nIn contrast, I didn't use feature_importance but used valid score to check if the specific features group is effective or not. For example, I made \"lagged_ca_diff_mean\", which means the average of difference between click_time and attributed_time in the previous day. Actually some of them have high feature_importance but the valid score doesn't improve, so I threw away all of them. I think it's very important to use both valid score and feature importances when selecting features.
Fair enough! I could understand your solution!
", "rawMarkdown": "Fair enough! I could understand your solution!", "votes": null }, { "id": "326951", "postDate": "05/10/2018 15:03:20", "content": "Hi mamasingkgs, how did you choose the depth or num_leaves in GBM, since you used over 500 features? In most of the public kernels(including mine), they set depth as 3-5 using only about 30 features.
", "rawMarkdown": "Hi mamasingkgs, how did you choose the depth or num_leaves in GBM, since you used over 500 features? In most of the public kernels(including mine), they set depth as 3-5 using only about 30 features.", "votes": null }, { "id": "326956", "postDate": "05/10/2018 15:10:04", "content": "This is my parameter.\nparams = {\n 'boosting_type': 'gbdt',\n 'objective': 'binary',\n 'tree_learner': 'serial',\n 'metric': 'binary_logloss',\n 'learning_rate': 0.05,\n 'max_bin': 255,\n 'num_leaves': 63,\n 'max_depth': -1, \n 'min_data_in_leaf': 1000, \n 'feature_fraction': 0.7, \n 'bagging_freq': 1,\n 'bagging_fraction': 0.7,\n 'lambda_l1': 1, \n 'lambda_l2': 1, \n 'sigmoid': 1,\n 'verbosity': 0, \n}
", "rawMarkdown": "This is my parameter.\nparams = {\n 'boosting_type': 'gbdt',\n 'objective': 'binary',\n 'tree_learner': 'serial',\n 'metric': 'binary_logloss',\n 'learning_rate': 0.05,\n 'max_bin': 255,\n 'num_leaves': 63,\n 'max_depth': -1, \n 'min_data_in_leaf': 1000, \n 'feature_fraction': 0.7, \n 'bagging_freq': 1,\n 'bagging_fraction': 0.7,\n 'lambda_l1': 1, \n 'lambda_l2': 1, \n 'sigmoid': 1,\n 'verbosity': 0, \n}", "votes": null }, { "id": "326964", "postDate": "05/10/2018 15:20:15", "content": "Thanks for sharing!!
", "rawMarkdown": "Thanks for sharing!!", "votes": null }, { "id": "327741", "postDate": "05/12/2018 10:28:39", "content": "I'm confused by lagged_dl_counts_ratio. in day1, this features are missing. If they are missing, is it good for training?
It doesn't give bad effect for me. If there is no sample or no positive sample in previous day, I set the value 0. so, there are no NaN samples in day2, day3, day4 and no problem happened.
", "rawMarkdown": "It doesn't give bad effect for me. If there is no sample or no positive sample in previous day, I set the value 0. so, there are no NaN samples in day2, day3, day4 and no problem happened.", "votes": null }, { "id": "329920", "postDate": "05/17/2018 15:03:24", "content": "What a amazing feature engineering!
", "rawMarkdown": "What a amazing feature engineering!", "votes": null } ], "comments": [ { "id": 326231, "author_name": "pocketsuteado", "author_url": "", "post_date": "05/09/2018 13:20:15", "content": "If we were to rank people on how hard they tried to win this competition, I am sure you will be ranked #1
\nYou worked day and night. I never saw you going to bed....
\nReally great teammate, hope you will win the next competition!
Thanks pocket! I'm very happy to team-up with you, I learned a lot from you :) I really hope you become grandmaster :)
", "votes": null, "replies": [] } ] }, { "id": 326248, "author_name": "huayupeng", "author_url": "", "post_date": "05/09/2018 13:33:59", "content": "wonderful. can you explain what is the meaning of \"tdf_top_counts_ratio\"??
", "votes": null, "replies": [ { "id": 326251, "author_name": "mamasinkgs", "author_url": "", "post_date": "05/09/2018 13:37:16", "content": "for example, if specific ip's click log is like this,
\n00:00:00
\n00:00:05
\n00:00:10
\n00:00:15
\n00:00:23
\n00:00:30
\n00:00:35
\nthey have 5 sec timedeltas 4 times. so, tdf_top_counts_ratio can be caluclated by this.
\n4/(7-1) = 0.67
\nThis is tdf_top_counts_ratio.
I get! 4 mean：the 5 timedelta have 4 times. 7 - 1 mean all deltas count...
", "votes": null, "replies": [] } ] }, { "id": 326250, "author_name": "cczaixian", "author_url": "", "post_date": "05/09/2018 13:37:04", "content": "Experiments on over 7000 features!!! Impressive! I assume you have a very strong and stable pipeline to work with these amount of features, right?
", "votes": null, "replies": [ { "id": 326257, "author_name": "mamasinkgs", "author_url": "", "post_date": "05/09/2018 13:42:06", "content": "I used n1-megamem-96 instance, which have 1.4TB RAM. It's not so expensive because I used preemptive instance.
", "votes": null, "replies": [] }, { "id": 326261, "author_name": "cczaixian", "author_url": "", "post_date": "05/09/2018 13:45:44", "content": "Thanks for reply! I mean the feature engineering and feature selection pipeline (framework/code). If you can share something about this.
", "votes": null, "replies": [] }, { "id": 326265, "author_name": "mamasinkgs", "author_url": "", "post_date": "05/09/2018 13:47:59", "content": "I did nothing special, just used .npy file to store features and load it and concat them, which took much time. but I didn't want to use DataFrame to avoid memory explodes...
", "votes": null, "replies": [] } ] }, { "id": 326254, "author_name": "onodera", "author_url": "", "post_date": "05/09/2018 13:40:21", "content": "Thank you for sharing your solution. Just curious,
\n\n\n\n\nI just tried all patterns
\n
means all combination of features? If we have 10 features here, you tried 10C1 + 10C2 + 10C3 ... 10C10? So you made 1022 models? Stepwise worked for me.
", "votes": null, "replies": [ { "id": 326262, "author_name": "mamasinkgs", "author_url": "", "post_date": "05/09/2018 13:46:25", "content": "No, It's impossible :( It means, For example, time_diff_k_for has 2^5 -1 = 31 patterns because there are (ip, os, device, app, channel). and nunique_counts_ratio has 180 patterns. I added all of them and select by feature_importances. I didn't do accurate feature selection, did very rough one.
", "votes": null, "replies": [] }, { "id": 326556, "author_name": "onodera", "author_url": "", "post_date": "05/09/2018 22:55:47", "content": "I believe some of features make your AUC worse. We can not use feature importance for selecting features at these kind of competition. Because datetime is different.
", "votes": null, "replies": [] }, { "id": 326627, "author_name": "mamasinkgs", "author_url": "", "post_date": "05/10/2018 04:38:44", "content": "I used feature importances to get the effective subset of the specific features group. For example, there are 180 patterns of \"nunique_counts_ratio\", but I didn't use all of them because if I use all of them training time becomes very long but the gain is small.
\nIn contrast, I didn't use feature_importance but used valid score to check if the specific features group is effective or not. For example, I made \"lagged_ca_diff_mean\", which means the average of difference between click_time and attributed_time in the previous day. Actually some of them have high feature_importance but the valid score doesn't improve, so I threw away all of them. I think it's very important to use both valid score and feature importances when selecting features.
Fair enough! I could understand your solution!
", "votes": null, "replies": [] } ] }, { "id": 326256, "author_name": "danijelk", "author_url": "", "post_date": "05/09/2018 13:41:20", "content": "I can only confirm what pocket said. Incredible how much time and energy you dedicated to this competition. You literally tried all the features. You'd have deserved to win. Too bad about the leak, I'm also really upset about this. But still, without you we wouldn't have finished #5. Thank you so much for that! Looking forward to meet you in another competition. Hopefully in the same a team because it will be really hard to beat you.
", "votes": null, "replies": [ { "id": 326269, "author_name": "mamasinkgs", "author_url": "", "post_date": "05/09/2018 13:52:39", "content": "Thanks Danijel, now you are grandmaster. :) I think I can't improve my model to 0.9836 without the discussion with you. I'm really looking forward to see you in another competition, and hopefully I want to see you in the real world :)
", "votes": null, "replies": [] } ] }, { "id": 326258, "author_name": "cpmpml", "author_url": "", "post_date": "05/09/2018 13:43:16", "content": "Thanks for sharing and congrats on the result. You have some interesting features I did not try.
", "votes": null, "replies": [ { "id": 326273, "author_name": "mamasinkgs", "author_url": "", "post_date": "05/09/2018 13:55:25", "content": "Thanks CPMP, your information in the discussion/kernel were really valuable and I was always impressed with you :)
", "votes": null, "replies": [] }, { "id": 326285, "author_name": "ericbenhamou", "author_url": "", "post_date": "05/09/2018 14:06:58", "content": "Agree with you that CPMP comments and posts are very valuable.
", "votes": null, "replies": [] } ] }, { "id": 326263, "author_name": "huayupeng", "author_url": "", "post_date": "05/09/2018 13:47:37", "content": "what is the means of \"back_10_mean_level_1_ip_os_device\"??
", "votes": null, "replies": [ { "id": 326267, "author_name": "mamasinkgs", "author_url": "", "post_date": "05/09/2018 13:50:15", "content": "Ah I completely forgot this. this is the rolling mean of timedeltas of 10 samples before the click.
", "votes": null, "replies": [] } ] }, { "id": 326271, "author_name": "xiaohuihui", "author_url": "", "post_date": "05/09/2018 13:54:02", "content": "thanks
", "votes": null, "replies": [] }, { "id": 326284, "author_name": "ericbenhamou", "author_url": "", "post_date": "05/09/2018 14:05:46", "content": "Thank you mamasingks for these feedbacks and hints. For noob like me it is highly valuable to read feedbacks from top kagglers. All the best to you
", "votes": null, "replies": [] }, { "id": 326951, "author_name": "fangao", "author_url": "", "post_date": "05/10/2018 15:03:20", "content": "Hi mamasingkgs, how did you choose the depth or num_leaves in GBM, since you used over 500 features? In most of the public kernels(including mine), they set depth as 3-5 using only about 30 features.
", "votes": null, "replies": [ { "id": 326956, "author_name": "mamasinkgs", "author_url": "", "post_date": "05/10/2018 15:10:04", "content": "This is my parameter.\nparams = {\n 'boosting_type': 'gbdt',\n 'objective': 'binary',\n 'tree_learner': 'serial',\n 'metric': 'binary_logloss',\n 'learning_rate': 0.05,\n 'max_bin': 255,\n 'num_leaves': 63,\n 'max_depth': -1, \n 'min_data_in_leaf': 1000, \n 'feature_fraction': 0.7, \n 'bagging_freq': 1,\n 'bagging_fraction': 0.7,\n 'lambda_l1': 1, \n 'lambda_l2': 1, \n 'sigmoid': 1,\n 'verbosity': 0, \n}
", "votes": null, "replies": [] }, { "id": 326964, "author_name": "fangao", "author_url": "", "post_date": "05/10/2018 15:20:15", "content": "Thanks for sharing!!
", "votes": null, "replies": [] } ] }, { "id": 327741, "author_name": "shenchengguang", "author_url": "", "post_date": "05/12/2018 10:28:39", "content": "I'm confused by lagged_dl_counts_ratio. in day1, this features are missing. If they are missing, is it good for training?
It doesn't give bad effect for me. If there is no sample or no positive sample in previous day, I set the value 0. so, there are no NaN samples in day2, day3, day4 and no problem happened.
", "votes": null, "replies": [] } ] }, { "id": 329920, "author_name": "shawnyxiao", "author_url": "", "post_date": "05/17/2018 15:03:24", "content": "What a amazing feature engineering!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "326225": "Thank you my briliant team, all teams who competed with us, and all people in the competition. \nand Congrats Danijel, who became grandmaster in this competition!