We'd like to thank TalkingData and Kaggle for organizing this exciting competition. This competition gave us a fantastic opportunity to learn how to deal with very large table data.
\n
\nHere is our solution.
\n
\nStrategy:
\nOur solution heavily depends on negative down-sampling [1, 2], which means we use all positive examples (i.e., is_attributed == 1) and down-sampled negative examples on model training. We down-sampled negative examples such that their size becomes equal to the number of positive ones. It discards about 99.8% of negative examples, but we didn't see much performance deterioration when we tested with our initial features. Moreover, we could get better performance when creating a submission by bagging five predictors trained on five sampled datasets created from different random seeds. This technique allowed us to use hundreds of features while keeping LGB training time less than 30 minutes.
\n
\nFeatures:
\nFirst, we started from features from Kernels. On Thanks pranav84 and other Kagglers for sharing their awesome insights! We did feature engineerings using all the data examples instead of the down-sampled ones.
\n- five raw categorical features (ip, os, app, channel, device)
\n- time categorical features (day, hour)
\n- some count features
\nThen, we created a bunch of features in a brute-force way. For each combination of five raw categorical features (ip, os, app, channel, and device), we created the following click series-based feature sets (i.e., each feature set consists of 31 (=(2^5) - 1) features):
\n- click count within next one/six hours
\n- forward/backward click time delta
\n- average attributed ratio of past clicks
\nWe didn't do feature selection. We just added all of them to our model. At that point, our LGB model's score was 0.9808.
\n
\nNext, we tried categorical feature embedding by using LDA/NMF/LSA. Here is the pseudo code to compute LDA topics of IPs related to app. (LDA is latent Dirichlet allocation)
apps_of_ip = {}\nfor sample in data_samples:\n apps_of_ip.setdefault(sample['ip'], []).append(str(sample['app']))\nips = list(apps_of_ip.keys())\napps_as_sentence = [' '.join(apps_of_ip[ip]) for ip in ips]\napps_as_matrix = CountTokenizer().fit_transform(apps_as_sentence)\ntopics_of_ips = LDA(n_components=5).fit_transform(apps_as_matrix)\n\n\nWe computed this feature for all the 20 (=5*(5-1)) combinations of 5 raw features and set the topic size to 5. This ended up with 100 new features. We also computed similar features using NMF and PCA, in total 300 new features. 0.9821 with a single LGB.
\n
\nAfter that, we removed all raw categorical features except app since we supposed embedding features cover information available from them. Surprisingly, this minor change made our public LB score jump up from 0.9821 to 0.9828. Actually, we don't know what causes this significant score improvement.
\nBesides features mentioned here, we created higher dimensional LDA features and features that try to address the duplicate sample problem. These features somewhat improve our public LB score.
\n
\nModels:
\nWe used day 7 & 8 for training and day 9 for validation, and chose the best number of iterations of LGB. Then, we trained a model on day 7 & 8 & 9 with the obtained number of iterations for creating submission. After we finished feature engineering, flowlight's five-bagged LGB model reached 0.98333 on public LB (and 0.98420 on private LB), which was trained on 646 features. As far as we remember, the memory usage for training this model was less than 100GB (<64GB will be possible with minor code modification).
\nI implemented a simple three layer NN model as some kernels do. It scored worse than LGB models by 0.0013 points with 0.005 down-sampling rate at first. Then, I realized it should be trained with more negative data samples, probably. However, we didn't afford to use so many examples because of the massive amount of our features (and my bad implementation, disk space, no GPU, close deadline, etc...). The final three-bagged NN model scored 0.98258 on public LB.
\nWe made our final submission with a rank-based weighted averaging. It is composed of seven bagged LGB models and a single bagged NN It scored 0.98343 on public LB.
\n
\n
\n[1] Practical Lessons from Predicting Clicks on Ads at\nFacebook
\n[2] Ad Click Prediction: a View from the Trenches
\n
\nEDIT: Added some extra explanations based on frequently asked questions in comments.
We'd like to thank TalkingData and Kaggle for organizing this exciting competition. This competition gave us a fantastic opportunity to learn how to deal with very large table data.
\n
\nHere is our solution.
\n
\nStrategy:
\nOur solution heavily depends on negative down-sampling [1, 2], which means we use all positive examples (i.e., is_attributed == 1) and down-sampled negative examples on model training. We down-sampled negative examples such that their size becomes equal to the number of positive ones. It discards about 99.8% of negative examples, but we didn't see much performance deterioration when we tested with our initial features. Moreover, we could get better performance when creating a submission by bagging five predictors trained on five sampled datasets created from different random seeds. This technique allowed us to use hundreds of features while keeping LGB training time less than 30 minutes.
\n
\nFeatures:
\nFirst, we started from features from Kernels. On Thanks pranav84 and other Kagglers for sharing their awesome insights! We did feature engineerings using all the data examples instead of the down-sampled ones.
\n- five raw categorical features (ip, os, app, channel, device)
\n- time categorical features (day, hour)
\n- some count features
\nThen, we created a bunch of features in a brute-force way. For each combination of five raw categorical features (ip, os, app, channel, and device), we created the following click series-based feature sets (i.e., each feature set consists of 31 (=(2^5) - 1) features):
\n- click count within next one/six hours
\n- forward/backward click time delta
\n- average attributed ratio of past clicks
\nWe didn't do feature selection. We just added all of them to our model. At that point, our LGB model's score was 0.9808.
\n
\nNext, we tried categorical feature embedding by using LDA/NMF/LSA. Here is the pseudo code to compute LDA topics of IPs related to app. (LDA is latent Dirichlet allocation)
apps_of_ip = {}\nfor sample in data_samples:\n apps_of_ip.setdefault(sample['ip'], []).append(str(sample['app']))\nips = list(apps_of_ip.keys())\napps_as_sentence = [' '.join(apps_of_ip[ip]) for ip in ips]\napps_as_matrix = CountTokenizer().fit_transform(apps_as_sentence)\ntopics_of_ips = LDA(n_components=5).fit_transform(apps_as_matrix)\n\n\nWe computed this feature for all the 20 (=5*(5-1)) combinations of 5 raw features and set the topic size to 5. This ended up with 100 new features. We also computed similar features using NMF and PCA, in total 300 new features. 0.9821 with a single LGB.
\n
\nAfter that, we removed all raw categorical features except app since we supposed embedding features cover information available from them. Surprisingly, this minor change made our public LB score jump up from 0.9821 to 0.9828. Actually, we don't know what causes this significant score improvement.
\nBesides features mentioned here, we created higher dimensional LDA features and features that try to address the duplicate sample problem. These features somewhat improve our public LB score.
\n
\nModels:
\nWe used day 7 & 8 for training and day 9 for validation, and chose the best number of iterations of LGB. Then, we trained a model on day 7 & 8 & 9 with the obtained number of iterations for creating submission. After we finished feature engineering, flowlight's five-bagged LGB model reached 0.98333 on public LB (and 0.98420 on private LB), which was trained on 646 features. As far as we remember, the memory usage for training this model was less than 100GB (<64GB will be possible with minor code modification).
\nI implemented a simple three layer NN model as some kernels do. It scored worse than LGB models by 0.0013 points with 0.005 down-sampling rate at first. Then, I realized it should be trained with more negative data samples, probably. However, we didn't afford to use so many examples because of the massive amount of our features (and my bad implementation, disk space, no GPU, close deadline, etc...). The final three-bagged NN model scored 0.98258 on public LB.
\nWe made our final submission with a rank-based weighted averaging. It is composed of seven bagged LGB models and a single bagged NN It scored 0.98343 on public LB.
\n
\n
\n[1] Practical Lessons from Predicting Clicks on Ads at\nFacebook
\n[2] Ad Click Prediction: a View from the Trenches
\n
\nEDIT: Added some extra explanations based on frequently asked questions in comments.
apps_of_ip = {}\nfor sample in data_samples:\n apps_of_ip.setdefault(sample['ip'], []).append(str(sample['app']))\nips = list(apps_of_ip.keys())\napps_as_sentence = [' '.join(apps_of_ip[ip]) for ip in ips]\napps_as_matrix = CountTokenizer().fit_transform(apps_as_sentence)\ntopics_of_ips = LDA(n_components=5).fit_transform(apps_as_matrix)\n\nWe computed this feature for all the 20 (=5*(5-1)) combinations of 5 raw features and set the topic size to 5. This ended up with 100 new features. We also computed similar features using NMF and PCA, in total 300 new features. 0.9821 with a single LGB. Congrats,a very good solution,your 1st place is well deserved,thanks for sharing. I think if we can combine the top teams' solutions ,we will get a very high score.
", "rawMarkdown": "Congrats,a very good solution,your 1st place is well deserved,thanks for sharing. I think if we can combine the top teams' solutions ,we will get a very high score.", "votes": 9, "replies": [ { "id": 326757, "postDate": "2018-05-10T09:23:19.427Z", "content": "I am not sure I qualify as one of the top teams, but I shared my 0.98379 private solution here. Maybe others could do the same so that we can blend?
", "rawMarkdown": "I am not sure I qualify as one of the top teams, but I shared my 0.98379 private solution [here][1]. Maybe others could do the same so that we can blend?\n\n [1]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56423", "votes": 4 }, { "id": 326761, "postDate": "2018-05-10T09:30:09.453Z", "content": "Absolutely should include you :)
To get top 50 is not so easy in this competition.
I feel the same. I am very impressed to see so many wonderful ideas that I have never thought of.
", "rawMarkdown": "I feel the same. I am very impressed to see so many wonderful ideas that I have never thought of. ", "votes": 4 } ] }, { "id": 326793, "postDate": "2018-05-10T10:55:21.323Z", "content": "Congratulations @ckomaki and @flowlight. Excellent approach and very well deserved win. And a big thank you for the mention :) This actually made my day!
", "rawMarkdown": "Congratulations @ckomaki and @flowlight. Excellent approach and very well deserved win. And a big thank you for the mention :) This actually made my day! ", "votes": 8, "replies": [ { "id": 326932, "postDate": "2018-05-10T14:27:23.777Z", "content": "@Pranav, It is a pity to see how Dirk share impacted you. Most people reused your shared kernels.
", "rawMarkdown": "@Pranav, It is a pity to see how Dirk share impacted you. Most people reused your shared kernels.", "votes": 1 }, { "id": 326946, "postDate": "2018-05-10T14:54:21.077Z", "content": "That's alrite @CPMP. Though Dirk impacted me too but I also blame myself for making stupid selections on judgement day :) But honestly, this competition was really a great learning experience for me specifically in feature engineering part.
\n\nShared learning from community actually helped me a lot and I'm sure that this will contribute positively for me in next competitions :)
", "rawMarkdown": "That's alrite @CPMP. Though Dirk impacted me too but I also blame myself for making stupid selections on judgement day :) But honestly, this competition was really a great learning experience for me specifically in feature engineering part. \n\nShared learning from community actually helped me a lot and I'm sure that this will contribute positively for me in next competitions :) ", "votes": 7 } ] }, { "id": 326714, "postDate": "2018-05-10T08:07:04.547Z", "content": "fantastic solution, thank you very much for sharing - Matrix factorization for feature interaction will be a main take away for me from this competition
", "rawMarkdown": "fantastic solution, thank you very much for sharing - Matrix factorization for feature interaction will be a main take away for me from this competition ", "votes": 3 }, { "id": 326705, "postDate": "2018-05-10T07:49:28.920Z", "content": "I must say that I feel stupid for not having tested downsampling of negative. I usually try it on very imbalanced datasets. Well done!
\n\nAre you computing your lsa models on downsampled data or on the original one?
", "rawMarkdown": "I must say that I feel stupid for not having tested downsampling of negative. I usually try it on very imbalanced datasets. Well done!\n\nAre you computing your lsa models on downsampled data or on the original one?", "votes": 4, "replies": [ { "id": 326729, "postDate": "2018-05-10T08:36:42.033Z", "content": "LSA features were computed on the original data :)
", "rawMarkdown": "LSA features were computed on the original data :) ", "votes": 2 }, { "id": 326731, "postDate": "2018-05-10T08:38:22.307Z", "content": "Thanks,.
\n\nYou almost make me try to rerun my stuff on downsampled data ;)
\n\nReally neat solution, very impressive. The gap between us is well justified.
", "rawMarkdown": "Thanks,.\n\nYou almost make me try to rerun my stuff on downsampled data ;) \n\nReally neat solution, very impressive. The gap between us is well justified.", "votes": 2 }, { "id": 326961, "postDate": "2018-05-10T15:14:50.697Z", "content": "Agree. It's a very neat solution, just like a clever and beautiful solution to a mathematics problem.
", "rawMarkdown": "Agree. It's a very neat solution, just like a clever and beautiful solution to a mathematics problem." }, { "id": 327295, "postDate": "2018-05-11T07:49:58.520Z", "content": "Me too! I feel stupider because I've implemented downsampling of negative 3 weeks ago but forgot to run it until the competition ended. XD
", "rawMarkdown": "Me too! I feel stupider because I've implemented downsampling of negative 3 weeks ago but forgot to run it until the competition ended. XD" } ] }, { "id": 327266, "postDate": "2018-05-11T06:45:08.493Z", "content": "Do you use sklearn.decomposition.LatentDirichletAllocation ? or others? It's very slow. How to train it quickly?
", "rawMarkdown": "Do you use sklearn.decomposition.LatentDirichletAllocation ? or others? It's very slow. How to train it quickly?", "votes": 1, "replies": [ { "id": 328142, "postDate": "2018-05-13T13:18:27.490Z", "content": "Yes, we used sklearn.decomposition.LatentDirichletAllocation. If my memory is correct, it takes about one hour for calculation on an (ip, app) pair. It was acceptable because we stored each feature in a file once we computed it and we don't have to run LDA for each training.
", "rawMarkdown": "Yes, we used sklearn.decomposition.LatentDirichletAllocation. If my memory is correct, it takes about one hour for calculation on an (ip, app) pair. It was acceptable because we stored each feature in a file once we computed it and we don't have to run LDA for each training. " }, { "id": 328154, "postDate": "2018-05-13T13:36:13.703Z", "content": "@xuan Class LDA has a parameter n_jobs. Have you tried it?\n@flowlight. Congrats and thanks for your sharing. Choosing a proper number of components often gave me a headache. I was wondering if there exist an empirical relationship between n_components and the shape (rows and columns) of the matrix. Or could you please tell me why you choose 5 for it? Is it just chosen arbitrarily or tuned?
", "rawMarkdown": "@xuan Class LDA has a parameter n_jobs. Have you tried it?\n@flowlight. Congrats and thanks for your sharing. Choosing a proper number of components often gave me a headache. I was wondering if there exist an empirical relationship between n_components and the shape (rows and columns) of the matrix. Or could you please tell me why you choose 5 for it? Is it just chosen arbitrarily or tuned?" }, { "id": 328199, "postDate": "2018-05-13T16:26:13.883Z", "content": "There are no strong reason why we chose 5 as n_components. Actually, we also added LDA with n_components = 20 before the deadline of this competition.
", "rawMarkdown": "There are no strong reason why we chose 5 as n_components. Actually, we also added LDA with n_components = 20 before the deadline of this competition. " }, { "id": 328319, "postDate": "2018-05-14T01:04:14.247Z", "content": "Thanks a lot for your reply!
", "rawMarkdown": "Thanks a lot for your reply!" }, { "id": 329285, "postDate": "2018-05-16T06:35:09.660Z", "content": "n_jobs = -1, can quikly a lot,I try
", "rawMarkdown": "n_jobs = -1, can quikly a lot,I try" } ] }, { "id": 327033, "postDate": "2018-05-10T17:56:24.310Z", "content": "Congratulations!Thanks for sharing.Also want to know why \"removed all raw categorical features except app\" work....
", "rawMarkdown": "Congratulations!Thanks for sharing.Also want to know why \"removed all raw categorical features except app\" work....", "votes": 1, "replies": [ { "id": 327870, "postDate": "2018-05-12T18:55:02.070Z", "content": "Removing all but app and os worked for me. Don't know why either.
", "rawMarkdown": "Removing all but app and os worked for me. Don't know why either." }, { "id": 327918, "postDate": "2018-05-12T22:23:35.913Z", "content": "My guess:
\n\nThe categorical features are simply labels and the numbers represented by the labels are more or less arbitrary. The model is successfully finding signal in the engineered features, such as count by ip, but the original categorical features are just adding more noise to the model, and when training, the model's learning of the actual relationships between engineered features and is_attributed is being diluted by the categorical features.
\n\nThere might be some information in the categorical features, maybe some of the app ids that have higher assigned numbers are correlated is is_attributed, but that is possibly over fitting?
\n\nFor reference, my model had very few engineered features and I struggled greatly to find any signal.
", "rawMarkdown": "My guess:\n\nThe categorical features are simply labels and the numbers represented by the labels are more or less arbitrary. The model is successfully finding signal in the engineered features, such as count by ip, but the original categorical features are just adding more noise to the model, and when training, the model's learning of the actual relationships between engineered features and is_attributed is being diluted by the categorical features. \n\nThere might be some information in the categorical features, maybe some of the app ids that have higher assigned numbers are correlated is is_attributed, but that is possibly over fitting?\n\nFor reference, my model had very few engineered features and I struggled greatly to find any signal. " }, { "id": 327977, "postDate": "2018-05-13T04:07:21.703Z", "content": "When you use a feature as a categorical in LightGBM the ordering of the numbers does not matter.
", "rawMarkdown": "When you use a feature as a categorical in LightGBM the ordering of the numbers does not matter.", "votes": 1 }, { "id": 328030, "postDate": "2018-05-13T07:26:32.317Z", "content": "My apologies, I thought I was reading about neural networks
", "rawMarkdown": "My apologies, I thought I was reading about neural networks" }, { "id": 329853, "postDate": "2018-05-17T09:43:48.857Z", "content": "Maybe raw categorical features information are captured by the more complex features. For example, ip_dev_os and dev or os may be higly correlated. Can this increase the noise instead of adding meaningful predicting value?
", "rawMarkdown": "Maybe raw categorical features information are captured by the more complex features. For example, ip_dev_os and dev or os may be higly correlated. Can this increase the noise instead of adding meaningful predicting value?" } ] }, { "id": 386003, "postDate": "2018-09-12T00:29:45.827Z", "content": "The feature engineering by LDA is brilliant . Thanks for sharing.
", "rawMarkdown": "The feature engineering by LDA is brilliant . Thanks for sharing. ", "votes": 2 }, { "id": 368871, "postDate": "2018-08-11T05:49:17.357Z", "content": "Hi there, just a quick question. What is a bagged LGB model? And did you use different random seeds for k-fold or not when creating seven LGB models? Many thanks!
", "rawMarkdown": "Hi there, just a quick question. What is a bagged LGB model? And did you use different random seeds for k-fold or not when creating seven LGB models? Many thanks!", "votes": 2 }, { "id": 336106, "postDate": "2018-05-31T04:26:19.170Z", "content": "@flowlight's repository:\nhttps://github.com/flowlight0/talkingdata-adtracking-fraud-detection
", "rawMarkdown": "@flowlight's repository:\nhttps://github.com/flowlight0/talkingdata-adtracking-fraud-detection\n", "votes": 2, "replies": [ { "id": 336798, "postDate": "2018-06-01T09:53:31.933Z", "content": "Thanks for attaching this link. I have forgotten to post a comment about my repo.
", "rawMarkdown": "Thanks for attaching this link. I have forgotten to post a comment about my repo. ", "votes": 2 }, { "id": 336816, "postDate": "2018-06-01T10:39:21.540Z", "content": "Thank you for sharing your repository. Your creative solution is valuable not only for current but also for future data scientists, researchers and machine learning engineers. Providing its repository makes it more informative and helpful.
", "rawMarkdown": "Thank you for sharing your repository. Your creative solution is valuable not only for current but also for future data scientists, researchers and machine learning engineers. Providing its repository makes it more informative and helpful." } ] }, { "id": 326966, "postDate": "2018-05-10T15:22:00.910Z", "content": "\n\n\n\n
\n- average attributed ratio of past clicks
\n
Doesn't this feature cause over-fit? Do you confirm or are you sure this feature really improve the score?\nCould you provide the feature importance of your trained model?
", "rawMarkdown": "> - average attributed ratio of past clicks\n\nDoesn't this feature cause over-fit? Do you confirm or are you sure this feature really improve the score?\nCould you provide the feature importance of your trained model?\n", "votes": 2, "replies": [ { "id": 326972, "postDate": "2018-05-10T15:30:11.180Z", "content": "Maybe I don't understand the sentence at all. What does \"past clicks\" mean?\nCould you show a pseudo code like following?:
\n\n\n", "rawMarkdown": "Maybe I don't understand the sentence at all. What does \"past clicks\" mean?\nCould you show a pseudo code like following?:\n\n> test_and_train_df.groupby(group)['is_attributed'].mean()\n", "votes": 1 }, { "id": 327100, "postDate": "2018-05-10T20:17:58.547Z", "content": "test_and_train_df.groupby(group)['is_attributed'].mean()
\n
I asked in other forum and got an answer. I show the idea as pseudo code.
\n\ns = '{}_mean_is_attributed'.format('_'.join(group))\nday9.join(day8.groupby(group)['is_attributed'].mean().rename(s), on=group)\nday10.join(day9.groupby(group)['is_attributed'].mean().rename(s), on=group)\nday11.join(day10.groupby(group)['is_attributed'].mean().rename(s), on=group)\nDo I understand correctly?
", "rawMarkdown": "I asked in other forum and got an answer. I show the idea as pseudo code.\n\n s = '{}_mean_is_attributed'.format('_'.join(group))\n day9.join(day8.groupby(group)['is_attributed'].mean().rename(s), on=group)\n day10.join(day9.groupby(group)['is_attributed'].mean().rename(s), on=group)\n day11.join(day10.groupby(group)['is_attributed'].mean().rename(s), on=group)\n\nDo I understand correctly?\n" }, { "id": 327293, "postDate": "2018-05-11T07:43:10.760Z", "content": "I tried this feature at first but afterwards I found a better way - using all data excluding the current day to get the attributed rates, which outperforms the one when you just use the data of the previous day. And I'm sure this won't cause overfitting, at least not for me.
", "rawMarkdown": "I tried this feature at first but afterwards I found a better way - using all data excluding the current day to get the attributed rates, which outperforms the one when you just use the data of the previous day. And I'm sure this won't cause overfitting, at least not for me.", "votes": 3 }, { "id": 328150, "postDate": "2018-05-13T13:27:07.870Z", "content": "For each click, we calculated average attributed ratio by using clicks that re more than one day before that click. I wanted to make distributions of these features same in both of train and test data.
", "rawMarkdown": "For each click, we calculated average attributed ratio by using clicks that re more than one day before that click. I wanted to make distributions of these features same in both of train and test data. ", "votes": 2 } ] }, { "id": 1674903, "postDate": "2022-02-03T21:24:20.890Z", "content": "Very good and clear guidelines indeed!)
", "rawMarkdown": "Very good and clear guidelines indeed!)" }, { "id": 436149, "postDate": "2018-12-09T17:27:32.023Z", "content": "damn, that's something I have been looking for. Perfect for a rookie.\nThank you
", "rawMarkdown": "damn, that's something I have been looking for. Perfect for a rookie.\nThank you" }, { "id": 372452, "postDate": "2018-08-19T12:22:25.630Z", "content": "The LDA move is ingenious. Hope I can get the idea and use it somewhere else too.
", "rawMarkdown": "The LDA move is ingenious. Hope I can get the idea and use it somewhere else too." }, { "id": 331424, "postDate": "2018-05-21T07:42:18.070Z", "content": "Insightful solution, thanks for sharing!
", "rawMarkdown": "Insightful solution, thanks for sharing!" }, { "id": 330683, "postDate": "2018-05-19T12:34:10.807Z", "content": "Congratulations! A noob and stupid question, why the down-sampling makes such a difference? Are you using the raw model trained with the balanced dataset? Or are you performing some adjustment to take into account this down-sampling (weighting examples, adjusting the trained model, etc)?
", "rawMarkdown": "Congratulations! A noob and stupid question, why the down-sampling makes such a difference? Are you using the raw model trained with the balanced dataset? Or are you performing some adjustment to take into account this down-sampling (weighting examples, adjusting the trained model, etc)?" }, { "id": 330515, "postDate": "2018-05-19T03:03:53.790Z", "content": "Congrats. Great work.
", "rawMarkdown": "Congrats. Great work." }, { "id": 330292, "postDate": "2018-05-18T13:53:41.727Z", "content": "Thanks for sharing. It's a great learning experience!
", "rawMarkdown": "Thanks for sharing. It's a great learning experience!" }, { "id": 330079, "postDate": "2018-05-18T02:10:25.867Z", "content": "Thanks for sharing! Learn a lot from negative down-sampling and categorical feature embedding! Cool jobs!
", "rawMarkdown": "Thanks for sharing! Learn a lot from negative down-sampling and categorical feature embedding! Cool jobs!" }, { "id": 329746, "postDate": "2018-05-17T05:16:52.597Z", "content": "Congrats. Great work. Well deserved.
", "rawMarkdown": "Congrats. Great work. Well deserved. " }, { "id": 329341, "postDate": "2018-05-16T09:22:29.083Z", "content": "great solution
", "rawMarkdown": "great solution" }, { "id": 329241, "postDate": "2018-05-16T04:34:46.700Z", "content": "Thanks for sharing. Few questions
\n\nI am glad I saw this post and learn something I haven't even thought of before. Great work. Congrats!!
", "rawMarkdown": "I am glad I saw this post and learn something I haven't even thought of before. Great work. Congrats!!" }, { "id": 328475, "postDate": "2018-05-14T12:18:34.257Z", "content": "Nice work
", "rawMarkdown": "Nice work" }, { "id": 328394, "postDate": "2018-05-14T07:32:35.227Z", "content": "Congrats！ and thanks for sharing your brilliant solutions.
\n\nI am a little confused about when you said “we created higher dimensional LDA features and features that try to address the duplicate sample problem”.
\n\nEnjoy the wins and i am looking forwards to your reply!
", "rawMarkdown": "Congrats！ and thanks for sharing your brilliant solutions.\n\nI am a little confused about when you said “we created higher dimensional LDA features and features that try to address the duplicate sample problem”.\n\n1. by saying \"higher dimensional\", do you mean some matrix factoraization over combined categorical features or sth else?\n2. i dont quite understand “duplicate sample problem” and how it would negatively effect the model. \n\nEnjoy the wins and i am looking forwards to your reply!" }, { "id": 328340, "postDate": "2018-05-14T02:46:21.483Z", "content": "Cool stuff lad
", "rawMarkdown": "Cool stuff lad" }, { "id": 328096, "postDate": "2018-05-13T10:35:45.397Z", "content": "Congrats and Thanks for sharing your strategy!
", "rawMarkdown": "Congrats and Thanks for sharing your strategy!" }, { "id": 327951, "postDate": "2018-05-13T02:34:59.637Z", "content": "Congtats！Thank you for your wonderful Insights!
", "rawMarkdown": "Congtats！Thank you for your wonderful Insights!" }, { "id": 327947, "postDate": "2018-05-13T02:05:19.730Z", "content": "one question about datetime features, did you guys use click_time or attributed time? or perhaps both?
", "rawMarkdown": "one question about datetime features, did you guys use click_time or attributed time? or perhaps both? ", "replies": [ { "id": 328145, "postDate": "2018-05-13T13:20:48.847Z", "content": "We used only click_time.
", "rawMarkdown": "We used only click_time. " } ] }, { "id": 327709, "postDate": "2018-05-12T08:40:17.743Z", "content": "Thanks for posting your solution. I am learning so much from the top solutions. Wonderful Insights
", "rawMarkdown": "Thanks for posting your solution. I am learning so much from the top solutions. Wonderful Insights" }, { "id": 327657, "postDate": "2018-05-12T05:28:52.587Z", "content": "\"we created the following click series-based feature sets (i.e., each feature set consists of 31 (=(2^5) - 1) features)\"
\n\njust wondering why is (2^5)-1 here?
", "rawMarkdown": " \"we created the following click series-based feature sets (i.e., each feature set consists of 31 (=(2^5) - 1) features)\"\n\njust wondering why is (2^5)-1 here? ", "replies": [ { "id": 327674, "postDate": "2018-05-12T06:45:05.137Z", "content": "binominal theorem
\n\n\n\n\n\n\\binom{5}{0} + \\binom{5}{1} + \\binom{5}{2} + \\binom{5}{3} + \\binom{5}{4} + \\binom{5}{5} \n- \\binom{5}{0}. \n=2^5-1\n
don't know if I get the point.
\n\nSorry, but could someone please tell me how to write a math formula in plain text format?
", "rawMarkdown": "binominal theorem\n\n> \\binom{5}{0} + \\binom{5}{1} + \\binom{5}{2} + \\binom{5}{3} + \\binom{5}{4} + \\binom{5}{5} \n> - \\binom{5}{0}. \n> =2^5-1\n\ndon't know if I get the point.\n\nSorry, but could someone please tell me how to write a math formula in plain text format?" }, { "id": 327727, "postDate": "2018-05-12T09:27:51.397Z", "content": "@Kelvin, There are 2^5 subsets of a set of 5 elements. If you discard the empty set then you get 2^5 - 1.
\n\n@Rayarrow I don't know how to use Latex or math markup here.
", "rawMarkdown": "@Kelvin, There are 2^5 subsets of a set of 5 elements. If you discard the empty set then you get 2^5 - 1.\n\n@Rayarrow I don't know how to use Latex or math markup here.", "votes": 3 }, { "id": 327730, "postDate": "2018-05-12T09:32:34.963Z", "content": "Thanks anyway for your reply.
", "rawMarkdown": "Thanks anyway for your reply.", "votes": 1 }, { "id": 327845, "postDate": "2018-05-12T16:28:28.760Z", "content": "that's the part im little confused, five raw categorical features are ip, os, app, channel, device, what 5 features of each come from and how come there is one empty set ? maybe I missed some info here? Sorry, I'm just trying to understand here.
", "rawMarkdown": "that's the part im little confused, five raw categorical features are ip, os, app, channel, device, what 5 features of each come from and how come there is one empty set ? maybe I missed some info here? Sorry, I'm just trying to understand here." }, { "id": 327853, "postDate": "2018-05-12T17:02:15.793Z", "content": "@Kelvin, some more details: it's called the power set, for n elements it's size is 2^n... Here's some code to see it explicitly:
\n\nfrom itertools import chain, combinations\n\ndef powerset(iterable):\n s = list(iterable)\n return chain.from_iterable(combinations(s, r) for r in range(len(s)+1))\n\nlist(enumerate(powerset(cols)))\nThey did it the Kaggle way and generated the same feature set for all of 1..31 :)
\n\n[(0, ()),\n (1, ('app',)),\n (2, ('channel',)),\n (3, ('ip',)),\n (4, ('device',)),\n (5, ('os',)),\n (6, ('app', 'channel')),\n (7, ('app', 'ip')),\n (8, ('app', 'device')),\n (9, ('app', 'os')),\n (10, ('channel', 'ip')),\n (11, ('channel', 'device')),\n (12, ('channel', 'os')),\n (13, ('ip', 'device')),\n (14, ('ip', 'os')),\n (15, ('device', 'os')),\n (16, ('app', 'channel', 'ip')),\n (17, ('app', 'channel', 'device')),\n (18, ('app', 'channel', 'os')),\n (19, ('app', 'ip', 'device')),\n (20, ('app', 'ip', 'os')),\n (21, ('app', 'device', 'os')),\n (22, ('channel', 'ip', 'device')),\n (23, ('channel', 'ip', 'os')),\n (24, ('channel', 'device', 'os')),\n (25, ('ip', 'device', 'os')),\n (26, ('app', 'channel', 'ip', 'device')),\n (27, ('app', 'channel', 'ip', 'os')),\n (28, ('app', 'channel', 'device', 'os')),\n (29, ('app', 'ip', 'device', 'os')),\n (30, ('channel', 'ip', 'device', 'os')),\n (31, ('app', 'channel', 'ip', 'device', 'os'))]\nThey also have one of the best team names, ['flowlight', 'komaki'].shuffle(), both get top billing, 50% of the time ;) Congrats on the win guys!
@James\noh i see, they actually list all possible elements in combinations, i got it now. appreciate it your help!
", "rawMarkdown": "@James\noh i see, they actually list all possible elements in combinations, i got it now. appreciate it your help!" }, { "id": 329342, "postDate": "2018-05-16T09:24:01.070Z", "content": "nice
", "rawMarkdown": "nice" } ] }, { "id": 327598, "postDate": "2018-05-12T00:32:33.163Z", "content": "Congratulations on the win.
\n\nThe paper 'A View from the Trenches' references using a 'feature importance weight' to scale down the learning from the positive results, so the network output didn't have a bias toward positive samples, did you implement something similar?
\n\nWouldn't a model trained on oversampled positive data predict more positives?
", "rawMarkdown": "Congratulations on the win.\n\nThe paper 'A View from the Trenches' references using a 'feature importance weight' to scale down the learning from the positive results, so the network output didn't have a bias toward positive samples, did you implement something similar? \n\nWouldn't a model trained on oversampled positive data predict more positives? ", "replies": [ { "id": 328313, "postDate": "2018-05-14T00:15:42.533Z", "content": "no. Predicting more positives wasn't a problem for AUC, luckily.
", "rawMarkdown": "no. Predicting more positives wasn't a problem for AUC, luckily." } ] }, { "id": 327205, "postDate": "2018-05-11T02:44:21.247Z", "content": "Maybe the reason that why removing all raw categorical features works is that your embedding features have captured more information than raw features. Thus raw feature become noise.
", "rawMarkdown": "Maybe the reason that why removing all raw categorical features works is that your embedding features have captured more information than raw features. Thus raw feature become noise." }, { "id": 327199, "postDate": "2018-05-11T02:15:26.623Z", "content": "Congratulations.Learn more from your solution. Especially categorical features processing method
", "rawMarkdown": "Congratulations.Learn more from your solution. Especially categorical features processing method " }, { "id": 327070, "postDate": "2018-05-10T19:21:33.617Z", "content": "Congrats! The methods are brilliant! Your methods and features are quite different from us, which is really amazing. I am sure I can learn a lot from you. Thanks!
", "rawMarkdown": "Congrats! The methods are brilliant! Your methods and features are quite different from us, which is really amazing. I am sure I can learn a lot from you. Thanks!" }, { "id": 327036, "postDate": "2018-05-10T18:01:38.417Z", "content": "Beautiful solution! Thank you very much for sharing and congrats!
", "rawMarkdown": "Beautiful solution! Thank you very much for sharing and congrats! " }, { "id": 326935, "postDate": "2018-05-10T14:31:18.643Z", "content": "hello，I'm a beginner in kaggle competitions. can you explain how to created a bunch of features in a brute-force way?\nThanks
", "rawMarkdown": "hello，I'm a beginner in kaggle competitions. can you explain how to created a bunch of features in a brute-force way?\nThanks" }, { "id": 326815, "postDate": "2018-05-10T11:31:29.200Z", "content": "Thanks for sharing guys. I admire the efficiency of the downsampling for training, yet the breadth of feature exploration. Nice work!
", "rawMarkdown": "Thanks for sharing guys. I admire the efficiency of the downsampling for training, yet the breadth of feature exploration. Nice work!" }, { "id": 326743, "postDate": "2018-05-10T09:00:47.137Z", "content": "Thanks and congrats!
\nAnd just a quick question: when you use subsample to decrease the size of negative labels, do you select them randomly every time you train model or just use the same sets every time?
Thank you. We did the former. We changed our training dataset for each training to get more diversity.
", "rawMarkdown": "Thank you. We did the former. We changed our training dataset for each training to get more diversity.", "votes": 4 } ] }, { "id": 326732, "postDate": "2018-05-10T08:41:17.423Z", "content": "I'm a beginner in kaggle competitions.I‘m wondered how to bag LGB model.Does it use the output of last layer feature transform as the input of current layer?
", "rawMarkdown": "I'm a beginner in kaggle competitions.I‘m wondered how to bag LGB model.Does it use the output of last layer feature transform as the input of current layer? ", "replies": [ { "id": 326741, "postDate": "2018-05-10T08:59:03.933Z", "content": "Basically we did the following two steps.
\n1. download past submissions from Kaggle
\n2. Run the following code
submissions = [pd.read_csv(path) for path in downloaded_submission_paths]\nsubmissions_as_rank = [submission.rank() for submission in submissions]\nensemble = np.mean(submissions_as_rank, axis=0)\n", "rawMarkdown": "Basically we did the following two steps.
submissions = [pd.read_csv(path) for path in downloaded_submission_paths]\nsubmissions_as_rank = [submission.rank() for submission in submissions]\nensemble = np.mean(submissions_as_rank, axis=0)\n\n\n", "votes": 6 }, { "id": 327183, "postDate": "2018-05-11T01:23:19.027Z", "content": "
Does ur LGB model use same features when u training?
", "rawMarkdown": "Does ur LGB model use same features when u training? " } ] }, { "id": 326715, "postDate": "2018-05-10T08:12:49.670Z", "content": "congtats！awesome work！
", "rawMarkdown": "congtats！awesome work！" }, { "id": 326710, "postDate": "2018-05-10T07:54:36.590Z", "content": "Congratulations! I have a question:your features are calculate on the full data or on the down-sample features? i mean your models diversity is not only train data difference but also feature differenct or just train data difference?
", "rawMarkdown": "Congratulations! I have a question:your features are calculate on the full data or on the down-sample features? i mean your models diversity is not only train data difference but also feature differenct or just train data difference?", "replies": [ { "id": 326727, "postDate": "2018-05-10T08:35:48.410Z", "content": "Thank you ChinaBoy. All the features were computed on the full data. Negative down-sampling was done only on model training.
", "rawMarkdown": "Thank you ChinaBoy. All the features were computed on the full data. Negative down-sampling was done only on model training. ", "votes": 1 } ] }, { "id": 326708, "postDate": "2018-05-10T07:52:08.940Z", "content": "Awesome! Your score with downsampling made me happy :)
", "rawMarkdown": "Awesome! Your score with downsampling made me happy :)" }, { "id": 326697, "postDate": "2018-05-10T07:33:35.203Z", "content": "Congrats and thx for sharing, these two papers are really nice. A question : brute-force way will bring many features, so how do you choose useful feature?
", "rawMarkdown": "Congrats and thx for sharing, these two papers are really nice. A question : brute-force way will bring many features, so how do you choose useful feature?", "replies": [ { "id": 326724, "postDate": "2018-05-10T08:33:31.140Z", "content": "Thank you, zr. Both of the two papers are good, but actually the second link was not what I wanted to put. I replaced it with the correct one. Sorry about this.
\n\nThanks to negative down-sampling, the many-features was not a so big problem to us, so we didn't do fine-grained feature selection. For example, when I came up with the LDA features, I added 100 new features to my model. Luckily, it worked very well, so I added all of the 100 features. These 100 features were used for the final submission model without any further testing.
\n\nI haven't cared about which one of them contribute the most though ideally I might have had to remove useless features.
", "rawMarkdown": "Thank you, zr. Both of the two papers are good, but actually the second link was not what I wanted to put. I replaced it with the correct one. Sorry about this. \n\nThanks to negative down-sampling, the many-features was not a so big problem to us, so we didn't do fine-grained feature selection. For example, when I came up with the LDA features, I added 100 new features to my model. Luckily, it worked very well, so I added all of the 100 features. These 100 features were used for the final submission model without any further testing. \n\nI haven't cared about which one of them contribute the most though ideally I might have had to remove useless features. \n", "votes": 2 } ] }, { "id": 326694, "postDate": "2018-05-10T07:26:32.350Z", "content": "Congratulations! I really like the idea with class balancing - I tried it in other way round boosting positive examples.
", "rawMarkdown": "Congratulations! I really like the idea with class balancing - I tried it in other way round boosting positive examples. " }, { "id": 326685, "postDate": "2018-05-10T07:17:17.843Z", "content": "Congrats on the result! Thanks for sharing. I am not surprised that relying heavily on matrix factorizations helped a lot.
", "rawMarkdown": "Congrats on the result! Thanks for sharing. I am not surprised that relying heavily on matrix factorizations helped a lot.", "replies": [ { "id": 326704, "postDate": "2018-05-10T07:46:54.943Z", "content": "Hi @CPMP ,can you explain about the intuitions of matrix factorizations??I think it is very amazing
", "rawMarkdown": "Hi @CPMP ,can you explain about the intuitions of matrix factorizations??I think it is very amazing" }, { "id": 326709, "postDate": "2018-05-10T07:52:55.247Z", "content": "@ChinaBoy, I thought of it when reading a solution from a previous competition, outbrain click prediction probably. The authors (was it 3 idiots?) said that using the set of apps for a given user helped a lot. In our competition here, representing this set would be too large, I therefore tried to find a way to get an approximation. That's how I used tSVD on the userxapp matrix. Note that tSVD is exactly the same model as the lsa model use by Komaki's team. The only difference is how data is input to it.
", "rawMarkdown": "@ChinaBoy, I thought of it when reading a solution from a previous competition, outbrain click prediction probably. The authors (was it 3 idiots?) said that using the set of apps for a given user helped a lot. In our competition here, representing this set would be too large, I therefore tried to find a way to get an approximation. That's how I used tSVD on the userxapp matrix. Note that tSVD is exactly the same model as the lsa model use by Komaki's team. The only difference is how data is input to it.", "votes": 5 }, { "id": 326711, "postDate": "2018-05-10T07:56:13.187Z", "content": "yes.i also think so~thank you for your reply
", "rawMarkdown": "yes.i also think so~thank you for your reply" } ] }, { "id": 627607, "postDate": "2019-09-16T07:07:58.857Z", "rawMarkdown": "", "isDeleted": true }, { "id": 330050, "postDate": "2018-05-17T23:00:49Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1071619, "postDate": "2020-11-07T05:52:35.600Z", "content": "Thank you for sharing your approach 😃
", "rawMarkdown": "Thank you for sharing your approach 😃" }, { "id": 436248, "postDate": "2018-12-10T01:25:41.773Z", "content": "thank you
", "rawMarkdown": "thank you" }, { "id": 401111, "postDate": "2018-10-09T12:57:56.033Z", "content": "Thank you for sharing. I will try it.
", "rawMarkdown": "Thank you for sharing. I will try it." }, { "id": 332137, "postDate": "2018-05-22T15:54:22.030Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 330726, "postDate": "2018-05-19T14:53:33.800Z", "content": "Congratulations!Thanks for sharing.
", "rawMarkdown": "Congratulations!Thanks for sharing." }, { "id": 328344, "postDate": "2018-05-14T03:00:09.620Z", "content": "Thank you for sharing!
", "rawMarkdown": "Thank you for sharing! " } ], "comments": [ { "id": 326747, "author_name": "bestfitting", "author_url": "", "post_date": "2018-05-10T09:09:32.277000", "content": "Congrats,a very good solution,your 1st place is well deserved,thanks for sharing. I think if we can combine the top teams' solutions ,we will get a very high score.
", "votes": 9, "replies": [ { "id": 326757, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-10T09:23:19.427000", "content": "I am not sure I qualify as one of the top teams, but I shared my 0.98379 private solution here. Maybe others could do the same so that we can blend?
", "votes": 4, "replies": [] }, { "id": 326761, "author_name": "bestfitting", "author_url": "", "post_date": "2018-05-10T09:30:09.453000", "content": "Absolutely should include you :)
To get top 50 is not so easy in this competition.
I feel the same. I am very impressed to see so many wonderful ideas that I have never thought of.
", "votes": 4, "replies": [] } ] }, { "id": 326793, "author_name": "Pranav Pandya", "author_url": "", "post_date": "2018-05-10T10:55:21.323000", "content": "Congratulations @ckomaki and @flowlight. Excellent approach and very well deserved win. And a big thank you for the mention :) This actually made my day!
", "votes": 8, "replies": [ { "id": 326932, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-10T14:27:23.777000", "content": "@Pranav, It is a pity to see how Dirk share impacted you. Most people reused your shared kernels.
", "votes": 1, "replies": [] }, { "id": 326946, "author_name": "Pranav Pandya", "author_url": "", "post_date": "2018-05-10T14:54:21.077000", "content": "That's alrite @CPMP. Though Dirk impacted me too but I also blame myself for making stupid selections on judgement day :) But honestly, this competition was really a great learning experience for me specifically in feature engineering part.
\n\nShared learning from community actually helped me a lot and I'm sure that this will contribute positively for me in next competitions :)
", "votes": 7, "replies": [] } ] }, { "id": 326714, "author_name": "Yifan Xie", "author_url": "", "post_date": "2018-05-10T08:07:04.547000", "content": "fantastic solution, thank you very much for sharing - Matrix factorization for feature interaction will be a main take away for me from this competition
", "votes": 3, "replies": [] }, { "id": 326705, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-10T07:49:28.920000", "content": "I must say that I feel stupid for not having tested downsampling of negative. I usually try it on very imbalanced datasets. Well done!
\n\nAre you computing your lsa models on downsampled data or on the original one?
", "votes": 4, "replies": [ { "id": 326729, "author_name": "Komaki", "author_url": "", "post_date": "2018-05-10T08:36:42.033000", "content": "LSA features were computed on the original data :)
", "votes": 2, "replies": [] }, { "id": 326731, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-10T08:38:22.307000", "content": "Thanks,.
\n\nYou almost make me try to rerun my stuff on downsampled data ;)
\n\nReally neat solution, very impressive. The gap between us is well justified.
", "votes": 2, "replies": [] }, { "id": 326961, "author_name": "seaguII", "author_url": "", "post_date": "2018-05-10T15:14:50.697000", "content": "Agree. It's a very neat solution, just like a clever and beautiful solution to a mathematics problem.
", "votes": 0, "replies": [] }, { "id": 327295, "author_name": "Rayarrow", "author_url": "", "post_date": "2018-05-11T07:49:58.520000", "content": "Me too! I feel stupider because I've implemented downsampling of negative 3 weeks ago but forgot to run it until the competition ended. XD
", "votes": 0, "replies": [] } ] }, { "id": 327266, "author_name": "xuan", "author_url": "", "post_date": "2018-05-11T06:45:08.493000", "content": "Do you use sklearn.decomposition.LatentDirichletAllocation ? or others? It's very slow. How to train it quickly?
", "votes": 1, "replies": [ { "id": 328142, "author_name": "flowlight", "author_url": "", "post_date": "2018-05-13T13:18:27.490000", "content": "Yes, we used sklearn.decomposition.LatentDirichletAllocation. If my memory is correct, it takes about one hour for calculation on an (ip, app) pair. It was acceptable because we stored each feature in a file once we computed it and we don't have to run LDA for each training.
", "votes": 0, "replies": [] }, { "id": 328154, "author_name": "Rayarrow", "author_url": "", "post_date": "2018-05-13T13:36:13.703000", "content": "@xuan Class LDA has a parameter n_jobs. Have you tried it?\n@flowlight. Congrats and thanks for your sharing. Choosing a proper number of components often gave me a headache. I was wondering if there exist an empirical relationship between n_components and the shape (rows and columns) of the matrix. Or could you please tell me why you choose 5 for it? Is it just chosen arbitrarily or tuned?
", "votes": 0, "replies": [] }, { "id": 328199, "author_name": "flowlight", "author_url": "", "post_date": "2018-05-13T16:26:13.883000", "content": "There are no strong reason why we chose 5 as n_components. Actually, we also added LDA with n_components = 20 before the deadline of this competition.
", "votes": 0, "replies": [] }, { "id": 328319, "author_name": "Rayarrow", "author_url": "", "post_date": "2018-05-14T01:04:14.247000", "content": "Thanks a lot for your reply!
", "votes": 0, "replies": [] }, { "id": 329285, "author_name": "xuan", "author_url": "", "post_date": "2018-05-16T06:35:09.660000", "content": "n_jobs = -1, can quikly a lot,I try
", "votes": 0, "replies": [] } ] }, { "id": 327033, "author_name": "plantsgo", "author_url": "", "post_date": "2018-05-10T17:56:24.310000", "content": "Congratulations!Thanks for sharing.Also want to know why \"removed all raw categorical features except app\" work....
", "votes": 1, "replies": [ { "id": 327870, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-12T18:55:02.070000", "content": "Removing all but app and os worked for me. Don't know why either.
", "votes": 0, "replies": [] }, { "id": 327918, "author_name": "macfarll", "author_url": "", "post_date": "2018-05-12T22:23:35.913000", "content": "My guess:
\n\nThe categorical features are simply labels and the numbers represented by the labels are more or less arbitrary. The model is successfully finding signal in the engineered features, such as count by ip, but the original categorical features are just adding more noise to the model, and when training, the model's learning of the actual relationships between engineered features and is_attributed is being diluted by the categorical features.
\n\nThere might be some information in the categorical features, maybe some of the app ids that have higher assigned numbers are correlated is is_attributed, but that is possibly over fitting?
\n\nFor reference, my model had very few engineered features and I struggled greatly to find any signal.
", "votes": 0, "replies": [] }, { "id": 327977, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-13T04:07:21.703000", "content": "When you use a feature as a categorical in LightGBM the ordering of the numbers does not matter.
", "votes": 1, "replies": [] }, { "id": 328030, "author_name": "macfarll", "author_url": "", "post_date": "2018-05-13T07:26:32.317000", "content": "My apologies, I thought I was reading about neural networks
", "votes": 0, "replies": [] }, { "id": 329853, "author_name": "Kate Gallo", "author_url": "", "post_date": "2018-05-17T09:43:48.857000", "content": "Maybe raw categorical features information are captured by the more complex features. For example, ip_dev_os and dev or os may be higly correlated. Can this increase the noise instead of adding meaningful predicting value?
", "votes": 0, "replies": [] } ] }, { "id": 386003, "author_name": "FredGu", "author_url": "", "post_date": "2018-09-12T00:29:45.827000", "content": "The feature engineering by LDA is brilliant . Thanks for sharing.
", "votes": 2, "replies": [] }, { "id": 368871, "author_name": "seaguII", "author_url": "", "post_date": "2018-08-11T05:49:17.357000", "content": "Hi there, just a quick question. What is a bagged LGB model? And did you use different random seeds for k-fold or not when creating seven LGB models? Many thanks!
", "votes": 2, "replies": [] }, { "id": 336106, "author_name": "Fujii Hironori", "author_url": "", "post_date": "2018-05-31T04:26:19.170000", "content": "@flowlight's repository:\nhttps://github.com/flowlight0/talkingdata-adtracking-fraud-detection
", "votes": 2, "replies": [ { "id": 336798, "author_name": "flowlight", "author_url": "", "post_date": "2018-06-01T09:53:31.933000", "content": "Thanks for attaching this link. I have forgotten to post a comment about my repo.
", "votes": 2, "replies": [] }, { "id": 336816, "author_name": "Fujii Hironori", "author_url": "", "post_date": "2018-06-01T10:39:21.540000", "content": "Thank you for sharing your repository. Your creative solution is valuable not only for current but also for future data scientists, researchers and machine learning engineers. Providing its repository makes it more informative and helpful.
", "votes": 0, "replies": [] } ] }, { "id": 326966, "author_name": "Fujii Hironori", "author_url": "", "post_date": "2018-05-10T15:22:00.910000", "content": "\n\n\n\n
\n- average attributed ratio of past clicks
\n
Doesn't this feature cause over-fit? Do you confirm or are you sure this feature really improve the score?\nCould you provide the feature importance of your trained model?
", "votes": 2, "replies": [ { "id": 326972, "author_name": "Fujii Hironori", "author_url": "", "post_date": "2018-05-10T15:30:11.180000", "content": "Maybe I don't understand the sentence at all. What does \"past clicks\" mean?\nCould you show a pseudo code like following?:
\n\n\n", "votes": 1, "replies": [] }, { "id": 327100, "author_name": "Fujii Hironori", "author_url": "", "post_date": "2018-05-10T20:17:58.547000", "content": "test_and_train_df.groupby(group)['is_attributed'].mean()
\n
I asked in other forum and got an answer. I show the idea as pseudo code.
\n\ns = '{}_mean_is_attributed'.format('_'.join(group))\nday9.join(day8.groupby(group)['is_attributed'].mean().rename(s), on=group)\nday10.join(day9.groupby(group)['is_attributed'].mean().rename(s), on=group)\nday11.join(day10.groupby(group)['is_attributed'].mean().rename(s), on=group)\nDo I understand correctly?
", "votes": 0, "replies": [] }, { "id": 327293, "author_name": "Rayarrow", "author_url": "", "post_date": "2018-05-11T07:43:10.760000", "content": "I tried this feature at first but afterwards I found a better way - using all data excluding the current day to get the attributed rates, which outperforms the one when you just use the data of the previous day. And I'm sure this won't cause overfitting, at least not for me.
", "votes": 3, "replies": [] }, { "id": 328150, "author_name": "flowlight", "author_url": "", "post_date": "2018-05-13T13:27:07.870000", "content": "For each click, we calculated average attributed ratio by using clicks that re more than one day before that click. I wanted to make distributions of these features same in both of train and test data.
", "votes": 2, "replies": [] } ] }, { "id": 1674903, "author_name": "Kirill Semenov", "author_url": "", "post_date": "2022-02-03T21:24:20.890000", "content": "Very good and clear guidelines indeed!)
", "votes": 0, "replies": [] }, { "id": 436149, "author_name": "Ievgen Ishchuk", "author_url": "", "post_date": "2018-12-09T17:27:32.023000", "content": "damn, that's something I have been looking for. Perfect for a rookie.\nThank you
", "votes": 0, "replies": [] }, { "id": 372452, "author_name": "dgg32", "author_url": "", "post_date": "2018-08-19T12:22:25.630000", "content": "The LDA move is ingenious. Hope I can get the idea and use it somewhere else too.
", "votes": 0, "replies": [] }, { "id": 331424, "author_name": "datasmith", "author_url": "", "post_date": "2018-05-21T07:42:18.070000", "content": "Insightful solution, thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 330683, "author_name": "Jordi Villar", "author_url": "", "post_date": "2018-05-19T12:34:10.807000", "content": "Congratulations! A noob and stupid question, why the down-sampling makes such a difference? Are you using the raw model trained with the balanced dataset? Or are you performing some adjustment to take into account this down-sampling (weighting examples, adjusting the trained model, etc)?
", "votes": 0, "replies": [] }, { "id": 330515, "author_name": "dean1977", "author_url": "", "post_date": "2018-05-19T03:03:53.790000", "content": "Congrats. Great work.
", "votes": 0, "replies": [] }, { "id": 330292, "author_name": "TaoL", "author_url": "", "post_date": "2018-05-18T13:53:41.727000", "content": "Thanks for sharing. It's a great learning experience!
", "votes": 0, "replies": [] }, { "id": 330079, "author_name": "Shawn Xiao", "author_url": "", "post_date": "2018-05-18T02:10:25.867000", "content": "Thanks for sharing! Learn a lot from negative down-sampling and categorical feature embedding! Cool jobs!
", "votes": 0, "replies": [] }, { "id": 329746, "author_name": "AmanSrivastava", "author_url": "", "post_date": "2018-05-17T05:16:52.597000", "content": "Congrats. Great work. Well deserved.
", "votes": 0, "replies": [] }, { "id": 329341, "author_name": "joejiong", "author_url": "", "post_date": "2018-05-16T09:22:29.083000", "content": "great solution
", "votes": 0, "replies": [] }, { "id": 329241, "author_name": "Hang", "author_url": "", "post_date": "2018-05-16T04:34:46.700000", "content": "Thanks for sharing. Few questions
\n\nI am glad I saw this post and learn something I haven't even thought of before. Great work. Congrats!!
", "votes": 0, "replies": [] }, { "id": 328475, "author_name": "Ajit Singh", "author_url": "", "post_date": "2018-05-14T12:18:34.257000", "content": "Nice work
", "votes": 0, "replies": [] }, { "id": 328394, "author_name": "tiopon", "author_url": "", "post_date": "2018-05-14T07:32:35.227000", "content": "Congrats！ and thanks for sharing your brilliant solutions.
\n\nI am a little confused about when you said “we created higher dimensional LDA features and features that try to address the duplicate sample problem”.
\n\nEnjoy the wins and i am looking forwards to your reply!
", "votes": 0, "replies": [] }, { "id": 328340, "author_name": "Tom Skrovan", "author_url": "", "post_date": "2018-05-14T02:46:21.483000", "content": "Cool stuff lad
", "votes": 0, "replies": [] }, { "id": 328096, "author_name": "Kengo Suzuki", "author_url": "", "post_date": "2018-05-13T10:35:45.397000", "content": "Congrats and Thanks for sharing your strategy!
", "votes": 0, "replies": [] }, { "id": 327951, "author_name": "s-kage", "author_url": "", "post_date": "2018-05-13T02:34:59.637000", "content": "Congtats！Thank you for your wonderful Insights!
", "votes": 0, "replies": [] }, { "id": 327947, "author_name": "Kelvin", "author_url": "", "post_date": "2018-05-13T02:05:19.730000", "content": "one question about datetime features, did you guys use click_time or attributed time? or perhaps both?
", "votes": 0, "replies": [ { "id": 328145, "author_name": "flowlight", "author_url": "", "post_date": "2018-05-13T13:20:48.847000", "content": "We used only click_time.
", "votes": 0, "replies": [] } ] }, { "id": 327709, "author_name": "Akif Rehman", "author_url": "", "post_date": "2018-05-12T08:40:17.743000", "content": "Thanks for posting your solution. I am learning so much from the top solutions. Wonderful Insights
", "votes": 0, "replies": [] }, { "id": 327657, "author_name": "", "author_url": "", "post_date": "2018-05-12T05:28:52.587000", "content": "", "votes": 0, "replies": [ { "id": 327674, "author_name": "", "author_url": "", "post_date": "2018-05-12T06:45:05.137000", "content": "", "votes": 0, "replies": [] }, { "id": 327727, "author_name": "", "author_url": "", "post_date": "2018-05-12T09:27:51.397000", "content": "", "votes": 3, "replies": [] }, { "id": 327730, "author_name": "", "author_url": "", "post_date": "2018-05-12T09:32:34.963000", "content": "", "votes": 1, "replies": [] }, { "id": 327845, "author_name": "", "author_url": "", "post_date": "2018-05-12T16:28:28.760000", "content": "", "votes": 0, "replies": [] }, { "id": 327853, "author_name": "", "author_url": "", "post_date": "2018-05-12T17:02:15.793000", "content": "", "votes": 7, "replies": [] }, { "id": 327866, "author_name": "", "author_url": "", "post_date": "2018-05-12T18:21:36.290000", "content": "", "votes": 0, "replies": [] }, { "id": 329342, "author_name": "", "author_url": "", "post_date": "2018-05-16T09:24:01.070000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 327598, "author_name": "", "author_url": "", "post_date": "2018-05-12T00:32:33.163000", "content": "", "votes": 0, "replies": [ { "id": 328313, "author_name": "", "author_url": "", "post_date": "2018-05-14T00:15:42.533000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 327205, "author_name": "", "author_url": "", "post_date": "2018-05-11T02:44:21.247000", "content": "", "votes": 0, "replies": [] }, { "id": 327199, "author_name": "", "author_url": "", "post_date": "2018-05-11T02:15:26.623000", "content": "", "votes": 0, "replies": [] }, { "id": 327070, "author_name": "", "author_url": "", "post_date": "2018-05-10T19:21:33.617000", "content": "", "votes": 0, "replies": [] }, { "id": 327036, "author_name": "", "author_url": "", "post_date": "2018-05-10T18:01:38.417000", "content": "", "votes": 0, "replies": [] }, { "id": 326935, "author_name": "", "author_url": "", "post_date": "2018-05-10T14:31:18.643000", "content": "", "votes": 0, "replies": [] }, { "id": 326815, "author_name": "", "author_url": "", "post_date": "2018-05-10T11:31:29.200000", "content": "", "votes": 0, "replies": [] }, { "id": 326743, "author_name": "", "author_url": "", "post_date": "2018-05-10T09:00:47.137000", "content": "", "votes": 0, "replies": [ { "id": 326781, "author_name": "", "author_url": "", "post_date": "2018-05-10T10:26:39.197000", "content": "", "votes": 4, "replies": [] } ] }, { "id": 326732, "author_name": "", "author_url": "", "post_date": "2018-05-10T08:41:17.423000", "content": "", "votes": 0, "replies": [ { "id": 326741, "author_name": "", "author_url": "", "post_date": "2018-05-10T08:59:03.933000", "content": "", "votes": 6, "replies": [] }, { "id": 327183, "author_name": "", "author_url": "", "post_date": "2018-05-11T01:23:19.027000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 326715, "author_name": "", "author_url": "", "post_date": "2018-05-10T08:12:49.670000", "content": "", "votes": 0, "replies": [] }, { "id": 326710, "author_name": "", "author_url": "", "post_date": "2018-05-10T07:54:36.590000", "content": "", "votes": 0, "replies": [ { "id": 326727, "author_name": "", "author_url": "", "post_date": "2018-05-10T08:35:48.410000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 326708, "author_name": "", "author_url": "", "post_date": "2018-05-10T07:52:08.940000", "content": "", "votes": 0, "replies": [] }, { "id": 326697, "author_name": "", "author_url": "", "post_date": "2018-05-10T07:33:35.203000", "content": "", "votes": 0, "replies": [ { "id": 326724, "author_name": "", "author_url": "", "post_date": "2018-05-10T08:33:31.140000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 326694, "author_name": "", "author_url": "", "post_date": "2018-05-10T07:26:32.350000", "content": "", "votes": 0, "replies": [] }, { "id": 326685, "author_name": "", "author_url": "", "post_date": "2018-05-10T07:17:17.843000", "content": "", "votes": 0, "replies": [ { "id": 326704, "author_name": "", "author_url": "", "post_date": "2018-05-10T07:46:54.943000", "content": "", "votes": 0, "replies": [] }, { "id": 326709, "author_name": "", "author_url": "", "post_date": "2018-05-10T07:52:55.247000", "content": "", "votes": 5, "replies": [] }, { "id": 326711, "author_name": "", "author_url": "", "post_date": "2018-05-10T07:56:13.187000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 627607, "author_name": "", "author_url": "", "post_date": "2019-09-16T07:07:58.857000", "content": "", "votes": 0, "replies": [] }, { "id": 330050, "author_name": "", "author_url": "", "post_date": "2018-05-17T23:00:49", "content": "", "votes": 0, "replies": [] }, { "id": 1071619, "author_name": "", "author_url": "", "post_date": "2020-11-07T05:52:35.600000", "content": "", "votes": 0, "replies": [] }, { "id": 436248, "author_name": "", "author_url": "", "post_date": "2018-12-10T01:25:41.773000", "content": "", "votes": 0, "replies": [] }, { "id": 401111, "author_name": "", "author_url": "", "post_date": "2018-10-09T12:57:56.033000", "content": "", "votes": 0, "replies": [] }, { "id": 332137, "author_name": "", "author_url": "", "post_date": "2018-05-22T15:54:22.030000", "content": "", "votes": 0, "replies": [] }, { "id": 330726, "author_name": "", "author_url": "", "post_date": "2018-05-19T14:53:33.800000", "content": "", "votes": 0, "replies": [] }, { "id": 328344, "author_name": "", "author_url": "", "post_date": "2018-05-14T03:00:09.620000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "326680": "We'd like to thank TalkingData and Kaggle for organizing this exciting competition. This competition gave us a fantastic opportunity to learn how to deal with very large table data.apps_of_ip = {}\nfor sample in data_samples:\n apps_of_ip.setdefault(sample['ip'], []).append(str(sample['app']))\nips = list(apps_of_ip.keys())\napps_as_sentence = [' '.join(apps_of_ip[ip]) for ip in ips]\napps_as_matrix = CountTokenizer().fit_transform(apps_as_sentence)\ntopics_of_ips = LDA(n_components=5).fit_transform(apps_as_matrix)\n\nWe computed this feature for all the 20 (=5*(5-1)) combinations of 5 raw features and set the topic size to 5. This ended up with 100 new features. We also computed similar features using NMF and PCA, in total 300 new features. 0.9821 with a single LGB.