![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F835774%2F3fea56de822a51982159f43811503050%2FScreenshot%202020-11-07%20at%2010.37.31%20PM.png?generation=1604768976117675&alt=media\")
With quite a large dataset at hand and a unique submission format there can be a lot of interesting ideas, models and validation approaches that can be used.
\nLarge datasets often give a stable performance and so far my experience has been positive.
\nOpening this thread for anyone willing to share their validation / leaderboard approaches and / or scores.
", "messages": [ { "id": 1058728, "postDate": "2020-10-24T06:51:36.513Z", "content": "With quite a large dataset at hand and a unique submission format there can be a lot of interesting ideas, models and validation approaches that can be used.
\nLarge datasets often give a stable performance and so far my experience has been positive.
\nOpening this thread for anyone willing to share their validation / leaderboard approaches and / or scores.
", "rawMarkdown": "With quite a large dataset at hand and a unique submission format there can be a lot of interesting ideas, models and validation approaches that can be used.\n\nLarge datasets often give a stable performance and so far my experience has been positive.\n\nOpening this thread for anyone willing to share their validation / leaderboard approaches and / or scores.", "votes": 88 }, { "id": 1112090, "postDate": "2020-12-14T09:16:54.680Z", "content": "Can you share the model size? I've reach around 0.792 using SAINT+ now but still using 128 d_model and 4 num_layers. Would increasing d_model help much?
", "rawMarkdown": "Can you share the model size? I've reach around 0.792 using SAINT+ now but still using 128 d_model and 4 num_layers. Would increasing d_model help much?" }, { "id": 1112097, "postDate": "2020-12-14T09:23:38.140Z", "content": "same as the paper: d_model = 512, num_layers = 4
", "rawMarkdown": "same as the paper: d_model = 512, num_layers = 4", "votes": 3 }, { "id": 1112109, "postDate": "2020-12-14T09:31:12.723Z", "content": "Thanks, I'll see how the scaled up version works for me as well. Issue is, one model training takes up almost all of the week's GPU quota :/
\nHave to do iterations once a week onlyโฆ
\nAnyway, best of luck. Include Lectures btw, they increased my score by ~0.01 i think
", "rawMarkdown": "Thanks, I'll see how the scaled up version works for me as well. Issue is, one model training takes up almost all of the week's GPU quota :/\n\nHave to do iterations once a week only...\n\nAnyway, best of luck. Include Lectures btw, they increased my score by ~0.01 i think", "votes": 3 }, { "id": 1112131, "postDate": "2020-12-14T10:07:05.447Z", "content": "legend ๐๐๐๐๐
", "rawMarkdown": "legend ๐๐๐๐๐", "votes": 1 }, { "id": 1112456, "postDate": "2020-12-14T16:11:45.440Z", "content": "thanks, I will also try saint++ model !
", "rawMarkdown": "thanks, I will also try saint++ model !", "votes": 4 }, { "id": 1112913, "postDate": "2020-12-15T02:27:40.230Z", "content": "Nice socre, do you mind share how you split train and valid ?
", "rawMarkdown": "Nice socre, do you mind share how you split train and valid ?" }, { "id": 1113487, "postDate": "2020-12-15T13:42:04.707Z", "content": "same as this split, dropping last 2.5M samples.
", "rawMarkdown": "same as [this](https://www.kaggle.com/its7171/cv-strategy) split, dropping last 2.5M samples.", "votes": 2 }, { "id": 1115459, "postDate": "2020-12-16T10:01:27.203Z", "content": "This is what looks like when you surrond by all genious people ๐
", "rawMarkdown": "This is what looks like when you surrond by all genious people ๐", "votes": 2 }, { "id": 1117434, "postDate": "2020-12-18T03:50:02.353Z", "content": "@sakami ; Just curious, If you won't mind, Can you share how are you handling larger sequences? Splitting them up of picking randomly (in sort of time-sorted fashion)?
\nTy!
Hi Sakami, did you get that score using only the features used in the paper or did you use some additional features. I also tried the the saint approach, but it cant beat my single lstm (0.797)
", "rawMarkdown": "Hi Sakami, did you get that score using only the features used in the paper or did you use some additional features. I also tried the the saint approach, but it cant beat my single lstm (0.797)", "votes": 3 }, { "id": 1124164, "postDate": "2020-12-23T17:52:23.783Z", "content": "Wow! Single LSTM scoring .797 is mind blowing!
", "rawMarkdown": "Wow! Single LSTM scoring .797 is mind blowing!" }, { "id": 1125306, "postDate": "2020-12-24T16:00:37.480Z", "content": "Sorry for late reply.
\n@adityaecdrid I'm using the same sampling method used in many public notebooks.
\n@mchahhou I'm using completely same features as the paper.
Last LB score increases:
\nLocal: 0.7609 LB: 0.759
\nLocal: 0.7757 LB: 0.772
\nLocal: 0.7817 LB: 0.780
\nLocal: 0.7866 LB: 0.786
I'm using 10m rows to train and 2.5m to validate per fold.
", "rawMarkdown": "Last LB score increases:\n\nLocal: **0.7609** LB: **0.759**\nLocal: **0.7757** LB: **0.772**\nLocal: **0.7817** LB: **0.780**\nLocal: **0.7866** LB: **0.786**\n\nI'm using 10m rows to train and 2.5m to validate per fold.", "votes": 19, "replies": [ { "id": 1072868, "postDate": "2020-11-08T19:28:41.137Z", "content": "@mingpan07 it would be great if you can share your approach how did you get this strong CV/LB correlation ?
", "rawMarkdown": "@mingpan07 it would be great if you can share your approach how did you get this strong CV/LB correlation ?" } ] }, { "id": 1058744, "postDate": "2020-10-24T07:29:02.003Z", "content": "Regarding CV/LB scores, my last three jumps on the LB were the following:
\nCV: 0.725487, LB: 0.761
\nCV: 0.740906, LB: 0.767
\nCV: 0.747638, LB: 0.772
My approach to validation is pretty simple, just the following to create the training/validation set. I use the remaining 90+% of the data for feature aggregates and the like. Even though the CV score isn't too similar to the LB, an increase in CV typically comes with an increase in LB:
\nprint(\"[1] Create Training Set\")\ntraining = combined_df.groupby(\"user_id\").tail(24)\ncombined_df = combined_df.drop(training.index)\n\nprint(\"[2] Split Training Set into Validation\")\nvalidation = training.groupby(\"user_id\").tail(6)\ntraining = training.drop(validation.index)\nUpdate:
\nCV: 0.790365, LB: 0.787
\nI switched up my validation strategy to instead use the one described in https://www.kaggle.com/its7171/cv-strategy. Seems to be relatively accurate from what I've noticed throughout the thread, typically <= 0.003 of a difference beween CV/LB. This score is calculated using 10M rows for training, 2.5M rows for validation. I'm not using any lecture features yet in the implementation.
\nUpdate #2:
\nCV: 0.793102, LB: 0.789
\nAdded a few more features, changed the framework of my code throughout to make it much easier to add/test new features. Still using only 10M rows, trying to see if I can hit ~0.795+ before I really look into training with the entire training set, given the estimated 0.01 boost from other people.
", "rawMarkdown": "Regarding CV/LB scores, my last three jumps on the LB were the following:\n\nCV: **0.725487**, LB: **0.761**\nCV: **0.740906**, LB: **0.767**\nCV: **0.747638**, LB: **0.772**\n\nMy approach to validation is pretty simple, just the following to create the training/validation set. I use the remaining 90+% of the data for feature aggregates and the like. Even though the CV score isn't too similar to the LB, an increase in CV typically comes with an increase in LB:\n\n```\nprint(\"[1] Create Training Set\")\ntraining = combined_df.groupby(\"user_id\").tail(24)\ncombined_df = combined_df.drop(training.index)\n\nprint(\"[2] Split Training Set into Validation\")\nvalidation = training.groupby(\"user_id\").tail(6)\ntraining = training.drop(validation.index)\n```\n\nUpdate:\n\nCV: **0.790365**, LB: **0.787**\n\nI switched up my validation strategy to instead use the one described in https://www.kaggle.com/its7171/cv-strategy. Seems to be relatively accurate from what I've noticed throughout the thread, typically <= 0.003 of a difference beween CV/LB. This score is calculated using 10M rows for training, 2.5M rows for validation. I'm not using any lecture features yet in the implementation.\n\nUpdate #2:\n\nCV: **0.793102**, LB: **0.789**\n\nAdded a few more features, changed the framework of my code throughout to make it much easier to add/test new features. Still using only 10M rows, trying to see if I can hit ~0.795+ before I really look into training with the entire training set, given the estimated 0.01 boost from other people.", "votes": 13 }, { "id": 1058962, "postDate": "2020-10-24T13:00:36.780Z", "content": "Local Validation: 0.770, LB: 0.766.
\nUpdate: Local 0.777, LB: 0.772.
That's with training on ~4% of the data, and validating on ~2%. My validation scheme is set up to mirror the way the test predictions need to be made, so I'm fairly confident in it. I think a bit of a gap is normal especially given the time element.
\nI strongly agree with what Alex said below about prioritizing finding the right features before using more of the data. 2% is already millions of samples and likely large enough to give reliable model feedback IMO (and even close to the same order as the actual test set). The key thing early in a competition is to be able to iterate on feature/model ideas quickly, and I'm pretty sure training time scales above linear with # of training rows for the typical gradient boosting model.
\nEdit: see update
", "rawMarkdown": "Local Validation: **0.770**, LB: **0.766**.\nUpdate: Local **0.777**, LB: **0.772**. \n\nThat's with training on ~4% of the data, and validating on ~2%. My validation scheme is set up to mirror the way the test predictions need to be made, so I'm fairly confident in it. I think a bit of a gap is normal especially given the time element.\n\nI strongly agree with what Alex said below about prioritizing finding the right features before using more of the data. 2% is already millions of samples and likely large enough to give reliable model feedback IMO (and even close to the same order as the actual test set). The key thing early in a competition is to be able to iterate on feature/model ideas quickly, and I'm pretty sure training time scales above linear with # of training rows for the typical gradient boosting model.\n\nEdit: see update ", "votes": 11, "replies": [ { "id": 1058966, "postDate": "2020-10-24T13:11:33.600Z", "content": "@aquatic Thanks for the info
\nAre you are now generating features for the 4% of data or for the whole data and train it with 4% of the data.
You definitely need to use all the training data for features to handle test well, so I do that. In particular, question level features come from all the data. One distinction though - my 6% train/val is a self-contained set of users, so I only really need that 6% of full user features to train/validate. For the rest of the data I have summarized user features.
", "rawMarkdown": "You definitely need to use all the training data for features to handle test well, so I do that. In particular, question level features come from all the data. One distinction though - my 6% train/val is a self-contained set of users, so I only really need that 6% of full user features to train/validate. For the rest of the data I have summarized user features.", "votes": 2 }, { "id": 1058975, "postDate": "2020-10-24T13:32:53.970Z", "content": "Would you mind to share how to handle the features for unseen user during test time? @spacelx @aquatic ? Or point me to a notebook that handle this?
", "rawMarkdown": "Would you mind to share how to handle the features for unseen user during test time? @spacelx @aquatic ? Or point me to a notebook that handle this?" }, { "id": 1058986, "postDate": "2020-10-24T13:50:28.873Z", "content": "I essentially have a table of users where I keep some useful attributes for each of themโฆ it starts out empty and gets filled by processing train and test data, simply adding a new row/entry for each previously unseen user and updating information as needed for already existing users.
", "rawMarkdown": "I essentially have a table of users where I keep some useful attributes for each of them... it starts out empty and gets filled by processing train and test data, simply adding a new row/entry for each previously unseen user and updating information as needed for already existing users.", "votes": 8 }, { "id": 1059004, "postDate": "2020-10-24T14:11:39.940Z", "content": "@yihdarshieh I actually have prepared a notebook like that, but unfortunately I don't feel comfortable sharing it yet because it scores fairly highly (top 50ish). I haven't seen any public examples as of yet. I'm hoping to share a little later on, at a point when it wouldn't disrupt the leaderboard as much.
\nMy method is similar to what Alex describes, where a table gets updated on a running basis as the test data comes in.
", "rawMarkdown": "@yihdarshieh I actually have prepared a notebook like that, but unfortunately I don't feel comfortable sharing it yet because it scores fairly highly (top 50ish). I haven't seen any public examples as of yet. I'm hoping to share a little later on, at a point when it wouldn't disrupt the leaderboard as much.\n\nMy method is similar to what Alex describes, where a table gets updated on a running basis as the test data comes in. ", "votes": 5 }, { "id": 1059011, "postDate": "2020-10-24T14:22:17.867Z", "content": "@aquatic No problem. I am just curious how this is done in FE. As I mentioned, I focus on sequential model, which should just use sequences as input, so there is no actual problem for unseen users. Of course, for unseen users, there is no history to use for the first prediction during test time - but the model, during training, will learn the distribution of targets in this case anyway. (You can think it as predicting the first word in a translation problem).
\nBeing said so, you could try to interoperate features among all users, like what's the ratio of answer correctly a question if this question is the first question in a user interaction sequences.
\n\nFeatures are entirely non-future-leaking
\n
Little confused here, let's say a user which you saw in train is also there in your val, so shall you use training stats and update that or use fresh stats? In case we don't use fresh, then we are leaking the user's past performance in val data, right? Am i missing something here? Ty!
\n\n\nWould you mind to share how to handle the features for unseen user during test time? Or point me to a notebook that handle this?
\n
One way to do this is what's already mentioned in this thread by Alex and Joe, we can index the user_stats_df's with the user_id (as the uid) for e.g.
", "rawMarkdown": ">Features are entirely non-future-leaking\n\nLittle confused here, let's say a user which you saw in train is also there in your val, so shall you use training stats and update that or use fresh stats? In case we don't use fresh, then we are leaking the user's past performance in val data, right? Am i missing something here? Ty!\n\n>Would you mind to share how to handle the features for unseen user during test time? Or point me to a notebook that handle this?\n\nOne way to do this is what's already mentioned in this thread by Alex and Joe, we can index the user_stats_df's with the user_id (as the uid) for e.g." }, { "id": 1059198, "postDate": "2020-10-24T19:10:41.137Z", "content": "\n\nIn case we don't use fresh, then we are leaking the user's past performance in val data, right?
\n
Using past performance info to predict the future is not a leak. Leaks refer to cases where the model has access to information that would not actually be available at prediction time, like future performance. In this problem, when we process and predict on the test data, we have access to past performance for users that aren't brand new. So long as the features you use are derived entirely from what the user has done in the past, you will be leak-free.
\nThe bigger picture way of thinking about this is that it's not really about the idea of an \"information leak\", but more a question of what's known at training time vs. what's known at prediction time. In most cases those two should mirror each other as much as possible for optimal results, both to prevent overfitting and underfitting. In particular, if you avoid using past information you're not protecting yourself from the harm of a leak, but instead denying your model relevant information that should almost certainly improve its performance.
\nAnother extension of this idea is that in a kaggle competition setting, it can be correct to use unrealistic, future-leaking features (talkingdata click prediction from a few years ago is a good example), so long as all parts of the timeline are accessible. Luckily, this competition gives you a much more realistic paradigm via the API submission process.
", "rawMarkdown": "> In case we don't use fresh, then we are leaking the user's past performance in val data, right?\n\nUsing past performance info to predict the future is not a leak. Leaks refer to cases where the model has access to information that would not actually be available at prediction time, like future performance. In this problem, when we process and predict on the test data, we have access to past performance for users that aren't brand new. So long as the features you use are derived entirely from what the user has done in the past, you will be leak-free.\n\nThe bigger picture way of thinking about this is that it's not really about the idea of an \"information leak\", but more a question of what's known at training time vs. what's known at prediction time. In most cases those two should mirror each other as much as possible for optimal results, both to prevent overfitting and underfitting. In particular, if you avoid using past information you're not protecting yourself from the harm of a leak, but instead denying your model relevant information that should almost certainly improve its performance.\n\nAnother extension of this idea is that in a kaggle competition setting, it can be correct to use unrealistic, future-leaking features (talkingdata click prediction from a few years ago is a good example), so long as all parts of the timeline are accessible. Luckily, this competition gives you a much more realistic paradigm via the API submission process.\n\n", "votes": 7 }, { "id": 1059205, "postDate": "2020-10-24T19:15:07.287Z", "content": "For a single user, I go row by row in order of time - for each row calculating all features based on the data available from previous rows (number of answers, number of correct answers -> mean correctness etc.). The features in each row then only depend on past information.
\nSo I end up having a very long list of some features (historic mean user correctness, mean question correctness, etc.) and the information whether this question was answered correctly and that's what I'm training on. User IDs are dropped so I couldn't even tell anymore which user is which. The goal is to find a general model that predicts the chance of a random user getting the question right based on that random user's past performance, pretty much.
\nSo of course the user's past performance will \"leak\" into the validation data if you want to call it that, but then which features would we even predict on if we didn't want to take into account past performance?
\nMy validation is a random split, so it might well be (and will undoubtedly happen) that I train on the 500th question answered by a user, using knowledge of the outcome of all their previous questions, and validate the result using, amongst other data, the 10th question answered by this user given features calculated from the outcome of their previous 9 questions.
\nDoes this make sense? I'm not sure whether that answers your question.
", "rawMarkdown": "For a single user, I go row by row in order of time - for each row calculating all features based on the data available from previous rows (number of answers, number of correct answers -> mean correctness etc.). The features in each row then only depend on past information.\n\nSo I end up having a very long list of some features (historic mean user correctness, mean question correctness, etc.) and the information whether this question was answered correctly and that's what I'm training on. User IDs are dropped so I couldn't even tell anymore which user is which. The goal is to find a general model that predicts the chance of a random user getting the question right based on that random user's past performance, pretty much.\n\nSo of course the user's past performance will \"leak\" into the validation data if you want to call it that, but then which features would we even predict on if we didn't want to take into account past performance?\n\nMy validation is a random split, so it might well be (and will undoubtedly happen) that I train on the 500th question answered by a user, using knowledge of the outcome of all their previous questions, and validate the result using, amongst other data, the 10th question answered by this user given features calculated from the outcome of their previous 9 questions.\n\nDoes this make sense? I'm not sure whether that answers your question.", "votes": 14 }, { "id": 1059400, "postDate": "2020-10-25T03:39:37.357Z", "content": "Thanks a lot Joe and Alex! It's much clear now!๐
", "rawMarkdown": "Thanks a lot Joe and Alex! It's much clear now!๐" }, { "id": 1059911, "postDate": "2020-10-25T15:24:17.317Z", "content": "\n\nFor a single user, I go row by row in order of time - for each row calculating all features based on the data available from previous rows (number of answers, number of correct answers -> mean correctness etc.). The features in each row then only depend on past information.
\n
@spacelx How are you able to do all this for 393,656 users without surpassing the 9h time limit? You only have 0.08 seconds available per user, right?
", "rawMarkdown": "> For a single user, I go row by row in order of time - for each row calculating all features based on the data available from previous rows (number of answers, number of correct answers -> mean correctness etc.). The features in each row then only depend on past information.\n\n@spacelx How are you able to do all this for 393,656 users without surpassing the 9h time limit? You only have 0.08 seconds available per user, right?", "votes": 1 }, { "id": 1059937, "postDate": "2020-10-25T15:51:21.710Z", "content": "My test preprocessing is currently split across 5 kernels, but all finish within about 1.5 hours - the split is to accommodate the large size of the output feature files which get close to 20 GB or so I guess. It could definitely finish in one kernel, maybe in 4-5 hours at a guess.
\nI don't process the data literally row by row, just row by row per user. In practical terms this means I keep a list of test set rows for each user, sorted by timestamp, and batch the processing by that. I.e., first batch is the first entry of each user, second batch the second entry of each user and so on.
\nThe kernel split is done on a user-basis, so each kernel handles about 80k users. So the first batch in the preprocessing loop will be some 80k rows (takes some 5 secs maybe), and the batches get smaller, in a somewhat exponential fashion. The most active users have some >15k rows (quite impressive dedication of them actually) so by the end I'm forced to go row by row, but that's not a big issue either.
", "rawMarkdown": "My test preprocessing is currently split across 5 kernels, but all finish within about 1.5 hours - the split is to accommodate the large size of the output feature files which get close to 20 GB or so I guess. It could definitely finish in one kernel, maybe in 4-5 hours at a guess.\n\nI don't process the data literally row by row, just row by row *per user*. In practical terms this means I keep a list of test set rows for each user, sorted by timestamp, and batch the processing by that. I.e., first batch is the first entry of each user, second batch the second entry of each user and so on.\n\nThe kernel split is done on a user-basis, so each kernel handles about 80k users. So the first batch in the preprocessing loop will be some 80k rows (takes some 5 secs maybe), and the batches get smaller, in a somewhat exponential fashion. The most active users have some >15k rows (quite impressive dedication of them actually) so by the end I'm forced to go row by row, but that's not a big issue either.", "votes": 4, "replies": [ { "id": 1060026, "postDate": "2020-10-25T17:32:37.023Z", "content": "\n\nMy test preprocessing is currently split across 5 kernels, but all finish within about 1.5 hours - the split is to accommodate the large size of the output feature files which get close to 20 GB or so I guess. It could definitely finish in one kernel, maybe in 4-5 hours at a guess.
\n
@spacelx During test time, how do you get to load these 20GB of features into memory and use them for the prediction? No OOM issue?
", "rawMarkdown": "> My test preprocessing is currently split across 5 kernels, but all finish within about 1.5 hours - the split is to accommodate the large size of the output feature files which get close to 20 GB or so I guess. It could definitely finish in one kernel, maybe in 4-5 hours at a guess.\n> \n\n@spacelx During test time, how do you get to load these 20GB of features into memory and use them for the prediction? No OOM issue?" }, { "id": 1060047, "postDate": "2020-10-25T18:03:54.940Z", "content": "Currently I'm simply not using all the data to train my model, just a fraction of it.
", "rawMarkdown": "Currently I'm simply not using all the data to train my model, just a fraction of it." }, { "id": 1060052, "postDate": "2020-10-25T18:09:59.013Z", "content": "I mean the prediction for submission. During inference, you don't need to use the features that are prepared for training?
", "rawMarkdown": "I mean the prediction for submission. During inference, you don't need to use the features that are prepared for training? " }, { "id": 1060102, "postDate": "2020-10-25T19:18:29.130Z", "content": "Once the model is trained the train features aren't needed anymore, so in the inference kernel I just load the pre-trained model. Test data needs to be preprocessed though before sticking it through the model.
", "rawMarkdown": "Once the model is trained the train features aren't needed anymore, so in the inference kernel I just load the pre-trained model. Test data needs to be preprocessed though before sticking it through the model.", "votes": 3 } ] }, { "id": 1059943, "postDate": "2020-10-25T15:58:03.757Z", "content": "\n\nThe most active users have some >15k rows (quite impressive dedication of them actually)
\n
They certainly have a bright future ๐
", "rawMarkdown": "> The most active users have some >15k rows (quite impressive dedication of them actually)\n\nThey certainly have a bright future ๐", "votes": 3 }, { "id": 1059967, "postDate": "2020-10-25T16:19:29.773Z", "content": "20 GB+ in features really means a hell lot of features ๐๐! Nice idea to split the work load's in multiple kernels for faster turn-around time as well!
", "rawMarkdown": "20 GB+ in features really means a hell lot of features ๐๐! Nice idea to split the work load's in multiple kernels for faster turn-around time as well!" }, { "id": 1059970, "postDate": "2020-10-25T16:23:33.240Z", "content": "Meh, it's not that many actuallyโฆ I should probably just switch to saving in a more efficient format than floats in plain csvโฆ
", "rawMarkdown": "Meh, it's not that many actually... I should probably just switch to saving in a more efficient format than floats in plain csv..." }, { "id": 1059991, "postDate": "2020-10-25T16:49:36.100Z", "content": "Why not store them as string's when saving them and while loading back, load as float point numbers?
", "rawMarkdown": "Why not store them as string's when saving them and while loading back, load as float point numbers?" }, { "id": 1059994, "postDate": "2020-10-25T16:55:47.160Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 1059999, "postDate": "2020-10-25T17:07:46.900Z", "content": "\n\nWhy not store them as string's when saving them and while loading back, load as float point numbers?
\n
In a .csv it saves the string representation of the floats (which can be pretty long), but it would be more space-efficient to save the floats themselves in binary format. I should really go and save in feather or something ๐
", "rawMarkdown": "> Why not store them as string's when saving them and while loading back, load as float point numbers?\n\nIn a .csv it saves the string representation of the floats (which can be pretty long), but it would be more space-efficient to save the floats themselves in binary format. I should really go and save in feather or something ๐", "votes": 1 }, { "id": 1060004, "postDate": "2020-10-25T17:09:53.990Z", "content": "They certainly have a bright future ๐
@rohanrao , if they come to Kaggle to compete, I won't be able to beat them ๐
", "rawMarkdown": "`They certainly have a bright future ๐`\n\n@rohanrao , if they come to Kaggle to compete, I won't be able to beat them ๐", "votes": 2 }, { "id": 1060024, "postDate": "2020-10-25T17:29:06.723Z", "content": "Yep, we can go for binary format for sure as numpy will help us there (never did it myself until today;) or array in Py but it's not used much in day-to-day activities. But do we need more than 4-5 decimals of precision?
", "rawMarkdown": "Yep, we can go for binary format for sure as numpy will help us there (never did it myself until today;) or array in Py but it's not used much in day-to-day activities. But do we need more than 4-5 decimals of precision?" }, { "id": 1060046, "postDate": "2020-10-25T18:00:44.913Z", "content": "\n\nBut do we need more than 4-5 decimals of precision?
\n
That's what I thought too, but a run with more decimals gave me a better score, and not a negligible increase at that. Or maybe I added a feature in between these runs and forgot about it.
\nAnyway, better safe than sorry is my current approachโฆ
Pandas has plenty of saving options so binary is not a problem at all from a practical point of view. Upside of .csv is that in Kaggle's dataset viewer you can directly see column histograms and check whether everything went right with scaling and such.
", "rawMarkdown": "> But do we need more than 4-5 decimals of precision?\n\nThat's what I thought too, but a run with more decimals gave me a better score, and not a negligible increase at that. Or maybe I added a feature in between these runs and forgot about it. \nAnyway, better safe than sorry is my current approach...\n\nPandas has plenty of saving options so binary is not a problem at all from a practical point of view. Upside of .csv is that in Kaggle's dataset viewer you can directly see column histograms and check whether everything went right with scaling and such.", "votes": 1 }, { "id": 1060092, "postDate": "2020-10-25T18:55:13.240Z", "content": "\n\nThe most active users have some >15k rows (quite impressive dedication of them actually)
\n
How do we know that they are not bots who were scrapping riddi? (out of topic, just kidding)
", "rawMarkdown": "> The most active users have some >15k rows (quite impressive dedication of them actually)\n\nHow do we know that they are not bots who were scrapping riddi? (out of topic, just kidding)" }, { "id": 1060926, "postDate": "2020-10-26T16:16:15.073Z", "content": "Is it possible to do ensembling and stacking in this challenge
\nbecause test data can only be called once right??
I would simply suggest, do something simpler first as that itself is quite complicated and challenging many times. And then one can build on top that!
", "rawMarkdown": "I would simply suggest, do something simpler first as that itself is quite complicated and challenging many times. And then one can build on top that!" }, { "id": 1105473, "postDate": "2020-12-07T22:58:00.777Z", "content": "@yihdarshieh @rohanrao Of course they have a bright future. If one of them is already here on kaggle, she just check the competion:
\n\"hmm, I created the training data for years, let's win this :P\"
single NN, inference for ~3h
\nI use this strategy.
\nCV: 0.772, LB: 0.776
Nice CV! Can you write few words about your NN! Is it transformer's based? Ty!
", "rawMarkdown": "Nice CV! Can you write few words about your NN! Is it transformer's based? Ty!" }, { "id": 1125940, "postDate": "2020-12-25T07:32:00.760Z", "content": "NN based, implemented with tensorflow, inference for 7h
\nsame cv strategy
\nCV: 0.803, LB: 0.799
Awesome Score! Good Luck!
", "rawMarkdown": "Awesome Score! Good Luck!" }, { "id": 1130563, "postDate": "2020-12-29T05:59:50.383Z", "content": "@nadare we have strong pipeline with single saint score of 78.6 we work on saint plus. let me know your thought about teamup
", "rawMarkdown": "@nadare we have strong pipeline with single saint score of 78.6 we work on saint plus. let me know your thought about teamup" } ] }, { "id": 1064170, "postDate": "2020-10-29T19:54:33.937Z", "content": "So far CV-LB correlation looks very stable - it's constantly +0.003 on LB compared to CV for all my submissions in the last 2 weeks.
", "rawMarkdown": "So far CV-LB correlation looks very stable - it's constantly +0.003 on LB compared to CV for all my submissions in the last 2 weeks.", "votes": 8 }, { "id": 1071447, "postDate": "2020-11-06T22:30:38.373Z", "content": "CV: .792
\nLB: .791
@mamasinkgs Could you please explain the validation strategy?๐
", "rawMarkdown": "@mamasinkgs Could you please explain the validation strategy?๐", "votes": 1 }, { "id": 1074479, "postDate": "2020-11-10T17:37:32.283Z", "content": "Okay, I will explain the details in the winning solution.๐
", "rawMarkdown": "Okay, I will explain the details in the winning solution.๐", "votes": 5 }, { "id": 1101258, "postDate": "2020-12-03T18:54:21.157Z", "content": "UPDATES:
\nCV: 0.8090
\nLB: 0.808X
stop plz! ๐
", "rawMarkdown": "stop plz! ๐", "votes": 6 }, { "id": 1102368, "postDate": "2020-12-04T21:09:07.627Z", "content": "@mamasinkgs Did you achieve this score using LGBM? I am asking because I am quite curious where the limit of GBDT model in this competition is. Thanks in advance!
", "rawMarkdown": "@mamasinkgs Did you achieve this score using LGBM? I am asking because I am quite curious where the limit of GBDT model in this competition is. Thanks in advance! " }, { "id": 1102597, "postDate": "2020-12-05T05:18:49.610Z", "content": "It's a top secret! But, please remember nyanp says his LGBM is .801, and Nikola Bacic (.806 in LB) says he is using Transformer. From their comments, I think Transformer may perform a bit better, but LGBM is competitive.
", "rawMarkdown": "It's a top secret! But, please remember nyanp says his LGBM is .801, and Nikola Bacic (.806 in LB) says he is using Transformer. From their comments, I think Transformer may perform a bit better, but LGBM is competitive.", "votes": 7 }, { "id": 1103287, "postDate": "2020-12-05T19:39:31.283Z", "content": "LGBM specifically, or GBM generally, including xgboost?
", "rawMarkdown": "LGBM specifically, or GBM generally, including xgboost?" }, { "id": 1103955, "postDate": "2020-12-06T13:00:45.217Z", "content": "I think 100M rows are too computationally expensive for XGBoost/CatBoost.
", "rawMarkdown": "I think 100M rows are too computationally expensive for XGBoost/CatBoost." }, { "id": 1111106, "postDate": "2020-12-13T12:12:52.303Z", "content": "UPDATES:
\nCV: 0.8125
\nLB: 0.811x
seems I'm overfitting :(
", "rawMarkdown": "UPDATES: \nCV: 0.8125\nLB: 0.811x\n\nseems I'm overfitting :(", "votes": 4 }, { "id": 1111116, "postDate": "2020-12-13T12:33:42.893Z", "content": "poor you :)
", "rawMarkdown": "poor you :)", "votes": 1 }, { "id": 1118037, "postDate": "2020-12-18T17:13:51.960Z", "content": "thus, nn might win in such case. I strongly believe.
", "rawMarkdown": "thus, nn might win in such case. I strongly believe." } ] }, { "id": 1117386, "postDate": "2020-12-18T02:11:25.650Z", "content": "CV: 0.802, LB: 0.802
\nsingle lightgbm model with 100+ features, nothing found in lectures๐. Anyone boost score from lectures features?
", "rawMarkdown": "CV: 0.802, LB: 0.802\n\nsingle lightgbm model with 100+ features, nothing found in lectures๐. Anyone boost score from lectures features?", "votes": 6, "replies": [ { "id": 1118049, "postDate": "2020-12-18T17:21:25.227Z", "content": "Hey @juzqyxs you're not alone, we're not using lectures, every attempt with lectures failed for us. Interested too with feeback on lectures features.
", "rawMarkdown": "Hey @juzqyxs you're not alone, we're not using lectures, every attempt with lectures failed for us. Interested too with feeback on lectures features.", "votes": 1 }, { "id": 1129008, "postDate": "2020-12-28T01:53:30.073Z", "content": "update, single lgb CV:0.806, LB:0.806, still without lectures features.
", "rawMarkdown": "update, single lgb CV:0.806, LB:0.806, still without lectures features.", "votes": 1 }, { "id": 1130744, "postDate": "2020-12-29T09:38:57.793Z", "content": "@juzqyxs @mpware Do you guys mean lectures don't improve your validation score or LB?
", "rawMarkdown": "@juzqyxs @mpware Do you guys mean lectures don't improve your validation score or LB?", "votes": 1 }, { "id": 1130784, "postDate": "2020-12-29T09:55:33.813Z", "content": "Your guess is right. So lectures improve your score?
", "rawMarkdown": "Your guess is right. So lectures improve your score?" }, { "id": 1130788, "postDate": "2020-12-29T09:57:05.403Z", "content": "It improves CV (a little bit) but not LB. I will give another try this week, I may have a bug somewhere.
\nBut lectures are only 2% of data so it might not be important.
@juzqyxs I asked an OR question. So which one?:)
\n@mpware 2% of data is actually important since we all compete for 3rd decimal on metric. I and several other people reported that it improves CV but not LB. Maybe we all have the same inference bug.
", "rawMarkdown": "@juzqyxs I asked an OR question. So which one?:) \n\n@mpware 2% of data is actually important since we all compete for 3rd decimal on metric. I and several other people reported that it improves CV but not LB. Maybe we all have the same inference bug.", "votes": 1 }, { "id": 1130864, "postDate": "2020-12-29T10:59:13.777Z", "content": "Because lectures features did not improve our local cv, we did not submit. If you generated features from lectures after user's answer records, it might improve your local cv, but the online lectures info maybe in next group, this could be the reason that LB not improve.
", "rawMarkdown": "Because lectures features did not improve our local cv, we did not submit. If you generated features from lectures after user's answer records, it might improve your local cv, but the online lectures info maybe in next group, this could be the reason that LB not improve." } ] }, { "id": 1122135, "postDate": "2020-12-22T08:09:12.127Z", "content": "CV: 0.808, LB: 0.805
\nLooking at everyone has better CV-LB gap, I either have a bug in inference or something else is wrong.
", "rawMarkdown": "CV: 0.808, LB: 0.805\n\nLooking at everyone has better CV-LB gap, I either have a bug in inference or something else is wrong.", "votes": 3, "replies": [ { "id": 1122149, "postDate": "2020-12-22T08:26:17.310Z", "content": "Can I form a team with you to study.
", "rawMarkdown": "Can I form a team with you to study." }, { "id": 1122263, "postDate": "2020-12-22T10:08:54.930Z", "content": "No.
\n", "rawMarkdown": "No.\n\n~~Your message must have at least 10 characters.~~", "votes": 1 }, { "id": 1122272, "postDate": "2020-12-22T10:18:49.190Z", "content": "During inference, you can have multiple questions for the same user within the same batch. If you've some kind of expanding features then are you updating those features in that case or do you wait for the next batch? I'm facing this issue with simular gap on my local simulator.
", "rawMarkdown": "During inference, you can have multiple questions for the same user within the same batch. If you've some kind of expanding features then are you updating those features in that case or do you wait for the next batch? I'm facing this issue with simular gap on my local simulator.", "votes": 1 }, { "id": 1122279, "postDate": "2020-12-22T10:27:48.510Z", "content": "It depends if multiple questions have different timestamps in the same batch. I assume they have the same timestamp?
", "rawMarkdown": "It depends if multiple questions have different timestamps in the same batch. I assume they have the same timestamp?", "votes": 1 }, { "id": 1122282, "postDate": "2020-12-22T10:29:51.690Z", "content": "Yes, same timestamp, all within the same task_container_id.
", "rawMarkdown": "Yes, same timestamp, all within the same task_container_id.\n", "votes": 1 }, { "id": 1122286, "postDate": "2020-12-22T10:31:18.420Z", "content": "I can confirm from my personal probe.
", "rawMarkdown": "I can confirm from my personal probe." }, { "id": 1122564, "postDate": "2020-12-22T14:29:16.927Z", "content": "update features ๏ผand update dict next batch.may be userful
", "rawMarkdown": "update features ๏ผand update dict next batch.may be userful ", "votes": 1 } ] }, { "id": 1071389, "postDate": "2020-11-06T20:18:28.260Z", "content": "Local: 0.7602 LB: 0.773
\nLocal: 0.7699 LB: 0.782
Local:766
\nLB:785
Local:763,LB:780
\nUpdate: Local:771,LB:790
CV:0.798
\nLB:0.771
\nValidating method: GroupKFold by user_id.
My last two submissions were
\nLocal CV: 0.7637, LB: 0.767
\nLocal CV: 0.7567, LB: 0.760
So both times my LB was about 0.033 higher than my local CV score. Not sure why but I'll take it.
", "rawMarkdown": "My last two submissions were\n\nLocal CV: **0.7637**, LB: **0.767**\nLocal CV: **0.7567**, LB: **0.760**\n\nSo both times my LB was about 0.033 higher than my local CV score. Not sure why but I'll take it.", "votes": 3, "replies": [ { "id": 1058828, "postDate": "2020-10-24T10:37:19.400Z", "content": "As for approaches, I'm still working with only small random selections of the train data - my last model was a quick-build on about 10% or maybe even less of the training data, validated with some 2% of it. Features are entirely non-future-leaking so I can happily train and validate on whichever rows I feel like without having to pay attention to that.
", "rawMarkdown": "As for approaches, I'm still working with only small random selections of the train data - my last model was a quick-build on about 10% or maybe even less of the training data, validated with some 2% of it. Features are entirely non-future-leaking so I can happily train and validate on whichever rows I feel like without having to pay attention to that.", "votes": 2 }, { "id": 1058867, "postDate": "2020-10-24T11:28:05.680Z", "content": "18th with 10% ๐
", "rawMarkdown": "18th with 10% ๐" }, { "id": 1058874, "postDate": "2020-10-24T11:34:28.970Z", "content": "Finding good features makes all the difference really. Sure I can get some increases using all the set (and I'll eventually do that), but finding good features is much more important in the beginning and will give you quite some improvements to work with.
", "rawMarkdown": "Finding good features makes all the difference really. Sure I can get some increases using all the set (and I'll eventually do that), but finding good features is much more important in the beginning and will give you quite some improvements to work with.", "votes": 4 }, { "id": 1058878, "postDate": "2020-10-24T11:36:35.203Z", "content": "Yeah, I know. I focus on sequential models though - haven't done any FE yet.
", "rawMarkdown": "Yeah, I know. I focus on sequential models though - haven't done any FE yet." }, { "id": 1058887, "postDate": "2020-10-24T11:42:39.193Z", "content": "I guess 10 % to train the model, but FE is on the full data , right ?
", "rawMarkdown": "I guess 10 % to train the model, but FE is on the full data , right ? " }, { "id": 1058899, "postDate": "2020-10-24T11:53:30.653Z", "content": "Ok so in more detail, I get features for the entire train set, going for each user by timestep essentially to get non-future-leaked features (like mean user accuracy and such, will be very noisy for the first few questions of each user but getting better over time).
\nThis gives a me a full train set of features as well as a post-train status of some parameters which are used to calculate the features of the test set and updated with each test row (imagine, number of answers given by user x or so).
So my model is currently only trained on like 10% of the train data, but when evaluating the test data with it I have the full cumulative history, so to say, of all previous events in order to calculate test features.
", "rawMarkdown": "Ok so in more detail, I get features for the entire train set, going for each user by timestep essentially to get non-future-leaked features (like mean user accuracy and such, will be very noisy for the first few questions of each user but getting better over time).\nThis gives a me a full train set of features as well as a post-train status of some parameters which are used to calculate the features of the test set and updated with each test row (imagine, number of answers given by user x or so).\n\nSo my model is currently only trained on like 10% of the train data, but when evaluating the test data with it I have the full cumulative history, so to say, of all previous events in order to calculate test features.", "votes": 4 } ] }, { "id": 1082908, "postDate": "2020-11-18T11:15:57.953Z", "content": "Local: 0.798 LB: not submitted (with time-series split)
\nThe road to 1st submission is a long way for me ๐
", "rawMarkdown": "Local: 0.798 LB: not submitted (with time-series split)\n\nThe road to 1st submission is a long way for me ๐", "votes": 4, "replies": [ { "id": 1082922, "postDate": "2020-11-18T11:35:39.917Z", "content": "STOP PLS!!!
", "rawMarkdown": "STOP PLS!!!", "votes": -8 }, { "id": 1101447, "postDate": "2020-12-03T22:53:18.930Z", "content": "updated:
\nCV: 0.803 LB: 0.801 with single LightGBM
", "rawMarkdown": "updated:\n\nCV: 0.803 LB: 0.801 with single LightGBM", "votes": 5 }, { "id": 1101478, "postDate": "2020-12-03T23:51:44.653Z", "content": "Nice result - how much do you think is related to:
\nThank you
", "rawMarkdown": "Nice result - how much do you think is related to:\n- training/validation split\n- model parameters\n- features\n\nThank you" }, { "id": 1101536, "postDate": "2020-12-04T02:03:27.743Z", "content": "How many rows are you training with? I'm currently at a CV/LB of 0.793/0.789 with a single LGB and 10M rows, and trying to figure out how much further I can push it before upping the number of rows.
", "rawMarkdown": "How many rows are you training with? I'm currently at a CV/LB of 0.793/0.789 with a single LGB and 10M rows, and trying to figure out how much further I can push it before upping the number of rows.", "votes": 1 }, { "id": 1101543, "postDate": "2020-12-04T02:13:10.890Z", "content": "amazing work
", "rawMarkdown": "amazing work" }, { "id": 1101577, "postDate": "2020-12-04T03:25:19.270Z", "content": "I always use full dataset for submission. In my case, using full datasets increases CV by +0.006 compared to 6M rows. All hyperparameters except for num_leaves and learning_rate (for speedup training) remain the default. I think feature engineering on the solid validation is much more important than model parameters in case of GBDT.
", "rawMarkdown": "I always use full dataset for submission. In my case, using full datasets increases CV by +0.006 compared to 6M rows. All hyperparameters except for num_leaves and learning_rate (for speedup training) remain the default. I think feature engineering on the solid validation is much more important than model parameters in case of GBDT.", "votes": 14 }, { "id": 1101580, "postDate": "2020-12-04T03:29:35.460Z", "content": "Perfect Feature Engineering๐๐๐
", "rawMarkdown": "Perfect Feature Engineering๐๐๐", "votes": 1 }, { "id": 1102849, "postDate": "2020-12-05T11:48:14.457Z", "content": "nyanp, did you really change num_leaves for full data training? or do you mean nround?
I forgot about the nround, it's based on early stopping, so of course it's different for full rows and 6M rows. As for the other parameters, I adjusted the num_leaves and learning_rate in 6M rows and used the same parameters in the full data as well.
", "rawMarkdown": "I forgot about the nround, it's based on early stopping, so of course it's different for full rows and 6M rows. As for the other parameters, I adjusted the num_leaves and learning_rate in 6M rows and used the same parameters in the full data as well.", "votes": 4 }, { "id": 1102865, "postDate": "2020-12-05T12:12:55.537Z", "content": "Ah ok thanks, I got it!
", "rawMarkdown": "Ah ok thanks, I got it!" } ] }, { "id": 1071733, "postDate": "2020-11-07T10:47:21.740Z", "content": "CV: 0.777
\nLB: 0.784
I have at the moment 0.7579 on validation and 0.763 on LB. I'm validating only against unseen users, and that's probably why it is quite higher on LB.
\nEDIT: I'm using 5 M rows, 33% of them as validation.
", "rawMarkdown": "I have at the moment 0.7579 on validation and 0.763 on LB. I'm validating only against unseen users, and that's probably why it is quite higher on LB.\n\nEDIT: I'm using 5 M rows, 33% of them as validation.", "votes": 4 }, { "id": 1127999, "postDate": "2020-12-27T05:08:31.843Z", "content": "Same as this cv strategy.
\nLocal: 0.8027, LB: 0.805x
My last 3 submissions (using holdout validation set):
\nLocal: 0.756 LB: 0.758
\nLocal: 0.748 LB: 0.751
\nLocal: 0.741 LB: 0.742
CV: 0.778 - LB: 0.779
", "rawMarkdown": "CV: 0.778 - LB: 0.779", "votes": 1 }, { "id": 1090124, "postDate": "2020-11-25T05:11:11.037Z", "content": "some last models:
\nCV 0.756//LB0.759
\nCV0.770//LB0.766
\nCV0.771//LB0.769
\nCV0.774//LB0.772
\nCV0.777//LB0.774
\nit Looks like the LB score is fluctuating around .003.
\nsingle LGBM LB0.801 sounds so magic,need more features..๐ช๐ช๐ช
but, seems you are lucky that you received a consistent improvement.
", "rawMarkdown": "but, seems you are lucky that you received a consistent improvement." }, { "id": 1125319, "postDate": "2020-12-24T16:13:37.280Z", "content": "update my features style ,now CV 0.781,LB0.781
", "rawMarkdown": "update my features style ,now CV 0.781,LB0.781" } ] }, { "id": 1079477, "postDate": "2020-11-16T05:37:12.520Z", "content": "Local: 0.7638 LB 0.765
\nUsing all training data and follow the cross-validation strategy in https://www.kaggle.com/its7171/cv-strategy
", "rawMarkdown": "Local: **0.7638** LB **0.765**\n\nUsing all training data and follow the cross-validation strategy in https://www.kaggle.com/its7171/cv-strategy", "votes": 1, "replies": [ { "id": 1089909, "postDate": "2020-11-24T22:16:51.303Z", "content": "I'm using the same cross-validation strategy, however I'm experiencing a gap in LB vs Local (0.757 vs 0.773). I noticed this gap exists only when I used cumulative user features. Do you experience the same ? Thanks!
", "rawMarkdown": "I'm using the same cross-validation strategy, however I'm experiencing a gap in LB vs Local (0.757 vs 0.773). I noticed this gap exists only when I used cumulative user features. Do you experience the same ? Thanks!" } ] }, { "id": 1078828, "postDate": "2020-11-15T11:02:56.707Z", "content": "Local CV: 0.740, LB: 0.727
", "rawMarkdown": "Local CV: 0.740, LB: 0.727", "votes": 1 }, { "id": 1075135, "postDate": "2020-11-11T12:23:35.723Z", "content": "CV: 0.763 LB: 0.771 with simple split by user_id
", "rawMarkdown": "CV: 0.763 LB: 0.771 with simple split by user_id", "votes": 1, "replies": [ { "id": 1089910, "postDate": "2020-11-24T22:17:27.473Z", "content": "Can you tell me how did you split by user_id ? Thanks!
", "rawMarkdown": "Can you tell me how did you split by user_id ? Thanks!" }, { "id": 1104145, "postDate": "2020-12-06T16:51:39.337Z", "content": "Update:
\nvalidation: 0.8064 LB: 0.808x
but at what costโฆ
", "rawMarkdown": "Update:\nvalidation: 0.8064 LB: 0.808x\n\nbut at what cost...", "votes": 2 } ] }, { "id": 1064367, "postDate": "2020-10-30T03:42:16.150Z", "content": "Local: 0.79305
\nLB: 0.792
Hi,
\nLocal : 0.733
\nLB: 0.758
Pretty classic validation scheme:
\nMy training set contains about 10M rows, 3M for validation. Features are aggretated on the remaining rows
The gap is similar to what @misfyre reported.
", "rawMarkdown": "The gap is similar to what @misfyre reported." }, { "id": 1063145, "postDate": "2020-10-28T14:55:56.747Z", "content": "@vopani I'm impressed by the strong correlation between your local validation and public LB. Are you using something similar to @misfyre ?
\nI've built my hold-out set in the same way so it makes sense that there is a similar gap. However I cannot reprocude his jump from 0.725 to 0.747 ๐
", "rawMarkdown": "@vopani I'm impressed by the strong correlation between your local validation and public LB. Are you using something similar to @misfyre ?\n\nI've built my hold-out set in the same way so it makes sense that there is a similar gap. However I cannot reprocude his jump from 0.725 to 0.747 ๐" }, { "id": 1063174, "postDate": "2020-10-28T15:38:20.943Z", "content": "Many others in this thread have reported much closer (and higher) correlation.
\nI just used a small random sample to start with (only made 5 successful submissions), so yet to explore how different validation setups work.
", "rawMarkdown": "Many others in this thread have reported much closer (and higher) correlation.\n\nI just used a small random sample to start with (only made 5 successful submissions), so yet to explore how different validation setups work.", "votes": 1 } ] }, { "id": 1083238, "postDate": "2020-11-18T18:35:58.437Z", "content": "I'm seeing very strong correlation between CV and LB so far, by using out-of-time validation and validating on the last 2M rows.
\nSome of my last models:
\nCV: 75.9 // LB: 76.0
\nCV: 77.0 // LB 76.9
\nCV: 77.6 // LB: 77.4
Haven't yet made the sub but the below stats is using 1M/8M rows for training and 2.5M rows for validation with 16 feats. Thanks a lot everyone for the inspirations!
\ntraining's auc: 0.769886 valid's auc: 0.765254 # 1M train, 2.5M rows val\n\ntraining's auc: 0.770383 valid's auc: 0.766083 # 8M train, 2.5M rows val\nSo i have finally made a sub for a lgbm trained with 8M rows, ~LB .762/.763 now.
", "rawMarkdown": "So i have finally made a sub for a lgbm trained with 8M rows, ~LB .762/.763 now." } ] }, { "id": 1071976, "postDate": "2020-11-07T17:09:38.927Z", "content": "Someone has a perfect 1.0 score on the LB (as it seems for now). [it's a glitch i hope, no point downvoting me] bug
\n
Very suspiciousโฆ
", "rawMarkdown": "Very suspicious..." }, { "id": 1071988, "postDate": "2020-11-07T17:27:01.507Z", "content": "I hope so, not sure why people would downvote, i just shared the information and TBH was amazed to see. Refreshed the page multiple times as well ๐
", "rawMarkdown": "I hope so, not sure why people would downvote, i just shared the information and TBH was amazed to see. Refreshed the page multiple times as well ๐ ", "votes": 1 }, { "id": 1071989, "postDate": "2020-11-07T17:28:04.103Z", "content": "This person has kindly shared a bug here: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/195815. Presumably it will just be fixed. It's not foul play, more like a white hat hacking sort of situation from what I can tell.
", "rawMarkdown": "This person has kindly shared a bug here: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/195815. Presumably it will just be fixed. It's not foul play, more like a white hat hacking sort of situation from what I can tell.", "votes": 2 }, { "id": 1071998, "postDate": "2020-11-07T17:35:46.010Z", "content": "edit-> placeholder [deleted]
", "rawMarkdown": "edit-> placeholder [deleted]" } ] }, { "id": 1061720, "postDate": "2020-10-27T08:58:21.313Z", "content": "nice implementation
", "rawMarkdown": "nice implementation", "votes": -11 }, { "id": 1061719, "postDate": "2020-10-27T08:58:00.180Z", "content": "Nice implementation
", "rawMarkdown": "Nice implementation", "votes": -11 }, { "id": 1109706, "postDate": "2020-12-12T01:30:00.810Z", "content": "CV 783 , LB : 783 , training with 7.5 million rows. However, my last big boost in cv was not a big boost in public leaderboard. I am working in geting a better data representation and looking for bugs in my last submissions.
", "rawMarkdown": "CV 783 , LB : 783 , training with 7.5 million rows. However, my last big boost in cv was not a big boost in public leaderboard. I am working in geting a better data representation and looking for bugs in my last submissions.", "replies": [ { "id": 1109745, "postDate": "2020-12-12T02:59:22.517Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1109772, "postDate": "2020-12-12T03:37:27.487Z", "content": "Lightgbm, no lecture features used
", "rawMarkdown": "Lightgbm, no lecture features used" } ] }, { "id": 1105943, "postDate": "2020-12-08T11:03:03.023Z", "content": "1.CV 0.767 / LB 0.766
\n2.CV 0.771 / LB 0.771
updated:
\nCV: 0.785 LB: 0.786 with single model
", "rawMarkdown": "updated:\n\nCV: 0.785 LB: 0.786 with single model", "votes": 1 }, { "id": 1120677, "postDate": "2020-12-21T03:41:29.830Z", "content": "How many features did you use? Did you train on kaggle's kernel๏ผthank you.
", "rawMarkdown": "How many features did you use? Did you train on kaggle's kernel๏ผthank you." }, { "id": 1120915, "postDate": "2020-12-21T07:41:33.670Z", "content": "It's around thirty features. Our team train the all model on local machine. Only use kaggle's kernel in inference.
", "rawMarkdown": "It's around thirty features. Our team train the all model on local machine. Only use kaggle's kernel in inference." } ] }, { "id": 1105028, "postDate": "2020-12-07T13:21:53.483Z", "content": "My CV is 0.756, but LB is 0.553. I can't find the reason.
\nWhen I drop variables, which is min/max/std/mean/skew of answer data of each user ID, CV is 0.701 and LB is 0.682.
\nI think maybe the additional variables are related with the problem, but I still don't know the exact reason.
\nAnyone having the similar experience?
Maybe you had some errors in feature engineering during the testing phase.
\nTherefore, your LB and CV are very different
User aggregated information may result in a data leakage.
", "rawMarkdown": "User aggregated information may result in a data leakage.", "votes": 2 } ] }, { "id": 1073485, "postDate": "2020-11-09T15:08:36.427Z", "content": "CV: 0.76 LB: 0.737
\nCV: 0.92 LB: 0.702
I don't get it, it makes no sense. My CV preparation is simple. I get the first 60% of the whole training set, for my training set, then do [ validation = X.groupby('user_id').tail(100) ]. I end up with 13 milion CV and 46 milion Train rows.
\nI did switch from Pandas to cuDF in my submission notebook to do faster merging and all, so unless cuDF is doing some wild messed up merges, I have no idea why such a wild difference. Almost giving up
", "rawMarkdown": "CV: 0.76 LB: 0.737\nCV: 0.92 LB: 0.702\n\nI don't get it, it makes no sense. My CV preparation is simple. I get the first 60% of the whole training set, for my training set, then do [ validation = X.groupby('user_id').tail(100) ]. I end up with 13 milion CV and 46 milion Train rows.\n\nI did switch from Pandas to cuDF in my submission notebook to do faster merging and all, so unless cuDF is doing some wild messed up merges, I have no idea why such a wild difference. Almost giving up", "replies": [ { "id": 1073562, "postDate": "2020-11-09T16:20:30.600Z", "content": "That second CV score looks extremely high and suggests target-leaking features or overlap between your train and validation to me. These sorts of discrepancies almost always end up coming back to data preparation IMO -- can't emphasize enough how important it is to carefully double and triple check basically everything you do with features (especially at inference time).
\nAlso, I'd suggest using a different validation setup - taking the last 100 records is going to remove all of the relatively light users from your training data, so your model might be badly biased. One possible approach is to do it based on percentage instead of raw number: e.g. take the last 10% of each user for val so that you get user-record distributions in train and val that mirror user frequency.
", "rawMarkdown": "That second CV score looks extremely high and suggests target-leaking features or overlap between your train and validation to me. These sorts of discrepancies almost always end up coming back to data preparation IMO -- can't emphasize enough how important it is to carefully double and triple check basically everything you do with features (especially at inference time). \n\nAlso, I'd suggest using a different validation setup - taking the last 100 records is going to remove all of the relatively light users from your training data, so your model might be badly biased. One possible approach is to do it based on percentage instead of raw number: e.g. take the last 10% of each user for val so that you get user-record distributions in train and val that mirror user frequency.", "votes": 7 }, { "id": 1073574, "postDate": "2020-11-09T16:33:35.817Z", "content": "It's not overlap nor leaking, I made sure to drop validation index and all that. I thought the score was really high because maybe 13 milion wasn't alot or maybe the examples were easy. So I wasn't expecting 0.92 on leaderboard, but not a drop from 0.737 to 0.702, since I only added 2 more features that do look really promising on CV and Feature Importance (gain and split for lgbm).
\nI'll try what you said on the last 10% of each user, for now I switched validation split to [ validation = X.tail(int(X.shape[0]*0.33)) ]. CV and Training Loss are really close now, and the mean correctness of both are very close also. Got 40 milion for training and 20 for CV. Maybe it does better now.
", "rawMarkdown": "It's not overlap nor leaking, I made sure to drop validation index and all that. I thought the score was really high because maybe 13 milion wasn't alot or maybe the examples were easy. So I wasn't expecting 0.92 on leaderboard, but not a drop from 0.737 to 0.702, since I only added 2 more features that do look really promising on CV and Feature Importance (gain and split for lgbm).\n\nI'll try what you said on the last 10% of each user, for now I switched validation split to [ validation = X.tail(int(X.shape[0]*0.33)) ]. CV and Training Loss are really close now, and the mean correctness of both are very close also. Got 40 milion for training and 20 for CV. Maybe it does better now." }, { "id": 1073578, "postDate": "2020-11-09T16:41:26.523Z", "content": "\n\nIMO -- can't emphasize enough how important it is to carefully double and triple check basically everything you do with features (especially at inference time).
\n
This happened yesterday on my local tests. I was doing something wrong and i wasn't aware of the same and i leaked something which resulted in a huge diff b/w train and val scores. (completely un intentionally) It took me around an hour to figure it out what went wrong.
\nThe idea here is that for each row, ensure you don't create any feature that you cannot compute at that point of time or you know about the same during that row. Plus try creating features which you can compute during inference as well easily first (to get a pipeline up) and then complicate stuffs.
", "rawMarkdown": ">IMO -- can't emphasize enough how important it is to carefully double and triple check basically everything you do with features (especially at inference time).\n\nThis happened yesterday on my local tests. I was doing something wrong and i wasn't aware of the same and i leaked something which resulted in a huge diff b/w train and val scores. (completely un intentionally) It took me around an hour to figure it out what went wrong.\n\nThe idea here is that for each row, ensure you don't create any feature that you cannot compute at that point of time or you know about the same during that row. Plus try creating features which you can compute during inference as well easily first (to get a pipeline up) and then complicate stuffs.", "votes": 2 } ] }, { "id": 1064345, "postDate": "2020-10-30T03:00:01.670Z", "content": "Im puzzled that with 10M trainset and 2.5M vset, Local: 0.7683 and LB: 0.69. I think I have got a overfitting dilemma.
", "rawMarkdown": "Im puzzled that with 10M trainset and 2.5M vset, Local: 0.7683 and LB: 0.69. I think I have got a overfitting dilemma." }, { "id": 1063886, "postDate": "2020-10-29T13:06:14.153Z", "content": "My LB and Local are radically different, the local one is 0.752 and on the LB is 0.680. Any idea ideas on why is that ?
", "rawMarkdown": "My LB and Local are radically different, the local one is 0.752 and on the LB is 0.680. Any idea ideas on why is that ? ", "replies": [ { "id": 1063896, "postDate": "2020-10-29T13:18:04.617Z", "content": "A gap this large points to data preparation issues, in my view. Check your code carefully to make sure that any feature preparation/transformation steps that you do on train are exactly the same as what do you on test (order of the columns when predicting, any label encoding/dummy variables/scaling including reusing the transformer fit on train instead of refitting it, null handling, etc.). Also might be worth checking that your prediction is formatted as expected as an actual probability, not a 0/1.
", "rawMarkdown": "A gap this large points to data preparation issues, in my view. Check your code carefully to make sure that any feature preparation/transformation steps that you do on train are exactly the same as what do you on test (order of the columns when predicting, any label encoding/dummy variables/scaling including reusing the transformer fit on train instead of refitting it, null handling, etc.). Also might be worth checking that your prediction is formatted as expected as an actual probability, not a 0/1. ", "votes": 4 }, { "id": 1064553, "postDate": "2020-10-30T08:41:15.093Z", "content": "I have checked that all the preparation/transformation steps I have the same as env_test do. But my local one is 0.768 with LB 0.690, what's wrong with such case? Will that be caused by wrong chosen features or wrong methods of data pretreatment?
", "rawMarkdown": "I have checked that all the preparation/transformation steps I have the same as env_test do. But my local one is 0.768 with LB 0.690, what's wrong with such case? Will that be caused by wrong chosen features or wrong methods of data pretreatment? " }, { "id": 1064680, "postDate": "2020-10-30T12:05:39.127Z", "content": "What closed the gap a bit for my case (not completely though), is that I extracted few features from the main CSV file, then I merged them during training. Since I didn't use the entire 100M row to extract these features, some content_ids were missing, thus resulting in a nan, leading the the decline of accuracy during scoring later on.
\nWhat I've done is simply use the entire dataset to extract features, and a portion of the dataset for training, which led to a bump in score 0.680 -> 0.724 vs local score 0.760
There is another thread 191413 on validation
", "rawMarkdown": "There is another thread [191413](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191413) on validation", "replies": [ { "id": 1066778, "postDate": "2020-11-02T03:00:53.600Z", "content": "But no one shared scores there ๐
", "rawMarkdown": "But no one shared scores there ๐", "votes": 1 } ] }, { "id": 1076266, "postDate": "2020-11-12T11:38:37.167Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1062749, "postDate": "2020-10-28T06:43:10.823Z", "rawMarkdown": "", "votes": -1, "isDeleted": true, "replies": [ { "id": 1063014, "postDate": "2020-10-28T12:33:03.190Z", "rawMarkdown": "", "votes": -1, "isDeleted": true, "replies": [ { "id": 1063086, "postDate": "2020-10-28T13:49:50.130Z", "content": "\n\n\nSubmissions are evaluated on area under the ROC curve between the predicted probability and the observed target.
\n
^From the competition's evaluation info [link]
", "rawMarkdown": "@dhyeypatel1234 :\n> Submissions are evaluated on area under the ROC curve between the predicted probability and the observed target.\n\n^From the competition's evaluation info [[link](https://www.kaggle.com/c/riiid-test-answer-prediction/overview/evaluation)]", "votes": 1 } ] } ] }, { "id": 1060810, "postDate": "2020-10-26T14:51:45.027Z", "rawMarkdown": "", "votes": -5, "isDeleted": true, "replies": [ { "id": 1060857, "postDate": "2020-10-26T15:17:49.560Z", "content": "โฆwhy plagiarize so transparently?
", "rawMarkdown": "...why plagiarize so transparently? " }, { "id": 1060863, "postDate": "2020-10-26T15:23:29.893Z", "content": "Did you justโฆโฆโฆโฆโฆ..๐๐๐
", "rawMarkdown": "Did you just.................๐๐๐" }, { "id": 1060868, "postDate": "2020-10-26T15:30:48.447Z", "content": "Or maybe he didn't realise he was logged in from the wrong id ๐
", "rawMarkdown": "Or maybe he didn't realise he was logged in from the wrong id ๐", "votes": 2 }, { "id": 1061170, "postDate": "2020-10-26T19:33:40.467Z", "content": "Looks like he has a nice armada of downvoting accounts ๐ฎ
", "rawMarkdown": "Looks like he has a nice armada of downvoting accounts ๐ฎ", "votes": 2 } ] } ], "comments": [ { "id": 1112090, "author_name": "sakami", "author_url": "", "post_date": "2020-12-14T09:16:54.680000", "content": "Can you share the model size? I've reach around 0.792 using SAINT+ now but still using 128 d_model and 4 num_layers. Would increasing d_model help much?
", "votes": 0, "replies": [] }, { "id": 1112097, "author_name": "sakami", "author_url": "", "post_date": "2020-12-14T09:23:38.140000", "content": "same as the paper: d_model = 512, num_layers = 4
", "votes": 3, "replies": [] }, { "id": 1112109, "author_name": "AbdurRafae", "author_url": "", "post_date": "2020-12-14T09:31:12.723000", "content": "Thanks, I'll see how the scaled up version works for me as well. Issue is, one model training takes up almost all of the week's GPU quota :/
\nHave to do iterations once a week onlyโฆ
\nAnyway, best of luck. Include Lectures btw, they increased my score by ~0.01 i think
", "votes": 3, "replies": [] }, { "id": 1112131, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-12-14T10:07:05.447000", "content": "legend ๐๐๐๐๐
", "votes": 1, "replies": [] }, { "id": 1112456, "author_name": "mamas", "author_url": "", "post_date": "2020-12-14T16:11:45.440000", "content": "thanks, I will also try saint++ model !
", "votes": 4, "replies": [] }, { "id": 1112913, "author_name": "cswwp", "author_url": "", "post_date": "2020-12-15T02:27:40.230000", "content": "Nice socre, do you mind share how you split train and valid ?
", "votes": 0, "replies": [] }, { "id": 1113487, "author_name": "sakami", "author_url": "", "post_date": "2020-12-15T13:42:04.707000", "content": "same as this split, dropping last 2.5M samples.
", "votes": 2, "replies": [] }, { "id": 1115459, "author_name": "AK", "author_url": "", "post_date": "2020-12-16T10:01:27.203000", "content": "This is what looks like when you surrond by all genious people ๐
", "votes": 2, "replies": [] }, { "id": 1117434, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-12-18T03:50:02.353000", "content": "@sakami ; Just curious, If you won't mind, Can you share how are you handling larger sequences? Splitting them up of picking randomly (in sort of time-sorted fashion)?
\nTy!
Hi Sakami, did you get that score using only the features used in the paper or did you use some additional features. I also tried the the saint approach, but it cant beat my single lstm (0.797)
", "votes": 3, "replies": [] }, { "id": 1124164, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-12-23T17:52:23.783000", "content": "Wow! Single LSTM scoring .797 is mind blowing!
", "votes": 0, "replies": [] }, { "id": 1125306, "author_name": "sakami", "author_url": "", "post_date": "2020-12-24T16:00:37.480000", "content": "Sorry for late reply.
\n@adityaecdrid I'm using the same sampling method used in many public notebooks.
\n@mchahhou I'm using completely same features as the paper.
Last LB score increases:
\nLocal: 0.7609 LB: 0.759
\nLocal: 0.7757 LB: 0.772
\nLocal: 0.7817 LB: 0.780
\nLocal: 0.7866 LB: 0.786
I'm using 10m rows to train and 2.5m to validate per fold.
", "votes": 19, "replies": [ { "id": 1072868, "author_name": "Abdur Rehman", "author_url": "", "post_date": "2020-11-08T19:28:41.137000", "content": "@mingpan07 it would be great if you can share your approach how did you get this strong CV/LB correlation ?
", "votes": 0, "replies": [] } ] }, { "id": 1058744, "author_name": "Nick Sarris", "author_url": "", "post_date": "2020-10-24T07:29:02.003000", "content": "Regarding CV/LB scores, my last three jumps on the LB were the following:
\nCV: 0.725487, LB: 0.761
\nCV: 0.740906, LB: 0.767
\nCV: 0.747638, LB: 0.772
My approach to validation is pretty simple, just the following to create the training/validation set. I use the remaining 90+% of the data for feature aggregates and the like. Even though the CV score isn't too similar to the LB, an increase in CV typically comes with an increase in LB:
\nprint(\"[1] Create Training Set\")\ntraining = combined_df.groupby(\"user_id\").tail(24)\ncombined_df = combined_df.drop(training.index)\n\nprint(\"[2] Split Training Set into Validation\")\nvalidation = training.groupby(\"user_id\").tail(6)\ntraining = training.drop(validation.index)\nUpdate:
\nCV: 0.790365, LB: 0.787
\nI switched up my validation strategy to instead use the one described in https://www.kaggle.com/its7171/cv-strategy. Seems to be relatively accurate from what I've noticed throughout the thread, typically <= 0.003 of a difference beween CV/LB. This score is calculated using 10M rows for training, 2.5M rows for validation. I'm not using any lecture features yet in the implementation.
\nUpdate #2:
\nCV: 0.793102, LB: 0.789
\nAdded a few more features, changed the framework of my code throughout to make it much easier to add/test new features. Still using only 10M rows, trying to see if I can hit ~0.795+ before I really look into training with the entire training set, given the estimated 0.01 boost from other people.
", "votes": 13, "replies": [] }, { "id": 1058962, "author_name": "Joe Eddy", "author_url": "", "post_date": "2020-10-24T13:00:36.780000", "content": "Local Validation: 0.770, LB: 0.766.
\nUpdate: Local 0.777, LB: 0.772.
That's with training on ~4% of the data, and validating on ~2%. My validation scheme is set up to mirror the way the test predictions need to be made, so I'm fairly confident in it. I think a bit of a gap is normal especially given the time element.
\nI strongly agree with what Alex said below about prioritizing finding the right features before using more of the data. 2% is already millions of samples and likely large enough to give reliable model feedback IMO (and even close to the same order as the actual test set). The key thing early in a competition is to be able to iterate on feature/model ideas quickly, and I'm pretty sure training time scales above linear with # of training rows for the typical gradient boosting model.
\nEdit: see update
", "votes": 11, "replies": [ { "id": 1058966, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-10-24T13:11:33.600000", "content": "@aquatic Thanks for the info
\nAre you are now generating features for the 4% of data or for the whole data and train it with 4% of the data.
You definitely need to use all the training data for features to handle test well, so I do that. In particular, question level features come from all the data. One distinction though - my 6% train/val is a self-contained set of users, so I only really need that 6% of full user features to train/validate. For the rest of the data I have summarized user features.
", "votes": 2, "replies": [] }, { "id": 1058975, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-10-24T13:32:53.970000", "content": "Would you mind to share how to handle the features for unseen user during test time? @spacelx @aquatic ? Or point me to a notebook that handle this?
", "votes": 0, "replies": [] }, { "id": 1058986, "author_name": "Alex Bader", "author_url": "", "post_date": "2020-10-24T13:50:28.873000", "content": "I essentially have a table of users where I keep some useful attributes for each of themโฆ it starts out empty and gets filled by processing train and test data, simply adding a new row/entry for each previously unseen user and updating information as needed for already existing users.
", "votes": 8, "replies": [] }, { "id": 1059004, "author_name": "Joe Eddy", "author_url": "", "post_date": "2020-10-24T14:11:39.940000", "content": "@yihdarshieh I actually have prepared a notebook like that, but unfortunately I don't feel comfortable sharing it yet because it scores fairly highly (top 50ish). I haven't seen any public examples as of yet. I'm hoping to share a little later on, at a point when it wouldn't disrupt the leaderboard as much.
\nMy method is similar to what Alex describes, where a table gets updated on a running basis as the test data comes in.
", "votes": 5, "replies": [] }, { "id": 1059011, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-10-24T14:22:17.867000", "content": "@aquatic No problem. I am just curious how this is done in FE. As I mentioned, I focus on sequential model, which should just use sequences as input, so there is no actual problem for unseen users. Of course, for unseen users, there is no history to use for the first prediction during test time - but the model, during training, will learn the distribution of targets in this case anyway. (You can think it as predicting the first word in a translation problem).
\nBeing said so, you could try to interoperate features among all users, like what's the ratio of answer correctly a question if this question is the first question in a user interaction sequences.
\n\nFeatures are entirely non-future-leaking
\n
Little confused here, let's say a user which you saw in train is also there in your val, so shall you use training stats and update that or use fresh stats? In case we don't use fresh, then we are leaking the user's past performance in val data, right? Am i missing something here? Ty!
\n\n\nWould you mind to share how to handle the features for unseen user during test time? Or point me to a notebook that handle this?
\n
One way to do this is what's already mentioned in this thread by Alex and Joe, we can index the user_stats_df's with the user_id (as the uid) for e.g.
", "votes": 0, "replies": [] }, { "id": 1059198, "author_name": "Joe Eddy", "author_url": "", "post_date": "2020-10-24T19:10:41.137000", "content": "\n\nIn case we don't use fresh, then we are leaking the user's past performance in val data, right?
\n
Using past performance info to predict the future is not a leak. Leaks refer to cases where the model has access to information that would not actually be available at prediction time, like future performance. In this problem, when we process and predict on the test data, we have access to past performance for users that aren't brand new. So long as the features you use are derived entirely from what the user has done in the past, you will be leak-free.
\nThe bigger picture way of thinking about this is that it's not really about the idea of an \"information leak\", but more a question of what's known at training time vs. what's known at prediction time. In most cases those two should mirror each other as much as possible for optimal results, both to prevent overfitting and underfitting. In particular, if you avoid using past information you're not protecting yourself from the harm of a leak, but instead denying your model relevant information that should almost certainly improve its performance.
\nAnother extension of this idea is that in a kaggle competition setting, it can be correct to use unrealistic, future-leaking features (talkingdata click prediction from a few years ago is a good example), so long as all parts of the timeline are accessible. Luckily, this competition gives you a much more realistic paradigm via the API submission process.
", "votes": 7, "replies": [] }, { "id": 1059205, "author_name": "Alex Bader", "author_url": "", "post_date": "2020-10-24T19:15:07.287000", "content": "For a single user, I go row by row in order of time - for each row calculating all features based on the data available from previous rows (number of answers, number of correct answers -> mean correctness etc.). The features in each row then only depend on past information.
\nSo I end up having a very long list of some features (historic mean user correctness, mean question correctness, etc.) and the information whether this question was answered correctly and that's what I'm training on. User IDs are dropped so I couldn't even tell anymore which user is which. The goal is to find a general model that predicts the chance of a random user getting the question right based on that random user's past performance, pretty much.
\nSo of course the user's past performance will \"leak\" into the validation data if you want to call it that, but then which features would we even predict on if we didn't want to take into account past performance?
\nMy validation is a random split, so it might well be (and will undoubtedly happen) that I train on the 500th question answered by a user, using knowledge of the outcome of all their previous questions, and validate the result using, amongst other data, the 10th question answered by this user given features calculated from the outcome of their previous 9 questions.
\nDoes this make sense? I'm not sure whether that answers your question.
", "votes": 14, "replies": [] }, { "id": 1059400, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-25T03:39:37.357000", "content": "Thanks a lot Joe and Alex! It's much clear now!๐
", "votes": 0, "replies": [] }, { "id": 1059911, "author_name": "Nya ๐", "author_url": "", "post_date": "2020-10-25T15:24:17.317000", "content": "\n\nFor a single user, I go row by row in order of time - for each row calculating all features based on the data available from previous rows (number of answers, number of correct answers -> mean correctness etc.). The features in each row then only depend on past information.
\n
@spacelx How are you able to do all this for 393,656 users without surpassing the 9h time limit? You only have 0.08 seconds available per user, right?
", "votes": 1, "replies": [] }, { "id": 1059937, "author_name": "Alex Bader", "author_url": "", "post_date": "2020-10-25T15:51:21.710000", "content": "My test preprocessing is currently split across 5 kernels, but all finish within about 1.5 hours - the split is to accommodate the large size of the output feature files which get close to 20 GB or so I guess. It could definitely finish in one kernel, maybe in 4-5 hours at a guess.
\nI don't process the data literally row by row, just row by row per user. In practical terms this means I keep a list of test set rows for each user, sorted by timestamp, and batch the processing by that. I.e., first batch is the first entry of each user, second batch the second entry of each user and so on.
\nThe kernel split is done on a user-basis, so each kernel handles about 80k users. So the first batch in the preprocessing loop will be some 80k rows (takes some 5 secs maybe), and the batches get smaller, in a somewhat exponential fashion. The most active users have some >15k rows (quite impressive dedication of them actually) so by the end I'm forced to go row by row, but that's not a big issue either.
", "votes": 4, "replies": [ { "id": 1060026, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-10-25T17:32:37.023000", "content": "\n\nMy test preprocessing is currently split across 5 kernels, but all finish within about 1.5 hours - the split is to accommodate the large size of the output feature files which get close to 20 GB or so I guess. It could definitely finish in one kernel, maybe in 4-5 hours at a guess.
\n
@spacelx During test time, how do you get to load these 20GB of features into memory and use them for the prediction? No OOM issue?
", "votes": 0, "replies": [] }, { "id": 1060047, "author_name": "Alex Bader", "author_url": "", "post_date": "2020-10-25T18:03:54.940000", "content": "Currently I'm simply not using all the data to train my model, just a fraction of it.
", "votes": 0, "replies": [] }, { "id": 1060052, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-10-25T18:09:59.013000", "content": "I mean the prediction for submission. During inference, you don't need to use the features that are prepared for training?
", "votes": 0, "replies": [] }, { "id": 1060102, "author_name": "Alex Bader", "author_url": "", "post_date": "2020-10-25T19:18:29.130000", "content": "Once the model is trained the train features aren't needed anymore, so in the inference kernel I just load the pre-trained model. Test data needs to be preprocessed though before sticking it through the model.
", "votes": 3, "replies": [] } ] }, { "id": 1059943, "author_name": "Vopani", "author_url": "", "post_date": "2020-10-25T15:58:03.757000", "content": "\n\nThe most active users have some >15k rows (quite impressive dedication of them actually)
\n
They certainly have a bright future ๐
", "votes": 3, "replies": [] }, { "id": 1059967, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-25T16:19:29.773000", "content": "20 GB+ in features really means a hell lot of features ๐๐! Nice idea to split the work load's in multiple kernels for faster turn-around time as well!
", "votes": 0, "replies": [] }, { "id": 1059970, "author_name": "Alex Bader", "author_url": "", "post_date": "2020-10-25T16:23:33.240000", "content": "Meh, it's not that many actuallyโฆ I should probably just switch to saving in a more efficient format than floats in plain csvโฆ
", "votes": 0, "replies": [] }, { "id": 1059991, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-25T16:49:36.100000", "content": "Why not store them as string's when saving them and while loading back, load as float point numbers?
", "votes": 0, "replies": [] }, { "id": 1059994, "author_name": "", "author_url": "", "post_date": "2020-10-25T16:55:47.160000", "content": "", "votes": -1, "replies": [] }, { "id": 1059999, "author_name": "Alex Bader", "author_url": "", "post_date": "2020-10-25T17:07:46.900000", "content": "\n\nWhy not store them as string's when saving them and while loading back, load as float point numbers?
\n
In a .csv it saves the string representation of the floats (which can be pretty long), but it would be more space-efficient to save the floats themselves in binary format. I should really go and save in feather or something ๐
", "votes": 1, "replies": [] }, { "id": 1060004, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-10-25T17:09:53.990000", "content": "They certainly have a bright future ๐
@rohanrao , if they come to Kaggle to compete, I won't be able to beat them ๐
", "votes": 2, "replies": [] }, { "id": 1060024, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-25T17:29:06.723000", "content": "Yep, we can go for binary format for sure as numpy will help us there (never did it myself until today;) or array in Py but it's not used much in day-to-day activities. But do we need more than 4-5 decimals of precision?
", "votes": 0, "replies": [] }, { "id": 1060046, "author_name": "Alex Bader", "author_url": "", "post_date": "2020-10-25T18:00:44.913000", "content": "\n\nBut do we need more than 4-5 decimals of precision?
\n
That's what I thought too, but a run with more decimals gave me a better score, and not a negligible increase at that. Or maybe I added a feature in between these runs and forgot about it.
\nAnyway, better safe than sorry is my current approachโฆ
Pandas has plenty of saving options so binary is not a problem at all from a practical point of view. Upside of .csv is that in Kaggle's dataset viewer you can directly see column histograms and check whether everything went right with scaling and such.
", "votes": 1, "replies": [] }, { "id": 1060092, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-25T18:55:13.240000", "content": "\n\nThe most active users have some >15k rows (quite impressive dedication of them actually)
\n
How do we know that they are not bots who were scrapping riddi? (out of topic, just kidding)
", "votes": 0, "replies": [] }, { "id": 1060926, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-10-26T16:16:15.073000", "content": "Is it possible to do ensembling and stacking in this challenge
\nbecause test data can only be called once right??
I would simply suggest, do something simpler first as that itself is quite complicated and challenging many times. And then one can build on top that!
", "votes": 0, "replies": [] }, { "id": 1105473, "author_name": "AmorfEvo", "author_url": "", "post_date": "2020-12-07T22:58:00.777000", "content": "@yihdarshieh @rohanrao Of course they have a bright future. If one of them is already here on kaggle, she just check the competion:
\n\"hmm, I created the training data for years, let's win this :P\"
single NN, inference for ~3h
\nI use this strategy.
\nCV: 0.772, LB: 0.776
Nice CV! Can you write few words about your NN! Is it transformer's based? Ty!
", "votes": 0, "replies": [] }, { "id": 1125940, "author_name": "nadare", "author_url": "", "post_date": "2020-12-25T07:32:00.760000", "content": "NN based, implemented with tensorflow, inference for 7h
\nsame cv strategy
\nCV: 0.803, LB: 0.799
Awesome Score! Good Luck!
", "votes": 0, "replies": [] }, { "id": 1130563, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-29T05:59:50.383000", "content": "@nadare we have strong pipeline with single saint score of 78.6 we work on saint plus. let me know your thought about teamup
", "votes": 0, "replies": [] } ] }, { "id": 1064170, "author_name": "alijs", "author_url": "", "post_date": "2020-10-29T19:54:33.937000", "content": "So far CV-LB correlation looks very stable - it's constantly +0.003 on LB compared to CV for all my submissions in the last 2 weeks.
", "votes": 8, "replies": [] }, { "id": 1071447, "author_name": "mamas", "author_url": "", "post_date": "2020-11-06T22:30:38.373000", "content": "CV: .792
\nLB: .791
@mamasinkgs Could you please explain the validation strategy?๐
", "votes": 1, "replies": [] }, { "id": 1074479, "author_name": "mamas", "author_url": "", "post_date": "2020-11-10T17:37:32.283000", "content": "Okay, I will explain the details in the winning solution.๐
", "votes": 5, "replies": [] }, { "id": 1101258, "author_name": "mamas", "author_url": "", "post_date": "2020-12-03T18:54:21.157000", "content": "UPDATES:
\nCV: 0.8090
\nLB: 0.808X
stop plz! ๐
", "votes": 6, "replies": [] }, { "id": 1102368, "author_name": "Tonghui Li", "author_url": "", "post_date": "2020-12-04T21:09:07.627000", "content": "@mamasinkgs Did you achieve this score using LGBM? I am asking because I am quite curious where the limit of GBDT model in this competition is. Thanks in advance!
", "votes": 0, "replies": [] }, { "id": 1102597, "author_name": "mamas", "author_url": "", "post_date": "2020-12-05T05:18:49.610000", "content": "It's a top secret! But, please remember nyanp says his LGBM is .801, and Nikola Bacic (.806 in LB) says he is using Transformer. From their comments, I think Transformer may perform a bit better, but LGBM is competitive.
", "votes": 7, "replies": [] }, { "id": 1103287, "author_name": "Caleb", "author_url": "", "post_date": "2020-12-05T19:39:31.283000", "content": "LGBM specifically, or GBM generally, including xgboost?
", "votes": 0, "replies": [] }, { "id": 1103955, "author_name": "mamas", "author_url": "", "post_date": "2020-12-06T13:00:45.217000", "content": "I think 100M rows are too computationally expensive for XGBoost/CatBoost.
", "votes": 0, "replies": [] }, { "id": 1111106, "author_name": "mamas", "author_url": "", "post_date": "2020-12-13T12:12:52.303000", "content": "UPDATES:
\nCV: 0.8125
\nLB: 0.811x
seems I'm overfitting :(
", "votes": 4, "replies": [] }, { "id": 1111116, "author_name": "Nikola Bacic", "author_url": "", "post_date": "2020-12-13T12:33:42.893000", "content": "poor you :)
", "votes": 1, "replies": [] }, { "id": 1118037, "author_name": "Zhenghan Chen", "author_url": "", "post_date": "2020-12-18T17:13:51.960000", "content": "thus, nn might win in such case. I strongly believe.
", "votes": 0, "replies": [] } ] }, { "id": 1117386, "author_name": "qyxs", "author_url": "", "post_date": "2020-12-18T02:11:25.650000", "content": "CV: 0.802, LB: 0.802
\nsingle lightgbm model with 100+ features, nothing found in lectures๐. Anyone boost score from lectures features?
", "votes": 6, "replies": [ { "id": 1118049, "author_name": "MPWARE", "author_url": "", "post_date": "2020-12-18T17:21:25.227000", "content": "Hey @juzqyxs you're not alone, we're not using lectures, every attempt with lectures failed for us. Interested too with feeback on lectures features.
", "votes": 1, "replies": [] }, { "id": 1129008, "author_name": "qyxs", "author_url": "", "post_date": "2020-12-28T01:53:30.073000", "content": "update, single lgb CV:0.806, LB:0.806, still without lectures features.
", "votes": 1, "replies": [] }, { "id": 1130744, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2020-12-29T09:38:57.793000", "content": "@juzqyxs @mpware Do you guys mean lectures don't improve your validation score or LB?
", "votes": 1, "replies": [] }, { "id": 1130784, "author_name": "qyxs", "author_url": "", "post_date": "2020-12-29T09:55:33.813000", "content": "Your guess is right. So lectures improve your score?
", "votes": 0, "replies": [] }, { "id": 1130788, "author_name": "MPWARE", "author_url": "", "post_date": "2020-12-29T09:57:05.403000", "content": "It improves CV (a little bit) but not LB. I will give another try this week, I may have a bug somewhere.
\nBut lectures are only 2% of data so it might not be important.
@juzqyxs I asked an OR question. So which one?:)
\n@mpware 2% of data is actually important since we all compete for 3rd decimal on metric. I and several other people reported that it improves CV but not LB. Maybe we all have the same inference bug.
", "votes": 1, "replies": [] }, { "id": 1130864, "author_name": "qyxs", "author_url": "", "post_date": "2020-12-29T10:59:13.777000", "content": "Because lectures features did not improve our local cv, we did not submit. If you generated features from lectures after user's answer records, it might improve your local cv, but the online lectures info maybe in next group, this could be the reason that LB not improve.
", "votes": 0, "replies": [] } ] }, { "id": 1122135, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2020-12-22T08:09:12.127000", "content": "CV: 0.808, LB: 0.805
\nLooking at everyone has better CV-LB gap, I either have a bug in inference or something else is wrong.
", "votes": 3, "replies": [ { "id": 1122149, "author_name": "ML_and_DL", "author_url": "", "post_date": "2020-12-22T08:26:17.310000", "content": "Can I form a team with you to study.
", "votes": 0, "replies": [] }, { "id": 1122263, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2020-12-22T10:08:54.930000", "content": "No.
\n", "votes": 1, "replies": [] }, { "id": 1122272, "author_name": "MPWARE", "author_url": "", "post_date": "2020-12-22T10:18:49.190000", "content": "During inference, you can have multiple questions for the same user within the same batch. If you've some kind of expanding features then are you updating those features in that case or do you wait for the next batch? I'm facing this issue with simular gap on my local simulator.
", "votes": 1, "replies": [] }, { "id": 1122279, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2020-12-22T10:27:48.510000", "content": "It depends if multiple questions have different timestamps in the same batch. I assume they have the same timestamp?
", "votes": 1, "replies": [] }, { "id": 1122282, "author_name": "MPWARE", "author_url": "", "post_date": "2020-12-22T10:29:51.690000", "content": "Yes, same timestamp, all within the same task_container_id.
", "votes": 1, "replies": [] }, { "id": 1122286, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-12-22T10:31:18.420000", "content": "I can confirm from my personal probe.
", "votes": 0, "replies": [] }, { "id": 1122564, "author_name": "qiaqia", "author_url": "", "post_date": "2020-12-22T14:29:16.927000", "content": "update features ๏ผand update dict next batch.may be userful
", "votes": 1, "replies": [] } ] }, { "id": 1071389, "author_name": "Branden Murray", "author_url": "", "post_date": "2020-11-06T20:18:28.260000", "content": "Local: 0.7602 LB: 0.773
\nLocal: 0.7699 LB: 0.782
Local:766
\nLB:785
Local:763,LB:780
\nUpdate: Local:771,LB:790
CV:0.798
\nLB:0.771
\nValidating method: GroupKFold by user_id.
My last two submissions were
\nLocal CV: 0.7637, LB: 0.767
\nLocal CV: 0.7567, LB: 0.760
So both times my LB was about 0.033 higher than my local CV score. Not sure why but I'll take it.
", "votes": 3, "replies": [ { "id": 1058828, "author_name": "Alex Bader", "author_url": "", "post_date": "2020-10-24T10:37:19.400000", "content": "As for approaches, I'm still working with only small random selections of the train data - my last model was a quick-build on about 10% or maybe even less of the training data, validated with some 2% of it. Features are entirely non-future-leaking so I can happily train and validate on whichever rows I feel like without having to pay attention to that.
", "votes": 2, "replies": [] }, { "id": 1058867, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-10-24T11:28:05.680000", "content": "18th with 10% ๐
", "votes": 0, "replies": [] }, { "id": 1058874, "author_name": "Alex Bader", "author_url": "", "post_date": "2020-10-24T11:34:28.970000", "content": "Finding good features makes all the difference really. Sure I can get some increases using all the set (and I'll eventually do that), but finding good features is much more important in the beginning and will give you quite some improvements to work with.
", "votes": 4, "replies": [] }, { "id": 1058878, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-10-24T11:36:35.203000", "content": "Yeah, I know. I focus on sequential models though - haven't done any FE yet.
", "votes": 0, "replies": [] }, { "id": 1058887, "author_name": "Ala Eddine Ayadi", "author_url": "", "post_date": "2020-10-24T11:42:39.193000", "content": "I guess 10 % to train the model, but FE is on the full data , right ?
", "votes": 0, "replies": [] }, { "id": 1058899, "author_name": "Alex Bader", "author_url": "", "post_date": "2020-10-24T11:53:30.653000", "content": "Ok so in more detail, I get features for the entire train set, going for each user by timestep essentially to get non-future-leaked features (like mean user accuracy and such, will be very noisy for the first few questions of each user but getting better over time).
\nThis gives a me a full train set of features as well as a post-train status of some parameters which are used to calculate the features of the test set and updated with each test row (imagine, number of answers given by user x or so).
So my model is currently only trained on like 10% of the train data, but when evaluating the test data with it I have the full cumulative history, so to say, of all previous events in order to calculate test features.
", "votes": 4, "replies": [] } ] }, { "id": 1082908, "author_name": "nyanp", "author_url": "", "post_date": "2020-11-18T11:15:57.953000", "content": "Local: 0.798 LB: not submitted (with time-series split)
\nThe road to 1st submission is a long way for me ๐
", "votes": 4, "replies": [ { "id": 1082922, "author_name": "mamas", "author_url": "", "post_date": "2020-11-18T11:35:39.917000", "content": "STOP PLS!!!
", "votes": -8, "replies": [] }, { "id": 1101447, "author_name": "nyanp", "author_url": "", "post_date": "2020-12-03T22:53:18.930000", "content": "updated:
\nCV: 0.803 LB: 0.801 with single LightGBM
", "votes": 5, "replies": [] }, { "id": 1101478, "author_name": "Caleb", "author_url": "", "post_date": "2020-12-03T23:51:44.653000", "content": "Nice result - how much do you think is related to:
\nThank you
", "votes": 0, "replies": [] }, { "id": 1101536, "author_name": "Nick Sarris", "author_url": "", "post_date": "2020-12-04T02:03:27.743000", "content": "How many rows are you training with? I'm currently at a CV/LB of 0.793/0.789 with a single LGB and 10M rows, and trying to figure out how much further I can push it before upping the number of rows.
", "votes": 1, "replies": [] }, { "id": 1101543, "author_name": "qiaqia", "author_url": "", "post_date": "2020-12-04T02:13:10.890000", "content": "amazing work
", "votes": 0, "replies": [] }, { "id": 1101577, "author_name": "nyanp", "author_url": "", "post_date": "2020-12-04T03:25:19.270000", "content": "I always use full dataset for submission. In my case, using full datasets increases CV by +0.006 compared to 6M rows. All hyperparameters except for num_leaves and learning_rate (for speedup training) remain the default. I think feature engineering on the solid validation is much more important than model parameters in case of GBDT.
", "votes": 14, "replies": [] }, { "id": 1101580, "author_name": "qiaqia", "author_url": "", "post_date": "2020-12-04T03:29:35.460000", "content": "Perfect Feature Engineering๐๐๐
", "votes": 1, "replies": [] }, { "id": 1102849, "author_name": "mamas", "author_url": "", "post_date": "2020-12-05T11:48:14.457000", "content": "nyanp, did you really change num_leaves for full data training? or do you mean nround?
I forgot about the nround, it's based on early stopping, so of course it's different for full rows and 6M rows. As for the other parameters, I adjusted the num_leaves and learning_rate in 6M rows and used the same parameters in the full data as well.
", "votes": 4, "replies": [] }, { "id": 1102865, "author_name": "mamas", "author_url": "", "post_date": "2020-12-05T12:12:55.537000", "content": "Ah ok thanks, I got it!
", "votes": 0, "replies": [] } ] }, { "id": 1071733, "author_name": "kurupical", "author_url": "", "post_date": "2020-11-07T10:47:21.740000", "content": "CV: 0.777
\nLB: 0.784
I have at the moment 0.7579 on validation and 0.763 on LB. I'm validating only against unseen users, and that's probably why it is quite higher on LB.
\nEDIT: I'm using 5 M rows, 33% of them as validation.
", "votes": 4, "replies": [] }, { "id": 1127999, "author_name": "HAO", "author_url": "", "post_date": "2020-12-27T05:08:31.843000", "content": "Same as this cv strategy.
\nLocal: 0.8027, LB: 0.805x
My last 3 submissions (using holdout validation set):
\nLocal: 0.756 LB: 0.758
\nLocal: 0.748 LB: 0.751
\nLocal: 0.741 LB: 0.742
CV: 0.778 - LB: 0.779
", "votes": 1, "replies": [] }, { "id": 1090124, "author_name": "qiaqia", "author_url": "", "post_date": "2020-11-25T05:11:11.037000", "content": "some last models:
\nCV 0.756//LB0.759
\nCV0.770//LB0.766
\nCV0.771//LB0.769
\nCV0.774//LB0.772
\nCV0.777//LB0.774
\nit Looks like the LB score is fluctuating around .003.
\nsingle LGBM LB0.801 sounds so magic,need more features..๐ช๐ช๐ช
but, seems you are lucky that you received a consistent improvement.
", "votes": 0, "replies": [] }, { "id": 1125319, "author_name": "qiaqia", "author_url": "", "post_date": "2020-12-24T16:13:37.280000", "content": "update my features style ,now CV 0.781,LB0.781
", "votes": 0, "replies": [] } ] }, { "id": 1079477, "author_name": "Ken Ho", "author_url": "", "post_date": "2020-11-16T05:37:12.520000", "content": "Local: 0.7638 LB 0.765
\nUsing all training data and follow the cross-validation strategy in https://www.kaggle.com/its7171/cv-strategy
", "votes": 1, "replies": [ { "id": 1089909, "author_name": "Abdessalem Boukil", "author_url": "", "post_date": "2020-11-24T22:16:51.303000", "content": "I'm using the same cross-validation strategy, however I'm experiencing a gap in LB vs Local (0.757 vs 0.773). I noticed this gap exists only when I used cumulative user features. Do you experience the same ? Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 1078828, "author_name": "Catadanna", "author_url": "", "post_date": "2020-11-15T11:02:56.707000", "content": "Local CV: 0.740, LB: 0.727
", "votes": 1, "replies": [] }, { "id": 1075135, "author_name": "Nikola Bacic", "author_url": "", "post_date": "2020-11-11T12:23:35.723000", "content": "CV: 0.763 LB: 0.771 with simple split by user_id
", "votes": 1, "replies": [ { "id": 1089910, "author_name": "Abdessalem Boukil", "author_url": "", "post_date": "2020-11-24T22:17:27.473000", "content": "Can you tell me how did you split by user_id ? Thanks!
", "votes": 0, "replies": [] }, { "id": 1104145, "author_name": "Nikola Bacic", "author_url": "", "post_date": "2020-12-06T16:51:39.337000", "content": "Update:
\nvalidation: 0.8064 LB: 0.808x
but at what costโฆ
", "votes": 2, "replies": [] } ] }, { "id": 1064367, "author_name": "HAO", "author_url": "", "post_date": "2020-10-30T03:42:16.150000", "content": "Local: 0.79305
\nLB: 0.792
Hi,
\nLocal : 0.733
\nLB: 0.758
Pretty classic validation scheme:
\nMy training set contains about 10M rows, 3M for validation. Features are aggretated on the remaining rows
The gap is similar to what @misfyre reported.
", "votes": 0, "replies": [] }, { "id": 1063145, "author_name": "Alex", "author_url": "", "post_date": "2020-10-28T14:55:56.747000", "content": "@vopani I'm impressed by the strong correlation between your local validation and public LB. Are you using something similar to @misfyre ?
\nI've built my hold-out set in the same way so it makes sense that there is a similar gap. However I cannot reprocude his jump from 0.725 to 0.747 ๐
", "votes": 0, "replies": [] }, { "id": 1063174, "author_name": "Vopani", "author_url": "", "post_date": "2020-10-28T15:38:20.943000", "content": "Many others in this thread have reported much closer (and higher) correlation.
\nI just used a small random sample to start with (only made 5 successful submissions), so yet to explore how different validation setups work.
", "votes": 1, "replies": [] } ] }, { "id": 1083238, "author_name": "Ravox", "author_url": "", "post_date": "2020-11-18T18:35:58.437000", "content": "I'm seeing very strong correlation between CV and LB so far, by using out-of-time validation and validating on the last 2M rows.
\nSome of my last models:
\nCV: 75.9 // LB: 76.0
\nCV: 77.0 // LB 76.9
\nCV: 77.6 // LB: 77.4
Haven't yet made the sub but the below stats is using 1M/8M rows for training and 2.5M rows for validation with 16 feats. Thanks a lot everyone for the inspirations!
\ntraining's auc: 0.769886 valid's auc: 0.765254 # 1M train, 2.5M rows val\n\ntraining's auc: 0.770383 valid's auc: 0.766083 # 8M train, 2.5M rows val\nSo i have finally made a sub for a lgbm trained with 8M rows, ~LB .762/.763 now.
", "votes": 0, "replies": [] } ] }, { "id": 1071976, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-11-07T17:09:38.927000", "content": "Someone has a perfect 1.0 score on the LB (as it seems for now). [it's a glitch i hope, no point downvoting me] bug
\n
Very suspiciousโฆ
", "votes": 0, "replies": [] }, { "id": 1071988, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-11-07T17:27:01.507000", "content": "I hope so, not sure why people would downvote, i just shared the information and TBH was amazed to see. Refreshed the page multiple times as well ๐
", "votes": 1, "replies": [] }, { "id": 1071989, "author_name": "Joe Eddy", "author_url": "", "post_date": "2020-11-07T17:28:04.103000", "content": "This person has kindly shared a bug here: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/195815. Presumably it will just be fixed. It's not foul play, more like a white hat hacking sort of situation from what I can tell.
", "votes": 2, "replies": [] }, { "id": 1071998, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-11-07T17:35:46.010000", "content": "edit-> placeholder [deleted]
", "votes": 0, "replies": [] } ] }, { "id": 1061720, "author_name": "", "author_url": "", "post_date": "2020-10-27T08:58:21.313000", "content": "", "votes": -11, "replies": [] }, { "id": 1061719, "author_name": "", "author_url": "", "post_date": "2020-10-27T08:58:00.180000", "content": "", "votes": -11, "replies": [] }, { "id": 1109706, "author_name": "", "author_url": "", "post_date": "2020-12-12T01:30:00.810000", "content": "", "votes": 0, "replies": [ { "id": 1109745, "author_name": "", "author_url": "", "post_date": "2020-12-12T02:59:22.517000", "content": "", "votes": 0, "replies": [] }, { "id": 1109772, "author_name": "", "author_url": "", "post_date": "2020-12-12T03:37:27.487000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1105943, "author_name": "", "author_url": "", "post_date": "2020-12-08T11:03:03.023000", "content": "", "votes": 0, "replies": [ { "id": 1115760, "author_name": "", "author_url": "", "post_date": "2020-12-16T14:57:09.357000", "content": "", "votes": 1, "replies": [] }, { "id": 1120677, "author_name": "", "author_url": "", "post_date": "2020-12-21T03:41:29.830000", "content": "", "votes": 0, "replies": [] }, { "id": 1120915, "author_name": "", "author_url": "", "post_date": "2020-12-21T07:41:33.670000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1105028, "author_name": "", "author_url": "", "post_date": "2020-12-07T13:21:53.483000", "content": "", "votes": 0, "replies": [ { "id": 1105526, "author_name": "", "author_url": "", "post_date": "2020-12-08T01:09:38.147000", "content": "", "votes": 0, "replies": [] }, { "id": 1105673, "author_name": "", "author_url": "", "post_date": "2020-12-08T04:36:46.043000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 1073485, "author_name": "", "author_url": "", "post_date": "2020-11-09T15:08:36.427000", "content": "", "votes": 0, "replies": [ { "id": 1073562, "author_name": "", "author_url": "", "post_date": "2020-11-09T16:20:30.600000", "content": "", "votes": 7, "replies": [] }, { "id": 1073574, "author_name": "", "author_url": "", "post_date": "2020-11-09T16:33:35.817000", "content": "", "votes": 0, "replies": [] }, { "id": 1073578, "author_name": "", "author_url": "", "post_date": "2020-11-09T16:41:26.523000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 1064345, "author_name": "", "author_url": "", "post_date": "2020-10-30T03:00:01.670000", "content": "", "votes": 0, "replies": [] }, { "id": 1063886, "author_name": "", "author_url": "", "post_date": "2020-10-29T13:06:14.153000", "content": "", "votes": 0, "replies": [ { "id": 1063896, "author_name": "", "author_url": "", "post_date": "2020-10-29T13:18:04.617000", "content": "", "votes": 4, "replies": [] }, { "id": 1064553, "author_name": "", "author_url": "", "post_date": "2020-10-30T08:41:15.093000", "content": "", "votes": 0, "replies": [] }, { "id": 1064680, "author_name": "", "author_url": "", "post_date": "2020-10-30T12:05:39.127000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 1058805, "author_name": "", "author_url": "", "post_date": "2020-10-24T10:03:28.310000", "content": "", "votes": 0, "replies": [ { "id": 1066778, "author_name": "", "author_url": "", "post_date": "2020-11-02T03:00:53.600000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 1076266, "author_name": "", "author_url": "", "post_date": "2020-11-12T11:38:37.167000", "content": "", "votes": 0, "replies": [] }, { "id": 1062749, "author_name": "", "author_url": "", "post_date": "2020-10-28T06:43:10.823000", "content": "", "votes": -1, "replies": [ { "id": 1063014, "author_name": "", "author_url": "", "post_date": "2020-10-28T12:33:03.190000", "content": "", "votes": -1, "replies": [ { "id": 1063086, "author_name": "", "author_url": "", "post_date": "2020-10-28T13:49:50.130000", "content": "", "votes": 1, "replies": [] } ] } ] }, { "id": 1060810, "author_name": "", "author_url": "", "post_date": "2020-10-26T14:51:45.027000", "content": "", "votes": -5, "replies": [ { "id": 1060857, "author_name": "", "author_url": "", "post_date": "2020-10-26T15:17:49.560000", "content": "", "votes": 0, "replies": [] }, { "id": 1060863, "author_name": "", "author_url": "", "post_date": "2020-10-26T15:23:29.893000", "content": "", "votes": 0, "replies": [] }, { "id": 1060868, "author_name": "", "author_url": "", "post_date": "2020-10-26T15:30:48.447000", "content": "", "votes": 2, "replies": [] }, { "id": 1061170, "author_name": "", "author_url": "", "post_date": "2020-10-26T19:33:40.467000", "content": "", "votes": 2, "replies": [] } ] } ], "raw_markdown_by_id": { "1058728": "With quite a large dataset at hand and a unique submission format there can be a lot of interesting ideas, models and validation approaches that can be used.\n\nLarge datasets often give a stable performance and so far my experience has been positive.\n\nOpening this thread for anyone willing to share their validation / leaderboard approaches and / or scores.", "1112090": "- CV: 0.80664, LB: 0.806\n- Single SAINT-based model\n- without lectures", "1059325": "Last LB score increases:\n\nLocal: **0.7609** LB: **0.759**\nLocal: **0.7757** LB: **0.772**\nLocal: **0.7817** LB: **0.780**\nLocal: **0.7866** LB: **0.786**\n\nI'm using 10m rows to train and 2.5m to validate per fold.", "1058744": "Regarding CV/LB scores, my last three jumps on the LB were the following:\n\nCV: **0.725487**, LB: **0.761**\nCV: **0.740906**, LB: **0.767**\nCV: **0.747638**, LB: **0.772**\n\nMy approach to validation is pretty simple, just the following to create the training/validation set. I use the remaining 90+% of the data for feature aggregates and the like. Even though the CV score isn't too similar to the LB, an increase in CV typically comes with an increase in LB:\n\n```\nprint(\"[1] Create Training Set\")\ntraining = combined_df.groupby(\"user_id\").tail(24)\ncombined_df = combined_df.drop(training.index)\n\nprint(\"[2] Split Training Set into Validation\")\nvalidation = training.groupby(\"user_id\").tail(6)\ntraining = training.drop(validation.index)\n```\n\nUpdate:\n\nCV: **0.790365**, LB: **0.787**\n\nI switched up my validation strategy to instead use the one described in https://www.kaggle.com/its7171/cv-strategy. Seems to be relatively accurate from what I've noticed throughout the thread, typically <= 0.003 of a difference beween CV/LB. This score is calculated using 10M rows for training, 2.5M rows for validation. I'm not using any lecture features yet in the implementation.\n\nUpdate #2:\n\nCV: **0.793102**, LB: **0.789**\n\nAdded a few more features, changed the framework of my code throughout to make it much easier to add/test new features. Still using only 10M rows, trying to see if I can hit ~0.795+ before I really look into training with the entire training set, given the estimated 0.01 boost from other people.", "1058962": "Local Validation: **0.770**, LB: **0.766**.\nUpdate: Local **0.777**, LB: **0.772**. \n\nThat's with training on ~4% of the data, and validating on ~2%. My validation scheme is set up to mirror the way the test predictions need to be made, so I'm fairly confident in it. I think a bit of a gap is normal especially given the time element.\n\nI strongly agree with what Alex said below about prioritizing finding the right features before using more of the data. 2% is already millions of samples and likely large enough to give reliable model feedback IMO (and even close to the same order as the actual test set). The key thing early in a competition is to be able to iterate on feature/model ideas quickly, and I'm pretty sure training time scales above linear with # of training rows for the typical gradient boosting model.\n\nEdit: see update ", "1093890": "single NN, inference for ~3h\nI use [this](https://www.kaggle.com/its7171/cv-strategy) strategy.\nCV: 0.772, LB: 0.776", "1064170": "So far CV-LB correlation looks very stable - it's constantly +0.003 on LB compared to CV for all my submissions in the last 2 weeks.", "1071447": "CV: .792 \nLB: .791", "1117386": "CV: 0.802, LB: 0.802\n\nsingle lightgbm model with 100+ features, nothing found in lectures๐. Anyone boost score from lectures features?", "1122135": "CV: 0.808, LB: 0.805\n\nLooking at everyone has better CV-LB gap, I either have a bug in inference or something else is wrong.", "1071389": "Local: **0.7602** LB: **0.773**\nLocal: **0.7699** LB: **0.782**", "1071175": "Local:766\nLB:785", "1066774": "Local:763,LB:780\nUpdate: Local:771,LB:790", "1066232": "CV:0.798\nLB:0.771\nValidating method: GroupKFold by user_id.", "1058825": "My last two submissions were\n\nLocal CV: **0.7637**, LB: **0.767**\nLocal CV: **0.7567**, LB: **0.760**\n\nSo both times my LB was about 0.033 higher than my local CV score. Not sure why but I'll take it.", "1082908": "Local: 0.798 LB: not submitted (with time-series split)\n\nThe road to 1st submission is a long way for me ๐", "1071733": "CV: 0.777\nLB: 0.784", "1065768": "I have at the moment 0.7579 on validation and 0.763 on LB. I'm validating only against unseen users, and that's probably why it is quite higher on LB.\n\nEDIT: I'm using 5 M rows, 33% of them as validation.", "1127999": "Same as [this](https://www.kaggle.com/its7171/cv-strategy) cv strategy.\nLocal: 0.8027, LB: 0.805x", "1058729": "My last 3 submissions (using holdout validation set):\n\nLocal: **0.756** LB: **0.758**\nLocal: **0.748** LB: **0.751**\nLocal: **0.741** LB: **0.742**", "1103563": "CV: 0.778 - LB: 0.779", "1090124": "some last models:\nCV 0.756//LB0.759\nCV0.770//LB0.766\nCV0.771//LB0.769\nCV0.774//LB0.772\nCV0.777//LB0.774\nit Looks like the LB score is fluctuating around .003.\nsingle LGBM LB0.801 sounds so magic,need more features..๐ช๐ช๐ช", "1079477": "Local: **0.7638** LB **0.765**\n\nUsing all training data and follow the cross-validation strategy in https://www.kaggle.com/its7171/cv-strategy", "1078828": "Local CV: 0.740, LB: 0.727", "1075135": "CV: 0.763 LB: 0.771 with simple split by user_id", "1064367": "Local: 0.79305\nLB: 0.792", "1062828": "Hi,\nLocal : 0.733\nLB: 0.758\n\nPretty classic validation scheme:\nMy training set contains about 10M rows, 3M for validation. Features are aggretated on the remaining rows", "1083238": "I'm seeing very strong correlation between CV and LB so far, by using out-of-time validation and validating on the last 2M rows. \nSome of my last models:\nCV: 75.9 // LB: 76.0\nCV: 77.0 // LB 76.9\nCV: 77.6 // LB: 77.4 ", "1074330": "Haven't yet made the sub but the below stats is using 1M/8M rows for training and 2.5M rows for validation with 16 feats. Thanks a lot everyone for the inspirations!\n\n```\ntraining's auc: 0.769886 valid's auc: 0.765254 # 1M train, 2.5M rows val\n\ntraining's auc: 0.770383 valid's auc: 0.766083 # 8M train, 2.5M rows val\n```", "1071976": "Someone has a perfect 1.0 score on the LB (as it seems for now). [it's a glitch i hope, no point downvoting me] [bug](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/195815)\n\n", "1061720": "nice implementation", "1061719": "Nice implementation", "1109706": "CV 783 , LB : 783 , training with 7.5 million rows. However, my last big boost in cv was not a big boost in public leaderboard. I am working in geting a better data representation and looking for bugs in my last submissions.", "1105943": "1.CV 0.767 / LB 0.766\n2.CV 0.771 / LB 0.771", "1105028": "My CV is 0.756, but LB is 0.553. I can't find the reason.\nWhen I drop variables, which is min/max/std/mean/skew of answer data of each user ID, CV is 0.701 and LB is 0.682.\nI think maybe the additional variables are related with the problem, but I still don't know the exact reason.\nAnyone having the similar experience?", "1073485": "CV: 0.76 LB: 0.737\nCV: 0.92 LB: 0.702\n\nI don't get it, it makes no sense. My CV preparation is simple. I get the first 60% of the whole training set, for my training set, then do [ validation = X.groupby('user_id').tail(100) ]. I end up with 13 milion CV and 46 milion Train rows.\n\nI did switch from Pandas to cuDF in my submission notebook to do faster merging and all, so unless cuDF is doing some wild messed up merges, I have no idea why such a wild difference. Almost giving up", "1064345": "Im puzzled that with 10M trainset and 2.5M vset, Local: 0.7683 and LB: 0.69. I think I have got a overfitting dilemma.", "1063886": "My LB and Local are radically different, the local one is 0.752 and on the LB is 0.680. Any idea ideas on why is that ? ", "1058805": "There is another thread [191413](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191413) on validation", "1076266": "", "1062749": "", "1060810": "" } }