I'm kinda out of ideas how to improve my model.
\nSo Just out of curious. What's everyone's best score single model?
\nStart from myself.
[update]
\n0.790
\nsingle lgb
\ntraining on 25% training data, 40+ features
\nno post-process
I'm kinda out of ideas how to improve my model.
\nSo Just out of curious. What's everyone's best score single model?
\nStart from myself.
[update]
\n0.790
\nsingle lgb
\ntraining on 25% training data, 40+ features
\nno post-process
Single Transformer(?)-like model, 0.813 in CV and 0.812 in LB.
", "rawMarkdown": "Single Transformer(?)-like model, 0.813 in CV and 0.812 in LB.", "votes": 25, "replies": [ { "id": 1117941, "postDate": "2020-12-18T15:53:30.773Z", "content": "Reveal the mystery na 😅! You are doing amazing! Looking forward for a great discussion once the comp ends!
", "rawMarkdown": "Reveal the mystery na 😅! You are doing amazing! Looking forward for a great discussion once the comp ends!" }, { "id": 1118010, "postDate": "2020-12-18T16:52:27.230Z", "content": "Oh…, Great work!
\nWould u tell us your tree model's result, if you don't mind ?
It's a still top secret, but I can definitely say it's worse than 0.812!
", "rawMarkdown": "It's a still top secret, but I can definitely say it's worse than 0.812!", "votes": 1 }, { "id": 1118060, "postDate": "2020-12-18T17:29:13.473Z", "content": "Thank u for replying.
", "rawMarkdown": "Thank u for replying. " }, { "id": 1118542, "postDate": "2020-12-19T07:04:06.160Z", "content": "Hi, great score.
\nCan you share how big the model is as in d_model and num_layers?
\nAlso are you using lectures? :D
I also have a transformer-esque model that scores around 0.792 but currently only have d_model 128 and num_layers 4 due to training time constraints. I wanted to know how much an improvement should I expect moving to d_model 512.
", "rawMarkdown": "Hi, great score.\n\nCan you share how big the model is as in d_model and num_layers?\nAlso are you using lectures? :D\n\nI also have a transformer-esque model that scores around 0.792 but currently only have d_model 128 and num_layers 4 due to training time constraints. I wanted to know how much an improvement should I expect moving to d_model 512.", "votes": 4 }, { "id": 1118972, "postDate": "2020-12-19T15:38:59.073Z", "content": "I will explain the detail in the winning solution. But, I can say your question is nice and one of the key of this competition.
", "rawMarkdown": "I will explain the detail in the winning solution. But, I can say your question is nice and one of the key of this competition.", "votes": 12 }, { "id": 1118999, "postDate": "2020-12-19T16:09:20.347Z", "content": "@abdurrafae I mean that's impressive, with 128 as d_model, 3 layers encoder and decoder, I get 0.769 but when I double it to 256 it jumps to 0.778. Can you hint at the changements you made to your transformer model so it jumps to 0.792
", "rawMarkdown": "@abdurrafae I mean that's impressive, with 128 as d_model, 3 layers encoder and decoder, I get 0.769 but when I double it to 256 it jumps to 0.778. Can you hint at the changements you made to your transformer model so it jumps to 0.792", "votes": 2 }, { "id": 1119059, "postDate": "2020-12-19T17:21:41.817Z", "content": "@mamasinkgs Looking forward to that winning solution :)
\n@abdessalemboukil I'd guess you are only training on the last window_len samples of each user. Try using more of the data to train and that should help.
", "rawMarkdown": "@mamasinkgs Looking forward to that winning solution :)\n\n@abdessalemboukil I'd guess you are only training on the last window_len samples of each user. Try using more of the data to train and that should help.", "votes": 3 }, { "id": 1126188, "postDate": "2020-12-25T12:16:26.123Z", "content": "Hi @mamasinkgs, my transformer model has AUC 0.812 in CV, but only achieves 0.785 in LB. I suppose it is because during inference more than one response for one user is asked to predict. That leads to the user response chain not complete at the end. For example in the current batch, 3 response of a user need to be predicted, the first question can be added to the previous user response chain perfectly, but the latter two can not, as the status of the first response is not available. I think that is the only difference between CV and LB inference. Will that affect the scores so much? If so can you share some tips on how to deal with it?
", "rawMarkdown": "Hi @mamasinkgs, my transformer model has AUC 0.812 in CV, but only achieves 0.785 in LB. I suppose it is because during inference more than one response for one user is asked to predict. That leads to the user response chain not complete at the end. For example in the current batch, 3 response of a user need to be predicted, the first question can be added to the previous user response chain perfectly, but the latter two can not, as the status of the first response is not available. I think that is the only difference between CV and LB inference. Will that affect the scores so much? If so can you share some tips on how to deal with it? ", "votes": 1 }, { "id": 1128226, "postDate": "2020-12-27T09:11:20.017Z", "content": "\nI can't share detailed information, but I think no difficult technique is required to get to 0.810. What are truly necessary are stable validation, careful implementation for dataloader/model/training loop, and a stable system that avoids bug in inference.
", "rawMarkdown": "I can't share detailed information, but I think no difficult technique is required to get to 0.810. What are truly necessary are stable validation, careful implementation for dataloader/model/training loop, and a stable system that avoids bug in inference.", "votes": 4 }, { "id": 1128348, "postDate": "2020-12-27T11:11:16.180Z", "content": "As you all know, In this competition, it's terribly hard to achieve these simple things, though…
", "rawMarkdown": "As you all know, In this competition, it's terribly hard to achieve these simple things, though...", "votes": 2 }, { "id": 1128349, "postDate": "2020-12-27T11:11:50.870Z", "content": "I believe, Simple is always complex.
", "rawMarkdown": "I believe, Simple is always complex.", "votes": 1 }, { "id": 1128352, "postDate": "2020-12-27T11:17:45.727Z", "content": "Yes, I found my code has a bug a few hours ago.
", "rawMarkdown": "Yes, I found my code has a bug a few hours ago." }, { "id": 1132782, "postDate": "2020-12-30T17:23:45.880Z", "content": "running into memory issues with 128 size embeddings and single transformer block. how did u achieve 256 and 512 with multiple layers? seq length is also 100. Anything beyond throws cuda error. Any tips?
", "rawMarkdown": "running into memory issues with 128 size embeddings and single transformer block. how did u achieve 256 and 512 with multiple layers? seq length is also 100. Anything beyond throws cuda error. Any tips?" }, { "id": 1135835, "postDate": "2021-01-02T14:50:30.693Z", "content": "Amazing model.👍
", "rawMarkdown": "Amazing model.👍" }, { "id": 1136808, "postDate": "2021-01-03T13:00:43.773Z", "content": "Can't get past 0.805. Look forward to seeing your solution.
", "rawMarkdown": "Can't get past 0.805. Look forward to seeing your solution." } ] }, { "id": 1116607, "postDate": "2020-12-17T10:14:37.107Z", "content": "0.786 Single LGB model with about 14-16 features.
", "rawMarkdown": "0.786 Single LGB model with about 14-16 features.", "votes": 10, "replies": [ { "id": 1116952, "postDate": "2020-12-17T15:28:05.017Z", "content": "this is impressive
", "rawMarkdown": "this is impressive" }, { "id": 1126645, "postDate": "2020-12-25T18:46:00.927Z", "content": "Thanks to my teammates! ❤️ Updated score is like .790 Single LGB model with 39 features. We are looking for a 4th member, so if you have a good single model score (would be great if it's a NN), please feel free to reach out while we have time!
\nEdits-: Inference pipeline is less than 4 hours.
", "rawMarkdown": "Thanks to my teammates! ❤️ Updated score is like .790 Single LGB model with 39 features. We are looking for a 4th member, so if you have a good single model score (would be great if it's a NN), please feel free to reach out while we have time!\n\nEdits-: Inference pipeline is less than 4 hours.", "votes": 1 } ] }, { "id": 1118044, "postDate": "2020-12-18T17:17:57.160Z", "content": "LGB (single model), CV=0.791, LB=0.791
\nTransformer (single model), CV=0.7861, LB=0.792
\nInference pipeline working now in less than 4 hours (most problems came from memory pressure), most bug fixed, competition can start now 😃
how many features are you using?
", "rawMarkdown": "how many features are you using?" }, { "id": 1126729, "postDate": "2020-12-25T21:38:37Z", "content": "44 features
", "rawMarkdown": "44 features", "votes": 1 } ] }, { "id": 1126778, "postDate": "2020-12-25T23:33:36.190Z", "content": "model: single lgbm
\nnumber of features: 20
\ntraining data size: 15000000
\nvalid data size: 2500000
\nCv: 0.7874
\nLb: 0.786
\nThis kernel helps me a lot, https://www.kaggle.com/ragnar123/riiid-model-lgbm
2020/12/30 update
\nNo of features : 21
\nLb: 0.788
2021/01/06
\n26 features
\nTrain size 15M
\nCv: 0.792, lb: 0.792
0.790 (~0.793-0.794 when I submit again)
\nSingle LightGBM, 10% training data (10M rows), no post-processing
User features are definitely the way to go for further improvement, also loops if you're not doing so already. Loops in general make testing and generating features much, much easier.
\nEdit: Modifying a few things, training on the full dataset, 0.801 CV currently
", "rawMarkdown": "0.790 (~0.793-0.794 when I submit again) \nSingle LightGBM, 10% training data (10M rows), no post-processing\n\nUser features are definitely the way to go for further improvement, also loops if you're not doing so already. Loops in general make testing and generating features much, much easier.\n\nEdit: Modifying a few things, training on the full dataset, 0.801 CV currently", "votes": 6, "replies": [ { "id": 1116226, "postDate": "2020-12-17T01:10:23.853Z", "content": "thanks for the advice, already use loops. I think I need to work harder on user features
", "rawMarkdown": "thanks for the advice, already use loops. I think I need to work harder on user features" }, { "id": 1117084, "postDate": "2020-12-17T18:03:35.987Z", "content": "Question for who used LightGBM or any other boosting algorithm: If you use the same features used for boosting to create a neural network simple classifier of three, two layers, how much the ROC would be ?
\nThanks!
", "rawMarkdown": "Question for who used LightGBM or any other boosting algorithm: If you use the same features used for boosting to create a neural network simple classifier of three, two layers, how much the ROC would be ? \n\nThanks!" }, { "id": 1117942, "postDate": "2020-12-18T15:55:02.553Z", "content": "\n\n0.790 (~0.793-0.794 when I submit again) Single LightGBM, 10% training data (10M rows), no post-processing
\n
Awesome! And what do you get on full dataset?
", "rawMarkdown": ">0.790 (~0.793-0.794 when I submit again) Single LightGBM, 10% training data (10M rows), no post-processing\n\nAwesome! And what do you get on full dataset?" }, { "id": 1118142, "postDate": "2020-12-18T19:12:51.917Z", "content": "Haven't tried, as I'll probably need to use some GCP credits. Hoping for a ~0.01 boost
", "rawMarkdown": "Haven't tried, as I'll probably need to use some GCP credits. Hoping for a ~0.01 boost" } ] }, { "id": 1136455, "postDate": "2021-01-03T04:56:06.223Z", "content": "Transformer model cv 0.797/ Lb 0.799
\nThe model was modified from great public notebook.
Which one?
", "rawMarkdown": "Which one?" }, { "id": 1136817, "postDate": "2021-01-03T13:06:17.363Z", "content": "I only modified the model architecture and SAKTDataset. My code is 80% same as following great notebook.
\nSAINT model example:
\nhttps://www.kaggle.com/m10515009/saint-model-training-lb-0-784/
0.809 validation score with single Saint+ model with additional feature on top of the above wonderful kernels. I submitted few hours ago, workers are spending a little extra time now. Let's see how LB goes,,,
\nepoch - 26 val_loss - 0.49171 acc - 0.75899 auc - 0.80944\nBest model with val auc: 0.8094391722949805\nloss - 0.4951: 100%\n4270/4270 [33:36<00:00, 2.12it/s]\n\nloss - 0.4912: 100%\n478/478 [00:45<00:00, 10.56it/s]\n\nepoch - 27 val_loss - 0.49159 acc - 0.75943 auc - 0.80968\nBest model with val auc: 0.8096761173613556\nloss - 0.5043: 100%\n4270/4270 [22:50<00:00, 3.12it/s]\n\nloss - 0.4873: 100%\n478/478 [02:15<00:00, 3.54it/s]\n\nepoch - 28 val_loss - 0.49179 acc - 0.75955 auc - 0.80980\nloss - 0.4687: 100%\n4270/4270 [12:06<00:00, 5.88it/s]\n\nloss - 0.4404: 100%\n478/478 [01:46<00:00, 4.47it/s]\n\nepoch - 29 val_loss - 0.49154 acc - 0.75959 auc - 0.80998\nBest model with val auc: 0.8099845981257148\nWow, if the validation is accurate, that'd be one hell of a model, good job!
", "rawMarkdown": "Wow, if the validation is accurate, that'd be one hell of a model, good job!" }, { "id": 1143255, "postDate": "2021-01-07T20:11:02.803Z", "content": "Still submission is going,,, Praying my validation is correct and wish no timeout 😹
", "rawMarkdown": "Still submission is going,,, Praying my validation is correct and wish no timeout 😹" }, { "id": 1143351, "postDate": "2021-01-07T21:17:56.450Z", "content": "Well, it turned out leakages.
\nGot LB 776 🙀
That's terrible man! Always submit at least week ago!
", "rawMarkdown": "That's terrible man! Always submit at least week ago!", "votes": 1 } ] }, { "id": 1118864, "postDate": "2020-12-19T13:25:38.223Z", "content": "Transformer model, CV: 0.790 and LB: 0.797
", "rawMarkdown": "Transformer model, CV: 0.790 and LB: 0.797", "votes": 3 }, { "id": 1117029, "postDate": "2020-12-17T16:54:05.133Z", "content": "Single model : 785 in lb, 787 in cv. 7 million rows. I have more ideas to test and hope to get a boost using all data :/
", "rawMarkdown": "Single model : 785 in lb, 787 in cv. 7 million rows. I have more ideas to test and hope to get a boost using all data :/ ", "votes": 3 }, { "id": 1116502, "postDate": "2020-12-17T08:06:48.710Z", "content": "0.789 Single LGB model with about 47 features in particular.
", "rawMarkdown": "0.789 Single LGB model with about 47 features in particular.", "votes": 3, "replies": [ { "id": 1116951, "postDate": "2020-12-17T15:27:54.880Z", "content": "wow, how many data you use to train then?
", "rawMarkdown": "wow, how many data you use to train then?" }, { "id": 1117462, "postDate": "2020-12-18T04:57:04.627Z", "content": "Entire data while training the model but to check the features i think 30M is enough or less than that
", "rawMarkdown": "Entire data while training the model but to check the features i think 30M is enough or less than that", "votes": 1 } ] }, { "id": 1132503, "postDate": "2020-12-30T12:57:53.073Z", "content": "saint+ model cv 0.795 lb 0.795
", "rawMarkdown": "saint+ model cv 0.795 lb 0.795", "votes": 4 }, { "id": 1125527, "postDate": "2020-12-24T19:32:33.740Z", "content": "I am late to the party but I have a few things going on:
\nI have a simple model where I am just trying to diagnose and make sure my pipeline is set up correctly. My first submission was CV = 0.772 / LB = 0.742. There were quite a few logic flaws using the test API and I just fixed all of them so let me see if it aligns more closely now.
If all goes well with (1) I have a single LGBM model with CV = 0.808 with full training data and about 30 features. I have quite a ways to go though before I get all the pipeline right but I have a good feeling about it.
Single LGB model with about 50 features, CV:0.784 and LB:0.787
", "rawMarkdown": "Single LGB model with about 50 features, CV:0.784 and LB:0.787", "votes": 4 }, { "id": 1131075, "postDate": "2020-12-29T14:30:47.173Z", "content": "Saint + like model, 0.797
", "rawMarkdown": "Saint + like model, 0.797", "votes": 1 }, { "id": 1130598, "postDate": "2020-12-29T06:50:25.873Z", "content": "model : lgbm
\ninference : ~3h
\ncv : .789 lb : .789
Was curious if lgb equally strong or similar to saint can give more upon ensemble ,so far I have got more after mixing weak and strong saint model
", "rawMarkdown": "Was curious if lgb equally strong or similar to saint can give more upon ensemble ,so far I have got more after mixing weak and strong saint model", "votes": 1 }, { "id": 1122026, "postDate": "2020-12-22T06:21:55.470Z", "content": "Sir, may I ask a question that how to avoid memory/time error using such big amount of features.
", "rawMarkdown": "Sir, may I ask a question that how to avoid memory/time error using such big amount of features.", "votes": 1, "replies": [ { "id": 1123201, "postDate": "2020-12-23T03:07:37.317Z", "content": "I processed data chunk by chunk, these are also many sharing and discussion about memory and speed.
", "rawMarkdown": "I processed data chunk by chunk, these are also many sharing and discussion about memory and speed." } ] }, { "id": 1117423, "postDate": "2020-12-18T03:18:56.870Z", "content": "Single model: 0.782 with transformer model
", "rawMarkdown": "Single model: 0.782 with transformer model", "votes": 1 }, { "id": 1116836, "postDate": "2020-12-17T13:55:30.763Z", "content": "What do you mean by no post-processing ?
", "rawMarkdown": "What do you mean by no post-processing ? ", "votes": 1, "replies": [ { "id": 1116950, "postDate": "2020-12-17T15:27:28.520Z", "content": "just directly use the mode prediction result
", "rawMarkdown": "just directly use the mode prediction result", "votes": 1 } ] }, { "id": 1116501, "postDate": "2020-12-17T08:06:48.710Z", "content": "0.789 Single LGB model with about 47 features in particular.
", "rawMarkdown": "0.789 Single LGB model with about 47 features in particular.", "votes": 1 }, { "id": 1136272, "postDate": "2021-01-02T22:41:58.383Z", "content": "My current score (0.795 lb) is with a single model trained on ~1% of the data (1m rows). Planning to spend the last 5 days training with all of the data, hope it'll be enough time.
", "rawMarkdown": "My current score (0.795 lb) is with a single model trained on ~1% of the data (1m rows). Planning to spend the last 5 days training with all of the data, hope it'll be enough time.", "votes": 2, "replies": [ { "id": 1136414, "postDate": "2021-01-03T03:51:39.707Z", "content": "It's impressive.
\nGreat work!
great work! which type of model you are using?
", "rawMarkdown": "great work! which type of model you are using?" } ] }, { "id": 1131079, "postDate": "2020-12-29T14:32:56.550Z", "content": "model: lgb (34 features)
\ninference: 3hr
\ncv: 0.792 lb: 0.791
Single LGBM with 38 features using 10% of training data. LB: 0.786
", "rawMarkdown": "Single LGBM with 38 features using 10% of training data. LB: 0.786 ", "votes": 2 }, { "id": 1143508, "postDate": "2021-01-07T23:53:30.840Z", "content": "cv790/lb793,transformer model，however i ran again yesterday, and now i found that cv boost 2k, frustratingly, i dont have submission times
", "rawMarkdown": "cv790/lb793,transformer model，however i ran again yesterday, and now i found that cv boost 2k, frustratingly, i dont have submission times" }, { "id": 1143195, "postDate": "2021-01-07T19:35:36.183Z", "content": "0.792 lb 0.793 cv single LGBM 23 features
", "rawMarkdown": "0.792 lb 0.793 cv single LGBM 23 features" }, { "id": 1126504, "postDate": "2020-12-25T16:58:49.070Z", "content": "Single LGBM, 15 features, 15% of training data, CV: 0.780.
\nMy submission is just a fork of a public Notebook, i've not submitted anything of my own yet, and i probably won't untill CV reaches to at least 0.790.
", "rawMarkdown": "Single LGBM, 15 features, 15% of training data, CV: 0.780. \n\nMy submission is just a fork of a public Notebook, i've not submitted anything of my own yet, and i probably won't untill CV reaches to at least 0.790." }, { "id": 1123396, "postDate": "2020-12-23T07:28:00.613Z", "content": "I just started and I'll update my status here. I'm using SAKT model with test set updates and cross-validation is groupkfold with 2 splits.
\nCV Score: AUC: 0.7423 Accuracy: 0.6775
\nLB Score: AUC: 0.766
Single Transformer(?)-like model, 0.813 in CV and 0.812 in LB.
", "votes": 25, "replies": [ { "id": 1117941, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-12-18T15:53:30.773000", "content": "Reveal the mystery na 😅! You are doing amazing! Looking forward for a great discussion once the comp ends!
", "votes": 0, "replies": [] }, { "id": 1118010, "author_name": "kaerururu", "author_url": "", "post_date": "2020-12-18T16:52:27.230000", "content": "Oh…, Great work!
\nWould u tell us your tree model's result, if you don't mind ?
It's a still top secret, but I can definitely say it's worse than 0.812!
", "votes": 1, "replies": [] }, { "id": 1118060, "author_name": "kaerururu", "author_url": "", "post_date": "2020-12-18T17:29:13.473000", "content": "Thank u for replying.
", "votes": 0, "replies": [] }, { "id": 1118542, "author_name": "AbdurRafae", "author_url": "", "post_date": "2020-12-19T07:04:06.160000", "content": "Hi, great score.
\nCan you share how big the model is as in d_model and num_layers?
\nAlso are you using lectures? :D
I also have a transformer-esque model that scores around 0.792 but currently only have d_model 128 and num_layers 4 due to training time constraints. I wanted to know how much an improvement should I expect moving to d_model 512.
", "votes": 4, "replies": [] }, { "id": 1118972, "author_name": "mamas", "author_url": "", "post_date": "2020-12-19T15:38:59.073000", "content": "I will explain the detail in the winning solution. But, I can say your question is nice and one of the key of this competition.
", "votes": 12, "replies": [] }, { "id": 1118999, "author_name": "Abdessalem Boukil", "author_url": "", "post_date": "2020-12-19T16:09:20.347000", "content": "@abdurrafae I mean that's impressive, with 128 as d_model, 3 layers encoder and decoder, I get 0.769 but when I double it to 256 it jumps to 0.778. Can you hint at the changements you made to your transformer model so it jumps to 0.792
", "votes": 2, "replies": [] }, { "id": 1119059, "author_name": "AbdurRafae", "author_url": "", "post_date": "2020-12-19T17:21:41.817000", "content": "@mamasinkgs Looking forward to that winning solution :)
\n@abdessalemboukil I'd guess you are only training on the last window_len samples of each user. Try using more of the data to train and that should help.
", "votes": 3, "replies": [] }, { "id": 1126188, "author_name": "alexxu", "author_url": "", "post_date": "2020-12-25T12:16:26.123000", "content": "Hi @mamasinkgs, my transformer model has AUC 0.812 in CV, but only achieves 0.785 in LB. I suppose it is because during inference more than one response for one user is asked to predict. That leads to the user response chain not complete at the end. For example in the current batch, 3 response of a user need to be predicted, the first question can be added to the previous user response chain perfectly, but the latter two can not, as the status of the first response is not available. I think that is the only difference between CV and LB inference. Will that affect the scores so much? If so can you share some tips on how to deal with it?
", "votes": 1, "replies": [] }, { "id": 1128226, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-12-27T09:11:20.017000", "content": "\nI can't share detailed information, but I think no difficult technique is required to get to 0.810. What are truly necessary are stable validation, careful implementation for dataloader/model/training loop, and a stable system that avoids bug in inference.
", "votes": 4, "replies": [] }, { "id": 1128348, "author_name": "mamas", "author_url": "", "post_date": "2020-12-27T11:11:16.180000", "content": "As you all know, In this competition, it's terribly hard to achieve these simple things, though…
", "votes": 2, "replies": [] }, { "id": 1128349, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-12-27T11:11:50.870000", "content": "I believe, Simple is always complex.
", "votes": 1, "replies": [] }, { "id": 1128352, "author_name": "mamas", "author_url": "", "post_date": "2020-12-27T11:17:45.727000", "content": "Yes, I found my code has a bug a few hours ago.
", "votes": 0, "replies": [] }, { "id": 1132782, "author_name": "Allohvk", "author_url": "", "post_date": "2020-12-30T17:23:45.880000", "content": "running into memory issues with 128 size embeddings and single transformer block. how did u achieve 256 and 512 with multiple layers? seq length is also 100. Anything beyond throws cuda error. Any tips?
", "votes": 0, "replies": [] }, { "id": 1135835, "author_name": "HenryHZY", "author_url": "", "post_date": "2021-01-02T14:50:30.693000", "content": "Amazing model.👍
", "votes": 0, "replies": [] }, { "id": 1136808, "author_name": "Junyuan Ye", "author_url": "", "post_date": "2021-01-03T13:00:43.773000", "content": "Can't get past 0.805. Look forward to seeing your solution.
", "votes": 0, "replies": [] } ] }, { "id": 1116607, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-12-17T10:14:37.107000", "content": "0.786 Single LGB model with about 14-16 features.
", "votes": 10, "replies": [ { "id": 1116952, "author_name": "Jacob", "author_url": "", "post_date": "2020-12-17T15:28:05.017000", "content": "this is impressive
", "votes": 0, "replies": [] }, { "id": 1126645, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-12-25T18:46:00.927000", "content": "Thanks to my teammates! ❤️ Updated score is like .790 Single LGB model with 39 features. We are looking for a 4th member, so if you have a good single model score (would be great if it's a NN), please feel free to reach out while we have time!
\nEdits-: Inference pipeline is less than 4 hours.
", "votes": 1, "replies": [] } ] }, { "id": 1118044, "author_name": "MPWARE", "author_url": "", "post_date": "2020-12-18T17:17:57.160000", "content": "LGB (single model), CV=0.791, LB=0.791
\nTransformer (single model), CV=0.7861, LB=0.792
\nInference pipeline working now in less than 4 hours (most problems came from memory pressure), most bug fixed, competition can start now 😃
how many features are you using?
", "votes": 0, "replies": [] }, { "id": 1126729, "author_name": "MPWARE", "author_url": "", "post_date": "2020-12-25T21:38:37", "content": "44 features
", "votes": 1, "replies": [] } ] }, { "id": 1126778, "author_name": "pin-rui", "author_url": "", "post_date": "2020-12-25T23:33:36.190000", "content": "model: single lgbm
\nnumber of features: 20
\ntraining data size: 15000000
\nvalid data size: 2500000
\nCv: 0.7874
\nLb: 0.786
\nThis kernel helps me a lot, https://www.kaggle.com/ragnar123/riiid-model-lgbm
2020/12/30 update
\nNo of features : 21
\nLb: 0.788
2021/01/06
\n26 features
\nTrain size 15M
\nCv: 0.792, lb: 0.792
0.790 (~0.793-0.794 when I submit again)
\nSingle LightGBM, 10% training data (10M rows), no post-processing
User features are definitely the way to go for further improvement, also loops if you're not doing so already. Loops in general make testing and generating features much, much easier.
\nEdit: Modifying a few things, training on the full dataset, 0.801 CV currently
", "votes": 6, "replies": [ { "id": 1116226, "author_name": "Jacob", "author_url": "", "post_date": "2020-12-17T01:10:23.853000", "content": "thanks for the advice, already use loops. I think I need to work harder on user features
", "votes": 0, "replies": [] }, { "id": 1117084, "author_name": "Abdessalem Boukil", "author_url": "", "post_date": "2020-12-17T18:03:35.987000", "content": "Question for who used LightGBM or any other boosting algorithm: If you use the same features used for boosting to create a neural network simple classifier of three, two layers, how much the ROC would be ?
\nThanks!
", "votes": 0, "replies": [] }, { "id": 1117942, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-12-18T15:55:02.553000", "content": "\n\n0.790 (~0.793-0.794 when I submit again) Single LightGBM, 10% training data (10M rows), no post-processing
\n
Awesome! And what do you get on full dataset?
", "votes": 0, "replies": [] }, { "id": 1118142, "author_name": "Nick Sarris", "author_url": "", "post_date": "2020-12-18T19:12:51.917000", "content": "Haven't tried, as I'll probably need to use some GCP credits. Hoping for a ~0.01 boost
", "votes": 0, "replies": [] } ] }, { "id": 1136455, "author_name": "Dean", "author_url": "", "post_date": "2021-01-03T04:56:06.223000", "content": "Transformer model cv 0.797/ Lb 0.799
\nThe model was modified from great public notebook.
Which one?
", "votes": 0, "replies": [] }, { "id": 1136817, "author_name": "Dean", "author_url": "", "post_date": "2021-01-03T13:06:17.363000", "content": "I only modified the model architecture and SAKTDataset. My code is 80% same as following great notebook.
\nSAINT model example:
\nhttps://www.kaggle.com/m10515009/saint-model-training-lb-0-784/
0.809 validation score with single Saint+ model with additional feature on top of the above wonderful kernels. I submitted few hours ago, workers are spending a little extra time now. Let's see how LB goes,,,
\nepoch - 26 val_loss - 0.49171 acc - 0.75899 auc - 0.80944\nBest model with val auc: 0.8094391722949805\nloss - 0.4951: 100%\n4270/4270 [33:36<00:00, 2.12it/s]\n\nloss - 0.4912: 100%\n478/478 [00:45<00:00, 10.56it/s]\n\nepoch - 27 val_loss - 0.49159 acc - 0.75943 auc - 0.80968\nBest model with val auc: 0.8096761173613556\nloss - 0.5043: 100%\n4270/4270 [22:50<00:00, 3.12it/s]\n\nloss - 0.4873: 100%\n478/478 [02:15<00:00, 3.54it/s]\n\nepoch - 28 val_loss - 0.49179 acc - 0.75955 auc - 0.80980\nloss - 0.4687: 100%\n4270/4270 [12:06<00:00, 5.88it/s]\n\nloss - 0.4404: 100%\n478/478 [01:46<00:00, 4.47it/s]\n\nepoch - 29 val_loss - 0.49154 acc - 0.75959 auc - 0.80998\nBest model with val auc: 0.8099845981257148\nWow, if the validation is accurate, that'd be one hell of a model, good job!
", "votes": 0, "replies": [] }, { "id": 1143255, "author_name": "Jihun Lorenzo Park", "author_url": "", "post_date": "2021-01-07T20:11:02.803000", "content": "Still submission is going,,, Praying my validation is correct and wish no timeout 😹
", "votes": 0, "replies": [] }, { "id": 1143351, "author_name": "Jihun Lorenzo Park", "author_url": "", "post_date": "2021-01-07T21:17:56.450000", "content": "Well, it turned out leakages.
\nGot LB 776 🙀
That's terrible man! Always submit at least week ago!
", "votes": 1, "replies": [] } ] }, { "id": 1118864, "author_name": "Jungwoo Park", "author_url": "", "post_date": "2020-12-19T13:25:38.223000", "content": "Transformer model, CV: 0.790 and LB: 0.797
", "votes": 3, "replies": [] }, { "id": 1117029, "author_name": "Icfstat", "author_url": "", "post_date": "2020-12-17T16:54:05.133000", "content": "Single model : 785 in lb, 787 in cv. 7 million rows. I have more ideas to test and hope to get a boost using all data :/
", "votes": 3, "replies": [] }, { "id": 1116502, "author_name": "AK", "author_url": "", "post_date": "2020-12-17T08:06:48.710000", "content": "0.789 Single LGB model with about 47 features in particular.
", "votes": 3, "replies": [ { "id": 1116951, "author_name": "Jacob", "author_url": "", "post_date": "2020-12-17T15:27:54.880000", "content": "wow, how many data you use to train then?
", "votes": 0, "replies": [] }, { "id": 1117462, "author_name": "AK", "author_url": "", "post_date": "2020-12-18T04:57:04.627000", "content": "Entire data while training the model but to check the features i think 30M is enough or less than that
", "votes": 1, "replies": [] } ] }, { "id": 1132503, "author_name": "chizhu", "author_url": "", "post_date": "2020-12-30T12:57:53.073000", "content": "saint+ model cv 0.795 lb 0.795
", "votes": 4, "replies": [] }, { "id": 1125527, "author_name": "RDizzl3", "author_url": "", "post_date": "2020-12-24T19:32:33.740000", "content": "I am late to the party but I have a few things going on:
\nI have a simple model where I am just trying to diagnose and make sure my pipeline is set up correctly. My first submission was CV = 0.772 / LB = 0.742. There were quite a few logic flaws using the test API and I just fixed all of them so let me see if it aligns more closely now.
If all goes well with (1) I have a single LGBM model with CV = 0.808 with full training data and about 30 features. I have quite a ways to go though before I get all the pipeline right but I have a good feeling about it.
Single LGB model with about 50 features, CV:0.784 and LB:0.787
", "votes": 4, "replies": [] }, { "id": 1131075, "author_name": "Abdessalem Boukil", "author_url": "", "post_date": "2020-12-29T14:30:47.173000", "content": "Saint + like model, 0.797
", "votes": 1, "replies": [] }, { "id": 1130598, "author_name": "zakopuro", "author_url": "", "post_date": "2020-12-29T06:50:25.873000", "content": "model : lgbm
\ninference : ~3h
\ncv : .789 lb : .789
Was curious if lgb equally strong or similar to saint can give more upon ensemble ,so far I have got more after mixing weak and strong saint model
", "votes": 1, "replies": [] }, { "id": 1122026, "author_name": "Zhenghan Chen", "author_url": "", "post_date": "2020-12-22T06:21:55.470000", "content": "Sir, may I ask a question that how to avoid memory/time error using such big amount of features.
", "votes": 1, "replies": [ { "id": 1123201, "author_name": "Jacob", "author_url": "", "post_date": "2020-12-23T03:07:37.317000", "content": "I processed data chunk by chunk, these are also many sharing and discussion about memory and speed.
", "votes": 0, "replies": [] } ] }, { "id": 1117423, "author_name": "xiaxiao03", "author_url": "", "post_date": "2020-12-18T03:18:56.870000", "content": "Single model: 0.782 with transformer model
", "votes": 1, "replies": [] }, { "id": 1116836, "author_name": "Abdessalem Boukil", "author_url": "", "post_date": "2020-12-17T13:55:30.763000", "content": "What do you mean by no post-processing ?
", "votes": 1, "replies": [ { "id": 1116950, "author_name": "Jacob", "author_url": "", "post_date": "2020-12-17T15:27:28.520000", "content": "just directly use the mode prediction result
", "votes": 1, "replies": [] } ] }, { "id": 1116501, "author_name": "AK", "author_url": "", "post_date": "2020-12-17T08:06:48.710000", "content": "0.789 Single LGB model with about 47 features in particular.
", "votes": 1, "replies": [] }, { "id": 1136272, "author_name": "Ming Pan", "author_url": "", "post_date": "2021-01-02T22:41:58.383000", "content": "My current score (0.795 lb) is with a single model trained on ~1% of the data (1m rows). Planning to spend the last 5 days training with all of the data, hope it'll be enough time.
", "votes": 2, "replies": [ { "id": 1136414, "author_name": "Dean", "author_url": "", "post_date": "2021-01-03T03:51:39.707000", "content": "It's impressive.
\nGreat work!
great work! which type of model you are using?
", "votes": 0, "replies": [] } ] }, { "id": 1131079, "author_name": "Jacky", "author_url": "", "post_date": "2020-12-29T14:32:56.550000", "content": "model: lgb (34 features)
\ninference: 3hr
\ncv: 0.792 lb: 0.791
Single LGBM with 38 features using 10% of training data. LB: 0.786
", "votes": 2, "replies": [] }, { "id": 1143508, "author_name": "captainqxy", "author_url": "", "post_date": "2021-01-07T23:53:30.840000", "content": "cv790/lb793,transformer model，however i ran again yesterday, and now i found that cv boost 2k, frustratingly, i dont have submission times
", "votes": 0, "replies": [] }, { "id": 1143195, "author_name": "Alyona Pasevieva", "author_url": "", "post_date": "2021-01-07T19:35:36.183000", "content": "0.792 lb 0.793 cv single LGBM 23 features
", "votes": 0, "replies": [] }, { "id": 1126504, "author_name": "Bekir", "author_url": "", "post_date": "2020-12-25T16:58:49.070000", "content": "Single LGBM, 15 features, 15% of training data, CV: 0.780.
\nMy submission is just a fork of a public Notebook, i've not submitted anything of my own yet, and i probably won't untill CV reaches to at least 0.790.
", "votes": 0, "replies": [] }, { "id": 1123396, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2020-12-23T07:28:00.613000", "content": "I just started and I'll update my status here. I'm using SAKT model with test set updates and cross-validation is groupkfold with 2 splits.
\nCV Score: AUC: 0.7423 Accuracy: 0.6775
\nLB Score: AUC: 0.766