![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F502737%2F094f322cf5eb45ed1f6d8e82cae3b6a0%2Fdownload.png?generation=1716717063194160&alt=media\")
Well, if u take from 60s to 91s week_num from train dataset and make from that the holdout test dataset, the adversarial validation will show u this type of probability distributions by week_num's. So, the idea is too simple to exploit the metric hack, adversarial validation is worste to predict first weeks from test dataset. And simple probability shift on this samples is good for boosting ur public LB score. But it seems this type of simple metric hack exploiting doesn't work for later periods, and probably host knows that more data in private LB is later week_num's. Also the public notebooks with metric hacking with simple mask of probability < threshold found about 5 percents of total test size and this also proves observation above.
\n
Hope host make private LB with only laters week_num's. Just wanna look at huge shakeup!! 🤟
", "messages": [ { "id": "2837147", "postDate": "05/26/2024 10:01:21", "content": "Well, if u take from 60s to 91s week_num from train dataset and make from that the holdout test dataset, the adversarial validation will show u this type of probability distributions by week_num's. So, the idea is too simple to exploit the metric hack, adversarial validation is worste to predict first weeks from test dataset. And simple probability shift on this samples is good for boosting ur public LB score. But it seems this type of simple metric hack exploiting doesn't work for later periods, and probably host knows that more data in private LB is later week_num's. Also the public notebooks with metric hacking with simple mask of probability < threshold found about 5 percents of total test size and this also proves observation above.
\n
Hope host make private LB with only laters week_num's. Just wanna look at huge shakeup!! 🤟
", "rawMarkdown": "Well, if u take from 60s to 91s week_num from train dataset and make from that the holdout test dataset, the adversarial validation will show u this type of probability distributions by week_num's. So, the idea is too simple to exploit the metric hack, adversarial validation is worste to predict first weeks from test dataset. And simple probability shift on this samples is good for boosting ur public LB score. But it seems this type of simple metric hack exploiting doesn't work for later periods, and probably host knows that more data in private LB is later week_num's. Also the public notebooks with metric hacking with simple mask of probability < threshold found about 5 percents of total test size and this also proves observation above.\n\n\n\nHope host make private LB with only laters week_num's. Just wanna look at huge shakeup!! 🤟", "votes": null }, { "id": "2837213", "postDate": "05/26/2024 11:17:18", "content": "I also align with you on this, I think we are all collectively in for a gargantuan shakeup 2 days later @sggpls
", "rawMarkdown": "I also align with you on this, I think we are all collectively in for a gargantuan shakeup 2 days later @sggpls", "votes": null }, { "id": "2838211", "postDate": "05/26/2024 23:44:41", "content": "Thanks for sharing @sggpls. It will be interesting to see what the maximum score is in the private LB. My best ensemble without the hack could not score better than 0.594. Perhaps the limit might be below 6.00 which is way lower than all those useless submissions with the hack😀
", "rawMarkdown": "Thanks for sharing @sggpls. It will be interesting to see what the maximum score is in the private LB. My best ensemble without the hack could not score better than 0.594. Perhaps the limit might be below 6.00 which is way lower than all those useless submissions with the hack😀", "votes": null }, { "id": "2838402", "postDate": "05/27/2024 02:49:57", "content": "Our best ensemble without hacking failed to score above 0.599+. It is possible that the limit could be lower than 0.612 which is much lower than all applications with the hack.
", "rawMarkdown": "Our best ensemble without hacking failed to score above 0.599+. It is possible that the limit could be lower than 0.612 which is much lower than all applications with the hack.", "votes": null }, { "id": "2838418", "postDate": "05/27/2024 03:12:56", "content": "If we lower the accuracy in the first week, wouldn’t the score improve to some extent, whether it’s public or private? Or is the data from the week immediately following the training data excluded in the private dataset?
", "rawMarkdown": "If we lower the accuracy in the first week, wouldn’t the score improve to some extent, whether it’s public or private? Or is the data from the week immediately following the training data excluded in the private dataset?", "votes": null }, { "id": "2838482", "postDate": "05/27/2024 04:12:04", "content": "Maybe a dumb question, but why do the public notebook threshold hack actually work? I have given some thoughts but could not figure out. You mentioned \"simple mask of probability < threshold found about 5 percents of total test size\", where does the 5 percents come from?
\nCan someone kindly explain it? Really interested in the math behind it.🤔
I hope so ! It would be funny and the revenge of the organizers…🤣
", "rawMarkdown": "I hope so ! It would be funny and the revenge of the organizers…🤣", "votes": null }, { "id": "2838598", "postDate": "05/27/2024 06:06:18", "content": "I bet solutions based on the 2ndary model trained online for the hack will end up with OOM.. and it would be kind of sweet.. ,)
", "rawMarkdown": "I bet solutions based on the 2ndary model trained online for the hack will end up with OOM.. and it would be kind of sweet.. ,)", "votes": null }, { "id": "2838869", "postDate": "05/27/2024 08:38:42", "content": "Somewhere one of the hosts mentioned that the public and private split, both contain data from near covid to post covid. This was to make the public leaderboard reliable. Can't seem to find it now. That being said I still believe that the winners will have strong base model (0.60+) and a little thresholding on top. However, thresholds also have to be chosen very carefully based on some logic :) so that you are not just guessing. Anyway All the best to all of us who have worked hard to build a good base model. May the best model win (at least be in the top 10). Let's choose our two submissions carefully.
", "rawMarkdown": "Somewhere one of the hosts mentioned that the public and private split, both contain data from near covid to post covid. This was to make the public leaderboard reliable. Can't seem to find it now. That being said I still believe that the winners will have strong base model (0.60+) and a little thresholding on top. However, thresholds also have to be chosen very carefully based on some logic :) so that you are not just guessing. Anyway All the best to all of us who have worked hard to build a good base model. May the best model win (at least be in the top 10). Let's choose our two submissions carefully.", "votes": null }, { "id": "2838874", "postDate": "05/27/2024 08:43:53", "content": "if its scored its already has been computed for private data too, afaik
", "rawMarkdown": "if its scored its already has been computed for private data too, afaik", "votes": null }, { "id": "2838891", "postDate": "05/27/2024 08:53:44", "content": "That's a critical thing to know! May I ask how do you know it?
", "rawMarkdown": "That's a critical thing to know! May I ask how do you know it?", "votes": null }, { "id": "2838907", "postDate": "05/27/2024 09:05:12", "content": "If yes is the answer for the second ur question or it is most of the samples from first week in public than there is a shake up ;)
", "rawMarkdown": "If yes is the answer for the second ur question or it is most of the samples from first week in public than there is a shake up ;)", "votes": null }, { "id": "2838980", "postDate": "05/27/2024 10:12:05", "content": "I believe that latest public hack affected only first 10-20 weeks. If hosts used all weeks for random public/private split, hack will work on private too, may be less effective (within .001-.005). Otherwise it will be a huge shake up :)
\nBig chances that public hack not very far from hacking limit (based on my simulation) and this gives us some equality and now we are competing based on initial strength of our models.
My model without any attempts to lower score has around .605 on LB and I believe it could be higher
", "rawMarkdown": "I believe that latest public hack affected only first 10-20 weeks. If hosts used all weeks for random public/private split, hack will work on private too, may be less effective (within .001-.005). Otherwise it will be a huge shake up :)\nBig chances that public hack not very far from hacking limit (based on my simulation) and this gives us some equality and now we are competing based on initial strength of our models. \n\nMy model without any attempts to lower score has around .605 on LB and I believe it could be higher", "votes": null }, { "id": "2838986", "postDate": "05/27/2024 10:18:54", "content": "As I know, hosts can see both public and private scores before the deadline, which means scores are prepared for full test, but you can only see the score for public part as a participant
", "rawMarkdown": "As I know, hosts can see both public and private scores before the deadline, which means scores are prepared for full test, but you can only see the score for public part as a participant", "votes": null }, { "id": "2839009", "postDate": "05/27/2024 10:37:42", "content": "Really? Wanna see that comment. On the contrary, I saw the host's comment that they wouldn't disclose how they divided the test dataset into public and private.
", "rawMarkdown": "Really? Wanna see that comment. On the contrary, I saw the host's comment that they wouldn't disclose how they divided the test dataset into public and private.", "votes": null }, { "id": "2839027", "postDate": "05/27/2024 10:48:16", "content": "That's true. Changing the ratio of the samples from the first week is a simple and effective way to inactivate the popular hacking method (the one with adversarial validation). Let's see how this competition ends :D
", "rawMarkdown": "That's true. Changing the ratio of the samples from the first week is a simple and effective way to inactivate the popular hacking method (the one with adversarial validation). Let's see how this competition ends :D", "votes": null }, { "id": "2839041", "postDate": "05/27/2024 11:03:49", "content": "Found it. I hope this is the case. Otherwise, the shakeup will be huge :) . Not only in terms of thresholding but also in terms of base model variance of prediction in time.
\nhttps://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/473765#2638764
Thanks a ton! This info is so important.
", "rawMarkdown": "Thanks a ton! This info is so important.", "votes": null }, { "id": "2839082", "postDate": "05/27/2024 11:29:54", "content": "More likely that in this answer Tomas wrote about train dataset as it was open to public
", "rawMarkdown": "More likely that in this answer Tomas wrote about train dataset as it was open to public", "votes": null }, { "id": "2839090", "postDate": "05/27/2024 11:34:43", "content": "@johnpateha Read Daniel's comment later in the discussion :)
", "rawMarkdown": "johnpateha Read Daniel's comment later in the discussion :)", "votes": null }, { "id": "2839111", "postDate": "05/27/2024 11:46:50", "content": "Thank you, you are right. It could be different weeks too, but likely they used random split for each of week
", "rawMarkdown": "Thank you, you are right. It could be different weeks too, but likely they used random split for each of week", "votes": null }, { "id": "2839117", "postDate": "05/27/2024 11:52:07", "content": "yes, otherwise I might have chosen the wrong competition as my first tabular on Kaggle 🤣
", "rawMarkdown": "yes, otherwise I might have chosen the wrong competition as my first tabular on Kaggle 🤣", "votes": null }, { "id": "2839123", "postDate": "05/27/2024 11:53:41", "content": "And how about the hack of retrieving WEEK_NUM and using them to lower score for initial weeks? I remember you made a post about it.
", "rawMarkdown": "And how about the hack of retrieving WEEK_NUM and using them to lower score for initial weeks? I remember you made a post about it.", "votes": null }, { "id": "2839158", "postDate": "05/27/2024 12:11:27", "content": "There many ways to restore weeks for hack, but final results were not very different for my models, I prefer to build better model
", "rawMarkdown": "There many ways to restore weeks for hack, but final results were not very different for my models, I prefer to build better model", "votes": null }, { "id": "2839165", "postDate": "05/27/2024 12:14:35", "content": "Every competition teaches us something. No matter how this competition ends, in my ranking, it is far from the worst. It’s hard to knock the M5 off the podium.
", "rawMarkdown": "Every competition teaches us something. No matter how this competition ends, in my ranking, it is far from the worst. It’s hard to knock the M5 off the podium.", "votes": null }, { "id": "2839192", "postDate": "05/27/2024 12:22:35", "content": "I'm not sure if a 'better model' in Public LB is worth exploring. If the public/private used all weeks for both then yes a better model is worth building provided a good hack is implemented. Otherwise, if the private/public were split based on week_num (private is later weeks) then I think the results will be random. The train set has a positive slope (and that's why we all are getting good stability in our CVs), and the public set has a negative one (hence the need for hacking), so in that case will depend on the starting period chosen for the private set.
", "rawMarkdown": "I'm not sure if a 'better model' in Public LB is worth exploring. If the public/private used all weeks for both then yes a better model is worth building provided a good hack is implemented. Otherwise, if the private/public were split based on week_num (private is later weeks) then I think the results will be random. The train set has a positive slope (and that's why we all are getting good stability in our CVs), and the public set has a negative one (hence the need for hacking), so in that case will depend on the starting period chosen for the private set.", "votes": null }, { "id": "2839209", "postDate": "05/27/2024 12:34:11", "content": "if private starts after public the impact of hack would be not so big. cutting 5% from scores lead to decrease Gini by around 0.1 It's too much for shorter period.
\nIt's possible that split was done by weeks, for example 1st week public, 2-3 - private, 4 - public, etc. In that case hack will work on private too with some fluctuation. But fluctuation is possible even if each weeks was splitted 30 x 70
Agreed, sir.
", "rawMarkdown": "Agreed, sir.", "votes": null } ], "comments": [ { "id": 2837213, "author_name": "ravi20076", "author_url": "", "post_date": "05/26/2024 11:17:18", "content": "I also align with you on this, I think we are all collectively in for a gargantuan shakeup 2 days later @sggpls
", "votes": null, "replies": [] }, { "id": 2838211, "author_name": "sheriytm", "author_url": "", "post_date": "05/26/2024 23:44:41", "content": "Thanks for sharing @sggpls. It will be interesting to see what the maximum score is in the private LB. My best ensemble without the hack could not score better than 0.594. Perhaps the limit might be below 6.00 which is way lower than all those useless submissions with the hack😀
", "votes": null, "replies": [] }, { "id": 2838402, "author_name": "alexxanderlarko", "author_url": "", "post_date": "05/27/2024 02:49:57", "content": "Our best ensemble without hacking failed to score above 0.599+. It is possible that the limit could be lower than 0.612 which is much lower than all applications with the hack.
", "votes": null, "replies": [] }, { "id": 2838418, "author_name": "atsuno", "author_url": "", "post_date": "05/27/2024 03:12:56", "content": "If we lower the accuracy in the first week, wouldn’t the score improve to some extent, whether it’s public or private? Or is the data from the week immediately following the training data excluded in the private dataset?
", "votes": null, "replies": [ { "id": 2838907, "author_name": "sggpls", "author_url": "", "post_date": "05/27/2024 09:05:12", "content": "If yes is the answer for the second ur question or it is most of the samples from first week in public than there is a shake up ;)
", "votes": null, "replies": [ { "id": 2839027, "author_name": "atsuno", "author_url": "", "post_date": "05/27/2024 10:48:16", "content": "That's true. Changing the ratio of the samples from the first week is a simple and effective way to inactivate the popular hacking method (the one with adversarial validation). Let's see how this competition ends :D
", "votes": null, "replies": [] } ] } ] }, { "id": 2838482, "author_name": "shyhandsome", "author_url": "", "post_date": "05/27/2024 04:12:04", "content": "Maybe a dumb question, but why do the public notebook threshold hack actually work? I have given some thoughts but could not figure out. You mentioned \"simple mask of probability < threshold found about 5 percents of total test size\", where does the 5 percents come from?
\nCan someone kindly explain it? Really interested in the math behind it.🤔
I hope so ! It would be funny and the revenge of the organizers…🤣
", "votes": null, "replies": [] }, { "id": 2838598, "author_name": "lohmaa", "author_url": "", "post_date": "05/27/2024 06:06:18", "content": "I bet solutions based on the 2ndary model trained online for the hack will end up with OOM.. and it would be kind of sweet.. ,)
", "votes": null, "replies": [ { "id": 2838874, "author_name": "bluepill", "author_url": "", "post_date": "05/27/2024 08:43:53", "content": "if its scored its already has been computed for private data too, afaik
", "votes": null, "replies": [ { "id": 2838891, "author_name": "lohmaa", "author_url": "", "post_date": "05/27/2024 08:53:44", "content": "That's a critical thing to know! May I ask how do you know it?
", "votes": null, "replies": [ { "id": 2838986, "author_name": "bluepill", "author_url": "", "post_date": "05/27/2024 10:18:54", "content": "As I know, hosts can see both public and private scores before the deadline, which means scores are prepared for full test, but you can only see the score for public part as a participant
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2838869, "author_name": "krishnapriya18", "author_url": "", "post_date": "05/27/2024 08:38:42", "content": "Somewhere one of the hosts mentioned that the public and private split, both contain data from near covid to post covid. This was to make the public leaderboard reliable. Can't seem to find it now. That being said I still believe that the winners will have strong base model (0.60+) and a little thresholding on top. However, thresholds also have to be chosen very carefully based on some logic :) so that you are not just guessing. Anyway All the best to all of us who have worked hard to build a good base model. May the best model win (at least be in the top 10). Let's choose our two submissions carefully.
", "votes": null, "replies": [ { "id": 2839009, "author_name": "atsuno", "author_url": "", "post_date": "05/27/2024 10:37:42", "content": "Really? Wanna see that comment. On the contrary, I saw the host's comment that they wouldn't disclose how they divided the test dataset into public and private.
", "votes": null, "replies": [ { "id": 2839041, "author_name": "krishnapriya18", "author_url": "", "post_date": "05/27/2024 11:03:49", "content": "Found it. I hope this is the case. Otherwise, the shakeup will be huge :) . Not only in terms of thresholding but also in terms of base model variance of prediction in time.
\nhttps://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/473765#2638764
Thanks a ton! This info is so important.
", "votes": null, "replies": [] }, { "id": 2839082, "author_name": "johnpateha", "author_url": "", "post_date": "05/27/2024 11:29:54", "content": "More likely that in this answer Tomas wrote about train dataset as it was open to public
", "votes": null, "replies": [ { "id": 2839090, "author_name": "krishnapriya18", "author_url": "", "post_date": "05/27/2024 11:34:43", "content": "@johnpateha Read Daniel's comment later in the discussion :)
", "votes": null, "replies": [ { "id": 2839111, "author_name": "johnpateha", "author_url": "", "post_date": "05/27/2024 11:46:50", "content": "Thank you, you are right. It could be different weeks too, but likely they used random split for each of week
", "votes": null, "replies": [ { "id": 2839117, "author_name": "krishnapriya18", "author_url": "", "post_date": "05/27/2024 11:52:07", "content": "yes, otherwise I might have chosen the wrong competition as my first tabular on Kaggle 🤣
", "votes": null, "replies": [ { "id": 2839165, "author_name": "johnpateha", "author_url": "", "post_date": "05/27/2024 12:14:35", "content": "Every competition teaches us something. No matter how this competition ends, in my ranking, it is far from the worst. It’s hard to knock the M5 off the podium.
", "votes": null, "replies": [ { "id": 2839230, "author_name": "krishnapriya18", "author_url": "", "post_date": "05/27/2024 12:46:28", "content": "Agreed, sir.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] } ] }, { "id": 2838980, "author_name": "johnpateha", "author_url": "", "post_date": "05/27/2024 10:12:05", "content": "I believe that latest public hack affected only first 10-20 weeks. If hosts used all weeks for random public/private split, hack will work on private too, may be less effective (within .001-.005). Otherwise it will be a huge shake up :)
\nBig chances that public hack not very far from hacking limit (based on my simulation) and this gives us some equality and now we are competing based on initial strength of our models.
My model without any attempts to lower score has around .605 on LB and I believe it could be higher
", "votes": null, "replies": [ { "id": 2839123, "author_name": "simoelm", "author_url": "", "post_date": "05/27/2024 11:53:41", "content": "And how about the hack of retrieving WEEK_NUM and using them to lower score for initial weeks? I remember you made a post about it.
", "votes": null, "replies": [ { "id": 2839158, "author_name": "johnpateha", "author_url": "", "post_date": "05/27/2024 12:11:27", "content": "There many ways to restore weeks for hack, but final results were not very different for my models, I prefer to build better model
", "votes": null, "replies": [ { "id": 2839192, "author_name": "simoelm", "author_url": "", "post_date": "05/27/2024 12:22:35", "content": "I'm not sure if a 'better model' in Public LB is worth exploring. If the public/private used all weeks for both then yes a better model is worth building provided a good hack is implemented. Otherwise, if the private/public were split based on week_num (private is later weeks) then I think the results will be random. The train set has a positive slope (and that's why we all are getting good stability in our CVs), and the public set has a negative one (hence the need for hacking), so in that case will depend on the starting period chosen for the private set.
", "votes": null, "replies": [ { "id": 2839209, "author_name": "johnpateha", "author_url": "", "post_date": "05/27/2024 12:34:11", "content": "if private starts after public the impact of hack would be not so big. cutting 5% from scores lead to decrease Gini by around 0.1 It's too much for shorter period.
\nIt's possible that split was done by weeks, for example 1st week public, 2-3 - private, 4 - public, etc. In that case hack will work on private too with some fluctuation. But fluctuation is possible even if each weeks was splitted 30 x 70