Dear @jetakow, @tomasjeline2 and @inversion,
\nOver the last few days concerns have emerged on the hackability of the metric even with the transformed test data (post1, post2, post3). I hate to say it, but this situation, when repeated, is not going to have a positive impact on the flow of the competition.
\nCould you please look into the issue and provide us with some feedback?
", "messages": [ { "id": 2771621, "postDate": "2024-04-24T10:33:25.883Z", "content": "Dear @jetakow, @tomasjeline2 and @inversion,
\nOver the last few days concerns have emerged on the hackability of the metric even with the transformed test data (post1, post2, post3). I hate to say it, but this situation, when repeated, is not going to have a positive impact on the flow of the competition.
\nCould you please look into the issue and provide us with some feedback?
", "rawMarkdown": "Dear @jetakow, @tomasjeline2 and @inversion, \n\nOver the last few days concerns have emerged on the hackability of the metric even with the transformed test data ([post1](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/496898), [post2](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/497003), [post3](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/497167)). I hate to say it, but this situation, when repeated, is not going to have a positive impact on the flow of the competition. \n\nCould you please look into the issue and provide us with some feedback?\n\n", "votes": 50 }, { "id": 2774365, "postDate": "2024-04-25T06:58:07.873Z", "content": "Sad. As experienced user I decided not to submit at early stage of competition. Decided to wait. So after first metric hacking and period of fixing I downloaded the data first time. But something inside of me said to me wait more please ;) And it happened again. Very sad. I sympathize with the guys who worked the whole distance. Still, Kaggle is not about hacking, lb probing and other dirty tricks.
", "rawMarkdown": "Sad. As experienced user I decided not to submit at early stage of competition. Decided to wait. So after first metric hacking and period of fixing I downloaded the data first time. But something inside of me said to me wait more please ;) And it happened again. Very sad. I sympathize with the guys who worked the whole distance. Still, Kaggle is not about hacking, lb probing and other dirty tricks.", "votes": 6 }, { "id": 2796412, "postDate": "2024-05-06T08:02:36.090Z", "content": "Dear Kagglers,
\nThere have been questions about leaderboard scores and the possibility to hack our metric recently. So here is our long-awaited comment.
\nFirst, I want to clarify that we have quite limited options how to monitor and investigate top leaderboard positions – we don’t see submitted code (unless author makes it public), only predictions for test dataset. Therefore, we don’t directly see how Kagglers achieve their score, but we must admit that there are few instances that seem suspicious.
\nWe have approached few Kagglers who were willing to share their insights about possibilities how to exploit our dataset. There are few approaches and attributes that can be used. Important finding is that some of them are related to our business and how it evolves over time. As a results, it’s almost impossible to remove such kind of information unless we delete or transform big part of dataset. We will not try to change the dataset again.
\nWe see and understand that it is common practice in Kaggle competitions to maximize the score using all available tools and vulnerabilities in dataset, even though it leads to not indented or not practically useful outcome. Not allowed (which can lead to disqualification) are cases of direct fraud, hack of Kaggle platform, etc. We believe submissions we see in the leaderboard do not fall into this category, therefore they are acceptable. This was also discussed with Kaggle team.
\nLastly, we still consider our stability metric as meaningful reflection of our business problem that is worth investigating. The metric itself is something new to this community with a great potential for optimization, allowing you to get ahead of the pack if tackled successfully.
\nTomas
", "rawMarkdown": "Dear Kagglers,\n\nThere have been questions about leaderboard scores and the possibility to hack our metric recently. So here is our long-awaited comment.\n\nFirst, I want to clarify that we have quite limited options how to monitor and investigate top leaderboard positions – we don’t see submitted code (unless author makes it public), only predictions for test dataset. Therefore, we don’t directly see how Kagglers achieve their score, but we must admit that there are few instances that seem suspicious.\n\nWe have approached few Kagglers who were willing to share their insights about possibilities how to exploit our dataset. There are few approaches and attributes that can be used. Important finding is that some of them are related to our business and how it evolves over time. As a results, it’s almost impossible to remove such kind of information unless we delete or transform big part of dataset. We will not try to change the dataset again.\n\nWe see and understand that it is common practice in Kaggle competitions to maximize the score using all available tools and vulnerabilities in dataset, even though it leads to not indented or not practically useful outcome. Not allowed (which can lead to disqualification) are cases of direct fraud, hack of Kaggle platform, etc. We believe submissions we see in the leaderboard do not fall into this category, therefore they are acceptable. This was also discussed with Kaggle team.\n\nLastly, we still consider our stability metric as meaningful reflection of our business problem that is worth investigating. The metric itself is something new to this community with a great potential for optimization, allowing you to get ahead of the pack if tackled successfully. \n\nTomas", "votes": 1, "replies": [ { "id": 2796443, "postDate": "2024-05-06T08:26:02.393Z", "content": "Does it mean that hacking the metric is allowed and won't be punished in any way?
", "rawMarkdown": "Does it mean that hacking the metric is allowed and won't be punished in any way?", "votes": 16, "replies": [ { "id": 2797587, "postDate": "2024-05-06T19:41:19.027Z", "content": "i read it the same.
\nhappy i didnt rejoin this competition.
A bit sad if metric hack is allowed. Have an idea on how to potentially (potentially since I have not tried it) do this quite eaisly. Might give it a go if there's no rules against it!
", "rawMarkdown": "A bit sad if metric hack is allowed. Have an idea on how to potentially (potentially since I have not tried it) do this quite eaisly. Might give it a go if there's no rules against it! ", "votes": 1 }, { "id": 2800533, "postDate": "2024-05-08T08:11:04.930Z", "content": "Hi @eivolkova,
\nyes, hack in a sense of utilization information that is part of our data will not be punished.
\nwe can punish and disqualified only \"real\" hacks, e.g., if somebody hack Kaggle platform and download (hidden) test dataset…
Update: Made a quick version of this without putting in any effort in making it good nor optimized. Still increased LB by like 0.015 (from a pretty bad lb score to begin with though). And sure, it could be due to chance but I don't believe so!
\nHave currently no GPU left, but will check later how much of an increase one could get by making it a bit smarter…
", "rawMarkdown": "Update: Made a quick version of this without putting in any effort in making it good nor optimized. Still increased LB by like 0.015 (from a pretty bad lb score to begin with though). And sure, it could be due to chance but I don't believe so! \n\nHave currently no GPU left, but will check later how much of an increase one could get by making it a bit smarter..." } ] }, { "id": 2797896, "postDate": "2024-05-07T02:35:03.543Z", "content": "Hello,
\nIf some teams use a trick that uses future information and improve their score by a lot, what will happen?
can you be more clear with what you mean? what you say regarding \"hacking of the kaggle platform\" obviously goes without saying.
\nearlier you said this:
\n\"If the top 100 solutions will be using some way of week num prediction, yes we will go over those hundred solutions in the top private leaderboard until we find a solution that doesn't have this approach \"
\ni interpreted it as that you will go through each of the notebooks and remove any which try to directly hack the metric. just asking whether it's worth the time to try to hack things (i imagine it's not very hard), as your current answer isn't very clear how this will be handled.
", "rawMarkdown": "can you be more clear with what you mean? what you say regarding \"hacking of the kaggle platform\" obviously goes without saying.\n\nearlier you said this:\n\n \"If the top 100 solutions will be using some way of week num prediction, yes we will go over those hundred solutions in the top private leaderboard until we find a solution that doesn't have this approach \"\n\ni interpreted it as that you will go through each of the notebooks and remove any which try to directly hack the metric. just asking whether it's worth the time to try to hack things (i imagine it's not very hard), as your current answer isn't very clear how this will be handled.", "votes": 4, "replies": [ { "id": 2800535, "postDate": "2024-05-08T08:14:44.027Z", "content": "Hi @at7459,
\nthe hack in a form which is discussed here (optimizing metric by detecting early cases and reducing their predictiveness to improve stability) is not against rules and will not be punished and disqualified
So, instead of improving models in the final weeks, you motivate kagglers to focus on improving find_most_similar_rows() function and artificially degrade scores if found? If that's the case, something is wrong with the problem statement.
next time maybe try not go back on things you said earlier
\nat this point you should just either scrap the metric entirely and go to purely auc or reveal \"WEEK NUM\" so that everyone can hack the metric without problems
\nas it stands, people who spent the entire competition trying to hack the metric have a signifcant advantage now compared to people who tried to solve the actual problem of stability
", "rawMarkdown": "next time maybe try not go back on things you said earlier\n\nat this point you should just either scrap the metric entirely and go to purely auc or reveal \"WEEK NUM\" so that everyone can hack the metric without problems\n\nas it stands, people who spent the entire competition trying to hack the metric have a signifcant advantage now compared to people who tried to solve the actual problem of stability", "votes": 22 }, { "id": 2800582, "postDate": "2024-05-08T08:59:40.413Z", "content": "Indeed, what's the point of hiding weeknum now that hacking the metric is permitted?
", "rawMarkdown": "Indeed, what's the point of hiding weeknum now that hacking the metric is permitted?", "votes": 3 }, { "id": 2800592, "postDate": "2024-05-08T09:06:33.887Z", "content": "I agree, I know that you have made efforts to prevent hacking but now that it is allowed and we know that it is possible, this only turns the last 20 days into seeing who hacks the metric better.
\nWhich doesn't make much sense in my opinion.
", "rawMarkdown": "I agree, I know that you have made efforts to prevent hacking but now that it is allowed and we know that it is possible, this only turns the last 20 days into seeing who hacks the metric better.\n\nWhich doesn't make much sense in my opinion.", "votes": 2 }, { "id": 2800625, "postDate": "2024-05-08T09:23:11.730Z", "content": "I understand it won't be changed now, but a very easy way to get rid of this hacking problem would be to just evaluate AUC on a later part of the test set. Like what model generalizes best on a test set at least 1 year after the latest training data point, or something like that.
\nEasy metric and probably useful?
", "rawMarkdown": "I understand it won't be changed now, but a very easy way to get rid of this hacking problem would be to just evaluate AUC on a later part of the test set. Like what model generalizes best on a test set at least 1 year after the latest training data point, or something like that. \n\nEasy metric and probably useful?\n\n", "votes": 2 }, { "id": 2800709, "postDate": "2024-05-08T10:12:28.310Z", "content": "@ern711 The idea is great, it's a simple and clear way to evaluate stability of a model.
\nBut changing the evaluation process with only 20 days left before the competition end is crazy…
", "rawMarkdown": "@ern711 The idea is great, it's a simple and clear way to evaluate stability of a model.\n\nBut changing the evaluation process with only 20 days left before the competition end is crazy...", "votes": 3 }, { "id": 2801058, "postDate": "2024-05-08T13:08:15.750Z", "content": "Sure @eivolkova, I understand they cannot really change it now.
\nBut at the same time they won't get the solution they are looking for, and I'm gonna try hacking the metric myself by the method I suggested just out of curiosity to see if one can eliminate the negative slope completely :P
", "rawMarkdown": "Sure @eivolkova, I understand they cannot really change it now. \n\nBut at the same time they won't get the solution they are looking for, and I'm gonna try hacking the metric myself by the method I suggested just out of curiosity to see if one can eliminate the negative slope completely :P " }, { "id": 2802757, "postDate": "2024-05-09T06:55:45.333Z", "content": "free the WEEK_NUM! the sooner it happens the sooner everyone goes back to building better models
", "rawMarkdown": "free the WEEK_NUM! the sooner it happens the sooner everyone goes back to building better models", "votes": 3 }, { "id": 2803351, "postDate": "2024-05-09T12:58:07.407Z", "content": "Hi @tomasjeline2 , if you publish the WEEK_NUM, you will prevent people from winning only by hacking the metric, and in a way, it will get you out of a difficult situation. At the end, you will have access to the best models (even if all are hacked) not only the best hacking…Otherwise, I'm afraid you won't get anything from your competition except how to hack your metric 😉
", "rawMarkdown": "Hi @tomasjeline2 , if you publish the WEEK_NUM, you will prevent people from winning only by hacking the metric, and in a way, it will get you out of a difficult situation. At the end, you will have access to the best models (even if all are hacked) not only the best hacking...Otherwise, I'm afraid you won't get anything from your competition except how to hack your metric 😉", "votes": 7 }, { "id": 2804398, "postDate": "2024-05-10T03:48:56.177Z", "content": "I'm trying to solve this business problem by not using any means of stability metric hacking. Unfortunately, there's no clear definition on \"stability\". I assume that \"stability\" refers to the ranking of the variance of a model. For example, if we use variance of the loss function over all datasets as a performance measure and test a set of models. The smallest loss indicates the more stable model. Also, model robustness, resilience, reliability and ability to generalize well in any out-of-distribution data are products of \"stability\". Did you have any chance to brainstorm the custom metric implemented by these competitions: https://www.kaggle.com/competitions/amex-default-prediction and https://www.kaggle.com/competitions/m5-forecasting-uncertainty. I think the combinations of those metrics or much improved combinations are the right metric for this competition.
", "rawMarkdown": "I'm trying to solve this business problem by not using any means of stability metric hacking. Unfortunately, there's no clear definition on \"stability\". I assume that \"stability\" refers to the ranking of the variance of a model. For example, if we use variance of the loss function over all datasets as a performance measure and test a set of models. The smallest loss indicates the more stable model. Also, model robustness, resilience, reliability and ability to generalize well in any out-of-distribution data are products of \"stability\". Did you have any chance to brainstorm the custom metric implemented by these competitions: https://www.kaggle.com/competitions/amex-default-prediction and https://www.kaggle.com/competitions/m5-forecasting-uncertainty. I think the combinations of those metrics or much improved combinations are the right metric for this competition." } ] }, { "id": 2798891, "postDate": "2024-05-07T13:39:52.680Z", "content": "Thanks for sharing. Please don't forget to remind people how you manage this competition if you ever host another one. It will be very helpful so they can avoid it and learn from my mistake :)
", "rawMarkdown": "Thanks for sharing. Please don't forget to remind people how you manage this competition if you ever host another one. It will be very helpful so they can avoid it and learn from my mistake :)", "votes": 9 }, { "id": 2798897, "postDate": "2024-05-07T13:41:24.060Z", "content": "It looks like the competition hosts have been discussing the evaluation metric again, yet it's not clear how they are going to handle metric hacking. By 'metric hacking,' I don't mean any server hacking; I simply mean utilizing the post-processing trick.
\nThere are only two options:
\nWe still do not know if metric hacking is considered a legitimate approach or not. All I want to know is what rules actually apply. Everybody should have the same options.
", "rawMarkdown": "It looks like the competition hosts have been discussing the evaluation metric again, yet it's not clear how they are going to handle metric hacking. By 'metric hacking,' I don't mean any server hacking; I simply mean utilizing the post-processing trick.\n\nThere are only two options:\n\n1. Metric hacking is allowed, and anybody can utilize it. The notebooks will be eligible.\nOR\n2. Metric hacking is not allowed, and nobody can utilize it. The notebooks will not be eligible.\n\nWe still do not know if metric hacking is considered a legitimate approach or not. All I want to know is what rules actually apply. Everybody should have the same options." } ] }, { "id": 2778081, "postDate": "2024-04-27T00:03:26.827Z", "content": "Thank you very much @kononenko.
\nI am just a passerby in this competition. Since the beginning of this competition, I have felt that the hosts' contributions have been outstanding and I have great respect for them.
\nHowever, contrary to the purpose of this competition, which is to compete for stability, the current situation is extremely unstable, and I believe that what is called \"報連相\" (report, communication, consultation) in Japanese is necessary. One response saves the participant.
", "rawMarkdown": "Thank you very much @kononenko.\n\nI am just a passerby in this competition. Since the beginning of this competition, I have felt that the hosts' contributions have been outstanding and I have great respect for them.\n\nHowever, contrary to the purpose of this competition, which is to compete for stability, the current situation is extremely unstable, and I believe that what is called \"報連相\" (report, communication, consultation) in Japanese is necessary. One response saves the participant.", "votes": 2, "replies": [ { "id": 2778088, "postDate": "2024-04-27T00:11:22.187Z", "content": "Yes, hosts were very supportive in the past, so I hope they will do something about the new way of metric hacking.
", "rawMarkdown": "Yes, hosts were very supportive in the past, so I hope they will do something about the new way of metric hacking.", "votes": 2, "replies": [ { "id": 2778719, "postDate": "2024-04-27T09:46:09.780Z", "content": "I explained them way how metric could be hacked and they need time to find best way how to eliminate it.
\nI hope next week will be more information/reaction
@johnpateha Nothing for the moment or am I missing it?
", "rawMarkdown": "@johnpateha Nothing for the moment or am I missing it?", "votes": 1 }, { "id": 2788605, "postDate": "2024-05-02T10:05:39.277Z", "content": "As far as I know, there is no response. The organizer should have checked the private LB, so this situation may be insignificant.
", "rawMarkdown": "As far as I know, there is no response. The organizer should have checked the private LB, so this situation may be insignificant.", "votes": 1 }, { "id": 2788701, "postDate": "2024-05-02T11:12:28.033Z", "content": "Host confirmed that they see the issue now, but need time to figure it out.
\nI believe next information will be public and availible for everyone.
I got 5% up on LB with my basic solution without any tuning of hack, so it hard to call insignificant.
", "rawMarkdown": "I got 5% up on LB with my basic solution without any tuning of hack, so it hard to call insignificant. ", "votes": 1 }, { "id": 2800631, "postDate": "2024-05-08T09:27:04.913Z", "content": "Hi @johnpateha , congrats for your last submission, it seems you were right ;-)
", "rawMarkdown": "Hi @johnpateha , congrats for your last submission, it seems you were right ;-)" } ] } ] }, { "id": 2794690, "postDate": "2024-05-05T12:51:35.577Z", "content": "Could there be a possibility that the metrics are not Hacked? and that the current top score of 0.639 is exactly the solution Host wants?
", "rawMarkdown": "Could there be a possibility that the metrics are not Hacked? and that the current top score of 0.639 is exactly the solution Host wants?", "replies": [ { "id": 2794708, "postDate": "2024-05-05T13:01:14.863Z", "content": "That's why the host said they will check the top submissions.
", "rawMarkdown": "That's why the host said they will check the top submissions." } ] }, { "id": 2793829, "postDate": "2024-05-05T01:59:31.123Z", "content": "Can Any one suggest good way to avoid metric hack to host!!
\nwill show some empathy for hosts..
", "rawMarkdown": "Can Any one suggest good way to avoid metric hack to host!!\n\nwill show some empathy for hosts.." }, { "id": 2798892, "postDate": "2024-05-07T13:40:03.487Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2776897, "postDate": "2024-04-26T11:47:33.020Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ], "comments": [ { "id": 2774365, "author_name": "Sergey Bryansky", "author_url": "", "post_date": "2024-04-25T06:58:07.873000", "content": "Sad. As experienced user I decided not to submit at early stage of competition. Decided to wait. So after first metric hacking and period of fixing I downloaded the data first time. But something inside of me said to me wait more please ;) And it happened again. Very sad. I sympathize with the guys who worked the whole distance. Still, Kaggle is not about hacking, lb probing and other dirty tricks.
", "votes": 6, "replies": [] }, { "id": 2796412, "author_name": "Tomas Jelinek", "author_url": "", "post_date": "2024-05-06T08:02:36.090000", "content": "Dear Kagglers,
\nThere have been questions about leaderboard scores and the possibility to hack our metric recently. So here is our long-awaited comment.
\nFirst, I want to clarify that we have quite limited options how to monitor and investigate top leaderboard positions – we don’t see submitted code (unless author makes it public), only predictions for test dataset. Therefore, we don’t directly see how Kagglers achieve their score, but we must admit that there are few instances that seem suspicious.
\nWe have approached few Kagglers who were willing to share their insights about possibilities how to exploit our dataset. There are few approaches and attributes that can be used. Important finding is that some of them are related to our business and how it evolves over time. As a results, it’s almost impossible to remove such kind of information unless we delete or transform big part of dataset. We will not try to change the dataset again.
\nWe see and understand that it is common practice in Kaggle competitions to maximize the score using all available tools and vulnerabilities in dataset, even though it leads to not indented or not practically useful outcome. Not allowed (which can lead to disqualification) are cases of direct fraud, hack of Kaggle platform, etc. We believe submissions we see in the leaderboard do not fall into this category, therefore they are acceptable. This was also discussed with Kaggle team.
\nLastly, we still consider our stability metric as meaningful reflection of our business problem that is worth investigating. The metric itself is something new to this community with a great potential for optimization, allowing you to get ahead of the pack if tackled successfully.
\nTomas
", "votes": 1, "replies": [ { "id": 2796443, "author_name": "Evgeniia Grigoreva", "author_url": "", "post_date": "2024-05-06T08:26:02.393000", "content": "Does it mean that hacking the metric is allowed and won't be punished in any way?
", "votes": 16, "replies": [ { "id": 2797587, "author_name": "Simon Veitner", "author_url": "", "post_date": "2024-05-06T19:41:19.027000", "content": "i read it the same.
\nhappy i didnt rejoin this competition.
A bit sad if metric hack is allowed. Have an idea on how to potentially (potentially since I have not tried it) do this quite eaisly. Might give it a go if there's no rules against it!
", "votes": 1, "replies": [] }, { "id": 2800533, "author_name": "Tomas Jelinek", "author_url": "", "post_date": "2024-05-08T08:11:04.930000", "content": "Hi @eivolkova,
\nyes, hack in a sense of utilization information that is part of our data will not be punished.
\nwe can punish and disqualified only \"real\" hacks, e.g., if somebody hack Kaggle platform and download (hidden) test dataset…
Update: Made a quick version of this without putting in any effort in making it good nor optimized. Still increased LB by like 0.015 (from a pretty bad lb score to begin with though). And sure, it could be due to chance but I don't believe so!
\nHave currently no GPU left, but will check later how much of an increase one could get by making it a bit smarter…
", "votes": 0, "replies": [] } ] }, { "id": 2797896, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-05-07T02:35:03.543000", "content": "Hello,
\nIf some teams use a trick that uses future information and improve their score by a lot, what will happen?
can you be more clear with what you mean? what you say regarding \"hacking of the kaggle platform\" obviously goes without saying.
\nearlier you said this:
\n\"If the top 100 solutions will be using some way of week num prediction, yes we will go over those hundred solutions in the top private leaderboard until we find a solution that doesn't have this approach \"
\ni interpreted it as that you will go through each of the notebooks and remove any which try to directly hack the metric. just asking whether it's worth the time to try to hack things (i imagine it's not very hard), as your current answer isn't very clear how this will be handled.
", "votes": 4, "replies": [ { "id": 2800535, "author_name": "Tomas Jelinek", "author_url": "", "post_date": "2024-05-08T08:14:44.027000", "content": "Hi @at7459,
\nthe hack in a form which is discussed here (optimizing metric by detecting early cases and reducing their predictiveness to improve stability) is not against rules and will not be punished and disqualified
So, instead of improving models in the final weeks, you motivate kagglers to focus on improving find_most_similar_rows() function and artificially degrade scores if found? If that's the case, something is wrong with the problem statement.
next time maybe try not go back on things you said earlier
\nat this point you should just either scrap the metric entirely and go to purely auc or reveal \"WEEK NUM\" so that everyone can hack the metric without problems
\nas it stands, people who spent the entire competition trying to hack the metric have a signifcant advantage now compared to people who tried to solve the actual problem of stability
", "votes": 22, "replies": [] }, { "id": 2800582, "author_name": "Gasset Mathieu", "author_url": "", "post_date": "2024-05-08T08:59:40.413000", "content": "Indeed, what's the point of hiding weeknum now that hacking the metric is permitted?
", "votes": 3, "replies": [] }, { "id": 2800592, "author_name": "Fnoa", "author_url": "", "post_date": "2024-05-08T09:06:33.887000", "content": "I agree, I know that you have made efforts to prevent hacking but now that it is allowed and we know that it is possible, this only turns the last 20 days into seeing who hacks the metric better.
\nWhich doesn't make much sense in my opinion.
", "votes": 2, "replies": [] }, { "id": 2800625, "author_name": "Ern711", "author_url": "", "post_date": "2024-05-08T09:23:11.730000", "content": "I understand it won't be changed now, but a very easy way to get rid of this hacking problem would be to just evaluate AUC on a later part of the test set. Like what model generalizes best on a test set at least 1 year after the latest training data point, or something like that.
\nEasy metric and probably useful?
", "votes": 2, "replies": [] }, { "id": 2800709, "author_name": "Evgeniia Grigoreva", "author_url": "", "post_date": "2024-05-08T10:12:28.310000", "content": "@ern711 The idea is great, it's a simple and clear way to evaluate stability of a model.
\nBut changing the evaluation process with only 20 days left before the competition end is crazy…
", "votes": 3, "replies": [] }, { "id": 2801058, "author_name": "Ern711", "author_url": "", "post_date": "2024-05-08T13:08:15.750000", "content": "Sure @eivolkova, I understand they cannot really change it now.
\nBut at the same time they won't get the solution they are looking for, and I'm gonna try hacking the metric myself by the method I suggested just out of curiosity to see if one can eliminate the negative slope completely :P
", "votes": 0, "replies": [] }, { "id": 2802757, "author_name": "Taizhuo Tang", "author_url": "", "post_date": "2024-05-09T06:55:45.333000", "content": "free the WEEK_NUM! the sooner it happens the sooner everyone goes back to building better models
", "votes": 3, "replies": [] }, { "id": 2803351, "author_name": "Laurent Pourchot", "author_url": "", "post_date": "2024-05-09T12:58:07.407000", "content": "Hi @tomasjeline2 , if you publish the WEEK_NUM, you will prevent people from winning only by hacking the metric, and in a way, it will get you out of a difficult situation. At the end, you will have access to the best models (even if all are hacked) not only the best hacking…Otherwise, I'm afraid you won't get anything from your competition except how to hack your metric 😉
", "votes": 7, "replies": [] }, { "id": 2804398, "author_name": "ph_one1", "author_url": "", "post_date": "2024-05-10T03:48:56.177000", "content": "I'm trying to solve this business problem by not using any means of stability metric hacking. Unfortunately, there's no clear definition on \"stability\". I assume that \"stability\" refers to the ranking of the variance of a model. For example, if we use variance of the loss function over all datasets as a performance measure and test a set of models. The smallest loss indicates the more stable model. Also, model robustness, resilience, reliability and ability to generalize well in any out-of-distribution data are products of \"stability\". Did you have any chance to brainstorm the custom metric implemented by these competitions: https://www.kaggle.com/competitions/amex-default-prediction and https://www.kaggle.com/competitions/m5-forecasting-uncertainty. I think the combinations of those metrics or much improved combinations are the right metric for this competition.
", "votes": 0, "replies": [] } ] }, { "id": 2798891, "author_name": "NxGTR", "author_url": "", "post_date": "2024-05-07T13:39:52.680000", "content": "Thanks for sharing. Please don't forget to remind people how you manage this competition if you ever host another one. It will be very helpful so they can avoid it and learn from my mistake :)
", "votes": 9, "replies": [] }, { "id": 2798897, "author_name": "Matous Famera", "author_url": "", "post_date": "2024-05-07T13:41:24.060000", "content": "It looks like the competition hosts have been discussing the evaluation metric again, yet it's not clear how they are going to handle metric hacking. By 'metric hacking,' I don't mean any server hacking; I simply mean utilizing the post-processing trick.
\nThere are only two options:
\nWe still do not know if metric hacking is considered a legitimate approach or not. All I want to know is what rules actually apply. Everybody should have the same options.
", "votes": 0, "replies": [] } ] }, { "id": 2778081, "author_name": "mh", "author_url": "", "post_date": "2024-04-27T00:03:26.827000", "content": "Thank you very much @kononenko.
\nI am just a passerby in this competition. Since the beginning of this competition, I have felt that the hosts' contributions have been outstanding and I have great respect for them.
\nHowever, contrary to the purpose of this competition, which is to compete for stability, the current situation is extremely unstable, and I believe that what is called \"報連相\" (report, communication, consultation) in Japanese is necessary. One response saves the participant.
", "votes": 2, "replies": [ { "id": 2778088, "author_name": "Oleksiy Kononenko", "author_url": "", "post_date": "2024-04-27T00:11:22.187000", "content": "Yes, hosts were very supportive in the past, so I hope they will do something about the new way of metric hacking.
", "votes": 2, "replies": [ { "id": 2778719, "author_name": "Evgeny Patekha", "author_url": "", "post_date": "2024-04-27T09:46:09.780000", "content": "I explained them way how metric could be hacked and they need time to find best way how to eliminate it.
\nI hope next week will be more information/reaction
@johnpateha Nothing for the moment or am I missing it?
", "votes": 1, "replies": [] }, { "id": 2788605, "author_name": "mh", "author_url": "", "post_date": "2024-05-02T10:05:39.277000", "content": "As far as I know, there is no response. The organizer should have checked the private LB, so this situation may be insignificant.
", "votes": 1, "replies": [] }, { "id": 2788701, "author_name": "Evgeny Patekha", "author_url": "", "post_date": "2024-05-02T11:12:28.033000", "content": "Host confirmed that they see the issue now, but need time to figure it out.
\nI believe next information will be public and availible for everyone.
I got 5% up on LB with my basic solution without any tuning of hack, so it hard to call insignificant.
", "votes": 1, "replies": [] }, { "id": 2800631, "author_name": "Laurent Pourchot", "author_url": "", "post_date": "2024-05-08T09:27:04.913000", "content": "Hi @johnpateha , congrats for your last submission, it seems you were right ;-)
", "votes": 0, "replies": [] } ] } ] }, { "id": 2794690, "author_name": "暗黑AGI", "author_url": "", "post_date": "2024-05-05T12:51:35.577000", "content": "Could there be a possibility that the metrics are not Hacked? and that the current top score of 0.639 is exactly the solution Host wants?
", "votes": 0, "replies": [ { "id": 2794708, "author_name": "Iqbal Syah Akbar", "author_url": "", "post_date": "2024-05-05T13:01:14.863000", "content": "That's why the host said they will check the top submissions.
", "votes": 0, "replies": [] } ] }, { "id": 2793829, "author_name": "Anil Kumar Reddy", "author_url": "", "post_date": "2024-05-05T01:59:31.123000", "content": "Can Any one suggest good way to avoid metric hack to host!!
\nwill show some empathy for hosts..
", "votes": 0, "replies": [] }, { "id": 2798892, "author_name": "", "author_url": "", "post_date": "2024-05-07T13:40:03.487000", "content": "", "votes": 0, "replies": [] }, { "id": 2776897, "author_name": "", "author_url": "", "post_date": "2024-04-26T11:47:33.020000", "content": "", "votes": -1, "replies": [] } ], "raw_markdown_by_id": { "2771621": "Dear @jetakow, @tomasjeline2 and @inversion, \n\nOver the last few days concerns have emerged on the hackability of the metric even with the transformed test data ([post1](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/496898), [post2](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/497003), [post3](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/497167)). I hate to say it, but this situation, when repeated, is not going to have a positive impact on the flow of the competition. \n\nCould you please look into the issue and provide us with some feedback?\n\n", "2774365": "Sad. As experienced user I decided not to submit at early stage of competition. Decided to wait. So after first metric hacking and period of fixing I downloaded the data first time. But something inside of me said to me wait more please ;) And it happened again. Very sad. I sympathize with the guys who worked the whole distance. Still, Kaggle is not about hacking, lb probing and other dirty tricks.", "2796412": "Dear Kagglers,\n\nThere have been questions about leaderboard scores and the possibility to hack our metric recently. So here is our long-awaited comment.\n\nFirst, I want to clarify that we have quite limited options how to monitor and investigate top leaderboard positions – we don’t see submitted code (unless author makes it public), only predictions for test dataset. Therefore, we don’t directly see how Kagglers achieve their score, but we must admit that there are few instances that seem suspicious.\n\nWe have approached few Kagglers who were willing to share their insights about possibilities how to exploit our dataset. There are few approaches and attributes that can be used. Important finding is that some of them are related to our business and how it evolves over time. As a results, it’s almost impossible to remove such kind of information unless we delete or transform big part of dataset. We will not try to change the dataset again.\n\nWe see and understand that it is common practice in Kaggle competitions to maximize the score using all available tools and vulnerabilities in dataset, even though it leads to not indented or not practically useful outcome. Not allowed (which can lead to disqualification) are cases of direct fraud, hack of Kaggle platform, etc. We believe submissions we see in the leaderboard do not fall into this category, therefore they are acceptable. This was also discussed with Kaggle team.\n\nLastly, we still consider our stability metric as meaningful reflection of our business problem that is worth investigating. The metric itself is something new to this community with a great potential for optimization, allowing you to get ahead of the pack if tackled successfully. \n\nTomas", "2778081": "Thank you very much @kononenko.\n\nI am just a passerby in this competition. Since the beginning of this competition, I have felt that the hosts' contributions have been outstanding and I have great respect for them.\n\nHowever, contrary to the purpose of this competition, which is to compete for stability, the current situation is extremely unstable, and I believe that what is called \"報連相\" (report, communication, consultation) in Japanese is necessary. One response saves the participant.", "2794690": "Could there be a possibility that the metrics are not Hacked? and that the current top score of 0.639 is exactly the solution Host wants?", "2793829": "Can Any one suggest good way to avoid metric hack to host!!\n\nwill show some empathy for hosts..", "2798892": "", "2776897": "" } }