Q: Why can't I submit a solution in my country?
\nA: We disabled the submissions as a temporary measure. It will be enabled again around 6.3.2024. https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/478716
Q: Questions regarding transformation of test data.
\nA: The transformation was done to train and test before the start of the competition and now we will transform test data in different manner. We won't specify what is the transformation exactly. We will keep column names, dtypes and the change is not going to be drastic.
Q: Any question related to private test set
\nA: We will not disclose any information about private test set. There is no point in asking questions about the private test set. We made the split in such a way that it will be fair for final evaluation.
Q: How long is the test set?
\nA: For the sake of stability evaluation it would not make sense to disclose you that information. This information would lead even to more \"hacks\" of the metric. Think about it this way. The model will be deployed and you don't know for how long it is going to be.
Q: What is the meaning of num_group1 and num_group2?
\nA: Those are indices that will help you to aggregate tables with depth=1,2. Please have a look at Data section.
Q: Any questions regarding stability metric and hacking.
\nPlease read https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/476867 first. We are working on a solution that would minimize such hacking by artificially worsening the score in the near future.
Q: Can't we just remove the falling rate from metric?
\nA: Omitting the falling rate would not be a satisfactory solution. Without the falling rate, we would greatly encourage unstable models in time. Those particular weights were selected so that they would perfectly represent the average decision of risk managers in HC. That is desirable as we will be able to use the knowledge gained from winners and compare it with our processes and models. Removing the falling rate would remove the stability component in time. This year we wanted to explore stability as the main topic.
Q: What is the reason for having those weights in the stability metric?
\nA: Those particular weights were selected so that they would perfectly represent the average decision of risk managers in HC. That is desirable as we will be able to use the knowledge gained from winners and compare it with our processes and models.
Q: Why is my notebook failing?
\nA: We don't have capacity to review every notebook, unfortunately. You can post it in discussion and maybe someone will help. In future, we can collect those errors into this thread.
Q: Dates are inconsistent and I see dates from the future in respect to date decision.
\nA: This might happen as all dates were transformed, but there might be some inconsistencies. Also, the underlying data from our primary sources can be flawed, sometimes it happens.
Q: What is the definition of the used target?
\nA: It's unpaid payment (one is enough) in certain time period. There is also some time tolerance (e.g. one day late is not default) and amount tolerance (if client paid $100 instead of $100.10). We can't disclose further details.
Q: Why can't I submit a solution in my country?
\nA: We disabled the submissions as a temporary measure. It will be enabled again around 6.3.2024. https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/478716
Q: Questions regarding transformation of test data.
\nA: The transformation was done to train and test before the start of the competition and now we will transform test data in different manner. We won't specify what is the transformation exactly. We will keep column names, dtypes and the change is not going to be drastic.
Q: Any question related to private test set
\nA: We will not disclose any information about private test set. There is no point in asking questions about the private test set. We made the split in such a way that it will be fair for final evaluation.
Q: How long is the test set?
\nA: For the sake of stability evaluation it would not make sense to disclose you that information. This information would lead even to more \"hacks\" of the metric. Think about it this way. The model will be deployed and you don't know for how long it is going to be.
Q: What is the meaning of num_group1 and num_group2?
\nA: Those are indices that will help you to aggregate tables with depth=1,2. Please have a look at Data section.
Q: Any questions regarding stability metric and hacking.
\nPlease read https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/476867 first. We are working on a solution that would minimize such hacking by artificially worsening the score in the near future.
Q: Can't we just remove the falling rate from metric?
\nA: Omitting the falling rate would not be a satisfactory solution. Without the falling rate, we would greatly encourage unstable models in time. Those particular weights were selected so that they would perfectly represent the average decision of risk managers in HC. That is desirable as we will be able to use the knowledge gained from winners and compare it with our processes and models. Removing the falling rate would remove the stability component in time. This year we wanted to explore stability as the main topic.
Q: What is the reason for having those weights in the stability metric?
\nA: Those particular weights were selected so that they would perfectly represent the average decision of risk managers in HC. That is desirable as we will be able to use the knowledge gained from winners and compare it with our processes and models.
Q: Why is my notebook failing?
\nA: We don't have capacity to review every notebook, unfortunately. You can post it in discussion and maybe someone will help. In future, we can collect those errors into this thread.
Q: Dates are inconsistent and I see dates from the future in respect to date decision.
\nA: This might happen as all dates were transformed, but there might be some inconsistencies. Also, the underlying data from our primary sources can be flawed, sometimes it happens.
Q: What is the definition of the used target?
\nA: It's unpaid payment (one is enough) in certain time period. There is also some time tolerance (e.g. one day late is not default) and amount tolerance (if client paid $100 instead of $100.10). We can't disclose further details.
\n\nOmitting the falling rate would not be a satisfactory solution. Without the falling rate, we would greatly encourage unstable models in time. Those particular weights were selected so that they would perfectly represent the average decision of risk manager in HC. That is desirable as we will be able to use the knowledge gained from winners and compare it with our processes and models. Removing the falling rate would remove the stability component in time. This year we wanted to explore stability as the main topic.
\n
The falling rate was ill conceived. There should be no scenario where a model that has consistently poor performance is deemed better than a model that sometimes has good performance and sometimes poor performance. To use such a metric is to misunderstand what the value of stability is. That is, the value of stability is never in being consistently bad, but in avoiding periods that are particularly bad. Once this is understood, it becomes obvious that the best way to encourage stability is by aggressively punishing periods of particularly bad performance. We can do this with a metric that rapidly become worse when very bad samples are present. As I suggested in the other thread, this can be done with a metric of the form:
\nmean( ( -log( MovingAverage(gini) ) )**exponent ) with exponent>=1
\nUnder this metric, periods of poor performance are heavily penalized without the silliness of punishing good rounds. Stability should require nothing more than this.
", "rawMarkdown": ">Omitting the falling rate would not be a satisfactory solution. Without the falling rate, we would greatly encourage unstable models in time. Those particular weights were selected so that they would perfectly represent the average decision of risk manager in HC. That is desirable as we will be able to use the knowledge gained from winners and compare it with our processes and models. Removing the falling rate would remove the stability component in time. This year we wanted to explore stability as the main topic.\n\nThe falling rate was ill conceived. There should be no scenario where a model that has consistently poor performance is deemed better than a model that sometimes has good performance and sometimes poor performance. To use such a metric is to misunderstand what the value of stability is. That is, the value of stability is never in being consistently bad, but in avoiding periods that are particularly bad. Once this is understood, it becomes obvious that the best way to encourage stability is by aggressively punishing periods of particularly bad performance. We can do this with a metric that rapidly become worse when very bad samples are present. As I suggested in the other thread, this can be done with a metric of the form: \n\nmean( ( -log( MovingAverage(gini) ) )**exponent ) with exponent>=1\n\nUnder this metric, periods of poor performance are heavily penalized without the silliness of punishing good rounds. Stability should require nothing more than this.", "votes": 8, "replies": [ { "id": 2653310, "postDate": "2024-02-15T10:49:34.977Z", "content": "\n\nThe falling rate was ill conceived. There should be no scenario where a model that has consistently poor performance is deemed better than a model that sometimes has good performance and sometimes poor performance. To use such a metric is to misunderstand what the value of stability is.
\n
Frankly, I think there is also some misunderstanding on your side. Consider this. From the perspective of risk managers at HC it is undesirable to have a model that is performing well let's say first 6 months and then it levels on for example 60%-80% of that performance. Even if the model performed better in terms of integral of weekly gini over time the model was unpredictable in sense that its performance lowered after the first 6 months. What is desirable is a model that has the same exact performance every week and if there is some decrease it is well predictable.
\nOmitting the falling rate term will allow models that will be performing well on some portion of test sample and then they will have decrease in performance that might not be even linear of exponential.
\nRegarding proposed metric, you could try to rewrite it using LaTeX with proper sums so it is clear what is applied to what. As it is now I am not sure if I understand it right.
\nRegarding the moving average idea, I believe that this might lower the amount of hacking, but it will not solve the problem completely. Consider following model that is performing well the first N weeks and during next 3 weeks the performance drops to 60% of that. By lowering the score artificially on the first N weeks you will improve the stability metric. That is aligned with risk managers' view of performance. It is not aligned with metric being fair and fun for everyone, we are working on that and I ask you for patience. The score from the metric that you suggested will improve when decreasing the score artificially first N weeks. If this is not the case, then I don't understand it perhaps correctly and you could explain it in finer detail.
", "rawMarkdown": ">The falling rate was ill conceived. There should be no scenario where a model that has consistently poor performance is deemed better than a model that sometimes has good performance and sometimes poor performance. To use such a metric is to misunderstand what the value of stability is. \n\nFrankly, I think there is also some misunderstanding on your side. Consider this. From the perspective of risk managers at HC it is undesirable to have a model that is performing well let's say first 6 months and then it levels on for example 60%-80% of that performance. Even if the model performed better in terms of integral of weekly gini over time the model was unpredictable in sense that its performance lowered after the first 6 months. What is desirable is a model that has the same exact performance every week and if there is some decrease it is well predictable. \n\nOmitting the falling rate term will allow models that will be performing well on some portion of test sample and then they will have decrease in performance that might not be even linear of exponential. \n\nRegarding proposed metric, you could try to rewrite it using LaTeX with proper sums so it is clear what is applied to what. As it is now I am not sure if I understand it right. \n\nRegarding the moving average idea, I believe that this might lower the amount of hacking, but it will not solve the problem completely. Consider following model that is performing well the first N weeks and during next 3 weeks the performance drops to 60% of that. By lowering the score artificially on the first N weeks you will improve the stability metric. That is aligned with risk managers' view of performance. It is not aligned with metric being fair and fun for everyone, we are working on that and I ask you for patience. The score from the metric that you suggested will improve when decreasing the score artificially first N weeks. If this is not the case, then I don't understand it perhaps correctly and you could explain it in finer detail. ", "votes": 3, "replies": [ { "id": 2653528, "postDate": "2024-02-15T13:58:10.207Z", "content": "I don’t think it will improve by lowering the score in first few weeks. Every lowering of gini score will result in a lower moving average. If moving average gets smaller, -log will get larger. The best performance is reached when all ginis in all relevant time windows are 1, then the above metric is 0 as log(1)=0.
\nThat badscores are punished heavily is also clear as the log decreases rapidly for arguments smaller than 1 (and even more so the closer we get to zero.)
\nPlease correct if I’m wrong @jacobyjaeger
\n\nFrankly, I think there is also some misunderstanding on your side. Consider this. From the perspective of risk managers at HC it is undesirable to have a model that is performing well let's say first 6 months and then it levels on for example 60%-80% of that performance.
\n
What you need is not to be able to predict the performance but to be able to put a good lower bound on what the performance could be. This is because the event of the model performing better than you might have prepared for is never a problem and as such does not need to be penalized. The set of models that can be given a high lower bound on their expected performance are those that have demonstrated a high lower bound during the validation period, which are those that perform well under a metric that heavily penalizes a given model's worst rounds as mine does.
\n\n\nRegarding proposed metric, you could try to rewrite it using LaTeX with proper sums so it is clear what is applied to what. As it is now I am not sure if I understand it right.
\n
Once predictions have been converted to ginis, there is only one axis that we need to be concerned with, the time axis. Here is a simple implementation of the metric. Higher scores are considered worse under this metric.
\n numpy np\n matplotlib pyplot plt\n\n\n ():\n (axis==)\n\n x = np.cumsum( x, axis= )\n\n x = np.concatenate([ *x[:], x ],)\n\n (x[n:] - x[:-n])/n\n\n\n ():\n \n\n ():\n\n scores = ( - np.log( MA( ginis, ma_len, axis= ) ) )**exponent\n\n scores\n\n metric \n\n\n\nN_GINIS = \nN_MA = \nEXPONENT = \n\nginis = np.random.rand(N_GINIS)\n\ntime = np.arange(N_GINIS)\n\n\nmetric = get_metric(EXPONENT, N_MA)\n\n\nscores = metric(ginis)\n\n\nplt.plot(time, ginis)\nplt.plot(time[N_MA//: N_GINIS-(N_MA-)//], scores)\n\n\nscore = np.mean(scores)\nscore\n\n\nOmitting the falling rate term will allow models that will be performing well on some portion of test sample and then they will have decrease in performance that might not be even linear of exponential.
\n
Any period of poor performance is heavily punished under my metric. But the falling rate can only detect periods of poor performance if they come at the end of the test period.
\n\n\nRegarding the moving average idea, I believe that this might lower the amount of hacking, but it will not solve the problem completely. Consider following model that is performing well the first N weeks and during next 3 weeks the performance drops to 60% of that. By lowering the score artificially on the first N weeks you will improve the stability metric. That is aligned with risk managers' view of performance.
\n
I'm not sure what you are trying to say here. Are you suggesting that performance on my metric would be improved by such hacking? That is not the case, it can be seen from the fact that the gradient wrt sample ginis is strictly negative that making any round's performance worse can only hurt performance under my metric.
\nOr are you saying that it is desirable that such hacking results in a higher score? That would be rather silly, you can make a declining performance look flat during the test period by making the early test rounds artificially worse, but that does not change the fact that true performance was declining which will become evident after the test period when you no longer have artificially deflated rounds left to compare the performance to.
", "rawMarkdown": ">Frankly, I think there is also some misunderstanding on your side. Consider this. From the perspective of risk managers at HC it is undesirable to have a model that is performing well let's say first 6 months and then it levels on for example 60%-80% of that performance. \n\nWhat you need is not to be able to predict the performance but to be able to put a good lower bound on what the performance could be. This is because the event of the model performing better than you might have prepared for is never a problem and as such does not need to be penalized. The set of models that can be given a high lower bound on their expected performance are those that have demonstrated a high lower bound during the validation period, which are those that perform well under a metric that heavily penalizes a given model's worst rounds as mine does.\n\n>Regarding proposed metric, you could try to rewrite it using LaTeX with proper sums so it is clear what is applied to what. As it is now I am not sure if I understand it right.\n\n\nOnce predictions have been converted to ginis, there is only one axis that we need to be concerned with, the time axis. Here is a simple implementation of the metric. Higher scores are considered worse under this metric. \n\n```python\n\nimport numpy as np\nfrom matplotlib import pyplot as plt\n\n\ndef MA( x, n, axis=0 ):\n assert(axis==0)\n \n x = np.cumsum( x, axis=0 )\n \n x = np.concatenate([ 0*x[:1], x ],0)\n \n return (x[n:] - x[:-n])/n\n\n\ndef get_metric(exponent=1, ma_len=5):\n '''\n A metric that heavily punishes rounds of poor performance.\n Higher exponent values punish bad rounds more disproportionately.\n Larger MA len's place focus on longer term performance.\n \n Lower mean scores mean better performance.\n '''\n \n def metric( ginis ):\n \n scores = ( - np.log( MA( ginis, ma_len, axis=0 ) ) )**exponent\n \n return scores\n \n return metric \n\n\n\nN_GINIS = 100\nN_MA = 5\nEXPONENT = 2\n\nginis = np.random.rand(N_GINIS)\n\ntime = np.arange(N_GINIS)\n\n\nmetric = get_metric(EXPONENT, N_MA)\n\n\nscores = metric(ginis)\n\n\nplt.plot(time, ginis)\nplt.plot(time[N_MA//2: N_GINIS-(N_MA-1)//2], scores)\n\n\nscore = np.mean(scores)\nscore\n```\n\n>Omitting the falling rate term will allow models that will be performing well on some portion of test sample and then they will have decrease in performance that might not be even linear of exponential.\n\nAny period of poor performance is heavily punished under my metric. But the falling rate can only detect periods of poor performance if they come at the end of the test period.\n\n\n>Regarding the moving average idea, I believe that this might lower the amount of hacking, but it will not solve the problem completely. Consider following model that is performing well the first N weeks and during next 3 weeks the performance drops to 60% of that. By lowering the score artificially on the first N weeks you will improve the stability metric. That is aligned with risk managers' view of performance.\n\nI'm not sure what you are trying to say here. Are you suggesting that performance on my metric would be improved by such hacking? That is not the case, it can be seen from the fact that the gradient wrt sample ginis is strictly negative that making any round's performance worse can only hurt performance under my metric. \n\nOr are you saying that it is desirable that such hacking results in a higher score? That would be rather silly, you can make a declining performance look flat during the test period by making the early test rounds artificially worse, but that does not change the fact that true performance was declining which will become evident after the test period when you no longer have artificially deflated rounds left to compare the performance to. " }, { "id": 2654192, "postDate": "2024-02-16T00:26:12.940Z", "content": "You're exactly right, Simon. We can check the sign of the gradient of the final score wrt the per round scores to prove that it is never desirable to make any round score worse. I suggest that kaggle should perform such a check on every competition metric so that we don't see hackable metrics again in the future.
", "rawMarkdown": "You're exactly right, Simon. We can check the sign of the gradient of the final score wrt the per round scores to prove that it is never desirable to make any round score worse. I suggest that kaggle should perform such a check on every competition metric so that we don't see hackable metrics again in the future.", "votes": 2 }, { "id": 2654518, "postDate": "2024-02-16T07:24:44.077Z", "content": "\n\nWhat you need is not to be able to predict the performance but to be able to put a good lower bound on what the performance could be. This is because the event of the model performing better than you might have prepared for is never a problem and as such does not need to be penalized.
\n
From business point of view this statement is not true. It's not only about the credit risk model, but whole underwriting setup, product parameters, profitability models and many other things. We believe we know why we defined stability as we did.
", "rawMarkdown": ">What you need is not to be able to predict the performance but to be able to put a good lower bound on what the performance could be. This is because the event of the model performing better than you might have prepared for is never a problem and as such does not need to be penalized. \n\nFrom business point of view this statement is not true. It's not only about the credit risk model, but whole underwriting setup, product parameters, profitability models and many other things. We believe we know why we defined stability as we did.", "votes": 3 }, { "id": 2654536, "postDate": "2024-02-16T07:47:46.097Z", "content": "In that case if a model performs \"too\" well in a given period, what is the negative consequence? What is the problem with it being \"too profitable\" one month?
", "rawMarkdown": "In that case if a model performs \"too\" well in a given period, what is the negative consequence? What is the problem with it being \"too profitable\" one month?", "votes": 3 }, { "id": 2654853, "postDate": "2024-02-16T13:36:07.927Z", "content": "The problem is not being too profitable, the problem is being unpredictable in production. What we want is to get a model that this predictable in performance for as long period as possible and ideally stable in time (in terms of performance). The score of a model is processed further with different processes. In other words, the score is not used directly to approve loans.
\nRegarding your previous post, I will test the proposed metric and reply to that later, I am working on the announced fix.
", "rawMarkdown": "The problem is not being too profitable, the problem is being unpredictable in production. What we want is to get a model that this predictable in performance for as long period as possible and ideally stable in time (in terms of performance). The score of a model is processed further with different processes. In other words, the score is not used directly to approve loans. \n\nRegarding your previous post, I will test the proposed metric and reply to that later, I am working on the announced fix. ", "votes": 1 }, { "id": 2654915, "postDate": "2024-02-16T14:52:13.080Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2662077, "postDate": "2024-02-21T17:28:55.827Z", "content": "Okay @jacobyjaeger I am back after testing it. I am here to provide you few examples where the current stability metric performs as we want it to perform, but your proposed metric does not select the correct option. Here are 5 pairs of simulated performance and A is always the preferred choice. The stability metric correctly says that the A is more preferred (with higher value of metric) and your metric says the opposite. Please have a look at this.
\n numpy np\n matplotlib pyplot plt\n\n ():\n x = np.arange((gini_in_time))\n y = gini_in_time\n a, b = np.polyfit(x, y, )\n y_hat = a*x + b\n residuals = y - y_hat\n res_std = np.std(residuals)\n avg_gini = np.mean(gini_in_time)\n avg_gini + w_fallingrate * (, a) + w_resstd * res_std\n\n ():\n x = np.cumsum(gini_in_time, axis=)\n x = np.concatenate([*x[:], x], )\n scores = -np.mean(-np.log(np.maximum((x[ma_len:] - x[:-ma_len])/ma_len, ))**exponent)\n scores \n\n\n(proposed_metric([, , , , ]), proposed_metric([, , , , ]))\n\n\n\ndata = [\n {: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : },\n {: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : },\n {: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : },\n {: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : },\n {: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : },\n]\n row data:\n target = row[]\n stability_metric_A = gini_stability(row[], w_fallingrate=, w_resstd=-)\n stability_metric_B = gini_stability(row[], w_fallingrate=, w_resstd=-)\n proposed_metric_A = proposed_metric(row[], exponent=, ma_len=)\n proposed_metric_B = proposed_metric(row[], exponent=, ma_len=)\n target == stability_metric_A > stability_metric_B proposed_metric_A < proposed_metric_B:\n ()\n ()\n plt.plot(row[], label=)\n plt.plot(row[], label=)\n plt.ylabel()\n plt.xlabel()\n plt.legend()\n plt.show()\nThe problem is that you're trying to extrapolate future performance beyond the test period which requires assigning negative weighting to early rounds. This is not suited to a competition because it is both hackable and noisy. The solution is to use a longer test period so you don't need to extrapolate.
\nAs I understand it, the test period already is not strictly future data, so why not just make it longer?
", "rawMarkdown": "The problem is that you're trying to extrapolate future performance beyond the test period which requires assigning negative weighting to early rounds. This is not suited to a competition because it is both hackable and noisy. The solution is to use a longer test period so you don't need to extrapolate.\n\nAs I understand it, the test period already is not strictly future data, so why not just make it longer?" }, { "id": 2662986, "postDate": "2024-02-22T09:11:07.990Z", "content": "I understand your points and I agree with the point that it is not perfectly suited for competition. You need to understand that The metric is done in a way that it very well approximates the decisions made in-house. This has the apparent benefit of directly considering the outcomes of the competition in our business. While the metric would only consider the timeframe on which kagglers operate it would not be directly applicable internally. This can be ofc disputed, but we have reasons for doing it in this way. Since we have problems with ppl hacking the LB we consider sacrificing this property if the new metric will be sufficiently similar. So far your metric is the closest one. https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/478699
", "rawMarkdown": "I understand your points and I agree with the point that it is not perfectly suited for competition. You need to understand that The metric is done in a way that it very well approximates the decisions made in-house. This has the apparent benefit of directly considering the outcomes of the competition in our business. While the metric would only consider the timeframe on which kagglers operate it would not be directly applicable internally. This can be ofc disputed, but we have reasons for doing it in this way. Since we have problems with ppl hacking the LB we consider sacrificing this property if the new metric will be sufficiently similar. So far your metric is the closest one. https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/478699" }, { "id": 2663102, "postDate": "2024-02-22T09:57:57Z", "content": "A metric with extrapolation should be about equal to the same metric without extrapolation taken over a longer test period. That is what extrapolation achieves, it compensates for a lack of future data.
\nUsing a longer test period should accomplish the same effect that extrapolating does, but better because you're using the actual future performance instead of guessing at it.
\nThere's nothing extrapolation does for you that extending the test period would not.
", "rawMarkdown": "A metric with extrapolation should be about equal to the same metric without extrapolation taken over a longer test period. That is what extrapolation achieves, it compensates for a lack of future data.\n\nUsing a longer test period should accomplish the same effect that extrapolating does, but better because you're using the actual future performance instead of guessing at it.\n\nThere's nothing extrapolation does for you that extending the test period would not." }, { "id": 2663109, "postDate": "2024-02-22T10:05:02.073Z", "content": "I agree only partially. For the sake of this competition, it would be ideal to have very long test sample so that we don't have to deal with the extrapolation anymore. Apart from the competition, the issue is that the decision of risk managers is usually about whether we should keep the model in production or not. Hence, they need to extrapolate the near future performance. How does it translate into the competition? Given that the metric approximates well in-house decisions winning model will be the one that is most likely kept in production after deployment on test set. If we knew beforehand that the deployment will be only for the period of test set, if would make sense to use only metric that takes the performance on test set without any kind of extrapolation. That is however rarely the case.
", "rawMarkdown": "I agree only partially. For the sake of this competition, it would be ideal to have very long test sample so that we don't have to deal with the extrapolation anymore. Apart from the competition, the issue is that the decision of risk managers is usually about whether we should keep the model in production or not. Hence, they need to extrapolate the near future performance. How does it translate into the competition? Given that the metric approximates well in-house decisions winning model will be the one that is most likely kept in production after deployment on test set. If we knew beforehand that the deployment will be only for the period of test set, if would make sense to use only metric that takes the performance on test set without any kind of extrapolation. That is however rarely the case. ", "votes": 1 }, { "id": 2663141, "postDate": "2024-02-22T10:38:35.467Z", "content": "You may have to guess at the future performance of models to decide whether to keep them running, but competition scores should not be guesses. They should be an evaluation of proven historical performance on real data.
\nIt's better to score on 6 months of real data than to score on 3 months of real data and whatever the linear fit predicts the next three months will be. If this differs from the equation you use in house it's because it's purpose differs from your needs in house. A competition needs a fair evaluation of performance without excessive randomness. This shouldn't be sacrificed just to make the evaluation period as recent as possible.
", "rawMarkdown": "You may have to guess at the future performance of models to decide whether to keep them running, but competition scores should not be guesses. They should be an evaluation of proven historical performance on real data. \n\nIt's better to score on 6 months of real data than to score on 3 months of real data and whatever the linear fit predicts the next three months will be. If this differs from the equation you use in house it's because it's purpose differs from your needs in house. A competition needs a fair evaluation of performance without excessive randomness. This shouldn't be sacrificed just to make the evaluation period as recent as possible.\n\n\n\n", "votes": 2 }, { "id": 2663413, "postDate": "2024-02-22T13:28:53.570Z", "content": "I don't think we could completely abandon the extrapolation nature of the current stability metric. It would indeed solve the hacking issue, but on the other hand, it would deviate from the intention of this competition. I can't see how we could make a metric that would have this extrapolation property and yet at the same it would be monotonic in respect to every prediction.
\nPerhaps there could be a compromise. We could lower the penalization that stems from the extrapolation somehow in new metric and at the same time have a metric that is monotonic in respect to majority of samples. We don't have to use gini by weeks as long as the alternative would be similar to gini. Think about it, we are willing to compromise, but we are unlikely to remove the intended extrapolation nature of the metric. Again the metric simulates rating of models when the decision is going to be made, whether it should be left in production or not.
", "rawMarkdown": "I don't think we could completely abandon the extrapolation nature of the current stability metric. It would indeed solve the hacking issue, but on the other hand, it would deviate from the intention of this competition. I can't see how we could make a metric that would have this extrapolation property and yet at the same it would be monotonic in respect to every prediction.\n\nPerhaps there could be a compromise. We could lower the penalization that stems from the extrapolation somehow in new metric and at the same time have a metric that is monotonic in respect to majority of samples. We don't have to use gini by weeks as long as the alternative would be similar to gini. Think about it, we are willing to compromise, but we are unlikely to remove the intended extrapolation nature of the metric. Again the metric simulates rating of models when the decision is going to be made, whether it should be left in production or not. ", "votes": 1 }, { "id": 2667400, "postDate": "2024-02-25T05:24:38.100Z", "content": "If the intention of the competition is to test stability, using a longer test period instead of extrapolation does not deviate from that intention, it better accomplishes it. To even call the metric a \"stability metric\" is misleading because it does not so much test stability as imperfectly simulate a longer test period. The metric I gave you is more of a stability metric in that it punishes models that experience periods of poor performance in a way that cannot be replicated by extending the test period.
\nTo say that the test set has to extrapolate because fund managers extrapolate seems like cargo cultism to me: a sort of naive copying without regard for the purpose or context of the original practice. While fund managers need to choose models that look like they will do well in the near future, competition scores should be based on real performance. At the end of the competition models can be retrained on new data and then projections can be made. A model that has done well for 3 months and is projected to do well for 3 months simply has not demonstrated better performance or stability than a model that has done well for 6 months.
\nThe dataset we have is also particularly unsuitable for extrapolation. The fact the features can disappear from the dataset creates a risk that a one time drop due to a feature leaving will be wrongly extrapolated as if it were a downward trend when no such trend is likely to materialize in reality. Obfuscating dates is also problematic because it precludes the creation of lagged time features that may otherwise improve models. And then there is of course the risk of de-obfuscation. You may think you can just disqualify any models that use such a technique, but it is always possible that features that appear legitimate also leak time information and choices that accomplish the goal of de-obfuscation are hidden behind what appear to be legitimate designs or designs that are too borderline to make a decision that isn't arbitrary. Kaggle has a long history of hosts thinking they have a bulletproof setup only to see their entire leaderboard populated by exploits.
\nI appreciate that you are willing to make changes, but if you are only willing change things unrelated to the problem nothing can be accomplished.
", "rawMarkdown": "If the intention of the competition is to test stability, using a longer test period instead of extrapolation does not deviate from that intention, it better accomplishes it. To even call the metric a \"stability metric\" is misleading because it does not so much test stability as imperfectly simulate a longer test period. The metric I gave you is more of a stability metric in that it punishes models that experience periods of poor performance in a way that cannot be replicated by extending the test period. \n\nTo say that the test set has to extrapolate because fund managers extrapolate seems like cargo cultism to me: a sort of naive copying without regard for the purpose or context of the original practice. While fund managers need to choose models that look like they will do well in the near future, competition scores should be based on real performance. At the end of the competition models can be retrained on new data and then projections can be made. A model that has done well for 3 months and is projected to do well for 3 months simply has not demonstrated better performance or stability than a model that has done well for 6 months. \n\nThe dataset we have is also particularly unsuitable for extrapolation. The fact the features can disappear from the dataset creates a risk that a one time drop due to a feature leaving will be wrongly extrapolated as if it were a downward trend when no such trend is likely to materialize in reality. Obfuscating dates is also problematic because it precludes the creation of lagged time features that may otherwise improve models. And then there is of course the risk of de-obfuscation. You may think you can just disqualify any models that use such a technique, but it is always possible that features that appear legitimate also leak time information and choices that accomplish the goal of de-obfuscation are hidden behind what appear to be legitimate designs or designs that are too borderline to make a decision that isn't arbitrary. Kaggle has a long history of hosts thinking they have a bulletproof setup only to see their entire leaderboard populated by exploits.\n\nI appreciate that you are willing to make changes, but if you are only willing change things unrelated to the problem nothing can be accomplished.", "votes": 1 }, { "id": 2750951, "postDate": "2024-04-13T23:18:07.583Z", "content": "This is worth looking at…
", "rawMarkdown": "This is worth looking at..." } ] } ] }, { "id": 2652189, "postDate": "2024-02-14T15:57:05.190Z", "content": "@jetakow
\nThe dates in the future are errors?
Or is it one of the following scenarios:
\nTy
", "rawMarkdown": "@jetakow \nThe dates in the future are errors?\n\nOr is it one of the following scenarios:\n\n- The dates are correct for various reasons even if they are in the future, as they pertain to an event that has actually occurred.\n- The dates are inconsistent because they result from calculations/extractions that sometimes lead to inconsistencies.\n\nTy", "votes": 1, "replies": [ { "id": 2653249, "postDate": "2024-02-15T10:12:01.880Z", "content": "We believe the majority will be the second case AND that we transformed all dates and there might be some inconsistencies as a result of that. There might be a case that the dates are from future in a sense that they indicate future date of something. Even if that is the case we don't think it will bring any significant information value.
", "rawMarkdown": "We believe the majority will be the second case AND that we transformed all dates and there might be some inconsistencies as a result of that. There might be a case that the dates are from future in a sense that they indicate future date of something. Even if that is the case we don't think it will bring any significant information value.", "votes": 2, "replies": [ { "id": 2671059, "postDate": "2024-02-27T09:12:08.220Z", "content": "@jetakow
\nActually transforming the dates will mess up a lot of features. For example, changing decision date will change the correct age of the applicant at the time of decision, which is a big discrepancy compared to train data. The same applies to FE around covid periods where people lost wages and were stood down…it will incorrectly apply these features to test data if the dates are altered.
I understand you concern and I agree that it will render a lot of features useless. We initially wanted to leave this information, but as you can see it was mainly exploited for metric hacking. Since the gain from metric hacking is about the same size as basic FE, we have to remove this information from dataset. It is still not certain, but very likely as of now. Kagglers will be given two additional weeks of time and they can use the time now to inspect train set further. Which is in my opinion very generous. The worst case scenario would be to leave it as it is now and just make the hack accessible to everyone.
", "rawMarkdown": "I understand you concern and I agree that it will render a lot of features useless. We initially wanted to leave this information, but as you can see it was mainly exploited for metric hacking. Since the gain from metric hacking is about the same size as basic FE, we have to remove this information from dataset. It is still not certain, but very likely as of now. Kagglers will be given two additional weeks of time and they can use the time now to inspect train set further. Which is in my opinion very generous. The worst case scenario would be to leave it as it is now and just make the hack accessible to everyone. ", "votes": 1 }, { "id": 2672017, "postDate": "2024-02-27T20:47:27.503Z", "content": "@jetakow OK will you please at least publish the dates that are no longer representative of the actual dates and whether at least the year or the month are correct? so for example if the concern is predicting WEEK_NUM then if the modified date is still within +/- 2 months of the correct date we can keep some features like age and COVID related behaviour. I am not asking to disclose the new transformation but just to indicate the fields that got new transformation and a rough indication of +/- N moths from the actual date.
", "rawMarkdown": "@jetakow OK will you please at least publish the dates that are no longer representative of the actual dates and whether at least the year or the month are correct? so for example if the concern is predicting WEEK_NUM then if the modified date is still within +/- 2 months of the correct date we can keep some features like age and COVID related behaviour. I am not asking to disclose the new transformation but just to indicate the fields that got new transformation and a rough indication of +/- N moths from the actual date." }, { "id": 2672103, "postDate": "2024-02-27T22:00:44.860Z", "content": "@fadylabib Thanks for asking, but no we will not do that. The point of transformation is to make it impossible to link to dates which there are now. So providing you this information would be the same as enabling you to exploit the metric hack again. Again, we know that by doing this we reduce the possible features that can be created. Also, please note that in production, the scoring is usually time-independent in that it doesn't matter at which time the scoring is done in the future.
", "rawMarkdown": "@fadylabib Thanks for asking, but no we will not do that. The point of transformation is to make it impossible to link to dates which there are now. So providing you this information would be the same as enabling you to exploit the metric hack again. Again, we know that by doing this we reduce the possible features that can be created. Also, please note that in production, the scoring is usually time-independent in that it doesn't matter at which time the scoring is done in the future. " }, { "id": 2672150, "postDate": "2024-02-28T00:23:49.490Z", "content": "@jetakow thank you. Will you be publishing the test columns that are having a different transformation from the train set or would this be done without explicitly saying which columns have different transforms ?
\nFor this part \"please note that in production, the scoring is usually time-independent\". I would disagree (I already mentioned that age is a factor and age will depend on the date e.g. if someone has / has not turned 18 even if the difference is just a few months) . Anyway, I understand that we will not know the dates, so please at least let us know which columns are no longer reliable.
", "rawMarkdown": "@jetakow thank you. Will you be publishing the test columns that are having a different transformation from the train set or would this be done without explicitly saying which columns have different transforms ?\n\nFor this part \"please note that in production, the scoring is usually time-independent\". I would disagree (I already mentioned that age is a factor and age will depend on the date e.g. if someone has / has not turned 18 even if the difference is just a few months) . Anyway, I understand that we will not know the dates, so please at least let us know which columns are no longer reliable.", "votes": 3 } ] } ] }, { "id": 2652301, "postDate": "2024-02-14T17:09:15.957Z", "content": "@jetakow congrats to you for the expert title. Thanks for being so proactive!
", "rawMarkdown": "@jetakow congrats to you for the expert title. Thanks for being so proactive!", "votes": 2, "replies": [ { "id": 2653245, "postDate": "2024-02-15T10:04:50.050Z", "content": "Thanks, I would never guess a year ago that I will get this first title thanks to kaggle competition :)
", "rawMarkdown": "Thanks, I would never guess a year ago that I will get this first title thanks to kaggle competition :)", "replies": [ { "id": 2821232, "postDate": "2024-05-18T02:50:51.567Z", "content": "Congrats Ravi
", "rawMarkdown": "Congrats Ravi" } ] } ] }, { "id": 2671055, "postDate": "2024-02-27T09:08:19.783Z", "content": "Regarding this point: The transformation was done to train and test before the start of the competition and now we will transform test data in different manner.
\nThis will create a big discrepancy between the two sets…for example if a solution is trying to do FE based on covid lock down periods (people were stood down, salaries were cut by 20%….etc). Having the dates processed differently in the test data will render all these creative FE ideas useless and they won't apply to test data. They will actually perform poorly if the dates were transformed.
\nMay I suggest that the we keep the focus on choosing the new metric that would avoid stability hacks without changing the test data. This proposed change is a big change that will make a lot of our efforts useless.
", "rawMarkdown": "Regarding this point: **The transformation was done to train and test before the start of the competition and now we will transform test data in different manner**. \n\nThis will create a big discrepancy between the two sets...for example if a solution is trying to do FE based on covid lock down periods (people were stood down, salaries were cut by 20%....etc). Having the dates processed differently in the test data will render all these creative FE ideas useless and they won't apply to test data. They will actually perform poorly if the dates were transformed.\n\nMay I suggest that the we keep the focus on choosing the new metric that would avoid stability hacks without changing the test data. This proposed change is a big change that will make a lot of our efforts useless.\n", "replies": [ { "id": 2671250, "postDate": "2024-02-27T12:09:08.233Z", "content": "I think there is no reason to panic. There has to be done some transformation so the test data are not the same as it is now. It should not change meaning of those features, so unless the FE is overfitted to train set or to LB, it won't be useless.
", "rawMarkdown": "I think there is no reason to panic. There has to be done some transformation so the test data are not the same as it is now. It should not change meaning of those features, so unless the FE is overfitted to train set or to LB, it won't be useless.", "votes": -1 } ] }, { "id": 2840141, "postDate": "2024-05-28T01:45:29.843Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2838965, "postDate": "2024-05-27T09:58:39.157Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2809713, "postDate": "2024-05-13T00:10:56.107Z", "rawMarkdown": "", "votes": 6, "isDeleted": true } ], "comments": [ { "id": 2652661, "author_name": "Jacoby Jaeger", "author_url": "", "post_date": "2024-02-14T23:10:47.477000", "content": "\n\nOmitting the falling rate would not be a satisfactory solution. Without the falling rate, we would greatly encourage unstable models in time. Those particular weights were selected so that they would perfectly represent the average decision of risk manager in HC. That is desirable as we will be able to use the knowledge gained from winners and compare it with our processes and models. Removing the falling rate would remove the stability component in time. This year we wanted to explore stability as the main topic.
\n
The falling rate was ill conceived. There should be no scenario where a model that has consistently poor performance is deemed better than a model that sometimes has good performance and sometimes poor performance. To use such a metric is to misunderstand what the value of stability is. That is, the value of stability is never in being consistently bad, but in avoiding periods that are particularly bad. Once this is understood, it becomes obvious that the best way to encourage stability is by aggressively punishing periods of particularly bad performance. We can do this with a metric that rapidly become worse when very bad samples are present. As I suggested in the other thread, this can be done with a metric of the form:
\nmean( ( -log( MovingAverage(gini) ) )**exponent ) with exponent>=1
\nUnder this metric, periods of poor performance are heavily penalized without the silliness of punishing good rounds. Stability should require nothing more than this.
", "votes": 8, "replies": [ { "id": 2653310, "author_name": "Daniel Herman", "author_url": "", "post_date": "2024-02-15T10:49:34.977000", "content": "\n\nThe falling rate was ill conceived. There should be no scenario where a model that has consistently poor performance is deemed better than a model that sometimes has good performance and sometimes poor performance. To use such a metric is to misunderstand what the value of stability is.
\n
Frankly, I think there is also some misunderstanding on your side. Consider this. From the perspective of risk managers at HC it is undesirable to have a model that is performing well let's say first 6 months and then it levels on for example 60%-80% of that performance. Even if the model performed better in terms of integral of weekly gini over time the model was unpredictable in sense that its performance lowered after the first 6 months. What is desirable is a model that has the same exact performance every week and if there is some decrease it is well predictable.
\nOmitting the falling rate term will allow models that will be performing well on some portion of test sample and then they will have decrease in performance that might not be even linear of exponential.
\nRegarding proposed metric, you could try to rewrite it using LaTeX with proper sums so it is clear what is applied to what. As it is now I am not sure if I understand it right.
\nRegarding the moving average idea, I believe that this might lower the amount of hacking, but it will not solve the problem completely. Consider following model that is performing well the first N weeks and during next 3 weeks the performance drops to 60% of that. By lowering the score artificially on the first N weeks you will improve the stability metric. That is aligned with risk managers' view of performance. It is not aligned with metric being fair and fun for everyone, we are working on that and I ask you for patience. The score from the metric that you suggested will improve when decreasing the score artificially first N weeks. If this is not the case, then I don't understand it perhaps correctly and you could explain it in finer detail.
", "votes": 3, "replies": [ { "id": 2653528, "author_name": "Simon Veitner", "author_url": "", "post_date": "2024-02-15T13:58:10.207000", "content": "I don’t think it will improve by lowering the score in first few weeks. Every lowering of gini score will result in a lower moving average. If moving average gets smaller, -log will get larger. The best performance is reached when all ginis in all relevant time windows are 1, then the above metric is 0 as log(1)=0.
\nThat badscores are punished heavily is also clear as the log decreases rapidly for arguments smaller than 1 (and even more so the closer we get to zero.)
\nPlease correct if I’m wrong @jacobyjaeger
\n\nFrankly, I think there is also some misunderstanding on your side. Consider this. From the perspective of risk managers at HC it is undesirable to have a model that is performing well let's say first 6 months and then it levels on for example 60%-80% of that performance.
\n
What you need is not to be able to predict the performance but to be able to put a good lower bound on what the performance could be. This is because the event of the model performing better than you might have prepared for is never a problem and as such does not need to be penalized. The set of models that can be given a high lower bound on their expected performance are those that have demonstrated a high lower bound during the validation period, which are those that perform well under a metric that heavily penalizes a given model's worst rounds as mine does.
\n\n\nRegarding proposed metric, you could try to rewrite it using LaTeX with proper sums so it is clear what is applied to what. As it is now I am not sure if I understand it right.
\n
Once predictions have been converted to ginis, there is only one axis that we need to be concerned with, the time axis. Here is a simple implementation of the metric. Higher scores are considered worse under this metric.
\n numpy np\n matplotlib pyplot plt\n\n\n ():\n (axis==)\n\n x = np.cumsum( x, axis= )\n\n x = np.concatenate([ *x[:], x ],)\n\n (x[n:] - x[:-n])/n\n\n\n ():\n \n\n ():\n\n scores = ( - np.log( MA( ginis, ma_len, axis= ) ) )**exponent\n\n scores\n\n metric \n\n\n\nN_GINIS = \nN_MA = \nEXPONENT = \n\nginis = np.random.rand(N_GINIS)\n\ntime = np.arange(N_GINIS)\n\n\nmetric = get_metric(EXPONENT, N_MA)\n\n\nscores = metric(ginis)\n\n\nplt.plot(time, ginis)\nplt.plot(time[N_MA//: N_GINIS-(N_MA-)//], scores)\n\n\nscore = np.mean(scores)\nscore\n\n\nOmitting the falling rate term will allow models that will be performing well on some portion of test sample and then they will have decrease in performance that might not be even linear of exponential.
\n
Any period of poor performance is heavily punished under my metric. But the falling rate can only detect periods of poor performance if they come at the end of the test period.
\n\n\nRegarding the moving average idea, I believe that this might lower the amount of hacking, but it will not solve the problem completely. Consider following model that is performing well the first N weeks and during next 3 weeks the performance drops to 60% of that. By lowering the score artificially on the first N weeks you will improve the stability metric. That is aligned with risk managers' view of performance.
\n
I'm not sure what you are trying to say here. Are you suggesting that performance on my metric would be improved by such hacking? That is not the case, it can be seen from the fact that the gradient wrt sample ginis is strictly negative that making any round's performance worse can only hurt performance under my metric.
\nOr are you saying that it is desirable that such hacking results in a higher score? That would be rather silly, you can make a declining performance look flat during the test period by making the early test rounds artificially worse, but that does not change the fact that true performance was declining which will become evident after the test period when you no longer have artificially deflated rounds left to compare the performance to.
", "votes": 0, "replies": [] }, { "id": 2654192, "author_name": "Jacoby Jaeger", "author_url": "", "post_date": "2024-02-16T00:26:12.940000", "content": "You're exactly right, Simon. We can check the sign of the gradient of the final score wrt the per round scores to prove that it is never desirable to make any round score worse. I suggest that kaggle should perform such a check on every competition metric so that we don't see hackable metrics again in the future.
", "votes": 2, "replies": [] }, { "id": 2654518, "author_name": "Tomas Jelinek", "author_url": "", "post_date": "2024-02-16T07:24:44.077000", "content": "\n\nWhat you need is not to be able to predict the performance but to be able to put a good lower bound on what the performance could be. This is because the event of the model performing better than you might have prepared for is never a problem and as such does not need to be penalized.
\n
From business point of view this statement is not true. It's not only about the credit risk model, but whole underwriting setup, product parameters, profitability models and many other things. We believe we know why we defined stability as we did.
", "votes": 3, "replies": [] }, { "id": 2654536, "author_name": "Jacoby Jaeger", "author_url": "", "post_date": "2024-02-16T07:47:46.097000", "content": "In that case if a model performs \"too\" well in a given period, what is the negative consequence? What is the problem with it being \"too profitable\" one month?
", "votes": 3, "replies": [] }, { "id": 2654853, "author_name": "Daniel Herman", "author_url": "", "post_date": "2024-02-16T13:36:07.927000", "content": "The problem is not being too profitable, the problem is being unpredictable in production. What we want is to get a model that this predictable in performance for as long period as possible and ideally stable in time (in terms of performance). The score of a model is processed further with different processes. In other words, the score is not used directly to approve loans.
\nRegarding your previous post, I will test the proposed metric and reply to that later, I am working on the announced fix.
", "votes": 1, "replies": [] }, { "id": 2654915, "author_name": "", "author_url": "", "post_date": "2024-02-16T14:52:13.080000", "content": "", "votes": 0, "replies": [] }, { "id": 2662077, "author_name": "Daniel Herman", "author_url": "", "post_date": "2024-02-21T17:28:55.827000", "content": "Okay @jacobyjaeger I am back after testing it. I am here to provide you few examples where the current stability metric performs as we want it to perform, but your proposed metric does not select the correct option. Here are 5 pairs of simulated performance and A is always the preferred choice. The stability metric correctly says that the A is more preferred (with higher value of metric) and your metric says the opposite. Please have a look at this.
\n numpy np\n matplotlib pyplot plt\n\n ():\n x = np.arange((gini_in_time))\n y = gini_in_time\n a, b = np.polyfit(x, y, )\n y_hat = a*x + b\n residuals = y - y_hat\n res_std = np.std(residuals)\n avg_gini = np.mean(gini_in_time)\n avg_gini + w_fallingrate * (, a) + w_resstd * res_std\n\n ():\n x = np.cumsum(gini_in_time, axis=)\n x = np.concatenate([*x[:], x], )\n scores = -np.mean(-np.log(np.maximum((x[ma_len:] - x[:-ma_len])/ma_len, ))**exponent)\n scores \n\n\n(proposed_metric([, , , , ]), proposed_metric([, , , , ]))\n\n\n\ndata = [\n {: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : },\n {: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : },\n {: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : },\n {: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : },\n {: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], : },\n]\n row data:\n target = row[]\n stability_metric_A = gini_stability(row[], w_fallingrate=, w_resstd=-)\n stability_metric_B = gini_stability(row[], w_fallingrate=, w_resstd=-)\n proposed_metric_A = proposed_metric(row[], exponent=, ma_len=)\n proposed_metric_B = proposed_metric(row[], exponent=, ma_len=)\n target == stability_metric_A > stability_metric_B proposed_metric_A < proposed_metric_B:\n ()\n ()\n plt.plot(row[], label=)\n plt.plot(row[], label=)\n plt.ylabel()\n plt.xlabel()\n plt.legend()\n plt.show()\nThe problem is that you're trying to extrapolate future performance beyond the test period which requires assigning negative weighting to early rounds. This is not suited to a competition because it is both hackable and noisy. The solution is to use a longer test period so you don't need to extrapolate.
\nAs I understand it, the test period already is not strictly future data, so why not just make it longer?
", "votes": 0, "replies": [] }, { "id": 2662986, "author_name": "Daniel Herman", "author_url": "", "post_date": "2024-02-22T09:11:07.990000", "content": "I understand your points and I agree with the point that it is not perfectly suited for competition. You need to understand that The metric is done in a way that it very well approximates the decisions made in-house. This has the apparent benefit of directly considering the outcomes of the competition in our business. While the metric would only consider the timeframe on which kagglers operate it would not be directly applicable internally. This can be ofc disputed, but we have reasons for doing it in this way. Since we have problems with ppl hacking the LB we consider sacrificing this property if the new metric will be sufficiently similar. So far your metric is the closest one. https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/478699
", "votes": 0, "replies": [] }, { "id": 2663102, "author_name": "Jacoby Jaeger", "author_url": "", "post_date": "2024-02-22T09:57:57", "content": "A metric with extrapolation should be about equal to the same metric without extrapolation taken over a longer test period. That is what extrapolation achieves, it compensates for a lack of future data.
\nUsing a longer test period should accomplish the same effect that extrapolating does, but better because you're using the actual future performance instead of guessing at it.
\nThere's nothing extrapolation does for you that extending the test period would not.
", "votes": 0, "replies": [] }, { "id": 2663109, "author_name": "Daniel Herman", "author_url": "", "post_date": "2024-02-22T10:05:02.073000", "content": "I agree only partially. For the sake of this competition, it would be ideal to have very long test sample so that we don't have to deal with the extrapolation anymore. Apart from the competition, the issue is that the decision of risk managers is usually about whether we should keep the model in production or not. Hence, they need to extrapolate the near future performance. How does it translate into the competition? Given that the metric approximates well in-house decisions winning model will be the one that is most likely kept in production after deployment on test set. If we knew beforehand that the deployment will be only for the period of test set, if would make sense to use only metric that takes the performance on test set without any kind of extrapolation. That is however rarely the case.
", "votes": 1, "replies": [] }, { "id": 2663141, "author_name": "Jacoby Jaeger", "author_url": "", "post_date": "2024-02-22T10:38:35.467000", "content": "You may have to guess at the future performance of models to decide whether to keep them running, but competition scores should not be guesses. They should be an evaluation of proven historical performance on real data.
\nIt's better to score on 6 months of real data than to score on 3 months of real data and whatever the linear fit predicts the next three months will be. If this differs from the equation you use in house it's because it's purpose differs from your needs in house. A competition needs a fair evaluation of performance without excessive randomness. This shouldn't be sacrificed just to make the evaluation period as recent as possible.
", "votes": 2, "replies": [] }, { "id": 2663413, "author_name": "Daniel Herman", "author_url": "", "post_date": "2024-02-22T13:28:53.570000", "content": "I don't think we could completely abandon the extrapolation nature of the current stability metric. It would indeed solve the hacking issue, but on the other hand, it would deviate from the intention of this competition. I can't see how we could make a metric that would have this extrapolation property and yet at the same it would be monotonic in respect to every prediction.
\nPerhaps there could be a compromise. We could lower the penalization that stems from the extrapolation somehow in new metric and at the same time have a metric that is monotonic in respect to majority of samples. We don't have to use gini by weeks as long as the alternative would be similar to gini. Think about it, we are willing to compromise, but we are unlikely to remove the intended extrapolation nature of the metric. Again the metric simulates rating of models when the decision is going to be made, whether it should be left in production or not.
", "votes": 1, "replies": [] }, { "id": 2667400, "author_name": "Jacoby Jaeger", "author_url": "", "post_date": "2024-02-25T05:24:38.100000", "content": "If the intention of the competition is to test stability, using a longer test period instead of extrapolation does not deviate from that intention, it better accomplishes it. To even call the metric a \"stability metric\" is misleading because it does not so much test stability as imperfectly simulate a longer test period. The metric I gave you is more of a stability metric in that it punishes models that experience periods of poor performance in a way that cannot be replicated by extending the test period.
\nTo say that the test set has to extrapolate because fund managers extrapolate seems like cargo cultism to me: a sort of naive copying without regard for the purpose or context of the original practice. While fund managers need to choose models that look like they will do well in the near future, competition scores should be based on real performance. At the end of the competition models can be retrained on new data and then projections can be made. A model that has done well for 3 months and is projected to do well for 3 months simply has not demonstrated better performance or stability than a model that has done well for 6 months.
\nThe dataset we have is also particularly unsuitable for extrapolation. The fact the features can disappear from the dataset creates a risk that a one time drop due to a feature leaving will be wrongly extrapolated as if it were a downward trend when no such trend is likely to materialize in reality. Obfuscating dates is also problematic because it precludes the creation of lagged time features that may otherwise improve models. And then there is of course the risk of de-obfuscation. You may think you can just disqualify any models that use such a technique, but it is always possible that features that appear legitimate also leak time information and choices that accomplish the goal of de-obfuscation are hidden behind what appear to be legitimate designs or designs that are too borderline to make a decision that isn't arbitrary. Kaggle has a long history of hosts thinking they have a bulletproof setup only to see their entire leaderboard populated by exploits.
\nI appreciate that you are willing to make changes, but if you are only willing change things unrelated to the problem nothing can be accomplished.
", "votes": 1, "replies": [] }, { "id": 2750951, "author_name": "Kabir Olawale Mohammed", "author_url": "", "post_date": "2024-04-13T23:18:07.583000", "content": "This is worth looking at…
", "votes": 0, "replies": [] } ] } ] }, { "id": 2652189, "author_name": "Davide Stenner", "author_url": "", "post_date": "2024-02-14T15:57:05.190000", "content": "@jetakow
\nThe dates in the future are errors?
Or is it one of the following scenarios:
\nTy
", "votes": 1, "replies": [ { "id": 2653249, "author_name": "Daniel Herman", "author_url": "", "post_date": "2024-02-15T10:12:01.880000", "content": "We believe the majority will be the second case AND that we transformed all dates and there might be some inconsistencies as a result of that. There might be a case that the dates are from future in a sense that they indicate future date of something. Even if that is the case we don't think it will bring any significant information value.
", "votes": 2, "replies": [ { "id": 2671059, "author_name": "ARMADA", "author_url": "", "post_date": "2024-02-27T09:12:08.220000", "content": "@jetakow
\nActually transforming the dates will mess up a lot of features. For example, changing decision date will change the correct age of the applicant at the time of decision, which is a big discrepancy compared to train data. The same applies to FE around covid periods where people lost wages and were stood down…it will incorrectly apply these features to test data if the dates are altered.
I understand you concern and I agree that it will render a lot of features useless. We initially wanted to leave this information, but as you can see it was mainly exploited for metric hacking. Since the gain from metric hacking is about the same size as basic FE, we have to remove this information from dataset. It is still not certain, but very likely as of now. Kagglers will be given two additional weeks of time and they can use the time now to inspect train set further. Which is in my opinion very generous. The worst case scenario would be to leave it as it is now and just make the hack accessible to everyone.
", "votes": 1, "replies": [] }, { "id": 2672017, "author_name": "ARMADA", "author_url": "", "post_date": "2024-02-27T20:47:27.503000", "content": "@jetakow OK will you please at least publish the dates that are no longer representative of the actual dates and whether at least the year or the month are correct? so for example if the concern is predicting WEEK_NUM then if the modified date is still within +/- 2 months of the correct date we can keep some features like age and COVID related behaviour. I am not asking to disclose the new transformation but just to indicate the fields that got new transformation and a rough indication of +/- N moths from the actual date.
", "votes": 0, "replies": [] }, { "id": 2672103, "author_name": "Daniel Herman", "author_url": "", "post_date": "2024-02-27T22:00:44.860000", "content": "@fadylabib Thanks for asking, but no we will not do that. The point of transformation is to make it impossible to link to dates which there are now. So providing you this information would be the same as enabling you to exploit the metric hack again. Again, we know that by doing this we reduce the possible features that can be created. Also, please note that in production, the scoring is usually time-independent in that it doesn't matter at which time the scoring is done in the future.
", "votes": 0, "replies": [] }, { "id": 2672150, "author_name": "ARMADA", "author_url": "", "post_date": "2024-02-28T00:23:49.490000", "content": "@jetakow thank you. Will you be publishing the test columns that are having a different transformation from the train set or would this be done without explicitly saying which columns have different transforms ?
\nFor this part \"please note that in production, the scoring is usually time-independent\". I would disagree (I already mentioned that age is a factor and age will depend on the date e.g. if someone has / has not turned 18 even if the difference is just a few months) . Anyway, I understand that we will not know the dates, so please at least let us know which columns are no longer reliable.
", "votes": 3, "replies": [] } ] } ] }, { "id": 2652301, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2024-02-14T17:09:15.957000", "content": "@jetakow congrats to you for the expert title. Thanks for being so proactive!
", "votes": 2, "replies": [ { "id": 2653245, "author_name": "Daniel Herman", "author_url": "", "post_date": "2024-02-15T10:04:50.050000", "content": "Thanks, I would never guess a year ago that I will get this first title thanks to kaggle competition :)
", "votes": 0, "replies": [ { "id": 2821232, "author_name": "vinothkarunakaran", "author_url": "", "post_date": "2024-05-18T02:50:51.567000", "content": "Congrats Ravi
", "votes": 0, "replies": [] } ] } ] }, { "id": 2671055, "author_name": "ARMADA", "author_url": "", "post_date": "2024-02-27T09:08:19.783000", "content": "Regarding this point: The transformation was done to train and test before the start of the competition and now we will transform test data in different manner.
\nThis will create a big discrepancy between the two sets…for example if a solution is trying to do FE based on covid lock down periods (people were stood down, salaries were cut by 20%….etc). Having the dates processed differently in the test data will render all these creative FE ideas useless and they won't apply to test data. They will actually perform poorly if the dates were transformed.
\nMay I suggest that the we keep the focus on choosing the new metric that would avoid stability hacks without changing the test data. This proposed change is a big change that will make a lot of our efforts useless.
", "votes": 0, "replies": [ { "id": 2671250, "author_name": "Daniel Herman", "author_url": "", "post_date": "2024-02-27T12:09:08.233000", "content": "I think there is no reason to panic. There has to be done some transformation so the test data are not the same as it is now. It should not change meaning of those features, so unless the FE is overfitted to train set or to LB, it won't be useless.
", "votes": -1, "replies": [] } ] }, { "id": 2840141, "author_name": "", "author_url": "", "post_date": "2024-05-28T01:45:29.843000", "content": "", "votes": 0, "replies": [] }, { "id": 2838965, "author_name": "", "author_url": "", "post_date": "2024-05-27T09:58:39.157000", "content": "", "votes": 0, "replies": [] }, { "id": 2809713, "author_name": "", "author_url": "", "post_date": "2024-05-13T00:10:56.107000", "content": "", "votes": 6, "replies": [] } ], "raw_markdown_by_id": { "2652153": "**Q: Why can't I submit a solution in my country?**\nA: We disabled the submissions as a temporary measure. It will be enabled again around 6.3.2024. https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/478716\n\n**Q: Questions regarding transformation of test data.**\nA: The transformation was done to train and test before the start of the competition and now we will transform test data in different manner. We won't specify what is the transformation exactly. We will keep column names, dtypes and the change is not going to be drastic. \n\n**Q: Any question related to private test set**\nA: We will not disclose any information about private test set. There is no point in asking questions about the private test set. We made the split in such a way that it will be fair for final evaluation.\n\n**Q: How long is the test set?**\nA: For the sake of stability evaluation it would not make sense to disclose you that information. This information would lead even to more \"hacks\" of the metric. Think about it this way. The model will be deployed and you don't know for how long it is going to be.\n\n**Q: What is the meaning of num_group1 and num_group2?**\nA: Those are indices that will help you to aggregate tables with depth=1,2. Please have a look at Data section.\n\n**Q: Any questions regarding stability metric and hacking.**\nPlease read https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/476867 first. We are working on a solution that would minimize such hacking by artificially worsening the score in the near future. \n\n**Q: Can't we just remove the falling rate from metric?**\nA: Omitting the falling rate would not be a satisfactory solution. Without the falling rate, we would greatly encourage unstable models in time. Those particular weights were selected so that they would perfectly represent the average decision of risk managers in HC. That is desirable as we will be able to use the knowledge gained from winners and compare it with our processes and models. Removing the falling rate would remove the stability component in time. This year we wanted to explore stability as the main topic.\n\n**Q: What is the reason for having those weights in the stability metric?**\nA: Those particular weights were selected so that they would perfectly represent the average decision of risk managers in HC. That is desirable as we will be able to use the knowledge gained from winners and compare it with our processes and models.\n\n**Q: Why is my notebook failing?**\nA: We don't have capacity to review every notebook, unfortunately. You can post it in discussion and maybe someone will help. In future, we can collect those errors into this thread. \n\n**Q: Dates are inconsistent and I see dates from the future in respect to date decision.**\nA: This might happen as all dates were transformed, but there might be some inconsistencies. Also, the underlying data from our primary sources can be flawed, sometimes it happens. \n\n**Q: What is the definition of the used target?**\nA: It's unpaid payment (one is enough) in certain time period. There is also some time tolerance (e.g. one day late is not default) and amount tolerance (if client paid $100 instead of $100.10). We can't disclose further details. ", "2652661": ">Omitting the falling rate would not be a satisfactory solution. Without the falling rate, we would greatly encourage unstable models in time. Those particular weights were selected so that they would perfectly represent the average decision of risk manager in HC. That is desirable as we will be able to use the knowledge gained from winners and compare it with our processes and models. Removing the falling rate would remove the stability component in time. This year we wanted to explore stability as the main topic.\n\nThe falling rate was ill conceived. There should be no scenario where a model that has consistently poor performance is deemed better than a model that sometimes has good performance and sometimes poor performance. To use such a metric is to misunderstand what the value of stability is. That is, the value of stability is never in being consistently bad, but in avoiding periods that are particularly bad. Once this is understood, it becomes obvious that the best way to encourage stability is by aggressively punishing periods of particularly bad performance. We can do this with a metric that rapidly become worse when very bad samples are present. As I suggested in the other thread, this can be done with a metric of the form: \n\nmean( ( -log( MovingAverage(gini) ) )**exponent ) with exponent>=1\n\nUnder this metric, periods of poor performance are heavily penalized without the silliness of punishing good rounds. Stability should require nothing more than this.", "2652189": "@jetakow \nThe dates in the future are errors?\n\nOr is it one of the following scenarios:\n\n- The dates are correct for various reasons even if they are in the future, as they pertain to an event that has actually occurred.\n- The dates are inconsistent because they result from calculations/extractions that sometimes lead to inconsistencies.\n\nTy", "2652301": "@jetakow congrats to you for the expert title. Thanks for being so proactive!", "2671055": "Regarding this point: **The transformation was done to train and test before the start of the competition and now we will transform test data in different manner**. \n\nThis will create a big discrepancy between the two sets...for example if a solution is trying to do FE based on covid lock down periods (people were stood down, salaries were cut by 20%....etc). Having the dates processed differently in the test data will render all these creative FE ideas useless and they won't apply to test data. They will actually perform poorly if the dates were transformed.\n\nMay I suggest that the we keep the focus on choosing the new metric that would avoid stability hacks without changing the test data. This proposed change is a big change that will make a lot of our efforts useless.\n", "2840141": "", "2838965": "", "2809713": "" } }