![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F549055%2Fa173b3e5c5d2a174d1cda8e87206a333%2Fsimilar_distributions.png?generation=1658084914837906&alt=media\")
There is no simple answer to this question. As is often the case, it is best done by combining models to find out how the ensemble fares on the leaderboard.
\nStill, we can get a pretty good idea whether the models would work well together, and I mentioned in the other post a simple script that can give us information about model divergence. Here are a couple of things to keep in mind when looking at the output of that script:
\nPearson's correlation score: 0.993078).https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
\nIn simple terms, KS two-sample test tells us whether two one-dimensional probability distributions are different. Applied to our problem, it tells us how different the predictions are from any two models. I explained in this post my interpretation of what KS-stat numbers mean, and I will show it by plotting several model comparisons below. They all show cumulative distribution function (CDF) of the two models that are either very related, somewhat related, or not related at all. In all cases there is a red line connecting parts of CDFs with greatest difference.
\nThese two models have a KS-stat of 0.018626.
KS-stat = 0.030106
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F549055%2F6afcf04cc78e56b83fe713f30bda1e12%2Fmedium_distributions.png?generation=1658085019099942&alt=media\")
KS-stat = 0.206750
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F549055%2Fc6750ac5e69a9c7d53eb3b37c6feac9b%2Fdifferent_distributions.png?generation=1658085052975274&alt=media\")
For the last plot, below is the output of my script showing all correlation values. They are high even for relatively dissimilar models.
\n Column to be measured: prediction\n Pearson's correlation score: 0.909037\n Kendall's correlation score: 0.821022\n Spearman's correlation score: 0.959914\n Kolmogorov-Smirnov test: KS-stat = 0.206750 p-value = 0.000e+00\nThere is no simple answer to this question. As is often the case, it is best done by combining models to find out how the ensemble fares on the leaderboard.
\nStill, we can get a pretty good idea whether the models would work well together, and I mentioned in the other post a simple script that can give us information about model divergence. Here are a couple of things to keep in mind when looking at the output of that script:
\nPearson's correlation score: 0.993078).https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
\nIn simple terms, KS two-sample test tells us whether two one-dimensional probability distributions are different. Applied to our problem, it tells us how different the predictions are from any two models. I explained in this post my interpretation of what KS-stat numbers mean, and I will show it by plotting several model comparisons below. They all show cumulative distribution function (CDF) of the two models that are either very related, somewhat related, or not related at all. In all cases there is a red line connecting parts of CDFs with greatest difference.
\nThese two models have a KS-stat of 0.018626.
KS-stat = 0.030106
KS-stat = 0.206750
For the last plot, below is the output of my script showing all correlation values. They are high even for relatively dissimilar models.
\n Column to be measured: prediction\n Pearson's correlation score: 0.909037\n Kendall's correlation score: 0.821022\n Spearman's correlation score: 0.959914\n Kolmogorov-Smirnov test: KS-stat = 0.206750 p-value = 0.000e+00\nthanks for share.
", "rawMarkdown": "thanks for share.", "votes": null } ], "comments": [ { "id": 1861071, "author_name": "zhehaoliang", "author_url": "", "post_date": "07/18/2022 18:48:50", "content": "thanks for share.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1859621": "There is no simple answer to this question. As is often the case, it is best done by combining models to find out how the ensemble fares on the leaderboard.\n\nStill, we can get a pretty good idea whether the models would work well together, and I mentioned [in the other post](https://www.kaggle.com/competitions/amex-default-prediction/discussion/337610) a simple script that can give us information about model divergence. Here are a couple of things to keep in mind when looking at the output of that script:\n\n - Correlation alone is a good measure when comparing regression models where the target values are spread over a large range. When the targets are in a 0-1 range (all classification tasks), and especially when there is target imbalance (as in this competition), almost all models will have high correlations. You can see the printout in that post (for example, it says `Pearson's correlation score: 0.993078`).\n - Kolmogorov-Smirnov two-sample test is usually more informative in such cases where model correlations are high.\n\nhttps://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test\n\nIn simple terms, KS two-sample test tells us whether two one-dimensional probability distributions are different. Applied to our problem, it tells us how different the predictions are from any two models. I explained [in this post](https://www.kaggle.com/competitions/amex-default-prediction/discussion/337610) my interpretation of what KS-stat numbers mean, and I will show it by plotting several model comparisons below. They all show cumulative distribution function (CDF) of the two models that are either very related, somewhat related, or not related at all. In all cases there is a red line connecting parts of CDFs with greatest difference.\n\nThese two models have a KS-stat of `0.018626`.\n\n\n\n`KS-stat = 0.030106`\n\n\n\n`KS-stat = 0.206750`\n\n\n\nFor the last plot, below is the output of my script showing all correlation values. They are high even for relatively dissimilar models.\n\n```\n Column to be measured: prediction\n Pearson's correlation score: 0.909037\n Kendall's correlation score: 0.821022\n Spearman's correlation score: 0.959914\n Kolmogorov-Smirnov test: KS-stat = 0.206750 p-value = 0.000e+00\n\n```", "1861071": "thanks for share." }, "source": "meta" }