I posted a script long time ago that calculates correlations between models. It was for a different competition and will need to be adjusted slightly, but it should work.
\nGenerally speaking, models that have Kolmogorov-Smirnov statistic greater than 0.02 could be useful for ensembling. They should ensemble well if KS-stat > 0.05, and should give a really nice score boost when KS-stat > 0.1. Models don't even have to be made by different methods, although it will likely be helpful to add neural network and linear models to LGB/XGB models. Still, even two LGB models made from ~900 and ~700 features can be different enough, as shown below.
\n Column to be measured: prediction\n Pearson's correlation score: 0.993078\n Kendall's correlation score: 0.918780\n Spearman's correlation score: 0.990127\n Kolmogorov-Smirnov test: KS-stat = 0.060161 p-value = 0.000e+00\nEDIT: A longer explanation for why diverse models ensemble better is given here, also by yours truly.
", "messages": [ { "id": "1858309", "postDate": "07/16/2022 19:51:02", "content": "I posted a script long time ago that calculates correlations between models. It was for a different competition and will need to be adjusted slightly, but it should work.
\nGenerally speaking, models that have Kolmogorov-Smirnov statistic greater than 0.02 could be useful for ensembling. They should ensemble well if KS-stat > 0.05, and should give a really nice score boost when KS-stat > 0.1. Models don't even have to be made by different methods, although it will likely be helpful to add neural network and linear models to LGB/XGB models. Still, even two LGB models made from ~900 and ~700 features can be different enough, as shown below.
\n Column to be measured: prediction\n Pearson's correlation score: 0.993078\n Kendall's correlation score: 0.918780\n Spearman's correlation score: 0.990127\n Kolmogorov-Smirnov test: KS-stat = 0.060161 p-value = 0.000e+00\nEDIT: A longer explanation for why diverse models ensemble better is given here, also by yours truly.
", "rawMarkdown": "I posted [**a script**](https://www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge/discussion/50827) long time ago that calculates correlations between models. It was for a different competition and will need to be adjusted slightly, but it should work.\n\nGenerally speaking, models that have Kolmogorov-Smirnov statistic greater than 0.02 could be useful for ensembling. They should ensemble well if KS-stat > 0.05, and should give a really nice score boost when KS-stat > 0.1. Models don't even have to be made by different methods, although it will likely be helpful to add neural network and linear models to LGB/XGB models. Still, even two LGB models made from ~900 and ~700 features can be different enough, as shown below.\n\n```\n Column to be measured: prediction\n Pearson's correlation score: 0.993078\n Kendall's correlation score: 0.918780\n Spearman's correlation score: 0.990127\n Kolmogorov-Smirnov test: KS-stat = 0.060161 p-value = 0.000e+00\n\n```\n**EDIT:** A longer explanation for why diverse models ensemble better is given [**here**](https://www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge/discussion/51058), also by yours truly.", "votes": null }, { "id": "1858661", "postDate": "07/17/2022 05:43:29", "content": "Very good post, thanks for sharing. This is a new learning for me. I hope to ensemble better with this approach.
", "rawMarkdown": "Very good post, thanks for sharing. This is a new learning for me. I hope to ensemble better with this approach.", "votes": null }, { "id": "1858834", "postDate": "07/17/2022 08:41:36", "content": "Thanks for posting! Will try this out.
", "rawMarkdown": "Thanks for posting! Will try this out.", "votes": null }, { "id": "1858969", "postDate": "07/17/2022 09:49:39", "content": "Thanks for sharing . One question (since the blog is not longer available), what exactly is passed in function corr? It says model1.csv and model2.csv, what exactly does these files have?
Those are your submission models in .csv format, where the first column (in this case customer_ID) is ignored. The original script compares several columns named ['toxic', 'severe_toxic', 'obscene', 'threat', 'insult', 'identity_hate'] which were targets in that competition. For this competition the only column to compare should be named prediction.
It determines various correlation measures between prediction columns of the two models, followed by a KS 2-sample test which tries to ascertain whether they came from the same distribution. Models that are highly correlated will have a KS-stat value close to 0.
So we are comparing correlation between the predictions of models. Understood. Thank you !
", "rawMarkdown": "So we are comparing correlation between the predictions of models. Understood. Thank you !", "votes": null }, { "id": "1860890", "postDate": "07/18/2022 16:25:36", "content": "Do use this and pearsons but does get tricky if you have lot of models and then u need to do permutation and combinations to mix them all together . In these case u need some iterative code that tries combinations and picks out the best based on these statistics .
", "rawMarkdown": "Do use this and pearsons but does get tricky if you have lot of models and then u need to do permutation and combinations to mix them all together . In these case u need some iterative code that tries combinations and picks out the best based on these statistics .", "votes": null }, { "id": "1860927", "postDate": "07/18/2022 16:50:34", "content": "Assuming that all models follow the naming convention sub*.csv and that python code is saved in correlations.py, we can make a bash script by pasting the following lines into text file combinations.sh:
#!/bin/bash\nfor i in sub*.csv\ndo\n for j in sub*.csv\n do\n if [ \"$i\" \\< \"$j\" ]\n then\n echo correlations.py $i $j >> correlations.txt\n fi\n done\ndone\nThen make the script executive and run from a directory containing all models:
\nchmod +x combinations.sh\n./combinations.sh\nThe results will be saved in correlations.txt.
The whole thing can be done within the python script by finding a list of files and putting them through a double loop.
", "rawMarkdown": "Assuming that all models follow the naming convention `sub*.csv` and that python code is saved in `correlations.py`, we can make a `bash` script by pasting the following lines into text file `combinations.sh`:\n\n```\n#!/bin/bash\nfor i in sub*.csv\ndo\n for j in sub*.csv\n do\n if [ \"$i\" \\< \"$j\" ]\n then\n echo correlations.py $i $j >> correlations.txt\n fi\n done\ndone\n```\n\nThen make the script executive and run from a directory containing all models:\n\n```\nchmod +x combinations.sh\n./combinations.sh\n\n```\nThe results will be saved in `correlations.txt`.\n\nThe whole thing can be done within the python script by finding a list of files and putting them through a double loop.", "votes": null }, { "id": "1860935", "postDate": "07/18/2022 16:53:41", "content": "Yup I do it via something like list(itertools.combinations(features, 2))
make a similar design pattern grid search.
", "rawMarkdown": "make a similar design pattern grid search.", "votes": null }, { "id": "1861578", "postDate": "07/19/2022 05:27:24", "content": "Great info!! Learned something new. Thanks.
", "rawMarkdown": "Great info!! Learned something new. Thanks.", "votes": null }, { "id": "1862023", "postDate": "07/19/2022 12:01:19", "content": "thanks sir @tilii7
", "rawMarkdown": "thanks sir @tilii7", "votes": null }, { "id": "1864141", "postDate": "07/20/2022 18:54:22", "content": "If I understand correctly the reason you use KS testing between the models is to ensure they aren't overlapping in their outputs? Would these outputs be features? I am very new to ensemble learning.
", "rawMarkdown": "If I understand correctly the reason you use KS testing between the models is to ensure they aren't overlapping in their outputs? Would these outputs be features? I am very new to ensemble learning.", "votes": null }, { "id": "1864163", "postDate": "07/20/2022 19:29:09", "content": "We are testing model files that are being submitted for Kaggle evaluations - those 924621 predictions at the end of each modeling cycle. The idea is that if two models correlate very well - if their distributions are very similar - they will not give a boost after ensembling.
", "rawMarkdown": "We are testing model files that are being submitted for Kaggle evaluations - those 924621 predictions at the end of each modeling cycle. The idea is that if two models correlate very well - if their distributions are very similar - they will not give a boost after ensembling.", "votes": null }, { "id": "1864176", "postDate": "07/20/2022 19:49:13", "content": "This is because they are explaining the same information? There is no benefit to having two models that are always in agreement?
", "rawMarkdown": "This is because they are explaining the same information? There is no benefit to having two models that are always in agreement?", "votes": null }, { "id": "1864201", "postDate": "07/20/2022 20:24:40", "content": "All good models are in general agreement, but they shouldn't be too similar for the ensembling to work well.
\nHere is a toy example with two models that have 3 predictions each:
\n[0.01, 0.3, 0.9]\n[0.011, 0.305, 0.895]\nLet's say that targets we are trying to predict are [0, 0, 1]. No matter how you combine the two models, whether by simple average or some kind of weighting (say, 0.9*model1 + 0.1*model2), you will always end up with a model that is hardly any better because the original two predictions are near-identical.
Let's say that instead we have these two quite different models to combine:
\n[0.015, 0.31, 0.935]\n[0.005, 0.255, 0.91]\nIn this case the first model is better at predicting target 1, while the second model is better at predicting target 0. When combined they will give a model that should be significantly better than either one alone. So we are trying to find those models that differ enough as to be able to assemble well.
\nI wrote a longer explanation about the need for model diversity here.
", "rawMarkdown": "All good models are in general agreement, but they shouldn't be too similar for the ensembling to work well.\n\nHere is a toy example with two models that have 3 predictions each:\n\n [0.01, 0.3, 0.9]\n [0.011, 0.305, 0.895]\n\nLet's say that targets we are trying to predict are `[0, 0, 1]`. No matter how you combine the two models, whether by simple average or some kind of weighting (say, `0.9*model1 + 0.1*model2`), you will always end up with a model that is hardly any better because the original two predictions are near-identical.\n\nLet's say that instead we have these two quite different models to combine:\n\n [0.015, 0.31, 0.935]\n [0.005, 0.255, 0.91]\n\nIn this case the first model is better at predicting target 1, while the second model is better at predicting target 0. When combined they will give a model that should be significantly better than either one alone. So we are trying to find those models that differ enough as to be able to assemble well.\n\nI wrote a longer explanation about the need for model diversity [**here**](https://www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge/discussion/51058).", "votes": null } ], "comments": [ { "id": 1858661, "author_name": "ravi20076", "author_url": "", "post_date": "07/17/2022 05:43:29", "content": "Very good post, thanks for sharing. This is a new learning for me. I hope to ensemble better with this approach.
", "votes": null, "replies": [] }, { "id": 1858834, "author_name": "nijianzhang", "author_url": "", "post_date": "07/17/2022 08:41:36", "content": "Thanks for posting! Will try this out.
", "votes": null, "replies": [] }, { "id": 1858969, "author_name": "nitishraj", "author_url": "", "post_date": "07/17/2022 09:49:39", "content": "Thanks for sharing . One question (since the blog is not longer available), what exactly is passed in function corr? It says model1.csv and model2.csv, what exactly does these files have?
Those are your submission models in .csv format, where the first column (in this case customer_ID) is ignored. The original script compares several columns named ['toxic', 'severe_toxic', 'obscene', 'threat', 'insult', 'identity_hate'] which were targets in that competition. For this competition the only column to compare should be named prediction.
It determines various correlation measures between prediction columns of the two models, followed by a KS 2-sample test which tries to ascertain whether they came from the same distribution. Models that are highly correlated will have a KS-stat value close to 0.
So we are comparing correlation between the predictions of models. Understood. Thank you !
", "votes": null, "replies": [] } ] }, { "id": 1860890, "author_name": "gauravbrills", "author_url": "", "post_date": "07/18/2022 16:25:36", "content": "Do use this and pearsons but does get tricky if you have lot of models and then u need to do permutation and combinations to mix them all together . In these case u need some iterative code that tries combinations and picks out the best based on these statistics .
", "votes": null, "replies": [ { "id": 1860927, "author_name": "tilii7", "author_url": "", "post_date": "07/18/2022 16:50:34", "content": "Assuming that all models follow the naming convention sub*.csv and that python code is saved in correlations.py, we can make a bash script by pasting the following lines into text file combinations.sh:
#!/bin/bash\nfor i in sub*.csv\ndo\n for j in sub*.csv\n do\n if [ \"$i\" \\< \"$j\" ]\n then\n echo correlations.py $i $j >> correlations.txt\n fi\n done\ndone\nThen make the script executive and run from a directory containing all models:
\nchmod +x combinations.sh\n./combinations.sh\nThe results will be saved in correlations.txt.
The whole thing can be done within the python script by finding a list of files and putting them through a double loop.
", "votes": null, "replies": [] }, { "id": 1860935, "author_name": "gauravbrills", "author_url": "", "post_date": "07/18/2022 16:53:41", "content": "Yup I do it via something like list(itertools.combinations(features, 2))
make a similar design pattern grid search.
", "votes": null, "replies": [] }, { "id": 1861578, "author_name": "cid007", "author_url": "", "post_date": "07/19/2022 05:27:24", "content": "Great info!! Learned something new. Thanks.
", "votes": null, "replies": [] }, { "id": 1862023, "author_name": "bilalsuppal", "author_url": "", "post_date": "07/19/2022 12:01:19", "content": "thanks sir @tilii7
", "votes": null, "replies": [] }, { "id": 1864141, "author_name": "ianolmstead", "author_url": "", "post_date": "07/20/2022 18:54:22", "content": "If I understand correctly the reason you use KS testing between the models is to ensure they aren't overlapping in their outputs? Would these outputs be features? I am very new to ensemble learning.
", "votes": null, "replies": [ { "id": 1864163, "author_name": "tilii7", "author_url": "", "post_date": "07/20/2022 19:29:09", "content": "We are testing model files that are being submitted for Kaggle evaluations - those 924621 predictions at the end of each modeling cycle. The idea is that if two models correlate very well - if their distributions are very similar - they will not give a boost after ensembling.
", "votes": null, "replies": [] }, { "id": 1864176, "author_name": "ianolmstead", "author_url": "", "post_date": "07/20/2022 19:49:13", "content": "This is because they are explaining the same information? There is no benefit to having two models that are always in agreement?
", "votes": null, "replies": [] }, { "id": 1864201, "author_name": "tilii7", "author_url": "", "post_date": "07/20/2022 20:24:40", "content": "All good models are in general agreement, but they shouldn't be too similar for the ensembling to work well.
\nHere is a toy example with two models that have 3 predictions each:
\n[0.01, 0.3, 0.9]\n[0.011, 0.305, 0.895]\nLet's say that targets we are trying to predict are [0, 0, 1]. No matter how you combine the two models, whether by simple average or some kind of weighting (say, 0.9*model1 + 0.1*model2), you will always end up with a model that is hardly any better because the original two predictions are near-identical.
Let's say that instead we have these two quite different models to combine:
\n[0.015, 0.31, 0.935]\n[0.005, 0.255, 0.91]\nIn this case the first model is better at predicting target 1, while the second model is better at predicting target 0. When combined they will give a model that should be significantly better than either one alone. So we are trying to find those models that differ enough as to be able to assemble well.
\nI wrote a longer explanation about the need for model diversity here.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1858309": "I posted [**a script**](https://www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge/discussion/50827) long time ago that calculates correlations between models. It was for a different competition and will need to be adjusted slightly, but it should work.\n\nGenerally speaking, models that have Kolmogorov-Smirnov statistic greater than 0.02 could be useful for ensembling. They should ensemble well if KS-stat > 0.05, and should give a really nice score boost when KS-stat > 0.1. Models don't even have to be made by different methods, although it will likely be helpful to add neural network and linear models to LGB/XGB models. Still, even two LGB models made from ~900 and ~700 features can be different enough, as shown below.\n\n```\n Column to be measured: prediction\n Pearson's correlation score: 0.993078\n Kendall's correlation score: 0.918780\n Spearman's correlation score: 0.990127\n Kolmogorov-Smirnov test: KS-stat = 0.060161 p-value = 0.000e+00\n\n```\n**EDIT:** A longer explanation for why diverse models ensemble better is given [**here**](https://www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge/discussion/51058), also by yours truly.", "1858661": "Very good post, thanks for sharing. This is a new learning for me. I hope to ensemble better with this approach.", "1858834": "Thanks for posting! Will try this out.", "1858969": "Thanks for sharing . One question (since the blog is not longer available), what exactly is passed in function `corr`? It says model1.csv and model2.csv, what exactly does these files have?", "1859547": "Those are your submission models in .csv format, where the first column (in this case `customer_ID`) is ignored. The original script compares several columns named `['toxic', 'severe_toxic', 'obscene', 'threat', 'insult', 'identity_hate']` which were targets in that competition. For this competition the only column to compare should be named `prediction`. \n\nIt determines various correlation measures between `prediction` columns of the two models, followed by a KS 2-sample test which tries to ascertain whether they came from the same distribution. Models that are highly correlated will have a KS-stat value close to 0.", "1859566": "So we are comparing correlation between the predictions of models. Understood. Thank you !", "1860890": "Do use this and pearsons but does get tricky if you have lot of models and then u need to do permutation and combinations to mix them all together . In these case u need some iterative code that tries combinations and picks out the best based on these statistics .", "1860927": "Assuming that all models follow the naming convention `sub*.csv` and that python code is saved in `correlations.py`, we can make a `bash` script by pasting the following lines into text file `combinations.sh`:\n\n```\n#!/bin/bash\nfor i in sub*.csv\ndo\n for j in sub*.csv\n do\n if [ \"$i\" \\< \"$j\" ]\n then\n echo correlations.py $i $j >> correlations.txt\n fi\n done\ndone\n```\n\nThen make the script executive and run from a directory containing all models:\n\n```\nchmod +x combinations.sh\n./combinations.sh\n\n```\nThe results will be saved in `correlations.txt`.\n\nThe whole thing can be done within the python script by finding a list of files and putting them through a double loop.", "1860935": "Yup I do it via something like `list(itertools.combinations(features, 2)) `", "1861061": "make a similar design pattern grid search.", "1861578": "Great info!! Learned something new. Thanks.", "1862023": "thanks sir @tilii7", "1864141": "If I understand correctly the reason you use KS testing between the models is to ensure they aren't overlapping in their outputs? Would these outputs be features? I am very new to ensemble learning.", "1864163": "We are testing model files that are being submitted for Kaggle evaluations - those 924621 predictions at the end of each modeling cycle. The idea is that if two models correlate very well - if their distributions are very similar - they will not give a boost after ensembling.", "1864176": "This is because they are explaining the same information? There is no benefit to having two models that are always in agreement?", "1864201": "All good models are in general agreement, but they shouldn't be too similar for the ensembling to work well.\n\nHere is a toy example with two models that have 3 predictions each:\n\n [0.01, 0.3, 0.9]\n [0.011, 0.305, 0.895]\n\nLet's say that targets we are trying to predict are `[0, 0, 1]`. No matter how you combine the two models, whether by simple average or some kind of weighting (say, `0.9*model1 + 0.1*model2`), you will always end up with a model that is hardly any better because the original two predictions are near-identical.\n\nLet's say that instead we have these two quite different models to combine:\n\n [0.015, 0.31, 0.935]\n [0.005, 0.255, 0.91]\n\nIn this case the first model is better at predicting target 1, while the second model is better at predicting target 0. When combined they will give a model that should be significantly better than either one alone. So we are trying to find those models that differ enough as to be able to assemble well.\n\nI wrote a longer explanation about the need for model diversity [**here**](https://www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge/discussion/51058)." }, "source": "meta" }