{ "id": 256225, "title": "Ensembling Ideas from previous competitions", "url": "/competitions/siim-covid19-detection/discussion/256225", "author_name": "Dr. Amritpal Singh", "post_date": "2021-07-31T13:59:46.927000", "votes": 5, "comment_count": 0, "views": 0, "content": "

I have curated a list of interesting ideas for ensemble models in computer vision. Some of them will be pretty simple ideas, the rest you might find them helpful.

\n

Collecting models for ensembling

\n\n

1. Varying Training Data

\n\n

2. Varying Models

\n\n

3. Varying Combinations

\n\n
\n

Interesting ideas implemented in previous Imaging competition

\n

1. Stacking with a LightGBM meta-model

\n

by @kozodoi - Link.- In Cassava Leaf Disease Classification
\nThe author performed Stacking with a LightGBM meta-model trained on the OOFs from the same CV folds, and combined 33 EfficientNet models(predicted one out of 5 classes) + 2 Pretrained ImageNet classifier predicting ImageNet classes (from 1 to 1000) + binary classifier predicting sick/healthy cassava (0/1).

\n

2. How to remove models from your pool?

\n\n

Reference - @kozodoi - in SIIM-ISIC Melanoma Classification, link

\n

3. Ensemble Pseudo Code

\n

by @cdeotte
\nHe talks about how to ensemble 50+ diverse models.
\nInsights shared

\n\n

Link to starter notebook

\n
 # START ENSEMBLE USING MODEL WITH LARGEST CV\n  Repeat until CV does not increase by 0.0003+ :\n    # TRY ADDING EVERY MODEL ONE AT A TIME AND REMEMBER \n    # HOW MUCH EACH INCREASES THE ENSEMBLE CV SCORE\n    for k in range( len(models) ):\n        for w in [0.01, 0.02, ..., 0.98, 0.99]:\n            # TRY ADDING MODEL k WITH WEIGHT w TO ENSEMBLE\n            trial = w * model[k,] + (1-w) * ensemble\n            auc_trial = roc_auc_score(true, trial)\n    # ADD ONE NEW MODEL TO ENSEMBLE THAT INCREASED CV THE MOST\n    # CHECK NEW CV SCORE. IF IT INCREASED REPEAT LOOP\n
\n

4. Ensemble using Weighted boxes fusion

\n\n
 pip install ensemble-boxes\n
\n

Another idea that I haven't studied yet -

\n\n

Thanks to all the amazing Kagglers that have shared their amazing work. 💛
\nI hope this is helpful.

", "messages": [ { "id": 1406105, "postDate": "2021-07-31T13:59:46.927Z", "content": "

I have curated a list of interesting ideas for ensemble models in computer vision. Some of them will be pretty simple ideas, the rest you might find them helpful.

\n

Collecting models for ensembling

\n\n

1. Varying Training Data

\n\n

2. Varying Models

\n\n

3. Varying Combinations

\n\n
\n

Interesting ideas implemented in previous Imaging competition

\n

1. Stacking with a LightGBM meta-model

\n

by @kozodoi - Link.- In Cassava Leaf Disease Classification
\nThe author performed Stacking with a LightGBM meta-model trained on the OOFs from the same CV folds, and combined 33 EfficientNet models(predicted one out of 5 classes) + 2 Pretrained ImageNet classifier predicting ImageNet classes (from 1 to 1000) + binary classifier predicting sick/healthy cassava (0/1).

\n

2. How to remove models from your pool?

\n\n

Reference - @kozodoi - in SIIM-ISIC Melanoma Classification, link

\n

3. Ensemble Pseudo Code

\n

by @cdeotte
\nHe talks about how to ensemble 50+ diverse models.
\nInsights shared

\n\n

Link to starter notebook

\n
 # START ENSEMBLE USING MODEL WITH LARGEST CV\n  Repeat until CV does not increase by 0.0003+ :\n    # TRY ADDING EVERY MODEL ONE AT A TIME AND REMEMBER \n    # HOW MUCH EACH INCREASES THE ENSEMBLE CV SCORE\n    for k in range( len(models) ):\n        for w in [0.01, 0.02, ..., 0.98, 0.99]:\n            # TRY ADDING MODEL k WITH WEIGHT w TO ENSEMBLE\n            trial = w * model[k,] + (1-w) * ensemble\n            auc_trial = roc_auc_score(true, trial)\n    # ADD ONE NEW MODEL TO ENSEMBLE THAT INCREASED CV THE MOST\n    # CHECK NEW CV SCORE. IF IT INCREASED REPEAT LOOP\n
\n

4. Ensemble using Weighted boxes fusion

\n\n
 pip install ensemble-boxes\n
\n

Another idea that I haven't studied yet -

\n\n

Thanks to all the amazing Kagglers that have shared their amazing work. 💛
\nI hope this is helpful.

", "rawMarkdown": "I have curated a list of interesting ideas for ensemble models in computer vision. Some of them will be pretty simple ideas, the rest you might find them helpful.\n\n## Collecting models for ensembling\n- Include High Variance models of Neural Network Models\n\n### 1. Varying Training Data\n- k-fold Cross-Validation Ensemble\n- Bootstrap Aggregation (Bagging) Ensemble\n- Random Training Subset Ensemble\n### 2. Varying Models\n- Multiple Training Run Ensemble\n- Hyperparameter Tuning Ensemble\n- Snapshot Ensemble\n- Horizontal Epochs Ensemble\n- Vertical Representational Ensemble\n### 3. Varying Combinations\n- Model Averaging Ensemble\n- Weighted Average Ensemble\n- Stacked Generalization (stacking) Ensemble\n- Boosting Ensemble\n- Model Weight Averaging Ensemble\n \n\n---\n\n# Interesting ideas implemented in previous Imaging competition\n\n### 1. Stacking with a LightGBM meta-model\nby @kozodoi - [Link](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/220751).- In Cassava Leaf Disease Classification \nThe author performed Stacking with a LightGBM meta-model trained on the OOFs from the same CV folds, and combined 33 EfficientNet models(predicted one out of 5 classes) + 2 Pretrained ImageNet classifier predicting ImageNet classes (from 1 to 1000) + binary classifier predicting sick/healthy cassava (0/1).\n\n### 2. How to remove models from your pool?\n- Removing models where the mean correlation of predictions with the other models demonstrated a large gap between OOF/test predictions\n - Explanation - Let take an example, where the correlation between their Out of Fold predictions(OOF) predictions is 0.9 but the correlation between their test predictions is 0.6. The gap is therefore 0.3. A large gap means that model predictions behave differently between local validation and test set, models might be overfitting the local data. \n- Removing models that ranked high in the adversarial validation model\n\nReference - @kozodoi - in SIIM-ISIC Melanoma Classification, [link](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175624)\n\n### 3. Ensemble Pseudo Code \nby @cdeotte \nHe talks about how to ensemble 50+ diverse models. \nInsights shared \n- Train all models using the same triple stratified folds seed = 42\n- Start with the model that has the largest CV and repeatedly try adding one model to increase a CV. \n- Whichever one additional model increases the CV the most (and at least 0.0003), keep that model and then iterate through all models again. Repeat this process until the CV score stops increasing (by at least 0.0003).\n\n[Link to starter notebook](https://www.kaggle.com/cdeotte/forward-selection-oof-ensemble-0-942-private/data)\n\n```\n # START ENSEMBLE USING MODEL WITH LARGEST CV\n Repeat until CV does not increase by 0.0003+ :\n # TRY ADDING EVERY MODEL ONE AT A TIME AND REMEMBER \n # HOW MUCH EACH INCREASES THE ENSEMBLE CV SCORE\n for k in range( len(models) ):\n for w in [0.01, 0.02, ..., 0.98, 0.99]:\n # TRY ADDING MODEL k WITH WEIGHT w TO ENSEMBLE\n trial = w * model[k,] + (1-w) * ensemble\n auc_trial = roc_auc_score(true, trial)\n # ADD ONE NEW MODEL TO ENSEMBLE THAT INCREASED CV THE MOST\n # CHECK NEW CV SCORE. IF IT INCREASED REPEAT LOOP\n\n```\n\n### 4. Ensemble using Weighted boxes fusion \n- [link](https://github.com/ZFTurbo/Weighted-Boxes-Fusion)\n- Install it using - \n```\n pip install ensemble-boxes\n```\n \nAnother idea that I haven't studied yet -\n- Random Transformation Ensembles (RTE) -by @kooaslansefat - [link](https://www.kaggle.com/c/shopee-product-matching/discussion/225128) \n \n\nThanks to all the amazing Kagglers that have shared their amazing work. 💛\nI hope this is helpful.", "votes": 5 } ], "comments": [], "raw_markdown_by_id": { "1406105": "I have curated a list of interesting ideas for ensemble models in computer vision. Some of them will be pretty simple ideas, the rest you might find them helpful.\n\n## Collecting models for ensembling\n- Include High Variance models of Neural Network Models\n\n### 1. Varying Training Data\n- k-fold Cross-Validation Ensemble\n- Bootstrap Aggregation (Bagging) Ensemble\n- Random Training Subset Ensemble\n### 2. Varying Models\n- Multiple Training Run Ensemble\n- Hyperparameter Tuning Ensemble\n- Snapshot Ensemble\n- Horizontal Epochs Ensemble\n- Vertical Representational Ensemble\n### 3. Varying Combinations\n- Model Averaging Ensemble\n- Weighted Average Ensemble\n- Stacked Generalization (stacking) Ensemble\n- Boosting Ensemble\n- Model Weight Averaging Ensemble\n \n\n---\n\n# Interesting ideas implemented in previous Imaging competition\n\n### 1. Stacking with a LightGBM meta-model\nby @kozodoi - [Link](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/220751).- In Cassava Leaf Disease Classification \nThe author performed Stacking with a LightGBM meta-model trained on the OOFs from the same CV folds, and combined 33 EfficientNet models(predicted one out of 5 classes) + 2 Pretrained ImageNet classifier predicting ImageNet classes (from 1 to 1000) + binary classifier predicting sick/healthy cassava (0/1).\n\n### 2. How to remove models from your pool?\n- Removing models where the mean correlation of predictions with the other models demonstrated a large gap between OOF/test predictions\n - Explanation - Let take an example, where the correlation between their Out of Fold predictions(OOF) predictions is 0.9 but the correlation between their test predictions is 0.6. The gap is therefore 0.3. A large gap means that model predictions behave differently between local validation and test set, models might be overfitting the local data. \n- Removing models that ranked high in the adversarial validation model\n\nReference - @kozodoi - in SIIM-ISIC Melanoma Classification, [link](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175624)\n\n### 3. Ensemble Pseudo Code \nby @cdeotte \nHe talks about how to ensemble 50+ diverse models. \nInsights shared \n- Train all models using the same triple stratified folds seed = 42\n- Start with the model that has the largest CV and repeatedly try adding one model to increase a CV. \n- Whichever one additional model increases the CV the most (and at least 0.0003), keep that model and then iterate through all models again. Repeat this process until the CV score stops increasing (by at least 0.0003).\n\n[Link to starter notebook](https://www.kaggle.com/cdeotte/forward-selection-oof-ensemble-0-942-private/data)\n\n```\n # START ENSEMBLE USING MODEL WITH LARGEST CV\n Repeat until CV does not increase by 0.0003+ :\n # TRY ADDING EVERY MODEL ONE AT A TIME AND REMEMBER \n # HOW MUCH EACH INCREASES THE ENSEMBLE CV SCORE\n for k in range( len(models) ):\n for w in [0.01, 0.02, ..., 0.98, 0.99]:\n # TRY ADDING MODEL k WITH WEIGHT w TO ENSEMBLE\n trial = w * model[k,] + (1-w) * ensemble\n auc_trial = roc_auc_score(true, trial)\n # ADD ONE NEW MODEL TO ENSEMBLE THAT INCREASED CV THE MOST\n # CHECK NEW CV SCORE. IF IT INCREASED REPEAT LOOP\n\n```\n\n### 4. Ensemble using Weighted boxes fusion \n- [link](https://github.com/ZFTurbo/Weighted-Boxes-Fusion)\n- Install it using - \n```\n pip install ensemble-boxes\n```\n \nAnother idea that I haven't studied yet -\n- Random Transformation Ensembles (RTE) -by @kooaslansefat - [link](https://www.kaggle.com/c/shopee-product-matching/discussion/225128) \n \n\nThanks to all the amazing Kagglers that have shared their amazing work. 💛\nI hope this is helpful." } }