I have many ensembled models scoring 0.8999 on private which could lead me to the top 100 and on the public, they scored 0.8999 to 0.9012. But I chose 0.9042 and ends up having 363th.
\nOne of the best unusual things in this competition was no blending works. I just started working for this competition 10 days before the deadline. And trained a few models but didn't make it. Because I trust LB.
\nI wanted to know more about different Ensembling approaches and how to evaluate the model for ensembling. If anyone has something it can be helpful for new kagglers like me. Share it here.
", "messages": [ { "id": 1215536, "postDate": "2021-02-23T17:51:36.193Z", "content": "I have many ensembled models scoring 0.8999 on private which could lead me to the top 100 and on the public, they scored 0.8999 to 0.9012. But I chose 0.9042 and ends up having 363th.
\nOne of the best unusual things in this competition was no blending works. I just started working for this competition 10 days before the deadline. And trained a few models but didn't make it. Because I trust LB.
\nI wanted to know more about different Ensembling approaches and how to evaluate the model for ensembling. If anyone has something it can be helpful for new kagglers like me. Share it here.
", "rawMarkdown": "I have many ensembled models scoring 0.8999 on private which could lead me to the top 100 and on the public, they scored 0.8999 to 0.9012. But I chose 0.9042 and ends up having 363th.\n\nOne of the best unusual things in this competition was no blending works. I just started working for this competition 10 days before the deadline. And trained a few models but didn't make it. Because I trust LB.\n\n#### How to approach Ensembling?\nI wanted to know more about different Ensembling approaches and how to evaluate the model for ensembling. If anyone has something it can be helpful for new kagglers like me. Share it here.", "votes": 4 }, { "id": 1215722, "postDate": "2021-02-23T22:24:29.660Z", "content": "Congratulations to your first medal!
\nThere are different ensembling methods:
\n- Bagging and Pasting:
\n- Boosting
\n- Stacking
\nIn general the more different your models within an ensemble is the better the accuracy. Because different models tend to do seperate errors and you can compress the error by using multiple models.
\nOn top of that there are two different ways in handling each output from each model within an ensemble:
\nSoft voting usually results in better accuracy and should be choosen if possible.
\nYou can also give them different weights, I usually give the more accurate model a little higher weight than the others.
", "rawMarkdown": "Congratulations to your first medal!\n\nThere are different ensembling methods:\n\n**- Bagging and Pasting:**\n1. Bagging trains N models on K randomly choosed train samples (with using the same train sample multiple times)\n2. Pasting trains N models on K randomly choosed train samples (without using the same train sample multiple times)\n\n**- Boosting**\n1. AdaBoosting trains N models on the trainset, models have to be trained sequentially because they are depended on each other. After training a model the model will be evaluated on the trainset, the training samples with the most errors get higher weighted and the next model learns these more\n2. GradientBoosting trains N models on the trainset, models have to be trained sequentially because they are depended on each other. After training a model the model will be evaluated on the trainset again and the differences will be used to train the next model. The ensemble is the sum of all predictions from each model.\n\n**- Stacking**\n1. Training N models on a subset of the trainset afterwards you predict on the rest of the trainset which you hold out. These predictions will be used to train another model, so called Blender, which tries to weight the prediction of your N models to the Groundtruth value.\n\nIn general the more different your models within an ensemble is the better the accuracy. Because different models tend to do seperate errors and you can compress the error by using multiple models.\n\nOn top of that there are two different ways in handling each output from each model within an ensemble:\n\n- Soft voting: Average of probabilities of each model\n- Hard voting: Majority voting which class gets predicted the most gets choosen\n\nSoft voting usually results in better accuracy and should be choosen if possible.\n\nYou can also give them different weights, I usually give the more accurate model a little higher weight than the others.", "votes": 2, "replies": [ { "id": 1215893, "postDate": "2021-02-24T03:02:51.300Z", "content": "Thank you for explaining different methods @aliabdin1. For this competition, I trained 5 folds on two different architectures and then used Soft Voting. It works well.
\n@aliabdin1, according to your description, am I correct that only hold out subset is used for training of Blender model for Stacking? (i.e. part that participated in CV is not used here)
", "rawMarkdown": "@aliabdin1, according to your description, am I correct that only hold out subset is used for training of Blender model for Stacking? (i.e. part that participated in CV is not used here)" }, { "id": 1225405, "postDate": "2021-03-03T15:43:48.583Z", "content": "@dmitrynovikov Yes that is right. You would use the hold out subset to evaluate your trained models on these and the predictions will be used to train your Blender model (Input: predictions from N trained models evaluated on holdout subset Target: ground truth data).
", "rawMarkdown": "@dmitrynovikov Yes that is right. You would use the hold out subset to evaluate your trained models on these and the predictions will be used to train your Blender model (Input: predictions from N trained models evaluated on holdout subset Target: ground truth data).", "votes": 1 }, { "id": 1225540, "postDate": "2021-03-03T17:13:26.313Z", "content": "Thank you for your explanation
", "rawMarkdown": "Thank you for your explanation" } ] }, { "id": 1216236, "postDate": "2021-02-24T06:47:18.247Z", "content": "Congratulations!!
", "rawMarkdown": "Congratulations!!" }, { "id": 1215544, "postDate": "2021-02-23T18:03:58.957Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1225116, "postDate": "2021-03-03T10:32:26.910Z", "content": "Thank you very much
", "rawMarkdown": "Thank you very much" } ], "comments": [ { "id": 1215722, "author_name": "Ali Abdin", "author_url": "", "post_date": "2021-02-23T22:24:29.660000", "content": "Congratulations to your first medal!
\nThere are different ensembling methods:
\n- Bagging and Pasting:
\n- Boosting
\n- Stacking
\nIn general the more different your models within an ensemble is the better the accuracy. Because different models tend to do seperate errors and you can compress the error by using multiple models.
\nOn top of that there are two different ways in handling each output from each model within an ensemble:
\nSoft voting usually results in better accuracy and should be choosen if possible.
\nYou can also give them different weights, I usually give the more accurate model a little higher weight than the others.
", "votes": 2, "replies": [ { "id": 1215893, "author_name": "Vatsal Mavani", "author_url": "", "post_date": "2021-02-24T03:02:51.300000", "content": "Thank you for explaining different methods @aliabdin1. For this competition, I trained 5 folds on two different architectures and then used Soft Voting. It works well.
\n@aliabdin1, according to your description, am I correct that only hold out subset is used for training of Blender model for Stacking? (i.e. part that participated in CV is not used here)
", "votes": 0, "replies": [] }, { "id": 1225405, "author_name": "Ali Abdin", "author_url": "", "post_date": "2021-03-03T15:43:48.583000", "content": "@dmitrynovikov Yes that is right. You would use the hold out subset to evaluate your trained models on these and the predictions will be used to train your Blender model (Input: predictions from N trained models evaluated on holdout subset Target: ground truth data).
", "votes": 1, "replies": [] }, { "id": 1225540, "author_name": "DN", "author_url": "", "post_date": "2021-03-03T17:13:26.313000", "content": "Thank you for your explanation
", "votes": 0, "replies": [] } ] }, { "id": 1216236, "author_name": "Himanshu Mehndiratta", "author_url": "", "post_date": "2021-02-24T06:47:18.247000", "content": "Congratulations!!
", "votes": 0, "replies": [] }, { "id": 1215544, "author_name": "", "author_url": "", "post_date": "2021-02-23T18:03:58.957000", "content": "", "votes": 0, "replies": [] }, { "id": 1225116, "author_name": "", "author_url": "", "post_date": "2021-03-03T10:32:26.910000", "content": "Thank you very much
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1215536": "I have many ensembled models scoring 0.8999 on private which could lead me to the top 100 and on the public, they scored 0.8999 to 0.9012. But I chose 0.9042 and ends up having 363th.\n\nOne of the best unusual things in this competition was no blending works. I just started working for this competition 10 days before the deadline. And trained a few models but didn't make it. Because I trust LB.\n\n#### How to approach Ensembling?\nI wanted to know more about different Ensembling approaches and how to evaluate the model for ensembling. If anyone has something it can be helpful for new kagglers like me. Share it here.", "1215722": "Congratulations to your first medal!\n\nThere are different ensembling methods:\n\n**- Bagging and Pasting:**\n1. Bagging trains N models on K randomly choosed train samples (with using the same train sample multiple times)\n2. Pasting trains N models on K randomly choosed train samples (without using the same train sample multiple times)\n\n**- Boosting**\n1. AdaBoosting trains N models on the trainset, models have to be trained sequentially because they are depended on each other. After training a model the model will be evaluated on the trainset, the training samples with the most errors get higher weighted and the next model learns these more\n2. GradientBoosting trains N models on the trainset, models have to be trained sequentially because they are depended on each other. After training a model the model will be evaluated on the trainset again and the differences will be used to train the next model. The ensemble is the sum of all predictions from each model.\n\n**- Stacking**\n1. Training N models on a subset of the trainset afterwards you predict on the rest of the trainset which you hold out. These predictions will be used to train another model, so called Blender, which tries to weight the prediction of your N models to the Groundtruth value.\n\nIn general the more different your models within an ensemble is the better the accuracy. Because different models tend to do seperate errors and you can compress the error by using multiple models.\n\nOn top of that there are two different ways in handling each output from each model within an ensemble:\n\n- Soft voting: Average of probabilities of each model\n- Hard voting: Majority voting which class gets predicted the most gets choosen\n\nSoft voting usually results in better accuracy and should be choosen if possible.\n\nYou can also give them different weights, I usually give the more accurate model a little higher weight than the others.", "1216236": "Congratulations!!", "1215544": "", "1225116": "Thank you very much" } }