Hi guys, I'm wondering how to pick up my model appropriately in order to apply ensemble method.
\nThese are my consideration to pick up the one.
\nUsing same construction of model.
\nSome people use separate models to apply ensemble, but I'm not familiar with the approach. Therefore, I'm using same structure of model.
Apply different hyperparameters.
If I use 3-fold ensemble, I choose separate options. For instance, The first is 24 bin, second 26, other 28..
\nOr different learning rate: 0.005, 0.0001, 0.0005, etc.,
This is my consideration to train model well.
\nAre there other options to distinguish these models?
", "messages": [ { "id": "2209252", "postDate": "04/04/2023 15:16:22", "content": "Hi guys, I'm wondering how to pick up my model appropriately in order to apply ensemble method.
\nThese are my consideration to pick up the one.
\nUsing same construction of model.
\nSome people use separate models to apply ensemble, but I'm not familiar with the approach. Therefore, I'm using same structure of model.
Apply different hyperparameters.
If I use 3-fold ensemble, I choose separate options. For instance, The first is 24 bin, second 26, other 28..
\nOr different learning rate: 0.005, 0.0001, 0.0005, etc.,
This is my consideration to train model well.
\nAre there other options to distinguish these models?
", "rawMarkdown": "Hi guys, I'm wondering how to pick up my model appropriately in order to apply ensemble method.\n\nThese are my consideration to pick up the one.\n\n1. Using same construction of model.\nSome people use separate models to apply ensemble, but I'm not familiar with the approach. Therefore, I'm using same structure of model.\n\n2. Apply different hyperparameters.\n\nIf I use 3-fold ensemble, I choose separate options. For instance, The first is 24 bin, second 26, other 28.. \nOr different learning rate: 0.005, 0.0001, 0.0005, etc.,\n\nThis is my consideration to train model well.\n\nAre there other options to distinguish these models?", "votes": null }, { "id": "2209343", "postDate": "04/04/2023 16:16:01", "content": "This LSTM kernel is using an ensemble from the same model, just at different epochs.
", "rawMarkdown": "This [LSTM kernel](https://www.kaggle.com/code/rsmits/tensorflow-lstm-model-inference) is using an ensemble from the same model, just at different epochs.", "votes": null }, { "id": "2210743", "postDate": "04/05/2023 15:39:19", "content": "@hyunsoolee1010 It is not a golden rule but ensembling different models could lead to a higher performance. There's nothing wrong with combining different models as long as you make sure that the output results are processed properly.
\nCombining the models saved on different epochs in one and the same training run is the easiest to do…it also saves you the computation time of running different training runs. The effect will not be really great.
\nCombining different models from a cross validation run will likely give you a better ensemble as the models are more diverse.
\nUsing different bin numbers is also an interresting idea but I have no idea how that compares to the ideas above.
", "rawMarkdown": "hyunsoolee1010 It is not a golden rule but ensembling different models could lead to a higher performance. There's nothing wrong with combining different models as long as you make sure that the output results are processed properly.\n\nCombining the models saved on different epochs in one and the same training run is the easiest to do...it also saves you the computation time of running different training runs. The effect will not be really great.\n\nCombining different models from a cross validation run will likely give you a better ensemble as the models are more diverse.\n\nUsing different bin numbers is also an interresting idea but I have no idea how that compares to the ideas above.", "votes": null }, { "id": "2210763", "postDate": "04/05/2023 15:49:17", "content": "Thanks for your suggestion, I'm gonna refer to it.
", "rawMarkdown": "Thanks for your suggestion, I'm gonna refer to it.", "votes": null }, { "id": "2210783", "postDate": "04/05/2023 16:01:54", "content": "Thanks for your advice. As you mention, different epochs in one model are an unexpected great result. (just one model -> 1.031. After ensembling others -> 1.027) I thought that it couldn't produce a better result, but it actually can. It's interesting because this means that slightly different weights can affect performance.
\nAt this time, combining other models is not familiar to me, but I should befriend it. At this time, can I consider only logits? or there are other factors that I have to do that?
\nActually, I don't know how much the number of bins can affect model performance. Therefore, it should be a consideration later.
", "rawMarkdown": "Thanks for your advice. As you mention, different epochs in one model are an unexpected great result. (just one model -> 1.031. After ensembling others -> 1.027) I thought that it couldn't produce a better result, but it actually can. It's interesting because this means that slightly different weights can affect performance.\n\nAt this time, combining other models is not familiar to me, but I should befriend it. At this time, can I consider only logits? or there are other factors that I have to do that?\n\nActually, I don't know how much the number of bins can affect model performance. Therefore, it should be a consideration later.", "votes": null }, { "id": "2210845", "postDate": "04/05/2023 16:41:32", "content": "If you want to combine the models with different bin_numbers then you have to first use the predictions from each model to construct the zenith and azimuth predictions. In the LSTM inference notebook you can then use the function 'weighted_vector_ensemble' to combine those predictions from models with different bin numbers.
\nThere is some more work to it but if you follow all the code then with the above hints you should be able to get it to work :-)
", "rawMarkdown": "If you want to combine the models with different bin_numbers then you have to first use the predictions from each model to construct the zenith and azimuth predictions. In the LSTM inference notebook you can then use the function 'weighted_vector_ensemble' to combine those predictions from models with different bin numbers.\n\nThere is some more work to it but if you follow all the code then with the above hints you should be able to get it to work :-)", "votes": null }, { "id": "2212546", "postDate": "04/06/2023 21:12:25", "content": "@hyunsoolee1010 BTW, this is the author of the kernel I mentioned :)
", "rawMarkdown": "hyunsoolee1010 BTW, this is the author of the kernel I mentioned :)", "votes": null } ], "comments": [ { "id": 2209343, "author_name": "araraonline", "author_url": "", "post_date": "04/04/2023 16:16:01", "content": "This LSTM kernel is using an ensemble from the same model, just at different epochs.
", "votes": null, "replies": [ { "id": 2210763, "author_name": "hyunsoolee1010", "author_url": "", "post_date": "04/05/2023 15:49:17", "content": "Thanks for your suggestion, I'm gonna refer to it.
", "votes": null, "replies": [] } ] }, { "id": 2210743, "author_name": "rsmits", "author_url": "", "post_date": "04/05/2023 15:39:19", "content": "@hyunsoolee1010 It is not a golden rule but ensembling different models could lead to a higher performance. There's nothing wrong with combining different models as long as you make sure that the output results are processed properly.
\nCombining the models saved on different epochs in one and the same training run is the easiest to do…it also saves you the computation time of running different training runs. The effect will not be really great.
\nCombining different models from a cross validation run will likely give you a better ensemble as the models are more diverse.
\nUsing different bin numbers is also an interresting idea but I have no idea how that compares to the ideas above.
", "votes": null, "replies": [ { "id": 2210783, "author_name": "hyunsoolee1010", "author_url": "", "post_date": "04/05/2023 16:01:54", "content": "Thanks for your advice. As you mention, different epochs in one model are an unexpected great result. (just one model -> 1.031. After ensembling others -> 1.027) I thought that it couldn't produce a better result, but it actually can. It's interesting because this means that slightly different weights can affect performance.
\nAt this time, combining other models is not familiar to me, but I should befriend it. At this time, can I consider only logits? or there are other factors that I have to do that?
\nActually, I don't know how much the number of bins can affect model performance. Therefore, it should be a consideration later.
", "votes": null, "replies": [ { "id": 2210845, "author_name": "rsmits", "author_url": "", "post_date": "04/05/2023 16:41:32", "content": "If you want to combine the models with different bin_numbers then you have to first use the predictions from each model to construct the zenith and azimuth predictions. In the LSTM inference notebook you can then use the function 'weighted_vector_ensemble' to combine those predictions from models with different bin numbers.
\nThere is some more work to it but if you follow all the code then with the above hints you should be able to get it to work :-)
", "votes": null, "replies": [ { "id": 2212546, "author_name": "araraonline", "author_url": "", "post_date": "04/06/2023 21:12:25", "content": "@hyunsoolee1010 BTW, this is the author of the kernel I mentioned :)
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2209252": "Hi guys, I'm wondering how to pick up my model appropriately in order to apply ensemble method.\n\nThese are my consideration to pick up the one.\n\n1. Using same construction of model.\nSome people use separate models to apply ensemble, but I'm not familiar with the approach. Therefore, I'm using same structure of model.\n\n2. Apply different hyperparameters.\n\nIf I use 3-fold ensemble, I choose separate options. For instance, The first is 24 bin, second 26, other 28.. \nOr different learning rate: 0.005, 0.0001, 0.0005, etc.,\n\nThis is my consideration to train model well.\n\nAre there other options to distinguish these models?", "2209343": "This [LSTM kernel](https://www.kaggle.com/code/rsmits/tensorflow-lstm-model-inference) is using an ensemble from the same model, just at different epochs.", "2210743": "hyunsoolee1010 It is not a golden rule but ensembling different models could lead to a higher performance. There's nothing wrong with combining different models as long as you make sure that the output results are processed properly.\n\nCombining the models saved on different epochs in one and the same training run is the easiest to do...it also saves you the computation time of running different training runs. The effect will not be really great.\n\nCombining different models from a cross validation run will likely give you a better ensemble as the models are more diverse.\n\nUsing different bin numbers is also an interresting idea but I have no idea how that compares to the ideas above.", "2210763": "Thanks for your suggestion, I'm gonna refer to it.", "2210783": "Thanks for your advice. As you mention, different epochs in one model are an unexpected great result. (just one model -> 1.031. After ensembling others -> 1.027) I thought that it couldn't produce a better result, but it actually can. It's interesting because this means that slightly different weights can affect performance.\n\nAt this time, combining other models is not familiar to me, but I should befriend it. At this time, can I consider only logits? or there are other factors that I have to do that?\n\nActually, I don't know how much the number of bins can affect model performance. Therefore, it should be a consideration later.", "2210845": "If you want to combine the models with different bin_numbers then you have to first use the predictions from each model to construct the zenith and azimuth predictions. In the LSTM inference notebook you can then use the function 'weighted_vector_ensemble' to combine those predictions from models with different bin numbers.\n\nThere is some more work to it but if you follow all the code then with the above hints you should be able to get it to work :-)", "2212546": "hyunsoolee1010 BTW, this is the author of the kernel I mentioned :)" }, "source": "meta" }