{ "id": 213278, "title": "How to weight the model，confusion matrix help me get 0.906", "url": "/competitions/cassava-leaf-disease-classification/discussion/213278", "author_name": "Hanson0910", "post_date": "2021-01-22T08:16:47.874000", "votes": 48, "comment_count": 15, "views": 0, "content": "

Using mult folds cross validation and mult model ensemble is a common method to improve score,but the performance of different models in different categories is different.If we give each model the same weight, I think it's unreasonable,so how to assign the right weight to each model is worth trying.By saving the confusion matrix of each model, a model with the highest accuracy in the specified category is selected,because we have five categories, we choose five models，finally, the sofamax function is obtained for each category of the five models，we can get the weight of each model in each category.In this way, my LB score increased from 0.905 to 0.906.
\none of the model confusion matrix as follows:
\n\"\"

", "messages": [ { "id": 1164225, "postDate": "2021-01-22T08:16:47.873Z", "content": "

Using mult folds cross validation and mult model ensemble is a common method to improve score,but the performance of different models in different categories is different.If we give each model the same weight, I think it's unreasonable,so how to assign the right weight to each model is worth trying.By saving the confusion matrix of each model, a model with the highest accuracy in the specified category is selected,because we have five categories, we choose five models，finally, the sofamax function is obtained for each category of the five models，we can get the weight of each model in each category.In this way, my LB score increased from 0.905 to 0.906.
\none of the model confusion matrix as follows:
\n\"\"

", "rawMarkdown": "**Using mult folds cross validation and mult model ensemble is a common method to improve score,but the performance of different models in different categories is different.If we give each model the same weight, I think it's unreasonable,so how to assign the right weight to each model is worth trying.By saving the confusion matrix of each model, a model with the highest accuracy in the specified category is selected,because we have five categories, we choose five models，finally, the sofamax function is obtained for each category of the five models，we can get the weight of each model in each category.In this way, my LB score increased from 0.905 to 0.906.**\none of the model confusion matrix as follows:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4848897%2F18faaf3e6d8fe439a94b908a497fd3ea%2Fcheckpoint-9.jpg?generation=1611305510368531&alt=media)", "votes": 48 }, { "id": 1164427, "postDate": "2021-01-22T11:15:28.057Z", "content": "

@hanson0910 I think by using the confusion matrix as a point to learn the weights might make the ensemble unstable and overfitting to the public lb . Instead a much better approach would be use a meta-classifier on top your predictions and let it decide the weights for each model's prediction in the ensemble , also training this meta-classifier on the same folds as your models makes it robust.

", "rawMarkdown": "@hanson0910 I think by using the confusion matrix as a point to learn the weights might make the ensemble unstable and overfitting to the public lb . Instead a much better approach would be use a meta-classifier on top your predictions and let it decide the weights for each model's prediction in the ensemble , also training this meta-classifier on the same folds as your models makes it robust. ", "votes": 10, "replies": [ { "id": 1164653, "postDate": "2021-01-22T13:56:04.300Z", "content": "

thank you for your great suggestion，your analysis is reasonable，i will try it.

", "rawMarkdown": "thank you for your great suggestion，your analysis is reasonable，i will try it.", "votes": 1 }, { "id": 1165932, "postDate": "2021-01-23T10:23:22.697Z", "content": "

@tanulsingh077 good idea, do you mind if you can link me some sample on how to use the meta classifier.

", "rawMarkdown": "@tanulsingh077 good idea, do you mind if you can link me some sample on how to use the meta classifier. " } ] }, { "id": 1165163, "postDate": "2021-01-22T18:36:35Z", "content": "

Very interesting approach. Thank you for sharing, I'm not sure if there's any paper for this, but if it does work, I think it would be a valuable to publish.

\n

Just to clarify:
\nA fold's prediction would be multiplied by the weight from confusion matrix, then all folds' prediction would be summed up, and the final inference would be the max value?

", "rawMarkdown": "Very interesting approach. Thank you for sharing, I'm not sure if there's any paper for this, but if it does work, I think it would be a valuable to publish. \n\nJust to clarify:\nA fold's prediction would be multiplied by the weight from confusion matrix, then all folds' prediction would be summed up, and the final inference would be the max value? ", "votes": 1 }, { "id": 1204797, "postDate": "2021-02-16T10:52:38.753Z", "content": "

@hanson0910 if the fold_0 model gives better predictions than everybody else on let's say '3'. Does that mean that the prediction of fold_0 model will have more weight than other fold models ? Unlike what we usually do like in 5 folds case we just average the predictions which essentially means each model gets 0.2 weight for all the predictions

", "rawMarkdown": "@hanson0910 if the fold_0 model gives better predictions than everybody else on let's say '3'. Does that mean that the prediction of fold_0 model will have more weight than other fold models ? Unlike what we usually do like in 5 folds case we just average the predictions which essentially means each model gets 0.2 weight for all the predictions" }, { "id": 1168763, "postDate": "2021-01-25T06:45:49.513Z", "content": "

model weights,with q

", "rawMarkdown": " model weights,with q" }, { "id": 1165258, "postDate": "2021-01-22T20:06:31.917Z", "content": "

Just an idea here, but if you would measure Collinearity between models. Such as model 0 and model 1 have different accuracies, but a low correlation between them, as in they get different images right at overall the same rate, you could argueably say that even though one has a higher accuracy, you probably could keep both at the same weight, because they are accurately labeling different images in your set.
\nIf you have 0 correlation between models, you could arguably (i'm being generous here) have 3 models at 33% accuracy which have 99% accuracy when combined.
\nDoes anyone else agree or have something to add?
\nOne thing that could be difficult to assess is when model 0 is right, and when model 1 is right, but that idea could probably solve the classification problem for good.

", "rawMarkdown": "Just an idea here, but if you would measure Collinearity between models. Such as model 0 and model 1 have different accuracies, but a low correlation between them, as in they get different images right at overall the same rate, you could argueably say that even though one has a higher accuracy, you probably could keep both at the same weight, because they are accurately labeling different images in your set.\nIf you have 0 correlation between models, you could arguably (i'm being generous here) have 3 models at 33% accuracy which have 99% accuracy when combined.\nDoes anyone else agree or have something to add?\nOne thing that could be difficult to assess is when model 0 is right, and when model 1 is right, but that idea could probably solve the classification problem for good.", "replies": [ { "id": 1165266, "postDate": "2021-01-22T20:15:11.147Z", "content": "

One thing i probably didn't made clear, is that two models with a high correlation actually don't add to the overall accuracy of the model, because if they are mostly predicting the same things for the same images.
\nIf you could actually train two models in a way to minimize their correlation you could probably improve most ensemble model accuracies.

", "rawMarkdown": "One thing i probably didn't made clear, is that two models with a high correlation actually don't add to the overall accuracy of the model, because if they are mostly predicting the same things for the same images.\nIf you could actually train two models in a way to minimize their correlation you could probably improve most ensemble model accuracies." } ] }, { "id": 1164614, "postDate": "2021-01-22T13:34:44.210Z", "content": "

Don't mind if I ask.. What kind of weights are we talking about? Class weights or model weights? If it's model weights, how do we go about modifying them?

", "rawMarkdown": "Don't mind if I ask.. What kind of weights are we talking about? Class weights or model weights? If it's model weights, how do we go about modifying them?", "replies": [ { "id": 1164662, "postDate": "2021-01-22T14:03:45.270Z", "content": "

It is the weight of model for each category, which is realized by the sofamax function of confusion matrix.

", "rawMarkdown": "It is the weight of model for each category, which is realized by the sofamax function of confusion matrix." }, { "id": 1164835, "postDate": "2021-01-22T15:28:17.113Z", "content": "

Ah right, thanks!

", "rawMarkdown": "Ah right, thanks!" } ] }, { "id": 1164374, "postDate": "2021-01-22T10:35:25.597Z", "content": "

@hanson0910 Thanks for sharing! I think it is great way to improve score. It can be really useful! Thanks for sharing again. +upvoted:)

", "rawMarkdown": "@hanson0910 Thanks for sharing! I think it is great way to improve score. It can be really useful! Thanks for sharing again. +upvoted:)", "replies": [ { "id": 1164654, "postDate": "2021-01-22T13:57:04.077Z", "content": "

sometimes work sometimes not ,anyway goog luck!

", "rawMarkdown": "sometimes work sometimes not ,anyway goog luck!", "votes": 2 } ] }, { "id": 1164643, "postDate": "2021-01-22T13:51:14.173Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1164642, "postDate": "2021-01-22T13:50:48.163Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1164427, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2021-01-22T11:15:28.057000", "content": "

@hanson0910 I think by using the confusion matrix as a point to learn the weights might make the ensemble unstable and overfitting to the public lb . Instead a much better approach would be use a meta-classifier on top your predictions and let it decide the weights for each model's prediction in the ensemble , also training this meta-classifier on the same folds as your models makes it robust.

", "votes": 10, "replies": [ { "id": 1164653, "author_name": "Hanson0910", "author_url": "", "post_date": "2021-01-22T13:56:04.300000", "content": "

thank you for your great suggestion，your analysis is reasonable，i will try it.

", "votes": 1, "replies": [] }, { "id": 1165932, "author_name": "gao-hongnan", "author_url": "", "post_date": "2021-01-23T10:23:22.697000", "content": "

@tanulsingh077 good idea, do you mind if you can link me some sample on how to use the meta classifier.

", "votes": 0, "replies": [] } ] }, { "id": 1165163, "author_name": "Jason Dolorso", "author_url": "", "post_date": "2021-01-22T18:36:35", "content": "

Very interesting approach. Thank you for sharing, I'm not sure if there's any paper for this, but if it does work, I think it would be a valuable to publish.

\n

Just to clarify:
\nA fold's prediction would be multiplied by the weight from confusion matrix, then all folds' prediction would be summed up, and the final inference would be the max value?

", "votes": 1, "replies": [] }, { "id": 1204797, "author_name": "Varun Gadre", "author_url": "", "post_date": "2021-02-16T10:52:38.753000", "content": "

@hanson0910 if the fold_0 model gives better predictions than everybody else on let's say '3'. Does that mean that the prediction of fold_0 model will have more weight than other fold models ? Unlike what we usually do like in 5 folds case we just average the predictions which essentially means each model gets 0.2 weight for all the predictions

", "votes": 0, "replies": [] }, { "id": 1168763, "author_name": "zhangyao0623", "author_url": "", "post_date": "2021-01-25T06:45:49.513000", "content": "

model weights,with q

", "votes": 0, "replies": [] }, { "id": 1165258, "author_name": "Gabriel Prado", "author_url": "", "post_date": "2021-01-22T20:06:31.917000", "content": "

Just an idea here, but if you would measure Collinearity between models. Such as model 0 and model 1 have different accuracies, but a low correlation between them, as in they get different images right at overall the same rate, you could argueably say that even though one has a higher accuracy, you probably could keep both at the same weight, because they are accurately labeling different images in your set.
\nIf you have 0 correlation between models, you could arguably (i'm being generous here) have 3 models at 33% accuracy which have 99% accuracy when combined.
\nDoes anyone else agree or have something to add?
\nOne thing that could be difficult to assess is when model 0 is right, and when model 1 is right, but that idea could probably solve the classification problem for good.

", "votes": 0, "replies": [ { "id": 1165266, "author_name": "Gabriel Prado", "author_url": "", "post_date": "2021-01-22T20:15:11.147000", "content": "

One thing i probably didn't made clear, is that two models with a high correlation actually don't add to the overall accuracy of the model, because if they are mostly predicting the same things for the same images.
\nIf you could actually train two models in a way to minimize their correlation you could probably improve most ensemble model accuracies.

", "votes": 0, "replies": [] } ] }, { "id": 1164614, "author_name": "Junyi Ng", "author_url": "", "post_date": "2021-01-22T13:34:44.210000", "content": "

Don't mind if I ask.. What kind of weights are we talking about? Class weights or model weights? If it's model weights, how do we go about modifying them?

", "votes": 0, "replies": [ { "id": 1164662, "author_name": "Hanson0910", "author_url": "", "post_date": "2021-01-22T14:03:45.270000", "content": "

It is the weight of model for each category, which is realized by the sofamax function of confusion matrix.

", "votes": 0, "replies": [] }, { "id": 1164835, "author_name": "Junyi Ng", "author_url": "", "post_date": "2021-01-22T15:28:17.113000", "content": "

Ah right, thanks!

", "votes": 0, "replies": [] } ] }, { "id": 1164374, "author_name": "Dongkyu Kim", "author_url": "", "post_date": "2021-01-22T10:35:25.597000", "content": "

@hanson0910 Thanks for sharing! I think it is great way to improve score. It can be really useful! Thanks for sharing again. +upvoted:)

", "votes": 0, "replies": [ { "id": 1164654, "author_name": "Hanson0910", "author_url": "", "post_date": "2021-01-22T13:57:04.077000", "content": "

sometimes work sometimes not ,anyway goog luck!

", "votes": 2, "replies": [] } ] }, { "id": 1164643, "author_name": "", "author_url": "", "post_date": "2021-01-22T13:51:14.173000", "content": "", "votes": 0, "replies": [] }, { "id": 1164642, "author_name": "", "author_url": "", "post_date": "2021-01-22T13:50:48.163000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1164225": "**Using mult folds cross validation and mult model ensemble is a common method to improve score,but the performance of different models in different categories is different.If we give each model the same weight, I think it's unreasonable,so how to assign the right weight to each model is worth trying.By saving the confusion matrix of each model, a model with the highest accuracy in the specified category is selected,because we have five categories, we choose five models，finally, the sofamax function is obtained for each category of the five models，we can get the weight of each model in each category.In this way, my LB score increased from 0.905 to 0.906.**\none of the model confusion matrix as follows:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4848897%2F18faaf3e6d8fe439a94b908a497fd3ea%2Fcheckpoint-9.jpg?generation=1611305510368531&alt=media)", "1164427": "@hanson0910 I think by using the confusion matrix as a point to learn the weights might make the ensemble unstable and overfitting to the public lb . Instead a much better approach would be use a meta-classifier on top your predictions and let it decide the weights for each model's prediction in the ensemble , also training this meta-classifier on the same folds as your models makes it robust. ", "1165163": "Very interesting approach. Thank you for sharing, I'm not sure if there's any paper for this, but if it does work, I think it would be a valuable to publish. \n\nJust to clarify:\nA fold's prediction would be multiplied by the weight from confusion matrix, then all folds' prediction would be summed up, and the final inference would be the max value? ", "1204797": "@hanson0910 if the fold_0 model gives better predictions than everybody else on let's say '3'. Does that mean that the prediction of fold_0 model will have more weight than other fold models ? Unlike what we usually do like in 5 folds case we just average the predictions which essentially means each model gets 0.2 weight for all the predictions", "1168763": " model weights,with q", "1165258": "Just an idea here, but if you would measure Collinearity between models. Such as model 0 and model 1 have different accuracies, but a low correlation between them, as in they get different images right at overall the same rate, you could argueably say that even though one has a higher accuracy, you probably could keep both at the same weight, because they are accurately labeling different images in your set.\nIf you have 0 correlation between models, you could arguably (i'm being generous here) have 3 models at 33% accuracy which have 99% accuracy when combined.\nDoes anyone else agree or have something to add?\nOne thing that could be difficult to assess is when model 0 is right, and when model 1 is right, but that idea could probably solve the classification problem for good.", "1164614": "Don't mind if I ask.. What kind of weights are we talking about? Class weights or model weights? If it's model weights, how do we go about modifying them?", "1164374": "@hanson0910 Thanks for sharing! I think it is great way to improve score. It can be really useful! Thanks for sharing again. +upvoted:)", "1164643": "", "1164642": "" } }