As competition is coming onto the end week, many teams must have already started with ensembling models.
\nI would also like to talk about our current best single models, which include ( Efficientnet v2l aux loss) for classification models. While for Object Detection, it's an ensemble of Yolo v5 variants.
\nThings that did pretty well for us so far:
\nComing onto ensemble,
\nWe have tried almost all sorts of ensembling with diverse models which include:
\nFor us, these usually give a good boost on study oof score(~ +0.004) but never reflects back on the leaderboard (not even comparable to a single model lb score).
\nI would really appreciate it if someone shares how well is ensemble performing for them in Study and Object Detection models and what techniques they are using for the ensemble.
\nI hope it might be helpful for some of the teams who are lagging behind or got frustrated in achieving a decent score. Sorry if you think I shared a bit too much in the second last week, but I believe most of the teams are already aware of most of the techniques I mentioned above and have already been discussed earlier in some of the topics.
", "messages": [ { "id": 1407104, "postDate": "2021-08-01T14:22:55.247Z", "content": "As competition is coming onto the end week, many teams must have already started with ensembling models.
\nI would also like to talk about our current best single models, which include ( Efficientnet v2l aux loss) for classification models. While for Object Detection, it's an ensemble of Yolo v5 variants.
\nThings that did pretty well for us so far:
\nComing onto ensemble,
\nWe have tried almost all sorts of ensembling with diverse models which include:
\nFor us, these usually give a good boost on study oof score(~ +0.004) but never reflects back on the leaderboard (not even comparable to a single model lb score).
\nI would really appreciate it if someone shares how well is ensemble performing for them in Study and Object Detection models and what techniques they are using for the ensemble.
\nI hope it might be helpful for some of the teams who are lagging behind or got frustrated in achieving a decent score. Sorry if you think I shared a bit too much in the second last week, but I believe most of the teams are already aware of most of the techniques I mentioned above and have already been discussed earlier in some of the topics.
", "rawMarkdown": "As competition is coming onto the end week, many teams must have already started with ensembling models. \n\nI would also like to talk about our current best single models, which include ( Efficientnet v2l aux loss) for classification models. While for Object Detection, it's an ensemble of Yolo v5 variants. \n \nThings that did pretty well for us so far:\n\n- Pseudo Labels with public data did well on classification models. \n- Aux loss ( BCE + lovasz loss)\n- Large Batch size\n- Label Smoothing\n- Larger Image size did better on Object Detection but not with Classification Models.\n- Cutmix / Mixup\n\nComing onto ensemble, \n\nWe have tried almost all sorts of ensembling with diverse models which include:\n- Mean/weighted blending\n- Geometric mean\n- Power blend\n- Stacking with classifier\n\nFor us, these usually give a good boost on study oof score(~ +0.004) but never reflects back on the leaderboard (not even comparable to a single model lb score).\n\nI would really appreciate it if someone shares how well is ensemble performing for them in Study and Object Detection models and what techniques they are using for the ensemble. \n\nI hope it might be helpful for some of the teams who are lagging behind or got frustrated in achieving a decent score. Sorry if you think I shared a bit too much in the second last week, but I believe most of the teams are already aware of most of the techniques I mentioned above and have already been discussed earlier in some of the topics.", "votes": 56 }, { "id": 1407200, "postDate": "2021-08-01T16:08:11.617Z", "content": "That's a lot of info you shared
\nFor us, these usually give a good boost on study oof score(~ +0.004) but never reflects back on the leaderboard (not even comparable to a single model lb score).\nSame here
\nWe used the same ensembling techniques as you did for the Study. OOF score increases ( ~ 0.01 ) but LB remains the same.
\nThanks for sharing your mini solution in the middle of the competition
https://ibb.co/brGRb67 (GT stands for ground truth)
\nThe boxes different models predict are quite closed, so it won't give much boost.
", "rawMarkdown": "https://ibb.co/brGRb67 (GT stands for ground truth)\n\nThe boxes different models predict are quite closed, so it won't give much boost.", "votes": 1 }, { "id": 1443360, "postDate": "2021-08-04T02:42:46.150Z", "content": "In my case, effnetv2+Transformer based models ensembling boosted CV+LB
", "rawMarkdown": "In my case, effnetv2+Transformer based models ensembling boosted CV+LB", "votes": 2, "replies": [ { "id": 1446556, "postDate": "2021-08-04T12:33:51.683Z", "content": "EffnetV2 is really outperforming old effnet models in term of training times but the accuracy and oof is almost similar to the old ones for me.
", "rawMarkdown": "EffnetV2 is really outperforming old effnet models in term of training times but the accuracy and oof is almost similar to the old ones for me." } ] }, { "id": 1407281, "postDate": "2021-08-01T17:38:41.637Z", "content": ".005 Boost currently on LB on classification models (2) ensemble. Classification Pseudo did not work on CV for us yet, so did not try LB…
", "rawMarkdown": ".005 Boost currently on LB on classification models (2) ensemble. Classification Pseudo did not work on CV for us yet, so did not try LB...", "votes": 2 }, { "id": 1407126, "postDate": "2021-08-01T14:46:35.543Z", "content": "That's indeed a lot of information shared 8 days before the deadline.
\nAlthough most of it is known stuff, you're probably giving too many details about your solution for your team's good (model names, img sizes).
Anyways, here's some food for thought, hope it helps :
\nthanks for your concern @theoviel and for sharing ensemble techniques. I just shared only some parts of our solutions that I thought are already known to others. Also, I feel study models have already reached a saturation point, and shouldn't make a major change in someone's score.
", "rawMarkdown": "thanks for your concern @theoviel and for sharing ensemble techniques. I just shared only some parts of our solutions that I thought are already known to others. Also, I feel study models have already reached a saturation point, and shouldn't make a major change in someone's score. ", "votes": 1 }, { "id": 1407272, "postDate": "2021-08-01T17:22:07.887Z", "content": "@nischaydnk you have to keep in mind that you also open the discussion forum with such a post and some other people might share important details
", "rawMarkdown": "@nischaydnk you have to keep in mind that you also open the discussion forum with such a post and some other people might share important details", "votes": 6 }, { "id": 1407613, "postDate": "2021-08-02T03:49:33.593Z", "content": "HI @philippsinger, I understood what you're trying to say, and I hope nobody shares their strategy publicly before the deadline. My intentions were not to harm any team's hard work including ours too but to open up a forum for casual discussions about final days strategy & ensemble.
\nMaybe the way I wrote the topic in bullet points made everyone think I shared a solution, which I didn't. Still, I will keep in my mind before posting anything on discussions next time.
i agree with @philippsinger you share these many details with some key filters with intent of either helping many or also knowing more from others when they share by getting carried away by flow of discussion.
", "rawMarkdown": "i agree with @philippsinger you share these many details with some key filters with intent of either helping many or also knowing more from others when they share by getting carried away by flow of discussion.\n", "votes": 3 } ] }, { "id": 1407984, "postDate": "2021-08-02T08:40:31.997Z", "content": "Hi,
\nI have a question about Aux loss because I don't know anything it. What is the Aux loss ? And, How to apply it ? If you have some example code or notebook, please share me if you don't mind. Thanks.
Please refer to this discussion for more information about Aux loss: https://www.kaggle.com/c/siim-covid19-detection/discussion/240233
", "rawMarkdown": "Please refer to this discussion for more information about Aux loss: https://www.kaggle.com/c/siim-covid19-detection/discussion/240233" } ] }, { "id": 1451271, "postDate": "2021-08-05T09:55:48.217Z", "content": "Thanks for posting :) @nishaydnk i appreciate your sharing although at this point of time. No wonder quite a people crossed 64.2 in matter 2 days . Do you know if there official confirmation from Host about not using covidxr2 because of licensing issue.
", "rawMarkdown": "Thanks for posting :) @nishaydnk i appreciate your sharing although at this point of time. No wonder quite a people crossed 64.2 in matter 2 days . Do you know if there official confirmation from Host about not using covidxr2 because of licensing issue." }, { "id": 1466584, "postDate": "2021-08-11T14:21:32.377Z", "content": "\n\nPseudo Labels with public data did well on classification models.
\n
Could you explain how this works? Do you use your model to predict the labels on the public test data, and then use those labels to train your model further? It is a bit unintuitive to me why this should help
", "rawMarkdown": "> Pseudo Labels with public data did well on classification models.\n\nCould you explain how this works? Do you use your model to predict the labels on the public test data, and then use those labels to train your model further? It is a bit unintuitive to me why this should help" }, { "id": 1446585, "postDate": "2021-08-04T12:38:49.330Z", "content": "I did simple average for both study and image classification models (1st model trained with comp. data and 2nd model trained with external data [covidx_cxr2]) giving me a boost of +0.005 on oof's and +0.008.
", "rawMarkdown": "I did simple average for both study and image classification models (1st model trained with comp. data and 2nd model trained with external data [covidx_cxr2]) giving me a boost of +0.005 on oof's and +0.008. ", "replies": [ { "id": 1448564, "postDate": "2021-08-04T17:57:02.070Z", "content": "I saw that the dataset covidx_cxr2 comes under \"License: Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)\" , do you think it is allowed to use in the competition as this is not allowed for commercial use??
", "rawMarkdown": "I saw that the dataset covidx_cxr2 comes under \"License: Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)\" , do you think it is allowed to use in the competition as this is not allowed for commercial use?? " }, { "id": 1448628, "postDate": "2021-08-04T18:09:29.440Z", "content": "No, I don't think we can use this dataset for this competition. I decided to remove it from my notebooks as it didn't get a pass through the discussion forum here. RICORD also comes under [Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)] and it is allowed to use as mentioned here but we didn't get any clarification about using MIMIC-CXR because there were few pre-requisites to download that data hence not publicly available .
", "rawMarkdown": "No, I don't think we can use this dataset for this competition. I decided to remove it from my notebooks as it didn't get a pass through the discussion forum [here](https://www.kaggle.com/c/siim-covid19-detection/discussion/240080). RICORD also comes under [Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)] and it is allowed to use as mentioned [here](https://www.kaggle.com/c/siim-covid19-detection/discussion/243804#1341447) but we didn't get any clarification about using MIMIC-CXR because there were few pre-requisites to download that data hence not publicly available .", "votes": 1 } ] }, { "id": 1427289, "postDate": "2021-08-03T08:18:30.193Z", "content": "I think the problem are the atypical and indeterminate cases, this is why improvements in local cross validation not necessarily reflect in the lb
", "rawMarkdown": "I think the problem are the atypical and indeterminate cases, this is why improvements in local cross validation not necessarily reflect in the lb", "replies": [ { "id": 1433379, "postDate": "2021-08-03T13:57:25.353Z", "content": "What if we regard indeterminate and atypical as two open set problems?
", "rawMarkdown": "What if we regard indeterminate and atypical as two open set problems?" }, { "id": 1449822, "postDate": "2021-08-05T01:04:12.867Z", "content": "@biglafe have you succeeded in treating those 2 as open set problems? It is an interesting approach.
", "rawMarkdown": "@biglafe have you succeeded in treating those 2 as open set problems? It is an interesting approach." } ] }, { "id": 1407117, "postDate": "2021-08-01T14:38:03.587Z", "content": "I couldn't make the ensemble work well either. It worked well in some of my models using small batch sizes, but they are not my best models. Using batch sizes like 32 produces models that don't aggregate in the ensemble. How large is your batch?
", "rawMarkdown": "I couldn't make the ensemble work well either. It worked well in some of my models using small batch sizes, but they are not my best models. Using batch sizes like 32 produces models that don't aggregate in the ensemble. How large is your batch?", "replies": [ { "id": 1407125, "postDate": "2021-08-01T14:46:07.637Z", "content": "Hmm we haven't any boost in lower batch size models either. Although, oof score is getting a boost which worries me a lot due to the bad correlation with the leaderboard. Our current models batch sizes are ranging between 32 - 48.
", "rawMarkdown": "Hmm we haven't any boost in lower batch size models either. Although, oof score is getting a boost which worries me a lot due to the bad correlation with the leaderboard. Our current models batch sizes are ranging between 32 - 48." }, { "id": 1407174, "postDate": "2021-08-01T15:30:54.640Z", "content": "@nischaydnk are you using accumulate gradient or TPU/multi GPUS to handle such batch size? Isn't your models having similar training and being very correlated?
", "rawMarkdown": "@nischaydnk are you using accumulate gradient or TPU/multi GPUS to handle such batch size? Isn't your models having similar training and being very correlated? " }, { "id": 1407177, "postDate": "2021-08-01T15:36:23.903Z", "content": "Based on past winners solutions it seems is good to train some models with cutmix and others without it for example.
", "rawMarkdown": "Based on past winners solutions it seems is good to train some models with cutmix and others without it for example." }, { "id": 1407182, "postDate": "2021-08-01T15:47:52.450Z", "content": "Training on multi gpus, We used different pipeline models for the ensemble.
", "rawMarkdown": "Training on multi gpus, We used different pipeline models for the ensemble.", "votes": 1 } ] }, { "id": 1453535, "postDate": "2021-08-05T22:45:32.533Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1443055, "postDate": "2021-08-04T01:34:08.687Z", "content": "Thanks for the great thread!
\nHas anyone had a good handle on NMS, Soft-NMS or Non-Maximum-Weighted?
\nI haven't handled them well (in the first place, NMS caused OOM), so if there is a technician, I would like to know these techniques even after the competition is over :)
That's a lot of info you shared
\nFor us, these usually give a good boost on study oof score(~ +0.004) but never reflects back on the leaderboard (not even comparable to a single model lb score).\nSame here
\nWe used the same ensembling techniques as you did for the Study. OOF score increases ( ~ 0.01 ) but LB remains the same.
\nThanks for sharing your mini solution in the middle of the competition
https://ibb.co/brGRb67 (GT stands for ground truth)
\nThe boxes different models predict are quite closed, so it won't give much boost.
", "votes": 1, "replies": [] }, { "id": 1443360, "author_name": "arutema47", "author_url": "", "post_date": "2021-08-04T02:42:46.150000", "content": "In my case, effnetv2+Transformer based models ensembling boosted CV+LB
", "votes": 2, "replies": [ { "id": 1446556, "author_name": "ilovepotatoes", "author_url": "", "post_date": "2021-08-04T12:33:51.683000", "content": "EffnetV2 is really outperforming old effnet models in term of training times but the accuracy and oof is almost similar to the old ones for me.
", "votes": 0, "replies": [] } ] }, { "id": 1407281, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2021-08-01T17:38:41.637000", "content": ".005 Boost currently on LB on classification models (2) ensemble. Classification Pseudo did not work on CV for us yet, so did not try LB…
", "votes": 2, "replies": [] }, { "id": 1407126, "author_name": "Theo Viel", "author_url": "", "post_date": "2021-08-01T14:46:35.543000", "content": "That's indeed a lot of information shared 8 days before the deadline.
\nAlthough most of it is known stuff, you're probably giving too many details about your solution for your team's good (model names, img sizes).
Anyways, here's some food for thought, hope it helps :
\nthanks for your concern @theoviel and for sharing ensemble techniques. I just shared only some parts of our solutions that I thought are already known to others. Also, I feel study models have already reached a saturation point, and shouldn't make a major change in someone's score.
", "votes": 1, "replies": [] }, { "id": 1407272, "author_name": "Psi", "author_url": "", "post_date": "2021-08-01T17:22:07.887000", "content": "@nischaydnk you have to keep in mind that you also open the discussion forum with such a post and some other people might share important details
", "votes": 6, "replies": [] }, { "id": 1407613, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2021-08-02T03:49:33.593000", "content": "HI @philippsinger, I understood what you're trying to say, and I hope nobody shares their strategy publicly before the deadline. My intentions were not to harm any team's hard work including ours too but to open up a forum for casual discussions about final days strategy & ensemble.
\nMaybe the way I wrote the topic in bullet points made everyone think I shared a solution, which I didn't. Still, I will keep in my mind before posting anything on discussions next time.
i agree with @philippsinger you share these many details with some key filters with intent of either helping many or also knowing more from others when they share by getting carried away by flow of discussion.
", "votes": 3, "replies": [] } ] }, { "id": 1407984, "author_name": "SeongwookLee", "author_url": "", "post_date": "2021-08-02T08:40:31.997000", "content": "Hi,
\nI have a question about Aux loss because I don't know anything it. What is the Aux loss ? And, How to apply it ? If you have some example code or notebook, please share me if you don't mind. Thanks.
Please refer to this discussion for more information about Aux loss: https://www.kaggle.com/c/siim-covid19-detection/discussion/240233
", "votes": 0, "replies": [] } ] }, { "id": 1451271, "author_name": "Jaideep", "author_url": "", "post_date": "2021-08-05T09:55:48.217000", "content": "Thanks for posting :) @nishaydnk i appreciate your sharing although at this point of time. No wonder quite a people crossed 64.2 in matter 2 days . Do you know if there official confirmation from Host about not using covidxr2 because of licensing issue.
", "votes": 0, "replies": [] }, { "id": 1466584, "author_name": "Victor", "author_url": "", "post_date": "2021-08-11T14:21:32.377000", "content": "\n\nPseudo Labels with public data did well on classification models.
\n
Could you explain how this works? Do you use your model to predict the labels on the public test data, and then use those labels to train your model further? It is a bit unintuitive to me why this should help
", "votes": 0, "replies": [] }, { "id": 1446585, "author_name": "ilovepotatoes", "author_url": "", "post_date": "2021-08-04T12:38:49.330000", "content": "I did simple average for both study and image classification models (1st model trained with comp. data and 2nd model trained with external data [covidx_cxr2]) giving me a boost of +0.005 on oof's and +0.008.
", "votes": 0, "replies": [ { "id": 1448564, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2021-08-04T17:57:02.070000", "content": "I saw that the dataset covidx_cxr2 comes under \"License: Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)\" , do you think it is allowed to use in the competition as this is not allowed for commercial use??
", "votes": 0, "replies": [] }, { "id": 1448628, "author_name": "ilovepotatoes", "author_url": "", "post_date": "2021-08-04T18:09:29.440000", "content": "No, I don't think we can use this dataset for this competition. I decided to remove it from my notebooks as it didn't get a pass through the discussion forum here. RICORD also comes under [Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)] and it is allowed to use as mentioned here but we didn't get any clarification about using MIMIC-CXR because there were few pre-requisites to download that data hence not publicly available .
", "votes": 1, "replies": [] } ] }, { "id": 1427289, "author_name": "hengck23", "author_url": "", "post_date": "2021-08-03T08:18:30.193000", "content": "I think the problem are the atypical and indeterminate cases, this is why improvements in local cross validation not necessarily reflect in the lb
", "votes": 0, "replies": [ { "id": 1433379, "author_name": "lafe", "author_url": "", "post_date": "2021-08-03T13:57:25.353000", "content": "What if we regard indeterminate and atypical as two open set problems?
", "votes": 0, "replies": [] }, { "id": 1449822, "author_name": "Liam Nguyen", "author_url": "", "post_date": "2021-08-05T01:04:12.867000", "content": "@biglafe have you succeeded in treating those 2 as open set problems? It is an interesting approach.
", "votes": 0, "replies": [] } ] }, { "id": 1407117, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-08-01T14:38:03.587000", "content": "I couldn't make the ensemble work well either. It worked well in some of my models using small batch sizes, but they are not my best models. Using batch sizes like 32 produces models that don't aggregate in the ensemble. How large is your batch?
", "votes": 0, "replies": [ { "id": 1407125, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2021-08-01T14:46:07.637000", "content": "Hmm we haven't any boost in lower batch size models either. Although, oof score is getting a boost which worries me a lot due to the bad correlation with the leaderboard. Our current models batch sizes are ranging between 32 - 48.
", "votes": 0, "replies": [] }, { "id": 1407174, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-08-01T15:30:54.640000", "content": "@nischaydnk are you using accumulate gradient or TPU/multi GPUS to handle such batch size? Isn't your models having similar training and being very correlated?
", "votes": 0, "replies": [] }, { "id": 1407177, "author_name": "IgorMuniz", "author_url": "", "post_date": "2021-08-01T15:36:23.903000", "content": "Based on past winners solutions it seems is good to train some models with cutmix and others without it for example.
", "votes": 0, "replies": [] }, { "id": 1407182, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2021-08-01T15:47:52.450000", "content": "Training on multi gpus, We used different pipeline models for the ensemble.
", "votes": 1, "replies": [] } ] }, { "id": 1453535, "author_name": "", "author_url": "", "post_date": "2021-08-05T22:45:32.533000", "content": "", "votes": 0, "replies": [] }, { "id": 1443055, "author_name": "", "author_url": "", "post_date": "2021-08-04T01:34:08.687000", "content": "Thanks for the great thread!
\nHas anyone had a good handle on NMS, Soft-NMS or Non-Maximum-Weighted?
\nI haven't handled them well (in the first place, NMS caused OOM), so if there is a technician, I would like to know these techniques even after the competition is over :)