I created k fold models using this code and on all my runs with different params and post processing, some of the folds (single-fold models) seem to be consistently giving worse results than other folds, and when we ensemble all folds, the results we get are toward the lower end of the spectrum.
\nIs there any particular explanation for why this might be happening? Also I don't exactly understand why the above code runs much slower and produces much larges files than the following kfold generators:
\nnotebook 1
\nnotebook 2
I created k fold models using this code and on all my runs with different params and post processing, some of the folds (single-fold models) seem to be consistently giving worse results than other folds, and when we ensemble all folds, the results we get are toward the lower end of the spectrum.
\nIs there any particular explanation for why this might be happening? Also I don't exactly understand why the above code runs much slower and produces much larges files than the following kfold generators:
\nnotebook 1
\nnotebook 2
Same here. Some folds are consistently better no matter what backbone is used. However, I believe I already split the folds with balanced cell types and the number of annotations. This may be because of inconsistent data quality, but I haven't found a way to be sure.
", "rawMarkdown": "Same here. Some folds are consistently better no matter what backbone is used. However, I believe I already split the folds with balanced cell types and the number of annotations. This may be because of inconsistent data quality, but I haven't found a way to be sure.", "votes": null }, { "id": "1616014", "postDate": "12/13/2021 04:16:43", "content": "you mean better on LB or CV?
", "rawMarkdown": "you mean better on LB or CV?", "votes": null }, { "id": "1616298", "postDate": "12/13/2021 10:13:39", "content": "Same here. When I ensemble multiple models, I always get score of the worst model.
", "rawMarkdown": "Same here. When I ensemble multiple models, I always get score of the worst model.", "votes": null }, { "id": "1616382", "postDate": "12/13/2021 11:40:56", "content": "actually for me cv was pretty much same for all the the models ~0.301 but different folds gave LB ranging from 0.27 to 0.31 so yeah pretty varied :/
", "rawMarkdown": "actually for me cv was pretty much same for all the the models ~0.301 but different folds gave LB ranging from 0.27 to 0.31 so yeah pretty varied :/", "votes": null }, { "id": "1617323", "postDate": "12/14/2021 03:09:30", "content": "Same here. Some folds are very hard to get a good lb score. I think there could be some tricky skills on the K-fold split.
\nAs for the running time about the k fold generator code, if you look deep into it, you can find the slower one is using the function \"enc =binary_mask_to_rle(mk)\". The other two are using pycocotools build-in method function. The latter one is much faster.
oh okay thanks for the help :) yes I think post processing will be very important. Or maybe training one class detectors and then using some ensemble technique would be more wise? Let's see 😄
", "rawMarkdown": "oh okay thanks for the help :) yes I think post processing will be very important. Or maybe training one class detectors and then using some ensemble technique would be more wise? Let's see 😄", "votes": null }, { "id": "1618706", "postDate": "12/15/2021 08:42:33", "content": "I found an efficient way to ensemble multiple models now. I can get better score than the best model's score in the ensemble.
", "rawMarkdown": "I found an efficient way to ensemble multiple models now. I can get better score than the best model's score in the ensemble.", "votes": null }, { "id": "1618784", "postDate": "12/15/2021 10:48:39", "content": "Can you share your ensemble strategy @gunesevitan?
", "rawMarkdown": "Can you share your ensemble strategy @gunesevitan?", "votes": null }, { "id": "1618792", "postDate": "12/15/2021 10:56:39", "content": "Sorry I can't share it at this stage of the competition. I'll make it open source when competition ends. I think it can be very useful in object detection and instance segmentation projects.
", "rawMarkdown": "Sorry I can't share it at this stage of the competition. I'll make it open source when competition ends. I think it can be very useful in object detection and instance segmentation projects.", "votes": null }, { "id": "1632948", "postDate": "12/30/2021 08:54:27", "content": "Curious!
\nPlease share after the competition is over.
well, this comment inspired me to try harder with ensembles, so I found reasonable straightforward method that gave us +0.008 so far.
", "rawMarkdown": "well, this comment inspired me to try harder with ensembles, so I found reasonable straightforward method that gave us +0.008 so far.", "votes": null }, { "id": "1633075", "postDate": "12/30/2021 11:44:40", "content": "The effective model ensemble skills in this competition will improve more scores. We look forward to your sharing!
", "rawMarkdown": "The effective model ensemble skills in this competition will improve more scores. We look forward to your sharing!", "votes": null } ], "comments": [ { "id": 1615831, "author_name": "peytonguo", "author_url": "", "post_date": "12/12/2021 18:37:48", "content": "Same here. Some folds are consistently better no matter what backbone is used. However, I believe I already split the folds with balanced cell types and the number of annotations. This may be because of inconsistent data quality, but I haven't found a way to be sure.
", "votes": null, "replies": [] }, { "id": 1616014, "author_name": "ptran1203", "author_url": "", "post_date": "12/13/2021 04:16:43", "content": "you mean better on LB or CV?
", "votes": null, "replies": [ { "id": 1616382, "author_name": "ferlockx", "author_url": "", "post_date": "12/13/2021 11:40:56", "content": "actually for me cv was pretty much same for all the the models ~0.301 but different folds gave LB ranging from 0.27 to 0.31 so yeah pretty varied :/
", "votes": null, "replies": [] } ] }, { "id": 1616298, "author_name": "gunesevitan", "author_url": "", "post_date": "12/13/2021 10:13:39", "content": "Same here. When I ensemble multiple models, I always get score of the worst model.
", "votes": null, "replies": [ { "id": 1618706, "author_name": "gunesevitan", "author_url": "", "post_date": "12/15/2021 08:42:33", "content": "I found an efficient way to ensemble multiple models now. I can get better score than the best model's score in the ensemble.
", "votes": null, "replies": [] }, { "id": 1618784, "author_name": "duykhanh99", "author_url": "", "post_date": "12/15/2021 10:48:39", "content": "Can you share your ensemble strategy @gunesevitan?
", "votes": null, "replies": [] }, { "id": 1618792, "author_name": "gunesevitan", "author_url": "", "post_date": "12/15/2021 10:56:39", "content": "Sorry I can't share it at this stage of the competition. I'll make it open source when competition ends. I think it can be very useful in object detection and instance segmentation projects.
", "votes": null, "replies": [] }, { "id": 1632948, "author_name": "wooseokshin", "author_url": "", "post_date": "12/30/2021 08:54:27", "content": "Curious!
\nPlease share after the competition is over.
well, this comment inspired me to try harder with ensembles, so I found reasonable straightforward method that gave us +0.008 so far.
", "votes": null, "replies": [] } ] }, { "id": 1617323, "author_name": "shinewine", "author_url": "", "post_date": "12/14/2021 03:09:30", "content": "Same here. Some folds are very hard to get a good lb score. I think there could be some tricky skills on the K-fold split.
\nAs for the running time about the k fold generator code, if you look deep into it, you can find the slower one is using the function \"enc =binary_mask_to_rle(mk)\". The other two are using pycocotools build-in method function. The latter one is much faster.
oh okay thanks for the help :) yes I think post processing will be very important. Or maybe training one class detectors and then using some ensemble technique would be more wise? Let's see 😄
", "votes": null, "replies": [] } ] }, { "id": 1633075, "author_name": "jingyq", "author_url": "", "post_date": "12/30/2021 11:44:40", "content": "The effective model ensemble skills in this competition will improve more scores. We look forward to your sharing!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1615560": "I created k fold models using [this](https://www.kaggle.com/ammarnassanalhajali/k-fold-crossvalidation-coco-dataset-generator/comments) code and on all my runs with different params and post processing, some of the folds (single-fold models) seem to be consistently giving worse results than other folds, and when we ensemble all folds, the results we get are toward the lower end of the spectrum. \n\nIs there any particular explanation for why this might be happening? Also I don't exactly understand why the above code runs much slower and produces much larges files than the following kfold generators: \n[notebook 1](https://www.kaggle.com/kisakitetta/sartorius-kfold-coco)\n[notebook 2](https://www.kaggle.com/mistag/sartorius-create-coco-annotations/notebook)", "1615831": "Same here. Some folds are consistently better no matter what backbone is used. However, I believe I already split the folds with balanced cell types and the number of annotations. This may be because of inconsistent data quality, but I haven't found a way to be sure.", "1616014": "you mean better on LB or CV?", "1616298": "Same here. When I ensemble multiple models, I always get score of the worst model.", "1616382": "actually for me cv was pretty much same for all the the models ~0.301 but different folds gave LB ranging from 0.27 to 0.31 so yeah pretty varied :/", "1617323": "Same here. Some folds are very hard to get a good lb score. I think there could be some tricky skills on the K-fold split.\nAs for the running time about the k fold generator code, if you look deep into it, you can find the slower one is using the function \"enc =binary_mask_to_rle(mk)\". The other two are using pycocotools build-in method function. The latter one is much faster.", "1617724": "oh okay thanks for the help :) yes I think post processing will be very important. Or maybe training one class detectors and then using some ensemble technique would be more wise? Let's see 😄", "1618706": "I found an efficient way to ensemble multiple models now. I can get better score than the best model's score in the ensemble.", "1618784": "Can you share your ensemble strategy @gunesevitan?", "1618792": "Sorry I can't share it at this stage of the competition. I'll make it open source when competition ends. I think it can be very useful in object detection and instance segmentation projects.", "1632948": "Curious!\nPlease share after the competition is over.", "1632961": "well, this comment inspired me to try harder with ensembles, so I found reasonable straightforward method that gave us +0.008 so far.", "1633075": "The effective model ensemble skills in this competition will improve more scores. We look forward to your sharing!" }, "source": "meta" }