I use the metric in the public detectron notebook.
\nBest single model CV: 0.2653 LB:0.296
\nWhat about yours?
I use the metric in the public detectron notebook.
\nBest single model CV: 0.2653 LB:0.296
\nWhat about yours?
CV 0.320 - LB 0.337 👀
\nI use a 5-fold split and the correlation is alright so far, but I submitted only 3 models so I can't really tell for sure.
", "rawMarkdown": "CV 0.320 - LB 0.337 👀\n\nI use a 5-fold split and the correlation is alright so far, but I submitted only 3 models so I can't really tell for sure.", "votes": 14, "replies": [ { "id": 1591347, "postDate": "2021-11-22T08:17:39.933Z", "content": "Amazing CV and LB. Seems the gap between my CV and LB is bit large👀
", "rawMarkdown": "Amazing CV and LB. Seems the gap between my CV and LB is bit large👀", "votes": 1 }, { "id": 1591380, "postDate": "2021-11-22T09:10:54.537Z", "content": "Mine was 0.334LB for a single model, I still cannot succeed in making use of ensembling. I have tested NMS, WBF, NMS at each stage as suggested by Kha Vu (https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/287774), but failed to make it work. Can I know: is 0.337 single fold score ? @theoviel
", "rawMarkdown": "Mine was 0.334LB for a single model, I still cannot succeed in making use of ensembling. I have tested NMS, WBF, NMS at each stage as suggested by Kha Vu (https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/287774), but failed to make it work. Can I know: is 0.337 single fold score ? @theoviel ", "votes": 6 }, { "id": 1591390, "postDate": "2021-11-22T09:33:10.607Z", "content": "I use all 5 folds, I moved to 5-folds as soon as I could, as I expected more robustness in my submissions. I didn't check single fold LB scores but I'm guessing they're close to my ensemble score.
", "rawMarkdown": "I use all 5 folds, I moved to 5-folds as soon as I could, as I expected more robustness in my submissions. I didn't check single fold LB scores but I'm guessing they're close to my ensemble score.", "votes": 5 }, { "id": 1591788, "postDate": "2021-11-22T15:53:38.833Z", "content": "are heavy encoders useful ? @theoviel
", "rawMarkdown": "are heavy encoders useful ? @theoviel ", "votes": -3 }, { "id": 1592079, "postDate": "2021-11-22T21:38:44.773Z", "content": "@theoviel if you submit a single fold someday, I'm interested to know what is the LB difference between your single fold and ensemble. Thx!
", "rawMarkdown": "@theoviel if you submit a single fold someday, I'm interested to know what is the LB difference between your single fold and ensemble. Thx!" }, { "id": 1597900, "postDate": "2021-11-28T03:20:37.047Z", "content": "Single fold getting more over folds ,but that is unsafe for private based till competition experience ,I think many scores are based on single folds only
", "rawMarkdown": "Single fold getting more over folds ,but that is unsafe for private based till competition experience ,I think many scores are based on single folds only " }, { "id": 1597901, "postDate": "2021-11-28T03:23:20.370Z", "content": "@namgalielei what is the issuevu faced with them wbf ,nms
", "rawMarkdown": "@namgalielei what is the issuevu faced with them wbf ,nms" }, { "id": 1597923, "postDate": "2021-11-28T04:09:22.617Z", "content": "As far as I inspected, when ensembling, it usually produced more box candidate and hence, led to more false positives.
", "rawMarkdown": "As far as I inspected, when ensembling, it usually produced more box candidate and hence, led to more false positives.", "votes": 1 }, { "id": 1603144, "postDate": "2021-12-02T11:21:26.117Z", "content": "which version of wbf u used @namgalielei in regular one we cant map fused box back to their masks
", "rawMarkdown": "which version of wbf u used @namgalielei in regular one we cant map fused box back to their masks" } ] }, { "id": 1597036, "postDate": "2021-11-27T06:19:36.610Z", "content": "In my experiments, CV/LB correlation is below:
\nCV 0.2746 - LB 0.315
\nCV 0.2827 - LB 0.320
\nCV 0.2880 - LB 0.324
\nCV 0.2983 - LB 0.327
\nCV 0.2995 - LB 0.328
\nCV 0.3002 - LB 0.330
\nCV 0.3013 - LB 0.331
These are single model scores.
", "rawMarkdown": "In my experiments, CV/LB correlation is below:\n\nCV 0.2746 - LB 0.315\nCV 0.2827 - LB 0.320\nCV 0.2880 - LB 0.324\nCV 0.2983 - LB 0.327\nCV 0.2995 - LB 0.328\nCV 0.3002 - LB 0.330\nCV 0.3013 - LB 0.331\n\nThese are single model scores.", "votes": 5 }, { "id": 1605644, "postDate": "2021-12-04T12:34:05.770Z", "content": "=============update==============
\nsingle model:
\nCV: 309~311 LB: 320~325
\nCV: 319~321 LB: 332
\nThe gap between cv and lb is unstable.Sometimes model with a 320 CV only gets 323 LB.
Hi @forcewithme What mAP code did you use?
", "rawMarkdown": "Hi @forcewithme What mAP code did you use?" } ] }, { "id": 1591089, "postDate": "2021-11-22T00:13:17.457Z", "content": "CV:295 LB:298
", "rawMarkdown": "CV:295 LB:298", "votes": 3, "replies": [ { "id": 1591111, "postDate": "2021-11-22T01:49:41.857Z", "content": "Very impressive cv！
", "rawMarkdown": "Very impressive cv！" } ] }, { "id": 1591372, "postDate": "2021-11-22T09:02:42.710Z", "content": "Single fold model: Valid 0.326 - LB 0.334. The correlation is relatively good. But some gain in valid did not translate into LB gain
", "rawMarkdown": "Single fold model: Valid 0.326 - LB 0.334. The correlation is relatively good. But some gain in valid did not translate into LB gain", "votes": 4, "replies": [ { "id": 1591398, "postDate": "2021-11-22T09:38:36.367Z", "content": "\n\nBut some gain in valid did not translate into LB gain
\n
Exactly the same, able to get 1% by fine tuning which is not translating into LB.
", "rawMarkdown": "> But some gain in valid did not translate into LB gain\n\nExactly the same, able to get 1% by fine tuning which is not translating into LB.", "votes": 1 }, { "id": 1594849, "postDate": "2021-11-25T07:01:24.257Z", "content": "@namgalielei @rednikotin . Same here, CV improve 0.02 but LB doesn't even improve 0.001
", "rawMarkdown": "@namgalielei @rednikotin . Same here, CV improve 0.02 but LB doesn't even improve 0.001" } ] }, { "id": 1592868, "postDate": "2021-11-23T12:33:17.733Z", "content": "hold-out (0.8 train - 0.2 val)
\nCV 0.301 - LB 0.314
metric
\nhttps://www.kaggle.com/theoviel/competition-metric-map-iou/notebook
single fold model
\nCV:0.2706, LB:0.294 with simple threshold, LB:0.3 with tunes threshold
Updated:
\nCV: 0.282 LB:0.311 with tuned threshold
would you mind to reveal why you can boost your CV from 0.27 to 0.282 (maybe increase number of epochs?) Thanks
", "rawMarkdown": "would you mind to reveal why you can boost your CV from 0.27 to 0.282 (maybe increase number of epochs?) Thanks", "votes": 1 }, { "id": 1621935, "postDate": "2021-12-18T07:49:28.167Z", "content": "Yes, increasing the number of epochs. Also added some augmentations.
", "rawMarkdown": "Yes, increasing the number of epochs. Also added some augmentations.", "votes": 1 }, { "id": 1621937, "postDate": "2021-12-18T07:53:50.453Z", "content": "Thanks for your reply!
", "rawMarkdown": "Thanks for your reply!", "votes": 1 } ] }, { "id": 1592185, "postDate": "2021-11-23T00:30:53.470Z", "content": "single fold (train:valid=0.8:0.2)
\nCV 0.271 / LB 0.294
\nCV 0.272 / LB 0.297
\nCV 0.279 / LB 0.300
My best single fold model:
\nCV: 0.265 - LB 0.298
CV 0.310 - LB 0.322
\nsame model, but more epoch
\nCV 0.312 - LB 0.311
\nCV 0.314 - LB 0.313
update:
\nThere seems to be a significant positive correlation.
\nCV:2719 LB:302
\nCV:2760 LB:304
\nCV:2777 LB:307
Hey, thanks for sharing these numbers. Are you using just hold-out set as validation set or are you doing k-fold CV as discussed here
", "rawMarkdown": "Hey, thanks for sharing these numbers. Are you using just hold-out set as validation set or are you doing k-fold CV as discussed [here](https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/285546)" }, { "id": 1590381, "postDate": "2021-11-21T07:29:35.560Z", "content": "hold out. Haven't used k-fold yet
", "rawMarkdown": "hold out. Haven't used k-fold yet", "votes": 1 }, { "id": 1590726, "postDate": "2021-11-21T15:37:28.853Z", "content": "thanks for sharing your results! indeed seems to correlate well with LB
\nmay I ask your split, i.e. 80/20, 90/10 ?
80/20.80/20.80/20.
", "rawMarkdown": "80/20.80/20.80/20.", "votes": 1 }, { "id": 1590910, "postDate": "2021-11-21T18:40:31.567Z", "content": "hi @atharvaingle can you please explain, what is hold-out. Im hearing this term for the first time. is it like, calculating mAP with different IoU for a single model on the validation data.
\nIm sorry if the question is very silly.
", "rawMarkdown": "hi @atharvaingle can you please explain, what is hold-out. Im hearing this term for the first time. is it like, calculating mAP with different IoU for a single model on the validation data.\n\nIm sorry if the question is very silly." }, { "id": 1590915, "postDate": "2021-11-21T18:51:00.493Z", "content": "Hold-out is just a small percentage of data kept for validating your trained model. Basically, it's nothing but 80-20 split as @forcewithme mentioned, where 80% of data is used for training and rest 20% is used for validation.
", "rawMarkdown": "Hold-out is just a small percentage of data kept for validating your trained model. Basically, it's nothing but 80-20 split as @forcewithme mentioned, where 80% of data is used for training and rest 20% is used for validation.", "votes": 1 }, { "id": 1590939, "postDate": "2021-11-21T19:13:09.317Z", "content": "Oh I see, thanks for answering ^_^
Mine went down. CV: 0.289 LB:0.283
", "rawMarkdown": "Mine went down. CV: 0.289 LB:0.283" }, { "id": 1592492, "postDate": "2021-11-23T05:33:34.437Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 1591341, "author_name": "Theo Viel", "author_url": "", "post_date": "2021-11-22T08:08:00.433000", "content": "CV 0.320 - LB 0.337 👀
\nI use a 5-fold split and the correlation is alright so far, but I submitted only 3 models so I can't really tell for sure.
", "votes": 14, "replies": [ { "id": 1591347, "author_name": "ForcewithMe", "author_url": "", "post_date": "2021-11-22T08:17:39.933000", "content": "Amazing CV and LB. Seems the gap between my CV and LB is bit large👀
", "votes": 1, "replies": [] }, { "id": 1591380, "author_name": "Liam Nguyen", "author_url": "", "post_date": "2021-11-22T09:10:54.537000", "content": "Mine was 0.334LB for a single model, I still cannot succeed in making use of ensembling. I have tested NMS, WBF, NMS at each stage as suggested by Kha Vu (https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/287774), but failed to make it work. Can I know: is 0.337 single fold score ? @theoviel
", "votes": 6, "replies": [] }, { "id": 1591390, "author_name": "Theo Viel", "author_url": "", "post_date": "2021-11-22T09:33:10.607000", "content": "I use all 5 folds, I moved to 5-folds as soon as I could, as I expected more robustness in my submissions. I didn't check single fold LB scores but I'm guessing they're close to my ensemble score.
", "votes": 5, "replies": [] }, { "id": 1591788, "author_name": "Jaideep", "author_url": "", "post_date": "2021-11-22T15:53:38.833000", "content": "are heavy encoders useful ? @theoviel
", "votes": -3, "replies": [] }, { "id": 1592079, "author_name": "Alexandre Cadrin-Chênevert", "author_url": "", "post_date": "2021-11-22T21:38:44.773000", "content": "@theoviel if you submit a single fold someday, I'm interested to know what is the LB difference between your single fold and ensemble. Thx!
", "votes": 0, "replies": [] }, { "id": 1597900, "author_name": "Jaideep", "author_url": "", "post_date": "2021-11-28T03:20:37.047000", "content": "Single fold getting more over folds ,but that is unsafe for private based till competition experience ,I think many scores are based on single folds only
", "votes": 0, "replies": [] }, { "id": 1597901, "author_name": "Jaideep", "author_url": "", "post_date": "2021-11-28T03:23:20.370000", "content": "@namgalielei what is the issuevu faced with them wbf ,nms
", "votes": 0, "replies": [] }, { "id": 1597923, "author_name": "Liam Nguyen", "author_url": "", "post_date": "2021-11-28T04:09:22.617000", "content": "As far as I inspected, when ensembling, it usually produced more box candidate and hence, led to more false positives.
", "votes": 1, "replies": [] }, { "id": 1603144, "author_name": "Jaideep", "author_url": "", "post_date": "2021-12-02T11:21:26.117000", "content": "which version of wbf u used @namgalielei in regular one we cant map fused box back to their masks
", "votes": 0, "replies": [] } ] }, { "id": 1597036, "author_name": "Yamame🐟", "author_url": "", "post_date": "2021-11-27T06:19:36.610000", "content": "In my experiments, CV/LB correlation is below:
\nCV 0.2746 - LB 0.315
\nCV 0.2827 - LB 0.320
\nCV 0.2880 - LB 0.324
\nCV 0.2983 - LB 0.327
\nCV 0.2995 - LB 0.328
\nCV 0.3002 - LB 0.330
\nCV 0.3013 - LB 0.331
These are single model scores.
", "votes": 5, "replies": [] }, { "id": 1605644, "author_name": "ForcewithMe", "author_url": "", "post_date": "2021-12-04T12:34:05.770000", "content": "=============update==============
\nsingle model:
\nCV: 309~311 LB: 320~325
\nCV: 319~321 LB: 332
\nThe gap between cv and lb is unstable.Sometimes model with a 320 CV only gets 323 LB.
Hi @forcewithme What mAP code did you use?
", "votes": 0, "replies": [] } ] }, { "id": 1591089, "author_name": "Manyu Li", "author_url": "", "post_date": "2021-11-22T00:13:17.457000", "content": "CV:295 LB:298
", "votes": 3, "replies": [ { "id": 1591111, "author_name": "ForcewithMe", "author_url": "", "post_date": "2021-11-22T01:49:41.857000", "content": "Very impressive cv！
", "votes": 0, "replies": [] } ] }, { "id": 1591372, "author_name": "Liam Nguyen", "author_url": "", "post_date": "2021-11-22T09:02:42.710000", "content": "Single fold model: Valid 0.326 - LB 0.334. The correlation is relatively good. But some gain in valid did not translate into LB gain
", "votes": 4, "replies": [ { "id": 1591398, "author_name": "Valentin Nikotin", "author_url": "", "post_date": "2021-11-22T09:38:36.367000", "content": "\n\nBut some gain in valid did not translate into LB gain
\n
Exactly the same, able to get 1% by fine tuning which is not translating into LB.
", "votes": 1, "replies": [] }, { "id": 1594849, "author_name": "ForcewithMe", "author_url": "", "post_date": "2021-11-25T07:01:24.257000", "content": "@namgalielei @rednikotin . Same here, CV improve 0.02 but LB doesn't even improve 0.001
", "votes": 0, "replies": [] } ] }, { "id": 1592868, "author_name": "yu4u", "author_url": "", "post_date": "2021-11-23T12:33:17.733000", "content": "hold-out (0.8 train - 0.2 val)
\nCV 0.301 - LB 0.314
metric
\nhttps://www.kaggle.com/theoviel/competition-metric-map-iou/notebook
single fold model
\nCV:0.2706, LB:0.294 with simple threshold, LB:0.3 with tunes threshold
Updated:
\nCV: 0.282 LB:0.311 with tuned threshold
would you mind to reveal why you can boost your CV from 0.27 to 0.282 (maybe increase number of epochs?) Thanks
", "votes": 1, "replies": [] }, { "id": 1621935, "author_name": "Hoda", "author_url": "", "post_date": "2021-12-18T07:49:28.167000", "content": "Yes, increasing the number of epochs. Also added some augmentations.
", "votes": 1, "replies": [] }, { "id": 1621937, "author_name": "NoChanged", "author_url": "", "post_date": "2021-12-18T07:53:50.453000", "content": "Thanks for your reply!
", "votes": 1, "replies": [] } ] }, { "id": 1592185, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-11-23T00:30:53.470000", "content": "single fold (train:valid=0.8:0.2)
\nCV 0.271 / LB 0.294
\nCV 0.272 / LB 0.297
\nCV 0.279 / LB 0.300
My best single fold model:
\nCV: 0.265 - LB 0.298
CV 0.310 - LB 0.322
\nsame model, but more epoch
\nCV 0.312 - LB 0.311
\nCV 0.314 - LB 0.313
update:
\nThere seems to be a significant positive correlation.
\nCV:2719 LB:302
\nCV:2760 LB:304
\nCV:2777 LB:307
Hey, thanks for sharing these numbers. Are you using just hold-out set as validation set or are you doing k-fold CV as discussed here
", "votes": 0, "replies": [] }, { "id": 1590381, "author_name": "ForcewithMe", "author_url": "", "post_date": "2021-11-21T07:29:35.560000", "content": "hold out. Haven't used k-fold yet
", "votes": 1, "replies": [] }, { "id": 1590726, "author_name": "Ioannis M", "author_url": "", "post_date": "2021-11-21T15:37:28.853000", "content": "thanks for sharing your results! indeed seems to correlate well with LB
\nmay I ask your split, i.e. 80/20, 90/10 ?
80/20.80/20.80/20.
", "votes": 1, "replies": [] }, { "id": 1590910, "author_name": "somuSan", "author_url": "", "post_date": "2021-11-21T18:40:31.567000", "content": "hi @atharvaingle can you please explain, what is hold-out. Im hearing this term for the first time. is it like, calculating mAP with different IoU for a single model on the validation data.
\nIm sorry if the question is very silly.
", "votes": 0, "replies": [] }, { "id": 1590915, "author_name": "Atharva Ingle", "author_url": "", "post_date": "2021-11-21T18:51:00.493000", "content": "Hold-out is just a small percentage of data kept for validating your trained model. Basically, it's nothing but 80-20 split as @forcewithme mentioned, where 80% of data is used for training and rest 20% is used for validation.
", "votes": 1, "replies": [] }, { "id": 1590939, "author_name": "somuSan", "author_url": "", "post_date": "2021-11-21T19:13:09.317000", "content": "Oh I see, thanks for answering ^_^
Mine went down. CV: 0.289 LB:0.283
", "votes": 0, "replies": [] }, { "id": 1592492, "author_name": "", "author_url": "", "post_date": "2021-11-23T05:33:34.437000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1589490": "I use the metric in the public detectron notebook. \nBest single model CV: 0.2653 LB:0.296\nWhat about yours?", "1591341": "CV 0.320 - LB 0.337 👀\n\nI use a 5-fold split and the correlation is alright so far, but I submitted only 3 models so I can't really tell for sure.", "1597036": "In my experiments, CV/LB correlation is below:\n\nCV 0.2746 - LB 0.315\nCV 0.2827 - LB 0.320\nCV 0.2880 - LB 0.324\nCV 0.2983 - LB 0.327\nCV 0.2995 - LB 0.328\nCV 0.3002 - LB 0.330\nCV 0.3013 - LB 0.331\n\nThese are single model scores.", "1605644": "=============update==============\nsingle model:\nCV: 309~311 LB: 320~325\nCV: 319~321 LB: 332\nThe gap between cv and lb is unstable.Sometimes model with a 320 CV only gets 323 LB. ", "1591089": "CV:295 LB:298", "1591372": "Single fold model: Valid 0.326 - LB 0.334. The correlation is relatively good. But some gain in valid did not translate into LB gain", "1592868": "hold-out (0.8 train - 0.2 val)\nCV 0.301 - LB 0.314\n\nmetric\nhttps://www.kaggle.com/theoviel/competition-metric-map-iou/notebook", "1592569": "single fold model\nCV:0.2706, LB:0.294 with simple threshold, LB:0.3 with tunes threshold", "1592185": "single fold (train:valid=0.8:0.2)\nCV 0.271 / LB 0.294\nCV 0.272 / LB 0.297\nCV 0.279 / LB 0.300", "1592058": "My best single fold model: \nCV: 0.265 - LB 0.298", "1618557": "CV 0.310 - LB 0.322\n\nsame model, but more epoch\nCV 0.312 - LB 0.311\nCV 0.314 - LB 0.313", "1589799": "update: \nThere seems to be a significant positive correlation.\nCV:2719 LB:302\nCV:2760 LB:304 \nCV:2777 LB:307", "1633606": "Mine went down. CV: 0.289 LB:0.283", "1592492": "" } }