Are you in the process of fixing a bug?
", "messages": [ { "id": 1320261, "postDate": "2021-05-24T00:35:03.520Z", "content": "Are you in the process of fixing a bug?
", "rawMarkdown": "Are you in the process of fixing a bug?", "votes": 39 }, { "id": 1331802, "postDate": "2021-06-01T18:21:40.430Z", "content": "Hi everyone! We've been working with the host on both the duplicates and other labeling inconsistencies that will also resolve the scoring issues you've identified. We expect these updates to be complete next week. I know this wait can be frustrating, but we hope only to have to make this update once and therefore are are taking our time with it. It does not change the problem or metric as defined, but you can expect that the scores will be accurate once we've remedied the inconsistencies/duplications.
", "rawMarkdown": "Hi everyone! We've been working with the host on both the duplicates and other labeling inconsistencies that will also resolve the scoring issues you've identified. We expect these updates to be complete next week. I know this wait can be frustrating, but we hope only to have to make this update once and therefore are are taking our time with it. It does not change the problem or metric as defined, but you can expect that the scores will be accurate once we've remedied the inconsistencies/duplications.", "votes": 25, "replies": [ { "id": 1337966, "postDate": "2021-06-06T04:33:53.267Z", "content": "thank you.i didn't know that its have worng data.
", "rawMarkdown": "thank you.i didn't know that its have worng data." }, { "id": 1339992, "postDate": "2021-06-07T15:22:16.650Z", "content": "Any update? @juliaelliott
", "rawMarkdown": "Any update? @juliaelliott ", "votes": 3 }, { "id": 1343248, "postDate": "2021-06-10T05:59:46.360Z", "content": "Waiting for your good news
", "rawMarkdown": "Waiting for your good news" }, { "id": 1346646, "postDate": "2021-06-12T14:23:14.953Z", "content": "any update on this?
", "rawMarkdown": "any update on this? ", "votes": 1 } ] }, { "id": 1320866, "postDate": "2021-05-24T11:37:26.473Z", "content": "@philculliton , @juliaelliott can you check this issue?
", "rawMarkdown": "@philculliton , @juliaelliott can you check this issue?", "votes": 8 }, { "id": 1320305, "postDate": "2021-05-24T01:44:43.853Z", "content": "Same as mine, very frustrated.
", "rawMarkdown": "Same as mine, very frustrated.", "votes": 3, "replies": [ { "id": 1320365, "postDate": "2021-05-24T03:07:32.530Z", "content": "The host should clarify the current status of the response. There is also a lack of clarity on data and metrics.
\nhttps://www.kaggle.com/c/siim-covid19-detection/discussion/239980
\nhttps://www.kaggle.com/c/siim-covid19-detection/discussion/241275
Same as mine. score 0.051 always.
", "rawMarkdown": "Same as mine. score 0.051 always.", "votes": 1 }, { "id": 1327240, "postDate": "2021-05-29T05:08:36.830Z", "content": "@nguyenbadung You LB score have improved by adding the image-level prediction? But why I got a bad LB result while adding the same operation. What's more, I have tried many detection models for image-level task but the LB results are the same.π By the way, the image-level prediction result is totally different.
", "rawMarkdown": "@nguyenbadung You LB score have improved by adding the image-level prediction? But why I got a bad LB result while adding the same operation. What's more, I have tried many detection models for image-level task but the LB results are the same.π By the way, the image-level prediction result is totally different.", "votes": 2 }, { "id": 1320532, "postDate": "2021-05-24T06:32:46.630Z", "content": "same, i got 0.000π³
", "rawMarkdown": "same, i got 0.000π³", "votes": 2 }, { "id": 1329562, "postDate": "2021-05-31T08:05:48.810Z", "content": "Did this problem already solved??
", "rawMarkdown": "Did this problem already solved??", "replies": [ { "id": 1329622, "postDate": "2021-05-31T08:56:06.547Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1329690, "postDate": "2021-05-31T09:56:26.180Z", "content": "still waiting π
", "rawMarkdown": "still waiting π", "votes": 2 }, { "id": 1337968, "postDate": "2021-06-06T04:34:57.763Z", "content": "me too. i'm waiting for data.
", "rawMarkdown": "me too. i'm waiting for data." } ] }, { "id": 1325947, "postDate": "2021-05-28T05:49:28.373Z", "content": "Any updates on this? Is scoring debugged?
", "rawMarkdown": "Any updates on this? Is scoring debugged?" }, { "id": 1322394, "postDate": "2021-05-25T12:32:59.370Z", "content": "Are the scores on Leaderboard just for image studies?
", "rawMarkdown": "Are the scores on Leaderboard just for image studies?", "replies": [ { "id": 1322438, "postDate": "2021-05-25T13:01:47.457Z", "content": "We can guess that our study level score is probably 0 or 0.001. However, there is no point in thinking about it because there is a bug. Since there is a bug in the scoring function, We can't trust the image level score either. We don't have time to spend on a buggy game, so we should spend our time on other things while the host doesn't reply to us.
", "rawMarkdown": "We can guess that our study level score is probably 0 or 0.001. However, there is no point in thinking about it because there is a bug. Since there is a bug in the scoring function, We can't trust the image level score either. We don't have time to spend on a buggy game, so we should spend our time on other things while the host doesn't reply to us.", "votes": 5 }, { "id": 1322520, "postDate": "2021-05-25T13:43:21.207Z", "content": "You are absolutely correct.
\nHope the Host clears all the problems soon enough.
Hi there, thanks for raising your concerns. We're continuing to investigate the scoring issues you've raised. We will be in touch if we discover or resolve any issues.
", "rawMarkdown": "Hi there, thanks for raising your concerns. We're continuing to investigate the scoring issues you've raised. We will be in touch if we discover or resolve any issues. ", "votes": 10 }, { "id": 1324370, "postDate": "2021-05-26T20:53:52.657Z", "content": "thanks for the info Julia, it is important for us to make sure this competition will be OK before we invest lots of time into it :)
", "rawMarkdown": "thanks for the info Julia, it is important for us to make sure this competition will be OK before we invest lots of time into it :)" }, { "id": 1324445, "postDate": "2021-05-27T00:50:29.050Z", "content": "Thanks for the reply! Now I can proceed with my analysis with no worries.
", "rawMarkdown": "Thanks for the reply! Now I can proceed with my analysis with no worries." }, { "id": 1326396, "postDate": "2021-05-28T12:54:40.737Z", "content": "\nsingle fold mAP@0.5 [6 classes]:
\nstudy level prediction + none 1 0 0 1 1 for all image level prediction
\nNegative for Pneumonia 0.710
\nTypical Appearance 0.769
\nIndeterminate Appearance 0.239
\nAtypical Appearance 0.269
\nnone 0.324
\nopacity 0
local mAP = 0.385 LB 0.383
\nstudy level prediction + image level prediction
\nNegative for Pneumonia 0.710
\nTypical Appearance 0.769
\nIndeterminate Appearance 0.239
\nAtypical Appearance 0.269
\nnone 0.286
\nopacity 0.450
local mAP: 0.453 but LB: 0.384
\nI checked few submissions but didn't get any point for the opacity class
", "rawMarkdown": "@juliaelliott @philculliton \n\nsingle fold mAP@0.5 [6 classes]:\n\n**study level prediction + none 1 0 0 1 1 for all image level prediction**\n\nNegative for Pneumonia 0.710\nTypical Appearance 0.769\nIndeterminate Appearance 0.239\nAtypical Appearance 0.269\nnone 0.324\nopacity 0\n\n**local mAP = 0.385 LB 0.383**\n\n**study level prediction + image level prediction**\n\nNegative for Pneumonia 0.710\nTypical Appearance 0.769\nIndeterminate Appearance 0.239\nAtypical Appearance 0.269\nnone 0.286\nopacity 0.450\n\n**local mAP: 0.453 but LB: 0.384**\n\nI checked few submissions but didn't get any point for the opacity class", "votes": 9, "replies": [ { "id": 1326610, "postDate": "2021-05-28T15:11:42.937Z", "content": "I also suspect a scoring bug with the image_level scoring. I tried to simply output the confidence value of \"Negative for Pneumonia\" in the \"none\" image_level class for all the associated images in a study. I was assuming a high correlation between these two classes.
\nI got absolutely no LB score change compared to the original submission with none 1 0 0 1 1 for all image_level cases.
\nIt's suspicious unless the \"none\" class doesn't count at all in the final score.
Hi @nguyenbadung, for image level prediction, what detection models did you use?
", "rawMarkdown": "Hi @nguyenbadung, for image level prediction, what detection models did you use?" }, { "id": 1337915, "postDate": "2021-06-06T02:57:46.807Z", "content": "@mrinath Efficientdet D3 image size 512 https://www.kaggle.com/shonenkov/training-efficientdet
", "rawMarkdown": "@mrinath Efficientdet D3 image size 512 https://www.kaggle.com/shonenkov/training-efficientdet", "votes": 3 }, { "id": 1338078, "postDate": "2021-06-06T06:46:35.003Z", "content": "Thanks, are you using the same version, as I have heard it is now out of date with respect to newer versions. @nguyenbadung
", "rawMarkdown": "Thanks, are you using the same version, as I have heard it is now out of date with respect to newer versions. @nguyenbadung ", "votes": 1 }, { "id": 1338154, "postDate": "2021-06-06T08:14:06.090Z", "content": "Yes, I use the same version https://www.kaggle.com/shonenkov/timm-efficientdet-pytorch
", "rawMarkdown": "Yes, I use the same version [https://www.kaggle.com/shonenkov/timm-efficientdet-pytorch](url)", "votes": 1 } ] } ], "comments": [ { "id": 1331802, "author_name": "Julia Elliott", "author_url": "", "post_date": "2021-06-01T18:21:40.430000", "content": "Hi everyone! We've been working with the host on both the duplicates and other labeling inconsistencies that will also resolve the scoring issues you've identified. We expect these updates to be complete next week. I know this wait can be frustrating, but we hope only to have to make this update once and therefore are are taking our time with it. It does not change the problem or metric as defined, but you can expect that the scores will be accurate once we've remedied the inconsistencies/duplications.
", "votes": 25, "replies": [ { "id": 1337966, "author_name": "tensor choko", "author_url": "", "post_date": "2021-06-06T04:33:53.267000", "content": "thank you.i didn't know that its have worng data.
", "votes": 0, "replies": [] }, { "id": 1339992, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-06-07T15:22:16.650000", "content": "Any update? @juliaelliott
", "votes": 3, "replies": [] }, { "id": 1343248, "author_name": "README", "author_url": "", "post_date": "2021-06-10T05:59:46.360000", "content": "Waiting for your good news
", "votes": 0, "replies": [] }, { "id": 1346646, "author_name": "Haw Keat", "author_url": "", "post_date": "2021-06-12T14:23:14.953000", "content": "any update on this?
", "votes": 1, "replies": [] } ] }, { "id": 1320866, "author_name": "DungNB", "author_url": "", "post_date": "2021-05-24T11:37:26.473000", "content": "@philculliton , @juliaelliott can you check this issue?
", "votes": 8, "replies": [] }, { "id": 1320305, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-05-24T01:44:43.853000", "content": "Same as mine, very frustrated.
", "votes": 3, "replies": [ { "id": 1320365, "author_name": "YujiAriyasu", "author_url": "", "post_date": "2021-05-24T03:07:32.530000", "content": "The host should clarify the current status of the response. There is also a lack of clarity on data and metrics.
\nhttps://www.kaggle.com/c/siim-covid19-detection/discussion/239980
\nhttps://www.kaggle.com/c/siim-covid19-detection/discussion/241275
Same as mine. score 0.051 always.
", "votes": 1, "replies": [] }, { "id": 1327240, "author_name": "blueboy-97", "author_url": "", "post_date": "2021-05-29T05:08:36.830000", "content": "@nguyenbadung You LB score have improved by adding the image-level prediction? But why I got a bad LB result while adding the same operation. What's more, I have tried many detection models for image-level task but the LB results are the same.π By the way, the image-level prediction result is totally different.
", "votes": 2, "replies": [] }, { "id": 1320532, "author_name": "MOONMOON", "author_url": "", "post_date": "2021-05-24T06:32:46.630000", "content": "same, i got 0.000π³
", "votes": 2, "replies": [] }, { "id": 1329562, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-05-31T08:05:48.810000", "content": "Did this problem already solved??
", "votes": 0, "replies": [ { "id": 1329622, "author_name": "", "author_url": "", "post_date": "2021-05-31T08:56:06.547000", "content": "", "votes": 0, "replies": [] }, { "id": 1329690, "author_name": "DungNB", "author_url": "", "post_date": "2021-05-31T09:56:26.180000", "content": "still waiting π
", "votes": 2, "replies": [] }, { "id": 1337968, "author_name": "tensor choko", "author_url": "", "post_date": "2021-06-06T04:34:57.763000", "content": "me too. i'm waiting for data.
", "votes": 0, "replies": [] } ] }, { "id": 1325947, "author_name": "Joseph AMIGO", "author_url": "", "post_date": "2021-05-28T05:49:28.373000", "content": "Any updates on this? Is scoring debugged?
", "votes": 0, "replies": [] }, { "id": 1322394, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-05-25T12:32:59.370000", "content": "Are the scores on Leaderboard just for image studies?
", "votes": 0, "replies": [ { "id": 1322438, "author_name": "YujiAriyasu", "author_url": "", "post_date": "2021-05-25T13:01:47.457000", "content": "We can guess that our study level score is probably 0 or 0.001. However, there is no point in thinking about it because there is a bug. Since there is a bug in the scoring function, We can't trust the image level score either. We don't have time to spend on a buggy game, so we should spend our time on other things while the host doesn't reply to us.
", "votes": 5, "replies": [] }, { "id": 1322520, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-05-25T13:43:21.207000", "content": "You are absolutely correct.
\nHope the Host clears all the problems soon enough.
Hi there, thanks for raising your concerns. We're continuing to investigate the scoring issues you've raised. We will be in touch if we discover or resolve any issues.
", "votes": 10, "replies": [] }, { "id": 1324370, "author_name": "Jacek Poplawski", "author_url": "", "post_date": "2021-05-26T20:53:52.657000", "content": "thanks for the info Julia, it is important for us to make sure this competition will be OK before we invest lots of time into it :)
", "votes": 0, "replies": [] }, { "id": 1324445, "author_name": "YujiAriyasu", "author_url": "", "post_date": "2021-05-27T00:50:29.050000", "content": "Thanks for the reply! Now I can proceed with my analysis with no worries.
", "votes": 0, "replies": [] }, { "id": 1326396, "author_name": "DungNB", "author_url": "", "post_date": "2021-05-28T12:54:40.737000", "content": "\nsingle fold mAP@0.5 [6 classes]:
\nstudy level prediction + none 1 0 0 1 1 for all image level prediction
\nNegative for Pneumonia 0.710
\nTypical Appearance 0.769
\nIndeterminate Appearance 0.239
\nAtypical Appearance 0.269
\nnone 0.324
\nopacity 0
local mAP = 0.385 LB 0.383
\nstudy level prediction + image level prediction
\nNegative for Pneumonia 0.710
\nTypical Appearance 0.769
\nIndeterminate Appearance 0.239
\nAtypical Appearance 0.269
\nnone 0.286
\nopacity 0.450
local mAP: 0.453 but LB: 0.384
\nI checked few submissions but didn't get any point for the opacity class
", "votes": 9, "replies": [ { "id": 1326610, "author_name": "Alexandre Cadrin-ChΓͺnevert", "author_url": "", "post_date": "2021-05-28T15:11:42.937000", "content": "I also suspect a scoring bug with the image_level scoring. I tried to simply output the confidence value of \"Negative for Pneumonia\" in the \"none\" image_level class for all the associated images in a study. I was assuming a high correlation between these two classes.
\nI got absolutely no LB score change compared to the original submission with none 1 0 0 1 1 for all image_level cases.
\nIt's suspicious unless the \"none\" class doesn't count at all in the final score.
Hi @nguyenbadung, for image level prediction, what detection models did you use?
", "votes": 0, "replies": [] }, { "id": 1337915, "author_name": "DungNB", "author_url": "", "post_date": "2021-06-06T02:57:46.807000", "content": "@mrinath Efficientdet D3 image size 512 https://www.kaggle.com/shonenkov/training-efficientdet
", "votes": 3, "replies": [] }, { "id": 1338078, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-06-06T06:46:35.003000", "content": "Thanks, are you using the same version, as I have heard it is now out of date with respect to newer versions. @nguyenbadung
", "votes": 1, "replies": [] }, { "id": 1338154, "author_name": "DungNB", "author_url": "", "post_date": "2021-06-06T08:14:06.090000", "content": "Yes, I use the same version https://www.kaggle.com/shonenkov/timm-efficientdet-pytorch
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1320261": "Are you in the process of fixing a bug?", "1331802": "Hi everyone! We've been working with the host on both the duplicates and other labeling inconsistencies that will also resolve the scoring issues you've identified. We expect these updates to be complete next week. I know this wait can be frustrating, but we hope only to have to make this update once and therefore are are taking our time with it. It does not change the problem or metric as defined, but you can expect that the scores will be accurate once we've remedied the inconsistencies/duplications.", "1320866": "@philculliton , @juliaelliott can you check this issue?", "1320305": "Same as mine, very frustrated.", "1323128": "Same as mine. score 0.051 always.", "1327240": "@nguyenbadung You LB score have improved by adding the image-level prediction? But why I got a bad LB result while adding the same operation. What's more, I have tried many detection models for image-level task but the LB results are the same.π By the way, the image-level prediction result is totally different.", "1320532": "same, i got 0.000π³", "1329562": "Did this problem already solved??", "1325947": "Any updates on this? Is scoring debugged?", "1322394": "Are the scores on Leaderboard just for image studies?" } }