Finally finished, shake up throwing out the top scores. And there are also people with zero score including me, what happened?
", "messages": [ { "id": "1301068", "postDate": "05/11/2021 00:04:30", "content": "Finally finished, shake up throwing out the top scores. And there are also people with zero score including me, what happened?
", "rawMarkdown": "Finally finished, shake up throwing out the top scores. And there are also people with zero score including me, what happened?", "votes": null }, { "id": "1301077", "postDate": "05/11/2021 00:11:20", "content": "For me any submission where I used deepflashV2 for my inference ended up getting a score of 0pts which of course is every recent submission that I made…no idea why this happened.
", "rawMarkdown": "For me any submission where I used deepflashV2 for my inference ended up getting a score of 0pts which of course is every recent submission that I made...no idea why this happened.", "votes": null }, { "id": "1301081", "postDate": "05/11/2021 00:13:37", "content": "Yes, the same for me, the deepflashV2 forked all get zero score….☹️
", "rawMarkdown": "Yes, the same for me, the deepflashV2 forked all get zero score....☹️", "votes": null }, { "id": "1301084", "postDate": "05/11/2021 00:15:40", "content": "Why would that be? I didn't even use the original code? Very frustrated - ended up with a 93.4 that got marked down to zero…
", "rawMarkdown": "Why would that be? I didn't even use the original code? Very frustrated - ended up with a 93.4 that got marked down to zero...", "votes": null }, { "id": "1301086", "postDate": "05/11/2021 00:17:45", "content": "It seems that some starter kernels had issues when submitting the results, it seems our deepflash2 kernel was also affected. We did speculate this could happen because the submissions were too fast. We updated our public notebook and tried to address our concerns, but our reach is limited.
\nThe (rest of the) shakeup was most likely due to hand labeling on the d4 dataset, which might have caused many participants to train on data inconsistent to the rest of the challenge.
", "rawMarkdown": "It seems that some starter kernels had issues when submitting the results, it seems our deepflash2 kernel was also affected. We did speculate this could happen because the submissions were too fast. We updated our public notebook and tried to address our concerns, but our reach is limited.\n\nThe (rest of the) shakeup was most likely due to hand labeling on the d4 dataset, which might have caused many participants to train on data inconsistent to the rest of the challenge.", "votes": null }, { "id": "1301093", "postDate": "05/11/2021 00:21:28", "content": "The way the deepflash2 notebook is made, if the inference cell fails (for instance due to a memory error), it will use the empty string \"\" as a final predictions. Hence scoring 0, instead of returning an error.
\nCorrect me if I'm wrong, but that would explain all the 0 scores.
I think it is probably an error. anything else. My notebooks are clean.
", "rawMarkdown": "I think it is probably an error. anything else. My notebooks are clean.", "votes": null }, { "id": "1301117", "postDate": "05/11/2021 00:32:55", "content": "large models rule. effnetb5 fared the best, in my case. Looks like the private dataset had some mis-labeled dark gloms, but not as many as d48. I prepared 2 kernels for the submission. A clean one and one trained with Zhano's hand-labels. Both scored with a one point difference. The best submission, which I did not mark, was a pure effnetb5, no pseudo, no dark gloms.
\nAnyway, given that I'm novice in the field and it's just a passion project, I'm happy with my results.
Thanks @theoviel , i got caught. Shame to me 😨. It explains the whole thing !!!
", "rawMarkdown": "Thanks @theoviel , i got caught. Shame to me 😨. It explains the whole thing !!!", "votes": null }, { "id": "1301129", "postDate": "05/11/2021 00:38:45", "content": "In our team case, we saved sample_submission with each iterrows(each image), so it didn't make error on Public LB, but It turned out that it just saved Public Images literally not Private Images.
", "rawMarkdown": "In our team case, we saved sample_submission with each iterrows(each image), so it didn't make error on Public LB, but It turned out that it just saved Public Images literally not Private Images.", "votes": null }, { "id": "1301137", "postDate": "05/11/2021 00:43:12", "content": "Yea, Chris saying that hand label and pseudo label does not work.
\nMy effnetb4 on 512 with simple augs trained with old dataset and 5 folds get 0.940 private lb, but not chosen…
There are many more zeros scores than u think. My notebooks were made in R, using pure tensorflow/keras models. All of them got zero. It isnt related to deepflash.
", "rawMarkdown": "There are many more zeros scores than u think. My notebooks were made in R, using pure tensorflow/keras models. All of them got zero. It isnt related to deepflash.", "votes": null }, { "id": "1301150", "postDate": "05/11/2021 00:50:43", "content": "Really wish there could be a 5 minute window to resubmit with new inference code….this impacts a huge amount of people and it's a shame all the good work ends up going to waste.
", "rawMarkdown": "Really wish there could be a 5 minute window to resubmit with new inference code....this impacts a huge amount of people and it's a shame all the good work ends up going to waste.", "votes": null }, { "id": "1301152", "postDate": "05/11/2021 00:53:11", "content": "too late for that, unfortunately. people started posting solutions, etc.
", "rawMarkdown": "too late for that, unfortunately. people started posting solutions, etc.", "votes": null }, { "id": "1301156", "postDate": "05/11/2021 00:55:57", "content": "@theoviel Even submit with csv notebook has 0 as private score . There is no memory issue in this case . All my submissions are zero , not just deepflash
", "rawMarkdown": "theoviel Even submit with csv notebook has 0 as private score . There is no memory issue in this case . All my submissions are zero , not just deepflash", "votes": null }, { "id": "1301166", "postDate": "05/11/2021 01:05:08", "content": "In your case, it makes sense @usharengaraju. The private dataset was hidden and only your code could make the predictions. So, the csv notebook wont work on the private leaderboard.
", "rawMarkdown": "In your case, it makes sense @usharengaraju. The private dataset was hidden and only your code could make the predictions. So, the csv notebook wont work on the private leaderboard.", "votes": null }, { "id": "1301352", "postDate": "05/11/2021 03:23:45", "content": "Pretty sad on such a thing. Guess it's myself to blame. Did feel that the pub LB ran kinda fast.
\nSeveral guys including myself did post question about whether private would be run. Got a lot of help from top guys, no complain. But found no final answer.
\nKinda sad to find out that one of very early submission got 0.947 in private (with a 0.907 public), but I guess I'm not alone in the zero scorers. That was some tweak of iafoss's so nothing to brag about. Along the way I did learn a lot from the public kernels that I can't complete on my own, so I guess I don't deserve a \"kaggle expert\" title anyway
\nI compared those with private and those with zero private, and found that they indeed loaded the same csv file.
\nFor beginners like me, I wish kaggle could give a hint when a submission got a vanishing score such as 0 in private in the future.
", "rawMarkdown": "Pretty sad on such a thing. Guess it's myself to blame. Did feel that the pub LB ran kinda fast. \n\nSeveral guys including myself did post question about whether private would be run. Got a lot of help from top guys, no complain. But found no final answer.\n\nKinda sad to find out that one of very early submission got 0.947 in private (with a 0.907 public), but I guess I'm not alone in the zero scorers. That was some tweak of iafoss's so nothing to brag about. Along the way I did learn a lot from the public kernels that I can't complete on my own, so I guess I don't deserve a \"kaggle expert\" title anyway\n\nI compared those with private and those with zero private, and found that they indeed loaded the same csv file.\n\nFor beginners like me, I wish kaggle could give a hint when a submission got a vanishing score such as 0 in private in the future.", "votes": null }, { "id": "1301515", "postDate": "05/11/2021 05:48:56", "content": "I am waiting to hear any confirmation but it is possible the issue with 0 scores has to do with the sample submission csv and there being another copy in the test folder (normally not the case). So only if you derived the ids from the image names did your submissions work. And of course if you got past the queue issues.
\nUPDATE - see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments v15 changes 4 days ago to use rasterio instead of tifffile due to potential large image in private test.
", "rawMarkdown": "I am waiting to hear any confirmation but it is possible the issue with 0 scores has to do with the sample submission csv and there being another copy in the test folder (normally not the case). So only if you derived the ids from the image names did your submissions work. And of course if you got past the queue issues. \n\nUPDATE - see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments v15 changes 4 days ago to use rasterio instead of tifffile due to potential large image in private test.", "votes": null }, { "id": "1301672", "postDate": "05/11/2021 07:30:23", "content": "Shake up is expected cuz too small LB, only 15 images
", "rawMarkdown": "Shake up is expected cuz too small LB, only 15 images", "votes": null } ], "comments": [ { "id": 1301077, "author_name": "goalieperson", "author_url": "", "post_date": "05/11/2021 00:11:20", "content": "For me any submission where I used deepflashV2 for my inference ended up getting a score of 0pts which of course is every recent submission that I made…no idea why this happened.
", "votes": null, "replies": [ { "id": 1301081, "author_name": "zungmann", "author_url": "", "post_date": "05/11/2021 00:13:37", "content": "Yes, the same for me, the deepflashV2 forked all get zero score….☹️
", "votes": null, "replies": [] }, { "id": 1301084, "author_name": "goalieperson", "author_url": "", "post_date": "05/11/2021 00:15:40", "content": "Why would that be? I didn't even use the original code? Very frustrated - ended up with a 93.4 that got marked down to zero…
", "votes": null, "replies": [] } ] }, { "id": 1301086, "author_name": "theudas", "author_url": "", "post_date": "05/11/2021 00:17:45", "content": "It seems that some starter kernels had issues when submitting the results, it seems our deepflash2 kernel was also affected. We did speculate this could happen because the submissions were too fast. We updated our public notebook and tried to address our concerns, but our reach is limited.
\nThe (rest of the) shakeup was most likely due to hand labeling on the d4 dataset, which might have caused many participants to train on data inconsistent to the rest of the challenge.
", "votes": null, "replies": [] }, { "id": 1301093, "author_name": "theoviel", "author_url": "", "post_date": "05/11/2021 00:21:28", "content": "The way the deepflash2 notebook is made, if the inference cell fails (for instance due to a memory error), it will use the empty string \"\" as a final predictions. Hence scoring 0, instead of returning an error.
\nCorrect me if I'm wrong, but that would explain all the 0 scores.
Thanks @theoviel , i got caught. Shame to me 😨. It explains the whole thing !!!
", "votes": null, "replies": [] }, { "id": 1301140, "author_name": "rpsantosakaggle", "author_url": "", "post_date": "05/11/2021 00:45:28", "content": "There are many more zeros scores than u think. My notebooks were made in R, using pure tensorflow/keras models. All of them got zero. It isnt related to deepflash.
", "votes": null, "replies": [] }, { "id": 1301156, "author_name": "usharengaraju", "author_url": "", "post_date": "05/11/2021 00:55:57", "content": "@theoviel Even submit with csv notebook has 0 as private score . There is no memory issue in this case . All my submissions are zero , not just deepflash
", "votes": null, "replies": [] }, { "id": 1301166, "author_name": "rpsantosakaggle", "author_url": "", "post_date": "05/11/2021 01:05:08", "content": "In your case, it makes sense @usharengaraju. The private dataset was hidden and only your code could make the predictions. So, the csv notebook wont work on the private leaderboard.
", "votes": null, "replies": [] } ] }, { "id": 1301094, "author_name": "rpsantosakaggle", "author_url": "", "post_date": "05/11/2021 00:22:08", "content": "I think it is probably an error. anything else. My notebooks are clean.
", "votes": null, "replies": [] }, { "id": 1301117, "author_name": "andrasferenczi", "author_url": "", "post_date": "05/11/2021 00:32:55", "content": "large models rule. effnetb5 fared the best, in my case. Looks like the private dataset had some mis-labeled dark gloms, but not as many as d48. I prepared 2 kernels for the submission. A clean one and one trained with Zhano's hand-labels. Both scored with a one point difference. The best submission, which I did not mark, was a pure effnetb5, no pseudo, no dark gloms.
\nAnyway, given that I'm novice in the field and it's just a passion project, I'm happy with my results.
Yea, Chris saying that hand label and pseudo label does not work.
\nMy effnetb4 on 512 with simple augs trained with old dataset and 5 folds get 0.940 private lb, but not chosen…
In our team case, we saved sample_submission with each iterrows(each image), so it didn't make error on Public LB, but It turned out that it just saved Public Images literally not Private Images.
", "votes": null, "replies": [] }, { "id": 1301150, "author_name": "goalieperson", "author_url": "", "post_date": "05/11/2021 00:50:43", "content": "Really wish there could be a 5 minute window to resubmit with new inference code….this impacts a huge amount of people and it's a shame all the good work ends up going to waste.
", "votes": null, "replies": [ { "id": 1301152, "author_name": "andrasferenczi", "author_url": "", "post_date": "05/11/2021 00:53:11", "content": "too late for that, unfortunately. people started posting solutions, etc.
", "votes": null, "replies": [] } ] }, { "id": 1301352, "author_name": "reaverlee", "author_url": "", "post_date": "05/11/2021 03:23:45", "content": "Pretty sad on such a thing. Guess it's myself to blame. Did feel that the pub LB ran kinda fast.
\nSeveral guys including myself did post question about whether private would be run. Got a lot of help from top guys, no complain. But found no final answer.
\nKinda sad to find out that one of very early submission got 0.947 in private (with a 0.907 public), but I guess I'm not alone in the zero scorers. That was some tweak of iafoss's so nothing to brag about. Along the way I did learn a lot from the public kernels that I can't complete on my own, so I guess I don't deserve a \"kaggle expert\" title anyway
\nI compared those with private and those with zero private, and found that they indeed loaded the same csv file.
\nFor beginners like me, I wish kaggle could give a hint when a submission got a vanishing score such as 0 in private in the future.
", "votes": null, "replies": [] }, { "id": 1301515, "author_name": "something4kag", "author_url": "", "post_date": "05/11/2021 05:48:56", "content": "I am waiting to hear any confirmation but it is possible the issue with 0 scores has to do with the sample submission csv and there being another copy in the test folder (normally not the case). So only if you derived the ids from the image names did your submissions work. And of course if you got past the queue issues.
\nUPDATE - see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments v15 changes 4 days ago to use rasterio instead of tifffile due to potential large image in private test.
", "votes": null, "replies": [] }, { "id": 1301672, "author_name": "leighplt", "author_url": "", "post_date": "05/11/2021 07:30:23", "content": "Shake up is expected cuz too small LB, only 15 images
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1301068": "Finally finished, shake up throwing out the top scores. And there are also people with zero score including me, what happened?", "1301077": "For me any submission where I used deepflashV2 for my inference ended up getting a score of 0pts which of course is every recent submission that I made...no idea why this happened.", "1301081": "Yes, the same for me, the deepflashV2 forked all get zero score....☹️", "1301084": "Why would that be? I didn't even use the original code? Very frustrated - ended up with a 93.4 that got marked down to zero...", "1301086": "It seems that some starter kernels had issues when submitting the results, it seems our deepflash2 kernel was also affected. We did speculate this could happen because the submissions were too fast. We updated our public notebook and tried to address our concerns, but our reach is limited.\n\nThe (rest of the) shakeup was most likely due to hand labeling on the d4 dataset, which might have caused many participants to train on data inconsistent to the rest of the challenge.", "1301093": "The way the deepflash2 notebook is made, if the inference cell fails (for instance due to a memory error), it will use the empty string \"\" as a final predictions. Hence scoring 0, instead of returning an error.\nCorrect me if I'm wrong, but that would explain all the 0 scores.", "1301094": "I think it is probably an error. anything else. My notebooks are clean.", "1301117": "large models rule. effnetb5 fared the best, in my case. Looks like the private dataset had some mis-labeled dark gloms, but not as many as d48. I prepared 2 kernels for the submission. A clean one and one trained with Zhano's hand-labels. Both scored with a one point difference. The best submission, which I did not mark, was a pure effnetb5, no pseudo, no dark gloms. \nAnyway, given that I'm novice in the field and it's just a passion project, I'm happy with my results.", "1301124": "Thanks @theoviel , i got caught. Shame to me 😨. It explains the whole thing !!!", "1301129": "In our team case, we saved sample_submission with each iterrows(each image), so it didn't make error on Public LB, but It turned out that it just saved Public Images literally not Private Images.", "1301137": "Yea, Chris saying that hand label and pseudo label does not work.\nMy effnetb4 on 512 with simple augs trained with old dataset and 5 folds get 0.940 private lb, but not chosen...", "1301140": "There are many more zeros scores than u think. My notebooks were made in R, using pure tensorflow/keras models. All of them got zero. It isnt related to deepflash.", "1301150": "Really wish there could be a 5 minute window to resubmit with new inference code....this impacts a huge amount of people and it's a shame all the good work ends up going to waste.", "1301152": "too late for that, unfortunately. people started posting solutions, etc.", "1301156": "theoviel Even submit with csv notebook has 0 as private score . There is no memory issue in this case . All my submissions are zero , not just deepflash", "1301166": "In your case, it makes sense @usharengaraju. The private dataset was hidden and only your code could make the predictions. So, the csv notebook wont work on the private leaderboard.", "1301352": "Pretty sad on such a thing. Guess it's myself to blame. Did feel that the pub LB ran kinda fast. \n\nSeveral guys including myself did post question about whether private would be run. Got a lot of help from top guys, no complain. But found no final answer.\n\nKinda sad to find out that one of very early submission got 0.947 in private (with a 0.907 public), but I guess I'm not alone in the zero scorers. That was some tweak of iafoss's so nothing to brag about. Along the way I did learn a lot from the public kernels that I can't complete on my own, so I guess I don't deserve a \"kaggle expert\" title anyway\n\nI compared those with private and those with zero private, and found that they indeed loaded the same csv file.\n\nFor beginners like me, I wish kaggle could give a hint when a submission got a vanishing score such as 0 in private in the future.", "1301515": "I am waiting to hear any confirmation but it is possible the issue with 0 scores has to do with the sample submission csv and there being another copy in the test folder (normally not the case). So only if you derived the ids from the image names did your submissions work. And of course if you got past the queue issues. \n\nUPDATE - see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments v15 changes 4 days ago to use rasterio instead of tifffile due to potential large image in private test.", "1301672": "Shake up is expected cuz too small LB, only 15 images" }, "source": "meta" }