We have read your feedback and investigated the high incidence of zero scores on the private leaderboard. This investigation determined that these were caused by unhandled errors in submission code, and the impact exacerbated by a large number of forks of a public notebook. We sympathize and never want to see this type of issue happen, but we also must uphold a standard of fairness and impartiality. This means we can not intervene to selectively fix issues with submitted code.
\nWe will perform our normal post-competition rules checks, but the scores and rankings will largely stand as they are. Going forward, we urge all code-competition participants to adopt best practices with respect to handling errors and validating outputs, some of which are discussed at https://www.kaggle.com/code-competition-debugging. Thank you for your participation and understanding.
", "messages": [ { "id": 1306482, "postDate": "2021-05-13T19:41:08.757Z", "content": "We have read your feedback and investigated the high incidence of zero scores on the private leaderboard. This investigation determined that these were caused by unhandled errors in submission code, and the impact exacerbated by a large number of forks of a public notebook. We sympathize and never want to see this type of issue happen, but we also must uphold a standard of fairness and impartiality. This means we can not intervene to selectively fix issues with submitted code.
\nWe will perform our normal post-competition rules checks, but the scores and rankings will largely stand as they are. Going forward, we urge all code-competition participants to adopt best practices with respect to handling errors and validating outputs, some of which are discussed at https://www.kaggle.com/code-competition-debugging. Thank you for your participation and understanding.
", "rawMarkdown": "We have read your feedback and investigated the high incidence of zero scores on the private leaderboard. This investigation determined that these were caused by unhandled errors in submission code, and the impact exacerbated by a large number of forks of a public notebook. We sympathize and never want to see this type of issue happen, but we also must uphold a standard of fairness and impartiality. This means we can not intervene to selectively fix issues with submitted code.\n\nWe will perform our normal post-competition rules checks, but the scores and rankings will largely stand as they are. Going forward, we urge all code-competition participants to adopt best practices with respect to handling errors and validating outputs, some of which are discussed at https://www.kaggle.com/code-competition-debugging. Thank you for your participation and understanding.", "votes": 47 }, { "id": 1307077, "postDate": "2021-05-14T08:39:03.160Z", "content": "I understand and agree with the result.
\nBut I think the most important thing right now is how to know if private submissions fail when public scored and succeeded.
\nFor example, add a new class of errors, \"Private Submission Scoring Error \" to let people know private submissions result can run correctly.
\nThis will avoid similar situations again
When is this endless competition over
", "rawMarkdown": "When is this endless competition over", "votes": 8 }, { "id": 1306761, "postDate": "2021-05-14T04:20:26Z", "content": "Phil, can you please give a justification for why notebook continues to run next cell after an unhandled exception happened? This is the only reason my notebook (which has nothing to do with the public one) scored 0? I gather all predictions in one cell and create a csv in the next cell. My assumption was that if the csv is created the previous cell finished without errors.
\nThis situation is impossible to debug without knowing such a peculiarity beforehand.
This is ridiculous. I had some with 0 private scores, others with pretty good private scores. I forgot to select my notebooks though. In other competition, I remember that top 2 scores will be chosen automatically.
", "rawMarkdown": "This is ridiculous. I had some with 0 private scores, others with pretty good private scores. I forgot to select my notebooks though. In other competition, I remember that top 2 scores will be chosen automatically." } ] }, { "id": 1311051, "postDate": "2021-05-17T06:08:36.150Z", "content": "Hi~ ๐ I want to konw when is the final leaderborad can be make sure?
", "rawMarkdown": "Hi~ ๐ I want to konw when is the final leaderborad can be make sure?", "replies": [ { "id": 1311064, "postDate": "2021-05-17T06:21:56.987Z", "content": "Nearly a week since the preliminary result was post๐ค
", "rawMarkdown": "Nearly a week since the preliminary result was post๐ค" } ] }, { "id": 1307802, "postDate": "2021-05-14T17:12:56.660Z", "content": "The answers are all in https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments?scriptVersionId=58391587, check the differences between v14 and v15 referring to its comments.
", "rawMarkdown": "The answers are all in https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments?scriptVersionId=58391587, check the differences between v14 and v15 referring to its comments.", "votes": 1 }, { "id": 1306845, "postDate": "2021-05-14T05:46:42.090Z", "content": "For the future and any that experienced 0 private scores regardless of using a forked notebook or other issues to do with silent fails in notebooks see this post -
\nHow to know if your private submissions will fail when public scored and succeeded
\nBut I will add for Kaggle for going forward, there has to be something done about public notebooks in the final week or 2 weeks of a competition. For any that managed to fork v15 changed in the final few days (3-4) and used that to get their non zero private score 0.9450+ and others that did not and got 0 what is the point in that?
\nAnd reruns and interventions do happen Rerunning the reruns
", "rawMarkdown": "For the future and any that experienced 0 private scores regardless of using a forked notebook or other issues to do with silent fails in notebooks see this post -\n\n[How to know if your private submissions will fail when public scored and succeeded](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238751)\n\nBut I will add for Kaggle for going forward, there has to be something done about public notebooks in the final week or 2 weeks of a competition. For any that managed to fork v15 changed in the final few days (3-4) and used that to get their non zero private score 0.9450+ and others that did not and got 0 what is the point in that? \n\nAnd reruns and interventions do happen [Rerunning the reruns](https://www.kaggle.com/c/nfl-big-data-bowl-2020/discussion/120142)", "votes": 1 }, { "id": 1307104, "postDate": "2021-05-14T08:52:05.690Z", "content": "\n\nThis investigation determined that these were caused by unhandled errors in submission code, and the impact exacerbated by a large number of forks of a public notebook.
\n
@philculliton
\nCheck this notebook
\nhttps://www.kaggle.com/sakvaua/testtifffilesubmit?scriptVersionId=62831320
\nI added the error handling code and it still results in 0 private submission. So this is not about the unhandled errors.
WOW! Even more. This happens only on private test run. If you run the notebook on the public test it works as expected. Raised errors are handled correctly and halt script execution.
", "rawMarkdown": "WOW! Even more. This happens only on private test run. If you run the notebook on the public test it works as expected. Raised errors are handled correctly and halt script execution." }, { "id": 1307501, "postDate": "2021-05-14T13:39:30.830Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1307522, "postDate": "2021-05-14T13:49:06.337Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1307573, "postDate": "2021-05-14T14:27:57.343Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1308579, "postDate": "2021-05-15T10:10:08.367Z", "rawMarkdown": "", "votes": -1, "isDeleted": true, "replies": [ { "id": 1308603, "postDate": "2021-05-15T10:31:45.563Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1307019, "postDate": "2021-05-14T07:52:02.207Z", "rawMarkdown": "", "votes": -1, "isDeleted": true, "replies": [ { "id": 1307061, "postDate": "2021-05-14T08:19:36.853Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1307228, "postDate": "2021-05-14T10:26:35.010Z", "content": "It is just unfair, isn't it? Maybe you should ask official to hide your \"NOT VERY GOOD RESULT\".
", "rawMarkdown": "It is just unfair, isn't it? Maybe you should ask official to hide your \"NOT VERY GOOD RESULT\"." }, { "id": 1307240, "postDate": "2021-05-14T10:33:50.510Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ] } ], "comments": [ { "id": 1307077, "author_name": "HengtaoTantai", "author_url": "", "post_date": "2021-05-14T08:39:03.160000", "content": "I understand and agree with the result.
\nBut I think the most important thing right now is how to know if private submissions fail when public scored and succeeded.
\nFor example, add a new class of errors, \"Private Submission Scoring Error \" to let people know private submissions result can run correctly.
\nThis will avoid similar situations again
When is this endless competition over
", "votes": 8, "replies": [] }, { "id": 1306761, "author_name": "DennisSakva", "author_url": "", "post_date": "2021-05-14T04:20:26", "content": "Phil, can you please give a justification for why notebook continues to run next cell after an unhandled exception happened? This is the only reason my notebook (which has nothing to do with the public one) scored 0? I gather all predictions in one cell and create a csv in the next cell. My assumption was that if the csv is created the previous cell finished without errors.
\nThis situation is impossible to debug without knowing such a peculiarity beforehand.
This is ridiculous. I had some with 0 private scores, others with pretty good private scores. I forgot to select my notebooks though. In other competition, I remember that top 2 scores will be chosen automatically.
", "votes": 0, "replies": [] } ] }, { "id": 1311051, "author_name": "Luya Wang", "author_url": "", "post_date": "2021-05-17T06:08:36.150000", "content": "Hi~ ๐ I want to konw when is the final leaderborad can be make sure?
", "votes": 0, "replies": [ { "id": 1311064, "author_name": "Lucius", "author_url": "", "post_date": "2021-05-17T06:21:56.987000", "content": "Nearly a week since the preliminary result was post๐ค
", "votes": 0, "replies": [] } ] }, { "id": 1307802, "author_name": "DA", "author_url": "", "post_date": "2021-05-14T17:12:56.660000", "content": "The answers are all in https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments?scriptVersionId=58391587, check the differences between v14 and v15 referring to its comments.
", "votes": 1, "replies": [] }, { "id": 1306845, "author_name": "something4kag", "author_url": "", "post_date": "2021-05-14T05:46:42.090000", "content": "For the future and any that experienced 0 private scores regardless of using a forked notebook or other issues to do with silent fails in notebooks see this post -
\nHow to know if your private submissions will fail when public scored and succeeded
\nBut I will add for Kaggle for going forward, there has to be something done about public notebooks in the final week or 2 weeks of a competition. For any that managed to fork v15 changed in the final few days (3-4) and used that to get their non zero private score 0.9450+ and others that did not and got 0 what is the point in that?
\nAnd reruns and interventions do happen Rerunning the reruns
", "votes": 1, "replies": [] }, { "id": 1307104, "author_name": "DennisSakva", "author_url": "", "post_date": "2021-05-14T08:52:05.690000", "content": "\n\nThis investigation determined that these were caused by unhandled errors in submission code, and the impact exacerbated by a large number of forks of a public notebook.
\n
@philculliton
\nCheck this notebook
\nhttps://www.kaggle.com/sakvaua/testtifffilesubmit?scriptVersionId=62831320
\nI added the error handling code and it still results in 0 private submission. So this is not about the unhandled errors.
WOW! Even more. This happens only on private test run. If you run the notebook on the public test it works as expected. Raised errors are handled correctly and halt script execution.
", "votes": 0, "replies": [] }, { "id": 1307501, "author_name": "", "author_url": "", "post_date": "2021-05-14T13:39:30.830000", "content": "", "votes": 0, "replies": [] }, { "id": 1307522, "author_name": "", "author_url": "", "post_date": "2021-05-14T13:49:06.337000", "content": "", "votes": 0, "replies": [] }, { "id": 1307573, "author_name": "", "author_url": "", "post_date": "2021-05-14T14:27:57.343000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1308579, "author_name": "", "author_url": "", "post_date": "2021-05-15T10:10:08.367000", "content": "", "votes": -1, "replies": [ { "id": 1308603, "author_name": "", "author_url": "", "post_date": "2021-05-15T10:31:45.563000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1307019, "author_name": "", "author_url": "", "post_date": "2021-05-14T07:52:02.207000", "content": "", "votes": -1, "replies": [ { "id": 1307061, "author_name": "", "author_url": "", "post_date": "2021-05-14T08:19:36.853000", "content": "", "votes": 0, "replies": [] }, { "id": 1307228, "author_name": "Lucius", "author_url": "", "post_date": "2021-05-14T10:26:35.010000", "content": "It is just unfair, isn't it? Maybe you should ask official to hide your \"NOT VERY GOOD RESULT\".
", "votes": 0, "replies": [] }, { "id": 1307240, "author_name": "", "author_url": "", "post_date": "2021-05-14T10:33:50.510000", "content": "", "votes": -1, "replies": [] } ] } ], "raw_markdown_by_id": { "1306482": "We have read your feedback and investigated the high incidence of zero scores on the private leaderboard. This investigation determined that these were caused by unhandled errors in submission code, and the impact exacerbated by a large number of forks of a public notebook. We sympathize and never want to see this type of issue happen, but we also must uphold a standard of fairness and impartiality. This means we can not intervene to selectively fix issues with submitted code.\n\nWe will perform our normal post-competition rules checks, but the scores and rankings will largely stand as they are. Going forward, we urge all code-competition participants to adopt best practices with respect to handling errors and validating outputs, some of which are discussed at https://www.kaggle.com/code-competition-debugging. Thank you for your participation and understanding.", "1307077": "I understand and agree with the result.\nBut I think the most important thing right now is how to know if private submissions fail when public scored and succeeded.\nFor example, add a new class of errors, \"Private Submission Scoring Error \" to let people know private submissions result can run correctly.\nThis will avoid similar situations again\n", "1306800": "When is this endless competition over", "1306761": "Phil, can you please give a justification for why notebook continues to run next cell after an unhandled exception happened? This is the only reason my notebook (which has nothing to do with the public one) scored 0? I gather all predictions in one cell and create a csv in the next cell. My assumption was that if the csv is created the previous cell finished without errors.\nThis situation is impossible to debug without knowing such a peculiarity beforehand.", "1311051": "Hi~ ๐ I want to konw when is the final leaderborad can be make sure?", "1307802": "The answers are all in https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments?scriptVersionId=58391587, check the differences between v14 and v15 referring to its comments.", "1306845": "For the future and any that experienced 0 private scores regardless of using a forked notebook or other issues to do with silent fails in notebooks see this post -\n\n[How to know if your private submissions will fail when public scored and succeeded](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238751)\n\nBut I will add for Kaggle for going forward, there has to be something done about public notebooks in the final week or 2 weeks of a competition. For any that managed to fork v15 changed in the final few days (3-4) and used that to get their non zero private score 0.9450+ and others that did not and got 0 what is the point in that? \n\nAnd reruns and interventions do happen [Rerunning the reruns](https://www.kaggle.com/c/nfl-big-data-bowl-2020/discussion/120142)", "1307104": "> This investigation determined that these were caused by unhandled errors in submission code, and the impact exacerbated by a large number of forks of a public notebook. \n\n@philculliton \nCheck this notebook\nhttps://www.kaggle.com/sakvaua/testtifffilesubmit?scriptVersionId=62831320\nI added the error handling code and it still results in 0 private submission. So this is not about the unhandled errors.", "1308579": "", "1307019": "" } }