Hi Guys..
\nIt's very hard for all of us who got ZEROS as the final private LB score. After the effort and time spent by 600+ kagglers, if we don't stand together on this, it all goes to waste.
\nIn one of the comments, Kaggle staff stated that ppl might have skipped private LB deliberately. I believe that there's some bug or issue.
\nI request all kagglers to kindly give your support so the bug or issue gets fixed ASAP.
\nThis would prevent such issues from occurring in the future.
I request Kaggle, the organizers and other kagglers to support us all.
", "messages": [ { "id": 1301556, "postDate": "2021-05-11T06:20:14.980Z", "content": "Hi Guys..
\nIt's very hard for all of us who got ZEROS as the final private LB score. After the effort and time spent by 600+ kagglers, if we don't stand together on this, it all goes to waste.
\nIn one of the comments, Kaggle staff stated that ppl might have skipped private LB deliberately. I believe that there's some bug or issue.
\nI request all kagglers to kindly give your support so the bug or issue gets fixed ASAP.
\nThis would prevent such issues from occurring in the future.
I request Kaggle, the organizers and other kagglers to support us all.
", "rawMarkdown": "Hi Guys..\n\nIt's very hard for all of us who got ZEROS as the final private LB score. After the effort and time spent by 600+ kagglers, if we don't stand together on this, it all goes to waste.\n\nIn one of the comments, Kaggle staff stated that ppl might have skipped private LB deliberately. I believe that there's some bug or issue.\n\nI request all kagglers to kindly give your support so the bug or issue gets fixed ASAP.\nThis would prevent such issues from occurring in the future.\n\nI request Kaggle, the organizers and other kagglers to support us all.\n", "votes": 22 }, { "id": 1301784, "postDate": "2021-05-11T08:42:31.320Z", "content": "I don't think there is any problem on Kaggle side, honestly. My friendly advice to all who think otherwise - double-/triple- check your inference code. Even better - make in public, so community can take a look and confirm presence/absence of the errors in your zero-scoring submissions.
\nAlthough one of our final submissions also got zero score, it became obvious there was a bug in our code.
The most common reason of getting zero score is a misuse of sample_submission.csv at the inference time. You should never even rely on set if image ids from this source when dealing with code competitions.
It hard to accept, that three month of efforts wasted for nothing, I personally came through the same frustration at DeepFakes challenge. Just deal with it and carry on to the next challenge.
\nHappy kaggling!
", "rawMarkdown": "I don't think there is any problem on Kaggle side, honestly. My friendly advice to all who think otherwise - double-/triple- check your inference code. Even better - make in public, so community can take a look and confirm presence/absence of the errors in your zero-scoring submissions.\nAlthough one of our final submissions also got zero score, it became obvious there was a bug in our code.\n\nThe most common reason of getting zero score is a misuse of `sample_submission.csv` at the inference time. You should never even rely on set if image ids from this source when dealing with code competitions.\n\nIt hard to accept, that three month of efforts wasted for nothing, I personally came through the same frustration at DeepFakes challenge. Just deal with it and carry on to the next challenge.\n\nHappy kaggling!", "votes": 15, "replies": [ { "id": 1301814, "postDate": "2021-05-11T08:54:41.167Z", "content": "I have successful commits with using of sample_submission.csv:
\nfor index, row in tqdm(submission_df.iterrows(),total=len(submission_df)):
But just from one moment all submissions are zero-scored…
", "rawMarkdown": "I have successful commits with using of sample_submission.csv:\n`for index, row in tqdm(submission_df.iterrows(),total=len(submission_df)):`\n\nBut just from one moment all submissions are zero-scored...\n", "votes": 1 }, { "id": 1301841, "postDate": "2021-05-11T09:09:21.320Z", "content": "Disclaimer: below is the my understanding how the scoring works.
\nA sample submission csv remains unchanged. What is changing is a content of a test folder. On public test it's 5 images (public only) and during submission, it's get's changed to private+public.
\nTo me, the correct way is to find list of images:
# pip install pytorch-toolbelt\nfrom pytorch_toolbelt.utils import fs\n\n# Gives you all .tiff files in the directory\ntest_images = fs.find_in_dir_with_ext(test_dir, \".tiff\")\nSo, since you load submission_df from sample_submission.csv you process only 5 images, regardless of how many images in test directory.
If we keep on accepting these things, I don't think we will get any long time solution. If there's no proper mechanism that lets me know whether my submission is valid or not.
\nHow do you expect me to trust that this won't happen again?
\nAtleast, if the pipeline had shown error for public score, I would have done some changes to fix it.
\nAfter spending good time & effort, seeing your score as 0 is heartbreaking.
Kaggle has to give answers or solution to this issue.
", "rawMarkdown": "If we keep on accepting these things, I don't think we will get any long time solution. If there's no proper mechanism that lets me know whether my submission is valid or not. \n\nHow do you expect me to trust that this won't happen again?\n\nAtleast, if the pipeline had shown error for public score, I would have done some changes to fix it.\nAfter spending good time & effort, seeing your score as 0 is heartbreaking. \n\nKaggle has to give answers or solution to this issue.", "votes": -1 }, { "id": 1301872, "postDate": "2021-05-11T09:27:43.320Z", "content": "How about running the code locally, using as a test folder a dummy one with different images and check that produced csv contains the same number of rows as the number of images in folder?
", "rawMarkdown": "How about running the code locally, using as a test folder a dummy one with different images and check that produced csv contains the same number of rows as the number of images in folder?", "votes": 3 }, { "id": 1301921, "postDate": "2021-05-11T10:10:37.060Z", "content": "I understand what you mean but want clarify what I wrote above: I have submissions with >0.9 for private with iterating over sample_submission.csv but from one moment (month ago) all other submission have zero scores.
", "rawMarkdown": "I understand what you mean but want clarify what I wrote above: I have submissions with >0.9 for private with iterating over sample_submission.csv but from one moment (month ago) all other submission have zero scores.", "votes": 5 }, { "id": 1302143, "postDate": "2021-05-11T12:06:39.743Z", "content": "Well, this notebook uses sample_submission and scored well above 0 (0.950). So there must be something else.
\nhttps://www.kaggle.com/shujun717/hubmap-3rd-place-inference
Also a reply from Kaggle
\nPhil CullitonKaggle Staff • 4 minutes ago • Options • Report • Reply
\nHi, thanks for posting this. All test image IDs were present in the private / hidden test set's sample_submission.csv file. The CSV does get swapped during private testing.
Fully agree, There is nothing wrong the Kaggle system…. and there is nothing wrong with that majority of people clone a kernel.
\nBut … if you clone it… well then the idea is to fully learn from it, analyze it and make sure that you know /experience the things that can go wrong. Á simple round of debugging the code and putting in the necessary exception handling will give some very usefull insights.
\nDon't blame Kaggle or the original writers of the kernel containing the issue.
", "rawMarkdown": "Fully agree, There is nothing wrong the Kaggle system.... and there is nothing wrong with that majority of people clone a kernel.\n\nBut ... if you clone it... well then the idea is to fully learn from it, analyze it and make sure that you know /experience the things that can go wrong. Á simple round of debugging the code and putting in the necessary exception handling will give some very usefull insights.\n\nDon't blame Kaggle or the original writers of the kernel containing the issue.", "votes": 5 }, { "id": 1302475, "postDate": "2021-05-11T15:09:53.747Z", "content": "There is something else. I used /dont used the sample_submission.csv. Zero score remains.
\nAlso, my code can predict all 20 images on the same run, without an error/ memory issues.
\nAlso, I use R notebooks, and it is supposed to work as much as python notebooks, once
\nthe submission.csv is the same from both pipelines.
The odd part is, once one have a valid public score, the private should be present as well. An error/warning should be returned so people could manage to workaround, months before deadline.
\nRight now, any configurantions i tried, results in zeroscore. I dont know much about the VM used to submission, but how hard it is to block valid submissions on public leaderboard if on private it fails?
I did not clone any kernel and I doubt that the kernel you're referring to was cloned by hundreds of users. There's something more going on there. And authors of the kernel didn't know what was not working and replaced one image library with another \"just in case\". This is hardly acceptable.
", "rawMarkdown": "I did not clone any kernel and I doubt that the kernel you're referring to was cloned by hundreds of users. There's something more going on there. And authors of the kernel didn't know what was not working and replaced one image library with another \"just in case\". This is hardly acceptable." }, { "id": 1304610, "postDate": "2021-05-12T18:24:56.383Z", "content": "this has been an ongoing issue for some time, but the wide adoption of a public notebook resulted in aggravates outcomes. BTW, almost 400 teams had zeros on the public LB as well. So in reality, there is only approx 200 folks who may have fallen victims to this issue
", "rawMarkdown": "this has been an ongoing issue for some time, but the wide adoption of a public notebook resulted in aggravates outcomes. BTW, almost 400 teams had zeros on the public LB as well. So in reality, there is only approx 200 folks who may have fallen victims to this issue", "votes": 1 } ] }, { "id": 1302596, "postDate": "2021-05-11T16:10:24.063Z", "content": "moral of the story: do not fork public notebooks and submit. Read the code, comments, and debug.
\nKaggle system works perfectly, the proof is that anyone with valid code and predictions on all test images got a score.
\nFeels bad, but it is what it is (imho).
Well said! Lol how are people downvoting this
", "rawMarkdown": "Well said! Lol how are people downvoting this" }, { "id": 1305562, "postDate": "2021-05-13T11:14:37.083Z", "content": "Look, this has nothing to do with some public notebook. I never used it and had this problem. Kaggle continues running notebooks cells after an exception was thrown by totally ignoring it. This is hardly a case of \"works perfectly\". How do you see a debugging process in this case? The notion that not everyone has this problem therefore it's not a problem or that there is some code that works correctly therefore it's not a problem is deeply flawed.
", "rawMarkdown": "Look, this has nothing to do with some public notebook. I never used it and had this problem. Kaggle continues running notebooks cells after an exception was thrown by totally ignoring it. This is hardly a case of \"works perfectly\". How do you see a debugging process in this case? The notion that not everyone has this problem therefore it's not a problem or that there is some code that works correctly therefore it's not a problem is deeply flawed.", "votes": 3 }, { "id": 1305622, "postDate": "2021-05-13T11:52:58.380Z", "content": "Hi @sakvaua , probably I didn't explain properly.
\nIt is related with public notebooks in most of the cases as you can read here: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238131
\nSolutions based on that notebook v14 failed. I guess some people just forked and run or modified briefly without checking if the notebook was running properly on the private dataset. Actually, this problem was discussed on the comments of the kernel BEFORE the end of the competition.
Here there are some tips for avoiding that problem: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238150
\nI personally recommend to check sample submission length, number of images in test/ and if the all images are loaded properly, during the private test run.
\nIf something in this process fails, you can raise an exception yourself. For instance, if the code cannot read all images, you don't save submision.csv, thus it will return an error in the dashboard saying \"Submission not found\". If not exceptions catched and/or all images were loaded, you save your submission and kaggle will show a score in the dashboard. These debugging methods have been used widely and are public.
\nHow do I know if my code reads all the images even if fails and continues running? You can check the shapes or catch exceptions.
\nAlso is easy to see if the notebook runs fast or not, it should take X time to run on public test, if you noticed that the notebook was taking X mins (you can see this at the submission dashboard) it is a bad indicator.
IMHO, the notion that there is a problem that affects certain people and I think I could not debug it, therefore, it's kaggle's problem and they have to solve it. So far, is not a valid argument either. Moreover, we all know code competitions difficulties when joining.
\nAlso you are right, \"works perfectly\" it is just an expression. Kaggle has many issues and could provide more feedback at the submission dashboard.
\nBest,
\nPD: I would not expect an answer from kaggle team, this is not the first time.
", "rawMarkdown": "Hi @sakvaua , probably I didn't explain properly. \n\nIt is related with public notebooks in most of the cases as you can read here: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238131\nSolutions based on that notebook v14 failed. I guess some people just forked and run or modified briefly without checking if the notebook was running properly on the private dataset. Actually, this problem was discussed on the comments of the kernel BEFORE the end of the competition.\n\nHere there are some tips for avoiding that problem: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238150\nI personally recommend to check sample submission length, number of images in test/ and if the all images are loaded properly, during the private test run.\nIf something in this process fails, you can raise an exception yourself. For instance, if the code cannot read all images, you don't save ```submision.csv```, thus it will return an error in the dashboard saying \"Submission not found\". If not exceptions catched and/or all images were loaded, you save your submission and kaggle will show a score in the dashboard. These debugging methods have been used widely and are public.\nHow do I know if my code reads all the images even if fails and continues running? You can check the shapes or catch exceptions.\nAlso is easy to see if the notebook runs fast or not, it should take X time to run on public test, if you noticed that the notebook was taking X mins (you can see this at the submission dashboard) it is a bad indicator.\n\nIMHO, the notion that there is a problem that affects certain people and I think I could not debug it, therefore, it's kaggle's problem and they have to solve it. So far, is not a valid argument either. Moreover, we all know code competitions difficulties when joining. \n\nAlso you are right, \"works perfectly\" it is just an expression. Kaggle has many issues and could provide more feedback at the submission dashboard.\n\nBest,\n\nPD: I would not expect an answer from kaggle team, this is not the first time.", "votes": 2 }, { "id": 1305663, "postDate": "2021-05-13T12:18:42.873Z", "content": "Now, after the competition ended and I saw that my best performing notebooks scored 0 I know that there was a problem. And I was able to debug it with trial and error. See here:
\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238534
\nBut this is all post-mortem. There's no way I could have known that beforehand. My scripts take many hours to finish and I don't monitor how much time they take. I, personally, ran them overnight because of that.
\nBut I insist that it is a Kaggle problem because ignoring all exceptions is not an expected behaviour from any high-level language that I know of except for assembler (which is not high end and we are not coding in assembler :) )
\nThe expected behaviour is to halt script execution when an unhandled exception happens. This is the expected behaviour in python, in Jupyter notebook and pretty much everything else. In the case of Kaggle, the script continues to run and generates a completely valid submission file for the public part, but not for the private.
I see. I agree that Kaggle code competitions must improve! They are 1yr old (I think the 1st one was launched 1 year ago) and Kaggle still improving. I proposed on \"product feedback\" to add more information in the submission dashboard like: running times, exceptions captured, etc. I have to calculate the running time for my subs by myself haha. These debugging problems have been around since the beginning of this kind of competition. What I meant before is that it is not impossible to score or debug (at a certain level), many kagglers have done it in this comp. IMHO the reality is that these problems are residual, yet they deserve an answer/solution as the next one:
\nI see no valid reason to continue running notebook cells after an unhandled exception happened. The script should abort and not proceed with the creation of the csv file using incomplete data.
About the execution after exceptions, probably it is better to run the notebook and return something rather than having the notebook hanging on the cloud, consuming resources… also returning exceptions in the submission dashboard must be tricky. In general, the number and variety of exceptions that the system should catch and report in the dashboard is important, if you reduce it to the most common ones (OOM, etc.), the message would be less informative… but hey! at least there will be a message.
\nI agree Kaggle should solve that particular problem in the near future, since I like very much code competitions for other reasons. However, they already have a lot of work with the anti-cheating system and leak detection haha.
\nBest and congrats for your achievements despite that problem.
", "rawMarkdown": "I see. I agree that Kaggle code competitions must improve! They are 1yr old (I think the 1st one was launched 1 year ago) and Kaggle still improving. I proposed on \"product feedback\" to add more information in the submission dashboard like: running times, exceptions captured, etc. I have to calculate the running time for my subs by myself haha. These debugging problems have been around since the beginning of this kind of competition. What I meant before is that it is not impossible to score or debug (at a certain level), many kagglers have done it in this comp. IMHO the reality is that these problems are residual, yet they deserve an answer/solution as the next one:\n\n`I see no valid reason to continue running notebook cells after an unhandled exception happened. The script should abort and not proceed with the creation of the csv file using incomplete data.`\n\nAbout the execution after exceptions, probably it is better to run the notebook and return something rather than having the notebook hanging on the cloud, consuming resources... also returning exceptions in the submission dashboard must be tricky. In general, the number and variety of exceptions that the system should catch and report in the dashboard is important, if you reduce it to the most common ones (OOM, etc.), the message would be less informative... but hey! at least there will be a message.\n\nI agree Kaggle should solve that particular problem in the near future, since I like very much code competitions for other reasons. However, they already have a lot of work with the anti-cheating system and leak detection haha. \n\nBest and congrats for your achievements despite that problem.\n", "votes": 1 } ] }, { "id": 1304724, "postDate": "2021-05-12T20:01:33.057Z", "content": "I had my pipeline, but I got the same problem. I replaced the read tiff function with the rasterio version, but the private LB score stayed 0. The funny thing was that the submission was run 2x more time as the save on public LB, so I had not even thought it could be a problem. But the lesson learned, I have to double-check everything.
\nIn these competitions, where the computing resources so much a bottleneck, it would be great if it would not be a code competition. Or from the host side, that would be kind, to not put bigger images in the private LB than to the public.
see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments v15 changes 4 days ago to use rasterio instead of tifffile due to potential large image in private test.
", "rawMarkdown": "see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments v15 changes 4 days ago to use rasterio instead of tifffile due to potential large image in private test. \n", "votes": 3, "replies": [ { "id": 1302151, "postDate": "2021-05-11T12:11:02.073Z", "content": "If what's written in the post is true - it's simply undebuggable. There's no way in hell we can be sure if our submission scored anything on private or not.
", "rawMarkdown": "If what's written in the post is true - it's simply undebuggable. There's no way in hell we can be sure if our submission scored anything on private or not.", "votes": 4 }, { "id": 1302211, "postDate": "2021-05-11T12:44:24.047Z", "content": "Fork the notebook, v15 and chg to your model and see if it scores other than 0. I tested a model and it definitely does. So using rasterio etc. works. However, having a public test set with at least similar sizes would have been helpful too.
", "rawMarkdown": "Fork the notebook, v15 and chg to your model and see if it scores other than 0. I tested a model and it definitely does. So using rasterio etc. works. However, having a public test set with at least similar sizes would have been helpful too. " }, { "id": 1302225, "postDate": "2021-05-11T12:51:53.183Z", "content": "I have a very different inference pipeline. Using someone elses pipeline after the end of the competition is hardly a solution to a rather massive problem of having a perfectly valid public leaderboard score and at the same time 0 private score. It shouldn't fail silently.
", "rawMarkdown": "I have a very different inference pipeline. Using someone elses pipeline after the end of the competition is hardly a solution to a rather massive problem of having a perfectly valid public leaderboard score and at the same time 0 private score. It shouldn't fail silently.", "votes": 2 }, { "id": 1302255, "postDate": "2021-05-11T13:06:10.120Z", "content": "Just an example notebook if you did not use rasterio for final sub to allow you to see if that helped. The issue appears to be an image in the private test that was much larger than anything seen in public test. Public test and LB were fine and you got a submission csv. And if you used sample submission csv and filled in the predictions, you could end up with just blanks if the image failed in the private test run.
\nIn terms of failing silently, I think some picked up that the submission run was just as quick as commit. Then investigating came up with changing to rasterio. If you had various inference strategies you may not have noticed. It is all a bit suss tho.
Believe that it's just bug and should be solved with the time…
", "rawMarkdown": "Believe that it's just bug and should be solved with the time...", "votes": 2 }, { "id": 1303757, "postDate": "2021-05-12T08:15:46.797Z", "content": "So any reaction from organizers and kaggle staff?(
", "rawMarkdown": "So any reaction from organizers and kaggle staff?( ", "votes": 2, "replies": [ { "id": 1304428, "postDate": "2021-05-12T15:59:16.487Z", "content": "desperately waiting for any response from kaggle :(
", "rawMarkdown": "desperately waiting for any response from kaggle :(", "votes": 1 } ] }, { "id": 1302306, "postDate": "2021-05-11T13:40:32.583Z", "content": "Dear Organizers,
\nPlease note that the exact requirements for the private test scoring to be non-zero is specified nowhere in the competition page or in any of the official discussion threads.
\nFrom other comments, one possible cause for the zero value private submission could be the use of Tifffle library instead of raterio but, this does not result in any kind of error or warning which could possibly help in identifying the failure during submission.
\nThe only requirements defined for the for the submission csv file is following the right format and I believe the public submission would have failed if we had not followed that. The fact that these instructions were nowhere on the platform which de-valuated many months of effort is clearly discouragement for participants who are beginners in Code Competitions.
\nIt is only fair for the submissions of participants who spend months working on this, to be fixed some way to re-run so the proper scores are reflected. Otherwise, I believe many amazing solutions would be lost just because of hidden OOM errors and not picking the right files from a hidden directory, which is really not the point of a research competition.
", "rawMarkdown": "Dear Organizers,\n\nPlease note that the exact requirements for the private test scoring to be non-zero is specified nowhere in the competition page or in any of the official discussion threads.\n\nFrom other comments, one possible cause for the zero value private submission could be the use of Tifffle library instead of raterio but, this does not result in any kind of error or warning which could possibly help in identifying the failure during submission.\n\nThe only requirements defined for the for the submission csv file is following the right format and I believe the public submission would have failed if we had not followed that. **The fact that these instructions were nowhere on the platform which de-valuated many months of effort is clearly discouragement for participants who are beginners in Code Competitions.**\n\n**It is only fair for the submissions of participants who spend months working on this, to be fixed some way to re-run so the proper scores are reflected. Otherwise, I believe many amazing solutions would be lost just because of hidden OOM errors and not picking the right files from a hidden directory, which is really not the point of a research competition.**\n\n", "votes": -2, "replies": [ { "id": 1302615, "postDate": "2021-05-11T16:21:39.940Z", "content": "Just a though…
\nI can certainly understand your frustration, however, the rules of the game have been the same for some time: you write a program that takes input from a specified location and generate output in a specific format. The job that runs your code will do its best to warn you if any errors are encountered (albeit could use some improvements). A score of 0 is not an error, and if your program does \"swallow\" an exception for whatever reason, the organizers cannot do much. Or can they?
I think the host of the competition doesnt care shit. It is 100% the host's problem with 600 0.000. Where is the warning?
", "rawMarkdown": "I think the host of the competition doesnt care shit. It is 100% the host's problem with 600 0.000. Where is the warning? ", "votes": -6 } ], "comments": [ { "id": 1301784, "author_name": "Eugene Khvedchenya", "author_url": "", "post_date": "2021-05-11T08:42:31.320000", "content": "I don't think there is any problem on Kaggle side, honestly. My friendly advice to all who think otherwise - double-/triple- check your inference code. Even better - make in public, so community can take a look and confirm presence/absence of the errors in your zero-scoring submissions.
\nAlthough one of our final submissions also got zero score, it became obvious there was a bug in our code.
The most common reason of getting zero score is a misuse of sample_submission.csv at the inference time. You should never even rely on set if image ids from this source when dealing with code competitions.
It hard to accept, that three month of efforts wasted for nothing, I personally came through the same frustration at DeepFakes challenge. Just deal with it and carry on to the next challenge.
\nHappy kaggling!
", "votes": 15, "replies": [ { "id": 1301814, "author_name": "anthony", "author_url": "", "post_date": "2021-05-11T08:54:41.167000", "content": "I have successful commits with using of sample_submission.csv:
\nfor index, row in tqdm(submission_df.iterrows(),total=len(submission_df)):
But just from one moment all submissions are zero-scored…
", "votes": 1, "replies": [] }, { "id": 1301841, "author_name": "Eugene Khvedchenya", "author_url": "", "post_date": "2021-05-11T09:09:21.320000", "content": "Disclaimer: below is the my understanding how the scoring works.
\nA sample submission csv remains unchanged. What is changing is a content of a test folder. On public test it's 5 images (public only) and during submission, it's get's changed to private+public.
\nTo me, the correct way is to find list of images:
# pip install pytorch-toolbelt\nfrom pytorch_toolbelt.utils import fs\n\n# Gives you all .tiff files in the directory\ntest_images = fs.find_in_dir_with_ext(test_dir, \".tiff\")\nSo, since you load submission_df from sample_submission.csv you process only 5 images, regardless of how many images in test directory.
If we keep on accepting these things, I don't think we will get any long time solution. If there's no proper mechanism that lets me know whether my submission is valid or not.
\nHow do you expect me to trust that this won't happen again?
\nAtleast, if the pipeline had shown error for public score, I would have done some changes to fix it.
\nAfter spending good time & effort, seeing your score as 0 is heartbreaking.
Kaggle has to give answers or solution to this issue.
", "votes": -1, "replies": [] }, { "id": 1301872, "author_name": "Eugene Khvedchenya", "author_url": "", "post_date": "2021-05-11T09:27:43.320000", "content": "How about running the code locally, using as a test folder a dummy one with different images and check that produced csv contains the same number of rows as the number of images in folder?
", "votes": 3, "replies": [] }, { "id": 1301921, "author_name": "anthony", "author_url": "", "post_date": "2021-05-11T10:10:37.060000", "content": "I understand what you mean but want clarify what I wrote above: I have submissions with >0.9 for private with iterating over sample_submission.csv but from one moment (month ago) all other submission have zero scores.
", "votes": 5, "replies": [] }, { "id": 1302143, "author_name": "DennisSakva", "author_url": "", "post_date": "2021-05-11T12:06:39.743000", "content": "Well, this notebook uses sample_submission and scored well above 0 (0.950). So there must be something else.
\nhttps://www.kaggle.com/shujun717/hubmap-3rd-place-inference
Also a reply from Kaggle
\nPhil CullitonKaggle Staff • 4 minutes ago • Options • Report • Reply
\nHi, thanks for posting this. All test image IDs were present in the private / hidden test set's sample_submission.csv file. The CSV does get swapped during private testing.
Fully agree, There is nothing wrong the Kaggle system…. and there is nothing wrong with that majority of people clone a kernel.
\nBut … if you clone it… well then the idea is to fully learn from it, analyze it and make sure that you know /experience the things that can go wrong. Á simple round of debugging the code and putting in the necessary exception handling will give some very usefull insights.
\nDon't blame Kaggle or the original writers of the kernel containing the issue.
", "votes": 5, "replies": [] }, { "id": 1302475, "author_name": "rpsantosa_kaggle", "author_url": "", "post_date": "2021-05-11T15:09:53.747000", "content": "There is something else. I used /dont used the sample_submission.csv. Zero score remains.
\nAlso, my code can predict all 20 images on the same run, without an error/ memory issues.
\nAlso, I use R notebooks, and it is supposed to work as much as python notebooks, once
\nthe submission.csv is the same from both pipelines.
The odd part is, once one have a valid public score, the private should be present as well. An error/warning should be returned so people could manage to workaround, months before deadline.
\nRight now, any configurantions i tried, results in zeroscore. I dont know much about the VM used to submission, but how hard it is to block valid submissions on public leaderboard if on private it fails?
I did not clone any kernel and I doubt that the kernel you're referring to was cloned by hundreds of users. There's something more going on there. And authors of the kernel didn't know what was not working and replaced one image library with another \"just in case\". This is hardly acceptable.
", "votes": 0, "replies": [] }, { "id": 1304610, "author_name": "andras", "author_url": "", "post_date": "2021-05-12T18:24:56.383000", "content": "this has been an ongoing issue for some time, but the wide adoption of a public notebook resulted in aggravates outcomes. BTW, almost 400 teams had zeros on the public LB as well. So in reality, there is only approx 200 folks who may have fallen victims to this issue
", "votes": 1, "replies": [] } ] }, { "id": 1302596, "author_name": "Nanashi", "author_url": "", "post_date": "2021-05-11T16:10:24.063000", "content": "moral of the story: do not fork public notebooks and submit. Read the code, comments, and debug.
\nKaggle system works perfectly, the proof is that anyone with valid code and predictions on all test images got a score.
\nFeels bad, but it is what it is (imho).
Well said! Lol how are people downvoting this
", "votes": 0, "replies": [] }, { "id": 1305562, "author_name": "DennisSakva", "author_url": "", "post_date": "2021-05-13T11:14:37.083000", "content": "Look, this has nothing to do with some public notebook. I never used it and had this problem. Kaggle continues running notebooks cells after an exception was thrown by totally ignoring it. This is hardly a case of \"works perfectly\". How do you see a debugging process in this case? The notion that not everyone has this problem therefore it's not a problem or that there is some code that works correctly therefore it's not a problem is deeply flawed.
", "votes": 3, "replies": [] }, { "id": 1305622, "author_name": "Nanashi", "author_url": "", "post_date": "2021-05-13T11:52:58.380000", "content": "Hi @sakvaua , probably I didn't explain properly.
\nIt is related with public notebooks in most of the cases as you can read here: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238131
\nSolutions based on that notebook v14 failed. I guess some people just forked and run or modified briefly without checking if the notebook was running properly on the private dataset. Actually, this problem was discussed on the comments of the kernel BEFORE the end of the competition.
Here there are some tips for avoiding that problem: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238150
\nI personally recommend to check sample submission length, number of images in test/ and if the all images are loaded properly, during the private test run.
\nIf something in this process fails, you can raise an exception yourself. For instance, if the code cannot read all images, you don't save submision.csv, thus it will return an error in the dashboard saying \"Submission not found\". If not exceptions catched and/or all images were loaded, you save your submission and kaggle will show a score in the dashboard. These debugging methods have been used widely and are public.
\nHow do I know if my code reads all the images even if fails and continues running? You can check the shapes or catch exceptions.
\nAlso is easy to see if the notebook runs fast or not, it should take X time to run on public test, if you noticed that the notebook was taking X mins (you can see this at the submission dashboard) it is a bad indicator.
IMHO, the notion that there is a problem that affects certain people and I think I could not debug it, therefore, it's kaggle's problem and they have to solve it. So far, is not a valid argument either. Moreover, we all know code competitions difficulties when joining.
\nAlso you are right, \"works perfectly\" it is just an expression. Kaggle has many issues and could provide more feedback at the submission dashboard.
\nBest,
\nPD: I would not expect an answer from kaggle team, this is not the first time.
", "votes": 2, "replies": [] }, { "id": 1305663, "author_name": "DennisSakva", "author_url": "", "post_date": "2021-05-13T12:18:42.873000", "content": "Now, after the competition ended and I saw that my best performing notebooks scored 0 I know that there was a problem. And I was able to debug it with trial and error. See here:
\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238534
\nBut this is all post-mortem. There's no way I could have known that beforehand. My scripts take many hours to finish and I don't monitor how much time they take. I, personally, ran them overnight because of that.
\nBut I insist that it is a Kaggle problem because ignoring all exceptions is not an expected behaviour from any high-level language that I know of except for assembler (which is not high end and we are not coding in assembler :) )
\nThe expected behaviour is to halt script execution when an unhandled exception happens. This is the expected behaviour in python, in Jupyter notebook and pretty much everything else. In the case of Kaggle, the script continues to run and generates a completely valid submission file for the public part, but not for the private.
I see. I agree that Kaggle code competitions must improve! They are 1yr old (I think the 1st one was launched 1 year ago) and Kaggle still improving. I proposed on \"product feedback\" to add more information in the submission dashboard like: running times, exceptions captured, etc. I have to calculate the running time for my subs by myself haha. These debugging problems have been around since the beginning of this kind of competition. What I meant before is that it is not impossible to score or debug (at a certain level), many kagglers have done it in this comp. IMHO the reality is that these problems are residual, yet they deserve an answer/solution as the next one:
\nI see no valid reason to continue running notebook cells after an unhandled exception happened. The script should abort and not proceed with the creation of the csv file using incomplete data.
About the execution after exceptions, probably it is better to run the notebook and return something rather than having the notebook hanging on the cloud, consuming resources… also returning exceptions in the submission dashboard must be tricky. In general, the number and variety of exceptions that the system should catch and report in the dashboard is important, if you reduce it to the most common ones (OOM, etc.), the message would be less informative… but hey! at least there will be a message.
\nI agree Kaggle should solve that particular problem in the near future, since I like very much code competitions for other reasons. However, they already have a lot of work with the anti-cheating system and leak detection haha.
\nBest and congrats for your achievements despite that problem.
", "votes": 1, "replies": [] } ] }, { "id": 1304724, "author_name": "Bessenyei Szilárd", "author_url": "", "post_date": "2021-05-12T20:01:33.057000", "content": "I had my pipeline, but I got the same problem. I replaced the read tiff function with the rasterio version, but the private LB score stayed 0. The funny thing was that the submission was run 2x more time as the save on public LB, so I had not even thought it could be a problem. But the lesson learned, I have to double-check everything.
\nIn these competitions, where the computing resources so much a bottleneck, it would be great if it would not be a code competition. Or from the host side, that would be kind, to not put bigger images in the private LB than to the public.
see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments v15 changes 4 days ago to use rasterio instead of tifffile due to potential large image in private test.
", "votes": 3, "replies": [ { "id": 1302151, "author_name": "DennisSakva", "author_url": "", "post_date": "2021-05-11T12:11:02.073000", "content": "If what's written in the post is true - it's simply undebuggable. There's no way in hell we can be sure if our submission scored anything on private or not.
", "votes": 4, "replies": [] }, { "id": 1302211, "author_name": "something4kag", "author_url": "", "post_date": "2021-05-11T12:44:24.047000", "content": "Fork the notebook, v15 and chg to your model and see if it scores other than 0. I tested a model and it definitely does. So using rasterio etc. works. However, having a public test set with at least similar sizes would have been helpful too.
", "votes": 0, "replies": [] }, { "id": 1302225, "author_name": "DennisSakva", "author_url": "", "post_date": "2021-05-11T12:51:53.183000", "content": "I have a very different inference pipeline. Using someone elses pipeline after the end of the competition is hardly a solution to a rather massive problem of having a perfectly valid public leaderboard score and at the same time 0 private score. It shouldn't fail silently.
", "votes": 2, "replies": [] }, { "id": 1302255, "author_name": "something4kag", "author_url": "", "post_date": "2021-05-11T13:06:10.120000", "content": "Just an example notebook if you did not use rasterio for final sub to allow you to see if that helped. The issue appears to be an image in the private test that was much larger than anything seen in public test. Public test and LB were fine and you got a submission csv. And if you used sample submission csv and filled in the predictions, you could end up with just blanks if the image failed in the private test run.
\nIn terms of failing silently, I think some picked up that the submission run was just as quick as commit. Then investigating came up with changing to rasterio. If you had various inference strategies you may not have noticed. It is all a bit suss tho.
Believe that it's just bug and should be solved with the time…
", "votes": 2, "replies": [] }, { "id": 1303757, "author_name": "anthony", "author_url": "", "post_date": "2021-05-12T08:15:46.797000", "content": "So any reaction from organizers and kaggle staff?(
", "votes": 2, "replies": [ { "id": 1304428, "author_name": "Swikwislkdjc", "author_url": "", "post_date": "2021-05-12T15:59:16.487000", "content": "desperately waiting for any response from kaggle :(
", "votes": 1, "replies": [] } ] }, { "id": 1302306, "author_name": "Sreevishnu Damodaran", "author_url": "", "post_date": "2021-05-11T13:40:32.583000", "content": "Dear Organizers,
\nPlease note that the exact requirements for the private test scoring to be non-zero is specified nowhere in the competition page or in any of the official discussion threads.
\nFrom other comments, one possible cause for the zero value private submission could be the use of Tifffle library instead of raterio but, this does not result in any kind of error or warning which could possibly help in identifying the failure during submission.
\nThe only requirements defined for the for the submission csv file is following the right format and I believe the public submission would have failed if we had not followed that. The fact that these instructions were nowhere on the platform which de-valuated many months of effort is clearly discouragement for participants who are beginners in Code Competitions.
\nIt is only fair for the submissions of participants who spend months working on this, to be fixed some way to re-run so the proper scores are reflected. Otherwise, I believe many amazing solutions would be lost just because of hidden OOM errors and not picking the right files from a hidden directory, which is really not the point of a research competition.
", "votes": -2, "replies": [ { "id": 1302615, "author_name": "andras", "author_url": "", "post_date": "2021-05-11T16:21:39.940000", "content": "Just a though…
\nI can certainly understand your frustration, however, the rules of the game have been the same for some time: you write a program that takes input from a specified location and generate output in a specific format. The job that runs your code will do its best to warn you if any errors are encountered (albeit could use some improvements). A score of 0 is not an error, and if your program does \"swallow\" an exception for whatever reason, the organizers cannot do much. Or can they?
I think the host of the competition doesnt care shit. It is 100% the host's problem with 600 0.000. Where is the warning?
", "votes": -6, "replies": [] } ], "raw_markdown_by_id": { "1301556": "Hi Guys..\n\nIt's very hard for all of us who got ZEROS as the final private LB score. After the effort and time spent by 600+ kagglers, if we don't stand together on this, it all goes to waste.\n\nIn one of the comments, Kaggle staff stated that ppl might have skipped private LB deliberately. I believe that there's some bug or issue.\n\nI request all kagglers to kindly give your support so the bug or issue gets fixed ASAP.\nThis would prevent such issues from occurring in the future.\n\nI request Kaggle, the organizers and other kagglers to support us all.\n", "1301784": "I don't think there is any problem on Kaggle side, honestly. My friendly advice to all who think otherwise - double-/triple- check your inference code. Even better - make in public, so community can take a look and confirm presence/absence of the errors in your zero-scoring submissions.\nAlthough one of our final submissions also got zero score, it became obvious there was a bug in our code.\n\nThe most common reason of getting zero score is a misuse of `sample_submission.csv` at the inference time. You should never even rely on set if image ids from this source when dealing with code competitions.\n\nIt hard to accept, that three month of efforts wasted for nothing, I personally came through the same frustration at DeepFakes challenge. Just deal with it and carry on to the next challenge.\n\nHappy kaggling!", "1302596": "moral of the story: do not fork public notebooks and submit. Read the code, comments, and debug.\nKaggle system works perfectly, the proof is that anyone with valid code and predictions on all test images got a score.\nFeels bad, but it is what it is (imho).", "1304724": "I had my pipeline, but I got the same problem. I replaced the read tiff function with the rasterio version, but the private LB score stayed 0. The funny thing was that the submission was run 2x more time as the save on public LB, so I had not even thought it could be a problem. But the lesson learned, I have to double-check everything.\nIn these competitions, where the computing resources so much a bottleneck, it would be great if it would not be a code competition. Or from the host side, that would be kind, to not put bigger images in the private LB than to the public.", "1301844": "see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments v15 changes 4 days ago to use rasterio instead of tifffile due to potential large image in private test. \n", "1301659": "Believe that it's just bug and should be solved with the time...", "1303757": "So any reaction from organizers and kaggle staff?( ", "1302306": "Dear Organizers,\n\nPlease note that the exact requirements for the private test scoring to be non-zero is specified nowhere in the competition page or in any of the official discussion threads.\n\nFrom other comments, one possible cause for the zero value private submission could be the use of Tifffle library instead of raterio but, this does not result in any kind of error or warning which could possibly help in identifying the failure during submission.\n\nThe only requirements defined for the for the submission csv file is following the right format and I believe the public submission would have failed if we had not followed that. **The fact that these instructions were nowhere on the platform which de-valuated many months of effort is clearly discouragement for participants who are beginners in Code Competitions.**\n\n**It is only fair for the submissions of participants who spend months working on this, to be fixed some way to re-run so the proper scores are reflected. Otherwise, I believe many amazing solutions would be lost just because of hidden OOM errors and not picking the right files from a hidden directory, which is really not the point of a research competition.**\n\n", "1302084": "I think the host of the competition doesnt care shit. It is 100% the host's problem with 600 0.000. Where is the warning? " } }