{ "id": 238002, "title": "Can anyone tell why this deepflash notebook is zero?", "url": "/competitions/hubmap-kidney-segmentation/discussion/238002", "author_name": "Swikwislkdjc", "post_date": "2021-05-11T00:23:54.587000", "votes": 11, "comment_count": 29, "views": 0, "content": "

https://www.kaggle.com/lililycai/infer-hubmap-deepflash2-bce?scriptVersionId=62398987
\nI used this notebook for submission, and the highest score in public in 0.940 which is top 14th in the ranking, but it becomes zero in private. It looks all my previous submission private score is zero 😂, very frustrated. And it looks 687/1216 -->56% of teams are zero score. Can the kaggle team or the host fix it? @leahscherschel

", "messages": [ { "id": 1301099, "postDate": "2021-05-11T00:23:54.587Z", "content": "

https://www.kaggle.com/lililycai/infer-hubmap-deepflash2-bce?scriptVersionId=62398987
\nI used this notebook for submission, and the highest score in public in 0.940 which is top 14th in the ranking, but it becomes zero in private. It looks all my previous submission private score is zero 😂, very frustrated. And it looks 687/1216 -->56% of teams are zero score. Can the kaggle team or the host fix it? @leahscherschel

", "rawMarkdown": "https://www.kaggle.com/lililycai/infer-hubmap-deepflash2-bce?scriptVersionId=62398987\nI used this notebook for submission, and the highest score in public in 0.940 which is top 14th in the ranking, but it becomes zero in private. It looks all my previous submission private score is zero 😂, very frustrated. And it looks 687/1216 -->56% of teams are zero score. Can the kaggle team or the host fix it? @leahscherschel ", "votes": 10 }, { "id": 1302722, "postDate": "2021-05-11T17:18:24.093Z", "content": "

I suggest you take attention to the differences of running time between inference run-all with online submission.If they take nearly the same time, there must be some trouble in submission. I found my submission time is nearly twice longer than run-all.

", "rawMarkdown": "I suggest you take attention to the differences of running time between inference run-all with online submission.If they take nearly the same time, there must be some trouble in submission. I found my submission time is nearly twice longer than run-all.", "votes": 4 }, { "id": 1303627, "postDate": "2021-05-12T06:49:01.087Z", "content": "

I have used Skimage (it uses tifffile). It might not be an OOM error. May be Skimage and tifffile need some additional codecs to open private test files, may be the files are compressed.

", "rawMarkdown": "I have used Skimage (it uses tifffile). It might not be an OOM error. May be Skimage and tifffile need some additional codecs to open private test files, may be the files are compressed. ", "votes": 1 }, { "id": 1301512, "postDate": "2021-05-11T05:47:50.830Z", "content": "

I am waiting to hear any confirmation but it is possible the issue with 0 scores has to do with the sample submission csv and there being another copy in the test folder (normally not the case). So only if you derived the ids from the image names did your submissions work. And of course if you got past the queue issues.

\n

UPDATE - see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments v15 changes 4 days ago to use rasterio instead of tifffile due to potential large image in private test.

", "rawMarkdown": "I am waiting to hear any confirmation but it is possible the issue with 0 scores has to do with the sample submission csv and there being another copy in the test folder (normally not the case). So only if you derived the ids from the image names did your submissions work. And of course if you got past the queue issues. \n\nUPDATE - see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments v15 changes 4 days ago to use rasterio instead of tifffile due to potential large image in private test. ", "votes": 1 }, { "id": 1301447, "postDate": "2021-05-11T04:48:18.990Z", "content": "

I think that problem is in this line:
\ndf_sample = pd.read_csv(cfg.data_path/'sample_submission.csv', index_col='id')
\nit reads ids from sample_submission.csv but ids of the private dataset are different. So there is no any ids from private dataset in subbmission.csv and scoring is 0.

", "rawMarkdown": "I think that problem is in this line:\ndf_sample = pd.read_csv(cfg.data_path/'sample_submission.csv', index_col='id')\nit reads ids from sample_submission.csv but ids of the private dataset are different. So there is no any ids from private dataset in subbmission.csv and scoring is 0.", "votes": -3, "replies": [ { "id": 1301478, "postDate": "2021-05-11T05:24:26.630Z", "content": "

sample_submission.csv is usually substituted with private test set during submission.

", "rawMarkdown": "sample_submission.csv is usually substituted with private test set during submission.", "votes": 2 }, { "id": 1301524, "postDate": "2021-05-11T05:56:56.067Z", "content": "

I know that 0.0 score can be if there are wrong ids in submission.csv. Probably sample_submission.csv does not have right ids for private dataset

", "rawMarkdown": "I know that 0.0 score can be if there are wrong ids in submission.csv. Probably sample_submission.csv does not have right ids for private dataset", "votes": -2 }, { "id": 1302753, "postDate": "2021-05-11T17:31:35.213Z", "content": "

I'm afraid that is not possible, or else we all would have gotten 0 results

", "rawMarkdown": "I'm afraid that is not possible, or else we all would have gotten 0 results" }, { "id": 1305397, "postDate": "2021-05-13T08:40:38.927Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1305456, "postDate": "2021-05-13T09:38:13.620Z", "content": "

We know that scoring is quite long. Looks like it calculates not only test images. Also all private scores was calculated instantly for all. This means that it was calculated before. So when you send the kernel it run on all images: test and private test. When you run your kernel you get id from sample_submission.csv but it does not have private test ids. So result is 0.

", "rawMarkdown": "We know that scoring is quite long. Looks like it calculates not only test images. Also all private scores was calculated instantly for all. This means that it was calculated before. So when you send the kernel it run on all images: test and private test. When you run your kernel you get id from sample_submission.csv but it does not have private test ids. So result is 0.", "votes": -2 }, { "id": 1305843, "postDate": "2021-05-13T14:03:00.240Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 1306052, "postDate": "2021-05-13T15:40:51.333Z", "content": "

sample_submission file is Ok.

", "rawMarkdown": "sample_submission file is Ok. " } ] }, { "id": 1305411, "postDate": "2021-05-13T08:52:37.380Z", "content": "

Hidden OOM issue on private data submission.

\n

Seems there are a lot updates on Kaggle during last time and handling of this issue was bad and probably become worse.

\n

Before it was better when it takes 9 hours to submit and then report an error…
\nAnd the saddest that this update take place during active competition with OOM problems from the beginning.

", "rawMarkdown": "Hidden OOM issue on private data submission.\n\nSeems there are a lot updates on Kaggle during last time and handling of this issue was bad and probably become worse.\n\nBefore it was better when it takes 9 hours to submit and then report an error…\nAnd the saddest that this update take place during active competition with OOM problems from the beginning." }, { "id": 1301192, "postDate": "2021-05-11T01:27:05.687Z", "content": "

I think there was a very large tiff file in the private test set. The tifffilelibrary couldn't read that file (maybe OOM problem) and the rasteriolibrary can read that file. Discussed here: https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/output

", "rawMarkdown": "I think there was a very large tiff file in the private test set. The `tifffile `library couldn't read that file (maybe OOM problem) and the `rasterio `library can read that file. Discussed here: https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/output", "replies": [ { "id": 1301203, "postDate": "2021-05-11T01:34:03.040Z", "content": "

There is no submission error shown in the kaggle submission page, and it looks normal because I can see my public score. No way to know this.

", "rawMarkdown": "There is no submission error shown in the kaggle submission page, and it looks normal because I can see my public score. No way to know this.", "votes": 2 }, { "id": 1301216, "postDate": "2021-05-11T01:39:17.280Z", "content": "

Yes, it was.
\nI observed that inference and submission durations are exactly same, so I think notebook doesn't predict private test set because private test set has 10 images. Then I tried reading tiff file instead of reading sample_submission.csv, but received an error that Submission Scoring Error.

", "rawMarkdown": "Yes, it was. \nI observed that inference and submission durations are exactly same, so I think notebook doesn't predict private test set because private test set has 10 images. Then I tried reading tiff file instead of reading sample_submission.csv, but received an error that Submission Scoring Error. " }, { "id": 1301258, "postDate": "2021-05-11T02:03:13.157Z", "content": "

could u share how to know the privagte has 10 images? I know my code can read/predict all 20 images ( 15+public) on the same run. It shouldnt be a problem.

", "rawMarkdown": "could u share how to know the privagte has 10 images? I know my code can read/predict all 20 images ( 15+public) on the same run. It shouldnt be a problem. " }, { "id": 1301267, "postDate": "2021-05-11T02:09:39.873Z", "content": "

You can find it from top of the public LB. There is an explanation of how many percent of data is used to calculate LB score. For this competition:

\n
This leaderboard is calculated with approximately 33% of the test data.\nThe final results will be based on the other 67%, so the final standings may be different.\n
", "rawMarkdown": "You can find it from top of the public LB. There is an explanation of how many percent of data is used to calculate LB score. For this competition:\n\n```\nThis leaderboard is calculated with approximately 33% of the test data.\nThe final results will be based on the other 67%, so the final standings may be different.\n```" }, { "id": 1301276, "postDate": "2021-05-11T02:16:05.950Z", "content": "

Not everyone know this trick, and not everyone sit there to wait. They may just go out or do something else during submission, and when come back, the page looks fine with no submission error. People don't pay attention unless there's an error showed. Kaggle team needs to fix this.

", "rawMarkdown": "Not everyone know this trick, and not everyone sit there to wait. They may just go out or do something else during submission, and when come back, the page looks fine with no submission error. People don't pay attention unless there's an error showed. Kaggle team needs to fix this. ", "votes": 2 }, { "id": 1301283, "postDate": "2021-05-11T02:23:42.360Z", "content": "

I read that differently. 2X the data wasnt regardless of the size of each image.

", "rawMarkdown": "I read that differently. 2X the data wasnt regardless of the size of each image. ", "votes": 2 }, { "id": 1301296, "postDate": "2021-05-11T02:34:37.403Z", "content": "

For deepflash, inference and submission were really faster. So I thought there is a bug. For R language problem, I don't know what happened.

", "rawMarkdown": "For deepflash, inference and submission were really faster. So I thought there is a bug. For R language problem, I don't know what happened. " } ] }, { "id": 1301191, "postDate": "2021-05-11T01:26:20.120Z", "content": "

@rpsantosakaggle @lililycai Phil mentioned the ones which deliberately skipped the private LB will have zero . So submission using csv is not allowed ?

", "rawMarkdown": "@rpsantosakaggle @lililycai Phil mentioned the ones which deliberately skipped the private LB will have zero . So submission using csv is not allowed ?", "replies": [ { "id": 1301198, "postDate": "2021-05-11T01:29:49.173Z", "content": "

I didn't submit only csv file. If you look at the kernel, it reads the test image files and then make predictions.

", "rawMarkdown": "I didn't submit only csv file. If you look at the kernel, it reads the test image files and then make predictions.", "votes": 2 }, { "id": 1301208, "postDate": "2021-05-11T01:35:39.070Z", "content": "

Not sure if this is your case. Many people just upload their predictions (from their computer for example) from public leaderboard into kernels and submitted just that, their uploaded csv. They cant do it to the private leaderboad because the private dataset is hidden. That was what i meant.

", "rawMarkdown": "Not sure if this is your case. Many people just upload their predictions (from their computer for example) from public leaderboard into kernels and submitted just that, their uploaded csv. They cant do it to the private leaderboad because the private dataset is hidden. That was what i meant. \n\n" }, { "id": 1301215, "postDate": "2021-05-11T01:39:15.083Z", "content": "

This is not my case. I know many people just submit a csv file, which is a wrong way, because it does not include private test set.

", "rawMarkdown": "This is not my case. I know many people just submit a csv file, which is a wrong way, because it does not include private test set. ", "votes": 2 }, { "id": 1301272, "postDate": "2021-05-11T02:14:22.363Z", "content": "

@lhagiimn has a point. How long did the submission take you? If a few minutes, as opposed to hours, his theory must hold.

", "rawMarkdown": "@lhagiimn has a point. How long did the submission take you? If a few minutes, as opposed to hours, his theory must hold." } ] }, { "id": 1305308, "postDate": "2021-05-13T07:50:33.547Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1301492, "postDate": "2021-05-11T05:36:00.283Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1301269, "postDate": "2021-05-11T02:12:14.257Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1301121, "postDate": "2021-05-11T00:34:40.540Z", "rawMarkdown": "", "votes": 3, "isDeleted": true } ], "comments": [ { "id": 1302722, "author_name": "朴大福", "author_url": "", "post_date": "2021-05-11T17:18:24.093000", "content": "

I suggest you take attention to the differences of running time between inference run-all with online submission.If they take nearly the same time, there must be some trouble in submission. I found my submission time is nearly twice longer than run-all.

", "votes": 4, "replies": [] }, { "id": 1303627, "author_name": "Elahi", "author_url": "", "post_date": "2021-05-12T06:49:01.087000", "content": "

I have used Skimage (it uses tifffile). It might not be an OOM error. May be Skimage and tifffile need some additional codecs to open private test files, may be the files are compressed.

", "votes": 1, "replies": [] }, { "id": 1301512, "author_name": "something4kag", "author_url": "", "post_date": "2021-05-11T05:47:50.830000", "content": "

I am waiting to hear any confirmation but it is possible the issue with 0 scores has to do with the sample submission csv and there being another copy in the test folder (normally not the case). So only if you derived the ids from the image names did your submissions work. And of course if you got past the queue issues.

\n

UPDATE - see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments v15 changes 4 days ago to use rasterio instead of tifffile due to potential large image in private test.

", "votes": 1, "replies": [] }, { "id": 1301447, "author_name": "Maxim Vedenev", "author_url": "", "post_date": "2021-05-11T04:48:18.990000", "content": "

I think that problem is in this line:
\ndf_sample = pd.read_csv(cfg.data_path/'sample_submission.csv', index_col='id')
\nit reads ids from sample_submission.csv but ids of the private dataset are different. So there is no any ids from private dataset in subbmission.csv and scoring is 0.

", "votes": -3, "replies": [ { "id": 1301478, "author_name": "Swikwislkdjc", "author_url": "", "post_date": "2021-05-11T05:24:26.630000", "content": "

sample_submission.csv is usually substituted with private test set during submission.

", "votes": 2, "replies": [] }, { "id": 1301524, "author_name": "Maxim Vedenev", "author_url": "", "post_date": "2021-05-11T05:56:56.067000", "content": "

I know that 0.0 score can be if there are wrong ids in submission.csv. Probably sample_submission.csv does not have right ids for private dataset

", "votes": -2, "replies": [] }, { "id": 1302753, "author_name": "andras", "author_url": "", "post_date": "2021-05-11T17:31:35.213000", "content": "

I'm afraid that is not possible, or else we all would have gotten 0 results

", "votes": 0, "replies": [] }, { "id": 1305397, "author_name": "", "author_url": "", "post_date": "2021-05-13T08:40:38.927000", "content": "", "votes": 0, "replies": [] }, { "id": 1305456, "author_name": "Maxim Vedenev", "author_url": "", "post_date": "2021-05-13T09:38:13.620000", "content": "

We know that scoring is quite long. Looks like it calculates not only test images. Also all private scores was calculated instantly for all. This means that it was calculated before. So when you send the kernel it run on all images: test and private test. When you run your kernel you get id from sample_submission.csv but it does not have private test ids. So result is 0.

", "votes": -2, "replies": [] }, { "id": 1305843, "author_name": "", "author_url": "", "post_date": "2021-05-13T14:03:00.240000", "content": "", "votes": -1, "replies": [] }, { "id": 1306052, "author_name": "anthony", "author_url": "", "post_date": "2021-05-13T15:40:51.333000", "content": "

sample_submission file is Ok.

", "votes": 0, "replies": [] } ] }, { "id": 1305411, "author_name": "anthony", "author_url": "", "post_date": "2021-05-13T08:52:37.380000", "content": "

Hidden OOM issue on private data submission.

\n

Seems there are a lot updates on Kaggle during last time and handling of this issue was bad and probably become worse.

\n

Before it was better when it takes 9 hours to submit and then report an error…
\nAnd the saddest that this update take place during active competition with OOM problems from the beginning.

", "votes": 0, "replies": [] }, { "id": 1301192, "author_name": "lhagiimn", "author_url": "", "post_date": "2021-05-11T01:27:05.687000", "content": "

I think there was a very large tiff file in the private test set. The tifffilelibrary couldn't read that file (maybe OOM problem) and the rasteriolibrary can read that file. Discussed here: https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/output

", "votes": 0, "replies": [ { "id": 1301203, "author_name": "Swikwislkdjc", "author_url": "", "post_date": "2021-05-11T01:34:03.040000", "content": "

There is no submission error shown in the kaggle submission page, and it looks normal because I can see my public score. No way to know this.

", "votes": 2, "replies": [] }, { "id": 1301216, "author_name": "lhagiimn", "author_url": "", "post_date": "2021-05-11T01:39:17.280000", "content": "

Yes, it was.
\nI observed that inference and submission durations are exactly same, so I think notebook doesn't predict private test set because private test set has 10 images. Then I tried reading tiff file instead of reading sample_submission.csv, but received an error that Submission Scoring Error.

", "votes": 0, "replies": [] }, { "id": 1301258, "author_name": "rpsantosa_kaggle", "author_url": "", "post_date": "2021-05-11T02:03:13.157000", "content": "

could u share how to know the privagte has 10 images? I know my code can read/predict all 20 images ( 15+public) on the same run. It shouldnt be a problem.

", "votes": 0, "replies": [] }, { "id": 1301267, "author_name": "lhagiimn", "author_url": "", "post_date": "2021-05-11T02:09:39.873000", "content": "

You can find it from top of the public LB. There is an explanation of how many percent of data is used to calculate LB score. For this competition:

\n
This leaderboard is calculated with approximately 33% of the test data.\nThe final results will be based on the other 67%, so the final standings may be different.\n
", "votes": 0, "replies": [] }, { "id": 1301276, "author_name": "Swikwislkdjc", "author_url": "", "post_date": "2021-05-11T02:16:05.950000", "content": "

Not everyone know this trick, and not everyone sit there to wait. They may just go out or do something else during submission, and when come back, the page looks fine with no submission error. People don't pay attention unless there's an error showed. Kaggle team needs to fix this.

", "votes": 2, "replies": [] }, { "id": 1301283, "author_name": "rpsantosa_kaggle", "author_url": "", "post_date": "2021-05-11T02:23:42.360000", "content": "

I read that differently. 2X the data wasnt regardless of the size of each image.

", "votes": 2, "replies": [] }, { "id": 1301296, "author_name": "lhagiimn", "author_url": "", "post_date": "2021-05-11T02:34:37.403000", "content": "

For deepflash, inference and submission were really faster. So I thought there is a bug. For R language problem, I don't know what happened.

", "votes": 0, "replies": [] } ] }, { "id": 1301191, "author_name": "Tensor Girl", "author_url": "", "post_date": "2021-05-11T01:26:20.120000", "content": "

@rpsantosakaggle @lililycai Phil mentioned the ones which deliberately skipped the private LB will have zero . So submission using csv is not allowed ?

", "votes": 0, "replies": [ { "id": 1301198, "author_name": "Swikwislkdjc", "author_url": "", "post_date": "2021-05-11T01:29:49.173000", "content": "

I didn't submit only csv file. If you look at the kernel, it reads the test image files and then make predictions.

", "votes": 2, "replies": [] }, { "id": 1301208, "author_name": "rpsantosa_kaggle", "author_url": "", "post_date": "2021-05-11T01:35:39.070000", "content": "

Not sure if this is your case. Many people just upload their predictions (from their computer for example) from public leaderboard into kernels and submitted just that, their uploaded csv. They cant do it to the private leaderboad because the private dataset is hidden. That was what i meant.

", "votes": 0, "replies": [] }, { "id": 1301215, "author_name": "Swikwislkdjc", "author_url": "", "post_date": "2021-05-11T01:39:15.083000", "content": "

This is not my case. I know many people just submit a csv file, which is a wrong way, because it does not include private test set.

", "votes": 2, "replies": [] }, { "id": 1301272, "author_name": "andras", "author_url": "", "post_date": "2021-05-11T02:14:22.363000", "content": "

@lhagiimn has a point. How long did the submission take you? If a few minutes, as opposed to hours, his theory must hold.

", "votes": 0, "replies": [] } ] }, { "id": 1305308, "author_name": "", "author_url": "", "post_date": "2021-05-13T07:50:33.547000", "content": "", "votes": 1, "replies": [] }, { "id": 1301492, "author_name": "", "author_url": "", "post_date": "2021-05-11T05:36:00.283000", "content": "", "votes": 0, "replies": [] }, { "id": 1301269, "author_name": "", "author_url": "", "post_date": "2021-05-11T02:12:14.257000", "content": "", "votes": 0, "replies": [] }, { "id": 1301121, "author_name": "", "author_url": "", "post_date": "2021-05-11T00:34:40.540000", "content": "", "votes": 3, "replies": [] } ], "raw_markdown_by_id": { "1301099": "https://www.kaggle.com/lililycai/infer-hubmap-deepflash2-bce?scriptVersionId=62398987\nI used this notebook for submission, and the highest score in public in 0.940 which is top 14th in the ranking, but it becomes zero in private. It looks all my previous submission private score is zero 😂, very frustrated. And it looks 687/1216 -->56% of teams are zero score. Can the kaggle team or the host fix it? @leahscherschel ", "1302722": "I suggest you take attention to the differences of running time between inference run-all with online submission.If they take nearly the same time, there must be some trouble in submission. I found my submission time is nearly twice longer than run-all.", "1303627": "I have used Skimage (it uses tifffile). It might not be an OOM error. May be Skimage and tifffile need some additional codecs to open private test files, may be the files are compressed. ", "1301512": "I am waiting to hear any confirmation but it is possible the issue with 0 scores has to do with the sample submission csv and there being another copy in the test folder (normally not the case). So only if you derived the ids from the image names did your submissions work. And of course if you got past the queue issues. \n\nUPDATE - see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/comments v15 changes 4 days ago to use rasterio instead of tifffile due to potential large image in private test. ", "1301447": "I think that problem is in this line:\ndf_sample = pd.read_csv(cfg.data_path/'sample_submission.csv', index_col='id')\nit reads ids from sample_submission.csv but ids of the private dataset are different. So there is no any ids from private dataset in subbmission.csv and scoring is 0.", "1305411": "Hidden OOM issue on private data submission.\n\nSeems there are a lot updates on Kaggle during last time and handling of this issue was bad and probably become worse.\n\nBefore it was better when it takes 9 hours to submit and then report an error…\nAnd the saddest that this update take place during active competition with OOM problems from the beginning.", "1301192": "I think there was a very large tiff file in the private test set. The `tifffile `library couldn't read that file (maybe OOM problem) and the `rasterio `library can read that file. Discussed here: https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/output", "1301191": "@rpsantosakaggle @lililycai Phil mentioned the ones which deliberately skipped the private LB will have zero . So submission using csv is not allowed ?", "1305308": "", "1301492": "", "1301269": "", "1301121": "" } }