{ "id": 455220, "title": "Submission Failures", "url": "/competitions/blood-vessel-segmentation/discussion/455220", "author_name": "Ryan Holbrook", "post_date": "2023-11-13T21:25:12.453000", "votes": 13, "comment_count": 40, "views": 0, "content": "

Hi everyone,

\n

It appears there's an issue with submissions accepting the 1 0 entries indicating empty masks. I'm still investigating the cause of the error, but I'll update here as soon as I know more.

\n

Thank you,
\nRyan

", "messages": [ { "id": 2523945, "postDate": "2023-11-13T21:25:12.453Z", "content": "

Hi everyone,

\n

It appears there's an issue with submissions accepting the 1 0 entries indicating empty masks. I'm still investigating the cause of the error, but I'll update here as soon as I know more.

\n

Thank you,
\nRyan

", "rawMarkdown": "Hi everyone,\n\nIt appears there's an issue with submissions accepting the `1 0` entries indicating empty masks. I'm still investigating the cause of the error, but I'll update here as soon as I know more.\n\nThank you,\nRyan", "votes": 13 }, { "id": 2535091, "postDate": "2023-11-23T05:12:16.150Z", "content": "

is there any other tricks to determine what is causing the submission scoring error??

\n

I have tried a lot of different things and none of my submissions seem to score correctly. It is probably something small that I am messing up, or maybe something is still wrong with the scoring method, It is hard to tell. I just want atleast one successful submission so that I can start making improvements to my model.

", "rawMarkdown": "is there any other tricks to determine what is causing the submission scoring error??\n\nI have tried a lot of different things and none of my submissions seem to score correctly. It is probably something small that I am messing up, or maybe something is still wrong with the scoring method, It is hard to tell. I just want atleast one successful submission so that I can start making improvements to my model.\n", "votes": 5, "replies": [ { "id": 2535440, "postDate": "2023-11-23T10:31:30.083Z", "content": "

increase the threshold of the mask during inference or add removing small regions

", "rawMarkdown": "increase the threshold of the mask during inference or add removing small regions ", "votes": 3, "replies": [ { "id": 2536261, "postDate": "2023-11-24T04:39:38.717Z", "content": "

Thank you, I finally received a score! Unfortunately the score was 0.000. Nowhere to go but up from here! 😅

", "rawMarkdown": "Thank you, I finally received a score! Unfortunately the score was 0.000. Nowhere to go but up from here! 😅", "votes": 1 }, { "id": 2549611, "postDate": "2023-12-05T11:42:47.197Z", "content": "

could you please your code? so far, I only got best 0 too.

", "rawMarkdown": "could you please your code? so far, I only got best 0 too." }, { "id": 2557038, "postDate": "2023-12-11T06:42:01.943Z", "content": "

Thank you! It solved my submission error.

", "rawMarkdown": "Thank you! It solved my submission error." } ] } ] }, { "id": 2526815, "postDate": "2023-11-16T04:32:54.837Z", "content": "

I'm still getting 'Submission Scoring Error', even though my notebook ran and generated a submission.csv file successfully.

", "rawMarkdown": "I'm still getting 'Submission Scoring Error', even though my notebook ran and generated a submission.csv file successfully.", "votes": 6, "replies": [ { "id": 2531206, "postDate": "2023-11-20T02:55:04.927Z", "content": "

same. for the curious, have provided a very simple errored-out notebook here.
\nhttps://www.kaggle.com/code/pangyuteng/frustrated-frank

", "rawMarkdown": "same. for the curious, have provided a very simple errored-out notebook here.\n[https://www.kaggle.com/code/pangyuteng/frustrated-frank](https://www.kaggle.com/code/pangyuteng/frustrated-frank)", "votes": 3 } ] }, { "id": 2525225, "postDate": "2023-11-14T23:33:50.920Z", "content": "

Hi everyone,

\n

I just deployed an updated version of the metric that I believe fixes the problem. Submissions with 1 0 for empty masks should no longer fail. I'm planning to run a rescore on all submissions with the updated metric sometime tomorrow once I can run some additional tests.

\n

Though I believe the update should fix submissions, I'm still investigating the root cause. (The problem doesn't seem to occur when the metric is run interactively, for instance, only during scoring.) Please let me know if you are still having trouble and I will investigate.

", "rawMarkdown": "Hi everyone,\n\nI just deployed an updated version of the metric that I believe fixes the problem. Submissions with `1 0` for empty masks should no longer fail. I'm planning to run a rescore on all submissions with the updated metric sometime tomorrow once I can run some additional tests.\n\nThough I believe the update should fix submissions, I'm still investigating the root cause. (The problem doesn't seem to occur when the metric is run interactively, for instance, only during scoring.) Please let me know if you are still having trouble and I will investigate.", "votes": 3, "replies": [ { "id": 2525231, "postDate": "2023-11-14T23:56:49.637Z", "content": "

The last two of my submissions took nearly an hour to calculate the score (\"Submission running\" status), before that, it was done in 10ish minutes, if you could check that the update you did to the metric is not the reason, that would be helpful

\n

Edit: I just got a \"Submission Scoring Error\" on a submission where the only change was image-size and weights, please look into that too, if this could be the reason, I do not use \"1 0\" for empty mask

", "rawMarkdown": "The last two of my submissions took nearly an hour to calculate the score (\"Submission running\" status), before that, it was done in 10ish minutes, if you could check that the update you did to the metric is not the reason, that would be helpful\n\nEdit: I just got a \"Submission Scoring Error\" on a submission where the only change was image-size and weights, please look into that too, if this could be the reason, I do not use \"1 0\" for empty mask", "replies": [ { "id": 2525270, "postDate": "2023-11-15T01:33:32.923Z", "content": "

Hi @harshitsheoran, I'm sorry to hear you're still having trouble. The longer runtimes might be due to a more complicated solution. Typically, the test submissions I've done have run in about 20-30 minutes. I wouldn't be surprised if sometimes it took up to an hour.

\n

It looks like the last submission failed due to an out-of-memory error in the scoring metric. It might be that a more complicated solution is again the culprit. I already added a number of changes to the reference metric to improve memory efficiency, and I'm not certain where to make further improvements, but I'll try to look again when I can.

\n

For reference, here is the code for the metric: Surface Dice Metric. That could help with testing submissions.

", "rawMarkdown": "Hi @harshitsheoran, I'm sorry to hear you're still having trouble. The longer runtimes might be due to a more complicated solution. Typically, the test submissions I've done have run in about 20-30 minutes. I wouldn't be surprised if sometimes it took up to an hour.\n\nIt looks like the last submission failed due to an out-of-memory error in the scoring metric. It might be that a more complicated solution is again the culprit. I already added a number of changes to the reference metric to improve memory efficiency, and I'm not certain where to make further improvements, but I'll try to look again when I can.\n\nFor reference, here is the code for the metric: [Surface Dice Metric](https://www.kaggle.com/code/metric/surface-dice-metric). That could help with testing submissions.", "votes": -1 }, { "id": 2525286, "postDate": "2023-11-15T02:12:38.140Z", "content": "

Thank you for replying, I did make a submit an hour ago, this submit was a repeat of a submit which ran successfully, and it failed now, I hope that the metric becomes more efficient

", "rawMarkdown": "Thank you for replying, I did make a submit an hour ago, this submit was a repeat of a submit which ran successfully, and it failed now, I hope that the metric becomes more efficient" }, { "id": 2525296, "postDate": "2023-11-15T02:23:13.923Z", "content": "

even same here @ryanholbrook previously it took 10 mins to complete a submission now it takes longer time

", "rawMarkdown": "even same here @ryanholbrook previously it took 10 mins to complete a submission now it takes longer time\n", "votes": 1 }, { "id": 2525847, "postDate": "2023-11-15T12:27:07.077Z", "content": "

@ryanholbrook I have failed a lot of submissions to \"Submission Scoring Error\", Even though I am only predicting one of the kidneys, this is first 24 hours of the competition and I can only imagine that the solution becomes more complicated as the competition progresses, Can you please give me an example on how to make my submission \"less-complicated\" so that it does not give the error

", "rawMarkdown": "@ryanholbrook I have failed a lot of submissions to \"Submission Scoring Error\", Even though I am only predicting one of the kidneys, this is first 24 hours of the competition and I can only imagine that the solution becomes more complicated as the competition progresses, Can you please give me an example on how to make my submission \"less-complicated\" so that it does not give the error", "votes": 3 }, { "id": 2526241, "postDate": "2023-11-15T16:54:08.113Z", "content": "

@harshitsheoran Yesterday, I encountered a new issue with Kaggle submissions. Previously, using \"1 1\" in the RLE allowed me to submit successfully, but now, even the same notebook version that worked before is failing.

\n

To troubleshoot, I wrote some code to verify the correctness of the RLE and to check for any missing IDs in the submission dataframe. If any IDs were missing, I appended them to the dataframe with an RLE of \"1 1\". Despite these efforts, I'm still facing scoring errors. New error : Submission Scoring Error

", "rawMarkdown": "@harshitsheoran Yesterday, I encountered a new issue with Kaggle submissions. Previously, using \"1 1\" in the RLE allowed me to submit successfully, but now, even the same notebook version that worked before is failing.\n\nTo troubleshoot, I wrote some code to verify the correctness of the RLE and to check for any missing IDs in the submission dataframe. If any IDs were missing, I appended them to the dataframe with an RLE of \"1 1\". Despite these efforts, I'm still facing scoring errors. New error : **Submission Scoring Error**", "votes": 1 }, { "id": 2526369, "postDate": "2023-11-15T18:20:35.187Z", "content": "

@abdrah @kashiwaba I had the same issue, it ruined a lot of submissions, but I managed to submit the correct one by making it less \"complicated\", just to test it, I changed the threshold from 0.5 to 0.9, this passed but incurred a massive penalty on score (0.423 -> 0.309) (I also changed \"1 1\" to \"1 0\" as that problem looks like it is fixed), so I would recommend trying out numbers like 0.6 or 0.7 to see if those works, I used all my submissions for today

", "rawMarkdown": "@abdrah @kashiwaba I had the same issue, it ruined a lot of submissions, but I managed to submit the correct one by making it less \"complicated\", just to test it, I changed the threshold from 0.5 to 0.9, this passed but incurred a massive penalty on score (0.423 -> 0.309) (I also changed \"1 1\" to \"1 0\" as that problem looks like it is fixed), so I would recommend trying out numbers like 0.6 or 0.7 to see if those works, I used all my submissions for today", "votes": 1 }, { "id": 2526381, "postDate": "2023-11-15T18:31:21.007Z", "content": "

That's a smart approach – increasing the threshold could help prevent the creation of overly large files. However, as you pointed out, this might affect the final score. Since I have only one submission attempt left, I'm considering trying a threshold of 0.8, just to see if it's effective. I usually opt for a threshold of 0.5.

", "rawMarkdown": "That's a smart approach – increasing the threshold could help prevent the creation of overly large files. However, as you pointed out, this might affect the final score. Since I have only one submission attempt left, I'm considering trying a threshold of 0.8, just to see if it's effective. I usually opt for a threshold of 0.5." }, { "id": 2526410, "postDate": "2023-11-15T19:04:02.093Z", "content": "

Good Luck!, please let us know how that goes, I usually opt for 0.5 threshold too

", "rawMarkdown": "Good Luck!, please let us know how that goes, I usually opt for 0.5 threshold too" }, { "id": 2527522, "postDate": "2023-11-16T15:36:39.823Z", "content": "

Unfortunately, my attempts were unsuccessful. I even reviewed the competition metrics, but that didn't help either. It's puzzling that some participants are achieving scores as high as 0.7 on the leaderboard.

", "rawMarkdown": "Unfortunately, my attempts were unsuccessful. I even reviewed the competition metrics, but that didn't help either. It's puzzling that some participants are achieving scores as high as 0.7 on the leaderboard.", "votes": 1 }, { "id": 2531888, "postDate": "2023-11-20T16:12:11.907Z", "content": "

@harshitsheoran what are the error logs you get with \"Submission Scoring Error\" ? I currently have this, do you have the same?

\n
\n

Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected.

\n
", "rawMarkdown": "@harshitsheoran what are the error logs you get with \"Submission Scoring Error\" ? I currently have this, do you have the same?\n> Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected." }, { "id": 2531909, "postDate": "2023-11-20T16:22:13.237Z", "content": "

Yes, but I think this statement might be a static and not reflective on the type of error

", "rawMarkdown": "Yes, but I think this statement might be a static and not reflective on the type of error", "votes": 1 } ] }, { "id": 2526240, "postDate": "2023-11-15T16:50:49.780Z", "content": "

@ryanholbrook
\nI also submitted a Notebook almost identical to the one that was graded successfully yesterday (exactly the same parameters and weights), but for some reason today it is getting a Scoring Error.

\n

I appreciate the quick response, but I would rather they revert to the previous scoring system until the underlying problem is identified than rush to release a half-baked one.

", "rawMarkdown": "@ryanholbrook \nI also submitted a Notebook almost identical to the one that was graded successfully yesterday (exactly the same parameters and weights), but for some reason today it is getting a Scoring Error.\n\nI appreciate the quick response, but I would rather they revert to the previous scoring system until the underlying problem is identified than rush to release a half-baked one.", "votes": 4, "replies": [ { "id": 2526247, "postDate": "2023-11-15T16:57:58.347Z", "content": "

@kashiwaba That's what I just said in the earlier comment : \"Yesterday, I encountered a new issue with Kaggle submissions. Previously, using \"1 1\" in the RLE allowed me to submit successfully, but now, even the same notebook version that worked before is failing. To troubleshoot, I wrote some code to verify the correctness of the RLE and to check for any missing IDs in the submission dataframe. If any IDs were missing, I appended them to the dataframe with an RLE of \"1 1\". Despite these efforts, I'm still facing scoring errors. New error : Submission Scoring Error\"

", "rawMarkdown": "@kashiwaba That's what I just said in the earlier comment : \"Yesterday, I encountered a new issue with Kaggle submissions. Previously, using \"1 1\" in the RLE allowed me to submit successfully, but now, even the same notebook version that worked before is failing. To troubleshoot, I wrote some code to verify the correctness of the RLE and to check for any missing IDs in the submission dataframe. If any IDs were missing, I appended them to the dataframe with an RLE of \"1 1\". Despite these efforts, I'm still facing scoring errors. **New error : Submission Scoring Error**\"", "votes": 2 }, { "id": 2526373, "postDate": "2023-11-15T18:23:09.837Z", "content": "

I liked the previous one better, it was faster and did not have OOM problems between Submission-Scoring, \"1 1\" hardly impacts the score anyway if everyone is forced to use that

", "rawMarkdown": "I liked the previous one better, it was faster and did not have OOM problems between Submission-Scoring, \"1 1\" hardly impacts the score anyway if everyone is forced to use that", "votes": 1 } ] }, { "id": 2596510, "postDate": "2024-01-11T06:15:09.277Z", "content": "

Hi @ryanholbrook , I am still facing submission scoring errors today, which is driving me crazy. Do you have any solutions?

", "rawMarkdown": "Hi @ryanholbrook , I am still facing submission scoring errors today, which is driving me crazy. Do you have any solutions?", "votes": 2 } ] }, { "id": 2536958, "postDate": "2023-11-24T16:16:31.373Z", "content": "

Hi, this is my first notebook only competition and currently (as of 11.24.23) not able to score on
\n the leaderboard other than with rle mask of \"1 0\". My submissions get the error:

\n
\n

Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. See more debugging tips

\n
\n

After training on one kidney i use the provided *score * (V22) function to validate on a second kidney and get score of ~ 0.50, so i would guess my submission format should be correct and as i do not get an OOM error at least on the \"local\" runs.
\nIt would be really great to get more information from the submissions failure so one knows if the error is on her/his side or if it is on the server side. Thxs and a great weekend to you all!

", "rawMarkdown": "Hi, this is my first notebook only competition and currently (as of 11.24.23) not able to score on\n the leaderboard other than with rle mask of \"1 0\". My submissions get the error: \n >Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. See more debugging tips\n\nAfter training on one kidney i use the provided *score * (V22) function to validate on a second kidney and get score of ~ 0.50, so i would guess my submission format should be correct and as i do not get an OOM error at least on the \"local\" runs. \nIt would be really great to get more information from the submissions failure so one knows if the error is on her/his side or if it is on the server side. Thxs and a great weekend to you all!\n\n", "votes": 4, "replies": [ { "id": 2539566, "postDate": "2023-11-27T06:27:46.280Z", "content": "

Yes, I agree, I trained one model submitted it, trained it further, all I did was change out the model for their one with more training, changed no other details in the code and it gets a submission error.

\n

I have tried using higher threshold, and also removing small objects but still get submission error. I have no idea how a dice score calculation can run out memory, also I don’t even know if that is what is failing me, it’s just difficult to determine what to fix to avoid submission scoring errors

", "rawMarkdown": "Yes, I agree, I trained one model submitted it, trained it further, all I did was change out the model for their one with more training, changed no other details in the code and it gets a submission error.\n\nI have tried using higher threshold, and also removing small objects but still get submission error. I have no idea how a dice score calculation can run out memory, also I don’t even know if that is what is failing me, it’s just difficult to determine what to fix to avoid submission scoring errors", "votes": 2 } ] }, { "id": 2531884, "postDate": "2023-11-20T16:09:13.587Z", "content": "

@ryanholbrook Can you please clarify what is the current status on the organizers' side ? Do you consider the metric computation as currently working ? Is it ok to say that one should try to make a \"simple enough\" surface prediction so that the computation goes through or are you currently working on a fix that would work regardless of the predicted structures (as long as it follows the submission format) ?

\n

Thanks for letting us know!

", "rawMarkdown": "@ryanholbrook Can you please clarify what is the current status on the organizers' side ? Do you consider the metric computation as currently working ? Is it ok to say that one should try to make a \"simple enough\" surface prediction so that the computation goes through or are you currently working on a fix that would work regardless of the predicted structures (as long as it follows the submission format) ?\n\nThanks for letting us know!", "votes": 4, "replies": [ { "id": 2532176, "postDate": "2023-11-20T19:49:03.147Z", "content": "

Hi @optimo, The metric computation itself is now correct and functional so far as I'm aware. There's still the issue of OOM errors when scoring some submissions. If we're able to come up with any improvements to the metric's memory usage, I will add them. This is something I'll continue to investigate. As I don't know if or when such improvements will be possible, for now, unfortunately, I'd say that making a submission simple enough to go through will have to be part of the competition challenge. For what it's worth, I have checked that a perfect submission will indeed score correctly.

", "rawMarkdown": "Hi @optimo, The metric computation itself is now correct and functional so far as I'm aware. There's still the issue of OOM errors when scoring some submissions. If we're able to come up with any improvements to the metric's memory usage, I will add them. This is something I'll continue to investigate. As I don't know if or when such improvements will be possible, for now, unfortunately, I'd say that making a submission simple enough to go through will have to be part of the competition challenge. For what it's worth, I have checked that a perfect submission will indeed score correctly.", "votes": 4, "replies": [ { "id": 2532774, "postDate": "2023-11-21T10:14:08.430Z", "content": "

Hello @ryanholbrook,

\n

I was not expecting this answer to be honest. It seems rather vague for an official rule of the competition to make a \"simple enough\" submission so that the metric computation does not OOM. Why would this notebook's solution used to be \"simple enough\" https://www.kaggle.com/code/kashiwaba/sennet-hoa-inference-unet-simple-baseline/notebook with the previous method but now is too complicated to be considered a valid solution (it's quite a simple baseline by the way - people are now changing thresholds from 0.99 to 0.97 to see what is considered \"simple enough\", this just does not make any sense)?

\n

Also, the goal of the competition is to come up with the best possible solution, not the best possible \"simple enough\" solution. I am sure the community can come up with a better implementation of the metric but some things might need clarifications:

\n", "rawMarkdown": "Hello @ryanholbrook,\n\nI was not expecting this answer to be honest. It seems rather vague for an official rule of the competition to make a \"simple enough\" submission so that the metric computation does not OOM. Why would this notebook's solution used to be \"simple enough\" https://www.kaggle.com/code/kashiwaba/sennet-hoa-inference-unet-simple-baseline/notebook with the previous method but now is too complicated to be considered a valid solution (it's quite a simple baseline by the way - people are now changing thresholds from 0.99 to 0.97 to see what is considered \"simple enough\", this just does not make any sense)?\n\nAlso, the goal of the competition is to come up with the best possible solution, not the best possible \"simple enough\" solution. I am sure the community can come up with a better implementation of the metric but some things might need clarifications:\n- Can you confirm that the current code for the metric is the version 22 of this notebook: https://www.kaggle.com/code/metric/surface-dice-metric/notebook ?\n- What are the specs of the environment where the metrics are computed ? same as the ones chosen to run the submission notebook ? How much cpu RAM and threads does it have ?\n- Would it be possible to add a specific error message to confirm that the Scoring Error comes from an OOM during metric's computation ?\n- Any other constraints for the metric's computation ?", "votes": 9 }, { "id": 2532983, "postDate": "2023-11-21T14:00:18.137Z", "content": "

The current version of the metric is now the same as used at the start of the competition. I'm not certain why submissions that completed before now do not complete, and why the 1 0 issue is no longer affecting submissions. It appears there was some kind of transient issue affecting the scoring environment that has now resolved.

\n", "rawMarkdown": "The current version of the metric is now the same as used at the start of the competition. I'm not certain why submissions that completed before now do not complete, and why the `1 0` issue is no longer affecting submissions. It appears there was some kind of transient issue affecting the scoring environment that has now resolved.\n\n- Yes, the most recent version of the notebook is the one used for the competition.\n- The environment used for scoring is the same as used in the CPU notebook environment. You can see full technical specifications in [this documentation](https://www.kaggle.com/docs/notebooks#technical-specifications).\n- I'll see what I can do about adding a more specific message for OOM errors.", "votes": 1 }, { "id": 2533045, "postDate": "2023-11-21T14:47:52.343Z", "content": "

Thank you @ryanholbrook, I hope we can find a solution to this because I see a real problem here.

\n

With the exact same submission, only changing threshold here is what I get:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ThresholdCV (kidney_3_sparse)LB
0.50.778OOM
0.850.642OOM
0.950.4660.451
\n

It's obvious that the correct predictions for my model should be around 0.5, the correct segmentations are also complex ones as blood vessels have small ramifications etc… So the current metric is a huge bottleneck to get to good solutions to this problem.

", "rawMarkdown": "Thank you @ryanholbrook, I hope we can find a solution to this because I see a real problem here.\n\nWith the exact same submission, only changing threshold here is what I get:\n| Threshold | CV (kidney_3_sparse) | LB |\n|:---------:|:------------------:|:--------:|\n| 0.5 | 0.778 | OOM |\n| 0.85 | 0.642 | OOM |\n| 0.95 | 0.466 | 0.451 |\n\nIt's obvious that the correct predictions for my model should be around 0.5, the correct segmentations are also complex ones as blood vessels have small ramifications etc... So the current metric is a huge bottleneck to get to good solutions to this problem.\n \n", "votes": 4 } ] } ] }, { "id": 2549360, "postDate": "2023-12-05T07:17:12.793Z", "content": "

Hi @ryanholbrook
\nI express my gratitude for your efforts regarding this competition.

\n

I'm encountering ongoing problems with submission scoring errors. 😭
\nAre there any alternative options such as improving the test environment by increasing memory?

", "rawMarkdown": "Hi @ryanholbrook \nI express my gratitude for your efforts regarding this competition.\n\nI'm encountering ongoing problems with submission scoring errors. 😭\nAre there any alternative options such as improving the test environment by increasing memory?", "votes": 1, "replies": [ { "id": 2596501, "postDate": "2024-01-11T06:05:44.037Z", "content": "

Hellow, How did you resolve submission scoring errors.?I am struggling with this matter. All my submissions are related to this issue

", "rawMarkdown": "Hellow, How did you resolve submission scoring errors.?I am struggling with this matter. All my submissions are related to this issue", "votes": 1 } ] }, { "id": 2525048, "postDate": "2023-11-14T18:14:31.190Z", "content": "

Thanks for looking into this! I've proposed a possible cause of this problem in my recent discussion post, see here. Would you mind taking a look and tell us if this is the underlying issue here?

", "rawMarkdown": "Thanks for looking into this! I've proposed a possible cause of this problem in my recent discussion post, see [here](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/455062). Would you mind taking a look and tell us if this is the underlying issue here? ", "votes": 1 }, { "id": 2524402, "postDate": "2023-11-14T07:55:31.827Z", "content": "

A simple fix can be to accept \"1 1\" for empty masks. This can be modified in the description as well.

", "rawMarkdown": "A simple fix can be to accept \"1 1\" for empty masks. This can be modified in the description as well. ", "votes": 1, "replies": [ { "id": 2524483, "postDate": "2023-11-14T09:21:44.013Z", "content": "

1 1 introduces a spurious pixel into prediction which in penalized (however small) by the matric

", "rawMarkdown": "1 1 introduces a spurious pixel into prediction which in penalized (however small) by the matric", "replies": [ { "id": 2524506, "postDate": "2023-11-14T09:49:30.873Z", "content": "

Yes it does if it is not accepted as empty mask. If used in encoding the actual solution… it won't be penalized

", "rawMarkdown": "Yes it does if it is not accepted as empty mask. If used in encoding the actual solution... it won't be penalized " }, { "id": 2524861, "postDate": "2023-11-14T15:16:58.770Z", "content": "

I'm not quite sure what you mean. 1 1 is not accepted as an empty mask becase it encodes a single pixel. 1 0 encodes an empty mask but for now it results in an exception.

", "rawMarkdown": "I'm not quite sure what you mean. 1 1 is not accepted as an empty mask becase it encodes a single pixel. 1 0 encodes an empty mask but for now it results in an exception." } ] } ] }, { "id": 2613484, "postDate": "2024-01-22T05:24:48.393Z", "content": "

I am still facing the submission scoring error
\n1.I have tried hardcoding the rle to \"1 0\" for all , and it gives a score of 0-> submission works

\n
    \n
  1. I have even generated dummy mask and rles based on these masks -> submission works
    2. \n
  2. submission fails when i try predicting my model
    \nI have used \"mask = remove_small_objects(pred,10)\"
    \nI can even produce the submission file.
    \nIs there anything else I am missing out on
    \nhere's a link to my notebook
    \nhttps://www.kaggle.com/code/ayshaasif/inference-kidney-human-vasculature-segmentation
    3. \n
", "rawMarkdown": "I am still facing the submission scoring error \n1.I have tried hardcoding the rle to \"1 0\" for all , and it gives a score of 0-> submission works\n2. I have even generated dummy mask and rles based on these masks -> submission works\n3. submission fails when i try predicting my model \n I have used \"mask = remove_small_objects(pred,10)\" \nI can even produce the submission file.\nIs there anything else I am missing out on \nhere's a link to my notebook\nhttps://www.kaggle.com/code/ayshaasif/inference-kidney-human-vasculature-segmentation\n" }, { "id": 2524039, "postDate": "2023-11-14T00:21:28.860Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2535091, "author_name": "Peter Warren", "author_url": "", "post_date": "2023-11-23T05:12:16.150000", "content": "

is there any other tricks to determine what is causing the submission scoring error??

\n

I have tried a lot of different things and none of my submissions seem to score correctly. It is probably something small that I am messing up, or maybe something is still wrong with the scoring method, It is hard to tell. I just want atleast one successful submission so that I can start making improvements to my model.

", "votes": 5, "replies": [ { "id": 2535440, "author_name": "Arunodhayan", "author_url": "", "post_date": "2023-11-23T10:31:30.083000", "content": "

increase the threshold of the mask during inference or add removing small regions

", "votes": 3, "replies": [ { "id": 2536261, "author_name": "Peter Warren", "author_url": "", "post_date": "2023-11-24T04:39:38.717000", "content": "

Thank you, I finally received a score! Unfortunately the score was 0.000. Nowhere to go but up from here! 😅

", "votes": 1, "replies": [] }, { "id": 2549611, "author_name": "dragon zhang", "author_url": "", "post_date": "2023-12-05T11:42:47.197000", "content": "

could you please your code? so far, I only got best 0 too.

", "votes": 0, "replies": [] }, { "id": 2557038, "author_name": "AMAN P7952", "author_url": "", "post_date": "2023-12-11T06:42:01.943000", "content": "

Thank you! It solved my submission error.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2526815, "author_name": "Joakim Arvidsson", "author_url": "", "post_date": "2023-11-16T04:32:54.837000", "content": "

I'm still getting 'Submission Scoring Error', even though my notebook ran and generated a submission.csv file successfully.

", "votes": 6, "replies": [ { "id": 2531206, "author_name": "pangyuteng", "author_url": "", "post_date": "2023-11-20T02:55:04.927000", "content": "

same. for the curious, have provided a very simple errored-out notebook here.
\nhttps://www.kaggle.com/code/pangyuteng/frustrated-frank

", "votes": 3, "replies": [] } ] }, { "id": 2525225, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-11-14T23:33:50.920000", "content": "

Hi everyone,

\n

I just deployed an updated version of the metric that I believe fixes the problem. Submissions with 1 0 for empty masks should no longer fail. I'm planning to run a rescore on all submissions with the updated metric sometime tomorrow once I can run some additional tests.

\n

Though I believe the update should fix submissions, I'm still investigating the root cause. (The problem doesn't seem to occur when the metric is run interactively, for instance, only during scoring.) Please let me know if you are still having trouble and I will investigate.

", "votes": 3, "replies": [ { "id": 2525231, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2023-11-14T23:56:49.637000", "content": "

The last two of my submissions took nearly an hour to calculate the score (\"Submission running\" status), before that, it was done in 10ish minutes, if you could check that the update you did to the metric is not the reason, that would be helpful

\n

Edit: I just got a \"Submission Scoring Error\" on a submission where the only change was image-size and weights, please look into that too, if this could be the reason, I do not use \"1 0\" for empty mask

", "votes": 0, "replies": [ { "id": 2525270, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-11-15T01:33:32.923000", "content": "

Hi @harshitsheoran, I'm sorry to hear you're still having trouble. The longer runtimes might be due to a more complicated solution. Typically, the test submissions I've done have run in about 20-30 minutes. I wouldn't be surprised if sometimes it took up to an hour.

\n

It looks like the last submission failed due to an out-of-memory error in the scoring metric. It might be that a more complicated solution is again the culprit. I already added a number of changes to the reference metric to improve memory efficiency, and I'm not certain where to make further improvements, but I'll try to look again when I can.

\n

For reference, here is the code for the metric: Surface Dice Metric. That could help with testing submissions.

", "votes": -1, "replies": [] }, { "id": 2525286, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2023-11-15T02:12:38.140000", "content": "

Thank you for replying, I did make a submit an hour ago, this submit was a repeat of a submit which ran successfully, and it failed now, I hope that the metric becomes more efficient

", "votes": 0, "replies": [] }, { "id": 2525296, "author_name": "Arunodhayan", "author_url": "", "post_date": "2023-11-15T02:23:13.923000", "content": "

even same here @ryanholbrook previously it took 10 mins to complete a submission now it takes longer time

", "votes": 1, "replies": [] }, { "id": 2525847, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2023-11-15T12:27:07.077000", "content": "

@ryanholbrook I have failed a lot of submissions to \"Submission Scoring Error\", Even though I am only predicting one of the kidneys, this is first 24 hours of the competition and I can only imagine that the solution becomes more complicated as the competition progresses, Can you please give me an example on how to make my submission \"less-complicated\" so that it does not give the error

", "votes": 3, "replies": [] }, { "id": 2526241, "author_name": "Parabellum", "author_url": "", "post_date": "2023-11-15T16:54:08.113000", "content": "

@harshitsheoran Yesterday, I encountered a new issue with Kaggle submissions. Previously, using \"1 1\" in the RLE allowed me to submit successfully, but now, even the same notebook version that worked before is failing.

\n

To troubleshoot, I wrote some code to verify the correctness of the RLE and to check for any missing IDs in the submission dataframe. If any IDs were missing, I appended them to the dataframe with an RLE of \"1 1\". Despite these efforts, I'm still facing scoring errors. New error : Submission Scoring Error

", "votes": 1, "replies": [] }, { "id": 2526369, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2023-11-15T18:20:35.187000", "content": "

@abdrah @kashiwaba I had the same issue, it ruined a lot of submissions, but I managed to submit the correct one by making it less \"complicated\", just to test it, I changed the threshold from 0.5 to 0.9, this passed but incurred a massive penalty on score (0.423 -> 0.309) (I also changed \"1 1\" to \"1 0\" as that problem looks like it is fixed), so I would recommend trying out numbers like 0.6 or 0.7 to see if those works, I used all my submissions for today

", "votes": 1, "replies": [] }, { "id": 2526381, "author_name": "Parabellum", "author_url": "", "post_date": "2023-11-15T18:31:21.007000", "content": "

That's a smart approach – increasing the threshold could help prevent the creation of overly large files. However, as you pointed out, this might affect the final score. Since I have only one submission attempt left, I'm considering trying a threshold of 0.8, just to see if it's effective. I usually opt for a threshold of 0.5.

", "votes": 0, "replies": [] }, { "id": 2526410, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2023-11-15T19:04:02.093000", "content": "

Good Luck!, please let us know how that goes, I usually opt for 0.5 threshold too

", "votes": 0, "replies": [] }, { "id": 2527522, "author_name": "Parabellum", "author_url": "", "post_date": "2023-11-16T15:36:39.823000", "content": "

Unfortunately, my attempts were unsuccessful. I even reviewed the competition metrics, but that didn't help either. It's puzzling that some participants are achieving scores as high as 0.7 on the leaderboard.

", "votes": 1, "replies": [] }, { "id": 2531888, "author_name": "Optimo", "author_url": "", "post_date": "2023-11-20T16:12:11.907000", "content": "

@harshitsheoran what are the error logs you get with \"Submission Scoring Error\" ? I currently have this, do you have the same?

\n
\n

Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected.

\n
", "votes": 0, "replies": [] }, { "id": 2531909, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2023-11-20T16:22:13.237000", "content": "

Yes, but I think this statement might be a static and not reflective on the type of error

", "votes": 1, "replies": [] } ] }, { "id": 2526240, "author_name": "YumeNeko", "author_url": "", "post_date": "2023-11-15T16:50:49.780000", "content": "

@ryanholbrook
\nI also submitted a Notebook almost identical to the one that was graded successfully yesterday (exactly the same parameters and weights), but for some reason today it is getting a Scoring Error.

\n

I appreciate the quick response, but I would rather they revert to the previous scoring system until the underlying problem is identified than rush to release a half-baked one.

", "votes": 4, "replies": [ { "id": 2526247, "author_name": "Parabellum", "author_url": "", "post_date": "2023-11-15T16:57:58.347000", "content": "

@kashiwaba That's what I just said in the earlier comment : \"Yesterday, I encountered a new issue with Kaggle submissions. Previously, using \"1 1\" in the RLE allowed me to submit successfully, but now, even the same notebook version that worked before is failing. To troubleshoot, I wrote some code to verify the correctness of the RLE and to check for any missing IDs in the submission dataframe. If any IDs were missing, I appended them to the dataframe with an RLE of \"1 1\". Despite these efforts, I'm still facing scoring errors. New error : Submission Scoring Error\"

", "votes": 2, "replies": [] }, { "id": 2526373, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2023-11-15T18:23:09.837000", "content": "

I liked the previous one better, it was faster and did not have OOM problems between Submission-Scoring, \"1 1\" hardly impacts the score anyway if everyone is forced to use that

", "votes": 1, "replies": [] } ] }, { "id": 2596510, "author_name": "WangXuC", "author_url": "", "post_date": "2024-01-11T06:15:09.277000", "content": "

Hi @ryanholbrook , I am still facing submission scoring errors today, which is driving me crazy. Do you have any solutions?

", "votes": 2, "replies": [] } ] }, { "id": 2536958, "author_name": "Chris Cross", "author_url": "", "post_date": "2023-11-24T16:16:31.373000", "content": "

Hi, this is my first notebook only competition and currently (as of 11.24.23) not able to score on
\n the leaderboard other than with rle mask of \"1 0\". My submissions get the error:

\n
\n

Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. See more debugging tips

\n
\n

After training on one kidney i use the provided *score * (V22) function to validate on a second kidney and get score of ~ 0.50, so i would guess my submission format should be correct and as i do not get an OOM error at least on the \"local\" runs.
\nIt would be really great to get more information from the submissions failure so one knows if the error is on her/his side or if it is on the server side. Thxs and a great weekend to you all!

", "votes": 4, "replies": [ { "id": 2539566, "author_name": "Peter Warren", "author_url": "", "post_date": "2023-11-27T06:27:46.280000", "content": "

Yes, I agree, I trained one model submitted it, trained it further, all I did was change out the model for their one with more training, changed no other details in the code and it gets a submission error.

\n

I have tried using higher threshold, and also removing small objects but still get submission error. I have no idea how a dice score calculation can run out memory, also I don’t even know if that is what is failing me, it’s just difficult to determine what to fix to avoid submission scoring errors

", "votes": 2, "replies": [] } ] }, { "id": 2531884, "author_name": "Optimo", "author_url": "", "post_date": "2023-11-20T16:09:13.587000", "content": "

@ryanholbrook Can you please clarify what is the current status on the organizers' side ? Do you consider the metric computation as currently working ? Is it ok to say that one should try to make a \"simple enough\" surface prediction so that the computation goes through or are you currently working on a fix that would work regardless of the predicted structures (as long as it follows the submission format) ?

\n

Thanks for letting us know!

", "votes": 4, "replies": [ { "id": 2532176, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-11-20T19:49:03.147000", "content": "

Hi @optimo, The metric computation itself is now correct and functional so far as I'm aware. There's still the issue of OOM errors when scoring some submissions. If we're able to come up with any improvements to the metric's memory usage, I will add them. This is something I'll continue to investigate. As I don't know if or when such improvements will be possible, for now, unfortunately, I'd say that making a submission simple enough to go through will have to be part of the competition challenge. For what it's worth, I have checked that a perfect submission will indeed score correctly.

", "votes": 4, "replies": [ { "id": 2532774, "author_name": "Optimo", "author_url": "", "post_date": "2023-11-21T10:14:08.430000", "content": "

Hello @ryanholbrook,

\n

I was not expecting this answer to be honest. It seems rather vague for an official rule of the competition to make a \"simple enough\" submission so that the metric computation does not OOM. Why would this notebook's solution used to be \"simple enough\" https://www.kaggle.com/code/kashiwaba/sennet-hoa-inference-unet-simple-baseline/notebook with the previous method but now is too complicated to be considered a valid solution (it's quite a simple baseline by the way - people are now changing thresholds from 0.99 to 0.97 to see what is considered \"simple enough\", this just does not make any sense)?

\n

Also, the goal of the competition is to come up with the best possible solution, not the best possible \"simple enough\" solution. I am sure the community can come up with a better implementation of the metric but some things might need clarifications:

\n", "votes": 9, "replies": [] }, { "id": 2532983, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-11-21T14:00:18.137000", "content": "

The current version of the metric is now the same as used at the start of the competition. I'm not certain why submissions that completed before now do not complete, and why the 1 0 issue is no longer affecting submissions. It appears there was some kind of transient issue affecting the scoring environment that has now resolved.

\n", "votes": 1, "replies": [] }, { "id": 2533045, "author_name": "Optimo", "author_url": "", "post_date": "2023-11-21T14:47:52.343000", "content": "

Thank you @ryanholbrook, I hope we can find a solution to this because I see a real problem here.

\n

With the exact same submission, only changing threshold here is what I get:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ThresholdCV (kidney_3_sparse)LB
0.50.778OOM
0.850.642OOM
0.950.4660.451
\n

It's obvious that the correct predictions for my model should be around 0.5, the correct segmentations are also complex ones as blood vessels have small ramifications etc… So the current metric is a huge bottleneck to get to good solutions to this problem.

", "votes": 4, "replies": [] } ] } ] }, { "id": 2549360, "author_name": "siwooyong", "author_url": "", "post_date": "2023-12-05T07:17:12.793000", "content": "

Hi @ryanholbrook
\nI express my gratitude for your efforts regarding this competition.

\n

I'm encountering ongoing problems with submission scoring errors. 😭
\nAre there any alternative options such as improving the test environment by increasing memory?

", "votes": 1, "replies": [ { "id": 2596501, "author_name": "WangXuC", "author_url": "", "post_date": "2024-01-11T06:05:44.037000", "content": "

Hellow, How did you resolve submission scoring errors.?I am struggling with this matter. All my submissions are related to this issue

", "votes": 1, "replies": [] } ] }, { "id": 2525048, "author_name": "Felix Konrad", "author_url": "", "post_date": "2023-11-14T18:14:31.190000", "content": "

Thanks for looking into this! I've proposed a possible cause of this problem in my recent discussion post, see here. Would you mind taking a look and tell us if this is the underlying issue here?

", "votes": 1, "replies": [] }, { "id": 2524402, "author_name": "HungryLearner", "author_url": "", "post_date": "2023-11-14T07:55:31.827000", "content": "

A simple fix can be to accept \"1 1\" for empty masks. This can be modified in the description as well.

", "votes": 1, "replies": [ { "id": 2524483, "author_name": "DennisSakva", "author_url": "", "post_date": "2023-11-14T09:21:44.013000", "content": "

1 1 introduces a spurious pixel into prediction which in penalized (however small) by the matric

", "votes": 0, "replies": [ { "id": 2524506, "author_name": "HungryLearner", "author_url": "", "post_date": "2023-11-14T09:49:30.873000", "content": "

Yes it does if it is not accepted as empty mask. If used in encoding the actual solution… it won't be penalized

", "votes": 0, "replies": [] }, { "id": 2524861, "author_name": "DennisSakva", "author_url": "", "post_date": "2023-11-14T15:16:58.770000", "content": "

I'm not quite sure what you mean. 1 1 is not accepted as an empty mask becase it encodes a single pixel. 1 0 encodes an empty mask but for now it results in an exception.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2613484, "author_name": "Aysha Asif", "author_url": "", "post_date": "2024-01-22T05:24:48.393000", "content": "

I am still facing the submission scoring error
\n1.I have tried hardcoding the rle to \"1 0\" for all , and it gives a score of 0-> submission works

\n
    \n
  1. I have even generated dummy mask and rles based on these masks -> submission works
    2. \n
  2. submission fails when i try predicting my model
    \nI have used \"mask = remove_small_objects(pred,10)\"
    \nI can even produce the submission file.
    \nIs there anything else I am missing out on
    \nhere's a link to my notebook
    \nhttps://www.kaggle.com/code/ayshaasif/inference-kidney-human-vasculature-segmentation
    3. \n
", "votes": 0, "replies": [] }, { "id": 2524039, "author_name": "", "author_url": "", "post_date": "2023-11-14T00:21:28.860000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2523945": "Hi everyone,\n\nIt appears there's an issue with submissions accepting the `1 0` entries indicating empty masks. I'm still investigating the cause of the error, but I'll update here as soon as I know more.\n\nThank you,\nRyan", "2535091": "is there any other tricks to determine what is causing the submission scoring error??\n\nI have tried a lot of different things and none of my submissions seem to score correctly. It is probably something small that I am messing up, or maybe something is still wrong with the scoring method, It is hard to tell. I just want atleast one successful submission so that I can start making improvements to my model.\n", "2526815": "I'm still getting 'Submission Scoring Error', even though my notebook ran and generated a submission.csv file successfully.", "2525225": "Hi everyone,\n\nI just deployed an updated version of the metric that I believe fixes the problem. Submissions with `1 0` for empty masks should no longer fail. I'm planning to run a rescore on all submissions with the updated metric sometime tomorrow once I can run some additional tests.\n\nThough I believe the update should fix submissions, I'm still investigating the root cause. (The problem doesn't seem to occur when the metric is run interactively, for instance, only during scoring.) Please let me know if you are still having trouble and I will investigate.", "2536958": "Hi, this is my first notebook only competition and currently (as of 11.24.23) not able to score on\n the leaderboard other than with rle mask of \"1 0\". My submissions get the error: \n >Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. See more debugging tips\n\nAfter training on one kidney i use the provided *score * (V22) function to validate on a second kidney and get score of ~ 0.50, so i would guess my submission format should be correct and as i do not get an OOM error at least on the \"local\" runs. \nIt would be really great to get more information from the submissions failure so one knows if the error is on her/his side or if it is on the server side. Thxs and a great weekend to you all!\n\n", "2531884": "@ryanholbrook Can you please clarify what is the current status on the organizers' side ? Do you consider the metric computation as currently working ? Is it ok to say that one should try to make a \"simple enough\" surface prediction so that the computation goes through or are you currently working on a fix that would work regardless of the predicted structures (as long as it follows the submission format) ?\n\nThanks for letting us know!", "2549360": "Hi @ryanholbrook \nI express my gratitude for your efforts regarding this competition.\n\nI'm encountering ongoing problems with submission scoring errors. 😭\nAre there any alternative options such as improving the test environment by increasing memory?", "2525048": "Thanks for looking into this! I've proposed a possible cause of this problem in my recent discussion post, see [here](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/455062). Would you mind taking a look and tell us if this is the underlying issue here? ", "2524402": "A simple fix can be to accept \"1 1\" for empty masks. This can be modified in the description as well. ", "2613484": "I am still facing the submission scoring error \n1.I have tried hardcoding the rle to \"1 0\" for all , and it gives a score of 0-> submission works\n2. I have even generated dummy mask and rles based on these masks -> submission works\n3. submission fails when i try predicting my model \n I have used \"mask = remove_small_objects(pred,10)\" \nI can even produce the submission file.\nIs there anything else I am missing out on \nhere's a link to my notebook\nhttps://www.kaggle.com/code/ayshaasif/inference-kidney-human-vasculature-segmentation\n", "2524039": "" } }