![\"hanging](\"https://i.imgur.com/h8nHJex.pngf\")
It has been frustrating to try and get any submissions through with this and other code competitions.
\nThe feedback from the errors is not intuitive. I understand the value of code competitions but if the public submission runs, then the test one should be able to run as well (the only exceptions should be time outs and even on that effort should be made for public and test to be comparable).
\nI have wasted 3 days trying to make this work and I still have no clue what the error is. Taking into account that GPU time is limited (half of my quota is used trying to debug so far and our time is limited as well), code competitions need to be restructured in my opinion.
\nAlso submissions are hanging (again).
\n\nEDIT: I have tracked the error for why the latest sub failed (This is not the only error I had faced though - I am sure about it). I forgot to put a header! I feel bad because this is a silly error - still a simple message or something could have prevented me for wasting so much time on it.
\nIt has been frustrating to try and get any submissions through with this and other code competitions.
\nThe feedback from the errors is not intuitive. I understand the value of code competitions but if the public submission runs, then the test one should be able to run as well (the only exceptions should be time outs and even on that effort should be made for public and test to be comparable).
\nI have wasted 3 days trying to make this work and I still have no clue what the error is. Taking into account that GPU time is limited (half of my quota is used trying to debug so far and our time is limited as well), code competitions need to be restructured in my opinion.
\nAlso submissions are hanging (again).
\n\nEDIT: I have tracked the error for why the latest sub failed (This is not the only error I had faced though - I am sure about it). I forgot to put a header! I feel bad because this is a silly error - still a simple message or something could have prevented me for wasting so much time on it.
\nI spent about a week. Finally I've got a score. If it hanging than it is a error. Not need to wait. Can be a lot of reasons for the error:
\nThis is really frustrating. Unfortunately, this has been my experience with the majority of Kaggle \"code\" competitions. I understand the rationale behind them, but in my mind the downsides far outweighs the benefits.
\nThe changes on the Kaggle platform over the past couple of years have already prompted a large exodus of legendary Kagglers into semi-retirement. It would be a shame if this trend continues.
", "rawMarkdown": "This is really frustrating. Unfortunately, this has been my experience with the majority of Kaggle \"code\" competitions. I understand the rationale behind them, but in my mind the downsides far outweighs the benefits. \n\nThe changes on the Kaggle platform over the past couple of years have already prompted a large exodus of legendary Kagglers into semi-retirement. It would be a shame if this trend continues.", "votes": null }, { "id": "1265293", "postDate": "04/06/2021 19:13:07", "content": "The reasons are many but you don't know the exact cause as it's hidden by the script. This makes debugging a nightmare.
", "rawMarkdown": "The reasons are many but you don't know the exact cause as it's hidden by the script. This makes debugging a nightmare.", "votes": null }, { "id": "1265742", "postDate": "04/07/2021 07:15:52", "content": "@kazanova for us, the problem was trying to access the csv file which is not upddated for private test set:
\nI think it is counter-intuitive to give information for public test set but not for private set. To avoid errors related to private set you need to only rely on the images themselves.
", "rawMarkdown": "kazanova for us, the problem was trying to access the csv file which is not upddated for private test set:\n- https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/228654\n- https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/201506\n\nI think it is counter-intuitive to give information for public test set but not for private set. To avoid errors related to private set you need to only rely on the images themselves.", "votes": null }, { "id": "1265839", "postDate": "04/07/2021 08:48:30", "content": "@vedenev i m new to join the party of Sub scoring error. I hve been checking through the discussions.
\nbased on this what i find is that sub can get into scoring error out because of
\n1) OOM -RAM - Sub csv not found
\n2) GPU full- Sub scoring error because not all image id are found for scoring as inference ends half way.
In my case i had monitored RAM max it reaches is till 9.8 GB and then instantly gets released for image size 33k by 43 k , so there is buffer of atleast 2.5 gb for any higher size images than this.
\nMy kernel runs for an hour before its gets into Sub scoring error.
\nCan there be any other reasons for this sub scoring error
", "rawMarkdown": "vedenev i m new to join the party of Sub scoring error. I hve been checking through the discussions.\nbased on this what i find is that sub can get into scoring error out because of\n1) OOM -RAM - Sub csv not found\n2) GPU full- Sub scoring error because not all image id are found for scoring as inference ends half way.\n\nIn my case i had monitored RAM max it reaches is till 9.8 GB and then instantly gets released for image size 33k by 43 k , so there is buffer of atleast 2.5 gb for any higher size images than this.\n\nMy kernel runs for an hour before its gets into Sub scoring error. \n\n Can there be any other reasons for this sub scoring error", "votes": null }, { "id": "1265873", "postDate": "04/07/2021 09:14:30", "content": "\n\n1) OOM -RAM - Sub csv not found
\n
Thx for sharing that. I have been struggling with this error now. I also think is probably OOM.
\nOn top of al the ambiguity surrounding the error, my submissions are also used-up, so I have limited chances to debug!
\n![\"subnotfound\"](\"https://i.imgur.com/W8AFbXB.png\")
I somewhat agree that the errors could be more verbose… yet I think in principal code/kernel competitions will become more common in the future
\nI think that difficulties like those associated with submission through a kernel are very similar to those you’d encounter when having to troubleshoot a problem in the real world.
\nThere are techniques to approach these types of problems like starting with very simple submissions and working up the complexity. Kaggle has also put together an extensive doc showing how to troubleshoot the various error messages.
\nI’ve beat my head against this proverbial wall many times and the way around it has always been to troubleshoot in a calm methodical manner that assumes nothing and checks everything and makes use of all available resources. I’m not an expert by any means, but often it is something simple like you mentioned above.
\nAs to why kernel competitions will be more common in the future: real world ML/DL tasks are more and more frequently being constrained to run in near real-time (or within a reasonable time allowance). Companies won’t often want algorithms that take hours for inference or have incredibly complicated pipelines (there are resourcing costs all over the place associated with this type of thing). They often want relatively fast, reproducible results and a modelling framework that allows for continuous improvement/development w.r.t retraining.
\nAs an aside, I think the kernel competitions create an environment where more focus is placed on “cleverness” and less on “how big are my GPUs”.
\nThat being said, I think as you proposed, some better error reporting would help alleviate many frustrations related to more obvious formatting problems.
\n—-
\nOn a side note:
\nI’d be interested to see competitions with very strict inference timing constraints. A competition that would reward those that could utilize SOTA compression/quantization to maintain accuracy while decreasing latency. I think that would be cool and I’d be excited to give it a shot.
I agree with the value of kernel competitions.
\nHowever,
\n\n\nI think that difficulties like those associated with submission through a kernel are very similar to those you’d encounter when having to troubleshoot a problem in the real world.
\n
In the real world, you normally have more info or have the ability to get more info regarding the error - like where the code fails. Now I have 1000 lines of code where the code could have failed anywhere and I have no clue what the failure is. In fact, I have never encountered something similar in the real world.
\n\n\nThere are techniques to approach these types of problems like starting with very simple submissions and working up the complexity.
\n
For me, this would mean, I can only do one competition at a time. Not necessarily saying is bad, but it is not what I was used to. I like to join many competitions and learn/pick up the different techniques/approaches presented and try them . Now I cannot do that. This is no fun! Needless to say that my quota is almost used-up and I have made very little progress. Even if I wanted to try something else, I cannot (given what I have in the pipeline). I have to wait for next week.
\nIt is good for kaggle to have a touch with the real world, but it is still a competition platform for data science - not an engineering competition platform ( ,there are other platforms if you want to do that ) where I need to spend half of the competition trying to troubleshoot the cryptic errors that come out of the system.
", "rawMarkdown": "I agree with the value of kernel competitions.\n\nHowever,\n\n>I think that difficulties like those associated with submission through a kernel are very similar to those you’d encounter when having to troubleshoot a problem in the real world.\n\nIn the real world, you normally have more info or have the ability to get more info regarding the error - like where the code fails. Now I have 1000 lines of code where the code could have failed anywhere and I have no clue what the failure is. In fact, I have never encountered something similar in the real world.\n\n>There are techniques to approach these types of problems like starting with very simple submissions and working up the complexity. \n\nFor me, this would mean, I can only do one competition at a time. Not necessarily saying is bad, but it is not what I was used to. I like to join many competitions and learn/pick up the different techniques/approaches presented and try them . Now I cannot do that. This is no fun! Needless to say that my quota is almost used-up and I have made very little progress. Even if I wanted to try something else, I cannot (given what I have in the pipeline). I have to wait for next week.\n\nIt is good for kaggle to have a touch with the real world, but it is still a competition platform for data science - not an engineering competition platform ( ,there are other platforms if you want to do that ) where I need to spend half of the competition trying to troubleshoot the cryptic errors that come out of the system.", "votes": null }, { "id": "1266034", "postDate": "04/07/2021 12:31:55", "content": "I can definitely respect that. I appreciate the reply. I hope they rework the error messages to make things clearer so that no one has to struggle! I can definitely appreciate wanting to do more without having to fight against cryptic errors.
", "rawMarkdown": "I can definitely respect that. I appreciate the reply. I hope they rework the error messages to make things clearer so that no one has to struggle! I can definitely appreciate wanting to do more without having to fight against cryptic errors.", "votes": null }, { "id": "1267760", "postDate": "04/08/2021 18:52:49", "content": "\n\ncode competitions are a pain!
\n
Sir, can I ultra-upvote you?. Thanks.
", "rawMarkdown": "> code competitions are a pain!\n\nSir, can I ultra-upvote you?. Thanks.", "votes": null }, { "id": "1267766", "postDate": "04/08/2021 19:00:29", "content": "\n\nSir, can I ultra-upvote you?. Thanks.
\n
Why, thank you sir!
", "rawMarkdown": "> Sir, can I ultra-upvote you?. Thanks.\n\nWhy, thank you sir!", "votes": null }, { "id": "1267905", "postDate": "04/08/2021 23:49:18", "content": "I can completely relate to this once I faced 10 such consecutive scoring errors in Riiid answer correctness prediction Competition .
\nI found out that on private test my kernel was exceeding 9hrs time limit , but during commit it was okay and I came to know this only because I used an emulator code published by a Kaggle GM which helped me track error.
\nSure Kaggle's code Competition and those involving module code like Api are really difficult to debug.They should improve verbosity of submission scoring errors as this errors are frustrating and takes away major time .
", "rawMarkdown": "I can completely relate to this once I faced 10 such consecutive scoring errors in Riiid answer correctness prediction Competition .\n\nI found out that on private test my kernel was exceeding 9hrs time limit , but during commit it was okay and I came to know this only because I used an emulator code published by a Kaggle GM which helped me track error.\n\nSure Kaggle's code Competition and those involving module code like Api are really difficult to debug.They should improve verbosity of submission scoring errors as this errors are frustrating and takes away major time .", "votes": null }, { "id": "1273750", "postDate": "04/14/2021 15:43:37", "content": "For those Submission Scoring Error, Endless running of submission, Submission CSV Not Found
\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227225
I spent about a week. Finally I've got a score. If it hanging than it is a error. Not need to wait. Can be a lot of reasons for the error:
\nThe reasons are many but you don't know the exact cause as it's hidden by the script. This makes debugging a nightmare.
", "votes": null, "replies": [] }, { "id": 1265839, "author_name": "jaideepvalani", "author_url": "", "post_date": "04/07/2021 08:48:30", "content": "@vedenev i m new to join the party of Sub scoring error. I hve been checking through the discussions.
\nbased on this what i find is that sub can get into scoring error out because of
\n1) OOM -RAM - Sub csv not found
\n2) GPU full- Sub scoring error because not all image id are found for scoring as inference ends half way.
In my case i had monitored RAM max it reaches is till 9.8 GB and then instantly gets released for image size 33k by 43 k , so there is buffer of atleast 2.5 gb for any higher size images than this.
\nMy kernel runs for an hour before its gets into Sub scoring error.
\nCan there be any other reasons for this sub scoring error
", "votes": null, "replies": [] }, { "id": 1265873, "author_name": "kazanova", "author_url": "", "post_date": "04/07/2021 09:14:30", "content": "\n\n1) OOM -RAM - Sub csv not found
\n
Thx for sharing that. I have been struggling with this error now. I also think is probably OOM.
\nOn top of al the ambiguity surrounding the error, my submissions are also used-up, so I have limited chances to debug!
\n
For those Submission Scoring Error, Endless running of submission, Submission CSV Not Found
\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227225
This is really frustrating. Unfortunately, this has been my experience with the majority of Kaggle \"code\" competitions. I understand the rationale behind them, but in my mind the downsides far outweighs the benefits.
\nThe changes on the Kaggle platform over the past couple of years have already prompted a large exodus of legendary Kagglers into semi-retirement. It would be a shame if this trend continues.
", "votes": null, "replies": [] }, { "id": 1265742, "author_name": "optimo", "author_url": "", "post_date": "04/07/2021 07:15:52", "content": "@kazanova for us, the problem was trying to access the csv file which is not upddated for private test set:
\nI think it is counter-intuitive to give information for public test set but not for private set. To avoid errors related to private set you need to only rely on the images themselves.
", "votes": null, "replies": [] }, { "id": 1265924, "author_name": "dschettler8845", "author_url": "", "post_date": "04/07/2021 10:37:03", "content": "I somewhat agree that the errors could be more verbose… yet I think in principal code/kernel competitions will become more common in the future
\nI think that difficulties like those associated with submission through a kernel are very similar to those you’d encounter when having to troubleshoot a problem in the real world.
\nThere are techniques to approach these types of problems like starting with very simple submissions and working up the complexity. Kaggle has also put together an extensive doc showing how to troubleshoot the various error messages.
\nI’ve beat my head against this proverbial wall many times and the way around it has always been to troubleshoot in a calm methodical manner that assumes nothing and checks everything and makes use of all available resources. I’m not an expert by any means, but often it is something simple like you mentioned above.
\nAs to why kernel competitions will be more common in the future: real world ML/DL tasks are more and more frequently being constrained to run in near real-time (or within a reasonable time allowance). Companies won’t often want algorithms that take hours for inference or have incredibly complicated pipelines (there are resourcing costs all over the place associated with this type of thing). They often want relatively fast, reproducible results and a modelling framework that allows for continuous improvement/development w.r.t retraining.
\nAs an aside, I think the kernel competitions create an environment where more focus is placed on “cleverness” and less on “how big are my GPUs”.
\nThat being said, I think as you proposed, some better error reporting would help alleviate many frustrations related to more obvious formatting problems.
\n—-
\nOn a side note:
\nI’d be interested to see competitions with very strict inference timing constraints. A competition that would reward those that could utilize SOTA compression/quantization to maintain accuracy while decreasing latency. I think that would be cool and I’d be excited to give it a shot.
I agree with the value of kernel competitions.
\nHowever,
\n\n\nI think that difficulties like those associated with submission through a kernel are very similar to those you’d encounter when having to troubleshoot a problem in the real world.
\n
In the real world, you normally have more info or have the ability to get more info regarding the error - like where the code fails. Now I have 1000 lines of code where the code could have failed anywhere and I have no clue what the failure is. In fact, I have never encountered something similar in the real world.
\n\n\nThere are techniques to approach these types of problems like starting with very simple submissions and working up the complexity.
\n
For me, this would mean, I can only do one competition at a time. Not necessarily saying is bad, but it is not what I was used to. I like to join many competitions and learn/pick up the different techniques/approaches presented and try them . Now I cannot do that. This is no fun! Needless to say that my quota is almost used-up and I have made very little progress. Even if I wanted to try something else, I cannot (given what I have in the pipeline). I have to wait for next week.
\nIt is good for kaggle to have a touch with the real world, but it is still a competition platform for data science - not an engineering competition platform ( ,there are other platforms if you want to do that ) where I need to spend half of the competition trying to troubleshoot the cryptic errors that come out of the system.
", "votes": null, "replies": [] }, { "id": 1266034, "author_name": "dschettler8845", "author_url": "", "post_date": "04/07/2021 12:31:55", "content": "I can definitely respect that. I appreciate the reply. I hope they rework the error messages to make things clearer so that no one has to struggle! I can definitely appreciate wanting to do more without having to fight against cryptic errors.
", "votes": null, "replies": [] } ] }, { "id": 1267760, "author_name": "carloshuertas", "author_url": "", "post_date": "04/08/2021 18:52:49", "content": "\n\ncode competitions are a pain!
\n
Sir, can I ultra-upvote you?. Thanks.
", "votes": null, "replies": [ { "id": 1267766, "author_name": "kazanova", "author_url": "", "post_date": "04/08/2021 19:00:29", "content": "\n\nSir, can I ultra-upvote you?. Thanks.
\n
Why, thank you sir!
", "votes": null, "replies": [] } ] }, { "id": 1267905, "author_name": "sayedathar11", "author_url": "", "post_date": "04/08/2021 23:49:18", "content": "I can completely relate to this once I faced 10 such consecutive scoring errors in Riiid answer correctness prediction Competition .
\nI found out that on private test my kernel was exceeding 9hrs time limit , but during commit it was okay and I came to know this only because I used an emulator code published by a Kaggle GM which helped me track error.
\nSure Kaggle's code Competition and those involving module code like Api are really difficult to debug.They should improve verbosity of submission scoring errors as this errors are frustrating and takes away major time .
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1264647": "It has been frustrating to try and get any submissions through with this and other code competitions. \n\nThe feedback from the errors is not intuitive. I understand the value of code competitions but if the public submission runs, then the test one should be able to run as well (the only exceptions should be time outs and even on that effort should be made for public and test to be comparable). \n\nI have wasted 3 days trying to make this work and I still have no clue what the error is. Taking into account that GPU time is limited (half of my quota is used trying to debug so far and our time is limited as well), code competitions need to be restructured in my opinion. \n\nAlso submissions are hanging (again). \n\ncc @addisonhoward \n\nEDIT: I have tracked the error for why the latest sub failed (This is not the only error I had faced though - I am sure about it). I forgot to put a header! I feel bad because this is a silly error - still a simple message or something could have prevented me for wasting so much time on it. \n\n", "1264673": "I spent about a week. Finally I've got a score. If it hanging than it is a error. Not need to wait. Can be a lot of reasons for the error:\n- out of memory\n- a bug in scoring algorithm at specific cases\n- error of your algorithm at specific images (data that scoring is done is deffer then train or test dataset, actually it is not public but private dataset )", "1264784": "This is really frustrating. Unfortunately, this has been my experience with the majority of Kaggle \"code\" competitions. I understand the rationale behind them, but in my mind the downsides far outweighs the benefits. \n\nThe changes on the Kaggle platform over the past couple of years have already prompted a large exodus of legendary Kagglers into semi-retirement. It would be a shame if this trend continues.", "1265293": "The reasons are many but you don't know the exact cause as it's hidden by the script. This makes debugging a nightmare.", "1265742": "kazanova for us, the problem was trying to access the csv file which is not upddated for private test set:\n- https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/228654\n- https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/201506\n\nI think it is counter-intuitive to give information for public test set but not for private set. To avoid errors related to private set you need to only rely on the images themselves.", "1265839": "vedenev i m new to join the party of Sub scoring error. I hve been checking through the discussions.\nbased on this what i find is that sub can get into scoring error out because of\n1) OOM -RAM - Sub csv not found\n2) GPU full- Sub scoring error because not all image id are found for scoring as inference ends half way.\n\nIn my case i had monitored RAM max it reaches is till 9.8 GB and then instantly gets released for image size 33k by 43 k , so there is buffer of atleast 2.5 gb for any higher size images than this.\n\nMy kernel runs for an hour before its gets into Sub scoring error. \n\n Can there be any other reasons for this sub scoring error", "1265873": ">1) OOM -RAM - Sub csv not found\n\nThx for sharing that. I have been struggling with this error now. I also think is probably OOM. \n\nOn top of al the ambiguity surrounding the error, my submissions are also used-up, so I have limited chances to debug!\n\n", "1265924": "I somewhat agree that the errors could be more verbose... yet I think in principal code/kernel competitions will become more common in the future\n\nI think that difficulties like those associated with submission through a kernel are very similar to those you’d encounter when having to troubleshoot a problem in the real world.\n\nThere are techniques to approach these types of problems like starting with very simple submissions and working up the complexity. Kaggle has also put together an extensive doc showing how to troubleshoot the various error messages.\n\nI’ve beat my head against this proverbial wall many times and the way around it has always been to troubleshoot in a calm methodical manner that assumes nothing and checks everything and makes use of all available resources. I’m not an expert by any means, but often it is something simple like you mentioned above.\n\nAs to why kernel competitions will be more common in the future: real world ML/DL tasks are more and more frequently being constrained to run in near real-time (or within a reasonable time allowance). Companies won’t often want algorithms that take hours for inference or have incredibly complicated pipelines (there are resourcing costs all over the place associated with this type of thing). They often want relatively fast, reproducible results and a modelling framework that allows for continuous improvement/development w.r.t retraining. \n\nAs an aside, I think the kernel competitions create an environment where more focus is placed on “cleverness” and less on “how big are my GPUs”. \n\nThat being said, I think as you proposed, some better error reporting would help alleviate many frustrations related to more obvious formatting problems.\n\n—-\n\nOn a side note: \nI’d be interested to see competitions with very strict inference timing constraints. A competition that would reward those that could utilize SOTA compression/quantization to maintain accuracy while decreasing latency. I think that would be cool and I’d be excited to give it a shot.", "1265939": "I agree with the value of kernel competitions.\n\nHowever,\n\n>I think that difficulties like those associated with submission through a kernel are very similar to those you’d encounter when having to troubleshoot a problem in the real world.\n\nIn the real world, you normally have more info or have the ability to get more info regarding the error - like where the code fails. Now I have 1000 lines of code where the code could have failed anywhere and I have no clue what the failure is. In fact, I have never encountered something similar in the real world.\n\n>There are techniques to approach these types of problems like starting with very simple submissions and working up the complexity. \n\nFor me, this would mean, I can only do one competition at a time. Not necessarily saying is bad, but it is not what I was used to. I like to join many competitions and learn/pick up the different techniques/approaches presented and try them . Now I cannot do that. This is no fun! Needless to say that my quota is almost used-up and I have made very little progress. Even if I wanted to try something else, I cannot (given what I have in the pipeline). I have to wait for next week.\n\nIt is good for kaggle to have a touch with the real world, but it is still a competition platform for data science - not an engineering competition platform ( ,there are other platforms if you want to do that ) where I need to spend half of the competition trying to troubleshoot the cryptic errors that come out of the system.", "1266034": "I can definitely respect that. I appreciate the reply. I hope they rework the error messages to make things clearer so that no one has to struggle! I can definitely appreciate wanting to do more without having to fight against cryptic errors.", "1267760": "> code competitions are a pain!\n\nSir, can I ultra-upvote you?. Thanks.", "1267766": "> Sir, can I ultra-upvote you?. Thanks.\n\nWhy, thank you sir!", "1267905": "I can completely relate to this once I faced 10 such consecutive scoring errors in Riiid answer correctness prediction Competition .\n\nI found out that on private test my kernel was exceeding 9hrs time limit , but during commit it was okay and I came to know this only because I used an emulator code published by a Kaggle GM which helped me track error.\n\nSure Kaggle's code Competition and those involving module code like Api are really difficult to debug.They should improve verbosity of submission scoring errors as this errors are frustrating and takes away major time .", "1273750": "For those Submission Scoring Error, Endless running of submission, Submission CSV Not Found\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227225" }, "source": "meta" }