I have same pipeline but using different models (all lightgbm models), the submission returning an error: Submission Scoring Error
\nAnyone facing the same issue?
", "messages": [ { "id": "1401215", "postDate": "07/27/2021 05:27:49", "content": "I have same pipeline but using different models (all lightgbm models), the submission returning an error: Submission Scoring Error
\nAnyone facing the same issue?
", "rawMarkdown": "I have same pipeline but using different models (all lightgbm models), the submission returning an error: **Submission Scoring Error**\n\n\nAnyone facing the same issue?", "votes": null }, { "id": "1401237", "postDate": "07/27/2021 06:13:13", "content": "Any kaggle staff can help? @wcukierski @juliaelliott
\nPossible to have a check on my notebook? It's simple notebook for inference only.
\nhttps://www.kaggle.com/lsw1993/999-sub (Made it visible to public for temporary review)
", "rawMarkdown": "Any kaggle staff can help? @wcukierski @juliaelliott \n\nPossible to have a check on my notebook? It's simple notebook for inference only. \n\nhttps://www.kaggle.com/lsw1993/999-sub (Made it visible to public for temporary review)", "votes": null }, { "id": "1401241", "postDate": "07/27/2021 06:20:34", "content": "Feel sucks - wasted all the submissions - but cannot get it working.
\nWhat's going on? It went well on test but the error only occurs during submission…. @@ Really hate this.
", "rawMarkdown": "Feel sucks - wasted all the submissions - but cannot get it working. \n\nWhat's going on? It went well on test but the error only occurs during submission.... @@ Really hate this.", "votes": null }, { "id": "1401464", "postDate": "07/27/2021 10:48:26", "content": "@lsw1993
\nYou can check your submission with API emulator. Hope this notebook helps:
\nhttps://www.kaggle.com/nyanpn/api-emulator-for-debugging-your-code-locally
Thanks but it's perfectly fine….. To put into context, it's same models with different parameters.
", "rawMarkdown": "https://ibb.co/BcQytH7\n\nThanks but it's perfectly fine..... To put into context, it's same models with different parameters.", "votes": null }, { "id": "1401592", "postDate": "07/27/2021 12:27:28", "content": "@wcukierski @juliaelliott
\nMy other queries:
Is there a limit to the Kaggle dataset I am using (or connecting to the submission)?
If test had returned smoothly, why my final submission will fail?
Is it possible not to count failed submissions?
Hey @lsw1993, we unfortunately cannot provide personal debugging support, but https://www.kaggle.com/code-competition-debugging provides answers to some of your questions. On (1), there is no limit to using a dataset you have attached to a notebook. Good luck!
", "rawMarkdown": "Hey @lsw1993, we unfortunately cannot provide personal debugging support, but https://www.kaggle.com/code-competition-debugging provides answers to some of your questions. On (1), there is no limit to using a dataset you have attached to a notebook. Good luck!", "votes": null }, { "id": "1401642", "postDate": "07/27/2021 13:10:11", "content": "(3)? Considering the competition is ending.
", "rawMarkdown": "(3)? Considering the competition is ending.", "votes": null }, { "id": "1401644", "postDate": "07/27/2021 13:12:16", "content": "Made my notebook public at: https://www.kaggle.com/lsw1993/999-sub
", "rawMarkdown": "Made my notebook public at: https://www.kaggle.com/lsw1993/999-sub", "votes": null }, { "id": "1401667", "postDate": "07/27/2021 13:31:23", "content": "Per the code debugging page:
\n\n", "rawMarkdown": "Per the code debugging page:\n\n> Submissions that error also count towards your team’s daily submission limit, otherwise such submissions could be used to mine for hidden information.", "votes": null }, { "id": "1401691", "postDate": "07/27/2021 13:51:41", "content": "\nSubmissions that error also count towards your team’s daily submission limit, otherwise such submissions could be used to mine for hidden information.
\n
Wouldn't it be systemic error/shortage if test is successful but not the final submission?
\nFurther more details to be shown would be easier for us to debug moving forwards in my opinion.
Perhaps a printed line of last run error would be suffice.
", "rawMarkdown": "wcukierski @juliaelliott\n\nWouldn't it be systemic error/shortage if test is successful but not the final submission? \nFurther more details to be shown would be easier for us to debug moving forwards in my opinion. \n\nPerhaps a printed line of last run error would be suffice.", "votes": null }, { "id": "1401845", "postDate": "07/27/2021 15:43:28", "content": "Will - it still amazes me that kaggle staff believes that folks can mine for hidden information - for sure the future is hidden - what the heck could someone learn by mining in this competition.
\nIt's fine that this is the company line but for the furtherance of my statistical training please provide a single sample of what could be learned by mining when the final private test has yet to occur and all the data used in the public LB has already been shared.
", "rawMarkdown": "[Will](https://www.kaggle.com/wcukierski) - it still amazes me that kaggle staff believes that folks can mine for hidden information - for sure the future is hidden - what the heck could someone learn by mining in this competition. \n\nIt's fine that this is the company line but for the furtherance of my statistical training please provide a single sample of what could be learned by mining when the final private test has yet to occur and all the data used in the public LB has already been shared.", "votes": null }, { "id": "1401926", "postDate": "07/27/2021 16:42:26", "content": "@pcjimmmy prior to \"sync rerun\" code competitions, competitions like this would suffer from large failure rates during the 2nd stage (as high as 40% in the worst cases). Participants would adapt to the realities of the stage one dataset but were not incentivized to future proof their models. When new data would arrive and there is a 0 where there had never been a 0 before, code failed, we can't intervene, and lots of unhappy folks were left without a rank.
\nThe benefit of a truly blind holdout set (whether the data is in the future or not) is that it front loads this \"tough love\" into stage one. For example, let's pretend the holdout set has a funny value that says a player is -92 ft tall. If you can see this fact, you might choose to patch or ignore that row. If you can't, you are pushed to make your code resilient to entire classes of bad inputs, missing values, impossible data events, and so on. The net result of this has been a dramatic reduction in rerun failure rates.
\nI don't deny this can be frustrating. I also agree the probing safeguards are less relevant to true predict-the-future competitions. The pain we endure now is to create robust ML models that will run correctly on data they've never seen.
", "rawMarkdown": "pcjimmmy prior to \"sync rerun\" code competitions, competitions like this would suffer from large failure rates during the 2nd stage (as high as 40% in the worst cases). Participants would adapt to the realities of the stage one dataset but were not incentivized to future proof their models. When new data would arrive and there is a 0 where there had never been a 0 before, code failed, we can't intervene, and lots of unhappy folks were left without a rank.\n\nThe benefit of a truly blind holdout set (whether the data is in the future or not) is that it front loads this \"tough love\" into stage one. For example, let's pretend the holdout set has a funny value that says a player is -92 ft tall. If you can see this fact, you might choose to patch or ignore that row. If you can't, you are pushed to make your code resilient to entire classes of bad inputs, missing values, impossible data events, and so on. The net result of this has been a dramatic reduction in rerun failure rates.\n\nI don't deny this can be frustrating. I also agree the probing safeguards are less relevant to true predict-the-future competitions. The pain we endure now is to create robust ML models that will run correctly on data they've never seen.", "votes": null }, { "id": "1401940", "postDate": "07/27/2021 16:55:46", "content": "OK - I guess I get it. Self driving cars can be made more effective if they crash and no error data is available to the engineer - hence they have to code against all possible errors. But of course only 5 crashes per day will be permitted :)
", "rawMarkdown": "OK - I guess I get it. Self driving cars can be made more effective if they crash and no error data is available to the engineer - hence they have to code against all possible errors. But of course only 5 crashes per day will be permitted :)", "votes": null }, { "id": "1401993", "postDate": "07/27/2021 17:42:03", "content": "Maybe the status can be made verbose, rather than just as such:
\n", "rawMarkdown": "Maybe the status can be made verbose, rather than just as such:\n\nhttps://ibb.co/zR4DYH5", "votes": null }, { "id": "1402890", "postDate": "07/28/2021 16:00:23", "content": "I can't seem to access the notebook
", "rawMarkdown": "I can't seem to access the notebook", "votes": null } ], "comments": [ { "id": 1401237, "author_name": "lsw1993", "author_url": "", "post_date": "07/27/2021 06:13:13", "content": "Any kaggle staff can help? @wcukierski @juliaelliott
\nPossible to have a check on my notebook? It's simple notebook for inference only.
\nhttps://www.kaggle.com/lsw1993/999-sub (Made it visible to public for temporary review)
", "votes": null, "replies": [ { "id": 1401592, "author_name": "lsw1993", "author_url": "", "post_date": "07/27/2021 12:27:28", "content": "@wcukierski @juliaelliott
\nMy other queries:
Is there a limit to the Kaggle dataset I am using (or connecting to the submission)?
If test had returned smoothly, why my final submission will fail?
Is it possible not to count failed submissions?
Hey @lsw1993, we unfortunately cannot provide personal debugging support, but https://www.kaggle.com/code-competition-debugging provides answers to some of your questions. On (1), there is no limit to using a dataset you have attached to a notebook. Good luck!
", "votes": null, "replies": [] }, { "id": 1401642, "author_name": "lsw1993", "author_url": "", "post_date": "07/27/2021 13:10:11", "content": "(3)? Considering the competition is ending.
", "votes": null, "replies": [] }, { "id": 1401667, "author_name": "wcukierski", "author_url": "", "post_date": "07/27/2021 13:31:23", "content": "Per the code debugging page:
\n\n", "votes": null, "replies": [] }, { "id": 1401691, "author_name": "lsw1993", "author_url": "", "post_date": "07/27/2021 13:51:41", "content": "\nSubmissions that error also count towards your team’s daily submission limit, otherwise such submissions could be used to mine for hidden information.
\n
Wouldn't it be systemic error/shortage if test is successful but not the final submission?
\nFurther more details to be shown would be easier for us to debug moving forwards in my opinion.
Perhaps a printed line of last run error would be suffice.
", "votes": null, "replies": [] }, { "id": 1401845, "author_name": "pcjimmmy", "author_url": "", "post_date": "07/27/2021 15:43:28", "content": "Will - it still amazes me that kaggle staff believes that folks can mine for hidden information - for sure the future is hidden - what the heck could someone learn by mining in this competition.
\nIt's fine that this is the company line but for the furtherance of my statistical training please provide a single sample of what could be learned by mining when the final private test has yet to occur and all the data used in the public LB has already been shared.
", "votes": null, "replies": [] }, { "id": 1401926, "author_name": "wcukierski", "author_url": "", "post_date": "07/27/2021 16:42:26", "content": "@pcjimmmy prior to \"sync rerun\" code competitions, competitions like this would suffer from large failure rates during the 2nd stage (as high as 40% in the worst cases). Participants would adapt to the realities of the stage one dataset but were not incentivized to future proof their models. When new data would arrive and there is a 0 where there had never been a 0 before, code failed, we can't intervene, and lots of unhappy folks were left without a rank.
\nThe benefit of a truly blind holdout set (whether the data is in the future or not) is that it front loads this \"tough love\" into stage one. For example, let's pretend the holdout set has a funny value that says a player is -92 ft tall. If you can see this fact, you might choose to patch or ignore that row. If you can't, you are pushed to make your code resilient to entire classes of bad inputs, missing values, impossible data events, and so on. The net result of this has been a dramatic reduction in rerun failure rates.
\nI don't deny this can be frustrating. I also agree the probing safeguards are less relevant to true predict-the-future competitions. The pain we endure now is to create robust ML models that will run correctly on data they've never seen.
", "votes": null, "replies": [] }, { "id": 1401940, "author_name": "pcjimmmy", "author_url": "", "post_date": "07/27/2021 16:55:46", "content": "OK - I guess I get it. Self driving cars can be made more effective if they crash and no error data is available to the engineer - hence they have to code against all possible errors. But of course only 5 crashes per day will be permitted :)
", "votes": null, "replies": [] }, { "id": 1401993, "author_name": "lsw1993", "author_url": "", "post_date": "07/27/2021 17:42:03", "content": "Maybe the status can be made verbose, rather than just as such:
\n", "votes": null, "replies": [] } ] }, { "id": 1401241, "author_name": "lsw1993", "author_url": "", "post_date": "07/27/2021 06:20:34", "content": "Feel sucks - wasted all the submissions - but cannot get it working.
\nWhat's going on? It went well on test but the error only occurs during submission…. @@ Really hate this.
", "votes": null, "replies": [] }, { "id": 1401464, "author_name": "nyanpn", "author_url": "", "post_date": "07/27/2021 10:48:26", "content": "@lsw1993
\nYou can check your submission with API emulator. Hope this notebook helps:
\nhttps://www.kaggle.com/nyanpn/api-emulator-for-debugging-your-code-locally
Thanks but it's perfectly fine….. To put into context, it's same models with different parameters.
", "votes": null, "replies": [] }, { "id": 1401644, "author_name": "lsw1993", "author_url": "", "post_date": "07/27/2021 13:12:16", "content": "Made my notebook public at: https://www.kaggle.com/lsw1993/999-sub
", "votes": null, "replies": [] }, { "id": 1402890, "author_name": "redfoongus", "author_url": "", "post_date": "07/28/2021 16:00:23", "content": "I can't seem to access the notebook
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1401215": "I have same pipeline but using different models (all lightgbm models), the submission returning an error: **Submission Scoring Error**\n\n\nAnyone facing the same issue?", "1401237": "Any kaggle staff can help? @wcukierski @juliaelliott \n\nPossible to have a check on my notebook? It's simple notebook for inference only. \n\nhttps://www.kaggle.com/lsw1993/999-sub (Made it visible to public for temporary review)", "1401241": "Feel sucks - wasted all the submissions - but cannot get it working. \n\nWhat's going on? It went well on test but the error only occurs during submission.... @@ Really hate this.", "1401464": "lsw1993 \nYou can check your submission with API emulator. Hope this notebook helps:\nhttps://www.kaggle.com/nyanpn/api-emulator-for-debugging-your-code-locally", "1401545": "https://ibb.co/BcQytH7\n\nThanks but it's perfectly fine..... To put into context, it's same models with different parameters.", "1401592": "wcukierski @juliaelliott \nMy other queries: \n1. Is there a limit to the Kaggle dataset I am using (or connecting to the submission)?\n\n2. If test had returned smoothly, why my final submission will fail?\n\n3. Is it possible not to count failed submissions?", "1401633": "Hey @lsw1993, we unfortunately cannot provide personal debugging support, but https://www.kaggle.com/code-competition-debugging provides answers to some of your questions. On (1), there is no limit to using a dataset you have attached to a notebook. Good luck!", "1401642": "(3)? Considering the competition is ending.", "1401644": "Made my notebook public at: https://www.kaggle.com/lsw1993/999-sub", "1401667": "Per the code debugging page:\n\n> Submissions that error also count towards your team’s daily submission limit, otherwise such submissions could be used to mine for hidden information.", "1401691": "wcukierski @juliaelliott\n\nWouldn't it be systemic error/shortage if test is successful but not the final submission? \nFurther more details to be shown would be easier for us to debug moving forwards in my opinion. \n\nPerhaps a printed line of last run error would be suffice.", "1401845": "[Will](https://www.kaggle.com/wcukierski) - it still amazes me that kaggle staff believes that folks can mine for hidden information - for sure the future is hidden - what the heck could someone learn by mining in this competition. \n\nIt's fine that this is the company line but for the furtherance of my statistical training please provide a single sample of what could be learned by mining when the final private test has yet to occur and all the data used in the public LB has already been shared.", "1401926": "pcjimmmy prior to \"sync rerun\" code competitions, competitions like this would suffer from large failure rates during the 2nd stage (as high as 40% in the worst cases). Participants would adapt to the realities of the stage one dataset but were not incentivized to future proof their models. When new data would arrive and there is a 0 where there had never been a 0 before, code failed, we can't intervene, and lots of unhappy folks were left without a rank.\n\nThe benefit of a truly blind holdout set (whether the data is in the future or not) is that it front loads this \"tough love\" into stage one. For example, let's pretend the holdout set has a funny value that says a player is -92 ft tall. If you can see this fact, you might choose to patch or ignore that row. If you can't, you are pushed to make your code resilient to entire classes of bad inputs, missing values, impossible data events, and so on. The net result of this has been a dramatic reduction in rerun failure rates.\n\nI don't deny this can be frustrating. I also agree the probing safeguards are less relevant to true predict-the-future competitions. The pain we endure now is to create robust ML models that will run correctly on data they've never seen.", "1401940": "OK - I guess I get it. Self driving cars can be made more effective if they crash and no error data is available to the engineer - hence they have to code against all possible errors. But of course only 5 crashes per day will be permitted :)", "1401993": "Maybe the status can be made verbose, rather than just as such:\n\nhttps://ibb.co/zR4DYH5", "1402890": "I can't seem to access the notebook" }, "source": "meta" }