Can we talk about how failed submissions still count toward the daily submission limit? Especially this challenge can be quite tricky to debug given the setup, and some problems can only be fixed by trial and error.
\nThis is especially true for the loop within which submissions have to be created, even more so since the public test set \"happens to not include any lectures\" and can't hence be fully utilized for bug fixing.
Can we talk about how failed submissions still count toward the daily submission limit? Especially this challenge can be quite tricky to debug given the setup, and some problems can only be fixed by trial and error.
\nThis is especially true for the loop within which submissions have to be created, even more so since the public test set \"happens to not include any lectures\" and can't hence be fully utilized for bug fixing.
Hi Alex: Yes. Submission Scoring Errors with information, \"No additional details provided for this error\", are a total pain 😬 I have two Submissions that \"Run all\" correctly with the Example data, but fail on the hidden data.
\nToday I had one successful submission (a resubmission of a previous submission to check that my procedure is correct) and four instances of \"Submission Scoring Error\". So far, have no clue what is causing these failures. These submissions only differ from accepted submissions in minor tweaks to some parameter values. So am attempting trial-and-error debugging. Of course, \"Cannot submit - Your team has used its daily Submission allowance (5) today, please try again tomorrow UTC (15 hours from now).\"
\nIf SSEs must count against our daily quota, then please let's have useful error messages, e.g., \"divide by zero at line/cell 23 in the submission.\"
", "rawMarkdown": "Hi Alex: Yes. Submission Scoring Errors with information, \"No additional details provided for this error\", are a total pain 😬 I have two Submissions that \"Run all\" correctly with the Example data, but fail on the hidden data. \n\nToday I had one successful submission (a resubmission of a previous submission to check that my procedure is correct) and four instances of \"Submission Scoring Error\". So far, have no clue what is causing these failures. These submissions only differ from accepted submissions in minor tweaks to some parameter values. So am attempting trial-and-error debugging. Of course, \"Cannot submit - Your team has used its daily Submission allowance (5) today, please try again tomorrow UTC (15 hours from now).\"\n\nIf SSEs must count against our daily quota, then please let's have useful error messages, e.g., \"divide by zero at line/cell 23 in the submission.\"", "votes": null }, { "id": "1058771", "postDate": "10/24/2020 08:36:01", "content": "We can't do that, otherwise everyone will raise Exceptions at will and let them return the answer basically… It's tricky.
\nNot counting failed submissions would also open the door to writing simple submissions and probe the private test set by letting the kernel fail on purpose if a certain condition is met unfortunately…
The simplest solution would still be to provide a useful dummy test set which covers everything that can happen in the actual test set (i.e., include lectures, dataframes with only lectures, or with only one entry or possibly empty…).
\nOne can also write these test by hand, it's a hassle but in this case needed and worth it.
It pains more when your kernel kinda fails at the very end for a single batch maybe for reason which you have to brainstorm. In my case, It runs for around 2 hours in sub and then crashes suddenly. :(
", "rawMarkdown": "It pains more when your kernel kinda fails at the very end for a single batch maybe for reason which you have to brainstorm. In my case, It runs for around 2 hours in sub and then crashes suddenly. :(", "votes": null }, { "id": "1094810", "postDate": "11/29/2020 00:29:06", "content": "My submission ended with an error. No clue whatsoever. I can run locally for a long time. No problem. I am using cuda and locally have 10GB available. What is the official way to contact the team to help identify the issue?
", "rawMarkdown": "My submission ended with an error. No clue whatsoever. I can run locally for a long time. No problem. I am using cuda and locally have 10GB available. What is the official way to contact the team to help identify the issue?", "votes": null }, { "id": "1094876", "postDate": "11/29/2020 03:46:45", "content": "Check your logic i would suggest because we do have successful subs for Dl based models as well. Ensure you don't have nan's and you are handling new_users/videos/non-videos rows, ensuring your SEQ_LEN doesn't go beyond etc..
", "rawMarkdown": "Check your logic i would suggest because we do have successful subs for Dl based models as well. Ensure you don't have nan's and you are handling new_users/videos/non-videos rows, ensuring your SEQ_LEN doesn't go beyond etc..", "votes": null } ], "comments": [ { "id": 1058769, "author_name": "mikel1", "author_url": "", "post_date": "10/24/2020 08:27:25", "content": "Hi Alex: Yes. Submission Scoring Errors with information, \"No additional details provided for this error\", are a total pain 😬 I have two Submissions that \"Run all\" correctly with the Example data, but fail on the hidden data.
\nToday I had one successful submission (a resubmission of a previous submission to check that my procedure is correct) and four instances of \"Submission Scoring Error\". So far, have no clue what is causing these failures. These submissions only differ from accepted submissions in minor tweaks to some parameter values. So am attempting trial-and-error debugging. Of course, \"Cannot submit - Your team has used its daily Submission allowance (5) today, please try again tomorrow UTC (15 hours from now).\"
\nIf SSEs must count against our daily quota, then please let's have useful error messages, e.g., \"divide by zero at line/cell 23 in the submission.\"
", "votes": null, "replies": [ { "id": 1058771, "author_name": "spacelx", "author_url": "", "post_date": "10/24/2020 08:36:01", "content": "We can't do that, otherwise everyone will raise Exceptions at will and let them return the answer basically… It's tricky.
\nNot counting failed submissions would also open the door to writing simple submissions and probe the private test set by letting the kernel fail on purpose if a certain condition is met unfortunately…
The simplest solution would still be to provide a useful dummy test set which covers everything that can happen in the actual test set (i.e., include lectures, dataframes with only lectures, or with only one entry or possibly empty…).
\nOne can also write these test by hand, it's a hassle but in this case needed and worth it.
It pains more when your kernel kinda fails at the very end for a single batch maybe for reason which you have to brainstorm. In my case, It runs for around 2 hours in sub and then crashes suddenly. :(
", "votes": null, "replies": [] }, { "id": 1094810, "author_name": "vadimmarchenko", "author_url": "", "post_date": "11/29/2020 00:29:06", "content": "My submission ended with an error. No clue whatsoever. I can run locally for a long time. No problem. I am using cuda and locally have 10GB available. What is the official way to contact the team to help identify the issue?
", "votes": null, "replies": [ { "id": 1094876, "author_name": "adityaecdrid", "author_url": "", "post_date": "11/29/2020 03:46:45", "content": "Check your logic i would suggest because we do have successful subs for Dl based models as well. Ensure you don't have nan's and you are handling new_users/videos/non-videos rows, ensuring your SEQ_LEN doesn't go beyond etc..
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1046281": "Can we talk about how failed submissions still count toward the daily submission limit? Especially this challenge can be quite tricky to debug given the setup, and some problems can only be fixed by trial and error. \nThis is especially true for the loop within which submissions have to be created, even more so since the public test set \"happens to not include any lectures\" and can't hence be fully utilized for bug fixing.", "1058769": "Hi Alex: Yes. Submission Scoring Errors with information, \"No additional details provided for this error\", are a total pain 😬 I have two Submissions that \"Run all\" correctly with the Example data, but fail on the hidden data. \n\nToday I had one successful submission (a resubmission of a previous submission to check that my procedure is correct) and four instances of \"Submission Scoring Error\". So far, have no clue what is causing these failures. These submissions only differ from accepted submissions in minor tweaks to some parameter values. So am attempting trial-and-error debugging. Of course, \"Cannot submit - Your team has used its daily Submission allowance (5) today, please try again tomorrow UTC (15 hours from now).\"\n\nIf SSEs must count against our daily quota, then please let's have useful error messages, e.g., \"divide by zero at line/cell 23 in the submission.\"", "1058771": "We can't do that, otherwise everyone will raise Exceptions at will and let them return the answer basically... It's tricky.\nNot counting failed submissions would also open the door to writing simple submissions and probe the private test set by letting the kernel fail on purpose if a certain condition is met unfortunately...\n\nThe simplest solution would still be to provide a useful dummy test set which covers everything that can happen in the actual test set (i.e., include lectures, dataframes with only lectures, or with only one entry or possibly empty...).\nOne can also write these test by hand, it's a hassle but in this case needed and worth it.", "1058785": "It pains more when your kernel kinda fails at the very end for a single batch maybe for reason which you have to brainstorm. In my case, It runs for around 2 hours in sub and then crashes suddenly. :(", "1094810": "My submission ended with an error. No clue whatsoever. I can run locally for a long time. No problem. I am using cuda and locally have 10GB available. What is the official way to contact the team to help identify the issue?", "1094876": "Check your logic i would suggest because we do have successful subs for Dl based models as well. Ensure you don't have nan's and you are handling new_users/videos/non-videos rows, ensuring your SEQ_LEN doesn't go beyond etc.." }, "source": "meta" }