{
  "id": 116409,
  "title": "Scoring error on submission - even with sample_submission.csv",
  "url": "/competitions/tensorflow2-question-answering/discussion/116409",
  "author_name": "",
  "post_date": "2019-11-08T22:25:43.997959900Z",
  "votes": 13,
  "comment_count": 19,
  "views": 0,
  "content": "<p>I'm having a lot of trouble submitting to this contest. Each submission fails with a scoring error. I went so far as to submit the sample_submission.csv provided in the instructions, and it still fails! What am I missing?</p>\n\n<p><a href=\"https://www.kaggle.com/alonbochman/qa-submission\">https://www.kaggle.com/alonbochman/qa-submission</a></p>\n\n<p>Thanks,\nAlon</p>",
  "messages": [
    {
      "id": "668803",
      "postDate": "11/08/2019 22:25:43",
      "content": "<p>I'm having a lot of trouble submitting to this contest. Each submission fails with a scoring error. I went so far as to submit the sample_submission.csv provided in the instructions, and it still fails! What am I missing?</p>\n\n<p><a href=\"https://www.kaggle.com/alonbochman/qa-submission\">https://www.kaggle.com/alonbochman/qa-submission</a></p>\n\n<p>Thanks,\nAlon</p>",
      "rawMarkdown": "I'm having a lot of trouble submitting to this contest. Each submission fails with a scoring error. I went so far as to submit the sample_submission.csv provided in the instructions, and it still fails! What am I missing?\n\nhttps://www.kaggle.com/alonbochman/qa-submission\n\nThanks,\nAlon",
      "votes": null
    },
    {
      "id": "668921",
      "postDate": "11/09/2019 05:02:58",
      "content": "<p><a href=\"/alonbochman\">@alonbochman</a> Sorry you were having issues with your submission, and thank you for reporting this issue. A bug was introduced last night. It has now been resolved and you should retry your submission without a problem. Please let us know if you hit any further unexpected errors.</p>",
      "rawMarkdown": "alonbochman Sorry you were having issues with your submission, and thank you for reporting this issue. A bug was introduced last night. It has now been resolved and you should retry your submission without a problem. Please let us know if you hit any further unexpected errors.",
      "votes": null
    },
    {
      "id": "669128",
      "postDate": "11/09/2019 14:24:34",
      "content": "<p>Thanks Julia. The sample submission works now, with a score of 0 as expected. However, it seems unfair to those of us that have wasted many submissions to find this bug. Can we get our submission counts reset? They don't matter now, but will when we are looking for teammates.</p>\n\n<p>Thanks,\nAlon</p>",
      "rawMarkdown": "Thanks Julia. The sample submission works now, with a score of 0 as expected. However, it seems unfair to those of us that have wasted many submissions to find this bug. Can we get our submission counts reset? They don't matter now, but will when we are looking for teammates.\n\nThanks,\nAlon",
      "votes": null
    },
    {
      "id": "669192",
      "postDate": "11/09/2019 16:52:41",
      "content": "<p><a href=\"/alonbochman\">@alonbochman</a> Sorry, as a policy, we do not make exceptions or adjustments to submissions counts, even in this case. Team merger eligibility is counted on an overall basis, so hopefully extra submissions associated with this one-day bug won’t be a hindrance for you in teaming.</p>",
      "rawMarkdown": "alonbochman Sorry, as a policy, we do not make exceptions or adjustments to submissions counts, even in this case. Team merger eligibility is counted on an overall basis, so hopefully extra submissions associated with this one-day bug won’t be a hindrance for you in teaming.",
      "votes": null
    },
    {
      "id": "673317",
      "postDate": "11/14/2019 20:05:18",
      "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> Still getting lots of scoring errors, including by resubmitting public kernels. Can you please let me know what is wrong with the following submission?</p>\n\n<p><a href=\"https://www.kaggle.com/alonbochman/qa-submission-v2?scriptVersionId=23483343\">https://www.kaggle.com/alonbochman/qa-submission-v2?scriptVersionId=23483343</a></p>\n\n<p>It would be super helpful if we could have a utility script to verify a submission file's format so we can avoid these scoring errors.</p>\n\n<p>Thanks,\nAlon </p>",
      "rawMarkdown": "juliaelliott Still getting lots of scoring errors, including by resubmitting public kernels. Can you please let me know what is wrong with the following submission?\n\nhttps://www.kaggle.com/alonbochman/qa-submission-v2?scriptVersionId=23483343\n\nIt would be super helpful if we could have a utility script to verify a submission file's format so we can avoid these scoring errors.\n\nThanks,\nAlon",
      "votes": null
    },
    {
      "id": "675706",
      "postDate": "11/18/2019 12:49:14",
      "content": "<p>Is there a limit to number of submissions a team can make?</p>",
      "rawMarkdown": "Is there a limit to number of submissions a team can make?",
      "votes": null
    },
    {
      "id": "675891",
      "postDate": "11/18/2019 18:19:39",
      "content": "<p>Hi Alon. So I think you may be misunderstanding how this code competition format is working with a private test set. In this format, you build code that is capable of being run on any test set that is fed to it. We provide a public test set as a proxy for what you can expect the private test set to be. But the score you receive during an interactive notebook session is based only on the public test set. When you submit from your notebook, we are re-running your code in the background against a private test set. And that code needs to produce a submission for that private test set in the right format or the entire submission will fail/error.</p>\n\n<p>Your notebook appears to be direct-reading a .CSV file and trying to make that as its submission. However, by doing that, your predictions only consist of the public test set and is not transferrable to a withheld private test set. This is where it errors. You must create code that is actually able to be run against a private set of inputs (in the same format as the public test set) and output a valid submission file for that test set.</p>",
      "rawMarkdown": "Hi Alon. So I think you may be misunderstanding how this code competition format is working with a private test set. In this format, you build code that is capable of being run on any test set that is fed to it. We provide a public test set as a proxy for what you can expect the private test set to be. But the score you receive during an interactive notebook session is based only on the public test set. When you submit from your notebook, we are re-running your code in the background against a private test set. And that code needs to produce a submission for that private test set in the right format or the entire submission will fail/error.\n\nYour notebook appears to be direct-reading a .CSV file and trying to make that as its submission. However, by doing that, your predictions only consist of the public test set and is not transferrable to a withheld private test set. This is where it errors. You must create code that is actually able to be run against a private set of inputs (in the same format as the public test set) and output a valid submission file for that test set.",
      "votes": null
    },
    {
      "id": "676028",
      "postDate": "11/18/2019 22:57:56",
      "content": "<p><a href=\"/arvindpdmn\">@arvindpdmn</a> 5 submissions per day.</p>",
      "rawMarkdown": "arvindpdmn 5 submissions per day.",
      "votes": null
    },
    {
      "id": "683881",
      "postDate": "11/28/2019 23:57:00",
      "content": "<p>so can i still make submission via .csv uploaded and it would give me a corect score since the bug has being fixed <a href=\"/alonbochman\">@alonbochman</a>  <a href=\"/juliaelliott\">@juliaelliott</a>  <a href=\"/arvindpdmn\">@arvindpdmn</a> </p>",
      "rawMarkdown": "so can i still make submission via .csv uploaded and it would give me a corect score since the bug has being fixed @alonbochman  @juliaelliott  @arvindpdmn",
      "votes": null
    },
    {
      "id": "684169",
      "postDate": "11/29/2019 10:41:33",
      "content": "<p>No, same question <a href=\"https://www.kaggle.com/c/tensorflow2-question-answering/discussion/119530#684168\">here</a>. </p>",
      "rawMarkdown": "No, same question [here](https://www.kaggle.com/c/tensorflow2-question-answering/discussion/119530#684168).",
      "votes": null
    },
    {
      "id": "704766",
      "postDate": "12/28/2019 01:45:44",
      "content": "<p>Honestly, this is incredibly frustrating! I am facing the same issue. I have spent my 3 out of 5 daily submission limit in 2 hours. I still don't know what the code needs to do to avoid this problem. The lack of error details means that I am flying blind on how to fix this. Since I can't see the error, I don't know if I'll be able to fix this in the next 2 or 2 million tries. </p>\n\n<p>Some questions I have:\n* Where is the private dataset? \n* How is the private dataset loaded into python memory? If my code doesn't load the private dataset, then what code does? Can I get a snippet of the code that loads the data?\n* Should we write a function that takes in one row at a time or one dataframe at a time or a path to a file ?</p>\n\n<p>It's not too much to ask for a small starter notebook (not the super complicated one they give you that hides all the complexity of I/O) that shows in simple python how to submit correctly. I am surprised how a company like Google can make such poor documentation, to be honest. I expect this from a company that doesn't deal with tech. </p>\n\n<p>Here is my notebook: <a href=\"https://www.kaggle.com/ankurgupta1985/randomly-selected\">https://www.kaggle.com/ankurgupta1985/randomly-selected</a>\nI am already chunking the data and not loading all of it in memory at one time. But, I assume that the final submission.csv is small enough that the corresponding dataframe would fit in memory. I have already asked friends who're experienced at Kaggle to help me and they're dumbfounded too. If anyone can help, I promise to consider naming my first born after you. Thank you very much!</p>",
      "rawMarkdown": "Honestly, this is incredibly frustrating! I am facing the same issue. I have spent my 3 out of 5 daily submission limit in 2 hours. I still don't know what the code needs to do to avoid this problem. The lack of error details means that I am flying blind on how to fix this. Since I can't see the error, I don't know if I'll be able to fix this in the next 2 or 2 million tries. \n\nSome questions I have:\n* Where is the private dataset? \n* How is the private dataset loaded into python memory? If my code doesn't load the private dataset, then what code does? Can I get a snippet of the code that loads the data?\n* Should we write a function that takes in one row at a time or one dataframe at a time or a path to a file ?\n\nIt's not too much to ask for a small starter notebook (not the super complicated one they give you that hides all the complexity of I/O) that shows in simple python how to submit correctly. I am surprised how a company like Google can make such poor documentation, to be honest. I expect this from a company that doesn't deal with tech. \n\nHere is my notebook: https://www.kaggle.com/ankurgupta1985/randomly-selected\nI am already chunking the data and not loading all of it in memory at one time. But, I assume that the final submission.csv is small enough that the corresponding dataframe would fit in memory. I have already asked friends who're experienced at Kaggle to help me and they're dumbfounded too. If anyone can help, I promise to consider naming my first born after you. Thank you very much!",
      "votes": null
    },
    {
      "id": "721668",
      "postDate": "01/17/2020 15:36:29",
      "content": "<p>Very well said  Ankur, It is really frustrating. \nAs you said even after 2 million attempts if it is continuous ... What else we can do?\nIt should be properly documented or the Kaggle team should respond ASAP ...</p>\n\n<p>I wrote this in a separate thread, almost 10 days back...</p>\n\n<p>But yet to get any response from Kaggle </p>",
      "rawMarkdown": "Very well said  Ankur, It is really frustrating. \nAs you said even after 2 million attempts if it is continuous ... What else we can do?\nIt should be properly documented or the Kaggle team should respond ASAP ...\n\nI wrote this in a separate thread, almost 10 days back...\n\nBut yet to get any response from Kaggle",
      "votes": null
    },
    {
      "id": "722061",
      "postDate": "01/18/2020 03:41:55",
      "content": "<p>I'm also a beginner to Kaggle. The system is not very friendly to those who're doing their first competition. I had to figure out many things by trial and error. Some things I understood:</p>\n\n<p>Private dataset is not available to us. It will be used for the final evaluation. It may become available once the competition ends.</p>\n\n<p>I have tried hard coding the answers and submitting. It doesn't work. The evaluation probably checks (using some heuristics) if you're predicting with a suitable ML model. If not, there's a submission error. Anyway, this is my best guess.</p>",
      "rawMarkdown": "I'm also a beginner to Kaggle. The system is not very friendly to those who're doing their first competition. I had to figure out many things by trial and error. Some things I understood:\n\nPrivate dataset is not available to us. It will be used for the final evaluation. It may become available once the competition ends.\n\nI have tried hard coding the answers and submitting. It doesn't work. The evaluation probably checks (using some heuristics) if you're predicting with a suitable ML model. If not, there's a submission error. Anyway, this is my best guess.",
      "votes": null
    },
    {
      "id": "722064",
      "postDate": "01/18/2020 03:47:53",
      "content": "<p><a href=\"/ankurgupta1985\">@ankurgupta1985</a> The private dataset’s location is intentionally privately held and only used on the re-run of your code at time of submission. But it is synchronously loaded in the same way the public test set is. Your code needs to run such that if the existing public test set is swapped out for a private test set (from its existing location), it can dynamically generate a <code>submission.csv</code>. So you can’t hardcode the id’s, for example. The <code>sample_submission</code> for the private test set is also available in the re-run instance. How you load the data is up to you as long as it adheres to the above principle.</p>",
      "rawMarkdown": "ankurgupta1985 The private dataset’s location is intentionally privately held and only used on the re-run of your code at time of submission. But it is synchronously loaded in the same way the public test set is. Your code needs to run such that if the existing public test set is swapped out for a private test set (from its existing location), it can dynamically generate a `submission.csv`. So you can’t hardcode the id’s, for example. The `sample_submission` for the private test set is also available in the re-run instance. How you load the data is up to you as long as it adheres to the above principle.",
      "votes": null
    },
    {
      "id": "722215",
      "postDate": "01/18/2020 09:22:54",
      "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a>  In <strong>Severstal competition</strong>  it was working fine for me before Oct 24, 2019. Currently, Kaggle changed the train CSV file ( by removing non-defective records but that is fine).</p>\n\n<p>problem is sample_submision file, currently, it is in a new format, we are confused with format new or old one.  Perhaps old format no issue... But New format is an issue because it is logically incorrect.</p>\n\n<p><a href=\"https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/125265\">Some more details</a></p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3335448%2F1b186578414174aeb746d9f163504e2b%2FScreenshot_1.png?generation=1579340738605479&amp;alt=media\" alt=\"\"></p>\n\n<p><a href=\"/juliaelliott\">@juliaelliott</a>  Perhaps you can understand the frustration of 23 days ...</p>",
      "rawMarkdown": "juliaelliott  In **Severstal competition**  it was working fine for me before Oct 24, 2019. Currently, Kaggle changed the train CSV file ( by removing non-defective records but that is fine).\n\nproblem is sample_submision file, currently, it is in a new format, we are confused with format new or old one.  Perhaps old format no issue... But New format is an issue because it is logically incorrect.\n\n\n[Some more details](https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/125265)\n\n\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3335448%2F1b186578414174aeb746d9f163504e2b%2FScreenshot_1.png?generation=1579340738605479&amp;alt=media)\n\n\n@juliaelliott  Perhaps you can understand the frustration of 23 days ...",
      "votes": null
    },
    {
      "id": "722475",
      "postDate": "01/18/2020 16:07:56",
      "content": "<p><a href=\"/siriguna\">@siriguna</a> Sorry, I'm not following you. This is the TF2.0 competition, not Severstal. The <code>sample_submission</code> in this competition has never changed since launch.</p>",
      "rawMarkdown": "siriguna Sorry, I'm not following you. This is the TF2.0 competition, not Severstal. The `sample_submission` in this competition has never changed since launch.",
      "votes": null
    },
    {
      "id": "722586",
      "postDate": "01/18/2020 18:58:38",
      "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> Thank you for the quick response, I understand it is TF2.0. Unfortunately, no one responding at Severstal. In spite of screaming for the past 23 days... </p>\n\n<p>Could you please influence someone in the Kaggle team to respond to this issue.</p>\n\n<p>Thanks in advance</p>",
      "rawMarkdown": "juliaelliott Thank you for the quick response, I understand it is TF2.0. Unfortunately, no one responding at Severstal. In spite of screaming for the past 23 days... \n\nCould you please influence someone in the Kaggle team to respond to this issue.\n\nThanks in advance",
      "votes": null
    },
    {
      "id": "724387",
      "postDate": "01/21/2020 05:48:02",
      "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> please someone help me ... at least state me it is just my problem <strong>or</strong> issue to everyone at Severstal competition  <strong>or</strong> after reopened for late submission, is anyone could able to submit successfully?</p>\n\n<p>Waiting for reply</p>\n\n<p>Thanks in advance  </p>",
      "rawMarkdown": "juliaelliott please someone help me ... at least state me it is just my problem **or** issue to everyone at Severstal competition  **or** after reopened for late submission, is anyone could able to submit successfully?\n\nWaiting for reply\n\nThanks in advance",
      "votes": null
    },
    {
      "id": "725126",
      "postDate": "01/21/2020 20:40:01",
      "content": "<p><a href=\"/siriguna\">@siriguna</a> This isn't the right place to be asking that question, as I am not directly aware of/the person responsible for that competition. Please check that competition's forum. After a quick look, it seems <a href=\"https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/126923\">this might answer your question</a>.</p>",
      "rawMarkdown": "siriguna This isn't the right place to be asking that question, as I am not directly aware of/the person responsible for that competition. Please check that competition's forum. After a quick look, it seems [this might answer your question](https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/126923).",
      "votes": null
    },
    {
      "id": "725451",
      "postDate": "01/22/2020 05:43:17",
      "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a>  thanks once again for quick reply. happen to fix the problem. Actually found some work around. (Even though it is kernel competition but submit the submission file only)</p>",
      "rawMarkdown": "juliaelliott  thanks once again for quick reply. happen to fix the problem. Actually found some work around. (Even though it is kernel competition but submit the submission file only)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 668921,
      "author_name": "juliaelliott",
      "author_url": "",
      "post_date": "11/09/2019 05:02:58",
      "content": "<p><a href=\"/alonbochman\">@alonbochman</a> Sorry you were having issues with your submission, and thank you for reporting this issue. A bug was introduced last night. It has now been resolved and you should retry your submission without a problem. Please let us know if you hit any further unexpected errors.</p>",
      "votes": null,
      "replies": [
        {
          "id": 669128,
          "author_name": "alonbochman",
          "author_url": "",
          "post_date": "11/09/2019 14:24:34",
          "content": "<p>Thanks Julia. The sample submission works now, with a score of 0 as expected. However, it seems unfair to those of us that have wasted many submissions to find this bug. Can we get our submission counts reset? They don't matter now, but will when we are looking for teammates.</p>\n\n<p>Thanks,\nAlon</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 669192,
          "author_name": "juliaelliott",
          "author_url": "",
          "post_date": "11/09/2019 16:52:41",
          "content": "<p><a href=\"/alonbochman\">@alonbochman</a> Sorry, as a policy, we do not make exceptions or adjustments to submissions counts, even in this case. Team merger eligibility is counted on an overall basis, so hopefully extra submissions associated with this one-day bug won’t be a hindrance for you in teaming.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 675706,
          "author_name": "arvindpdmn",
          "author_url": "",
          "post_date": "11/18/2019 12:49:14",
          "content": "<p>Is there a limit to number of submissions a team can make?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 676028,
          "author_name": "juliaelliott",
          "author_url": "",
          "post_date": "11/18/2019 22:57:56",
          "content": "<p><a href=\"/arvindpdmn\">@arvindpdmn</a> 5 submissions per day.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 683881,
          "author_name": "",
          "author_url": "",
          "post_date": "11/28/2019 23:57:00",
          "content": "<p>so can i still make submission via .csv uploaded and it would give me a corect score since the bug has being fixed <a href=\"/alonbochman\">@alonbochman</a>  <a href=\"/juliaelliott\">@juliaelliott</a>  <a href=\"/arvindpdmn\">@arvindpdmn</a> </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 684169,
          "author_name": "kashnitsky",
          "author_url": "",
          "post_date": "11/29/2019 10:41:33",
          "content": "<p>No, same question <a href=\"https://www.kaggle.com/c/tensorflow2-question-answering/discussion/119530#684168\">here</a>. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 704766,
          "author_name": "ankurgupta1985",
          "author_url": "",
          "post_date": "12/28/2019 01:45:44",
          "content": "<p>Honestly, this is incredibly frustrating! I am facing the same issue. I have spent my 3 out of 5 daily submission limit in 2 hours. I still don't know what the code needs to do to avoid this problem. The lack of error details means that I am flying blind on how to fix this. Since I can't see the error, I don't know if I'll be able to fix this in the next 2 or 2 million tries. </p>\n\n<p>Some questions I have:\n* Where is the private dataset? \n* How is the private dataset loaded into python memory? If my code doesn't load the private dataset, then what code does? Can I get a snippet of the code that loads the data?\n* Should we write a function that takes in one row at a time or one dataframe at a time or a path to a file ?</p>\n\n<p>It's not too much to ask for a small starter notebook (not the super complicated one they give you that hides all the complexity of I/O) that shows in simple python how to submit correctly. I am surprised how a company like Google can make such poor documentation, to be honest. I expect this from a company that doesn't deal with tech. </p>\n\n<p>Here is my notebook: <a href=\"https://www.kaggle.com/ankurgupta1985/randomly-selected\">https://www.kaggle.com/ankurgupta1985/randomly-selected</a>\nI am already chunking the data and not loading all of it in memory at one time. But, I assume that the final submission.csv is small enough that the corresponding dataframe would fit in memory. I have already asked friends who're experienced at Kaggle to help me and they're dumbfounded too. If anyone can help, I promise to consider naming my first born after you. Thank you very much!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 721668,
          "author_name": "siriguna",
          "author_url": "",
          "post_date": "01/17/2020 15:36:29",
          "content": "<p>Very well said  Ankur, It is really frustrating. \nAs you said even after 2 million attempts if it is continuous ... What else we can do?\nIt should be properly documented or the Kaggle team should respond ASAP ...</p>\n\n<p>I wrote this in a separate thread, almost 10 days back...</p>\n\n<p>But yet to get any response from Kaggle </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 722061,
          "author_name": "arvindpdmn",
          "author_url": "",
          "post_date": "01/18/2020 03:41:55",
          "content": "<p>I'm also a beginner to Kaggle. The system is not very friendly to those who're doing their first competition. I had to figure out many things by trial and error. Some things I understood:</p>\n\n<p>Private dataset is not available to us. It will be used for the final evaluation. It may become available once the competition ends.</p>\n\n<p>I have tried hard coding the answers and submitting. It doesn't work. The evaluation probably checks (using some heuristics) if you're predicting with a suitable ML model. If not, there's a submission error. Anyway, this is my best guess.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 722064,
          "author_name": "juliaelliott",
          "author_url": "",
          "post_date": "01/18/2020 03:47:53",
          "content": "<p><a href=\"/ankurgupta1985\">@ankurgupta1985</a> The private dataset’s location is intentionally privately held and only used on the re-run of your code at time of submission. But it is synchronously loaded in the same way the public test set is. Your code needs to run such that if the existing public test set is swapped out for a private test set (from its existing location), it can dynamically generate a <code>submission.csv</code>. So you can’t hardcode the id’s, for example. The <code>sample_submission</code> for the private test set is also available in the re-run instance. How you load the data is up to you as long as it adheres to the above principle.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 722215,
          "author_name": "siriguna",
          "author_url": "",
          "post_date": "01/18/2020 09:22:54",
          "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a>  In <strong>Severstal competition</strong>  it was working fine for me before Oct 24, 2019. Currently, Kaggle changed the train CSV file ( by removing non-defective records but that is fine).</p>\n\n<p>problem is sample_submision file, currently, it is in a new format, we are confused with format new or old one.  Perhaps old format no issue... But New format is an issue because it is logically incorrect.</p>\n\n<p><a href=\"https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/125265\">Some more details</a></p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3335448%2F1b186578414174aeb746d9f163504e2b%2FScreenshot_1.png?generation=1579340738605479&amp;alt=media\" alt=\"\"></p>\n\n<p><a href=\"/juliaelliott\">@juliaelliott</a>  Perhaps you can understand the frustration of 23 days ...</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 722475,
          "author_name": "juliaelliott",
          "author_url": "",
          "post_date": "01/18/2020 16:07:56",
          "content": "<p><a href=\"/siriguna\">@siriguna</a> Sorry, I'm not following you. This is the TF2.0 competition, not Severstal. The <code>sample_submission</code> in this competition has never changed since launch.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 722586,
          "author_name": "siriguna",
          "author_url": "",
          "post_date": "01/18/2020 18:58:38",
          "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> Thank you for the quick response, I understand it is TF2.0. Unfortunately, no one responding at Severstal. In spite of screaming for the past 23 days... </p>\n\n<p>Could you please influence someone in the Kaggle team to respond to this issue.</p>\n\n<p>Thanks in advance</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 724387,
          "author_name": "siriguna",
          "author_url": "",
          "post_date": "01/21/2020 05:48:02",
          "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> please someone help me ... at least state me it is just my problem <strong>or</strong> issue to everyone at Severstal competition  <strong>or</strong> after reopened for late submission, is anyone could able to submit successfully?</p>\n\n<p>Waiting for reply</p>\n\n<p>Thanks in advance  </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 725126,
          "author_name": "juliaelliott",
          "author_url": "",
          "post_date": "01/21/2020 20:40:01",
          "content": "<p><a href=\"/siriguna\">@siriguna</a> This isn't the right place to be asking that question, as I am not directly aware of/the person responsible for that competition. Please check that competition's forum. After a quick look, it seems <a href=\"https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/126923\">this might answer your question</a>.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 725451,
          "author_name": "siriguna",
          "author_url": "",
          "post_date": "01/22/2020 05:43:17",
          "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a>  thanks once again for quick reply. happen to fix the problem. Actually found some work around. (Even though it is kernel competition but submit the submission file only)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 673317,
      "author_name": "alonbochman",
      "author_url": "",
      "post_date": "11/14/2019 20:05:18",
      "content": "<p><a href=\"/juliaelliott\">@juliaelliott</a> Still getting lots of scoring errors, including by resubmitting public kernels. Can you please let me know what is wrong with the following submission?</p>\n\n<p><a href=\"https://www.kaggle.com/alonbochman/qa-submission-v2?scriptVersionId=23483343\">https://www.kaggle.com/alonbochman/qa-submission-v2?scriptVersionId=23483343</a></p>\n\n<p>It would be super helpful if we could have a utility script to verify a submission file's format so we can avoid these scoring errors.</p>\n\n<p>Thanks,\nAlon </p>",
      "votes": null,
      "replies": [
        {
          "id": 675891,
          "author_name": "juliaelliott",
          "author_url": "",
          "post_date": "11/18/2019 18:19:39",
          "content": "<p>Hi Alon. So I think you may be misunderstanding how this code competition format is working with a private test set. In this format, you build code that is capable of being run on any test set that is fed to it. We provide a public test set as a proxy for what you can expect the private test set to be. But the score you receive during an interactive notebook session is based only on the public test set. When you submit from your notebook, we are re-running your code in the background against a private test set. And that code needs to produce a submission for that private test set in the right format or the entire submission will fail/error.</p>\n\n<p>Your notebook appears to be direct-reading a .CSV file and trying to make that as its submission. However, by doing that, your predictions only consist of the public test set and is not transferrable to a withheld private test set. This is where it errors. You must create code that is actually able to be run against a private set of inputs (in the same format as the public test set) and output a valid submission file for that test set.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "668803": "I'm having a lot of trouble submitting to this contest. Each submission fails with a scoring error. I went so far as to submit the sample_submission.csv provided in the instructions, and it still fails! What am I missing?\n\nhttps://www.kaggle.com/alonbochman/qa-submission\n\nThanks,\nAlon",
    "668921": "alonbochman Sorry you were having issues with your submission, and thank you for reporting this issue. A bug was introduced last night. It has now been resolved and you should retry your submission without a problem. Please let us know if you hit any further unexpected errors.",
    "669128": "Thanks Julia. The sample submission works now, with a score of 0 as expected. However, it seems unfair to those of us that have wasted many submissions to find this bug. Can we get our submission counts reset? They don't matter now, but will when we are looking for teammates.\n\nThanks,\nAlon",
    "669192": "alonbochman Sorry, as a policy, we do not make exceptions or adjustments to submissions counts, even in this case. Team merger eligibility is counted on an overall basis, so hopefully extra submissions associated with this one-day bug won’t be a hindrance for you in teaming.",
    "673317": "juliaelliott Still getting lots of scoring errors, including by resubmitting public kernels. Can you please let me know what is wrong with the following submission?\n\nhttps://www.kaggle.com/alonbochman/qa-submission-v2?scriptVersionId=23483343\n\nIt would be super helpful if we could have a utility script to verify a submission file's format so we can avoid these scoring errors.\n\nThanks,\nAlon",
    "675706": "Is there a limit to number of submissions a team can make?",
    "675891": "Hi Alon. So I think you may be misunderstanding how this code competition format is working with a private test set. In this format, you build code that is capable of being run on any test set that is fed to it. We provide a public test set as a proxy for what you can expect the private test set to be. But the score you receive during an interactive notebook session is based only on the public test set. When you submit from your notebook, we are re-running your code in the background against a private test set. And that code needs to produce a submission for that private test set in the right format or the entire submission will fail/error.\n\nYour notebook appears to be direct-reading a .CSV file and trying to make that as its submission. However, by doing that, your predictions only consist of the public test set and is not transferrable to a withheld private test set. This is where it errors. You must create code that is actually able to be run against a private set of inputs (in the same format as the public test set) and output a valid submission file for that test set.",
    "676028": "arvindpdmn 5 submissions per day.",
    "683881": "so can i still make submission via .csv uploaded and it would give me a corect score since the bug has being fixed @alonbochman  @juliaelliott  @arvindpdmn",
    "684169": "No, same question [here](https://www.kaggle.com/c/tensorflow2-question-answering/discussion/119530#684168).",
    "704766": "Honestly, this is incredibly frustrating! I am facing the same issue. I have spent my 3 out of 5 daily submission limit in 2 hours. I still don't know what the code needs to do to avoid this problem. The lack of error details means that I am flying blind on how to fix this. Since I can't see the error, I don't know if I'll be able to fix this in the next 2 or 2 million tries. \n\nSome questions I have:\n* Where is the private dataset? \n* How is the private dataset loaded into python memory? If my code doesn't load the private dataset, then what code does? Can I get a snippet of the code that loads the data?\n* Should we write a function that takes in one row at a time or one dataframe at a time or a path to a file ?\n\nIt's not too much to ask for a small starter notebook (not the super complicated one they give you that hides all the complexity of I/O) that shows in simple python how to submit correctly. I am surprised how a company like Google can make such poor documentation, to be honest. I expect this from a company that doesn't deal with tech. \n\nHere is my notebook: https://www.kaggle.com/ankurgupta1985/randomly-selected\nI am already chunking the data and not loading all of it in memory at one time. But, I assume that the final submission.csv is small enough that the corresponding dataframe would fit in memory. I have already asked friends who're experienced at Kaggle to help me and they're dumbfounded too. If anyone can help, I promise to consider naming my first born after you. Thank you very much!",
    "721668": "Very well said  Ankur, It is really frustrating. \nAs you said even after 2 million attempts if it is continuous ... What else we can do?\nIt should be properly documented or the Kaggle team should respond ASAP ...\n\nI wrote this in a separate thread, almost 10 days back...\n\nBut yet to get any response from Kaggle",
    "722061": "I'm also a beginner to Kaggle. The system is not very friendly to those who're doing their first competition. I had to figure out many things by trial and error. Some things I understood:\n\nPrivate dataset is not available to us. It will be used for the final evaluation. It may become available once the competition ends.\n\nI have tried hard coding the answers and submitting. It doesn't work. The evaluation probably checks (using some heuristics) if you're predicting with a suitable ML model. If not, there's a submission error. Anyway, this is my best guess.",
    "722064": "ankurgupta1985 The private dataset’s location is intentionally privately held and only used on the re-run of your code at time of submission. But it is synchronously loaded in the same way the public test set is. Your code needs to run such that if the existing public test set is swapped out for a private test set (from its existing location), it can dynamically generate a `submission.csv`. So you can’t hardcode the id’s, for example. The `sample_submission` for the private test set is also available in the re-run instance. How you load the data is up to you as long as it adheres to the above principle.",
    "722215": "juliaelliott  In **Severstal competition**  it was working fine for me before Oct 24, 2019. Currently, Kaggle changed the train CSV file ( by removing non-defective records but that is fine).\n\nproblem is sample_submision file, currently, it is in a new format, we are confused with format new or old one.  Perhaps old format no issue... But New format is an issue because it is logically incorrect.\n\n\n[Some more details](https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/125265)\n\n\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3335448%2F1b186578414174aeb746d9f163504e2b%2FScreenshot_1.png?generation=1579340738605479&amp;alt=media)\n\n\n@juliaelliott  Perhaps you can understand the frustration of 23 days ...",
    "722475": "siriguna Sorry, I'm not following you. This is the TF2.0 competition, not Severstal. The `sample_submission` in this competition has never changed since launch.",
    "722586": "juliaelliott Thank you for the quick response, I understand it is TF2.0. Unfortunately, no one responding at Severstal. In spite of screaming for the past 23 days... \n\nCould you please influence someone in the Kaggle team to respond to this issue.\n\nThanks in advance",
    "724387": "juliaelliott please someone help me ... at least state me it is just my problem **or** issue to everyone at Severstal competition  **or** after reopened for late submission, is anyone could able to submit successfully?\n\nWaiting for reply\n\nThanks in advance",
    "725126": "siriguna This isn't the right place to be asking that question, as I am not directly aware of/the person responsible for that competition. Please check that competition's forum. After a quick look, it seems [this might answer your question](https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/126923).",
    "725451": "juliaelliott  thanks once again for quick reply. happen to fix the problem. Actually found some work around. (Even though it is kernel competition but submit the submission file only)"
  },
  "source": "meta"
}