{
  "id": 103070,
  "title": "Kernels errors resolved (August 6, 15:50 PDT)",
  "url": "/competitions/aptos2019-blindness-detection/discussion/103070",
  "author_name": "Sohier Dane",
  "post_date": "2019-08-06T23:06:20.148000",
  "votes": -8,
  "comment_count": 13,
  "views": 0,
  "content": "<p>Apologies for another round of kernels errors, we know the last few days have been frustrating. Our engineers have just deployed a patch that should fix the latest issue, which was live for about 19 hours. We are looking into steps we can take to reduce both the error rate and the time it takes for us to properly detect and escalates these issues internally.  </p>",
  "messages": [
    {
      "id": 593653,
      "postDate": "2019-08-06T23:49:01.543Z",
      "content": "<p>Given the recent cluster of errors, a suggestion might be to add more submissions or extend the submission deadline?</p>",
      "rawMarkdown": "Given the recent cluster of errors, a suggestion might be to add more submissions or extend the submission deadline?",
      "votes": 7
    },
    {
      "id": 593674,
      "postDate": "2019-08-07T01:07:11.057Z",
      "content": "<p>still having issue of submission, i notice i had one submission been hanging from the morning (~12hrs) and all the subsequent submissions resulted in instant submission error (im only doing fast submission for public test only)<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1878085%2F2b37ca293df4f84db1b09c5b117736f7%2FScreenshot%20from%202019-08-06%2021-05-02.png?generation=1565140055762983&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "still having issue of submission, i notice i had one submission been hanging from the morning (~12hrs) and all the subsequent submissions resulted in instant submission error (im only doing fast submission for public test only)![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1878085%2F2b37ca293df4f84db1b09c5b117736f7%2FScreenshot%20from%202019-08-06%2021-05-02.png?generation=1565140055762983&amp;alt=media)\n",
      "votes": 1
    },
    {
      "id": 593644,
      "postDate": "2019-08-06T23:06:20.147Z",
      "content": "<p>Apologies for another round of kernels errors, we know the last few days have been frustrating. Our engineers have just deployed a patch that should fix the latest issue, which was live for about 19 hours. We are looking into steps we can take to reduce both the error rate and the time it takes for us to properly detect and escalates these issues internally.  </p>",
      "rawMarkdown": "Apologies for another round of kernels errors, we know the last few days have been frustrating. Our engineers have just deployed a patch that should fix the latest issue, which was live for about 19 hours. We are looking into steps we can take to reduce both the error rate and the time it takes for us to properly detect and escalates these issues internally.  ",
      "votes": -8
    },
    {
      "id": 594134,
      "postDate": "2019-08-07T16:11:07.607Z",
      "content": "<p>Apologies for the problems continuing after my initial post, it turned out that there were two underlying problems in this instance. The remainder of the issues should have stopped around 7:20 PM PDT last night when we deployed a second fix.</p>",
      "rawMarkdown": "Apologies for the problems continuing after my initial post, it turned out that there were two underlying problems in this instance. The remainder of the issues should have stopped around 7:20 PM PDT last night when we deployed a second fix.",
      "replies": [
        {
          "id": 594150,
          "postDate": "2019-08-07T16:30:45.223Z",
          "content": "<p>at least mine still didnt work, just made two submission (one was from previous submission that had worked<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1878085%2F2eb92a33a694f2765fbbf35a51945ddc%2FScreen%20Shot%202019-08-07%20at%2012.29.44%20PM.png?generation=1565195438930393&amp;alt=media\" alt=\"\">\n)</p>",
          "rawMarkdown": "at least mine still didnt work, just made two submission (one was from previous submission that had worked![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1878085%2F2eb92a33a694f2765fbbf35a51945ddc%2FScreen%20Shot%202019-08-07%20at%2012.29.44%20PM.png?generation=1565195438930393&amp;alt=media)\n)"
        },
        {
          "id": 594152,
          "postDate": "2019-08-07T16:38:58.200Z",
          "content": "<p>I'll take a look now; overall error rates look low at the moment but I would expect reruns of previously functioning kernels to pass. Thank you for the report!</p>",
          "rawMarkdown": "I'll take a look now; overall error rates look low at the moment but I would expect reruns of previously functioning kernels to pass. Thank you for the report!",
          "votes": 1
        },
        {
          "id": 594240,
          "postDate": "2019-08-07T18:44:55.620Z",
          "content": "<p><a href=\"/samshipengs\">@samshipengs</a> we're showing that the dataset associated with that kernel has been deleted, that is the most likely reason for the error in this case.</p>",
          "rawMarkdown": "@samshipengs we're showing that the dataset associated with that kernel has been deleted, that is the most likely reason for the error in this case."
        },
        {
          "id": 594304,
          "postDate": "2019-08-07T20:08:56.417Z",
          "content": "<p>hmm interesting, i have deleted and then re-added files (although with the same folder name and file name), and it's showing the newly added dataset is present... Maybe i will just make a new kernel to submit</p>",
          "rawMarkdown": "hmm interesting, i have deleted and then re-added files (although with the same folder name and file name), and it's showing the newly added dataset is present... Maybe i will just make a new kernel to submit"
        },
        {
          "id": 594306,
          "postDate": "2019-08-07T20:14:17.730Z",
          "content": "<p>ok, so if i start a new (exactly same) kernel, the submission works, seems like the file deletes and re-adds history somehow messed things up </p>",
          "rawMarkdown": "ok, so if i start a new (exactly same) kernel, the submission works, seems like the file deletes and re-adds history somehow messed things up "
        }
      ]
    },
    {
      "id": 594064,
      "postDate": "2019-08-07T14:07:51.303Z",
      "content": "<p>It seems that is running fine for the moment, thanks</p>",
      "rawMarkdown": "It seems that is running fine for the moment, thanks"
    },
    {
      "id": 593691,
      "postDate": "2019-08-07T01:45:34.563Z",
      "content": "<p>the issue still exists <a href=\"/sohier\">@sohier</a> <a href=\"/inversion\">@inversion</a> </p>",
      "rawMarkdown": "the issue still exists @sohier @inversion "
    },
    {
      "id": 593676,
      "postDate": "2019-08-07T01:09:27.780Z",
      "content": "<p>Still having this error... had a kernel that took 31-34 minutes to run ~17 hours ago but now its been running for 1hr+</p>",
      "rawMarkdown": "Still having this error... had a kernel that took 31-34 minutes to run ~17 hours ago but now its been running for 1hr+"
    },
    {
      "id": 593938,
      "postDate": "2019-08-07T10:07:46.230Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 593655,
      "postDate": "2019-08-07T00:00:16.870Z",
      "rawMarkdown": "",
      "votes": 3,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 593653,
      "author_name": "ilovescience",
      "author_url": "",
      "post_date": "2019-08-06T23:49:01.543000",
      "content": "<p>Given the recent cluster of errors, a suggestion might be to add more submissions or extend the submission deadline?</p>",
      "votes": 7,
      "replies": []
    },
    {
      "id": 593674,
      "author_name": "samshipengs",
      "author_url": "",
      "post_date": "2019-08-07T01:07:11.057000",
      "content": "<p>still having issue of submission, i notice i had one submission been hanging from the morning (~12hrs) and all the subsequent submissions resulted in instant submission error (im only doing fast submission for public test only)<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1878085%2F2b37ca293df4f84db1b09c5b117736f7%2FScreenshot%20from%202019-08-06%2021-05-02.png?generation=1565140055762983&amp;alt=media\" alt=\"\"></p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 594134,
      "author_name": "Sohier Dane",
      "author_url": "",
      "post_date": "2019-08-07T16:11:07.607000",
      "content": "<p>Apologies for the problems continuing after my initial post, it turned out that there were two underlying problems in this instance. The remainder of the issues should have stopped around 7:20 PM PDT last night when we deployed a second fix.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 594150,
          "author_name": "samshipengs",
          "author_url": "",
          "post_date": "2019-08-07T16:30:45.223000",
          "content": "<p>at least mine still didnt work, just made two submission (one was from previous submission that had worked<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1878085%2F2eb92a33a694f2765fbbf35a51945ddc%2FScreen%20Shot%202019-08-07%20at%2012.29.44%20PM.png?generation=1565195438930393&amp;alt=media\" alt=\"\">\n)</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 594152,
          "author_name": "Sohier Dane",
          "author_url": "",
          "post_date": "2019-08-07T16:38:58.200000",
          "content": "<p>I'll take a look now; overall error rates look low at the moment but I would expect reruns of previously functioning kernels to pass. Thank you for the report!</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 594240,
          "author_name": "Sohier Dane",
          "author_url": "",
          "post_date": "2019-08-07T18:44:55.620000",
          "content": "<p><a href=\"/samshipengs\">@samshipengs</a> we're showing that the dataset associated with that kernel has been deleted, that is the most likely reason for the error in this case.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 594304,
          "author_name": "samshipengs",
          "author_url": "",
          "post_date": "2019-08-07T20:08:56.417000",
          "content": "<p>hmm interesting, i have deleted and then re-added files (although with the same folder name and file name), and it's showing the newly added dataset is present... Maybe i will just make a new kernel to submit</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 594306,
          "author_name": "samshipengs",
          "author_url": "",
          "post_date": "2019-08-07T20:14:17.730000",
          "content": "<p>ok, so if i start a new (exactly same) kernel, the submission works, seems like the file deletes and re-adds history somehow messed things up </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 594064,
      "author_name": "Petru Radu",
      "author_url": "",
      "post_date": "2019-08-07T14:07:51.303000",
      "content": "<p>It seems that is running fine for the moment, thanks</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 593691,
      "author_name": "Ravi Kumar Vadapalli",
      "author_url": "",
      "post_date": "2019-08-07T01:45:34.563000",
      "content": "<p>the issue still exists <a href=\"/sohier\">@sohier</a> <a href=\"/inversion\">@inversion</a> </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 593676,
      "author_name": "sh",
      "author_url": "",
      "post_date": "2019-08-07T01:09:27.780000",
      "content": "<p>Still having this error... had a kernel that took 31-34 minutes to run ~17 hours ago but now its been running for 1hr+</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 593938,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-08-07T10:07:46.230000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 593655,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-08-07T00:00:16.870000",
      "content": "",
      "votes": 3,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "593653": "Given the recent cluster of errors, a suggestion might be to add more submissions or extend the submission deadline?",
    "593674": "still having issue of submission, i notice i had one submission been hanging from the morning (~12hrs) and all the subsequent submissions resulted in instant submission error (im only doing fast submission for public test only)![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1878085%2F2b37ca293df4f84db1b09c5b117736f7%2FScreenshot%20from%202019-08-06%2021-05-02.png?generation=1565140055762983&amp;alt=media)\n",
    "593644": "Apologies for another round of kernels errors, we know the last few days have been frustrating. Our engineers have just deployed a patch that should fix the latest issue, which was live for about 19 hours. We are looking into steps we can take to reduce both the error rate and the time it takes for us to properly detect and escalates these issues internally.  ",
    "594134": "Apologies for the problems continuing after my initial post, it turned out that there were two underlying problems in this instance. The remainder of the issues should have stopped around 7:20 PM PDT last night when we deployed a second fix.",
    "594064": "It seems that is running fine for the moment, thanks",
    "593691": "the issue still exists @sohier @inversion ",
    "593676": "Still having this error... had a kernel that took 31-34 minutes to run ~17 hours ago but now its been running for 1hr+",
    "593938": "",
    "593655": ""
  }
}