{
  "id": 399423,
  "title": "submission timeout and how to fix it",
  "url": "/competitions/icecube-neutrinos-in-deep-ice/discussion/399423",
  "author_name": "",
  "post_date": "2023-04-04T03:56:51.523489100Z",
  "votes": 4,
  "comment_count": 6,
  "views": 0,
  "content": "<p>getting multiple submission timeout error, </p>\n<p>trying to figure out if its caused by queue of submissions or there are lots of hidden test data to test. <br>\nif it is the size of hidden test dataset, can [host] provide the <strong>range of the total number of events</strong> is going to test ? so in our code we can estimate how long it may take, instead failing after 9 hours scoring without knowing the cause of the timeout.</p>\n<p>my notebook takes about [744.5s] to run without scoring.</p>",
  "messages": [
    {
      "id": "2208435",
      "postDate": "04/04/2023 03:56:51",
      "content": "<p>getting multiple submission timeout error, </p>\n<p>trying to figure out if its caused by queue of submissions or there are lots of hidden test data to test. <br>\nif it is the size of hidden test dataset, can [host] provide the <strong>range of the total number of events</strong> is going to test ? so in our code we can estimate how long it may take, instead failing after 9 hours scoring without knowing the cause of the timeout.</p>\n<p>my notebook takes about [744.5s] to run without scoring.</p>",
      "rawMarkdown": "getting multiple submission timeout error, \n\ntrying to figure out if its caused by queue of submissions or there are lots of hidden test data to test. \nif it is the size of hidden test dataset, can [host] provide the **range of the total number of events** is going to test ? so in our code we can estimate how long it may take, instead failing after 9 hours scoring without knowing the cause of the timeout.\n\nmy notebook takes about [744.5s] to run without scoring.",
      "votes": null
    },
    {
      "id": "2208643",
      "postDate": "04/04/2023 08:22:13",
      "content": "<p>I train model at local and only do data process and finan infer, so I haven't meet timeout yet👀</p>",
      "rawMarkdown": "I train model at local and only do data process and finan infer, so I haven't meet timeout yet👀",
      "votes": null
    },
    {
      "id": "2208672",
      "postDate": "04/04/2023 08:48:15",
      "content": "<p>i got 3 timeout, made some change to reduce the size of test dataset( i guess, the scoring process is injecting the [hidden] test data here) after each timeout, but still timeout, i am running 4th scoring, its about 5 hours now, still not getting score result.</p>",
      "rawMarkdown": "i got 3 timeout, made some change to reduce the size of test dataset( i guess, the scoring process is injecting the [hidden] test data here) after each timeout, but still timeout, i am running 4th scoring, its about 5 hours now, still not getting score result.",
      "votes": null
    },
    {
      "id": "2208712",
      "postDate": "04/04/2023 09:14:22",
      "content": "<p>From a data page - <code>expect to see roughly one million events in the hidden test set, split between multiple batches.</code></p>",
      "rawMarkdown": "From a data page - ```expect to see roughly one million events in the hidden test set, split between multiple batches.```",
      "votes": null
    },
    {
      "id": "2208783",
      "postDate": "04/04/2023 09:48:48",
      "content": "<p>you are right, in the data section, it does indicate one million events, my bad. </p>",
      "rawMarkdown": "you are right, in the data section, it does indicate one million events, my bad.",
      "votes": null
    },
    {
      "id": "2210754",
      "postDate": "04/05/2023 15:44:54",
      "content": "<p><a href=\"https://www.kaggle.com/tyeestudio\" target=\"_blank\">@tyeestudio</a> What you can do to validate the issue is create your own custom 'testset' consisting of about 5 seperate trainfiles.</p>\n<p>That should help you.</p>\n<p>But based on the time it takes for the public test set (the 744 seconds…) that is already way to long….it contains only 3 records. Try to identify with some debugging code which part(s) take to long.</p>\n<p>To give you some idea. I've tried one larger ensemble… On the public testset it took roughly 100 seconds. Submission run time was about 2 hours.</p>",
      "rawMarkdown": "tyeestudio What you can do to validate the issue is create your own custom 'testset' consisting of about 5 seperate trainfiles.\n\nThat should help you.\n\nBut based on the time it takes for the public test set (the 744 seconds...) that is already way to long....it contains only 3 records. Try to identify with some debugging code which part(s) take to long.\n\nTo give you some idea. I've tried one larger ensemble... On the public testset it took roughly 100 seconds. Submission run time was about 2 hours.",
      "votes": null
    },
    {
      "id": "2214189",
      "postDate": "04/08/2023 08:22:19",
      "content": "<p><a href=\"https://www.kaggle.com/rsmits\" target=\"_blank\">@rsmits</a> ,  thanks for the suggestion. i use the similar approach to emulate the submission process by using one of  the whole train-file ( which contains about 400,000 ? unique session_id ) to test how long it may take, but i was running out the p100 time. </p>\n<p>my train time is about 698 seconds of out of 744 seconds, and per prediction time is about 32m, in theory total process time is about a little before 9 hours. </p>\n<p>yes, i removed some code to make the process runs quicker, currently there is one running in the submission queue.  </p>",
      "rawMarkdown": "rsmits ,  thanks for the suggestion. i use the similar approach to emulate the submission process by using one of  the whole train-file ( which contains about 400,000 ? unique session_id ) to test how long it may take, but i was running out the p100 time. \n\nmy train time is about 698 seconds of out of 744 seconds, and per prediction time is about 32m, in theory total process time is about a little before 9 hours. \n\nyes, i removed some code to make the process runs quicker, currently there is one running in the submission queue.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2208643,
      "author_name": "roger92",
      "author_url": "",
      "post_date": "04/04/2023 08:22:13",
      "content": "<p>I train model at local and only do data process and finan infer, so I haven't meet timeout yet👀</p>",
      "votes": null,
      "replies": [
        {
          "id": 2208672,
          "author_name": "tyeestudio",
          "author_url": "",
          "post_date": "04/04/2023 08:48:15",
          "content": "<p>i got 3 timeout, made some change to reduce the size of test dataset( i guess, the scoring process is injecting the [hidden] test data here) after each timeout, but still timeout, i am running 4th scoring, its about 5 hours now, still not getting score result.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2208712,
      "author_name": "martynoveduard",
      "author_url": "",
      "post_date": "04/04/2023 09:14:22",
      "content": "<p>From a data page - <code>expect to see roughly one million events in the hidden test set, split between multiple batches.</code></p>",
      "votes": null,
      "replies": [
        {
          "id": 2208783,
          "author_name": "tyeestudio",
          "author_url": "",
          "post_date": "04/04/2023 09:48:48",
          "content": "<p>you are right, in the data section, it does indicate one million events, my bad. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2210754,
      "author_name": "rsmits",
      "author_url": "",
      "post_date": "04/05/2023 15:44:54",
      "content": "<p><a href=\"https://www.kaggle.com/tyeestudio\" target=\"_blank\">@tyeestudio</a> What you can do to validate the issue is create your own custom 'testset' consisting of about 5 seperate trainfiles.</p>\n<p>That should help you.</p>\n<p>But based on the time it takes for the public test set (the 744 seconds…) that is already way to long….it contains only 3 records. Try to identify with some debugging code which part(s) take to long.</p>\n<p>To give you some idea. I've tried one larger ensemble… On the public testset it took roughly 100 seconds. Submission run time was about 2 hours.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2214189,
          "author_name": "tyeestudio",
          "author_url": "",
          "post_date": "04/08/2023 08:22:19",
          "content": "<p><a href=\"https://www.kaggle.com/rsmits\" target=\"_blank\">@rsmits</a> ,  thanks for the suggestion. i use the similar approach to emulate the submission process by using one of  the whole train-file ( which contains about 400,000 ? unique session_id ) to test how long it may take, but i was running out the p100 time. </p>\n<p>my train time is about 698 seconds of out of 744 seconds, and per prediction time is about 32m, in theory total process time is about a little before 9 hours. </p>\n<p>yes, i removed some code to make the process runs quicker, currently there is one running in the submission queue.  </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2208435": "getting multiple submission timeout error, \n\ntrying to figure out if its caused by queue of submissions or there are lots of hidden test data to test. \nif it is the size of hidden test dataset, can [host] provide the **range of the total number of events** is going to test ? so in our code we can estimate how long it may take, instead failing after 9 hours scoring without knowing the cause of the timeout.\n\nmy notebook takes about [744.5s] to run without scoring.",
    "2208643": "I train model at local and only do data process and finan infer, so I haven't meet timeout yet👀",
    "2208672": "i got 3 timeout, made some change to reduce the size of test dataset( i guess, the scoring process is injecting the [hidden] test data here) after each timeout, but still timeout, i am running 4th scoring, its about 5 hours now, still not getting score result.",
    "2208712": "From a data page - ```expect to see roughly one million events in the hidden test set, split between multiple batches.```",
    "2208783": "you are right, in the data section, it does indicate one million events, my bad.",
    "2210754": "tyeestudio What you can do to validate the issue is create your own custom 'testset' consisting of about 5 seperate trainfiles.\n\nThat should help you.\n\nBut based on the time it takes for the public test set (the 744 seconds...) that is already way to long....it contains only 3 records. Try to identify with some debugging code which part(s) take to long.\n\nTo give you some idea. I've tried one larger ensemble... On the public testset it took roughly 100 seconds. Submission run time was about 2 hours.",
    "2214189": "rsmits ,  thanks for the suggestion. i use the similar approach to emulate the submission process by using one of  the whole train-file ( which contains about 400,000 ? unique session_id ) to test how long it may take, but i was running out the p100 time. \n\nmy train time is about 698 seconds of out of 744 seconds, and per prediction time is about 32m, in theory total process time is about a little before 9 hours. \n\nyes, i removed some code to make the process runs quicker, currently there is one running in the submission queue."
  },
  "source": "meta"
}