{
  "id": 388356,
  "title": "Questions about scoring run-time",
  "url": "/competitions/predict-student-performance-from-game-play/discussion/388356",
  "author_name": "",
  "post_date": "2023-02-17T04:02:32.260289200Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi everyone! Hope you guys are happy Kaggle-ing!</p>\n<p>I have two questions about scoring run time that I couldn't seem to find answers in the \"Submission\" tab nor the competition's description</p>\n<ul>\n<li>Is there a time-limit on scoring time? I'd be surprised if there isn't, so I best guess is that it's similar to notebook limit - 9 hours.</li>\n<li>Is there a way to know how long does a scoring take? It'd be great to be able to estimate how much bandwidth we have left for increasing our model's and pipeline's complexity.</li>\n</ul>\n<p>Thank you!</p>",
  "messages": [
    {
      "id": "2147966",
      "postDate": "02/17/2023 04:02:32",
      "content": "<p>Hi everyone! Hope you guys are happy Kaggle-ing!</p>\n<p>I have two questions about scoring run time that I couldn't seem to find answers in the \"Submission\" tab nor the competition's description</p>\n<ul>\n<li>Is there a time-limit on scoring time? I'd be surprised if there isn't, so I best guess is that it's similar to notebook limit - 9 hours.</li>\n<li>Is there a way to know how long does a scoring take? It'd be great to be able to estimate how much bandwidth we have left for increasing our model's and pipeline's complexity.</li>\n</ul>\n<p>Thank you!</p>",
      "rawMarkdown": "Hi everyone! Hope you guys are happy Kaggle-ing!\n\nI have two questions about scoring run time that I couldn't seem to find answers in the \"Submission\" tab nor the competition's description\n- Is there a time-limit on scoring time? I'd be surprised if there isn't, so I best guess is that it's similar to notebook limit - 9 hours.\n- Is there a way to know how long does a scoring take? It'd be great to be able to estimate how much bandwidth we have left for increasing our model's and pipeline's complexity.\n\nThank you!",
      "votes": null
    },
    {
      "id": "2148120",
      "postDate": "02/17/2023 06:40:18",
      "content": "<p>hello <a href=\"https://www.kaggle.com/hoangnguyen719\" target=\"_blank\">@hoangnguyen719</a> :</p>\n<ol>\n<li><code>CPU Notebook &lt;= 9 hours run-time</code> in code requirements  tab</li>\n<li>According to the host : <br>\n<code>Note that the hidden test set is roughly as large as the training set; you should expect it will take much longer to run on than the three test samples provided.</code><br>\nand we have 3 sessions in the interactive mode so you can approximate the scoring time.</li>\n</ol>",
      "rawMarkdown": "hello @hoangnguyen719 :\n1. `CPU Notebook <= 9 hours run-time` in code requirements  tab\n2. According to the host : \n`Note that the hidden test set is roughly as large as the training set; you should expect it will take much longer to run on than the three test samples provided.`\nand we have 3 sessions in the interactive mode so you can approximate the scoring time.",
      "votes": null
    },
    {
      "id": "2148127",
      "postDate": "02/17/2023 06:52:59",
      "content": "<p>Thanks <a href=\"https://www.kaggle.com/ihebch\" target=\"_blank\">@ihebch</a> !</p>\n<blockquote>\n  <p>CPU Notebook &lt;= 9 hours run-time in code requirements tab</p>\n</blockquote>\n<p>To clarify, I was referring to the notebook submission's runtime, not the notebook commit's runtime. Are you saying that both of them has a limit of 9 hours?</p>\n<blockquote>\n  <p>According to the host :<br>\n  Note that the hidden test set is roughly as large as the training set; you should expect it will take much longer to run on than the three test samples provided.<br>\n  and we have 3 sessions in the interactive mode so you can approximate the scoring time.</p>\n</blockquote>\n<p>Because we are using the host's API whose runtime is unknown to us, I was worried that the scoring process may take much longer than training. We can mimic the API's flow by splitting and iterating over the train set, but since we don't know the size of a batch, the runtime can still vary.</p>",
      "rawMarkdown": "Thanks @ihebch !\n\n> CPU Notebook <= 9 hours run-time in code requirements tab\n\nTo clarify, I was referring to the notebook submission's runtime, not the notebook commit's runtime. Are you saying that both of them has a limit of 9 hours?\n\n> According to the host :\nNote that the hidden test set is roughly as large as the training set; you should expect it will take much longer to run on than the three test samples provided.\nand we have 3 sessions in the interactive mode so you can approximate the scoring time.\n\nBecause we are using the host's API whose runtime is unknown to us, I was worried that the scoring process may take much longer than training. We can mimic the API's flow by splitting and iterating over the train set, but since we don't know the size of a batch, the runtime can still vary.",
      "votes": null
    },
    {
      "id": "2148131",
      "postDate": "02/17/2023 07:03:00",
      "content": "<p>1 . Yes both have a limit of 9 hours.<br>\n2 . We have access to the first three sessions in the debug mode using the api so we can approximate the scoring time since the number of sessions in the test is near the training ones<code>(nbr_training_session * inference_time_of_three_samples/3)</code> , but it's not very accurate since the sessions_length varies.</p>",
      "rawMarkdown": "1 . Yes both have a limit of 9 hours.\n2 . We have access to the first three sessions in the debug mode using the api so we can approximate the scoring time since the number of sessions in the test is near the training ones`(nbr_training_session * inference_time_of_three_samples/3)` , but it's not very accurate since the sessions_length varies.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2148120,
      "author_name": "ihebch",
      "author_url": "",
      "post_date": "02/17/2023 06:40:18",
      "content": "<p>hello <a href=\"https://www.kaggle.com/hoangnguyen719\" target=\"_blank\">@hoangnguyen719</a> :</p>\n<ol>\n<li><code>CPU Notebook &lt;= 9 hours run-time</code> in code requirements  tab</li>\n<li>According to the host : <br>\n<code>Note that the hidden test set is roughly as large as the training set; you should expect it will take much longer to run on than the three test samples provided.</code><br>\nand we have 3 sessions in the interactive mode so you can approximate the scoring time.</li>\n</ol>",
      "votes": null,
      "replies": [
        {
          "id": 2148127,
          "author_name": "hoangnguyen719",
          "author_url": "",
          "post_date": "02/17/2023 06:52:59",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/ihebch\" target=\"_blank\">@ihebch</a> !</p>\n<blockquote>\n  <p>CPU Notebook &lt;= 9 hours run-time in code requirements tab</p>\n</blockquote>\n<p>To clarify, I was referring to the notebook submission's runtime, not the notebook commit's runtime. Are you saying that both of them has a limit of 9 hours?</p>\n<blockquote>\n  <p>According to the host :<br>\n  Note that the hidden test set is roughly as large as the training set; you should expect it will take much longer to run on than the three test samples provided.<br>\n  and we have 3 sessions in the interactive mode so you can approximate the scoring time.</p>\n</blockquote>\n<p>Because we are using the host's API whose runtime is unknown to us, I was worried that the scoring process may take much longer than training. We can mimic the API's flow by splitting and iterating over the train set, but since we don't know the size of a batch, the runtime can still vary.</p>",
          "votes": null,
          "replies": [
            {
              "id": 2148131,
              "author_name": "ihebch",
              "author_url": "",
              "post_date": "02/17/2023 07:03:00",
              "content": "<p>1 . Yes both have a limit of 9 hours.<br>\n2 . We have access to the first three sessions in the debug mode using the api so we can approximate the scoring time since the number of sessions in the test is near the training ones<code>(nbr_training_session * inference_time_of_three_samples/3)</code> , but it's not very accurate since the sessions_length varies.</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2147966": "Hi everyone! Hope you guys are happy Kaggle-ing!\n\nI have two questions about scoring run time that I couldn't seem to find answers in the \"Submission\" tab nor the competition's description\n- Is there a time-limit on scoring time? I'd be surprised if there isn't, so I best guess is that it's similar to notebook limit - 9 hours.\n- Is there a way to know how long does a scoring take? It'd be great to be able to estimate how much bandwidth we have left for increasing our model's and pipeline's complexity.\n\nThank you!",
    "2148120": "hello @hoangnguyen719 :\n1. `CPU Notebook <= 9 hours run-time` in code requirements  tab\n2. According to the host : \n`Note that the hidden test set is roughly as large as the training set; you should expect it will take much longer to run on than the three test samples provided.`\nand we have 3 sessions in the interactive mode so you can approximate the scoring time.",
    "2148127": "Thanks @ihebch !\n\n> CPU Notebook <= 9 hours run-time in code requirements tab\n\nTo clarify, I was referring to the notebook submission's runtime, not the notebook commit's runtime. Are you saying that both of them has a limit of 9 hours?\n\n> According to the host :\nNote that the hidden test set is roughly as large as the training set; you should expect it will take much longer to run on than the three test samples provided.\nand we have 3 sessions in the interactive mode so you can approximate the scoring time.\n\nBecause we are using the host's API whose runtime is unknown to us, I was worried that the scoring process may take much longer than training. We can mimic the API's flow by splitting and iterating over the train set, but since we don't know the size of a batch, the runtime can still vary.",
    "2148131": "1 . Yes both have a limit of 9 hours.\n2 . We have access to the first three sessions in the debug mode using the api so we can approximate the scoring time since the number of sessions in the test is near the training ones`(nbr_training_session * inference_time_of_three_samples/3)` , but it's not very accurate since the sessions_length varies."
  },
  "source": "meta"
}