{
  "id": 377290,
  "title": "How long it will took to PreProcess Test Data?",
  "url": "/competitions/nfl-player-contact-detection/discussion/377290",
  "author_name": "Selvakumar Perumal",
  "post_date": "2023-01-10T16:35:48.804000",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I am doing this on kaggle kernel only, I created 4 separate notebooks to preprocess training data, each one took 10 hours nearly, because of biggest training dataset, kaggle kernel can't able to do at single time, Means kaggle is limited to 12 hours, I don't know the test dataset size, if someone know comment</p>",
  "messages": [
    {
      "id": 2094321,
      "postDate": "2023-01-10T17:42:42.110Z",
      "content": "<p>The data tab says that the test set is 61 plays. The train set is 240 plays. </p>\n<p>If your preprocessing took 10hr * 4 = 40 hours on train, then it sounds like it would take about 10 hours on the test set (which is longer than the 9 hour submit runtime limit) - so you'll have to find a different way to preprocess the data to get what you need.</p>",
      "rawMarkdown": "The data tab says that the test set is 61 plays. The train set is 240 plays. \n\nIf your preprocessing took 10hr * 4 = 40 hours on train, then it sounds like it would take about 10 hours on the test set (which is longer than the 9 hour submit runtime limit) - so you'll have to find a different way to preprocess the data to get what you need.",
      "votes": 3,
      "replies": [
        {
          "id": 2094960,
          "postDate": "2023-01-11T06:05:16.213Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 2097694,
      "postDate": "2023-01-12T21:19:53.533Z",
      "content": "<p>You may want to look into vectorization and loop optimization. Python for loops are notoriously slow, and eliminating a loop will often speed up that section of code by orders of magnitude. </p>",
      "rawMarkdown": "You may want to look into vectorization and loop optimization. Python for loops are notoriously slow, and eliminating a loop will often speed up that section of code by orders of magnitude. ",
      "votes": 1
    },
    {
      "id": 2094168,
      "postDate": "2023-01-10T16:35:48.803Z",
      "content": "<p>I am doing this on kaggle kernel only, I created 4 separate notebooks to preprocess training data, each one took 10 hours nearly, because of biggest training dataset, kaggle kernel can't able to do at single time, Means kaggle is limited to 12 hours, I don't know the test dataset size, if someone know comment</p>",
      "rawMarkdown": "I am doing this on kaggle kernel only, I created 4 separate notebooks to preprocess training data, each one took 10 hours nearly, because of biggest training dataset, kaggle kernel can't able to do at single time, Means kaggle is limited to 12 hours, I don't know the test dataset size, if someone know comment",
      "votes": 2
    }
  ],
  "comments": [
    {
      "id": 2094321,
      "author_name": "chris",
      "author_url": "",
      "post_date": "2023-01-10T17:42:42.110000",
      "content": "<p>The data tab says that the test set is 61 plays. The train set is 240 plays. </p>\n<p>If your preprocessing took 10hr * 4 = 40 hours on train, then it sounds like it would take about 10 hours on the test set (which is longer than the 9 hour submit runtime limit) - so you'll have to find a different way to preprocess the data to get what you need.</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2094960,
          "author_name": "",
          "author_url": "",
          "post_date": "2023-01-11T06:05:16.213000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2097694,
      "author_name": "DanielPeshkov",
      "author_url": "",
      "post_date": "2023-01-12T21:19:53.533000",
      "content": "<p>You may want to look into vectorization and loop optimization. Python for loops are notoriously slow, and eliminating a loop will often speed up that section of code by orders of magnitude. </p>",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2094321": "The data tab says that the test set is 61 plays. The train set is 240 plays. \n\nIf your preprocessing took 10hr * 4 = 40 hours on train, then it sounds like it would take about 10 hours on the test set (which is longer than the 9 hour submit runtime limit) - so you'll have to find a different way to preprocess the data to get what you need.",
    "2097694": "You may want to look into vectorization and loop optimization. Python for loops are notoriously slow, and eliminating a loop will often speed up that section of code by orders of magnitude. ",
    "2094168": "I am doing this on kaggle kernel only, I created 4 separate notebooks to preprocess training data, each one took 10 hours nearly, because of biggest training dataset, kaggle kernel can't able to do at single time, Means kaggle is limited to 12 hours, I don't know the test dataset size, if someone know comment"
  }
}