{
  "id": 174415,
  "title": "Problem with test set preprocessing ",
  "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/174415",
  "author_name": "",
  "post_date": "2020-08-13T13:15:36.589489900Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I havent been able to submit my results beacause there is somehow a problem with my test set preprocessing.</p>\n<p>the code is below.</p>\n<p>This runs just fine when i commit and then when i try to run it on the hidden set it crashes..</p>\n<p>someone have any thoughts?</p>",
  "messages": [
    {
      "id": "969114",
      "postDate": "08/13/2020 13:15:36",
      "content": "<p>I havent been able to submit my results beacause there is somehow a problem with my test set preprocessing.</p>\n<p>the code is below.</p>\n<p>This runs just fine when i commit and then when i try to run it on the hidden set it crashes..</p>\n<p>someone have any thoughts?</p>",
      "rawMarkdown": "I havent been able to submit my results beacause there is somehow a problem with my test set preprocessing.\n\nthe code is below.\n\nThis runs just fine when i commit and then when i try to run it on the hidden set it crashes..\n\nsomeone have any thoughts?",
      "votes": null
    },
    {
      "id": "969362",
      "postDate": "08/13/2020 16:26:04",
      "content": "<p>You might be running out of disk space. I think you are trying to store all the images in full resolution, non-compressed. I don't know if anybody has figured out the size of the test dataset.</p>\n<p>Depending on your needs, considered resizing the images. Consider the datatype of your saved data. Can you store as int8? Consider compression.</p>\n<p>You can try your code with \"train\" instead of \"test\" to see if you run into a disk space problem with the train data alone.</p>\n<p>-Rich</p>",
      "rawMarkdown": "You might be running out of disk space. I think you are trying to store all the images in full resolution, non-compressed. I don't know if anybody has figured out the size of the test dataset.\n\nDepending on your needs, considered resizing the images. Consider the datatype of your saved data. Can you store as int8? Consider compression.\n\nYou can try your code with \"train\" instead of \"test\" to see if you run into a disk space problem with the train data alone.\n\n-Rich",
      "votes": null
    },
    {
      "id": "981928",
      "postDate": "08/22/2020 21:15:58",
      "content": "<p>Thanks <br>\nquadcore/Richard Epstein!!</p>\n<p>That was actually the problems. Though its weird cause all of the training set in jpeg is 1.1GB so i thought the test set would be smaller then 5GB…..</p>",
      "rawMarkdown": "Thanks \nquadcore/Richard Epstein!!\n\nThat was actually the problems. Though its weird cause all of the training set in jpeg is 1.1GB so i thought the test set would be smaller then 5GB.....",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 969362,
      "author_name": "richardepstein",
      "author_url": "",
      "post_date": "08/13/2020 16:26:04",
      "content": "<p>You might be running out of disk space. I think you are trying to store all the images in full resolution, non-compressed. I don't know if anybody has figured out the size of the test dataset.</p>\n<p>Depending on your needs, considered resizing the images. Consider the datatype of your saved data. Can you store as int8? Consider compression.</p>\n<p>You can try your code with \"train\" instead of \"test\" to see if you run into a disk space problem with the train data alone.</p>\n<p>-Rich</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 981928,
      "author_name": "omerhacohen",
      "author_url": "",
      "post_date": "08/22/2020 21:15:58",
      "content": "<p>Thanks <br>\nquadcore/Richard Epstein!!</p>\n<p>That was actually the problems. Though its weird cause all of the training set in jpeg is 1.1GB so i thought the test set would be smaller then 5GB…..</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "969114": "I havent been able to submit my results beacause there is somehow a problem with my test set preprocessing.\n\nthe code is below.\n\nThis runs just fine when i commit and then when i try to run it on the hidden set it crashes..\n\nsomeone have any thoughts?",
    "969362": "You might be running out of disk space. I think you are trying to store all the images in full resolution, non-compressed. I don't know if anybody has figured out the size of the test dataset.\n\nDepending on your needs, considered resizing the images. Consider the datatype of your saved data. Can you store as int8? Consider compression.\n\nYou can try your code with \"train\" instead of \"test\" to see if you run into a disk space problem with the train data alone.\n\n-Rich",
    "981928": "Thanks \nquadcore/Richard Epstein!!\n\nThat was actually the problems. Though its weird cause all of the training set in jpeg is 1.1GB so i thought the test set would be smaller then 5GB....."
  },
  "source": "meta"
}