{
  "id": 154339,
  "title": "Training Data Creation",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/154339",
  "author_name": "",
  "post_date": "2020-05-28T03:45:59.741350100Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi!</p>\n\n<p>I am new to Convolutional Networks using Deep Learning. I know the method \"ImageDataGenerator\" where we cluster training images using a folder name. However, in this Kaggle competition we are asked to work with the memory not exceeding more than 5 GB. I am unable to do Image Data Generation by sorting the images as the image folder size is extremely large nor I am able to sort source images (as they are read-only).</p>\n\n<p>Is there some alternative way to prepare images for training?</p>\n\n<p>Thank  You!</p>",
  "messages": [
    {
      "id": "864520",
      "postDate": "05/28/2020 03:45:59",
      "content": "<p>Hi!</p>\n\n<p>I am new to Convolutional Networks using Deep Learning. I know the method \"ImageDataGenerator\" where we cluster training images using a folder name. However, in this Kaggle competition we are asked to work with the memory not exceeding more than 5 GB. I am unable to do Image Data Generation by sorting the images as the image folder size is extremely large nor I am able to sort source images (as they are read-only).</p>\n\n<p>Is there some alternative way to prepare images for training?</p>\n\n<p>Thank  You!</p>",
      "rawMarkdown": "Hi!\n\nI am new to Convolutional Networks using Deep Learning. I know the method \"ImageDataGenerator\" where we cluster training images using a folder name. However, in this Kaggle competition we are asked to work with the memory not exceeding more than 5 GB. I am unable to do Image Data Generation by sorting the images as the image folder size is extremely large nor I am able to sort source images (as they are read-only).\n\nIs there some alternative way to prepare images for training?\n\nThank  You!",
      "votes": null
    },
    {
      "id": "864537",
      "postDate": "05/28/2020 04:02:01",
      "content": "<p>Regardless of the other discussion about images being too slow to load because of how big they are, I am not sure why you want to sort the images ?</p>\n\n<p>Currently if you want to use kernels I'd suggest to first design what your preprocess should be, like the size you want to use and use a Kernel to do that preprocessing first outputing everything to a zipfile. Then use the output as a dataset for a model. An alternative is to try to use the TFrecords I guess ?</p>",
      "rawMarkdown": "Regardless of the other discussion about images being too slow to load because of how big they are, I am not sure why you want to sort the images ?\n\nCurrently if you want to use kernels I'd suggest to first design what your preprocess should be, like the size you want to use and use a Kernel to do that preprocessing first outputing everything to a zipfile. Then use the output as a dataset for a model. An alternative is to try to use the TFrecords I guess ?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 864537,
      "author_name": "arroqc",
      "author_url": "",
      "post_date": "05/28/2020 04:02:01",
      "content": "<p>Regardless of the other discussion about images being too slow to load because of how big they are, I am not sure why you want to sort the images ?</p>\n\n<p>Currently if you want to use kernels I'd suggest to first design what your preprocess should be, like the size you want to use and use a Kernel to do that preprocessing first outputing everything to a zipfile. Then use the output as a dataset for a model. An alternative is to try to use the TFrecords I guess ?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "864520": "Hi!\n\nI am new to Convolutional Networks using Deep Learning. I know the method \"ImageDataGenerator\" where we cluster training images using a folder name. However, in this Kaggle competition we are asked to work with the memory not exceeding more than 5 GB. I am unable to do Image Data Generation by sorting the images as the image folder size is extremely large nor I am able to sort source images (as they are read-only).\n\nIs there some alternative way to prepare images for training?\n\nThank  You!",
    "864537": "Regardless of the other discussion about images being too slow to load because of how big they are, I am not sure why you want to sort the images ?\n\nCurrently if you want to use kernels I'd suggest to first design what your preprocess should be, like the size you want to use and use a Kernel to do that preprocessing first outputing everything to a zipfile. Then use the output as a dataset for a model. An alternative is to try to use the TFrecords I guess ?"
  },
  "source": "meta"
}