{
  "id": 134970,
  "title": "ImageDataGenerators for Keras",
  "url": "/competitions/deepfake-detection-challenge/discussion/134970",
  "author_name": "",
  "post_date": "2020-03-11T12:08:35.956677400Z",
  "votes": null,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Hey folks,</p>\n\n<p>Looks like I need some help. I have my entire dataset in a dataframe and I use Imagedatagenerator's flow_from_dataframe while training model with single image.</p>\n\n<p>However, right now, I plan on trying out a few LSTM models and was pretty much confused on how to use imagedatagenerator for getting hold of multiple images. If anyone has already used this part, could you kindly help out a bit.</p>\n\n<p>Thanks,\nAkash</p>",
  "messages": [
    {
      "id": "768988",
      "postDate": "03/11/2020 12:08:35",
      "content": "<p>Hey folks,</p>\n\n<p>Looks like I need some help. I have my entire dataset in a dataframe and I use Imagedatagenerator's flow_from_dataframe while training model with single image.</p>\n\n<p>However, right now, I plan on trying out a few LSTM models and was pretty much confused on how to use imagedatagenerator for getting hold of multiple images. If anyone has already used this part, could you kindly help out a bit.</p>\n\n<p>Thanks,\nAkash</p>",
      "rawMarkdown": "Hey folks,\n\nLooks like I need some help. I have my entire dataset in a dataframe and I use Imagedatagenerator's flow_from_dataframe while training model with single image.\n\nHowever, right now, I plan on trying out a few LSTM models and was pretty much confused on how to use imagedatagenerator for getting hold of multiple images. If anyone has already used this part, could you kindly help out a bit.\n\nThanks,\nAkash",
      "votes": null
    },
    {
      "id": "769301",
      "postDate": "03/11/2020 18:38:24",
      "content": "<p>In my opinion, custom generators are better. You have more control. I don't think you can use image data generator can output 3D data (num_frames,size,size).</p>",
      "rawMarkdown": "In my opinion, custom generators are better. You have more control. I don't think you can use image data generator can output 3D data (num_frames,size,size).",
      "votes": null
    },
    {
      "id": "770192",
      "postDate": "03/12/2020 17:00:34",
      "content": "<p>Ah. I see. Thanks for the suggestion.</p>",
      "rawMarkdown": "Ah. I see. Thanks for the suggestion.",
      "votes": null
    },
    {
      "id": "771220",
      "postDate": "03/13/2020 21:57:04",
      "content": "<p>You'll need to create a custom generator. Here is a good example of how to do so, by having your generator inheriting from the keras sequence class, which will prevent a lot of issues when using multi-thread fit:\n[https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly]</p>",
      "rawMarkdown": "You'll need to create a custom generator. Here is a good example of how to do so, by having your generator inheriting from the keras sequence class, which will prevent a lot of issues when using multi-thread fit:\n[https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly]",
      "votes": null
    },
    {
      "id": "771235",
      "postDate": "03/13/2020 22:35:00",
      "content": "<p>Personally, I prefer rather function instead of class. You can do it by:\n<code>\ndef generator():\n    while True:\n        yield get_next_x(), get_next_y()\n</code></p>",
      "rawMarkdown": "Personally, I prefer rather function instead of class. You can do it by:\n```\ndef generator():\n    while True:\n        yield get_next_x(), get_next_y()\n```",
      "votes": null
    },
    {
      "id": "771239",
      "postDate": "03/13/2020 22:45:38",
      "content": "<p><a href=\"/unkownhihi\">@unkownhihi</a> If you want to use multiprocessing=True in the fit method, that  approach will cause weird problems and mess up your training.</p>",
      "rawMarkdown": "unkownhihi If you want to use multiprocessing=True in the fit method, that  approach will cause weird problems and mess up your training.",
      "votes": null
    },
    {
      "id": "771242",
      "postDate": "03/13/2020 22:47:52",
      "content": "<p>I never use multiprocessing lol, cuz kaggle only has one vcpu and multiprocessing would make it worse.</p>",
      "rawMarkdown": "I never use multiprocessing lol, cuz kaggle only has one vcpu and multiprocessing would make it worse.",
      "votes": null
    },
    {
      "id": "771246",
      "postDate": "03/13/2020 23:18:06",
      "content": "<p>Hello,\nhere is a simple example how you can write your own generator to process images and generate batches in a way you want: <a href=\"https://towardsdatascience.com/writing-custom-keras-generators-fe815d992c5a\">https://towardsdatascience.com/writing-custom-keras-generators-fe815d992c5a</a></p>",
      "rawMarkdown": "Hello,\nhere is a simple example how you can write your own generator to process images and generate batches in a way you want: https://towardsdatascience.com/writing-custom-keras-generators-fe815d992c5a",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 769301,
      "author_name": "unkownhihi",
      "author_url": "",
      "post_date": "03/11/2020 18:38:24",
      "content": "<p>In my opinion, custom generators are better. You have more control. I don't think you can use image data generator can output 3D data (num_frames,size,size).</p>",
      "votes": null,
      "replies": [
        {
          "id": 770192,
          "author_name": "akashnandi",
          "author_url": "",
          "post_date": "03/12/2020 17:00:34",
          "content": "<p>Ah. I see. Thanks for the suggestion.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 771220,
      "author_name": "ngcferreira",
      "author_url": "",
      "post_date": "03/13/2020 21:57:04",
      "content": "<p>You'll need to create a custom generator. Here is a good example of how to do so, by having your generator inheriting from the keras sequence class, which will prevent a lot of issues when using multi-thread fit:\n[https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly]</p>",
      "votes": null,
      "replies": [
        {
          "id": 771235,
          "author_name": "unkownhihi",
          "author_url": "",
          "post_date": "03/13/2020 22:35:00",
          "content": "<p>Personally, I prefer rather function instead of class. You can do it by:\n<code>\ndef generator():\n    while True:\n        yield get_next_x(), get_next_y()\n</code></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 771239,
          "author_name": "ngcferreira",
          "author_url": "",
          "post_date": "03/13/2020 22:45:38",
          "content": "<p><a href=\"/unkownhihi\">@unkownhihi</a> If you want to use multiprocessing=True in the fit method, that  approach will cause weird problems and mess up your training.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 771242,
          "author_name": "unkownhihi",
          "author_url": "",
          "post_date": "03/13/2020 22:47:52",
          "content": "<p>I never use multiprocessing lol, cuz kaggle only has one vcpu and multiprocessing would make it worse.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 771246,
      "author_name": "vpaslay",
      "author_url": "",
      "post_date": "03/13/2020 23:18:06",
      "content": "<p>Hello,\nhere is a simple example how you can write your own generator to process images and generate batches in a way you want: <a href=\"https://towardsdatascience.com/writing-custom-keras-generators-fe815d992c5a\">https://towardsdatascience.com/writing-custom-keras-generators-fe815d992c5a</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "768988": "Hey folks,\n\nLooks like I need some help. I have my entire dataset in a dataframe and I use Imagedatagenerator's flow_from_dataframe while training model with single image.\n\nHowever, right now, I plan on trying out a few LSTM models and was pretty much confused on how to use imagedatagenerator for getting hold of multiple images. If anyone has already used this part, could you kindly help out a bit.\n\nThanks,\nAkash",
    "769301": "In my opinion, custom generators are better. You have more control. I don't think you can use image data generator can output 3D data (num_frames,size,size).",
    "770192": "Ah. I see. Thanks for the suggestion.",
    "771220": "You'll need to create a custom generator. Here is a good example of how to do so, by having your generator inheriting from the keras sequence class, which will prevent a lot of issues when using multi-thread fit:\n[https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly]",
    "771235": "Personally, I prefer rather function instead of class. You can do it by:\n```\ndef generator():\n    while True:\n        yield get_next_x(), get_next_y()\n```",
    "771239": "unkownhihi If you want to use multiprocessing=True in the fit method, that  approach will cause weird problems and mess up your training.",
    "771242": "I never use multiprocessing lol, cuz kaggle only has one vcpu and multiprocessing would make it worse.",
    "771246": "Hello,\nhere is a simple example how you can write your own generator to process images and generate batches in a way you want: https://towardsdatascience.com/writing-custom-keras-generators-fe815d992c5a"
  },
  "source": "meta"
}