{
  "id": 99186,
  "title": "Running fit on Image Generator",
  "url": "/competitions/aptos2019-blindness-detection/discussion/99186",
  "author_name": "",
  "post_date": "2019-07-09T12:03:17.653944600Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>When we have to apply ZCA whitening on the images using ImageDataGenerator API of keras, we have to run fit on the whole training data. But in order to run the fit we need to load the whole training data in numpy array. I have doubt how will we fit it when we cannot load the whole data or when we are uising flow_from_dataframe?</p>",
  "messages": [
    {
      "id": "571262",
      "postDate": "07/09/2019 12:03:17",
      "content": "<p>When we have to apply ZCA whitening on the images using ImageDataGenerator API of keras, we have to run fit on the whole training data. But in order to run the fit we need to load the whole training data in numpy array. I have doubt how will we fit it when we cannot load the whole data or when we are uising flow_from_dataframe?</p>",
      "rawMarkdown": "When we have to apply ZCA whitening on the images using ImageDataGenerator API of keras, we have to run fit on the whole training data. But in order to run the fit we need to load the whole training data in numpy array. I have doubt how will we fit it when we cannot load the whole data or when we are uising flow_from_dataframe?",
      "votes": null
    },
    {
      "id": "571362",
      "postDate": "07/09/2019 14:28:26",
      "content": "<p>Try taking 20% of all classes and fit it.</p>",
      "rawMarkdown": "Try taking 20% of all classes and fit it.",
      "votes": null
    },
    {
      "id": "571655",
      "postDate": "07/10/2019 00:08:28",
      "content": "<p>It implies that I have to load the data anyway then the motive of using such an API is lost altogether since I can some percent of the data manually preprocess it (for satisfaction or for black box removal) with above par customization and delete it from the memory and loop over it again and again. Thanks btw.</p>",
      "rawMarkdown": "It implies that I have to load the data anyway then the motive of using such an API is lost altogether since I can some percent of the data manually preprocess it (for satisfaction or for black box removal) with above par customization and delete it from the memory and loop over it again and again. Thanks btw.",
      "votes": null
    },
    {
      "id": "580943",
      "postDate": "07/21/2019 05:38:35",
      "content": "<p>would not making a custom data generator and calling fit_generator help?</p>",
      "rawMarkdown": "would not making a custom data generator and calling fit_generator help?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 571362,
      "author_name": "mktripathi456",
      "author_url": "",
      "post_date": "07/09/2019 14:28:26",
      "content": "<p>Try taking 20% of all classes and fit it.</p>",
      "votes": null,
      "replies": [
        {
          "id": 571655,
          "author_name": "akash2sharma",
          "author_url": "",
          "post_date": "07/10/2019 00:08:28",
          "content": "<p>It implies that I have to load the data anyway then the motive of using such an API is lost altogether since I can some percent of the data manually preprocess it (for satisfaction or for black box removal) with above par customization and delete it from the memory and loop over it again and again. Thanks btw.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 580943,
      "author_name": "aliakbar09a",
      "author_url": "",
      "post_date": "07/21/2019 05:38:35",
      "content": "<p>would not making a custom data generator and calling fit_generator help?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "571262": "When we have to apply ZCA whitening on the images using ImageDataGenerator API of keras, we have to run fit on the whole training data. But in order to run the fit we need to load the whole training data in numpy array. I have doubt how will we fit it when we cannot load the whole data or when we are uising flow_from_dataframe?",
    "571362": "Try taking 20% of all classes and fit it.",
    "571655": "It implies that I have to load the data anyway then the motive of using such an API is lost altogether since I can some percent of the data manually preprocess it (for satisfaction or for black box removal) with above par customization and delete it from the memory and loop over it again and again. Thanks btw.",
    "580943": "would not making a custom data generator and calling fit_generator help?"
  },
  "source": "meta"
}