{
  "id": 230563,
  "title": "Taking too much time to fit the model",
  "url": "/competitions/plant-pathology-2021-fgvc8/discussion/230563",
  "author_name": "",
  "post_date": "2021-04-04T14:21:07.380866900Z",
  "votes": 1,
  "comment_count": 5,
  "views": 0,
  "content": "<p>After fiting the model it takes way too much time to run the epochs. Probably because of the large dataset. Is there any way to fasten up the process? </p>",
  "messages": [
    {
      "id": "1262620",
      "postDate": "04/04/2021 14:21:07",
      "content": "<p>After fiting the model it takes way too much time to run the epochs. Probably because of the large dataset. Is there any way to fasten up the process? </p>",
      "rawMarkdown": "After fiting the model it takes way too much time to run the epochs. Probably because of the large dataset. Is there any way to fasten up the process?",
      "votes": null
    },
    {
      "id": "1263679",
      "postDate": "04/05/2021 15:37:54",
      "content": "<p>The actual image shape is bigger as well as the dataset is also bigger. <a href=\"https://www.kaggle.com/shailkardani\" target=\"_blank\">@shailkardani</a> you can use this resize dataset. <a href=\"https://www.kaggle.com/ankursingh12/resized-plant2021\" target=\"_blank\">https://www.kaggle.com/ankursingh12/resized-plant2021</a></p>",
      "rawMarkdown": "The actual image shape is bigger as well as the dataset is also bigger. @shailkardani you can use this resize dataset. https://www.kaggle.com/ankursingh12/resized-plant2021",
      "votes": null
    },
    {
      "id": "1269667",
      "postDate": "04/10/2021 18:51:46",
      "content": "<p>There're some things you can do to increase training speed. For example, you can reduce image size, or use less complex model.</p>",
      "rawMarkdown": "There're some things you can do to increase training speed. For example, you can reduce image size, or use less complex model.",
      "votes": null
    },
    {
      "id": "1271395",
      "postDate": "04/12/2021 15:11:27",
      "content": "<p>This should help. Thanks ,I'll check it out.</p>",
      "rawMarkdown": "This should help. Thanks ,I'll check it out.",
      "votes": null
    },
    {
      "id": "1272753",
      "postDate": "04/13/2021 18:18:48",
      "content": "<p>What are you using to train the model?</p>",
      "rawMarkdown": "What are you using to train the model?",
      "votes": null
    },
    {
      "id": "1274336",
      "postDate": "04/15/2021 07:54:44",
      "content": "<p>May be because you are using ImageDataGenerator , Each image is quite big in the dataset so during training the model predicts with GPU and loads and augments the image with CPU . Thus GPU cant get completely activated as it is always taking sometime to load the images with CPU giving the GPU some break and also because we are loading images and training our model at the same time thus it takes some extra time to load the image .you can either augment the images yourself as numpy array, or use this notebook <a href=\"https://www.kaggle.com/datasciencegeek/resize-images\" target=\"_blank\">https://www.kaggle.com/datasciencegeek/resize-images</a>( Use the output files as data) and just change the path. Hope it helps . Here are my logs after using the notebook<br>\n<code>Epoch 1/10 466/466 [==============================] - 229s 492ms/step - loss: 0.1528 - acc: 0.6191 - val_loss: 0.1517 - val_acc: 0.6186 Epoch 2/10 466/466 [==============================] - 229s 492ms/step - loss: 0.1514 - acc: 0.6212 - val_loss: 0.1501 - val_acc: 0.6296 Epoch 3/10 466/466 [==============================] - 228s 490ms/step - loss: 0.1524 - acc: 0.6235 - val_loss: 0.1517 - val_acc: 0.6173 Epoch 4/10 466/466 [==============================] - 228s 490ms/step - loss: 0.1511 - acc: 0.6238 - val_loss: 0.1488 - val_acc: 0.6299 Epoch 5/10 466/466 [==============================] - 229s 491ms/step - loss: 0.1513 - acc: 0.6233 - val_loss: 0.1508 - val_acc: 0.6294 Epoch 6/10 466/466 [==============================] - 229s 492ms/step - loss: 0.1501 - acc: 0.6270 - val_loss: 0.1483 - val_acc: 0.6275 Epoch 7/10 466/466 [==============================] - 230s 494ms/step - loss: 0.1490 - acc: 0.6302 - val_loss: 0.1509 - val_acc: 0.6259 Epoch 8/10 382/466 [=======================&gt;......] - ETA: 32s - loss: 0.1495 - acc: 0.6268\n</code></p>",
      "rawMarkdown": "May be because you are using ImageDataGenerator , Each image is quite big in the dataset so during training the model predicts with GPU and loads and augments the image with CPU . Thus GPU cant get completely activated as it is always taking sometime to load the images with CPU giving the GPU some break and also because we are loading images and training our model at the same time thus it takes some extra time to load the image .you can either augment the images yourself as numpy array, or use this notebook https://www.kaggle.com/datasciencegeek/resize-images( Use the output files as data) and just change the path. Hope it helps . Here are my logs after using the notebook\n`Epoch 1/10 466/466 [==============================] - 229s 492ms/step - loss: 0.1528 - acc: 0.6191 - val_loss: 0.1517 - val_acc: 0.6186 Epoch 2/10 466/466 [==============================] - 229s 492ms/step - loss: 0.1514 - acc: 0.6212 - val_loss: 0.1501 - val_acc: 0.6296 Epoch 3/10 466/466 [==============================] - 228s 490ms/step - loss: 0.1524 - acc: 0.6235 - val_loss: 0.1517 - val_acc: 0.6173 Epoch 4/10 466/466 [==============================] - 228s 490ms/step - loss: 0.1511 - acc: 0.6238 - val_loss: 0.1488 - val_acc: 0.6299 Epoch 5/10 466/466 [==============================] - 229s 491ms/step - loss: 0.1513 - acc: 0.6233 - val_loss: 0.1508 - val_acc: 0.6294 Epoch 6/10 466/466 [==============================] - 229s 492ms/step - loss: 0.1501 - acc: 0.6270 - val_loss: 0.1483 - val_acc: 0.6275 Epoch 7/10 466/466 [==============================] - 230s 494ms/step - loss: 0.1490 - acc: 0.6302 - val_loss: 0.1509 - val_acc: 0.6259 Epoch 8/10 382/466 [=======================>......] - ETA: 32s - loss: 0.1495 - acc: 0.6268\n`",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1263679,
      "author_name": "afridi10",
      "author_url": "",
      "post_date": "04/05/2021 15:37:54",
      "content": "<p>The actual image shape is bigger as well as the dataset is also bigger. <a href=\"https://www.kaggle.com/shailkardani\" target=\"_blank\">@shailkardani</a> you can use this resize dataset. <a href=\"https://www.kaggle.com/ankursingh12/resized-plant2021\" target=\"_blank\">https://www.kaggle.com/ankursingh12/resized-plant2021</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 1271395,
          "author_name": "shailkardani",
          "author_url": "",
          "post_date": "04/12/2021 15:11:27",
          "content": "<p>This should help. Thanks ,I'll check it out.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1269667,
      "author_name": "atamazian",
      "author_url": "",
      "post_date": "04/10/2021 18:51:46",
      "content": "<p>There're some things you can do to increase training speed. For example, you can reduce image size, or use less complex model.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1272753,
      "author_name": "jl18pg052",
      "author_url": "",
      "post_date": "04/13/2021 18:18:48",
      "content": "<p>What are you using to train the model?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1274336,
      "author_name": "swaralipibose",
      "author_url": "",
      "post_date": "04/15/2021 07:54:44",
      "content": "<p>May be because you are using ImageDataGenerator , Each image is quite big in the dataset so during training the model predicts with GPU and loads and augments the image with CPU . Thus GPU cant get completely activated as it is always taking sometime to load the images with CPU giving the GPU some break and also because we are loading images and training our model at the same time thus it takes some extra time to load the image .you can either augment the images yourself as numpy array, or use this notebook <a href=\"https://www.kaggle.com/datasciencegeek/resize-images\" target=\"_blank\">https://www.kaggle.com/datasciencegeek/resize-images</a>( Use the output files as data) and just change the path. Hope it helps . Here are my logs after using the notebook<br>\n<code>Epoch 1/10 466/466 [==============================] - 229s 492ms/step - loss: 0.1528 - acc: 0.6191 - val_loss: 0.1517 - val_acc: 0.6186 Epoch 2/10 466/466 [==============================] - 229s 492ms/step - loss: 0.1514 - acc: 0.6212 - val_loss: 0.1501 - val_acc: 0.6296 Epoch 3/10 466/466 [==============================] - 228s 490ms/step - loss: 0.1524 - acc: 0.6235 - val_loss: 0.1517 - val_acc: 0.6173 Epoch 4/10 466/466 [==============================] - 228s 490ms/step - loss: 0.1511 - acc: 0.6238 - val_loss: 0.1488 - val_acc: 0.6299 Epoch 5/10 466/466 [==============================] - 229s 491ms/step - loss: 0.1513 - acc: 0.6233 - val_loss: 0.1508 - val_acc: 0.6294 Epoch 6/10 466/466 [==============================] - 229s 492ms/step - loss: 0.1501 - acc: 0.6270 - val_loss: 0.1483 - val_acc: 0.6275 Epoch 7/10 466/466 [==============================] - 230s 494ms/step - loss: 0.1490 - acc: 0.6302 - val_loss: 0.1509 - val_acc: 0.6259 Epoch 8/10 382/466 [=======================&gt;......] - ETA: 32s - loss: 0.1495 - acc: 0.6268\n</code></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1262620": "After fiting the model it takes way too much time to run the epochs. Probably because of the large dataset. Is there any way to fasten up the process?",
    "1263679": "The actual image shape is bigger as well as the dataset is also bigger. @shailkardani you can use this resize dataset. https://www.kaggle.com/ankursingh12/resized-plant2021",
    "1269667": "There're some things you can do to increase training speed. For example, you can reduce image size, or use less complex model.",
    "1271395": "This should help. Thanks ,I'll check it out.",
    "1272753": "What are you using to train the model?",
    "1274336": "May be because you are using ImageDataGenerator , Each image is quite big in the dataset so during training the model predicts with GPU and loads and augments the image with CPU . Thus GPU cant get completely activated as it is always taking sometime to load the images with CPU giving the GPU some break and also because we are loading images and training our model at the same time thus it takes some extra time to load the image .you can either augment the images yourself as numpy array, or use this notebook https://www.kaggle.com/datasciencegeek/resize-images( Use the output files as data) and just change the path. Hope it helps . Here are my logs after using the notebook\n`Epoch 1/10 466/466 [==============================] - 229s 492ms/step - loss: 0.1528 - acc: 0.6191 - val_loss: 0.1517 - val_acc: 0.6186 Epoch 2/10 466/466 [==============================] - 229s 492ms/step - loss: 0.1514 - acc: 0.6212 - val_loss: 0.1501 - val_acc: 0.6296 Epoch 3/10 466/466 [==============================] - 228s 490ms/step - loss: 0.1524 - acc: 0.6235 - val_loss: 0.1517 - val_acc: 0.6173 Epoch 4/10 466/466 [==============================] - 228s 490ms/step - loss: 0.1511 - acc: 0.6238 - val_loss: 0.1488 - val_acc: 0.6299 Epoch 5/10 466/466 [==============================] - 229s 491ms/step - loss: 0.1513 - acc: 0.6233 - val_loss: 0.1508 - val_acc: 0.6294 Epoch 6/10 466/466 [==============================] - 229s 492ms/step - loss: 0.1501 - acc: 0.6270 - val_loss: 0.1483 - val_acc: 0.6275 Epoch 7/10 466/466 [==============================] - 230s 494ms/step - loss: 0.1490 - acc: 0.6302 - val_loss: 0.1509 - val_acc: 0.6259 Epoch 8/10 382/466 [=======================>......] - ETA: 32s - loss: 0.1495 - acc: 0.6268\n`"
  },
  "source": "meta"
}