{ "id": 200239, "title": "OOM in Kaggle Notebook", "url": "/competitions/cassava-leaf-disease-classification/discussion/200239", "author_name": "", "post_date": "2020-11-29T15:38:30.616854300Z", "votes": 2, "comment_count": 3, "views": 0, "content": "

Hi,

\n

Is there any suggestion while building model in Kaggle notebook, I am getting out of memory even with 30% samples of total data given in this competition? I am using keras data generator and making image in grayscale and dividing layers with 255.

", "messages": [ { "id": "1095429", "postDate": "11/29/2020 15:38:30", "content": "

Hi,

\n

Is there any suggestion while building model in Kaggle notebook, I am getting out of memory even with 30% samples of total data given in this competition? I am using keras data generator and making image in grayscale and dividing layers with 255.

", "rawMarkdown": "Hi,\n\nIs there any suggestion while building model in Kaggle notebook, I am getting out of memory even with 30% samples of total data given in this competition? I am using keras data generator and making image in grayscale and dividing layers with 255.", "votes": null }, { "id": "1095687", "postDate": "11/29/2020 21:07:35", "content": "

Following things are often the reason for OOM:

\n", "rawMarkdown": "Following things are often the reason for OOM:\n- very big image sizes\n- big model (with a lot of parameters)\n- large batch size", "votes": null }, { "id": "1096286", "postDate": "11/30/2020 11:28:53", "content": "

Don't worry: you don't need to:

\n
    \n
  1. go for grayscale
    2. \n
  2. reduce the training set
    3. \n
\n

when you're training with a large set of images you need to divide the set in mini-batches, and in each training step only a mini-batch is processed.

\n

You have to set appropriately the batch_size parameter. For example something like: 64

\n

consider that the batch size will also depends on the size of the images after pre-rpocessing that you send to the NN. The larger the image size, the lower should be the batch_size

\n

Don't put too low the batch size otherwise the GPU won't have enough work to do and the training will be very slow.

", "rawMarkdown": "Don't worry: you don't need to:\n1. go for grayscale\n2. reduce the training set\n\nwhen you're training with a large set of images you need to divide the set in mini-batches, and in each training step only a mini-batch is processed.\n\nYou have to set appropriately the batch_size parameter. For example something like: 64\n\nconsider that the batch size will also depends on the size of the images after pre-rpocessing that you send to the NN. The larger the image size, the lower should be the batch_size\n\nDon't put too low the batch size otherwise the GPU won't have enough work to do and the training will be very slow.", "votes": null }, { "id": "1096292", "postDate": "11/30/2020 11:36:56", "content": "

Thank you Ali and Luigi for your suggestion. I had tried the model by reducing batch size as suggested by Ali. Now I will also see if I can use color image and play with batch size to run my model with complete data.

", "rawMarkdown": "Thank you Ali and Luigi for your suggestion. I had tried the model by reducing batch size as suggested by Ali. Now I will also see if I can use color image and play with batch size to run my model with complete data.", "votes": null } ], "comments": [ { "id": 1095687, "author_name": "aliabdin1", "author_url": "", "post_date": "11/29/2020 21:07:35", "content": "

Following things are often the reason for OOM:

\n", "votes": null, "replies": [] }, { "id": 1096286, "author_name": "luigisaetta", "author_url": "", "post_date": "11/30/2020 11:28:53", "content": "

Don't worry: you don't need to:

\n
    \n
  1. go for grayscale
    2. \n
  2. reduce the training set
    3. \n
\n

when you're training with a large set of images you need to divide the set in mini-batches, and in each training step only a mini-batch is processed.

\n

You have to set appropriately the batch_size parameter. For example something like: 64

\n

consider that the batch size will also depends on the size of the images after pre-rpocessing that you send to the NN. The larger the image size, the lower should be the batch_size

\n

Don't put too low the batch size otherwise the GPU won't have enough work to do and the training will be very slow.

", "votes": null, "replies": [] }, { "id": 1096292, "author_name": "rams143", "author_url": "", "post_date": "11/30/2020 11:36:56", "content": "

Thank you Ali and Luigi for your suggestion. I had tried the model by reducing batch size as suggested by Ali. Now I will also see if I can use color image and play with batch size to run my model with complete data.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1095429": "Hi,\n\nIs there any suggestion while building model in Kaggle notebook, I am getting out of memory even with 30% samples of total data given in this competition? I am using keras data generator and making image in grayscale and dividing layers with 255.", "1095687": "Following things are often the reason for OOM:\n- very big image sizes\n- big model (with a lot of parameters)\n- large batch size", "1096286": "Don't worry: you don't need to:\n1. go for grayscale\n2. reduce the training set\n\nwhen you're training with a large set of images you need to divide the set in mini-batches, and in each training step only a mini-batch is processed.\n\nYou have to set appropriately the batch_size parameter. For example something like: 64\n\nconsider that the batch size will also depends on the size of the images after pre-rpocessing that you send to the NN. The larger the image size, the lower should be the batch_size\n\nDon't put too low the batch size otherwise the GPU won't have enough work to do and the training will be very slow.", "1096292": "Thank you Ali and Luigi for your suggestion. I had tried the model by reducing batch size as suggested by Ali. Now I will also see if I can use color image and play with batch size to run my model with complete data." }, "source": "meta" }