{
  "id": 223979,
  "title": "Kaggle notebook running out of memory",
  "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/223979",
  "author_name": "",
  "post_date": "2021-03-06T08:49:21.261680300Z",
  "votes": 3,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I'm trying to train a simple Resnet200D model using TIMM on kaggle notebook but it keeps on crashing due to memory overload. Are you guys able to train on kaggle notebooks or are all you of training locally?</p>",
  "messages": [
    {
      "id": "1228286",
      "postDate": "03/06/2021 08:49:21",
      "content": "<p>I'm trying to train a simple Resnet200D model using TIMM on kaggle notebook but it keeps on crashing due to memory overload. Are you guys able to train on kaggle notebooks or are all you of training locally?</p>",
      "rawMarkdown": "I'm trying to train a simple Resnet200D model using TIMM on kaggle notebook but it keeps on crashing due to memory overload. Are you guys able to train on kaggle notebooks or are all you of training locally?",
      "votes": null
    },
    {
      "id": "1228647",
      "postDate": "03/06/2021 16:07:48",
      "content": "<p>It is possible to train a resnet200d model in notebook, but you need to make sure you are only loading in a small number of images at a time and you have gpu on. If you try to do the calculation in CPU or try to load in too many images then you will run out of memory. There are a few public notebooks that show this process. </p>\n<p><a href=\"https://www.kaggle.com/underwearfitting/single-fold-training-of-resnet200d-lb0-965\" target=\"_blank\">https://www.kaggle.com/underwearfitting/single-fold-training-of-resnet200d-lb0-965</a></p>",
      "rawMarkdown": "It is possible to train a resnet200d model in notebook, but you need to make sure you are only loading in a small number of images at a time and you have gpu on. If you try to do the calculation in CPU or try to load in too many images then you will run out of memory. There are a few public notebooks that show this process. \n\nhttps://www.kaggle.com/underwearfitting/single-fold-training-of-resnet200d-lb0-965",
      "votes": null
    },
    {
      "id": "1229017",
      "postDate": "03/07/2021 01:28:05",
      "content": "<p>Even with a batch size of 1 and gpu support it crashes. </p>\n<p><a href=\"https://www.kaggle.com/khalidanwaar/resnet-200d-timm\" target=\"_blank\">https://www.kaggle.com/khalidanwaar/resnet-200d-timm</a></p>",
      "rawMarkdown": "Even with a batch size of 1 and gpu support it crashes. \n\nhttps://www.kaggle.com/khalidanwaar/resnet-200d-timm",
      "votes": null
    },
    {
      "id": "1230461",
      "postDate": "03/08/2021 07:10:53",
      "content": "<p>Gone through same problem,<br>\nactually you will need to reduce the image size drastically to fit the whole data in gpu memory<br>\nthat's why even using batch_size = 1 won't work</p>\n<p>Why?<br>\nbecause this dataset is large and even each image is of very high resolution (i.e. ~2500x2600)<br>\nAlso the resnet200D model is too big so while saving computations after every nn layer and batch on gpu, it fills up the memory and the model will throw this error</p>\n<p>so the larger your image tensor is the higher time consumption will be ( since the Matrix Multiplication has time complexity of O(n^3))</p>\n<p>try image size as 50x50 (which results in very bad accuracy but still just for knowledge)<br>\nalso try using single channel rather than 3 channels to save more memory</p>",
      "rawMarkdown": "Gone through same problem,\nactually you will need to reduce the image size drastically to fit the whole data in gpu memory\nthat's why even using batch_size = 1 won't work\n\nWhy?\nbecause this dataset is large and even each image is of very high resolution (i.e. ~2500x2600)\nAlso the resnet200D model is too big so while saving computations after every nn layer and batch on gpu, it fills up the memory and the model will throw this error\n\nso the larger your image tensor is the higher time consumption will be ( since the Matrix Multiplication has time complexity of O(n^3))\n\ntry image size as 50x50 (which results in very bad accuracy but still just for knowledge)\nalso try using single channel rather than 3 channels to save more memory",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1228647,
      "author_name": "ryches",
      "author_url": "",
      "post_date": "03/06/2021 16:07:48",
      "content": "<p>It is possible to train a resnet200d model in notebook, but you need to make sure you are only loading in a small number of images at a time and you have gpu on. If you try to do the calculation in CPU or try to load in too many images then you will run out of memory. There are a few public notebooks that show this process. </p>\n<p><a href=\"https://www.kaggle.com/underwearfitting/single-fold-training-of-resnet200d-lb0-965\" target=\"_blank\">https://www.kaggle.com/underwearfitting/single-fold-training-of-resnet200d-lb0-965</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 1229017,
          "author_name": "khalidanwaar",
          "author_url": "",
          "post_date": "03/07/2021 01:28:05",
          "content": "<p>Even with a batch size of 1 and gpu support it crashes. </p>\n<p><a href=\"https://www.kaggle.com/khalidanwaar/resnet-200d-timm\" target=\"_blank\">https://www.kaggle.com/khalidanwaar/resnet-200d-timm</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1230461,
      "author_name": "sahilambekar777",
      "author_url": "",
      "post_date": "03/08/2021 07:10:53",
      "content": "<p>Gone through same problem,<br>\nactually you will need to reduce the image size drastically to fit the whole data in gpu memory<br>\nthat's why even using batch_size = 1 won't work</p>\n<p>Why?<br>\nbecause this dataset is large and even each image is of very high resolution (i.e. ~2500x2600)<br>\nAlso the resnet200D model is too big so while saving computations after every nn layer and batch on gpu, it fills up the memory and the model will throw this error</p>\n<p>so the larger your image tensor is the higher time consumption will be ( since the Matrix Multiplication has time complexity of O(n^3))</p>\n<p>try image size as 50x50 (which results in very bad accuracy but still just for knowledge)<br>\nalso try using single channel rather than 3 channels to save more memory</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1228286": "I'm trying to train a simple Resnet200D model using TIMM on kaggle notebook but it keeps on crashing due to memory overload. Are you guys able to train on kaggle notebooks or are all you of training locally?",
    "1228647": "It is possible to train a resnet200d model in notebook, but you need to make sure you are only loading in a small number of images at a time and you have gpu on. If you try to do the calculation in CPU or try to load in too many images then you will run out of memory. There are a few public notebooks that show this process. \n\nhttps://www.kaggle.com/underwearfitting/single-fold-training-of-resnet200d-lb0-965",
    "1229017": "Even with a batch size of 1 and gpu support it crashes. \n\nhttps://www.kaggle.com/khalidanwaar/resnet-200d-timm",
    "1230461": "Gone through same problem,\nactually you will need to reduce the image size drastically to fit the whole data in gpu memory\nthat's why even using batch_size = 1 won't work\n\nWhy?\nbecause this dataset is large and even each image is of very high resolution (i.e. ~2500x2600)\nAlso the resnet200D model is too big so while saving computations after every nn layer and batch on gpu, it fills up the memory and the model will throw this error\n\nso the larger your image tensor is the higher time consumption will be ( since the Matrix Multiplication has time complexity of O(n^3))\n\ntry image size as 50x50 (which results in very bad accuracy but still just for knowledge)\nalso try using single channel rather than 3 channels to save more memory"
  },
  "source": "meta"
}