{
  "id": 553643,
  "title": "memory exceeded, help !!",
  "url": "/competitions/czii-cryo-et-object-identification/discussion/553643",
  "author_name": "",
  "post_date": "2024-12-27T12:57:00.345685900Z",
  "votes": null,
  "comment_count": 7,
  "views": 0,
  "content": "<p>I have experience in training models with TensorFlow. So, I converted the dataset into NumPy arrays with a size (7, 184, 630, 630, 1). Then, I feed it into a 3D-UNet model inspired by <a href=\"https://arxiv.org/pdf/1606.06650\" target=\"_blank\">this paper</a>.</p>\n<p>Now, I encounter a problem where the RAM usage exceeds 30 GB. I do not know why it takes up so much memory for a NumPy array of around 2 GB.</p>\n<p>This is the <a href=\"https://www.kaggle.com/code/nveshaan/notebookebec8b91b1\" target=\"_blank\">notebook</a> where I ran the code.</p>\n<p>Could anyone please point out what went wrong here?</p>\n<p>I'd be happy to provide any info you need. Thanks !!!</p>",
  "messages": [
    {
      "id": "3081952",
      "postDate": "12/27/2024 12:57:00",
      "content": "<p>I have experience in training models with TensorFlow. So, I converted the dataset into NumPy arrays with a size (7, 184, 630, 630, 1). Then, I feed it into a 3D-UNet model inspired by <a href=\"https://arxiv.org/pdf/1606.06650\" target=\"_blank\">this paper</a>.</p>\n<p>Now, I encounter a problem where the RAM usage exceeds 30 GB. I do not know why it takes up so much memory for a NumPy array of around 2 GB.</p>\n<p>This is the <a href=\"https://www.kaggle.com/code/nveshaan/notebookebec8b91b1\" target=\"_blank\">notebook</a> where I ran the code.</p>\n<p>Could anyone please point out what went wrong here?</p>\n<p>I'd be happy to provide any info you need. Thanks !!!</p>",
      "rawMarkdown": "I have experience in training models with TensorFlow. So, I converted the dataset into NumPy arrays with a size (7, 184, 630, 630, 1). Then, I feed it into a 3D-UNet model inspired by [this paper](https://arxiv.org/pdf/1606.06650).\n\nNow, I encounter a problem where the RAM usage exceeds 30 GB. I do not know why it takes up so much memory for a NumPy array of around 2 GB.\n\nThis is the [notebook](https://www.kaggle.com/code/nveshaan/notebookebec8b91b1) where I ran the code.\n\nCould anyone please point out what went wrong here?\n\nI'd be happy to provide any info you need. Thanks !!!",
      "votes": null
    },
    {
      "id": "3081953",
      "postDate": "12/27/2024 12:58:59",
      "content": "<p>EDIT: I tried lowering the resolution and setting the batch size to 1. It still didn't work.</p>",
      "rawMarkdown": "EDIT: I tried lowering the resolution and setting the batch size to 1. It still didn't work.",
      "votes": null
    },
    {
      "id": "3082012",
      "postDate": "12/27/2024 14:12:08",
      "content": "<p>Don't feed the whole image. Do a crop. Like 64x64x64</p>",
      "rawMarkdown": "Don't feed the whole image. Do a crop. Like 64x64x64",
      "votes": null
    },
    {
      "id": "3082015",
      "postDate": "12/27/2024 14:16:19",
      "content": "<p>The input can be 2GB. But all the intermediate tensors produced at each layer toguether with their gradients wich are necessary for back propagation can cause OOM. Even without gradients for inference only. Train on crops.</p>",
      "rawMarkdown": "The input can be 2GB. But all the intermediate tensors produced at each layer toguether with their gradients wich are necessary for back propagation can cause OOM. Even without gradients for inference only. Train on crops.",
      "votes": null
    },
    {
      "id": "3082098",
      "postDate": "12/27/2024 16:51:41",
      "content": "<p>thanks, will give it a try</p>",
      "rawMarkdown": "thanks, will give it a try",
      "votes": null
    },
    {
      "id": "3082099",
      "postDate": "12/27/2024 16:52:04",
      "content": "<p>oh i get it, tysm</p>",
      "rawMarkdown": "oh i get it, tysm",
      "votes": null
    },
    {
      "id": "3083159",
      "postDate": "12/29/2024 05:28:33",
      "content": "<p>Hey it fits in P100, T4 GPU. If you use batch_size = 1 and scanning_window_shape = (80, 160, 160).</p>\n<p>This was the best I could fit.</p>\n<p>I use ResidualUNet3D model implemented in the github repo : <a href=\"https://github.com/wolny/pytorch-3dunet/tree/master\" target=\"_blank\">https://github.com/wolny/pytorch-3dunet/tree/master</a></p>\n<p>Though the issue depends on many factors.</p>",
      "rawMarkdown": "Hey it fits in P100, T4 GPU. If you use batch_size = 1 and scanning_window_shape = (80, 160, 160).\n\nThis was the best I could fit.\n\nI use ResidualUNet3D model implemented in the github repo : https://github.com/wolny/pytorch-3dunet/tree/master\n\nThough the issue depends on many factors.",
      "votes": null
    },
    {
      "id": "3084067",
      "postDate": "12/30/2024 11:19:41",
      "content": "<p>oh yes, i found out that it actually depends on the window size, thanks</p>",
      "rawMarkdown": "oh yes, i found out that it actually depends on the window size, thanks",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3081953,
      "author_name": "nveshaan",
      "author_url": "",
      "post_date": "12/27/2024 12:58:59",
      "content": "<p>EDIT: I tried lowering the resolution and setting the batch size to 1. It still didn't work.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3083159,
          "author_name": "vigneshwar472",
          "author_url": "",
          "post_date": "12/29/2024 05:28:33",
          "content": "<p>Hey it fits in P100, T4 GPU. If you use batch_size = 1 and scanning_window_shape = (80, 160, 160).</p>\n<p>This was the best I could fit.</p>\n<p>I use ResidualUNet3D model implemented in the github repo : <a href=\"https://github.com/wolny/pytorch-3dunet/tree/master\" target=\"_blank\">https://github.com/wolny/pytorch-3dunet/tree/master</a></p>\n<p>Though the issue depends on many factors.</p>",
          "votes": null,
          "replies": [
            {
              "id": 3084067,
              "author_name": "nveshaan",
              "author_url": "",
              "post_date": "12/30/2024 11:19:41",
              "content": "<p>oh yes, i found out that it actually depends on the window size, thanks</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3082012,
      "author_name": "sakvaua",
      "author_url": "",
      "post_date": "12/27/2024 14:12:08",
      "content": "<p>Don't feed the whole image. Do a crop. Like 64x64x64</p>",
      "votes": null,
      "replies": [
        {
          "id": 3082098,
          "author_name": "nveshaan",
          "author_url": "",
          "post_date": "12/27/2024 16:51:41",
          "content": "<p>thanks, will give it a try</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3082015,
      "author_name": "sacuscreed",
      "author_url": "",
      "post_date": "12/27/2024 14:16:19",
      "content": "<p>The input can be 2GB. But all the intermediate tensors produced at each layer toguether with their gradients wich are necessary for back propagation can cause OOM. Even without gradients for inference only. Train on crops.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3082099,
          "author_name": "nveshaan",
          "author_url": "",
          "post_date": "12/27/2024 16:52:04",
          "content": "<p>oh i get it, tysm</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3081952": "I have experience in training models with TensorFlow. So, I converted the dataset into NumPy arrays with a size (7, 184, 630, 630, 1). Then, I feed it into a 3D-UNet model inspired by [this paper](https://arxiv.org/pdf/1606.06650).\n\nNow, I encounter a problem where the RAM usage exceeds 30 GB. I do not know why it takes up so much memory for a NumPy array of around 2 GB.\n\nThis is the [notebook](https://www.kaggle.com/code/nveshaan/notebookebec8b91b1) where I ran the code.\n\nCould anyone please point out what went wrong here?\n\nI'd be happy to provide any info you need. Thanks !!!",
    "3081953": "EDIT: I tried lowering the resolution and setting the batch size to 1. It still didn't work.",
    "3082012": "Don't feed the whole image. Do a crop. Like 64x64x64",
    "3082015": "The input can be 2GB. But all the intermediate tensors produced at each layer toguether with their gradients wich are necessary for back propagation can cause OOM. Even without gradients for inference only. Train on crops.",
    "3082098": "thanks, will give it a try",
    "3082099": "oh i get it, tysm",
    "3083159": "Hey it fits in P100, T4 GPU. If you use batch_size = 1 and scanning_window_shape = (80, 160, 160).\n\nThis was the best I could fit.\n\nI use ResidualUNet3D model implemented in the github repo : https://github.com/wolny/pytorch-3dunet/tree/master\n\nThough the issue depends on many factors.",
    "3084067": "oh yes, i found out that it actually depends on the window size, thanks"
  },
  "source": "meta"
}