{
  "id": 403390,
  "title": "LSTM out of memory",
  "url": "/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/403390",
  "author_name": "Quim Quadrada",
  "post_date": "2023-04-22T19:39:14.934000",
  "votes": 1,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi,</p>\n<p>I'm struggling with an out of memory problem even in local. I am usin LSTM, I saved the models so the notebook just pick them from the data, I delete as much variables as I can, I have even tried to save in output chuncks of the dataset, but no way. Memory increases and increases and after a couple of hours, it stops. LSTM processes one batch per prediction position, but the batches are deleted. It seems that the results array gets bigger without stop, even if i crop it and save in output.</p>\n<p>I think it is a matter of LSTM, as with such a huge datasets, the memory increases without stopping.</p>\n<p>Anyone has found the same?</p>",
  "messages": [
    {
      "id": 2230837,
      "postDate": "2023-04-22T19:39:14.933Z",
      "content": "<p>Hi,</p>\n<p>I'm struggling with an out of memory problem even in local. I am usin LSTM, I saved the models so the notebook just pick them from the data, I delete as much variables as I can, I have even tried to save in output chuncks of the dataset, but no way. Memory increases and increases and after a couple of hours, it stops. LSTM processes one batch per prediction position, but the batches are deleted. It seems that the results array gets bigger without stop, even if i crop it and save in output.</p>\n<p>I think it is a matter of LSTM, as with such a huge datasets, the memory increases without stopping.</p>\n<p>Anyone has found the same?</p>",
      "rawMarkdown": "Hi,\n\nI'm struggling with an out of memory problem even in local. I am usin LSTM, I saved the models so the notebook just pick them from the data, I delete as much variables as I can, I have even tried to save in output chuncks of the dataset, but no way. Memory increases and increases and after a couple of hours, it stops. LSTM processes one batch per prediction position, but the batches are deleted. It seems that the results array gets bigger without stop, even if i crop it and save in output.\n\nI think it is a matter of LSTM, as with such a huge datasets, the memory increases without stopping.\n\nAnyone has found the same?",
      "votes": 1
    },
    {
      "id": 2231003,
      "postDate": "2023-04-23T02:17:30.733Z",
      "content": "<p>you may post your code👀, on the other hand, you also can use pytorch_lightning to rewrite your code, the pkg can standardize most of the training process, which may solve your problem</p>",
      "rawMarkdown": "you may post your code👀, on the other hand, you also can use pytorch_lightning to rewrite your code, the pkg can standardize most of the training process, which may solve your problem",
      "votes": 2,
      "replies": [
        {
          "id": 2231657,
          "postDate": "2023-04-23T14:10:12.607Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 2231662,
          "postDate": "2023-04-23T14:19:12.423Z",
          "content": "<p>I train apart locally. So when I submt, I just need to load the trained models and predict the values.</p>",
          "rawMarkdown": "I train apart locally. So when I submt, I just need to load the trained models and predict the values.",
          "replies": [
            {
              "id": 2241363,
              "postDate": "2023-05-01T12:34:24.443Z",
              "content": "<p>if you are using GPU for submitting, RAM is very limited with GPU turned on, and test time series are very long so allocating the memory for the sequences causes memory error</p>",
              "rawMarkdown": "if you are using GPU for submitting, RAM is very limited with GPU turned on, and test time series are very long so allocating the memory for the sequences causes memory error"
            }
          ]
        }
      ]
    },
    {
      "id": 2287882,
      "postDate": "2023-06-05T02:19:42.257Z",
      "content": "<blockquote>\n  <p>Memory increases and increases and after a couple of hours, it stops.</p>\n</blockquote>\n<p>I meet the same problem. Have you fixed it, sir? <a href=\"https://www.kaggle.com/quimquadrada\" target=\"_blank\">@quimquadrada</a> </p>",
      "rawMarkdown": ">Memory increases and increases and after a couple of hours, it stops.\n\nI meet the same problem. Have you fixed it, sir? @quimquadrada ",
      "votes": -1
    }
  ],
  "comments": [
    {
      "id": 2231003,
      "author_name": "RogerOcean",
      "author_url": "",
      "post_date": "2023-04-23T02:17:30.733000",
      "content": "<p>you may post your code👀, on the other hand, you also can use pytorch_lightning to rewrite your code, the pkg can standardize most of the training process, which may solve your problem</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2231657,
          "author_name": "",
          "author_url": "",
          "post_date": "2023-04-23T14:10:12.607000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2231662,
          "author_name": "Quim Quadrada",
          "author_url": "",
          "post_date": "2023-04-23T14:19:12.423000",
          "content": "<p>I train apart locally. So when I submt, I just need to load the trained models and predict the values.</p>",
          "votes": 0,
          "replies": [
            {
              "id": 2241363,
              "author_name": "Alberto Annoni",
              "author_url": "",
              "post_date": "2023-05-01T12:34:24.443000",
              "content": "<p>if you are using GPU for submitting, RAM is very limited with GPU turned on, and test time series are very long so allocating the memory for the sequences causes memory error</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2287882,
      "author_name": "Joseph Zhou",
      "author_url": "",
      "post_date": "2023-06-05T02:19:42.257000",
      "content": "<blockquote>\n  <p>Memory increases and increases and after a couple of hours, it stops.</p>\n</blockquote>\n<p>I meet the same problem. Have you fixed it, sir? <a href=\"https://www.kaggle.com/quimquadrada\" target=\"_blank\">@quimquadrada</a> </p>",
      "votes": -1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2230837": "Hi,\n\nI'm struggling with an out of memory problem even in local. I am usin LSTM, I saved the models so the notebook just pick them from the data, I delete as much variables as I can, I have even tried to save in output chuncks of the dataset, but no way. Memory increases and increases and after a couple of hours, it stops. LSTM processes one batch per prediction position, but the batches are deleted. It seems that the results array gets bigger without stop, even if i crop it and save in output.\n\nI think it is a matter of LSTM, as with such a huge datasets, the memory increases without stopping.\n\nAnyone has found the same?",
    "2231003": "you may post your code👀, on the other hand, you also can use pytorch_lightning to rewrite your code, the pkg can standardize most of the training process, which may solve your problem",
    "2287882": ">Memory increases and increases and after a couple of hours, it stops.\n\nI meet the same problem. Have you fixed it, sir? @quimquadrada "
  }
}