{
  "id": 81573,
  "title": "Kernel continously crashing due to RAM over utilization",
  "url": "/competitions/vsb-power-line-fault-detection/discussion/81573",
  "author_name": "",
  "post_date": "2019-02-22T14:40:44.905933400Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I am trying to execute the Kernel and it is continuously crashing due to RAM utilization. I had imported the gc and called the gc.collect().</p>\n\n<p>I had wrote the below code to break and process the train.parquet in eight cycle, but what I am observing is that even after execution of the first cycle the garbage is not getting collected and system is not releasing the RAM, so in the middle of second run the Kernel is crashing.</p>\n\n<p>chunk_size=len(metadata_train)\nfor ini, end in [(0, int(chunk_size/8)), (int(chunk_size/8), int(chunk_size/4)), (int(chunk_size/4), int(chunk_size/8*3)), (int(chunk_size/8*3), (int(chunk_size/2))), (int(chunk_size/2), (int(chunk_size/4*3))), (int(chunk_size/4*3), chunk_size)]:</p>\n\n<h1>for ini, end in [(0, 2)]:</h1>\n\n<pre><code>X_temp = prep_data(ini, end)\nX.append(X_temp)\ngc.collect()\n</code></pre>\n\n<p>The value of ini, end is as below.\n0\n1089\n1089\n2178\n2178\n3267\n3267\n4356\n4356\n6534\n6534\n8712</p>\n\n<p>As, mentioned earlier the code is working correctly for range 0-1089, but when executing 1089-2178 it is falling as the RAM utilized in the 0-1089 was not getting released.</p>\n\n<p>Please help, how to do the garbage collection and release the RAM after each execution cycle.</p>",
  "messages": [
    {
      "id": "476687",
      "postDate": "02/22/2019 14:40:44",
      "content": "<p>I am trying to execute the Kernel and it is continuously crashing due to RAM utilization. I had imported the gc and called the gc.collect().</p>\n\n<p>I had wrote the below code to break and process the train.parquet in eight cycle, but what I am observing is that even after execution of the first cycle the garbage is not getting collected and system is not releasing the RAM, so in the middle of second run the Kernel is crashing.</p>\n\n<p>chunk_size=len(metadata_train)\nfor ini, end in [(0, int(chunk_size/8)), (int(chunk_size/8), int(chunk_size/4)), (int(chunk_size/4), int(chunk_size/8*3)), (int(chunk_size/8*3), (int(chunk_size/2))), (int(chunk_size/2), (int(chunk_size/4*3))), (int(chunk_size/4*3), chunk_size)]:</p>\n\n<h1>for ini, end in [(0, 2)]:</h1>\n\n<pre><code>X_temp = prep_data(ini, end)\nX.append(X_temp)\ngc.collect()\n</code></pre>\n\n<p>The value of ini, end is as below.\n0\n1089\n1089\n2178\n2178\n3267\n3267\n4356\n4356\n6534\n6534\n8712</p>\n\n<p>As, mentioned earlier the code is working correctly for range 0-1089, but when executing 1089-2178 it is falling as the RAM utilized in the 0-1089 was not getting released.</p>\n\n<p>Please help, how to do the garbage collection and release the RAM after each execution cycle.</p>",
      "rawMarkdown": "I am trying to execute the Kernel and it is continuously crashing due to RAM utilization. I had imported the gc and called the gc.collect().\n\nI had wrote the below code to break and process the train.parquet in eight cycle, but what I am observing is that even after execution of the first cycle the garbage is not getting collected and system is not releasing the RAM, so in the middle of second run the Kernel is crashing.\n\nchunk_size=len(metadata_train)\nfor ini, end in [(0, int(chunk_size/8)), (int(chunk_size/8), int(chunk_size/4)), (int(chunk_size/4), int(chunk_size/8*3)), (int(chunk_size/8*3), (int(chunk_size/2))), (int(chunk_size/2), (int(chunk_size/4*3))), (int(chunk_size/4*3), chunk_size)]:\n#for ini, end in [(0, 2)]:\n    X_temp = prep_data(ini, end)\n    X.append(X_temp)\n    gc.collect()\n\nThe value of ini, end is as below.\n0\n1089\n1089\n2178\n2178\n3267\n3267\n4356\n4356\n6534\n6534\n8712\n\nAs, mentioned earlier the code is working correctly for range 0-1089, but when executing 1089-2178 it is falling as the RAM utilized in the 0-1089 was not getting released.\n\nPlease help, how to do the garbage collection and release the RAM after each execution cycle.",
      "votes": null
    },
    {
      "id": "477480",
      "postDate": "02/24/2019 17:14:12",
      "content": "<p>What helps for me is to explicitly delete the temporary data and then run gc.collect() (don't ask me why it works), e.g. something like this:</p>\n\n<pre><code>X_temp = read_parquet(ini, end)\nX[ini:end] = prep_data(X_temp)\ndel X_temp; gc.collect()\n</code></pre>\n\n<p>I am able to read and process both test and train data this way in one go.</p>",
      "rawMarkdown": "What helps for me is to explicitly delete the temporary data and then run gc.collect() (don't ask me why it works), e.g. something like this:\n\n    X_temp = read_parquet(ini, end)\n    X[ini:end] = prep_data(X_temp)\n    del X_temp; gc.collect()\n\nI am able to read and process both test and train data this way in one go.",
      "votes": null
    },
    {
      "id": "477827",
      "postDate": "02/25/2019 10:25:13",
      "content": "<p>Thanks Joop, I will definitely code like wise.</p>\n\n<p>Also, I am looking into converting functions into generator, will keep posted if any improvement is found.</p>",
      "rawMarkdown": "Thanks Joop, I will definitely code like wise.\n\nAlso, I am looking into converting functions into generator, will keep posted if any improvement is found.",
      "votes": null
    },
    {
      "id": "477859",
      "postDate": "02/25/2019 11:38:16",
      "content": "<p>Found the issue, I was using the PyWavelets for the denoising and PyWaletes have an known memory leak issue.</p>\n\n<p><a href=\"https://github.com/PyWavelets/pywt/issues/180\">https://github.com/PyWavelets/pywt/issues/180</a></p>\n\n<p>For, this reason whatever I tried gc.collect, python generator, del(variables) the issue was not getting resolved, also there is no PyWavelets functionality to de-allocate/free memory.</p>",
      "rawMarkdown": "Found the issue, I was using the PyWavelets for the denoising and PyWaletes have an known memory leak issue.\n\nhttps://github.com/PyWavelets/pywt/issues/180\n\nFor, this reason whatever I tried gc.collect, python generator, del(variables) the issue was not getting resolved, also there is no PyWavelets functionality to de-allocate/free memory.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 477480,
      "author_name": "proef2",
      "author_url": "",
      "post_date": "02/24/2019 17:14:12",
      "content": "<p>What helps for me is to explicitly delete the temporary data and then run gc.collect() (don't ask me why it works), e.g. something like this:</p>\n\n<pre><code>X_temp = read_parquet(ini, end)\nX[ini:end] = prep_data(X_temp)\ndel X_temp; gc.collect()\n</code></pre>\n\n<p>I am able to read and process both test and train data this way in one go.</p>",
      "votes": null,
      "replies": [
        {
          "id": 477827,
          "author_name": "adityamanna",
          "author_url": "",
          "post_date": "02/25/2019 10:25:13",
          "content": "<p>Thanks Joop, I will definitely code like wise.</p>\n\n<p>Also, I am looking into converting functions into generator, will keep posted if any improvement is found.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 477859,
      "author_name": "adityamanna",
      "author_url": "",
      "post_date": "02/25/2019 11:38:16",
      "content": "<p>Found the issue, I was using the PyWavelets for the denoising and PyWaletes have an known memory leak issue.</p>\n\n<p><a href=\"https://github.com/PyWavelets/pywt/issues/180\">https://github.com/PyWavelets/pywt/issues/180</a></p>\n\n<p>For, this reason whatever I tried gc.collect, python generator, del(variables) the issue was not getting resolved, also there is no PyWavelets functionality to de-allocate/free memory.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "476687": "I am trying to execute the Kernel and it is continuously crashing due to RAM utilization. I had imported the gc and called the gc.collect().\n\nI had wrote the below code to break and process the train.parquet in eight cycle, but what I am observing is that even after execution of the first cycle the garbage is not getting collected and system is not releasing the RAM, so in the middle of second run the Kernel is crashing.\n\nchunk_size=len(metadata_train)\nfor ini, end in [(0, int(chunk_size/8)), (int(chunk_size/8), int(chunk_size/4)), (int(chunk_size/4), int(chunk_size/8*3)), (int(chunk_size/8*3), (int(chunk_size/2))), (int(chunk_size/2), (int(chunk_size/4*3))), (int(chunk_size/4*3), chunk_size)]:\n#for ini, end in [(0, 2)]:\n    X_temp = prep_data(ini, end)\n    X.append(X_temp)\n    gc.collect()\n\nThe value of ini, end is as below.\n0\n1089\n1089\n2178\n2178\n3267\n3267\n4356\n4356\n6534\n6534\n8712\n\nAs, mentioned earlier the code is working correctly for range 0-1089, but when executing 1089-2178 it is falling as the RAM utilized in the 0-1089 was not getting released.\n\nPlease help, how to do the garbage collection and release the RAM after each execution cycle.",
    "477480": "What helps for me is to explicitly delete the temporary data and then run gc.collect() (don't ask me why it works), e.g. something like this:\n\n    X_temp = read_parquet(ini, end)\n    X[ini:end] = prep_data(X_temp)\n    del X_temp; gc.collect()\n\nI am able to read and process both test and train data this way in one go.",
    "477827": "Thanks Joop, I will definitely code like wise.\n\nAlso, I am looking into converting functions into generator, will keep posted if any improvement is found.",
    "477859": "Found the issue, I was using the PyWavelets for the denoising and PyWaletes have an known memory leak issue.\n\nhttps://github.com/PyWavelets/pywt/issues/180\n\nFor, this reason whatever I tried gc.collect, python generator, del(variables) the issue was not getting resolved, also there is no PyWavelets functionality to de-allocate/free memory."
  },
  "source": "meta"
}