{
  "id": 191386,
  "title": "Kernel RAM size",
  "url": "/competitions/riiid-test-answer-prediction/discussion/191386",
  "author_name": "",
  "post_date": "2020-10-16T07:51:20.552182600Z",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi I cannot understand why the RAM is increasing while I'm removing some data :</p>\n<p><code>train = pd.read_hdf(\"../input/riiid-train-data-multiple-formats/riiid_train.h5\",</code><br>\n<code>usecols=[1, 2, 3, 4, 7, 8, 9],</code><br>\n<code>dtype={'timestamp': 'int64',</code><br>\n<code>user_id': 'int32',</code><br>\n<code>content_id': 'int16',</code><br>\n<code>content_type_id': 'int8',</code><br>\n<code>answered_correctly':'int8',</code><br>\n<code>prior_question_elapsed_time': 'float32',</code><br>\n<code>prior_question_had_explanation': 'boolean'})</code></p>\n<p>=&gt; RAM utilization: 6.9GB</p>\n<p><code>train.drop(['task_container_id', 'user_answer'], axis=1, inplace=True)</code><br>\n<code>train = train[train.content_type_id == False]</code><br>\n<code>train = train.sort_values(['timestamp'], ascending=True)</code><br>\n<code>train.drop(['timestamp', 'content_type_id'], axis=1, inplace=True)</code></p>\n<p>=&gt; RAM utilization: 9.4GB</p>\n<p>I've just dropped some colmuns so why RAM utilization increased ?</p>",
  "messages": [
    {
      "id": "1051136",
      "postDate": "10/16/2020 07:51:20",
      "content": "<p>Hi I cannot understand why the RAM is increasing while I'm removing some data :</p>\n<p><code>train = pd.read_hdf(\"../input/riiid-train-data-multiple-formats/riiid_train.h5\",</code><br>\n<code>usecols=[1, 2, 3, 4, 7, 8, 9],</code><br>\n<code>dtype={'timestamp': 'int64',</code><br>\n<code>user_id': 'int32',</code><br>\n<code>content_id': 'int16',</code><br>\n<code>content_type_id': 'int8',</code><br>\n<code>answered_correctly':'int8',</code><br>\n<code>prior_question_elapsed_time': 'float32',</code><br>\n<code>prior_question_had_explanation': 'boolean'})</code></p>\n<p>=&gt; RAM utilization: 6.9GB</p>\n<p><code>train.drop(['task_container_id', 'user_answer'], axis=1, inplace=True)</code><br>\n<code>train = train[train.content_type_id == False]</code><br>\n<code>train = train.sort_values(['timestamp'], ascending=True)</code><br>\n<code>train.drop(['timestamp', 'content_type_id'], axis=1, inplace=True)</code></p>\n<p>=&gt; RAM utilization: 9.4GB</p>\n<p>I've just dropped some colmuns so why RAM utilization increased ?</p>",
      "rawMarkdown": "Hi I cannot understand why the RAM is increasing while I'm removing some data :\n\n`train = pd.read_hdf(\"../input/riiid-train-data-multiple-formats/riiid_train.h5\",`\n`                   usecols=[1, 2, 3, 4, 7, 8, 9],`\n`                   dtype={'timestamp': 'int64',`\n`                         user_id': 'int32',`\n`                          content_id': 'int16',`\n`                          content_type_id': 'int8',`\n`                          answered_correctly':'int8',`\n`                          prior_question_elapsed_time': 'float32',`\n`                          prior_question_had_explanation': 'boolean'})`\n\n=> RAM utilization: 6.9GB\n\n`train.drop(['task_container_id', 'user_answer'], axis=1, inplace=True)`\n`train = train[train.content_type_id == False]`\n`train = train.sort_values(['timestamp'], ascending=True)`\n`train.drop(['timestamp', 'content_type_id'], axis=1, inplace=True)`\n\n=> RAM utilization: 9.4GB\n\nI've just dropped some colmuns so why RAM utilization increased ?",
      "votes": null
    },
    {
      "id": "1051162",
      "postDate": "10/16/2020 08:18:47",
      "content": "<p>I ran the exact code on Kaggle Notebook and RAM dropped to 5.6GB (after going up to 9.5GB). Maybe just wait few seconds or run an empty cell to ensure RAM utilization gets refreshed?</p>",
      "rawMarkdown": "I ran the exact code on Kaggle Notebook and RAM dropped to 5.6GB (after going up to 9.5GB). Maybe just wait few seconds or run an empty cell to ensure RAM utilization gets refreshed?",
      "votes": null
    },
    {
      "id": "1051171",
      "postDate": "10/16/2020 08:28:00",
      "content": "<p>Yeah I tried both ideas but RAM utilization is still stuck at 9GB</p>\n<p>edit: I removed train.head() which appear twice in my notebook and it solved the issue. I cannot figure out why it impacts RAM utilization so much</p>",
      "rawMarkdown": "Yeah I tried both ideas but RAM utilization is still stuck at 9GB\n\nedit: I removed train.head() which appear twice in my notebook and it solved the issue. I cannot figure out why it impacts RAM utilization so much",
      "votes": null
    },
    {
      "id": "1101017",
      "postDate": "12/03/2020 15:02:03",
      "content": "<p>I have this same problem. </p>\n<p>I don't use head() in my notebook and still can't get RAM to drop. </p>\n<p>Did anyone figure out how to solve this?</p>",
      "rawMarkdown": "I have this same problem. \n\nI don't use head() in my notebook and still can't get RAM to drop. \n\nDid anyone figure out how to solve this?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1051162,
      "author_name": "rohanrao",
      "author_url": "",
      "post_date": "10/16/2020 08:18:47",
      "content": "<p>I ran the exact code on Kaggle Notebook and RAM dropped to 5.6GB (after going up to 9.5GB). Maybe just wait few seconds or run an empty cell to ensure RAM utilization gets refreshed?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1051171,
          "author_name": "alexj21",
          "author_url": "",
          "post_date": "10/16/2020 08:28:00",
          "content": "<p>Yeah I tried both ideas but RAM utilization is still stuck at 9GB</p>\n<p>edit: I removed train.head() which appear twice in my notebook and it solved the issue. I cannot figure out why it impacts RAM utilization so much</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1101017,
          "author_name": "thomasliptay",
          "author_url": "",
          "post_date": "12/03/2020 15:02:03",
          "content": "<p>I have this same problem. </p>\n<p>I don't use head() in my notebook and still can't get RAM to drop. </p>\n<p>Did anyone figure out how to solve this?</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1051136": "Hi I cannot understand why the RAM is increasing while I'm removing some data :\n\n`train = pd.read_hdf(\"../input/riiid-train-data-multiple-formats/riiid_train.h5\",`\n`                   usecols=[1, 2, 3, 4, 7, 8, 9],`\n`                   dtype={'timestamp': 'int64',`\n`                         user_id': 'int32',`\n`                          content_id': 'int16',`\n`                          content_type_id': 'int8',`\n`                          answered_correctly':'int8',`\n`                          prior_question_elapsed_time': 'float32',`\n`                          prior_question_had_explanation': 'boolean'})`\n\n=> RAM utilization: 6.9GB\n\n`train.drop(['task_container_id', 'user_answer'], axis=1, inplace=True)`\n`train = train[train.content_type_id == False]`\n`train = train.sort_values(['timestamp'], ascending=True)`\n`train.drop(['timestamp', 'content_type_id'], axis=1, inplace=True)`\n\n=> RAM utilization: 9.4GB\n\nI've just dropped some colmuns so why RAM utilization increased ?",
    "1051162": "I ran the exact code on Kaggle Notebook and RAM dropped to 5.6GB (after going up to 9.5GB). Maybe just wait few seconds or run an empty cell to ensure RAM utilization gets refreshed?",
    "1051171": "Yeah I tried both ideas but RAM utilization is still stuck at 9GB\n\nedit: I removed train.head() which appear twice in my notebook and it solved the issue. I cannot figure out why it impacts RAM utilization so much",
    "1101017": "I have this same problem. \n\nI don't use head() in my notebook and still can't get RAM to drop. \n\nDid anyone figure out how to solve this?"
  },
  "source": "meta"
}