{
  "id": 189377,
  "title": "Dataset: Train PKL (5s read 6GB RAM)",
  "url": "/competitions/riiid-test-answer-prediction/discussion/189377",
  "author_name": "",
  "post_date": "2020-10-07T12:44:05.696330Z",
  "votes": 13,
  "comment_count": 2,
  "views": 0,
  "content": "<p>As reading the original train.csv takes forever and also doesn't fit in RAM without downcasting I created a PKL with the downcasted data  types.</p>\n<p>The public dataset can be found <a href=\"https://www.kaggle.com/markwijkhuizen/riiid-answer-correctness-prediction-files\" target=\"_blank\">here</a></p>\n<p>The PKL takes ~5s to read and consumes 6GB RAM.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4433335%2F1d05a03733b0038807a27d1677743f95%2Fdtypes.png?generation=1602077583774662&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": "1040911",
      "postDate": "10/07/2020 12:44:05",
      "content": "<p>As reading the original train.csv takes forever and also doesn't fit in RAM without downcasting I created a PKL with the downcasted data  types.</p>\n<p>The public dataset can be found <a href=\"https://www.kaggle.com/markwijkhuizen/riiid-answer-correctness-prediction-files\" target=\"_blank\">here</a></p>\n<p>The PKL takes ~5s to read and consumes 6GB RAM.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4433335%2F1d05a03733b0038807a27d1677743f95%2Fdtypes.png?generation=1602077583774662&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "As reading the original train.csv takes forever and also doesn't fit in RAM without downcasting I created a PKL with the downcasted data  types.\n\nThe public dataset can be found [here](https://www.kaggle.com/markwijkhuizen/riiid-answer-correctness-prediction-files)\n\nThe PKL takes ~5s to read and consumes 6GB RAM.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4433335%2F1d05a03733b0038807a27d1677743f95%2Fdtypes.png?generation=1602077583774662&alt=media)",
      "votes": null
    },
    {
      "id": "1041185",
      "postDate": "10/07/2020 15:43:19",
      "content": "<p>thanks!</p>\n<p>for those who have a recursive loading error in python, add this statement before loading the pickle file</p>\n<p>sys.setrecursionlimit(10**6)</p>\n<p>reference: <a href=\"https://www.geeksforgeeks.org/python-handling-recursion-limit/\" target=\"_blank\">https://www.geeksforgeeks.org/python-handling-recursion-limit/</a></p>",
      "rawMarkdown": "thanks!\n\nfor those who have a recursive loading error in python, add this statement before loading the pickle file\n\nsys.setrecursionlimit(10**6)\n\nreference: https://www.geeksforgeeks.org/python-handling-recursion-limit/",
      "votes": null
    },
    {
      "id": "1041507",
      "postDate": "10/07/2020 19:29:17",
      "content": "<p>Please see <a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/188908\" target=\"_blank\">solution</a> to load the entire original train file without throwing out of memory error in a kernel.</p>",
      "rawMarkdown": "Please see [solution](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/188908) to load the entire original train file without throwing out of memory error in a kernel.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1041185,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "10/07/2020 15:43:19",
      "content": "<p>thanks!</p>\n<p>for those who have a recursive loading error in python, add this statement before loading the pickle file</p>\n<p>sys.setrecursionlimit(10**6)</p>\n<p>reference: <a href=\"https://www.geeksforgeeks.org/python-handling-recursion-limit/\" target=\"_blank\">https://www.geeksforgeeks.org/python-handling-recursion-limit/</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1041507,
      "author_name": "sirishks",
      "author_url": "",
      "post_date": "10/07/2020 19:29:17",
      "content": "<p>Please see <a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/188908\" target=\"_blank\">solution</a> to load the entire original train file without throwing out of memory error in a kernel.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1040911": "As reading the original train.csv takes forever and also doesn't fit in RAM without downcasting I created a PKL with the downcasted data  types.\n\nThe public dataset can be found [here](https://www.kaggle.com/markwijkhuizen/riiid-answer-correctness-prediction-files)\n\nThe PKL takes ~5s to read and consumes 6GB RAM.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4433335%2F1d05a03733b0038807a27d1677743f95%2Fdtypes.png?generation=1602077583774662&alt=media)",
    "1041185": "thanks!\n\nfor those who have a recursive loading error in python, add this statement before loading the pickle file\n\nsys.setrecursionlimit(10**6)\n\nreference: https://www.geeksforgeeks.org/python-handling-recursion-limit/",
    "1041507": "Please see [solution](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/188908) to load the entire original train file without throwing out of memory error in a kernel."
  },
  "source": "meta"
}