{
  "id": 207636,
  "title": "Use a database to avoid memory burden",
  "url": "/competitions/riiid-test-answer-prediction/discussion/207636",
  "author_name": "Jude TCHAYE",
  "post_date": "2020-12-30T16:35:33.647000",
  "votes": 2,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hi guys! </p>\n<p>Even at the last moments of the competition, I realize that many people are still facing submission errors due to memory overheads.  If this is your case,  I recommend you to try the SQLite database instead of pandas merge during your data aggregation. In addition to requiring less RAM, inserting and updating data in the SQLite files is amazingly fast. You can check the following notebook for more details:<br>\n<a href=\"https://www.kaggle.com/tchaye59/riiid-work-with-the-full-state-using-sqlalchemy\" target=\"_blank\">https://www.kaggle.com/tchaye59/riiid-work-with-the-full-state-using-sqlalchemy</a></p>",
  "messages": [
    {
      "id": 1132740,
      "postDate": "2020-12-30T16:35:33.647Z",
      "content": "<p>Hi guys! </p>\n<p>Even at the last moments of the competition, I realize that many people are still facing submission errors due to memory overheads.  If this is your case,  I recommend you to try the SQLite database instead of pandas merge during your data aggregation. In addition to requiring less RAM, inserting and updating data in the SQLite files is amazingly fast. You can check the following notebook for more details:<br>\n<a href=\"https://www.kaggle.com/tchaye59/riiid-work-with-the-full-state-using-sqlalchemy\" target=\"_blank\">https://www.kaggle.com/tchaye59/riiid-work-with-the-full-state-using-sqlalchemy</a></p>",
      "rawMarkdown": "Hi guys! \n\nEven at the last moments of the competition, I realize that many people are still facing submission errors due to memory overheads.  If this is your case,  I recommend you to try the SQLite database instead of pandas merge during your data aggregation. In addition to requiring less RAM, inserting and updating data in the SQLite files is amazingly fast. You can check the following notebook for more details:\nhttps://www.kaggle.com/tchaye59/riiid-work-with-the-full-state-using-sqlalchemy",
      "votes": 2
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1132740": "Hi guys! \n\nEven at the last moments of the competition, I realize that many people are still facing submission errors due to memory overheads.  If this is your case,  I recommend you to try the SQLite database instead of pandas merge during your data aggregation. In addition to requiring less RAM, inserting and updating data in the SQLite files is amazingly fast. You can check the following notebook for more details:\nhttps://www.kaggle.com/tchaye59/riiid-work-with-the-full-state-using-sqlalchemy"
  }
}