{
  "id": 400431,
  "title": "Memory utilization reduced by 30%..,now no memory error with competition notebook.",
  "url": "/competitions/predict-student-performance-from-game-play/discussion/400431",
  "author_name": "",
  "post_date": "2023-04-08T11:05:43.872840800Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p><strong>Need to see how fullscreen / music / hq (picture quality) contribute to the overall result of game play , for memory efficiency it can be removed or park for initial model building.</strong></p>\n<p>Also default categories which pandas assigns automatically consumes more memory , so specifying categories which are memory efficient reduces memory usage by 50%</p>\n<h2>**Memory usage experiment On 16GB BOx</h2>\n<p>**with normal pandas loading \"df.read\":</p>\n<p>Memory usage before loading 3.8GB -- 23.3%<br>\nMemory usage after loading  10.9GB -- 66.2% (increase of ~43%)</p>\n<p>**With removed columns and specifying datatypes</p>\n<p>**Memory usage before loading 3.8GB -- 23.3%<br>\nMemory usage after loading  6.0GB -- 36.5% (increase of ~13%)</p>\n<p><a href=\"https://www.kaggle.com/sakvaua\" target=\"_blank\">Acknowledgement </a></p>",
  "messages": [
    {
      "id": "2214315",
      "postDate": "04/08/2023 11:05:43",
      "content": "<p><strong>Need to see how fullscreen / music / hq (picture quality) contribute to the overall result of game play , for memory efficiency it can be removed or park for initial model building.</strong></p>\n<p>Also default categories which pandas assigns automatically consumes more memory , so specifying categories which are memory efficient reduces memory usage by 50%</p>\n<h2>**Memory usage experiment On 16GB BOx</h2>\n<p>**with normal pandas loading \"df.read\":</p>\n<p>Memory usage before loading 3.8GB -- 23.3%<br>\nMemory usage after loading  10.9GB -- 66.2% (increase of ~43%)</p>\n<p>**With removed columns and specifying datatypes</p>\n<p>**Memory usage before loading 3.8GB -- 23.3%<br>\nMemory usage after loading  6.0GB -- 36.5% (increase of ~13%)</p>\n<p><a href=\"https://www.kaggle.com/sakvaua\" target=\"_blank\">Acknowledgement </a></p>",
      "rawMarkdown": "**Need to see how fullscreen / music / hq (picture quality) contribute to the overall result of game play , for memory efficiency it can be removed or park for initial model building.**\n\nAlso default categories which pandas assigns automatically consumes more memory , so specifying categories which are memory efficient reduces memory usage by 50%\n\n## **Memory usage experiment On 16GB BOx\n\n**with normal pandas loading \"df.read\":\n\nMemory usage before loading 3.8GB -- 23.3%\nMemory usage after loading  10.9GB -- 66.2% (increase of ~43%)\n\n\n**With removed columns and specifying datatypes\n\n**Memory usage before loading 3.8GB -- 23.3%\nMemory usage after loading  6.0GB -- 36.5% (increase of ~13%)\n\n[Acknowledgement ]( https://www.kaggle.com/sakvaua)",
      "votes": null
    },
    {
      "id": "2217554",
      "postDate": "04/11/2023 02:26:10",
      "content": "<p>That's a great improvement in memory usage. I agree that some columns like fullscreen, music and hq may not be very relevant for predicting student performance and can be removed or parked for initial model building. I also like your idea of specifying datatypes for pandas columns to reduce memory usage.</p>\n<p>I wonder if you have tried any other techniques to optimize memory usage, such as using feather format, reducing precision, or using sparse matrices. I think these techniques can also help to save memory and speed up the data processing.</p>",
      "rawMarkdown": "That's a great improvement in memory usage. I agree that some columns like fullscreen, music and hq may not be very relevant for predicting student performance and can be removed or parked for initial model building. I also like your idea of specifying datatypes for pandas columns to reduce memory usage.\n\nI wonder if you have tried any other techniques to optimize memory usage, such as using feather format, reducing precision, or using sparse matrices. I think these techniques can also help to save memory and speed up the data processing.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2217554,
      "author_name": "yus002",
      "author_url": "",
      "post_date": "04/11/2023 02:26:10",
      "content": "<p>That's a great improvement in memory usage. I agree that some columns like fullscreen, music and hq may not be very relevant for predicting student performance and can be removed or parked for initial model building. I also like your idea of specifying datatypes for pandas columns to reduce memory usage.</p>\n<p>I wonder if you have tried any other techniques to optimize memory usage, such as using feather format, reducing precision, or using sparse matrices. I think these techniques can also help to save memory and speed up the data processing.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2214315": "**Need to see how fullscreen / music / hq (picture quality) contribute to the overall result of game play , for memory efficiency it can be removed or park for initial model building.**\n\nAlso default categories which pandas assigns automatically consumes more memory , so specifying categories which are memory efficient reduces memory usage by 50%\n\n## **Memory usage experiment On 16GB BOx\n\n**with normal pandas loading \"df.read\":\n\nMemory usage before loading 3.8GB -- 23.3%\nMemory usage after loading  10.9GB -- 66.2% (increase of ~43%)\n\n\n**With removed columns and specifying datatypes\n\n**Memory usage before loading 3.8GB -- 23.3%\nMemory usage after loading  6.0GB -- 36.5% (increase of ~13%)\n\n[Acknowledgement ]( https://www.kaggle.com/sakvaua)",
    "2217554": "That's a great improvement in memory usage. I agree that some columns like fullscreen, music and hq may not be very relevant for predicting student performance and can be removed or parked for initial model building. I also like your idea of specifying datatypes for pandas columns to reduce memory usage.\n\nI wonder if you have tried any other techniques to optimize memory usage, such as using feather format, reducing precision, or using sparse matrices. I think these techniques can also help to save memory and speed up the data processing."
  },
  "source": "meta"
}