{
  "id": 396551,
  "title": "Your notebook tried to allocate more memory than is available. It has restarted.",
  "url": "/competitions/predict-student-performance-from-game-play/discussion/396551",
  "author_name": "",
  "post_date": "2023-03-22T03:49:30.651417200Z",
  "votes": null,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Hi, I got stuck about memory in kaggle notebook when I try to load dataset even if I use def reduce_memory_usage(df): I still load the data. How could I solve this problem thank you</p>",
  "messages": [
    {
      "id": "2191544",
      "postDate": "03/22/2023 03:49:30",
      "content": "<p>Hi, I got stuck about memory in kaggle notebook when I try to load dataset even if I use def reduce_memory_usage(df): I still load the data. How could I solve this problem thank you</p>",
      "rawMarkdown": "Hi, I got stuck about memory in kaggle notebook when I try to load dataset even if I use def reduce_memory_usage(df): I still load the data. How could I solve this problem thank you",
      "votes": null
    },
    {
      "id": "2191546",
      "postDate": "03/22/2023 03:52:38",
      "content": "<p>We can load train data and train model in Kaggle notebook with 32GB+ RAM and then save models. Then use second notebook (which will have 8GB RAM) to load model and infer test.</p>",
      "rawMarkdown": "We can load train data and train model in Kaggle notebook with 32GB+ RAM and then save models. Then use second notebook (which will have 8GB RAM) to load model and infer test.",
      "votes": null
    },
    {
      "id": "2191552",
      "postDate": "03/22/2023 03:59:52",
      "content": "<p>So, I need to load the train data for train model first right the data size is more than 4 GB. it still out of memory</p>",
      "rawMarkdown": "So, I need to load the train data for train model first right the data size is more than 4 GB. it still out of memory",
      "votes": null
    },
    {
      "id": "2191644",
      "postDate": "03/22/2023 05:24:46",
      "content": "<p>To make it clear, you could create another dataset and attach this new dataset to your notebook instead of the competition dataset. This will enlarge your RAM, providing enough memory for your training part.</p>",
      "rawMarkdown": "To make it clear, you could create another dataset and attach this new dataset to your notebook instead of the competition dataset. This will enlarge your RAM, providing enough memory for your training part.",
      "votes": null
    },
    {
      "id": "2191670",
      "postDate": "03/22/2023 06:03:58",
      "content": "<p>Thx!!!!!👍👍👍👍</p>",
      "rawMarkdown": "Thx!!!!!👍👍👍👍",
      "votes": null
    },
    {
      "id": "2192872",
      "postDate": "03/23/2023 00:43:23",
      "content": "<p>Hi Dear <a href=\"https://www.kaggle.com/thitiwat\" target=\"_blank\">@thitiwat</a> <br>\nIf you are still experiencing memory issues even after using the reduce_memory_usage() function, there are a few other approaches you can try:</p>\n<p><strong>Reduce the size of the dataset</strong>: If you are working with a large dataset, try to reduce its size by selecting a subset of the data or removing unnecessary columns/features.</p>\n<p><strong>Use a generator to load the data</strong>: Instead of loading the entire dataset into memory, you can use a generator to load the data in chunks. This can be done using the pandas read_csv() function with the chunksize parameter.</p>\n<p><strong>Use a cloud-based solution</strong>: If your computer doesn't have enough memory to handle the dataset, consider using a cloud-based solution like Google Colab or AWS. These platforms provide access to powerful hardware and can handle large datasets.</p>\n<p><strong>Optimize your code</strong>: Check your code for any inefficiencies or unnecessary computations that could be contributing to the memory issue. Make sure you are using the most memory-efficient data structures and algorithms.</p>\n<p><strong>Restart the kernel</strong>: Sometimes, simply restarting the kernel can help to clear up any memory issues. Try restarting the kernel and then re-running your code to see if that resolves the problem.<br>\nI hope these solution will be able to work with.</p>",
      "rawMarkdown": "Hi Dear @thitiwat \nIf you are still experiencing memory issues even after using the reduce_memory_usage() function, there are a few other approaches you can try:\n\n**Reduce the size of the dataset**: If you are working with a large dataset, try to reduce its size by selecting a subset of the data or removing unnecessary columns/features.\n\n**Use a generator to load the data**: Instead of loading the entire dataset into memory, you can use a generator to load the data in chunks. This can be done using the pandas read_csv() function with the chunksize parameter.\n\n**Use a cloud-based solution**: If your computer doesn't have enough memory to handle the dataset, consider using a cloud-based solution like Google Colab or AWS. These platforms provide access to powerful hardware and can handle large datasets.\n\n**Optimize your code**: Check your code for any inefficiencies or unnecessary computations that could be contributing to the memory issue. Make sure you are using the most memory-efficient data structures and algorithms.\n\n**Restart the kernel**: Sometimes, simply restarting the kernel can help to clear up any memory issues. Try restarting the kernel and then re-running your code to see if that resolves the problem.\nI hope these solution will be able to work with.",
      "votes": null
    },
    {
      "id": "2193147",
      "postDate": "03/23/2023 05:36:37",
      "content": "<p>Hi there, did this work?</p>",
      "rawMarkdown": "Hi there, did this work?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2191546,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "03/22/2023 03:52:38",
      "content": "<p>We can load train data and train model in Kaggle notebook with 32GB+ RAM and then save models. Then use second notebook (which will have 8GB RAM) to load model and infer test.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2191552,
          "author_name": "thitiwat",
          "author_url": "",
          "post_date": "03/22/2023 03:59:52",
          "content": "<p>So, I need to load the train data for train model first right the data size is more than 4 GB. it still out of memory</p>",
          "votes": null,
          "replies": [
            {
              "id": 2191644,
              "author_name": "yunqicao",
              "author_url": "",
              "post_date": "03/22/2023 05:24:46",
              "content": "<p>To make it clear, you could create another dataset and attach this new dataset to your notebook instead of the competition dataset. This will enlarge your RAM, providing enough memory for your training part.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2191670,
                  "author_name": "thitiwat",
                  "author_url": "",
                  "post_date": "03/22/2023 06:03:58",
                  "content": "<p>Thx!!!!!👍👍👍👍</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 2193147,
                      "author_name": "shwetangimehta",
                      "author_url": "",
                      "post_date": "03/23/2023 05:36:37",
                      "content": "<p>Hi there, did this work?</p>",
                      "votes": null,
                      "replies": []
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 2192872,
      "author_name": "tariqbashir",
      "author_url": "",
      "post_date": "03/23/2023 00:43:23",
      "content": "<p>Hi Dear <a href=\"https://www.kaggle.com/thitiwat\" target=\"_blank\">@thitiwat</a> <br>\nIf you are still experiencing memory issues even after using the reduce_memory_usage() function, there are a few other approaches you can try:</p>\n<p><strong>Reduce the size of the dataset</strong>: If you are working with a large dataset, try to reduce its size by selecting a subset of the data or removing unnecessary columns/features.</p>\n<p><strong>Use a generator to load the data</strong>: Instead of loading the entire dataset into memory, you can use a generator to load the data in chunks. This can be done using the pandas read_csv() function with the chunksize parameter.</p>\n<p><strong>Use a cloud-based solution</strong>: If your computer doesn't have enough memory to handle the dataset, consider using a cloud-based solution like Google Colab or AWS. These platforms provide access to powerful hardware and can handle large datasets.</p>\n<p><strong>Optimize your code</strong>: Check your code for any inefficiencies or unnecessary computations that could be contributing to the memory issue. Make sure you are using the most memory-efficient data structures and algorithms.</p>\n<p><strong>Restart the kernel</strong>: Sometimes, simply restarting the kernel can help to clear up any memory issues. Try restarting the kernel and then re-running your code to see if that resolves the problem.<br>\nI hope these solution will be able to work with.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2191544": "Hi, I got stuck about memory in kaggle notebook when I try to load dataset even if I use def reduce_memory_usage(df): I still load the data. How could I solve this problem thank you",
    "2191546": "We can load train data and train model in Kaggle notebook with 32GB+ RAM and then save models. Then use second notebook (which will have 8GB RAM) to load model and infer test.",
    "2191552": "So, I need to load the train data for train model first right the data size is more than 4 GB. it still out of memory",
    "2191644": "To make it clear, you could create another dataset and attach this new dataset to your notebook instead of the competition dataset. This will enlarge your RAM, providing enough memory for your training part.",
    "2191670": "Thx!!!!!👍👍👍👍",
    "2192872": "Hi Dear @thitiwat \nIf you are still experiencing memory issues even after using the reduce_memory_usage() function, there are a few other approaches you can try:\n\n**Reduce the size of the dataset**: If you are working with a large dataset, try to reduce its size by selecting a subset of the data or removing unnecessary columns/features.\n\n**Use a generator to load the data**: Instead of loading the entire dataset into memory, you can use a generator to load the data in chunks. This can be done using the pandas read_csv() function with the chunksize parameter.\n\n**Use a cloud-based solution**: If your computer doesn't have enough memory to handle the dataset, consider using a cloud-based solution like Google Colab or AWS. These platforms provide access to powerful hardware and can handle large datasets.\n\n**Optimize your code**: Check your code for any inefficiencies or unnecessary computations that could be contributing to the memory issue. Make sure you are using the most memory-efficient data structures and algorithms.\n\n**Restart the kernel**: Sometimes, simply restarting the kernel can help to clear up any memory issues. Try restarting the kernel and then re-running your code to see if that resolves the problem.\nI hope these solution will be able to work with.",
    "2193147": "Hi there, did this work?"
  },
  "source": "meta"
}