The train dataset is very large. Not able to train the data, as it ends up using all of the RAM and the notebook restarts every time I try to train my model. Any idea about a solution for this problem?
", "messages": [ { "id": "2523045", "postDate": "11/13/2023 07:58:07", "content": "The train dataset is very large. Not able to train the data, as it ends up using all of the RAM and the notebook restarts every time I try to train my model. Any idea about a solution for this problem?
", "rawMarkdown": "The train dataset is very large. Not able to train the data, as it ends up using all of the RAM and the notebook restarts every time I try to train my model. Any idea about a solution for this problem?", "votes": null }, { "id": "2523121", "postDate": "11/13/2023 09:16:38", "content": "You will need to train models on a local environment with a good RAM and GPU @scienceenthusiast
\nProbably Kaggle kernels may not be the correct environment for this process.
As mentioned by Ravi - training models on kaggle is often an issue in most of the competitions.
\nYou did not tell us where your doing your training - if it is on local environment and you run out of RAM there are options available for either Windows or Linux that can use SSD space for RAM - let us know were your running your training!
", "rawMarkdown": "As mentioned by [Ravi](https://www.kaggle.com/ravi20076) - training models on kaggle is often an issue in most of the competitions. \n\nYou did not tell us where your doing your training - if it is on local environment and you run out of RAM there are options available for either Windows or Linux that can use SSD space for RAM - let us know were your running your training!", "votes": null }, { "id": "2523485", "postDate": "11/13/2023 14:21:54", "content": "I am running on Kaggle kernel. I will try in the local environment. Thank you.
", "rawMarkdown": "I am running on Kaggle kernel. I will try in the local environment. Thank you.", "votes": null }, { "id": "2523534", "postDate": "11/13/2023 14:53:31", "content": "There are likely some shared notebooks that show how you can do things on kaggle for training. Methods used depend on type of model, ie, tensorflow vs lgbm, etc. Look for a shared notebook that trains using your model type. Can look in other competitions for those tips also - as mentioned its a common issue. I always train local as my skill set to use the needed methods on kaggle is limited. Good luck.
", "rawMarkdown": "There are likely some shared notebooks that show how you can do things on kaggle for training. Methods used depend on type of model, ie, tensorflow vs lgbm, etc. Look for a shared notebook that trains using your model type. Can look in other competitions for those tips also - as mentioned its a common issue. I always train local as my skill set to use the needed methods on kaggle is limited. Good luck.", "votes": null } ], "comments": [ { "id": 2523121, "author_name": "ravi20076", "author_url": "", "post_date": "11/13/2023 09:16:38", "content": "You will need to train models on a local environment with a good RAM and GPU @scienceenthusiast
\nProbably Kaggle kernels may not be the correct environment for this process.
As mentioned by Ravi - training models on kaggle is often an issue in most of the competitions.
\nYou did not tell us where your doing your training - if it is on local environment and you run out of RAM there are options available for either Windows or Linux that can use SSD space for RAM - let us know were your running your training!
", "votes": null, "replies": [ { "id": 2523485, "author_name": "scienceenthusiast", "author_url": "", "post_date": "11/13/2023 14:21:54", "content": "I am running on Kaggle kernel. I will try in the local environment. Thank you.
", "votes": null, "replies": [ { "id": 2523534, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/13/2023 14:53:31", "content": "There are likely some shared notebooks that show how you can do things on kaggle for training. Methods used depend on type of model, ie, tensorflow vs lgbm, etc. Look for a shared notebook that trains using your model type. Can look in other competitions for those tips also - as mentioned its a common issue. I always train local as my skill set to use the needed methods on kaggle is limited. Good luck.
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2523045": "The train dataset is very large. Not able to train the data, as it ends up using all of the RAM and the notebook restarts every time I try to train my model. Any idea about a solution for this problem?", "2523121": "You will need to train models on a local environment with a good RAM and GPU @scienceenthusiast \nProbably Kaggle kernels may not be the correct environment for this process.", "2523460": "As mentioned by [Ravi](https://www.kaggle.com/ravi20076) - training models on kaggle is often an issue in most of the competitions. \n\nYou did not tell us where your doing your training - if it is on local environment and you run out of RAM there are options available for either Windows or Linux that can use SSD space for RAM - let us know were your running your training!", "2523485": "I am running on Kaggle kernel. I will try in the local environment. Thank you.", "2523534": "There are likely some shared notebooks that show how you can do things on kaggle for training. Methods used depend on type of model, ie, tensorflow vs lgbm, etc. Look for a shared notebook that trains using your model type. Can look in other competitions for those tips also - as mentioned its a common issue. I always train local as my skill set to use the needed methods on kaggle is limited. Good luck." }, "source": "meta" }