When I try to submit my notebook and submission file, it keeps failing with the error \"notebook out of memory\" and thus cannot generate my score, even though my notebook works well. I have tried to narrow down my dataset and features, but it didn't work. Anybody meets the same problem? How do you fix it? Thanks!
", "messages": [ { "id": 2722080, "postDate": "2024-03-29T11:40:57.933Z", "content": "Check these solutions and the comments https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/486259
", "rawMarkdown": "Check these solutions and the comments https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/486259", "votes": 1, "replies": [ { "id": 2726436, "postDate": "2024-04-01T07:43:52.773Z", "content": "Thank you, I'll try to follow the steps in this link.
", "rawMarkdown": "Thank you, I'll try to follow the steps in this link. " } ] }, { "id": 2711261, "postDate": "2024-03-22T20:09:39.423Z", "content": "If yous training process is going smoothly, and you have a memory issue during submission, It could mean that it failing due to the test data size,
\n2 ways you can address this, 1) by deleting every train data after the model is trained and only then load the test data and do the predictions. 2) make separate notebooks for training the model and for prediction.
Thank you! I've tried the first method you suggested but the issue persisted, so I'm gonna trying the second method.
", "rawMarkdown": "Thank you! I've tried the first method you suggested but the issue persisted, so I'm gonna trying the second method. " } ] }, { "id": 2710246, "postDate": "2024-03-22T05:49:10.070Z", "content": "I can give you some advice:
\nFirstly, you can add a file to your last successful run of code, one by one, to determine where the problem lies
\nSecondly, when you are doing data processing, if a file is not needed, it should be cleaned up in a timely manner (del file, gc.collect())
\nThirdly, you can try using polars instead of pandas, which can handle files with larger memory
\nGood luck to you
", "rawMarkdown": "I can give you some advice:\n\nFirstly, you can add a file to your last successful run of code, one by one, to determine where the problem lies\n\nSecondly, when you are doing data processing, if a file is not needed, it should be cleaned up in a timely manner (del file, gc.collect())\n\nThirdly, you can try using polars instead of pandas, which can handle files with larger memory\n\n\n\nGood luck to you", "votes": 1, "replies": [ { "id": 2710260, "postDate": "2024-03-22T06:01:53.103Z", "content": "Thank you! I'll try ASAP.
", "rawMarkdown": "Thank you! I'll try ASAP." }, { "id": 2710558, "postDate": "2024-03-22T10:59:43.943Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ] }, { "id": 2710137, "postDate": "2024-03-22T03:51:49.277Z", "content": "When I try to submit my notebook and submission file, it keeps failing with the error \"notebook out of memory\" and thus cannot generate my score, even though my notebook works well. I have tried to narrow down my dataset and features, but it didn't work. Anybody meets the same problem? How do you fix it? Thanks!
", "rawMarkdown": "When I try to submit my notebook and submission file, it keeps failing with the error \"notebook out of memory\" and thus cannot generate my score, even though my notebook works well. I have tried to narrow down my dataset and features, but it didn't work. Anybody meets the same problem? How do you fix it? Thanks! ", "votes": 1 } ], "comments": [ { "id": 2722080, "author_name": "Danu A.", "author_url": "", "post_date": "2024-03-29T11:40:57.933000", "content": "Check these solutions and the comments https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/486259
", "votes": 1, "replies": [ { "id": 2726436, "author_name": "Tianjun Ma", "author_url": "", "post_date": "2024-04-01T07:43:52.773000", "content": "Thank you, I'll try to follow the steps in this link.
", "votes": 0, "replies": [] } ] }, { "id": 2711261, "author_name": "Shreyas Bhatt", "author_url": "", "post_date": "2024-03-22T20:09:39.423000", "content": "If yous training process is going smoothly, and you have a memory issue during submission, It could mean that it failing due to the test data size,
\n2 ways you can address this, 1) by deleting every train data after the model is trained and only then load the test data and do the predictions. 2) make separate notebooks for training the model and for prediction.
Thank you! I've tried the first method you suggested but the issue persisted, so I'm gonna trying the second method.
", "votes": 0, "replies": [] } ] }, { "id": 2710246, "author_name": "yunsuxiaozi", "author_url": "", "post_date": "2024-03-22T05:49:10.070000", "content": "I can give you some advice:
\nFirstly, you can add a file to your last successful run of code, one by one, to determine where the problem lies
\nSecondly, when you are doing data processing, if a file is not needed, it should be cleaned up in a timely manner (del file, gc.collect())
\nThirdly, you can try using polars instead of pandas, which can handle files with larger memory
\nGood luck to you
", "votes": 1, "replies": [ { "id": 2710260, "author_name": "Tianjun Ma", "author_url": "", "post_date": "2024-03-22T06:01:53.103000", "content": "Thank you! I'll try ASAP.
", "votes": 0, "replies": [] }, { "id": 2710558, "author_name": "", "author_url": "", "post_date": "2024-03-22T10:59:43.943000", "content": "", "votes": -1, "replies": [] } ] } ], "raw_markdown_by_id": { "2722080": "Check these solutions and the comments https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/486259", "2711261": "If yous training process is going smoothly, and you have a memory issue during submission, It could mean that it failing due to the test data size,\n2 ways you can address this, 1) by deleting every train data after the model is trained and only then load the test data and do the predictions. 2) make separate notebooks for training the model and for prediction.", "2710246": "I can give you some advice:\n\nFirstly, you can add a file to your last successful run of code, one by one, to determine where the problem lies\n\nSecondly, when you are doing data processing, if a file is not needed, it should be cleaned up in a timely manner (del file, gc.collect())\n\nThirdly, you can try using polars instead of pandas, which can handle files with larger memory\n\n\n\nGood luck to you", "2710137": "When I try to submit my notebook and submission file, it keeps failing with the error \"notebook out of memory\" and thus cannot generate my score, even though my notebook works well. I have tried to narrow down my dataset and features, but it didn't work. Anybody meets the same problem? How do you fix it? Thanks! " } }