\nHere is the my last notebook logs:
\n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11169812%2F8da7687a754867890f6e82cef212aa60%2FScreenshot%202024-03-25%20125635.png?generation=1711357270650885&alt=media\")
Hello,
\n@jetakow I created and run my notebook successfully but when I try to submit it failed (gives me error run out of memory). I can't solve the problem I tried a few things like use small percent of train data, use some variables etc.. but I get the same error.
\nHere is the my last notebook logs:
\n
When I try to submit it:
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11169812%2Fc63571bc906d2047c1906a0eb6d78c64%2FScreenshot%202024-03-25%20125745.png?generation=1711357315066178&alt=media\")
All the time I get the same error:
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11169812%2F33f225c21821a5f45b84db754820fa87%2FScreenshot%202024-03-25%20125829.png?generation=1711357372381572&alt=media\")
Hello,
\n@jetakow I created and run my notebook successfully but when I try to submit it failed (gives me error run out of memory). I can't solve the problem I tried a few things like use small percent of train data, use some variables etc.. but I get the same error.
\nHere is the my last notebook logs:
\n
When I try to submit it:
\n
All the time I get the same error:
\n
Could you please create a minimal viable example of such a notebook and share the code?
", "rawMarkdown": "Could you please create a minimal viable example of such a notebook and share the code?", "votes": null }, { "id": "2715664", "postDate": "03/25/2024 15:59:05", "content": "Few points need to be considered,
\n1) is your notebook working successfully without submitting? so that it can be confirmed the fail is due to test data, it could be size or may even be small bug in code in handelling the test data.
\n2) closely monitor the memory usage, if ur notebook is sing almost full RAM than due to addition of test data it can fail.
There could be more issues but these few need to be checked as mostly the issue may fall in above two points. if not than we can look further.
", "rawMarkdown": "Few points need to be considered,\n1) is your notebook working successfully without submitting? so that it can be confirmed the fail is due to test data, it could be size or may even be small bug in code in handelling the test data.\n2) closely monitor the memory usage, if ur notebook is sing almost full RAM than due to addition of test data it can fail.\n\nThere could be more issues but these few need to be checked as mostly the issue may fall in above two points. if not than we can look further.", "votes": null }, { "id": "2716595", "postDate": "03/26/2024 05:52:15", "content": "If it is okay, I can share my notebook with you
", "rawMarkdown": "If it is okay, I can share my notebook with you", "votes": null }, { "id": "2716597", "postDate": "03/26/2024 05:53:20", "content": "1) yes, it is working successfully.
\n2) I am trying, but can't find a solution
Ok so in your notebook are you deleting the files once u have extracted features from them? and clearing the memory taken by them, once a file is read and features are extracted, it is good to delete it and clear space by: del 'variable' then gc.collect().
\nAssuming that u are reaching to the model training and the model gets trained, now we can delete entientire training data, including the training features. So we have enough space for model prediction.
I changed my notebook and I don't use train data just 60-70% of test data and a pre-trained model. Pre-trained model memory is about 250 kb. I got the same error
", "rawMarkdown": "I changed my notebook and I don't use train data just 60-70% of test data and a pre-trained model. Pre-trained model memory is about 250 kb. I got the same error", "votes": null }, { "id": "2719171", "postDate": "03/27/2024 14:53:24", "content": "There is one more way you can pin point where the problem is by commenting part of the code and only the porting of code that u might have suspect be working. for submission can create a dummy file with size of test data, with all zeros to get successful submission.
\nSo, you can divide the code section into a reading data/files section, than the processing and model making section, than the prediction section.
\nstart with top reading secting, and comment everything else, than the dummy submission.
\nIf it works, than uncomment the next section, and this progressively checking how far the code is able to execute.
\nThis might take few submissions but will help you find out which section of code is troubling.
Also along with the commenting you can use the 'try except' to handle exceptions where you feel suitable. to debg the code or to ensure it will pass through that section of code.
", "rawMarkdown": "Also along with the commenting you can use the 'try except' to handle exceptions where you feel suitable. to debg the code or to ensure it will pass through that section of code.", "votes": null }, { "id": "2722078", "postDate": "03/29/2024 11:39:47", "content": "We discussed it here https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/486259
", "rawMarkdown": "We discussed it here https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/486259", "votes": null }, { "id": "2760394", "postDate": "04/19/2024 09:34:00", "content": "@jetakow hello,
\nlibrary(tidyverse)
\nlibrary(vroom)
test_base <- vroom(\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/test/test_base.csv\")
\ntest_base$score <- c(0.16556765,0.64040754,0.04545037,0.32004708,0.75208873,0.10967650,0.09592315,0.01648286,0.42362419,0.06741692)
\nsubmission <- test_base %>% select(case_id, score)
\nsubmission
submission %>% write.csv(\"submission.csv\")
\nI use this code for just submission but again it failed.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11169812%2Fe8fcecd4a0306fa1ed6699c43c67dbae%2FScreenshot%202024-04-19%20133331.png?generation=1713519237888076&alt=media\")
Could you please create a minimal viable example of such a notebook and share the code?
", "votes": null, "replies": [ { "id": 2716595, "author_name": "elvinrustam", "author_url": "", "post_date": "03/26/2024 05:52:15", "content": "If it is okay, I can share my notebook with you
", "votes": null, "replies": [] }, { "id": 2760394, "author_name": "elvinrustam", "author_url": "", "post_date": "04/19/2024 09:34:00", "content": "@jetakow hello,
\nlibrary(tidyverse)
\nlibrary(vroom)
test_base <- vroom(\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/test/test_base.csv\")
\ntest_base$score <- c(0.16556765,0.64040754,0.04545037,0.32004708,0.75208873,0.10967650,0.09592315,0.01648286,0.42362419,0.06741692)
\nsubmission <- test_base %>% select(case_id, score)
\nsubmission
submission %>% write.csv(\"submission.csv\")
\nI use this code for just submission but again it failed.
\n
Few points need to be considered,
\n1) is your notebook working successfully without submitting? so that it can be confirmed the fail is due to test data, it could be size or may even be small bug in code in handelling the test data.
\n2) closely monitor the memory usage, if ur notebook is sing almost full RAM than due to addition of test data it can fail.
There could be more issues but these few need to be checked as mostly the issue may fall in above two points. if not than we can look further.
", "votes": null, "replies": [ { "id": 2716597, "author_name": "elvinrustam", "author_url": "", "post_date": "03/26/2024 05:53:20", "content": "1) yes, it is working successfully.
\n2) I am trying, but can't find a solution
Ok so in your notebook are you deleting the files once u have extracted features from them? and clearing the memory taken by them, once a file is read and features are extracted, it is good to delete it and clear space by: del 'variable' then gc.collect().
\nAssuming that u are reaching to the model training and the model gets trained, now we can delete entientire training data, including the training features. So we have enough space for model prediction.
I changed my notebook and I don't use train data just 60-70% of test data and a pre-trained model. Pre-trained model memory is about 250 kb. I got the same error
", "votes": null, "replies": [ { "id": 2719171, "author_name": "shreyas9181", "author_url": "", "post_date": "03/27/2024 14:53:24", "content": "There is one more way you can pin point where the problem is by commenting part of the code and only the porting of code that u might have suspect be working. for submission can create a dummy file with size of test data, with all zeros to get successful submission.
\nSo, you can divide the code section into a reading data/files section, than the processing and model making section, than the prediction section.
\nstart with top reading secting, and comment everything else, than the dummy submission.
\nIf it works, than uncomment the next section, and this progressively checking how far the code is able to execute.
\nThis might take few submissions but will help you find out which section of code is troubling.
Also along with the commenting you can use the 'try except' to handle exceptions where you feel suitable. to debg the code or to ensure it will pass through that section of code.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2722078, "author_name": "eu1234", "author_url": "", "post_date": "03/29/2024 11:39:47", "content": "We discussed it here https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/486259
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2715079": "Hello,\n\n@jetakow I created and run my notebook successfully but when I try to submit it failed (gives me error run out of memory). I can't solve the problem I tried a few things like use small percent of train data, use some variables etc.. but I get the same error. \nHere is the my last notebook logs:\n \n\nWhen I try to submit it:\n\n\nAll the time I get the same error:\n", "2715323": "Could you please create a minimal viable example of such a notebook and share the code?", "2715664": "Few points need to be considered,\n1) is your notebook working successfully without submitting? so that it can be confirmed the fail is due to test data, it could be size or may even be small bug in code in handelling the test data.\n2) closely monitor the memory usage, if ur notebook is sing almost full RAM than due to addition of test data it can fail.\n\nThere could be more issues but these few need to be checked as mostly the issue may fall in above two points. if not than we can look further.", "2716595": "If it is okay, I can share my notebook with you", "2716597": "1) yes, it is working successfully.\n2) I am trying, but can't find a solution", "2717551": "Ok so in your notebook are you deleting the files once u have extracted features from them? and clearing the memory taken by them, once a file is read and features are extracted, it is good to delete it and clear space by: del 'variable' then gc.collect().\nAssuming that u are reaching to the model training and the model gets trained, now we can delete entientire training data, including the training features. So we have enough space for model prediction.", "2718545": "I changed my notebook and I don't use train data just 60-70% of test data and a pre-trained model. Pre-trained model memory is about 250 kb. I got the same error", "2719171": "There is one more way you can pin point where the problem is by commenting part of the code and only the porting of code that u might have suspect be working. for submission can create a dummy file with size of test data, with all zeros to get successful submission.\nSo, you can divide the code section into a reading data/files section, than the processing and model making section, than the prediction section.\nstart with top reading secting, and comment everything else, than the dummy submission.\nIf it works, than uncomment the next section, and this progressively checking how far the code is able to execute.\nThis might take few submissions but will help you find out which section of code is troubling.", "2719522": "Also along with the commenting you can use the 'try except' to handle exceptions where you feel suitable. to debg the code or to ensure it will pass through that section of code.", "2722078": "We discussed it here https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/486259", "2760394": "jetakow hello,\n\nlibrary(tidyverse)\nlibrary(vroom)\n\n\ntest_base <- vroom(\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/test/test_base.csv\")\n\n\ntest_base$score <- c(0.16556765,0.64040754,0.04545037,0.32004708,0.75208873,0.10967650,0.09592315,0.01648286,0.42362419,0.06741692)\n\n\nsubmission <- test_base %>% select(case_id, score)\nsubmission\n\n\nsubmission %>% write.csv(\"submission.csv\")\n\nI use this code for just submission but again it failed.\n\n" }, "source": "meta" }