Hello everyone!
\nMost of my recent submissions are failed with memory errors, I decided to inspect what is wrong and submitted code which previously scored successfully, but now it has failed with out-of-memory error.
\nAlso, previously, I have encountered situation when failed notebooks were scored after resubmitting.
\nDoes anyone have same issues?
", "messages": [ { "id": 2828780, "postDate": "2024-05-22T09:08:24.797Z", "content": "Hello everyone!
\nMost of my recent submissions are failed with memory errors, I decided to inspect what is wrong and submitted code which previously scored successfully, but now it has failed with out-of-memory error.
\nAlso, previously, I have encountered situation when failed notebooks were scored after resubmitting.
\nDoes anyone have same issues?
", "rawMarkdown": "Hello everyone!\n\nMost of my recent submissions are failed with memory errors, I decided to inspect what is wrong and submitted code which previously scored successfully, but now it has failed with out-of-memory error.\n\nAlso, previously, I have encountered situation when failed notebooks were scored after resubmitting.\n\nDoes anyone have same issues?", "votes": 14 }, { "id": 2828825, "postDate": "2024-05-22T09:44:32.880Z", "content": "https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/505491
\nThis should help. Let me know if it works :)
Thanks, I have seen and upvoted your post before. I am removing all unnecessary dataframes, but it still fails. Thats especially weird for me, because code without memory optimization submits successfully
", "rawMarkdown": "Thanks, I have seen and upvoted your post before. I am removing all unnecessary dataframes, but it still fails. Thats especially weird for me, because code without memory optimization submits successfully", "replies": [ { "id": 2828941, "postDate": "2024-05-22T10:21:15.240Z", "content": "oh, ok. that's weird.
", "rawMarkdown": "oh, ok. that's weird." } ] } ] }, { "id": 2837387, "postDate": "2024-05-26T13:03:19.053Z", "content": "After all, has anyone solved the phenomenon where the exact same code sometimes succeeds and sometimes fails due to out-of-memory? Can this be solved by modifying the code or is it a problem with the kaggle execution environment?
\nI wish someone could give us a solution for this memory issue even after the competition is over. I am more interested in this than the metric hacking method of 1st place solution.
", "rawMarkdown": "After all, has anyone solved the phenomenon where the exact same code sometimes succeeds and sometimes fails due to out-of-memory? Can this be solved by modifying the code or is it a problem with the kaggle execution environment?\n\nI wish someone could give us a solution for this memory issue even after the competition is over. I am more interested in this than the metric hacking method of 1st place solution.", "votes": 1 }, { "id": 2835028, "postDate": "2024-05-25T06:25:20.030Z", "content": "It seems to be a common issue …
", "rawMarkdown": "It seems to be a common issue ...", "votes": 1 }, { "id": 2834152, "postDate": "2024-05-24T15:20:38.110Z", "content": "Same here. I have submitted a notebook with similar memory usage to previously successful submissions but failed multiple times with OOM. The worst part is the logs still show \"successfully ran\", so we can't identify which part actually causes the issue.
", "rawMarkdown": "Same here. I have submitted a notebook with similar memory usage to previously successful submissions but failed multiple times with OOM. The worst part is the logs still show \"successfully ran\", so we can't identify which part actually causes the issue. ", "votes": 1 }, { "id": 2829028, "postDate": "2024-05-22T11:18:55.590Z", "content": "I also have a submission that was working fine until yesterday, but today it has failed twice in a row. I'm also trying my best to delete unnecessary variables. is there more test data to calculate in LB or something?
\nBut how is that possible?
I had no choice but to stop using the VotingModel class I had been using to aggregate and infer my learned models, and managed to get the submissions to succeed by loading the models one by one and deleting them when I had finished inferring them. Well, it takes longer than before.
", "rawMarkdown": "I had no choice but to stop using the VotingModel class I had been using to aggregate and infer my learned models, and managed to get the submissions to succeed by loading the models one by one and deleting them when I had finished inferring them. Well, it takes longer than before." } ] }, { "id": 2828816, "postDate": "2024-05-22T09:35:38Z", "content": "Yes. Especially yesterday, submissions that worked before failed three times in a row for me.
\nSeems like the ram usage varies a bit from time to time, and if its already quite high, it might fail every now and then. But that's just my guess, I have really no idea.
\nA bit annoying since I'm trying out new ideas now in the last days, and now I might instead have to spend more time on memory things..
", "rawMarkdown": "Yes. Especially yesterday, submissions that worked before failed three times in a row for me. \n\nSeems like the ram usage varies a bit from time to time, and if its already quite high, it might fail every now and then. But that's just my guess, I have really no idea.\n\nA bit annoying since I'm trying out new ideas now in the last days, and now I might instead have to spend more time on memory things..\n\n\n\n", "votes": 1 }, { "id": 2837154, "postDate": "2024-05-26T10:12:50.540Z", "content": "Ya me too, my memory always run out while doing the coding, but luckily everything is still going well, GG
", "rawMarkdown": "Ya me too, my memory always run out while doing the coding, but luckily everything is still going well, GG" }, { "id": 2836702, "postDate": "2024-05-26T04:42:20.170Z", "content": "Hello, I faced the same issue here. I successfully uploaded my notebook, and the log also shows that it ran successfully. However, it always shows an out-of-memory error when I submit my notebook. Is it because this competition submission's RAM condition is more strict than uploading the notebook?
", "rawMarkdown": "Hello, I faced the same issue here. I successfully uploaded my notebook, and the log also shows that it ran successfully. However, it always shows an out-of-memory error when I submit my notebook. Is it because this competition submission's RAM condition is more strict than uploading the notebook?", "replies": [ { "id": 2837052, "postDate": "2024-05-26T08:43:35.023Z", "content": "Because test run computes data for test set only for 10 samples, but for scoring its computed on whole dataset, thus it takes more memory to store. You have to manually clean RAM
", "rawMarkdown": "Because test run computes data for test set only for 10 samples, but for scoring its computed on whole dataset, thus it takes more memory to store. You have to manually clean RAM", "votes": 1, "replies": [ { "id": 2838564, "postDate": "2024-05-27T05:35:20.303Z", "content": "Thank you for the reply!
", "rawMarkdown": "Thank you for the reply!" } ] } ] }, { "id": 2829940, "postDate": "2024-05-22T22:42:42.360Z", "content": "When I submit the same code, sometimes it works, sometimes it gets an error.
", "rawMarkdown": "When I submit the same code, sometimes it works, sometimes it gets an error." }, { "id": 2829642, "postDate": "2024-05-22T17:49:53.200Z", "content": "Hi @bluepill , I don't know if it is the case but lightgbm with GPU faces sometimes unexpected issues.
", "rawMarkdown": "Hi @bluepill , I don't know if it is the case but lightgbm with GPU faces sometimes unexpected issues.", "replies": [ { "id": 2829650, "postDate": "2024-05-22T17:59:16.087Z", "content": "I switched to CPU since lgbm GPU performance is worse
", "rawMarkdown": "I switched to CPU since lgbm GPU performance is worse" } ] }, { "id": 2829475, "postDate": "2024-05-22T15:48:17.240Z", "content": "Yeah, I also have the same problem. It is so weird that the notebook keeps running even after it is out of memory (no score recorded). And also it was working fine before I submitted it. I tried to use 80% of the train data but it still had errors.
\nI already made a post in: https://www.kaggle.com/discussions/product-feedback/506603
", "rawMarkdown": "Yeah, I also have the same problem. It is so weird that the notebook keeps running even after it is out of memory (no score recorded). And also it was working fine before I submitted it. I tried to use 80% of the train data but it still had errors.\n\nI already made a post in: https://www.kaggle.com/discussions/product-feedback/506603", "replies": [ { "id": 2829485, "postDate": "2024-05-22T15:52:01.720Z", "content": "What I find strange when testing it locally is that the variation is the same there. Sometimes it uses just half of the ram, other times is almost uses all of it. And it's the same code 🤷♂️
", "rawMarkdown": "What I find strange when testing it locally is that the variation is the same there. Sometimes it uses just half of the ram, other times is almost uses all of it. And it's the same code 🤷♂️" } ] }, { "id": 2829234, "postDate": "2024-05-22T13:44:31.130Z", "content": "mb @tomasjeline2 could clarify if test data volume has been increased or sample changed?
", "rawMarkdown": "mb @tomasjeline2 could clarify if test data volume has been increased or sample changed?", "replies": [ { "id": 2833731, "postDate": "2024-05-24T11:25:53.007Z", "content": "Hi @bluepill,
\nsorry for late reply. No, we haven't change anything with test data since restart of the competition. Volume of data, applied transformation, public-private split - all remains same and no changes have been done.
Same here. Now I'm working on the out-of-memory issue by submitting several times…
", "rawMarkdown": "Same here. Now I'm working on the out-of-memory issue by submitting several times...", "replies": [ { "id": 2834700, "postDate": "2024-05-25T00:10:36.473Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 2828825, "author_name": "Krishna Priya", "author_url": "", "post_date": "2024-05-22T09:44:32.880000", "content": "https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/505491
\nThis should help. Let me know if it works :)
Thanks, I have seen and upvoted your post before. I am removing all unnecessary dataframes, but it still fails. Thats especially weird for me, because code without memory optimization submits successfully
", "votes": 0, "replies": [ { "id": 2828941, "author_name": "Krishna Priya", "author_url": "", "post_date": "2024-05-22T10:21:15.240000", "content": "oh, ok. that's weird.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2837387, "author_name": "ano", "author_url": "", "post_date": "2024-05-26T13:03:19.053000", "content": "After all, has anyone solved the phenomenon where the exact same code sometimes succeeds and sometimes fails due to out-of-memory? Can this be solved by modifying the code or is it a problem with the kaggle execution environment?
\nI wish someone could give us a solution for this memory issue even after the competition is over. I am more interested in this than the metric hacking method of 1st place solution.
", "votes": 1, "replies": [] }, { "id": 2835028, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-05-25T06:25:20.030000", "content": "It seems to be a common issue …
", "votes": 1, "replies": [] }, { "id": 2834152, "author_name": "LYCHEE", "author_url": "", "post_date": "2024-05-24T15:20:38.110000", "content": "Same here. I have submitted a notebook with similar memory usage to previously successful submissions but failed multiple times with OOM. The worst part is the logs still show \"successfully ran\", so we can't identify which part actually causes the issue.
", "votes": 1, "replies": [] }, { "id": 2829028, "author_name": "kyotoman", "author_url": "", "post_date": "2024-05-22T11:18:55.590000", "content": "I also have a submission that was working fine until yesterday, but today it has failed twice in a row. I'm also trying my best to delete unnecessary variables. is there more test data to calculate in LB or something?
\nBut how is that possible?
I had no choice but to stop using the VotingModel class I had been using to aggregate and infer my learned models, and managed to get the submissions to succeed by loading the models one by one and deleting them when I had finished inferring them. Well, it takes longer than before.
", "votes": 0, "replies": [] } ] }, { "id": 2828816, "author_name": "Ern711", "author_url": "", "post_date": "2024-05-22T09:35:38", "content": "Yes. Especially yesterday, submissions that worked before failed three times in a row for me.
\nSeems like the ram usage varies a bit from time to time, and if its already quite high, it might fail every now and then. But that's just my guess, I have really no idea.
\nA bit annoying since I'm trying out new ideas now in the last days, and now I might instead have to spend more time on memory things..
", "votes": 1, "replies": [] }, { "id": 2837154, "author_name": "Carlos SCJ", "author_url": "", "post_date": "2024-05-26T10:12:50.540000", "content": "Ya me too, my memory always run out while doing the coding, but luckily everything is still going well, GG
", "votes": 0, "replies": [] }, { "id": 2836702, "author_name": "AustinCKR", "author_url": "", "post_date": "2024-05-26T04:42:20.170000", "content": "Hello, I faced the same issue here. I successfully uploaded my notebook, and the log also shows that it ran successfully. However, it always shows an out-of-memory error when I submit my notebook. Is it because this competition submission's RAM condition is more strict than uploading the notebook?
", "votes": 0, "replies": [ { "id": 2837052, "author_name": "Andrey Chankin", "author_url": "", "post_date": "2024-05-26T08:43:35.023000", "content": "Because test run computes data for test set only for 10 samples, but for scoring its computed on whole dataset, thus it takes more memory to store. You have to manually clean RAM
", "votes": 1, "replies": [ { "id": 2838564, "author_name": "AustinCKR", "author_url": "", "post_date": "2024-05-27T05:35:20.303000", "content": "Thank you for the reply!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2829940, "author_name": "", "author_url": "", "post_date": "2024-05-22T22:42:42.360000", "content": "When I submit the same code, sometimes it works, sometimes it gets an error.
", "votes": 0, "replies": [] }, { "id": 2829642, "author_name": "Laurent Pourchot", "author_url": "", "post_date": "2024-05-22T17:49:53.200000", "content": "Hi @bluepill , I don't know if it is the case but lightgbm with GPU faces sometimes unexpected issues.
", "votes": 0, "replies": [ { "id": 2829650, "author_name": "Andrey Chankin", "author_url": "", "post_date": "2024-05-22T17:59:16.087000", "content": "I switched to CPU since lgbm GPU performance is worse
", "votes": 0, "replies": [] } ] }, { "id": 2829475, "author_name": "Evan", "author_url": "", "post_date": "2024-05-22T15:48:17.240000", "content": "Yeah, I also have the same problem. It is so weird that the notebook keeps running even after it is out of memory (no score recorded). And also it was working fine before I submitted it. I tried to use 80% of the train data but it still had errors.
\nI already made a post in: https://www.kaggle.com/discussions/product-feedback/506603
", "votes": 0, "replies": [ { "id": 2829485, "author_name": "Ern711", "author_url": "", "post_date": "2024-05-22T15:52:01.720000", "content": "What I find strange when testing it locally is that the variation is the same there. Sometimes it uses just half of the ram, other times is almost uses all of it. And it's the same code 🤷♂️
", "votes": 0, "replies": [] } ] }, { "id": 2829234, "author_name": "Andrey Chankin", "author_url": "", "post_date": "2024-05-22T13:44:31.130000", "content": "mb @tomasjeline2 could clarify if test data volume has been increased or sample changed?
", "votes": 0, "replies": [ { "id": 2833731, "author_name": "Tomas Jelinek", "author_url": "", "post_date": "2024-05-24T11:25:53.007000", "content": "Hi @bluepill,
\nsorry for late reply. No, we haven't change anything with test data since restart of the competition. Volume of data, applied transformation, public-private split - all remains same and no changes have been done.
Same here. Now I'm working on the out-of-memory issue by submitting several times…
", "votes": 0, "replies": [ { "id": 2834700, "author_name": "", "author_url": "", "post_date": "2024-05-25T00:10:36.473000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2828780": "Hello everyone!\n\nMost of my recent submissions are failed with memory errors, I decided to inspect what is wrong and submitted code which previously scored successfully, but now it has failed with out-of-memory error.\n\nAlso, previously, I have encountered situation when failed notebooks were scored after resubmitting.\n\nDoes anyone have same issues?", "2828825": "https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/505491\nThis should help. Let me know if it works :)", "2837387": "After all, has anyone solved the phenomenon where the exact same code sometimes succeeds and sometimes fails due to out-of-memory? Can this be solved by modifying the code or is it a problem with the kaggle execution environment?\n\nI wish someone could give us a solution for this memory issue even after the competition is over. I am more interested in this than the metric hacking method of 1st place solution.", "2835028": "It seems to be a common issue ...", "2834152": "Same here. I have submitted a notebook with similar memory usage to previously successful submissions but failed multiple times with OOM. The worst part is the logs still show \"successfully ran\", so we can't identify which part actually causes the issue. ", "2829028": "I also have a submission that was working fine until yesterday, but today it has failed twice in a row. I'm also trying my best to delete unnecessary variables. is there more test data to calculate in LB or something?\nBut how is that possible?", "2828816": "Yes. Especially yesterday, submissions that worked before failed three times in a row for me. \n\nSeems like the ram usage varies a bit from time to time, and if its already quite high, it might fail every now and then. But that's just my guess, I have really no idea.\n\nA bit annoying since I'm trying out new ideas now in the last days, and now I might instead have to spend more time on memory things..\n\n\n\n", "2837154": "Ya me too, my memory always run out while doing the coding, but luckily everything is still going well, GG", "2836702": "Hello, I faced the same issue here. I successfully uploaded my notebook, and the log also shows that it ran successfully. However, it always shows an out-of-memory error when I submit my notebook. Is it because this competition submission's RAM condition is more strict than uploading the notebook?", "2829940": "When I submit the same code, sometimes it works, sometimes it gets an error.", "2829642": "Hi @bluepill , I don't know if it is the case but lightgbm with GPU faces sometimes unexpected issues.", "2829475": "Yeah, I also have the same problem. It is so weird that the notebook keeps running even after it is out of memory (no score recorded). And also it was working fine before I submitted it. I tried to use 80% of the train data but it still had errors.\n\nI already made a post in: https://www.kaggle.com/discussions/product-feedback/506603", "2829234": "mb @tomasjeline2 could clarify if test data volume has been increased or sample changed?", "2829058": "Same here. Now I'm working on the out-of-memory issue by submitting several times..." } }