I tried to use Kaggle notebook to preprocess the data (just load the train dataset only and do two pd.merge) and I used the dtypes which can minimize the memory usage for every df, but the notebook still crashed due to memory issue.
\nThen I tried on my local jupyterlab and did the same operation but it didn't raise any errors or crash.
\nMy computer also has 16GB RAM, so why it crashed in Kaggle notebook but worked fine locally?
I tried to use Kaggle notebook to preprocess the data (just load the train dataset only and do two pd.merge) and I used the dtypes which can minimize the memory usage for every df, but the notebook still crashed due to memory issue.
\nThen I tried on my local jupyterlab and did the same operation but it didn't raise any errors or crash.
\nMy computer also has 16GB RAM, so why it crashed in Kaggle notebook but worked fine locally?
Did you try to process a training dataset a bit before merge to reduce its size and then perform merge operation? I'm doing it in my notebook and everything is going well - reaching close to 16GB. Also I'm removing not needed variables and data as soon as they are not required.
", "rawMarkdown": "Did you try to process a training dataset a bit before merge to reduce its size and then perform merge operation? I'm doing it in my notebook and everything is going well - reaching close to 16GB. Also I'm removing not needed variables and data as soon as they are not required.", "votes": null }, { "id": "1057249", "postDate": "10/22/2020 14:06:34", "content": "Yes, I actually reduced the size (select question's rows) and also drop the cols not needed but still encountered this issue and it worked just fine locally even though my computer was running other apps.
\nI just wonder why they have the same RAM (16G), but kaggle notebook didn't work, and I think I can process the data offline and upload it onto Kaggle, can I?
I faced same problem, I search online but didn't find any answers or solutions. I believe it is an issue related to the virtual 16 GB RAM given by kaggle (i.e. not related to the code or pandas).
\nIf you find any explanation for the problem please let me know.
Haven't found any solutions yet, maybe the only thing I can do is to process the data offline and upload it onto Kaggle notebook if allowed
", "rawMarkdown": "Haven't found any solutions yet, maybe the only thing I can do is to process the data offline and upload it onto Kaggle notebook if allowed", "votes": null }, { "id": "1116259", "postDate": "12/17/2020 02:08:46", "content": "Maybe this is explained here https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/193113 by @kaushal2896 as follows:
\n\"That's because with GPU enabled, you only get 13GB of RAM and when GPU is disabled, you get 16GB of RAM.\"
", "rawMarkdown": "Maybe this is explained here [https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/193113](url) by @kaushal2896 as follows:\n\n\"That's because with GPU enabled, you only get 13GB of RAM and when GPU is disabled, you get 16GB of RAM.\"", "votes": null } ], "comments": [ { "id": 1057095, "author_name": "danpeczek", "author_url": "", "post_date": "10/22/2020 11:09:11", "content": "Did you try to process a training dataset a bit before merge to reduce its size and then perform merge operation? I'm doing it in my notebook and everything is going well - reaching close to 16GB. Also I'm removing not needed variables and data as soon as they are not required.
", "votes": null, "replies": [ { "id": 1057249, "author_name": "patrickyip", "author_url": "", "post_date": "10/22/2020 14:06:34", "content": "Yes, I actually reduced the size (select question's rows) and also drop the cols not needed but still encountered this issue and it worked just fine locally even though my computer was running other apps.
\nI just wonder why they have the same RAM (16G), but kaggle notebook didn't work, and I think I can process the data offline and upload it onto Kaggle, can I?
I faced same problem, I search online but didn't find any answers or solutions. I believe it is an issue related to the virtual 16 GB RAM given by kaggle (i.e. not related to the code or pandas).
\nIf you find any explanation for the problem please let me know.
Haven't found any solutions yet, maybe the only thing I can do is to process the data offline and upload it onto Kaggle notebook if allowed
", "votes": null, "replies": [] }, { "id": 1116259, "author_name": "hinepo", "author_url": "", "post_date": "12/17/2020 02:08:46", "content": "Maybe this is explained here https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/193113 by @kaushal2896 as follows:
\n\"That's because with GPU enabled, you only get 13GB of RAM and when GPU is disabled, you get 16GB of RAM.\"
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1056882": "I tried to use Kaggle notebook to preprocess the data (just load the train dataset only and do two pd.merge) and I used the dtypes which can minimize the memory usage for every df, but the notebook still crashed due to memory issue.\nThen I tried on my local jupyterlab and did the same operation but it didn't raise any errors or crash.\nMy computer also has 16GB RAM, so why it crashed in Kaggle notebook but worked fine locally?", "1057095": "Did you try to process a training dataset a bit before merge to reduce its size and then perform merge operation? I'm doing it in my notebook and everything is going well - reaching close to 16GB. Also I'm removing not needed variables and data as soon as they are not required.", "1057249": "Yes, I actually reduced the size (select question's rows) and also drop the cols not needed but still encountered this issue and it worked just fine locally even though my computer was running other apps.\nI just wonder why they have the same RAM (16G), but kaggle notebook didn't work, and I think I can process the data offline and upload it onto Kaggle, can I?", "1058368": "I faced same problem, I search online but didn't find any answers or solutions. I believe it is an issue related to the virtual 16 GB RAM given by kaggle (i.e. not related to the code or pandas).\nIf you find any explanation for the problem please let me know.", "1058822": "Haven't found any solutions yet, maybe the only thing I can do is to process the data offline and upload it onto Kaggle notebook if allowed", "1116259": "Maybe this is explained here [https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/193113](url) by @kaushal2896 as follows:\n\n\"That's because with GPU enabled, you only get 13GB of RAM and when GPU is disabled, you get 16GB of RAM.\"" }, "source": "meta" }