This is my first time participating in this competition, so I may be ignorant on many points, but I would like to ask a question.
\nI am encountering \"Notebook Out of Memory Errors\" and have not been able to resolve it. My understanding is that the memory usage should be within 32GB, and I am not using a GPU.
\nInitially, I thought the memory usage exceeded 32GB due to the processing steps, so I reduced the data used for training and resubmitted the notebook. As a result, the memory usage during the processing steps reached a maximum of about 25GB, but the error was not resolved.
\nI understand that in the competition, the notebook is applied to a private dataset for evaluation. Does this mean that the memory usage, including during this private evaluation, must also be within 32GB?
\nHowever, there are times when a score is generated even with the original data, which clearly used more memory than currently, and this confuses me. I would appreciate it if you could also explain the general method to determine if the memory usage is within the specified limit.
", "messages": [ { "id": "2816262", "postDate": "05/16/2024 08:12:07", "content": "This is my first time participating in this competition, so I may be ignorant on many points, but I would like to ask a question.
\nI am encountering \"Notebook Out of Memory Errors\" and have not been able to resolve it. My understanding is that the memory usage should be within 32GB, and I am not using a GPU.
\nInitially, I thought the memory usage exceeded 32GB due to the processing steps, so I reduced the data used for training and resubmitted the notebook. As a result, the memory usage during the processing steps reached a maximum of about 25GB, but the error was not resolved.
\nI understand that in the competition, the notebook is applied to a private dataset for evaluation. Does this mean that the memory usage, including during this private evaluation, must also be within 32GB?
\nHowever, there are times when a score is generated even with the original data, which clearly used more memory than currently, and this confuses me. I would appreciate it if you could also explain the general method to determine if the memory usage is within the specified limit.
", "rawMarkdown": "This is my first time participating in this competition, so I may be ignorant on many points, but I would like to ask a question.\n\nI am encountering \"Notebook Out of Memory Errors\" and have not been able to resolve it. My understanding is that the memory usage should be within 32GB, and I am not using a GPU.\n\nInitially, I thought the memory usage exceeded 32GB due to the processing steps, so I reduced the data used for training and resubmitted the notebook. As a result, the memory usage during the processing steps reached a maximum of about 25GB, but the error was not resolved.\n\nI understand that in the competition, the notebook is applied to a private dataset for evaluation. Does this mean that the memory usage, including during this private evaluation, must also be within 32GB?\n\nHowever, there are times when a score is generated even with the original data, which clearly used more memory than currently, and this confuses me. I would appreciate it if you could also explain the general method to determine if the memory usage is within the specified limit.", "votes": null }, { "id": "2816606", "postDate": "05/16/2024 12:29:29", "content": "The reason I think this way is that after the log indicating the completion of all processing in my notebook, the following logs and warnings appear. Is the processing during the conversion of the notebook disclosed anywhere?
\n305.1s 40 /opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2930: FutureWarning: --Exporter.preprocessors=[\"remove_papermill_header.RemovePapermillHeader\"] for containers is deprecated in traitlets 5.0. You can pass--Exporter.preprocessors item... multiple times to add items to a list.\n305.1s 41 warn(\n305.1s 42 [NbConvertApp] WARNING | Config optionkernel_spec_manager_classnot recognized byNbConvertApp.\n305.2s 43 [NbConvertApp] Converting notebook __notebook__.ipynb to notebook\n305.6s 44 [NbConvertApp] Writing 28517 bytes to __notebook__.ipynb\n307.3s 45 /opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2930: FutureWarning: --Exporter.preprocessors=[\"nbconvert.preprocessors.ExtractOutputPreprocessor\"] for containers is deprecated in traitlets 5.0. You can pass--Exporter.preprocessors item... multiple times to add items to a list.\n307.3s 46 warn(\n307.3s 47 [NbConvertApp] WARNING | Config optionkernel_spec_manager_classnot recognized byNbConvertApp.\n307.4s 48 [NbConvertApp] Converting notebook __notebook__.ipynb to html\n308.4s 49 [NbConvertApp] Writing 357764 bytes to __results__.html
\n\nDoes this mean that the memory usage, including during this private evaluation, must also be within 32GB?
\n
Yes, that's correct. If you are including train data in your inference notebook, the inference with the included train dataset and the private dataset must be within the memory limit of 32 GB.
\nAlso encountered OOM errors several days ago. Below are some tips that help me resolve this issue:
\ndel var; gc.collect() after you are done with using that vardef predict:\n probas = \n i range(, len(X_test), batch_size):\n batch = X_test\n proba_batch = model.predict\n probas.extend(proba_batch)\n\n return pd.\nThank you very much.
\nI had already been deleting unused variables, using garbage collection, and loading the trained model as needed, but I hadn't implemented batch processing yet. I'll give that a try. Your help has been invaluable.
Also, consider the option of separate notebooks for model training and prediction
", "rawMarkdown": "Also, consider the option of separate notebooks for model training and prediction", "votes": null }, { "id": "2818186", "postDate": "05/17/2024 10:34:42", "content": "try run different heavy memory parts in seperated memory leaving zero memory traces like below, only saving and loading needed info to next codes.
\n%%writefile run1.py\n\nthen run
\n!python run1.py
etc….
", "rawMarkdown": "try run different heavy memory parts in seperated memory leaving zero memory traces like below, only saving and loading needed info to next codes.\n\n```python\n%%writefile run1.py\n#code#\n```\n\nthen run\n\n`!python run1.py`\n\netc....", "votes": null } ], "comments": [ { "id": 2816606, "author_name": "kissshotorca", "author_url": "", "post_date": "05/16/2024 12:29:29", "content": "The reason I think this way is that after the log indicating the completion of all processing in my notebook, the following logs and warnings appear. Is the processing during the conversion of the notebook disclosed anywhere?
\n305.1s 40 /opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2930: FutureWarning: --Exporter.preprocessors=[\"remove_papermill_header.RemovePapermillHeader\"] for containers is deprecated in traitlets 5.0. You can pass--Exporter.preprocessors item... multiple times to add items to a list.\n305.1s 41 warn(\n305.1s 42 [NbConvertApp] WARNING | Config optionkernel_spec_manager_classnot recognized byNbConvertApp.\n305.2s 43 [NbConvertApp] Converting notebook __notebook__.ipynb to notebook\n305.6s 44 [NbConvertApp] Writing 28517 bytes to __notebook__.ipynb\n307.3s 45 /opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2930: FutureWarning: --Exporter.preprocessors=[\"nbconvert.preprocessors.ExtractOutputPreprocessor\"] for containers is deprecated in traitlets 5.0. You can pass--Exporter.preprocessors item... multiple times to add items to a list.\n307.3s 46 warn(\n307.3s 47 [NbConvertApp] WARNING | Config optionkernel_spec_manager_classnot recognized byNbConvertApp.\n307.4s 48 [NbConvertApp] Converting notebook __notebook__.ipynb to html\n308.4s 49 [NbConvertApp] Writing 357764 bytes to __results__.html
\n\nDoes this mean that the memory usage, including during this private evaluation, must also be within 32GB?
\n
Yes, that's correct. If you are including train data in your inference notebook, the inference with the included train dataset and the private dataset must be within the memory limit of 32 GB.
\nAlso encountered OOM errors several days ago. Below are some tips that help me resolve this issue:
\ndel var; gc.collect() after you are done with using that vardef predict:\n probas = \n i range(, len(X_test), batch_size):\n batch = X_test\n proba_batch = model.predict\n probas.extend(proba_batch)\n\n return pd.\nThank you very much.
\nI had already been deleting unused variables, using garbage collection, and loading the trained model as needed, but I hadn't implemented batch processing yet. I'll give that a try. Your help has been invaluable.
Also, consider the option of separate notebooks for model training and prediction
", "votes": null, "replies": [] }, { "id": 2818186, "author_name": "kirderf", "author_url": "", "post_date": "05/17/2024 10:34:42", "content": "try run different heavy memory parts in seperated memory leaving zero memory traces like below, only saving and loading needed info to next codes.
\n%%writefile run1.py\n\nthen run
\n!python run1.py
etc….
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2816262": "This is my first time participating in this competition, so I may be ignorant on many points, but I would like to ask a question.\n\nI am encountering \"Notebook Out of Memory Errors\" and have not been able to resolve it. My understanding is that the memory usage should be within 32GB, and I am not using a GPU.\n\nInitially, I thought the memory usage exceeded 32GB due to the processing steps, so I reduced the data used for training and resubmitted the notebook. As a result, the memory usage during the processing steps reached a maximum of about 25GB, but the error was not resolved.\n\nI understand that in the competition, the notebook is applied to a private dataset for evaluation. Does this mean that the memory usage, including during this private evaluation, must also be within 32GB?\n\nHowever, there are times when a score is generated even with the original data, which clearly used more memory than currently, and this confuses me. I would appreciate it if you could also explain the general method to determine if the memory usage is within the specified limit.", "2816606": "The reason I think this way is that after the log indicating the completion of all processing in my notebook, the following logs and warnings appear. Is the processing during the conversion of the notebook disclosed anywhere?\n\n`305.1s\t40\t/opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2930: FutureWarning: --Exporter.preprocessors=[\"remove_papermill_header.RemovePapermillHeader\"] for containers is deprecated in traitlets 5.0. You can pass `--Exporter.preprocessors item` ... multiple times to add items to a list.\n305.1s\t41\t warn(\n305.1s\t42\t[NbConvertApp] WARNING | Config option `kernel_spec_manager_class` not recognized by `NbConvertApp`.\n305.2s\t43\t[NbConvertApp] Converting notebook __notebook__.ipynb to notebook\n305.6s\t44\t[NbConvertApp] Writing 28517 bytes to __notebook__.ipynb\n307.3s\t45\t/opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2930: FutureWarning: --Exporter.preprocessors=[\"nbconvert.preprocessors.ExtractOutputPreprocessor\"] for containers is deprecated in traitlets 5.0. You can pass `--Exporter.preprocessors item` ... multiple times to add items to a list.\n307.3s\t46\t warn(\n307.3s\t47\t[NbConvertApp] WARNING | Config option `kernel_spec_manager_class` not recognized by `NbConvertApp`.\n307.4s\t48\t[NbConvertApp] Converting notebook __notebook__.ipynb to html\n308.4s\t49\t[NbConvertApp] Writing 357764 bytes to __results__.html`", "2816695": ">Does this mean that the memory usage, including during this private evaluation, must also be within 32GB?\n\nYes, that's correct. If you are including train data in your inference notebook, the inference with the included train dataset and the private dataset must be within the memory limit of 32 GB. \n\nAlso encountered OOM errors several days ago. Below are some tips that help me resolve this issue: \n\n1. Actively call `del var; gc.collect()` after you are done with using that `var`\n2. Predict in batches, e.g., please see the code snippet at the end\n3. Possibly separate the training and inference into two notebooks \n4. If the purpose of including the train dataset is just to select the features used during the training stage, instead of loading the whole train dataset, we can just save the train feature columns to a .csv file, and load that .csv file during inference, which can save a lot of memory and time loading the entire train dataset during the inference stage\n\n```\ndef predict_in_batches(model, X_test, batch_size=10000):\n probas = []\n for i in range(0, len(X_test), batch_size):\n batch = X_test[i:i+batch_size]\n proba_batch = model.predict_proba(batch)[:, 1]\n probas.extend(proba_batch)\n\n return pd.Series(probas, index=X_test.index, name=\"score\")\n```", "2816924": "Thank you very much.\nI had already been deleting unused variables, using garbage collection, and loading the trained model as needed, but I hadn't implemented batch processing yet. I'll give that a try. Your help has been invaluable.", "2817170": "Also, consider the option of separate notebooks for model training and prediction", "2818186": "try run different heavy memory parts in seperated memory leaving zero memory traces like below, only saving and loading needed info to next codes.\n\n```python\n%%writefile run1.py\n#code#\n```\n\nthen run\n\n`!python run1.py`\n\netc...." }, "source": "meta" }