{
  "id": 498690,
  "title": "OOM during the competition submission phase",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/498690",
  "author_name": "",
  "post_date": "2024-04-29T08:41:51.873325100Z",
  "votes": 2,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Why does my code run successfully in the notebook, and it runs successfully in scoring phase, but I encounter an Out Of Memory (OOM) error during the competition submission phase?</p>",
  "messages": [
    {
      "id": "2782289",
      "postDate": "04/29/2024 08:41:51",
      "content": "<p>Why does my code run successfully in the notebook, and it runs successfully in scoring phase, but I encounter an Out Of Memory (OOM) error during the competition submission phase?</p>",
      "rawMarkdown": "Why does my code run successfully in the notebook, and it runs successfully in scoring phase, but I encounter an Out Of Memory (OOM) error during the competition submission phase?",
      "votes": null
    },
    {
      "id": "2782590",
      "postDate": "04/29/2024 11:25:21",
      "content": "<p>Because the test data is big, and your code occupies a lot of memory in proccessing data.</p>",
      "rawMarkdown": "Because the test data is big, and your code occupies a lot of memory in proccessing data.",
      "votes": null
    },
    {
      "id": "2783765",
      "postDate": "04/30/2024 00:03:16",
      "content": "<p>The error occurred during the scoring phase, the execution phase were successful.Do you think it is the large size of the training set data or the testing set data that is causing this issue?</p>",
      "rawMarkdown": "The error occurred during the scoring phase, the execution phase were successful.Do you think it is the large size of the training set data or the testing set data that is causing this issue?",
      "votes": null
    },
    {
      "id": "2783953",
      "postDate": "04/30/2024 03:13:34",
      "content": "<p>yes,  the test data is big.</p>",
      "rawMarkdown": "yes,  the test data is big.",
      "votes": null
    },
    {
      "id": "2784798",
      "postDate": "04/30/2024 13:55:26",
      "content": "<p>The following error occurred during the scoring phase，What does it mean?</p>\n<p>/opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2930: FutureWarning: --Exporter.preprocessors=[\"remove_papermill_header.RemovePapermillHeader\"] for containers is deprecated in traitlets 5.0. You can pass <code>--Exporter.preprocessors item</code> … multiple times to add items to a list.<br>\n366.3s    750   warn(<br>\n366.3s    751 [NbConvertApp] WARNING | Config option <code>kernel_spec_manager_class</code> not recognized by <code>NbConvertApp</code>.<br>\n366.4s    752 [NbConvertApp] Converting notebook <strong>notebook</strong>.ipynb to notebook<br>\n366.9s    753 [NbConvertApp] Writing 182342 bytes to <strong>notebook</strong>.ipynb<br>\n368.5s    754 /opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2930: FutureWarning: --Exporter.preprocessors=[\"nbconvert.preprocessors.ExtractOutputPreprocessor\"] for containers is deprecated in traitlets 5.0. You can pass <code>--Exporter.preprocessors item</code> … multiple times to add items to a list.<br>\n368.5s    755   warn(<br>\n368.5s    756 [NbConvertApp] WARNING | Config option <code>kernel_spec_manager_class</code> not recognized by <code>NbConvertApp</code>.<br>\n368.5s    757 [NbConvertApp] Converting notebook <strong>notebook</strong>.ipynb to html<br>\n369.6s    758 [NbConvertApp] Support files will be in <strong>results___files/\n369.6s    759 [NbConvertApp] Making directory __results___files\n369.6s    760 [NbConvertApp] Writing 423943 bytes to __results</strong>.html</p>",
      "rawMarkdown": "The following error occurred during the scoring phase，What does it mean?\n\n\n/opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2930: FutureWarning: --Exporter.preprocessors=[\"remove_papermill_header.RemovePapermillHeader\"] for containers is deprecated in traitlets 5.0. You can pass `--Exporter.preprocessors item` ... multiple times to add items to a list.\n366.3s\t750\t  warn(\n366.3s\t751\t[NbConvertApp] WARNING | Config option `kernel_spec_manager_class` not recognized by `NbConvertApp`.\n366.4s\t752\t[NbConvertApp] Converting notebook __notebook__.ipynb to notebook\n366.9s\t753\t[NbConvertApp] Writing 182342 bytes to __notebook__.ipynb\n368.5s\t754\t/opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2930: FutureWarning: --Exporter.preprocessors=[\"nbconvert.preprocessors.ExtractOutputPreprocessor\"] for containers is deprecated in traitlets 5.0. You can pass `--Exporter.preprocessors item` ... multiple times to add items to a list.\n368.5s\t755\t  warn(\n368.5s\t756\t[NbConvertApp] WARNING | Config option `kernel_spec_manager_class` not recognized by `NbConvertApp`.\n368.5s\t757\t[NbConvertApp] Converting notebook __notebook__.ipynb to html\n369.6s\t758\t[NbConvertApp] Support files will be in __results___files/\n369.6s\t759\t[NbConvertApp] Making directory __results___files\n369.6s\t760\t[NbConvertApp] Writing 423943 bytes to __results__.html",
      "votes": null
    },
    {
      "id": "2789558",
      "postDate": "05/02/2024 18:00:33",
      "content": "<p>Delete the training set from memory before loading the test set </p>",
      "rawMarkdown": "Delete the training set from memory before loading the test set",
      "votes": null
    },
    {
      "id": "2802914",
      "postDate": "05/09/2024 08:24:06",
      "content": "<p>About 1.35 millions rows has testing data size !! 90 % of training data , host said in one of the comments ..</p>",
      "rawMarkdown": "About 1.35 millions rows has testing data size !! 90 % of training data , host said in one of the comments ..",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2782590,
      "author_name": "yangq369",
      "author_url": "",
      "post_date": "04/29/2024 11:25:21",
      "content": "<p>Because the test data is big, and your code occupies a lot of memory in proccessing data.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2783765,
          "author_name": "zhangzil2025",
          "author_url": "",
          "post_date": "04/30/2024 00:03:16",
          "content": "<p>The error occurred during the scoring phase, the execution phase were successful.Do you think it is the large size of the training set data or the testing set data that is causing this issue?</p>",
          "votes": null,
          "replies": [
            {
              "id": 2783953,
              "author_name": "huangshibao",
              "author_url": "",
              "post_date": "04/30/2024 03:13:34",
              "content": "<p>yes,  the test data is big.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2784798,
                  "author_name": "zhangzil2025",
                  "author_url": "",
                  "post_date": "04/30/2024 13:55:26",
                  "content": "<p>The following error occurred during the scoring phase，What does it mean?</p>\n<p>/opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2930: FutureWarning: --Exporter.preprocessors=[\"remove_papermill_header.RemovePapermillHeader\"] for containers is deprecated in traitlets 5.0. You can pass <code>--Exporter.preprocessors item</code> … multiple times to add items to a list.<br>\n366.3s    750   warn(<br>\n366.3s    751 [NbConvertApp] WARNING | Config option <code>kernel_spec_manager_class</code> not recognized by <code>NbConvertApp</code>.<br>\n366.4s    752 [NbConvertApp] Converting notebook <strong>notebook</strong>.ipynb to notebook<br>\n366.9s    753 [NbConvertApp] Writing 182342 bytes to <strong>notebook</strong>.ipynb<br>\n368.5s    754 /opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2930: FutureWarning: --Exporter.preprocessors=[\"nbconvert.preprocessors.ExtractOutputPreprocessor\"] for containers is deprecated in traitlets 5.0. You can pass <code>--Exporter.preprocessors item</code> … multiple times to add items to a list.<br>\n368.5s    755   warn(<br>\n368.5s    756 [NbConvertApp] WARNING | Config option <code>kernel_spec_manager_class</code> not recognized by <code>NbConvertApp</code>.<br>\n368.5s    757 [NbConvertApp] Converting notebook <strong>notebook</strong>.ipynb to html<br>\n369.6s    758 [NbConvertApp] Support files will be in <strong>results___files/\n369.6s    759 [NbConvertApp] Making directory __results___files\n369.6s    760 [NbConvertApp] Writing 423943 bytes to __results</strong>.html</p>",
                  "votes": null,
                  "replies": []
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 2789558,
      "author_name": "eu1234",
      "author_url": "",
      "post_date": "05/02/2024 18:00:33",
      "content": "<p>Delete the training set from memory before loading the test set </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2802914,
      "author_name": "kunduruanil",
      "author_url": "",
      "post_date": "05/09/2024 08:24:06",
      "content": "<p>About 1.35 millions rows has testing data size !! 90 % of training data , host said in one of the comments ..</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2782289": "Why does my code run successfully in the notebook, and it runs successfully in scoring phase, but I encounter an Out Of Memory (OOM) error during the competition submission phase?",
    "2782590": "Because the test data is big, and your code occupies a lot of memory in proccessing data.",
    "2783765": "The error occurred during the scoring phase, the execution phase were successful.Do you think it is the large size of the training set data or the testing set data that is causing this issue?",
    "2783953": "yes,  the test data is big.",
    "2784798": "The following error occurred during the scoring phase，What does it mean?\n\n\n/opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2930: FutureWarning: --Exporter.preprocessors=[\"remove_papermill_header.RemovePapermillHeader\"] for containers is deprecated in traitlets 5.0. You can pass `--Exporter.preprocessors item` ... multiple times to add items to a list.\n366.3s\t750\t  warn(\n366.3s\t751\t[NbConvertApp] WARNING | Config option `kernel_spec_manager_class` not recognized by `NbConvertApp`.\n366.4s\t752\t[NbConvertApp] Converting notebook __notebook__.ipynb to notebook\n366.9s\t753\t[NbConvertApp] Writing 182342 bytes to __notebook__.ipynb\n368.5s\t754\t/opt/conda/lib/python3.10/site-packages/traitlets/traitlets.py:2930: FutureWarning: --Exporter.preprocessors=[\"nbconvert.preprocessors.ExtractOutputPreprocessor\"] for containers is deprecated in traitlets 5.0. You can pass `--Exporter.preprocessors item` ... multiple times to add items to a list.\n368.5s\t755\t  warn(\n368.5s\t756\t[NbConvertApp] WARNING | Config option `kernel_spec_manager_class` not recognized by `NbConvertApp`.\n368.5s\t757\t[NbConvertApp] Converting notebook __notebook__.ipynb to html\n369.6s\t758\t[NbConvertApp] Support files will be in __results___files/\n369.6s\t759\t[NbConvertApp] Making directory __results___files\n369.6s\t760\t[NbConvertApp] Writing 423943 bytes to __results__.html",
    "2789558": "Delete the training set from memory before loading the test set",
    "2802914": "About 1.35 millions rows has testing data size !! 90 % of training data , host said in one of the comments .."
  },
  "source": "meta"
}