{
  "id": 487960,
  "title": "Memory limit for competition? ",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/487960",
  "author_name": "",
  "post_date": "2024-03-31T14:24:53.213703200Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hey all anyone else have issue with memory errors during submission?<br>\nMy notebook runs without issue and creates a submission.csv file in less then 12 hour, while processing and predicting train datasets.  But competition submissions fails with memory error. </p>\n<p>I check the memory profile of my notebook , peak memory is approximately  26.5 gb during run time so I wonder if there is a memory limit less then that for this competition, I didn't read such restriction in description and proceeded to work thinking I will have access to 30 GB memory but now this submission errors happening I am wondering if there is a limit I am unaware of??<br>\nThere is no fail log, because notebook completes successfully but sum how submission failes… Note I did read actual test data is approximately 80% of train data. So to ensured pipeline won't exceed 30 gb when processing, I tested my pipeline on whole train set on kaggle envorinment and it successfully completed without memory failure so I am very much lost about what can be the issue. </p>",
  "messages": [
    {
      "id": "2725413",
      "postDate": "03/31/2024 14:24:53",
      "content": "<p>Hey all anyone else have issue with memory errors during submission?<br>\nMy notebook runs without issue and creates a submission.csv file in less then 12 hour, while processing and predicting train datasets.  But competition submissions fails with memory error. </p>\n<p>I check the memory profile of my notebook , peak memory is approximately  26.5 gb during run time so I wonder if there is a memory limit less then that for this competition, I didn't read such restriction in description and proceeded to work thinking I will have access to 30 GB memory but now this submission errors happening I am wondering if there is a limit I am unaware of??<br>\nThere is no fail log, because notebook completes successfully but sum how submission failes… Note I did read actual test data is approximately 80% of train data. So to ensured pipeline won't exceed 30 gb when processing, I tested my pipeline on whole train set on kaggle envorinment and it successfully completed without memory failure so I am very much lost about what can be the issue. </p>",
      "rawMarkdown": "Hey all anyone else have issue with memory errors during submission?\nMy notebook runs without issue and creates a submission.csv file in less then 12 hour, while processing and predicting train datasets.  But competition submissions fails with memory error. \n\nI check the memory profile of my notebook , peak memory is approximately  26.5 gb during run time so I wonder if there is a memory limit less then that for this competition, I didn't read such restriction in description and proceeded to work thinking I will have access to 30 GB memory but now this submission errors happening I am wondering if there is a limit I am unaware of??\nThere is no fail log, because notebook completes successfully but sum how submission failes... Note I did read actual test data is approximately 80% of train data. So to ensured pipeline won't exceed 30 gb when processing, I tested my pipeline on whole train set on kaggle envorinment and it successfully completed without memory failure so I am very much lost about what can be the issue.",
      "votes": null
    },
    {
      "id": "2725425",
      "postDate": "03/31/2024 14:29:47",
      "content": "<p>\"the hidden test_base.csv contains approximately 90% of the numbers of case_id values of train_base.csv\"</p>\n<p>But for each case id there can be more data than in the train set.</p>",
      "rawMarkdown": "\"the hidden test_base.csv contains approximately 90% of the numbers of case_id values of train_base.csv\"\n\nBut for each case id there can be more data than in the train set.",
      "votes": null
    },
    {
      "id": "2725428",
      "postDate": "03/31/2024 14:32:03",
      "content": "<p>Ah ok, I understand thank you 👍</p>",
      "rawMarkdown": "Ah ok, I understand thank you 👍",
      "votes": null
    },
    {
      "id": "2727001",
      "postDate": "04/01/2024 14:31:32",
      "content": "<p>Many solutions are here and in the comments <a href=\"https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/486259\" target=\"_blank\">https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/486259</a></p>",
      "rawMarkdown": "Many solutions are here and in the comments https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/486259",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2725425,
      "author_name": "eivolkova",
      "author_url": "",
      "post_date": "03/31/2024 14:29:47",
      "content": "<p>\"the hidden test_base.csv contains approximately 90% of the numbers of case_id values of train_base.csv\"</p>\n<p>But for each case id there can be more data than in the train set.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2725428,
          "author_name": "pelinkeskin",
          "author_url": "",
          "post_date": "03/31/2024 14:32:03",
          "content": "<p>Ah ok, I understand thank you 👍</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2727001,
      "author_name": "eu1234",
      "author_url": "",
      "post_date": "04/01/2024 14:31:32",
      "content": "<p>Many solutions are here and in the comments <a href=\"https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/486259\" target=\"_blank\">https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/486259</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2725413": "Hey all anyone else have issue with memory errors during submission?\nMy notebook runs without issue and creates a submission.csv file in less then 12 hour, while processing and predicting train datasets.  But competition submissions fails with memory error. \n\nI check the memory profile of my notebook , peak memory is approximately  26.5 gb during run time so I wonder if there is a memory limit less then that for this competition, I didn't read such restriction in description and proceeded to work thinking I will have access to 30 GB memory but now this submission errors happening I am wondering if there is a limit I am unaware of??\nThere is no fail log, because notebook completes successfully but sum how submission failes... Note I did read actual test data is approximately 80% of train data. So to ensured pipeline won't exceed 30 gb when processing, I tested my pipeline on whole train set on kaggle envorinment and it successfully completed without memory failure so I am very much lost about what can be the issue.",
    "2725425": "\"the hidden test_base.csv contains approximately 90% of the numbers of case_id values of train_base.csv\"\n\nBut for each case id there can be more data than in the train set.",
    "2725428": "Ah ok, I understand thank you 👍",
    "2727001": "Many solutions are here and in the comments https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/486259"
  },
  "source": "meta"
}