{
  "id": 180414,
  "title": "How i handled the notebook timeout",
  "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/180414",
  "author_name": "Mehdi GASMI",
  "post_date": "2020-09-04T22:41:40.114000",
  "votes": 3,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I have 10 submissions with an 8 hour wait time for the notebook while with the local test it took 3 minutes.</p>\n<p>I made several attempts to post sample submissions after each step to see which step my code was bugged at, and after several attempts I noticed that I had poor RAM management due to my data preprocessing pipeline who took 100MB of RAM for the 5 patients and it took over 16GB for the 200 patients.</p>\n<p>To solve this, I applied reduction_mem_usage</p>\n<p>Check this : <a href=\"https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65\" target=\"_blank\">https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65</a></p>\n<p>Which reduced memory usage by 30%.</p>\n<p>and it reduces the sample_submission memory usage by 25%</p>",
  "messages": [
    {
      "id": 998647,
      "postDate": "2020-09-04T22:41:40.113Z",
      "content": "<p>I have 10 submissions with an 8 hour wait time for the notebook while with the local test it took 3 minutes.</p>\n<p>I made several attempts to post sample submissions after each step to see which step my code was bugged at, and after several attempts I noticed that I had poor RAM management due to my data preprocessing pipeline who took 100MB of RAM for the 5 patients and it took over 16GB for the 200 patients.</p>\n<p>To solve this, I applied reduction_mem_usage</p>\n<p>Check this : <a href=\"https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65\" target=\"_blank\">https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65</a></p>\n<p>Which reduced memory usage by 30%.</p>\n<p>and it reduces the sample_submission memory usage by 25%</p>",
      "rawMarkdown": "I have 10 submissions with an 8 hour wait time for the notebook while with the local test it took 3 minutes.\n\nI made several attempts to post sample submissions after each step to see which step my code was bugged at, and after several attempts I noticed that I had poor RAM management due to my data preprocessing pipeline who took 100MB of RAM for the 5 patients and it took over 16GB for the 200 patients.\n\nTo solve this, I applied reduction_mem_usage\n\nCheck this : https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65\n\nWhich reduced memory usage by 30%.\n\nand it reduces the sample_submission memory usage by 25%",
      "votes": 3
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "998647": "I have 10 submissions with an 8 hour wait time for the notebook while with the local test it took 3 minutes.\n\nI made several attempts to post sample submissions after each step to see which step my code was bugged at, and after several attempts I noticed that I had poor RAM management due to my data preprocessing pipeline who took 100MB of RAM for the 5 patients and it took over 16GB for the 200 patients.\n\nTo solve this, I applied reduction_mem_usage\n\nCheck this : https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65\n\nWhich reduced memory usage by 30%.\n\nand it reduces the sample_submission memory usage by 25%"
  }
}