{
  "id": 199404,
  "title": "How to Run Prediction on Private Test Set Without OOM Error",
  "url": "/competitions/hubmap-kidney-segmentation/discussion/199404",
  "author_name": "",
  "post_date": "2020-11-25T16:24:13.024387500Z",
  "votes": 3,
  "comment_count": 7,
  "views": 0,
  "content": "<p>My submission script always ends up with OOM error. It seems like tifffile only supports whole-silde image reading, which results in large memory usage. OpenSlide, which supports efficient patch reading, runs into error even when reading public test images. How do you solve the memory issue running on the private test set?</p>",
  "messages": [
    {
      "id": "1090867",
      "postDate": "11/25/2020 16:24:13",
      "content": "<p>My submission script always ends up with OOM error. It seems like tifffile only supports whole-silde image reading, which results in large memory usage. OpenSlide, which supports efficient patch reading, runs into error even when reading public test images. How do you solve the memory issue running on the private test set?</p>",
      "rawMarkdown": "My submission script always ends up with OOM error. It seems like tifffile only supports whole-silde image reading, which results in large memory usage. OpenSlide, which supports efficient patch reading, runs into error even when reading public test images. How do you solve the memory issue running on the private test set?",
      "votes": null
    },
    {
      "id": "1090908",
      "postDate": "11/25/2020 17:05:20",
      "content": "<p><a href=\"https://www.kaggle.com/kaimingk\" target=\"_blank\">@kaimingk</a> For me private test submission completes in about 40 mins using GPU.</p>",
      "rawMarkdown": "kaimingk For me private test submission completes in about 40 mins using GPU.",
      "votes": null
    },
    {
      "id": "1090947",
      "postDate": "11/25/2020 17:26:20",
      "content": "<p>Maybe you aren't using GPU?</p>",
      "rawMarkdown": "Maybe you aren't using GPU?",
      "votes": null
    },
    {
      "id": "1091409",
      "postDate": "11/26/2020 02:03:56",
      "content": "<p>I did turn the GPU on, but the notebook was still timed out.</p>",
      "rawMarkdown": "I did turn the GPU on, but the notebook was still timed out.",
      "votes": null
    },
    {
      "id": "1091412",
      "postDate": "11/26/2020 02:08:18",
      "content": "<p>Thanks for the reference. I think it might be that I read the image in a whole slide fashion and the memory usage slows down the program.</p>",
      "rawMarkdown": "Thanks for the reference. I think it might be that I read the image in a whole slide fashion and the memory usage slows down the program.",
      "votes": null
    },
    {
      "id": "1091633",
      "postDate": "11/26/2020 06:38:00",
      "content": "<p>Maybe the kernel got a 'OOM' error.</p>",
      "rawMarkdown": "Maybe the kernel got a 'OOM' error.",
      "votes": null
    },
    {
      "id": "1091854",
      "postDate": "11/26/2020 10:43:53",
      "content": "<p>Looks like so. I just submitted a random prediction without any file reading and it succeeded.</p>",
      "rawMarkdown": "Looks like so. I just submitted a random prediction without any file reading and it succeeded.",
      "votes": null
    },
    {
      "id": "1102323",
      "postDate": "12/04/2020 19:45:02",
      "content": "<p>I was getting OOM errors as well, and I solved it using the disk space as working area, as described in this notebook. <br>\n<a href=\"https://www.kaggle.com/marcosnovaes/hubmap-memory-efficient-submission-using-disk/\" target=\"_blank\">https://www.kaggle.com/marcosnovaes/hubmap-memory-efficient-submission-using-disk/</a></p>\n<p>Hope it helps!</p>\n<p>--Marcos</p>",
      "rawMarkdown": "I was getting OOM errors as well, and I solved it using the disk space as working area, as described in this notebook. \n[https://www.kaggle.com/marcosnovaes/hubmap-memory-efficient-submission-using-disk/](https://www.kaggle.com/marcosnovaes/hubmap-memory-efficient-submission-using-disk/)\n\nHope it helps!\n\n--Marcos",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1090908,
      "author_name": "yovinyahathugoda",
      "author_url": "",
      "post_date": "11/25/2020 17:05:20",
      "content": "<p><a href=\"https://www.kaggle.com/kaimingk\" target=\"_blank\">@kaimingk</a> For me private test submission completes in about 40 mins using GPU.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1091412,
          "author_name": "kaimingk",
          "author_url": "",
          "post_date": "11/26/2020 02:08:18",
          "content": "<p>Thanks for the reference. I think it might be that I read the image in a whole slide fashion and the memory usage slows down the program.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1090947,
      "author_name": "matthewmasters",
      "author_url": "",
      "post_date": "11/25/2020 17:26:20",
      "content": "<p>Maybe you aren't using GPU?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1091409,
          "author_name": "kaimingk",
          "author_url": "",
          "post_date": "11/26/2020 02:03:56",
          "content": "<p>I did turn the GPU on, but the notebook was still timed out.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1091633,
      "author_name": "scp173",
      "author_url": "",
      "post_date": "11/26/2020 06:38:00",
      "content": "<p>Maybe the kernel got a 'OOM' error.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1091854,
          "author_name": "kaimingk",
          "author_url": "",
          "post_date": "11/26/2020 10:43:53",
          "content": "<p>Looks like so. I just submitted a random prediction without any file reading and it succeeded.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1102323,
      "author_name": "marcosnovaes",
      "author_url": "",
      "post_date": "12/04/2020 19:45:02",
      "content": "<p>I was getting OOM errors as well, and I solved it using the disk space as working area, as described in this notebook. <br>\n<a href=\"https://www.kaggle.com/marcosnovaes/hubmap-memory-efficient-submission-using-disk/\" target=\"_blank\">https://www.kaggle.com/marcosnovaes/hubmap-memory-efficient-submission-using-disk/</a></p>\n<p>Hope it helps!</p>\n<p>--Marcos</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1090867": "My submission script always ends up with OOM error. It seems like tifffile only supports whole-silde image reading, which results in large memory usage. OpenSlide, which supports efficient patch reading, runs into error even when reading public test images. How do you solve the memory issue running on the private test set?",
    "1090908": "kaimingk For me private test submission completes in about 40 mins using GPU.",
    "1090947": "Maybe you aren't using GPU?",
    "1091409": "I did turn the GPU on, but the notebook was still timed out.",
    "1091412": "Thanks for the reference. I think it might be that I read the image in a whole slide fashion and the memory usage slows down the program.",
    "1091633": "Maybe the kernel got a 'OOM' error.",
    "1091854": "Looks like so. I just submitted a random prediction without any file reading and it succeeded.",
    "1102323": "I was getting OOM errors as well, and I solved it using the disk space as working area, as described in this notebook. \n[https://www.kaggle.com/marcosnovaes/hubmap-memory-efficient-submission-using-disk/](https://www.kaggle.com/marcosnovaes/hubmap-memory-efficient-submission-using-disk/)\n\nHope it helps!\n\n--Marcos"
  },
  "source": "meta"
}