{
  "id": 333931,
  "title": "How to prevent a Notebook timeout error?",
  "url": "/competitions/hubmap-organ-segmentation/discussion/333931",
  "author_name": "",
  "post_date": "2022-06-29T02:54:49.436469900Z",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I have trained my model of 256x256 sized tiles taken from the larger dataset images. For inferencing, I resize the image into 2560x2560, divide it into tiles of 10x10 to get a tensor of [100,3,256,256] using Pytorch's Unfold function(it should be more memory efficient that way). </p>\n<p>This is now like feeding a batch size of 100 into the model. And therefore I have skipped using the Dataset and Dataloader classes. By this method, my inferencing is taking more than the alloted time, i.e. 9 hours. <br>\nWhat can be the solution to this problem?</p>\n<p>Thanks!</p>",
  "messages": [
    {
      "id": "1836742",
      "postDate": "06/29/2022 02:54:49",
      "content": "<p>I have trained my model of 256x256 sized tiles taken from the larger dataset images. For inferencing, I resize the image into 2560x2560, divide it into tiles of 10x10 to get a tensor of [100,3,256,256] using Pytorch's Unfold function(it should be more memory efficient that way). </p>\n<p>This is now like feeding a batch size of 100 into the model. And therefore I have skipped using the Dataset and Dataloader classes. By this method, my inferencing is taking more than the alloted time, i.e. 9 hours. <br>\nWhat can be the solution to this problem?</p>\n<p>Thanks!</p>",
      "rawMarkdown": "I have trained my model of 256x256 sized tiles taken from the larger dataset images. For inferencing, I resize the image into 2560x2560, divide it into tiles of 10x10 to get a tensor of [100,3,256,256] using Pytorch's Unfold function(it should be more memory efficient that way). \n\nThis is now like feeding a batch size of 100 into the model. And therefore I have skipped using the Dataset and Dataloader classes. By this method, my inferencing is taking more than the alloted time, i.e. 9 hours. \nWhat can be the solution to this problem?\n\nThanks!",
      "votes": null
    },
    {
      "id": "1836753",
      "postDate": "06/29/2022 03:51:48",
      "content": "<p>Code to get the divide the image into tiles</p>\n<pre><code>def make_grid(idx,crop= False):\n    '''\n    idx = 11\n    img_t = np.transpose(output[idx].to(torch.uint8).numpy(),(1,2,0))\n    print(img_t.dtype)\n    plt.imshow(img_t)\n\n    return a tensor of shape = (no_of_image,3,grid_w,grid_h) and dtype torch.float32\n    '''\n\n    img = io.imread(f\"../input/hubmap-organ-segmentation/test_images/{idx}.tiff\")\n\n    img = transform.resize(img,(2560,2560))\n\n\n    if crop:\n        img = img[250:-250,250:-250,:]\n\n    img_h,img_w = img.shape[0],img.shape[1]\n\n    img = torch.tensor(np.transpose(img,(2,0,1))).unsqueeze(0).to(torch.float32)\n\n    grid_h,grid_w = 256,256\n    number_of_images = (img_h/grid_h)*(img_w/grid_w)\n\n    unfold = torch.nn.Unfold(kernel_size = (grid_h,grid_w),stride = (grid_h,grid_w))\n\n    output = torch.transpose(unfold(img),2,1).reshape([1,int(number_of_images),3,grid_h,grid_w]).squeeze()\n\n    return output\n</code></pre>",
      "rawMarkdown": "Code to get the divide the image into tiles\n```\ndef make_grid(idx,crop= False):\n    '''\n    idx = 11\n    img_t = np.transpose(output[idx].to(torch.uint8).numpy(),(1,2,0))\n    print(img_t.dtype)\n    plt.imshow(img_t)\n    \n    return a tensor of shape = (no_of_image,3,grid_w,grid_h) and dtype torch.float32\n    '''\n\n    img = io.imread(f\"../input/hubmap-organ-segmentation/test_images/{idx}.tiff\")\n\n    img = transform.resize(img,(2560,2560))\n\n    \n    if crop:\n        img = img[250:-250,250:-250,:]\n        \n    img_h,img_w = img.shape[0],img.shape[1]\n    \n    img = torch.tensor(np.transpose(img,(2,0,1))).unsqueeze(0).to(torch.float32)\n\n    grid_h,grid_w = 256,256\n    number_of_images = (img_h/grid_h)*(img_w/grid_w)\n    \n    unfold = torch.nn.Unfold(kernel_size = (grid_h,grid_w),stride = (grid_h,grid_w))\n\n    output = torch.transpose(unfold(img),2,1).reshape([1,int(number_of_images),3,grid_h,grid_w]).squeeze()\n\n    return output\n```",
      "votes": null
    },
    {
      "id": "1837188",
      "postDate": "06/29/2022 12:12:10",
      "content": "<p>Analysis of which package is best for reading .tiff files can be found <a href=\"https://www.kaggle.com/code/nishantbhansali/tiff-file-timeanalysis\" target=\"_blank\">here</a><br>\n(Read -&gt; convert it to tensor)</p>\n<h1>Results</h1>\n<ul>\n<li>skimage :17.06 ms</li>\n<li>tifffile : 17.03 ms</li>\n<li>PIL : 63.5ms</li>\n<li>Rasterio: 234 ms</li>\n<li>Opencv: 111ms</li>\n</ul>",
      "rawMarkdown": "Analysis of which package is best for reading .tiff files can be found [here](https://www.kaggle.com/code/nishantbhansali/tiff-file-timeanalysis)\n(Read -> convert it to tensor)\n# Results\n- skimage :17.06 ms\n- tifffile : 17.03 ms\n- PIL : 63.5ms\n- Rasterio: 234 ms\n- Opencv: 111ms",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1836753,
      "author_name": "nishantbhansali",
      "author_url": "",
      "post_date": "06/29/2022 03:51:48",
      "content": "<p>Code to get the divide the image into tiles</p>\n<pre><code>def make_grid(idx,crop= False):\n    '''\n    idx = 11\n    img_t = np.transpose(output[idx].to(torch.uint8).numpy(),(1,2,0))\n    print(img_t.dtype)\n    plt.imshow(img_t)\n\n    return a tensor of shape = (no_of_image,3,grid_w,grid_h) and dtype torch.float32\n    '''\n\n    img = io.imread(f\"../input/hubmap-organ-segmentation/test_images/{idx}.tiff\")\n\n    img = transform.resize(img,(2560,2560))\n\n\n    if crop:\n        img = img[250:-250,250:-250,:]\n\n    img_h,img_w = img.shape[0],img.shape[1]\n\n    img = torch.tensor(np.transpose(img,(2,0,1))).unsqueeze(0).to(torch.float32)\n\n    grid_h,grid_w = 256,256\n    number_of_images = (img_h/grid_h)*(img_w/grid_w)\n\n    unfold = torch.nn.Unfold(kernel_size = (grid_h,grid_w),stride = (grid_h,grid_w))\n\n    output = torch.transpose(unfold(img),2,1).reshape([1,int(number_of_images),3,grid_h,grid_w]).squeeze()\n\n    return output\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1837188,
      "author_name": "nishantbhansali",
      "author_url": "",
      "post_date": "06/29/2022 12:12:10",
      "content": "<p>Analysis of which package is best for reading .tiff files can be found <a href=\"https://www.kaggle.com/code/nishantbhansali/tiff-file-timeanalysis\" target=\"_blank\">here</a><br>\n(Read -&gt; convert it to tensor)</p>\n<h1>Results</h1>\n<ul>\n<li>skimage :17.06 ms</li>\n<li>tifffile : 17.03 ms</li>\n<li>PIL : 63.5ms</li>\n<li>Rasterio: 234 ms</li>\n<li>Opencv: 111ms</li>\n</ul>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1836742": "I have trained my model of 256x256 sized tiles taken from the larger dataset images. For inferencing, I resize the image into 2560x2560, divide it into tiles of 10x10 to get a tensor of [100,3,256,256] using Pytorch's Unfold function(it should be more memory efficient that way). \n\nThis is now like feeding a batch size of 100 into the model. And therefore I have skipped using the Dataset and Dataloader classes. By this method, my inferencing is taking more than the alloted time, i.e. 9 hours. \nWhat can be the solution to this problem?\n\nThanks!",
    "1836753": "Code to get the divide the image into tiles\n```\ndef make_grid(idx,crop= False):\n    '''\n    idx = 11\n    img_t = np.transpose(output[idx].to(torch.uint8).numpy(),(1,2,0))\n    print(img_t.dtype)\n    plt.imshow(img_t)\n    \n    return a tensor of shape = (no_of_image,3,grid_w,grid_h) and dtype torch.float32\n    '''\n\n    img = io.imread(f\"../input/hubmap-organ-segmentation/test_images/{idx}.tiff\")\n\n    img = transform.resize(img,(2560,2560))\n\n    \n    if crop:\n        img = img[250:-250,250:-250,:]\n        \n    img_h,img_w = img.shape[0],img.shape[1]\n    \n    img = torch.tensor(np.transpose(img,(2,0,1))).unsqueeze(0).to(torch.float32)\n\n    grid_h,grid_w = 256,256\n    number_of_images = (img_h/grid_h)*(img_w/grid_w)\n    \n    unfold = torch.nn.Unfold(kernel_size = (grid_h,grid_w),stride = (grid_h,grid_w))\n\n    output = torch.transpose(unfold(img),2,1).reshape([1,int(number_of_images),3,grid_h,grid_w]).squeeze()\n\n    return output\n```",
    "1837188": "Analysis of which package is best for reading .tiff files can be found [here](https://www.kaggle.com/code/nishantbhansali/tiff-file-timeanalysis)\n(Read -> convert it to tensor)\n# Results\n- skimage :17.06 ms\n- tifffile : 17.03 ms\n- PIL : 63.5ms\n- Rasterio: 234 ms\n- Opencv: 111ms"
  },
  "source": "meta"
}