With the nature of the competition we need to find the masks which are of pixel dimensions such as at least 25k by 35k. These type of file consumes almost all the available memory and causes to fail with memory run out error when dealing with them. Even I have tried saving the predicted masks in disks and loading them back, but it doesn't seems to work as well. Can someone tell me is there any workaround for this? Or am I doing something wrong here? Many thanks.
", "messages": [ { "id": "1284296", "postDate": "04/25/2021 19:12:40", "content": "With the nature of the competition we need to find the masks which are of pixel dimensions such as at least 25k by 35k. These type of file consumes almost all the available memory and causes to fail with memory run out error when dealing with them. Even I have tried saving the predicted masks in disks and loading them back, but it doesn't seems to work as well. Can someone tell me is there any workaround for this? Or am I doing something wrong here? Many thanks.
", "rawMarkdown": "With the nature of the competition we need to find the masks which are of pixel dimensions such as at least 25k by 35k. These type of file consumes almost all the available memory and causes to fail with memory run out error when dealing with them. Even I have tried saving the predicted masks in disks and loading them back, but it doesn't seems to work as well. Can someone tell me is there any workaround for this? Or am I doing something wrong here? Many thanks.", "votes": null }, { "id": "1285819", "postDate": "04/27/2021 09:34:19", "content": "Use uint8 mask
\npreds = np.zeros(dataset.shape, dtype=np.uint8)
\nFloat mask is too big for whole image
Will this also help when reading large tiff files from the HDD?
", "rawMarkdown": "Will this also help when reading large tiff files from the HDD?", "votes": null }, { "id": "1286124", "postDate": "04/27/2021 15:35:50", "content": "No. Use rasterio and read data with the window
", "rawMarkdown": "No. Use rasterio and read data with the window", "votes": null }, { "id": "1295137", "postDate": "05/06/2021 08:22:23", "content": "You can use also zarr library. It is storing arrays in memory chunked and compressed.
", "rawMarkdown": "You can use also zarr library. It is storing arrays in memory chunked and compressed.", "votes": null }, { "id": "1295263", "postDate": "05/06/2021 10:43:20", "content": "Thanks for the advice, but if we use it, then it need to take special actions to install that library when we submit the notebook because internet is disabled while submitting.
", "rawMarkdown": "Thanks for the advice, but if we use it, then it need to take special actions to install that library when we submit the notebook because internet is disabled while submitting.", "votes": null }, { "id": "1295265", "postDate": "05/06/2021 10:45:49", "content": "I tried using it. But it seems every time the tiff files are open with only single band.
\ndataset = rasterio.open(\"filename.tiff\")
\ndataset.count ==> gives 1, not 3
I've not used rasterio before, so may be Im doing something wrong here, but couldn't figure it out.
", "rawMarkdown": "I tried using it. But it seems every time the tiff files are open with only single band.\n\ndataset = rasterio.open(\"filename.tiff\")\ndataset.count ==> gives 1, not 3\n\nI've not used rasterio before, so may be Im doing something wrong here, but couldn't figure it out.", "votes": null }, { "id": "1295291", "postDate": "05/06/2021 10:59:55", "content": "Look at open notebooks - there is an example with similar code somewhere
\n`for i, filename in tqdm(enumerate(p.glob('test/.tiff')), \n total = len(list(p.glob('test/.tiff')))):
\nprint(f'{i+1} Predicting {filename.stem}')\ndataset = rasterio.open(filename.as_posix(), transform = identity)\nslices = make_grid(dataset.shape, window=WINDOW, min_overlap=MIN_OVERLAP)\npreds = np.zeros(dataset.shape, dtype=np.uint8)\nif dataset.count != 3:\n print('Image file with subdatasets as channels')\n layers = [rasterio.open(subd) for subd in dataset.subdatasets]\npbar = tqdm(total=len(slices))\nfor (x1,x2,y1,y2) in slices: \n if dataset.count == 3:\n image = dataset.read([1,2,3],\n window=Window.from_slices((x1,x2),(y1,y2)))\n image = np.moveaxis(image, 0, -1)\n else:\n image = np.zeros((WINDOW, WINDOW, 3), dtype=np.uint8)\n for fl in range(3):\n image[:,:,fl] = layers[fl].read(window=Window.from_slices((x1,x2),(y1,y2))) \n image = cv2.resize(image, (NEW_SIZE, NEW_SIZE),interpolation = cv2.INTER_AREA) \n image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) \n hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)\n h_, s, v = cv2.split(hsv)\n if (s>s_th).sum() <= p_th or image.sum() <= p_th: \n pbar.update(1)\n continue \n image2 = np.zeros(image.shape, dtype=np.uint8)\n np.copyto(image2,image)\n image = np.expand_dims(image, 0)\n\n pred = None\n\n for fold_model in fold_models:\n if pred is None:\n pred = np.squeeze(fold_model.predict(image))\n else:\n pred += np.squeeze(fold_model.predict(image))\n\n pred = pred/(len(fold_models))\n\n pred = cv2.resize(pred, (WINDOW, WINDOW))\n\n preds[x1:x2,y1:y2] |= mask_filter2((pred > 0.5).astype(np.uint8)) \n pbar.update(1)\nsubm[i] = {'id':filename.stem, 'predicted': rle_numba_encode(preds)}\nprint(np.sum(preds))\ndel preds\ngc.collect(); `\nWell…. a big thanks for this. I just did some work with this and it seems to be working. Appreciate the time taken to share your knowledge. As you mentioned I did search and found following kernel, but your part of the code should be added in to there. I'm just sharing it hoping that someone may need it.
\nhttps://www.kaggle.com/alinaherderich/submisson-alina-resnet34-v2
", "rawMarkdown": "Well.... a big thanks for this. I just did some work with this and it seems to be working. Appreciate the time taken to share your knowledge. As you mentioned I did search and found following kernel, but your part of the code should be added in to there. I'm just sharing it hoping that someone may need it.\n\nhttps://www.kaggle.com/alinaherderich/submisson-alina-resnet34-v2", "votes": null } ], "comments": [ { "id": 1285819, "author_name": "aleksandrkruchinin", "author_url": "", "post_date": "04/27/2021 09:34:19", "content": "Use uint8 mask
\npreds = np.zeros(dataset.shape, dtype=np.uint8)
\nFloat mask is too big for whole image
Will this also help when reading large tiff files from the HDD?
", "votes": null, "replies": [] }, { "id": 1286124, "author_name": "aleksandrkruchinin", "author_url": "", "post_date": "04/27/2021 15:35:50", "content": "No. Use rasterio and read data with the window
", "votes": null, "replies": [] }, { "id": 1295265, "author_name": "gssdatamanager", "author_url": "", "post_date": "05/06/2021 10:45:49", "content": "I tried using it. But it seems every time the tiff files are open with only single band.
\ndataset = rasterio.open(\"filename.tiff\")
\ndataset.count ==> gives 1, not 3
I've not used rasterio before, so may be Im doing something wrong here, but couldn't figure it out.
", "votes": null, "replies": [] }, { "id": 1295291, "author_name": "aleksandrkruchinin", "author_url": "", "post_date": "05/06/2021 10:59:55", "content": "Look at open notebooks - there is an example with similar code somewhere
\n`for i, filename in tqdm(enumerate(p.glob('test/.tiff')), \n total = len(list(p.glob('test/.tiff')))):
\nprint(f'{i+1} Predicting {filename.stem}')\ndataset = rasterio.open(filename.as_posix(), transform = identity)\nslices = make_grid(dataset.shape, window=WINDOW, min_overlap=MIN_OVERLAP)\npreds = np.zeros(dataset.shape, dtype=np.uint8)\nif dataset.count != 3:\n print('Image file with subdatasets as channels')\n layers = [rasterio.open(subd) for subd in dataset.subdatasets]\npbar = tqdm(total=len(slices))\nfor (x1,x2,y1,y2) in slices: \n if dataset.count == 3:\n image = dataset.read([1,2,3],\n window=Window.from_slices((x1,x2),(y1,y2)))\n image = np.moveaxis(image, 0, -1)\n else:\n image = np.zeros((WINDOW, WINDOW, 3), dtype=np.uint8)\n for fl in range(3):\n image[:,:,fl] = layers[fl].read(window=Window.from_slices((x1,x2),(y1,y2))) \n image = cv2.resize(image, (NEW_SIZE, NEW_SIZE),interpolation = cv2.INTER_AREA) \n image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) \n hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)\n h_, s, v = cv2.split(hsv)\n if (s>s_th).sum() <= p_th or image.sum() <= p_th: \n pbar.update(1)\n continue \n image2 = np.zeros(image.shape, dtype=np.uint8)\n np.copyto(image2,image)\n image = np.expand_dims(image, 0)\n\n pred = None\n\n for fold_model in fold_models:\n if pred is None:\n pred = np.squeeze(fold_model.predict(image))\n else:\n pred += np.squeeze(fold_model.predict(image))\n\n pred = pred/(len(fold_models))\n\n pred = cv2.resize(pred, (WINDOW, WINDOW))\n\n preds[x1:x2,y1:y2] |= mask_filter2((pred > 0.5).astype(np.uint8)) \n pbar.update(1)\nsubm[i] = {'id':filename.stem, 'predicted': rle_numba_encode(preds)}\nprint(np.sum(preds))\ndel preds\ngc.collect(); `\nWell…. a big thanks for this. I just did some work with this and it seems to be working. Appreciate the time taken to share your knowledge. As you mentioned I did search and found following kernel, but your part of the code should be added in to there. I'm just sharing it hoping that someone may need it.
\nhttps://www.kaggle.com/alinaherderich/submisson-alina-resnet34-v2
", "votes": null, "replies": [] } ] }, { "id": 1295137, "author_name": "trytolose", "author_url": "", "post_date": "05/06/2021 08:22:23", "content": "You can use also zarr library. It is storing arrays in memory chunked and compressed.
", "votes": null, "replies": [ { "id": 1295263, "author_name": "gssdatamanager", "author_url": "", "post_date": "05/06/2021 10:43:20", "content": "Thanks for the advice, but if we use it, then it need to take special actions to install that library when we submit the notebook because internet is disabled while submitting.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1284296": "With the nature of the competition we need to find the masks which are of pixel dimensions such as at least 25k by 35k. These type of file consumes almost all the available memory and causes to fail with memory run out error when dealing with them. Even I have tried saving the predicted masks in disks and loading them back, but it doesn't seems to work as well. Can someone tell me is there any workaround for this? Or am I doing something wrong here? Many thanks.", "1285819": "Use uint8 mask \n`preds = np.zeros(dataset.shape, dtype=np.uint8)`\nFloat mask is too big for whole image", "1285922": "Will this also help when reading large tiff files from the HDD?", "1286124": "No. Use rasterio and read data with the window", "1295137": "You can use also zarr library. It is storing arrays in memory chunked and compressed.", "1295263": "Thanks for the advice, but if we use it, then it need to take special actions to install that library when we submit the notebook because internet is disabled while submitting.", "1295265": "I tried using it. But it seems every time the tiff files are open with only single band.\n\ndataset = rasterio.open(\"filename.tiff\")\ndataset.count ==> gives 1, not 3\n\nI've not used rasterio before, so may be Im doing something wrong here, but couldn't figure it out.", "1295291": "Look at open notebooks - there is an example with similar code somewhere\n\n`for i, filename in tqdm(enumerate(p.glob('test/*.tiff')), \n total = len(list(p.glob('test/*.tiff')))):\n \n print(f'{i+1} Predicting {filename.stem}')\n dataset = rasterio.open(filename.as_posix(), transform = identity)\n slices = make_grid(dataset.shape, window=WINDOW, min_overlap=MIN_OVERLAP)\n preds = np.zeros(dataset.shape, dtype=np.uint8)\n if dataset.count != 3:\n print('Image file with subdatasets as channels')\n layers = [rasterio.open(subd) for subd in dataset.subdatasets]\n pbar = tqdm(total=len(slices))\n for (x1,x2,y1,y2) in slices: \n if dataset.count == 3:\n image = dataset.read([1,2,3],\n window=Window.from_slices((x1,x2),(y1,y2)))\n image = np.moveaxis(image, 0, -1)\n else:\n image = np.zeros((WINDOW, WINDOW, 3), dtype=np.uint8)\n for fl in range(3):\n image[:,:,fl] = layers[fl].read(window=Window.from_slices((x1,x2),(y1,y2))) \n image = cv2.resize(image, (NEW_SIZE, NEW_SIZE),interpolation = cv2.INTER_AREA) \n image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) \n hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)\n h_, s, v = cv2.split(hsv)\n if (s>s_th).sum() <= p_th or image.sum() <= p_th: \n pbar.update(1)\n continue \n image2 = np.zeros(image.shape, dtype=np.uint8)\n np.copyto(image2,image)\n image = np.expand_dims(image, 0)\n \n pred = None\n \n for fold_model in fold_models:\n if pred is None:\n pred = np.squeeze(fold_model.predict(image))\n else:\n pred += np.squeeze(fold_model.predict(image))\n \n pred = pred/(len(fold_models))\n \n pred = cv2.resize(pred, (WINDOW, WINDOW))\n\n preds[x1:x2,y1:y2] |= mask_filter2((pred > 0.5).astype(np.uint8)) \n pbar.update(1)\n subm[i] = {'id':filename.stem, 'predicted': rle_numba_encode(preds)}\n print(np.sum(preds))\n del preds\n gc.collect(); `", "1295831": "Well.... a big thanks for this. I just did some work with this and it seems to be working. Appreciate the time taken to share your knowledge. As you mentioned I did search and found following kernel, but your part of the code should be added in to there. I'm just sharing it hoping that someone may need it.\n\nhttps://www.kaggle.com/alinaherderich/submisson-alina-resnet34-v2" }, "source": "meta" }