the best way is to refer to some submission codes that are successful in giving public score.
\n1) Does the actual test set maintain the same folder structure as shown in the example set?
\n2) Is there a possibility of encountering non-TIFF images in the test set?
\n = glob(f \n f :\n image = cv2.imread(f. ...)\n3) Are the dimensions of the images in the test set relatively similar to those in the training set?
\nfile = glob( \n f file:\n image = cv2.imread(f. ...)\n H,W = image.shape\n H 3x larger than train image:\n notImplementError \n4) Assuming the test set follows the same folder structure, should we anticipate the same number of slices per - subject or kidney as in the training set?
\nnumber of slices should be different.
suggestion:
\nwrite a dummy submission code to predict all mask image.
\nif it pass, then you know how to access the test images.
\nprediction = np.zeros_like(image)\n \ne.g.
\nhttps://www.kaggle.com/code/kashiwaba/sennet-hoa-inference-unet-simple-baseline
DATASET_FOLDER = \nls_images = (os(DATASET_FOLDER, , , , ))\n\nThese are great suggestions. I tried a submission with random 1s and 0s for each image. The code is below. But this failed after 5 hours. Any suggestions?
\n os\n numpy np\n pandas pd\n cv2\n glob glob\n tqdm tqdm\n torch\n torch.nn nn\n torch.utils.data Dataset, DataLoader\n albumentations A\n\n numpy np\n\n ():\n \n random_numbers = np.random.random()\n\n \n p = random_numbers / np.(random_numbers)\n\n \n random_data = np.random.choice([, ], size=img.shape, p=p)\n\n random_data\n\n ():\n \n pixels = img.flatten()\n pixels = np.concatenate([[], pixels, []])\n runs = np.where(pixels[:] != pixels[:-])[] + \n runs[::] -= runs[::]\n rle = .join((x) x runs)\n rle==:\n rle = \n rle\n\n\ntest_images = glob()\ntest_images.sort()\n=[]\nrle=[]\n image test_images:\n name = image.split()[]\n slice_number = image.split()[].split()[]\n rle_encoded = rle_encode(get_mask(cv2.imread(image)))\n .append(name++slice_number)\n rle.append(rle_encoded)\n\ndf = pd.DataFrame({: , : rle})\ndf.to_csv(, index=)\nSo your code here is correct. I Modified 2 things and here is the outcome:
\nSo as noted in other discussions (e.g. https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/455894#2528861 ), your error is most likely caused due to a problem with the metric score needing a lot of memory from the small randomly generated masks.
", "rawMarkdown": "So your code here is correct. I Modified 2 things and here is the outcome:\n1. Fixed rle_encoded to \"1 0\" and the submission competed with a score of 0 in less than 10 minutes.\n1. Fixed rle_encoded to \"1 1\" and the submission competed with a score of 0 in around 30 minutes.\n\nSo as noted in other discussions (e.g. https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/455894#2528861 ), your error is most likely caused due to a problem with the metric score needing a lot of memory from the small randomly generated masks.", "votes": null } ], "comments": [ { "id": 2529603, "author_name": "hengck23", "author_url": "", "post_date": "11/18/2023 11:22:23", "content": "the best way is to refer to some submission codes that are successful in giving public score.
\n1) Does the actual test set maintain the same folder structure as shown in the example set?
\n2) Is there a possibility of encountering non-TIFF images in the test set?
\n = glob(f \n f :\n image = cv2.imread(f. ...)\n3) Are the dimensions of the images in the test set relatively similar to those in the training set?
\nfile = glob( \n f file:\n image = cv2.imread(f. ...)\n H,W = image.shape\n H 3x larger than train image:\n notImplementError \n4) Assuming the test set follows the same folder structure, should we anticipate the same number of slices per - subject or kidney as in the training set?
\nnumber of slices should be different.
suggestion:
\nwrite a dummy submission code to predict all mask image.
\nif it pass, then you know how to access the test images.
\nprediction = np.zeros_like(image)\n \ne.g.
\nhttps://www.kaggle.com/code/kashiwaba/sennet-hoa-inference-unet-simple-baseline
DATASET_FOLDER = \nls_images = (os(DATASET_FOLDER, , , , ))\n\nThese are great suggestions. I tried a submission with random 1s and 0s for each image. The code is below. But this failed after 5 hours. Any suggestions?
\n os\n numpy np\n pandas pd\n cv2\n glob glob\n tqdm tqdm\n torch\n torch.nn nn\n torch.utils.data Dataset, DataLoader\n albumentations A\n\n numpy np\n\n ():\n \n random_numbers = np.random.random()\n\n \n p = random_numbers / np.(random_numbers)\n\n \n random_data = np.random.choice([, ], size=img.shape, p=p)\n\n random_data\n\n ():\n \n pixels = img.flatten()\n pixels = np.concatenate([[], pixels, []])\n runs = np.where(pixels[:] != pixels[:-])[] + \n runs[::] -= runs[::]\n rle = .join((x) x runs)\n rle==:\n rle = \n rle\n\n\ntest_images = glob()\ntest_images.sort()\n=[]\nrle=[]\n image test_images:\n name = image.split()[]\n slice_number = image.split()[].split()[]\n rle_encoded = rle_encode(get_mask(cv2.imread(image)))\n .append(name++slice_number)\n rle.append(rle_encoded)\n\ndf = pd.DataFrame({: , : rle})\ndf.to_csv(, index=)\nSo your code here is correct. I Modified 2 things and here is the outcome:
\nSo as noted in other discussions (e.g. https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/455894#2528861 ), your error is most likely caused due to a problem with the metric score needing a lot of memory from the small randomly generated masks.
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2529372": "1. Does the actual test set maintain the same folder structure as shown in the example set?\n2. Is there a possibility of encountering non-TIFF images in the test set?\n3. Are the dimensions of the images in the test set relatively similar to those in the training set?\n4. Assuming the test set follows the same folder structure, should we anticipate the same number of slices per subject or kidney as in the training set?", "2529603": "the best way is to refer to some submission codes that are successful in giving public score.\n\n\n1) Does the actual test set maintain the same folder structure as shown in the example set?\n- yes\n\n2) Is there a possibility of encountering non-TIFF images in the test set?\n- No. but this is not an issue because we will usually glob for a list of files and use e.g. opencv to read file:\n\n```\nfile = glob(f'{test_folder}\\*' # glob for any format\nfor f in file:\n image = cv2.imread(f. ...)\n\n```\n\n3) Are the dimensions of the images in the test set relatively similar to those in the training set?\n- not the same, but should be similar. you can write submission code to test.\n\n```\nfile = glob(f'{test_folder}\\*' # glob for any format\nfor f in file:\n image = cv2.imread(f. ...)\n H,W = image.shape\n if H is 3x larger than train image:\n raise notImplementError #this will cause submission failure\n\n```\n\n\n4) Assuming the test set follows the same folder structure, should we anticipate the same number of slices per - subject or kidney as in the training set?\nnumber of slices should be different.\n\n----\nsuggestion:\n\nwrite a dummy submission code to predict all mask image.\nif it pass, then you know how to access the test images.\n\n```\n... read all test images ...\nprediction = np.zeros_like(image)\n... encode in rle and make submission csv.\n\n```\n---\n\ne.g.\nhttps://www.kaggle.com/code/kashiwaba/sennet-hoa-inference-unet-simple-baseline\n```\nDATASET_FOLDER = \"/kaggle/input/blood-vessel-segmentation\"\nls_images = glob(os.path.join(DATASET_FOLDER, \"test\", \"*\", \"*\", \"*.tif\"))\nprint(f\"found images: {len(ls_images)}\")\n```", "2530285": "These are great suggestions. I tried a submission with random 1s and 0s for each image. The code is below. But this failed after 5 hours. Any suggestions?\n\n```python\nimport os\nimport numpy as np\nimport pandas as pd\nimport cv2\nfrom glob import glob\nfrom tqdm import tqdm\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset, DataLoader\nimport albumentations as A\n\nimport numpy as np\n\ndef get_mask(img):\n # Generate two random numbers\n random_numbers = np.random.random(2)\n\n # Normalize them to sum up to 1\n p = random_numbers / np.sum(random_numbers)\n\n # Use this distribution in np.random.choice\n random_data = np.random.choice([0, 1], size=img.shape, p=p)\n \n return random_data\n\ndef rle_encode(img):\n '''\n img: numpy array, 1 - mask, 0 - background\n Returns run length as string formated\n '''\n pixels = img.flatten()\n pixels = np.concatenate([[0], pixels, [0]])\n runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n runs[1::2] -= runs[::2]\n rle = ' '.join(str(x) for x in runs)\n if rle=='':\n rle = '1 1'\n return rle\n\n\ntest_images = glob(\"/kaggle/input/blood-vessel-segmentation/test/*/*/*.tif\")\ntest_images.sort()\nid=[]\nrle=[]\nfor image in test_images:\n name = image.split('/')[5]\n slice_number = image.split('/')[7].split('.')[0]\n rle_encoded = rle_encode(get_mask(cv2.imread(image)))\n id.append(name+'_'+slice_number)\n rle.append(rle_encoded)\n\ndf = pd.DataFrame({'id': id, 'rle': rle})\ndf.to_csv('submission.csv', index=False)\n```", "2530565": "So your code here is correct. I Modified 2 things and here is the outcome:\n1. Fixed rle_encoded to \"1 0\" and the submission competed with a score of 0 in less than 10 minutes.\n1. Fixed rle_encoded to \"1 1\" and the submission competed with a score of 0 in around 30 minutes.\n\nSo as noted in other discussions (e.g. https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/455894#2528861 ), your error is most likely caused due to a problem with the metric score needing a lot of memory from the small randomly generated masks." }, "source": "meta" }