{"metadata":{"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":61446,"databundleVersionId":6962461,"sourceType":"competition"},{"sourceId":7558821,"sourceType":"datasetVersion","datasetId":4401668},{"sourceId":7564822,"sourceType":"datasetVersion","datasetId":4404786},{"sourceId":7572769,"sourceType":"datasetVersion","datasetId":4408596},{"sourceId":159504505,"sourceType":"kernelVersion"}],"dockerImageVersionId":30636,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.12"},"papermill":{"default_parameters":{},"duration":41.744078,"end_time":"2024-01-18T18:06:14.390261","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-01-18T18:05:32.646183","version":"2.4.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Le notebook suivant montre comment intégrer le post processing dans la fonction prédict afin de limiter la mémoire utilisée.","metadata":{}},{"cell_type":"code","source":"!pip install --no-index --find-links='/kaggle/input/download-smp-ok' segmentation-models-pytorch","metadata":{"papermill":{"duration":20.861386,"end_time":"2024-01-18T18:05:57.260854","exception":false,"start_time":"2024-01-18T18:05:36.399468","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-06T12:48:46.412135Z","iopub.execute_input":"2024-02-06T12:48:46.412423Z","iopub.status.idle":"2024-02-06T12:49:05.715775Z","shell.execute_reply.started":"2024-02-06T12:48:46.412398Z","shell.execute_reply":"2024-02-06T12:49:05.714840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\nfrom torch.utils.data import Dataset\nimport tifffile as tiff\nimport cv2\nimport albumentations as A\nimport numpy as np\nimport pandas as pd\nimport os\nfrom torch.utils.data import DataLoader\nimport segmentation_models_pytorch as smp\n\nimport ssl\nssl._create_default_https_context = ssl._create_unverified_context\n","metadata":{"papermill":{"duration":3.34564,"end_time":"2024-01-18T18:06:07.902438","exception":false,"start_time":"2024-01-18T18:06:04.556798","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-06T12:49:11.459036Z","iopub.execute_input":"2024-02-06T12:49:11.459380Z","iopub.status.idle":"2024-02-06T12:49:18.882396Z","shell.execute_reply.started":"2024-02-06T12:49:11.459351Z","shell.execute_reply":"2024-02-06T12:49:18.881464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"height = 512\nwidth = 512\nsize = (height,width)","metadata":{"execution":{"iopub.status.busy":"2024-02-06T12:50:15.835749Z","iopub.execute_input":"2024-02-06T12:50:15.836630Z","iopub.status.idle":"2024-02-06T12:50:15.840764Z","shell.execute_reply.started":"2024-02-06T12:50:15.836591Z","shell.execute_reply":"2024-02-06T12:50:15.839763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preprocessing image","metadata":{"papermill":{"duration":0.010097,"end_time":"2024-01-18T18:06:07.923417","exception":false,"start_time":"2024-01-18T18:06:07.913320","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def preprocess_image(path):\n    \n    img = cv2.imread(path, cv2.IMREAD_UNCHANGED)\n    img = np.tile(img[...,None],[1, 1, 3]) \n    img = img.astype('float32') \n    mx = np.max(img)\n    if mx:\n        img/=mx #(2**16)\n        \n    img = np.transpose(img, (2, 0, 1))\n    img_ten = torch.tensor(img)\n    return img_ten","metadata":{"papermill":{"duration":0.019208,"end_time":"2024-01-18T18:06:07.952645","exception":false,"start_time":"2024-01-18T18:06:07.933437","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-06T12:50:35.984336Z","iopub.execute_input":"2024-02-06T12:50:35.985139Z","iopub.status.idle":"2024-02-06T12:50:35.992016Z","shell.execute_reply.started":"2024-02-06T12:50:35.985095Z","shell.execute_reply":"2024-02-06T12:50:35.990890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Activate GPU","metadata":{"papermill":{"duration":0.010242,"end_time":"2024-01-18T18:06:08.022542","exception":false,"start_time":"2024-01-18T18:06:08.012300","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def get_default_device():\n    \"\"\"Pick GPU if available, else CPU\"\"\"\n    if torch.cuda.is_available():\n        return torch.device('cuda')\n    else:\n        return torch.device('cpu')\ndevice = get_default_device()\ndevice","metadata":{"papermill":{"duration":0.114646,"end_time":"2024-01-18T18:06:08.148699","exception":false,"start_time":"2024-01-18T18:06:08.034053","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-06T12:50:44.593670Z","iopub.execute_input":"2024-02-06T12:50:44.594070Z","iopub.status.idle":"2024-02-06T12:50:44.649815Z","shell.execute_reply.started":"2024-02-06T12:50:44.594041Z","shell.execute_reply":"2024-02-06T12:50:44.648711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Functions Images without labels","metadata":{"papermill":{"duration":0.010652,"end_time":"2024-01-18T18:06:08.170826","exception":false,"start_time":"2024-01-18T18:06:08.160174","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## DataSet","metadata":{"papermill":{"duration":0.010589,"end_time":"2024-01-18T18:06:08.192538","exception":false,"start_time":"2024-01-18T18:06:08.181949","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class CustomDatasetImageOnly(Dataset):\n    def __init__(self, image_files, input_size=size, augmentation_transforms=None):\n        self.image_files = image_files\n        self.input_size = input_size\n        self.augmentation_transforms = augmentation_transforms\n\n    def __len__(self):\n        return len(self.image_files)\n\n    def __getitem__(self, idx):\n    \n        image_path = self.image_files[idx]\n        image = preprocess_image(image_path)\n\n        if self.augmentation_transforms:\n            image = self.augmentation_transforms(image)\n\n        return image","metadata":{"papermill":{"duration":0.019872,"end_time":"2024-01-18T18:06:08.222659","exception":false,"start_time":"2024-01-18T18:06:08.202787","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-06T12:52:39.457472Z","iopub.execute_input":"2024-02-06T12:52:39.458397Z","iopub.status.idle":"2024-02-06T12:52:39.464848Z","shell.execute_reply.started":"2024-02-06T12:52:39.458360Z","shell.execute_reply":"2024-02-06T12:52:39.463861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Augmentation","metadata":{"papermill":{"duration":0.009885,"end_time":"2024-01-18T18:06:08.242627","exception":false,"start_time":"2024-01-18T18:06:08.232742","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def augment_image_without_mask(image):\n    \n    image_np = image.permute(1, 2, 0).numpy()\n    transform = A.Compose([\n            A.Resize(height,width, interpolation=cv2.INTER_NEAREST)])\n\n    augmented = transform(image=image_np)\n    augmented_image = augmented['image']\n\n    augmented_image = torch.tensor(augmented_image, dtype=torch.float32).permute(2, 0, 1)\n\n    return augmented_image","metadata":{"papermill":{"duration":0.021933,"end_time":"2024-01-18T18:06:08.274872","exception":false,"start_time":"2024-01-18T18:06:08.252939","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-06T12:52:42.679060Z","iopub.execute_input":"2024-02-06T12:52:42.679758Z","iopub.status.idle":"2024-02-06T12:52:42.685359Z","shell.execute_reply.started":"2024-02-06T12:52:42.679724Z","shell.execute_reply":"2024-02-06T12:52:42.684338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def remove_small_objects(img, min_size):\n    # Find all connected components (labels)\n    num_labels, labels, stats, centroids = cv2.connectedComponentsWithStats(img, cv2.CV_32S, connectivity=8)\n\n    # Create a mask where small objects are removed\n    new_img = np.zeros_like(img)\n    for label in range(1, num_labels):\n        if stats[label, cv2.CC_STAT_AREA] >= min_size:\n            new_img[labels == label] = 1\n\n    return new_img","metadata":{"execution":{"iopub.status.busy":"2024-02-06T12:52:43.634099Z","iopub.execute_input":"2024-02-06T12:52:43.634827Z","iopub.status.idle":"2024-02-06T12:52:43.640177Z","shell.execute_reply.started":"2024-02-06T12:52:43.634795Z","shell.execute_reply":"2024-02-06T12:52:43.639280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def resized_images(predicted_image, true_image_size):\n    image_shape = true_image_size\n    img_resized_test = cv2.resize(predicted_image, (true_image_size[1],true_image_size[0]), interpolation = cv2.INTER_NEAREST)\n    return resized_image","metadata":{"execution":{"iopub.status.busy":"2024-02-06T12:52:45.028344Z","iopub.execute_input":"2024-02-06T12:52:45.029308Z","iopub.status.idle":"2024-02-06T12:52:45.033925Z","shell.execute_reply.started":"2024-02-06T12:52:45.029270Z","shell.execute_reply":"2024-02-06T12:52:45.032926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prediction","metadata":{"papermill":{"duration":0.010516,"end_time":"2024-01-18T18:06:08.296054","exception":false,"start_time":"2024-01-18T18:06:08.285538","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def prediction(model, dataloader, image_shape_list, threshold=None):\n    \"\"\" return a list with all predictions of the batch \"\"\"\n    \n    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n    model.to(device)\n\n    model.eval()\n    predictions = []\n\n    for sample, image_shape in zip(iter(dataloader), image_shape_list):\n\n        inputs = sample.to(device)\n        \n        with torch.set_grad_enabled(False):\n            outputs = model(inputs)\n\n            if threshold != None:\n                # Appliquer un seuil aux sorties du modèle\n                binary_predictions = (outputs > threshold)\n\n                y_pred = binary_predictions.data.cpu().numpy().astype(np.uint8)\n                y_pred = np.transpose(y_pred, (0, 2, 3, 1))            \n                y_pred = cv2.resize(y_pred[0,:,:,:], (image_shape[1],image_shape[0]), interpolation = cv2.INTER_NEAREST)\n                y_pred = remove_small_objects(y_pred, min_size=7)\n                \n                predictions.append(y_pred)\n            \n            else:\n\n                y_pred = outputs.data.cpu().numpy().astype(np.uint8)\n                y_pred = np.transpose(y_pred, (0, 2, 3, 1))\n                y_pred = remove_small_objects(y_pred, min_size=10)\n                y_pred = cv2.resize(y_pred, (image_shape[1],image_shape[0]), interpolation = cv2.INTER_NEAREST)\n                predictions.append(y_pred)\n    \n\n    return predictions","metadata":{"papermill":{"duration":0.023039,"end_time":"2024-01-18T18:06:08.330455","exception":false,"start_time":"2024-01-18T18:06:08.307416","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-06T13:00:17.860521Z","iopub.execute_input":"2024-02-06T13:00:17.861446Z","iopub.status.idle":"2024-02-06T13:00:17.872158Z","shell.execute_reply.started":"2024-02-06T13:00:17.861410Z","shell.execute_reply":"2024-02-06T13:00:17.871203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## RLE encoding","metadata":{"papermill":{"duration":0.010693,"end_time":"2024-01-18T18:06:08.352008","exception":false,"start_time":"2024-01-18T18:06:08.341315","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def rle_encode(img):\n    '''\n    img: numpy array, 1 - mask, 0 - background\n    Returns run length as string formated\n    '''\n\n    pixels = img.flatten()\n    pixels = np.concatenate([[0], pixels, [0]])\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n\n    # Format the result as \"id, rle\"\n    result = ' '.join(map(str, runs))\n\n    if result == '':\n        result = '1 0'\n    \n    return result","metadata":{"papermill":{"duration":0.02084,"end_time":"2024-01-18T18:06:08.383455","exception":false,"start_time":"2024-01-18T18:06:08.362615","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-06T12:52:52.099493Z","iopub.execute_input":"2024-02-06T12:52:52.099879Z","iopub.status.idle":"2024-02-06T12:52:52.106367Z","shell.execute_reply.started":"2024-02-06T12:52:52.099851Z","shell.execute_reply":"2024-02-06T12:52:52.105401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Model","metadata":{"papermill":{"duration":0.010429,"end_time":"2024-01-18T18:06:08.404544","exception":false,"start_time":"2024-01-18T18:06:08.394115","status":"completed"},"tags":[]}},{"cell_type":"code","source":"PATH = '/kaggle/input/trained-model-1/trained_model_1.pth'\nENCODER = 'se_resnext50_32x4d'\nACTIVATION = 'sigmoid'\n\nmodel_fpn = smp.Unet(\n    encoder_name=ENCODER, \n    encoder_weights=None, \n    classes=1, \n    activation=ACTIVATION,\n)\n\n# à voir si ajoute ce paramètre : in_channels=3, \n\nmodel_fpn.load_state_dict(torch.load(PATH))\nmodel_fpn.eval()","metadata":{"papermill":{"duration":1.860579,"end_time":"2024-01-18T18:06:10.275816","exception":false,"start_time":"2024-01-18T18:06:08.415237","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-06T12:52:53.267805Z","iopub.execute_input":"2024-02-06T12:52:53.268192Z","iopub.status.idle":"2024-02-06T12:52:55.095951Z","shell.execute_reply.started":"2024-02-06T12:52:53.268162Z","shell.execute_reply":"2024-02-06T12:52:55.095066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prédiction sur le jeu de test","metadata":{}},{"cell_type":"code","source":"TEST_BASE_PATH = '/kaggle/input/blood-vessel-segmentation/test/'\nimage_files_test = []\nimages_sizes_list = []\nfor dirname, _, filenames in os.walk(TEST_BASE_PATH):\n    for filename in filenames:\n        images_path = os.path.join(dirname, filename)\n        image_files_test.append(images_path)\n        image_shape = cv2.imread(images_path, cv2.IMREAD_UNCHANGED).shape\n        images_sizes_list.append(image_shape)","metadata":{"execution":{"iopub.status.busy":"2024-02-06T12:52:57.844463Z","iopub.execute_input":"2024-02-06T12:52:57.845196Z","iopub.status.idle":"2024-02-06T12:52:58.063278Z","shell.execute_reply.started":"2024-02-06T12:52:57.845161Z","shell.execute_reply":"2024-02-06T12:52:58.062448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_dataset = CustomDatasetImageOnly(image_files_test, augmentation_transforms=augment_image_without_mask)\ntest_dataloader = DataLoader(test_dataset, batch_size=1)\nseuil = 0.05\nimg_result = prediction(model_fpn, test_dataloader, images_sizes_list, seuil)","metadata":{"execution":{"iopub.status.busy":"2024-02-06T13:00:22.932724Z","iopub.execute_input":"2024-02-06T13:00:22.933331Z","iopub.status.idle":"2024-02-06T13:00:23.701642Z","shell.execute_reply.started":"2024-02-06T13:00:22.933299Z","shell.execute_reply":"2024-02-06T13:00:23.700250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submissions_data = []\n\nfor i, k in enumerate(range(len(img_result))):\n    rle_result = rle_encode(img_result[i])\n    submissions_data.append({\"id\": i, \"rle\": rle_result})\n\n# Convert the list to a DataFrame\nsubmissions_df = pd.DataFrame(submissions_data)\n\nid_list = []\nfor path in image_files_test:\n    part1_temp = path.split('/images/')[0]\n    part2_temp = path.split('/images/')[1]\n    part1 = part1_temp.split('/')[-1]\n    part2 = part2_temp.split('.tif')[0]\n    id_list.append(f'{part1}_{part2}')\nsubmissions_df[\"id\"] = id_list\n\n# Save the DataFrame to a CSV file\nsubmissions_df.to_csv(\"/kaggle/working/submission.csv\", index=False)\nsubmissions_df","metadata":{"execution":{"iopub.status.busy":"2024-02-06T12:58:31.100623Z","iopub.execute_input":"2024-02-06T12:58:31.100998Z","iopub.status.idle":"2024-02-06T12:58:31.137371Z","shell.execute_reply.started":"2024-02-06T12:58:31.100964Z","shell.execute_reply":"2024-02-06T12:58:31.136429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}