{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"\n What it is?\n
And actually this is my private notebook for the competition. I trained models and came up with ideas, but then I had less time, and more people who have some problems with submitting results. Therefore, I decided to publish this work specifically for those who wrote comments about the problems under my works. I recommend having two notebooks. One for training and the second for sending results. This notebook is an example for submitting results. I hope this helps.
\n
","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"\n Constants\n
","metadata":{}},{"cell_type":"code","source":"# Files \n__SAMPLE_SBMISSION_PATH = \"/kaggle/input/hubmap-hacking-the-human-vasculature/sample_submission.csv\"\n__TILE_META_PATH = \"/kaggle/input/hubmap-hacking-the-human-vasculature/tile_meta.csv\"\n__WSI_META_PATH = \"/kaggle/input/hubmap-hacking-the-human-vasculature/wsi_meta.csv\"\n__ANNOTATION_PATH = \"/kaggle/input/hubmap-hacking-the-human-vasculature/polygons.jsonl\"\n__BEST_MODEL_PATH = \"/kaggle/input/hubmap-models/mobilenet_v2_UnetPlusPlus/epoch=2_val_loss=0.20_val_accuracy=0.96.ckpt\"\n\n# Folders\n__TRAIN_PATH = \"/kaggle/input/hubmap-hacking-the-human-vasculature/train\"\n__TEST_PATH = \"/kaggle/input/hubmap-hacking-the-human-vasculature/test\"\nMODELS_DIRPATH = \"/kaggle/input/hubmap-models/\"","metadata":{"execution":{"iopub.status.busy":"2023-06-25T23:32:39.286436Z","iopub.execute_input":"2023-06-25T23:32:39.287417Z","iopub.status.idle":"2023-06-25T23:32:39.300575Z","shell.execute_reply.started":"2023-06-25T23:32:39.287372Z","shell.execute_reply":"2023-06-25T23:32:39.299654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n And how can I use the right tools if the internet is banned in the competition? 🙁\n
In the competition, many have difficulty installing the necessary tools and using them, because the Internet is prohibited, but it is prohibited only so that the test data is not stolen (they are uploaded to the test folder during the submission of the result). This means that you can use the tools you want, but you must add them as data. You can find my ultralytics and pycocotools dataset here. Importantly, the Internet is still present in this notebook to load the model, however, after training it, you will save the model, add it to the data and easily use it for forecasting without the Internet.
\n
","metadata":{}},{"cell_type":"code","source":"import shutil\nimport os\nimport sys\nfrom colorama import Fore","metadata":{"execution":{"iopub.status.busy":"2023-06-25T23:32:39.302676Z","iopub.execute_input":"2023-06-25T23:32:39.303251Z","iopub.status.idle":"2023-06-25T23:32:39.313617Z","shell.execute_reply.started":"2023-06-25T23:32:39.303216Z","shell.execute_reply":"2023-06-25T23:32:39.31275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class SetupPipline:\n def __init__(self, display: bool = True):\n self.pycocotools = self.__pycocotools()\n self.ultralytics = self.__ultralytics()\n \n @staticmethod\n def __ultralytics() -> str:\n sys.path.append(\"/kaggle/input/hubmap-tools-ultralytics-and-pycocotools/ultralytics/ultralytics\") \n return \"successfully\"\n \n @staticmethod\n def __pycocotools() -> str:\n if not os.path.exists(\"/kaggle/working/packages\"):\n shutil.copytree(\"/kaggle/input/hubmap-tools-ultralytics-and-pycocotools/pycocotools/pycocotools\", \"/kaggle/working/packages\")\n os.chdir(\"/kaggle/working/packages/pycocotools-2.0.6/\")\n os.system(\"python setup.py install\")\n os.system(\"pip install . --no-index --find-links /kaggle/working/packages/\")\n os.chdir(\"/kaggle/working\")\n return \"successfully\"\n \n def display(self) -> None:\n print(Fore.GREEN+f\"\\nPycocotools was installed {self.pycocotools}\")\n print(f\"Ultralytics was installed {self.ultralytics}\"+Fore.WHITE)","metadata":{"execution":{"iopub.status.busy":"2023-06-25T23:32:39.315423Z","iopub.execute_input":"2023-06-25T23:32:39.316321Z","iopub.status.idle":"2023-06-25T23:32:39.325104Z","shell.execute_reply.started":"2023-06-25T23:32:39.316288Z","shell.execute_reply":"2023-06-25T23:32:39.324248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pipline = SetupPipline()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-06-25T23:32:39.328436Z","iopub.execute_input":"2023-06-25T23:32:39.328996Z","iopub.status.idle":"2023-06-25T23:33:27.183318Z","shell.execute_reply.started":"2023-06-25T23:32:39.328963Z","shell.execute_reply":"2023-06-25T23:33:27.182351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pipline.display()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n Encode Binary Mask\n
","metadata":{}},{"cell_type":"code","source":"import base64\nimport numpy as np\nimport torch\nfrom pycocotools import _mask as coco_mask\nimport typing as t\nimport zlib\nimport pandas as pd\nimport torchvision.transforms as T\nfrom ultralytics import YOLO\nfrom PIL import Image","metadata":{"execution":{"iopub.status.busy":"2023-06-25T23:33:27.184718Z","iopub.execute_input":"2023-06-25T23:33:27.185152Z","iopub.status.idle":"2023-06-25T23:33:39.396572Z","shell.execute_reply.started":"2023-06-25T23:33:27.185116Z","shell.execute_reply":"2023-06-25T23:33:39.395575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class EncodeBinaryMask:\n @staticmethod\n def __checking_mask(mask: np.ndarray) -> np.ndarray:\n if mask.dtype != np.bool:\n raise ValueError(\n \"expects a binary mask, received dtype == %s\" %\n mask.dtype\n )\n return mask\n\n @staticmethod\n def __convert_mask(mask: np.ndarray):\n mask_to_encode = mask.astype(np.uint8)\n mask_to_encode = np.asfortranarray(mask_to_encode)\n return mask_to_encode\n\n @staticmethod\n def __compress_encode(encoded_mask) -> t.Text:\n binary_str = zlib.compress(encoded_mask, zlib.Z_BEST_COMPRESSION)\n base64_str = base64.b64encode(binary_str)\n return base64_str\n\n def __call__(self, mask: np.ndarray) -> t.Text:\n mask = self.__checking_mask(mask)\n mask_to_encode = self.__convert_mask(mask)\n encoded_mask = coco_mask.encode(mask_to_encode)[0][\"counts\"]\n base64_str = self.__compress_encode(encoded_mask)\n return base64_str","metadata":{"execution":{"iopub.status.busy":"2023-06-25T23:33:39.398029Z","iopub.execute_input":"2023-06-25T23:33:39.398684Z","iopub.status.idle":"2023-06-25T23:33:39.407595Z","shell.execute_reply.started":"2023-06-25T23:33:39.398649Z","shell.execute_reply":"2023-06-25T23:33:39.406569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n Submission\n
","metadata":{}},{"cell_type":"code","source":"class Submission:\n def __init__(self, dirpath: str, model: torch.nn.Module):\n self.__eval_transforms = self.get_transforms()\n self.__model = model\n self.__encoder = EncodeBinaryMask()\n self.__dirpath = dirpath\n self.__filenames = os.listdir(dirpath)\n self.height = 512\n self.width = 512\n \n self.__submission_dict = {\n \"id\": [],\n \"height\": [],\n \"width\": [],\n \"prediction_string\": []\n }\n \n self.submission = None\n \n @staticmethod\n def get_transforms():\n return T.Compose([\n T.ToTensor(),\n T.Resize(size=(512, 512)),\n T.Normalize(mean=[0.485, 0.456, 0.406],\n std=[0.229, 0.224, 0.225])\n ])\n\n def __len__(self):\n return len(self.__filenames)\n\n def __get_columns(self) -> None:\n for filename in self.__filenames:\n path = self.__get_image_path(filename)\n masks = self.__forward(path)\n identifier, height, width, prediction_string = self.__get_cells(filename, masks)\n self.__update_columns(identifier, height, width, prediction_string)\n\n def __update_columns(self, identifier: str, height: int, width: int, prediction_string: str) -> None:\n self.__submission_dict[\"id\"].append(identifier)\n self.__submission_dict[\"height\"].append(height)\n self.__submission_dict[\"width\"].append(width)\n self.__submission_dict[\"prediction_string\"].append(prediction_string)\n\n def __get_cells(self, filename: str, masks: list):\n prediction_string = \"\"\n prediction_string = self.__get_prediction_string(masks, prediction_string)\n identifier = filename.split(\".\")[0]\n return identifier, self.height, self.width, prediction_string\n\n def __get_prediction_string(self, masks: list, prediction_string: str) -> str:\n if masks:\n for outputs in masks:\n mask = outputs[\"mask\"]\n mask = np.where(mask > 0.5, 1, 0).astype(np.bool)\n base64_str = self.__encoder(mask)\n confidence = outputs[\"confidence\"]\n prediction_string += f\"0 {confidence} {base64_str.decode('utf-8')} \"\n else:\n return \"\"\n return prediction_string\n\n def __get_image_path(self, filename: str) -> str:\n return os.path.join(\n self.__dirpath, filename\n )\n\n def __get_image(self, path: str) -> torch.Tensor:\n image = Image.open(path)\n image = np.asarray(image)\n image = self.__eval_transforms(image)\n return image\n\n def __forward(self, image: torch.tensor) -> list:\n masks = self.__model(image) \n return masks \n\n def submit(self) -> None:\n if not self.submission:\n self.__get_columns()\n self.submission = pd.DataFrame(self.__submission_dict)\n self.submission = self.submission.set_index('id')\n self.submission.to_csv(\"submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-06-25T23:33:39.410664Z","iopub.execute_input":"2023-06-25T23:33:39.411622Z","iopub.status.idle":"2023-06-25T23:33:39.433592Z","shell.execute_reply.started":"2023-06-25T23:33:39.411587Z","shell.execute_reply":"2023-06-25T23:33:39.432434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n Wrapper class for YOLO model\n
","metadata":{}},{"cell_type":"code","source":"class BestYolo:\n def __init__(self, num: int, conf: float = 0.05):\n self.models_dirpath = MODELS_DIRPATH\n self.model = self.get_model(num)\n self.conf = conf\n \n def get_model(self, num: int) -> YOLO:\n path = os.path.join(self.models_dirpath, f\"{num}/best.pt\")\n return YOLO(path)\n \n def __call__(self, source) -> list[dict, ...]:\n sublist = []\n result = self.model(source)[0]\n if result.masks:\n for i in range(len(result.masks.data)):\n conf = round(float(result.boxes.conf[i]), 2)\n mask = np.expand_dims(result.masks.data[i].cpu().numpy(), axis=0).transpose(1,2,0)\n \n if int(result.boxes.cls[i]) == 0 and conf >= self.conf:\n sublist.append({\"mask\": mask, \"confidence\": conf})\n else:\n continue\n return sublist\n else:\n return None","metadata":{"execution":{"iopub.status.busy":"2023-06-25T23:33:39.435358Z","iopub.execute_input":"2023-06-25T23:33:39.435762Z","iopub.status.idle":"2023-06-25T23:33:39.447676Z","shell.execute_reply.started":"2023-06-25T23:33:39.435728Z","shell.execute_reply":"2023-06-25T23:33:39.446701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n Let's start!\n
","metadata":{}},{"cell_type":"code","source":"model = BestYolo(5)\nsub = Submission(dirpath=__TEST_PATH, model=model)\nsub.submit()","metadata":{"execution":{"iopub.status.busy":"2023-06-25T23:33:39.449298Z","iopub.execute_input":"2023-06-25T23:33:39.449746Z","iopub.status.idle":"2023-06-25T23:33:57.698368Z","shell.execute_reply.started":"2023-06-25T23:33:39.449715Z","shell.execute_reply":"2023-06-25T23:33:57.697419Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub.submission.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n Well, now I think the question will definitely not remain. Is it true? 🙂\n
","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}}]}