{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# <B><u>Whale&Dolphin Repeatedly Cropped Images (1/4)</u></B>","metadata":{}},{"cell_type":"markdown","source":"# <B>[1] Introduction</B>","metadata":{}},{"cell_type":"markdown","source":"There is no annotation data, which shows position of whale and dolphin, in \"Happywhale - Whale and Dolphin Identification\" competition dataset.So annotation data and cropped images are created in the series of the following kernels,\n- [\"Whale&Dolphin Background Removed Images (1/4)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-background-removed-images-1-4),\n- [\"Whale&Dolphin Background Removed Images (2/4)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-background-removed-images-2-4),\n- [\"Whale&Dolphin Background Removed Images (3/4)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-background-removed-images-3-4),\n- [\"Whale&Dolphin Background Removed Images (4/4)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-background-removed-images-4-4),\n- [\"Annotation Data for Detecting Whale&Dolphin\"](https://www.kaggle.com/code/acchiko/annotation-data-for-detecting-whale-dolphin),\n- [\"Whale&Dolphin Cropped Images (1/3)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-cropped-images-1-3),\n- [\"Whale&Dolphin Cropped Images (2/3)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-cropped-images-2-3),\n- [\"Whale&Dolphin Cropped Images (3/3)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-cropped-images-3-3).\n\nBut failed annotation data with large space was created for some images. So same procedure (removing background of image, calculating bounding box, cropping bounding box) is repeated for the cropped images, which were created in the above kernels, for improving the failed annotation data in the series of the following kernels,\n- [\"Whale&Dolphin Repeatedly Cropped Images (1/4)\"](https://www.kaggle.com/acchiko/whale-dolphin-repeatedly-cropped-images-1-4) (This kernel),\n- [\"Whale&Dolphin Repeatedly Cropped Images (2/4)\"](https://www.kaggle.com/acchiko/whale-dolphin-repeatedly-cropped-images-2-4),\n- [\"Whale&Dolphin Repeatedly Cropped Images (3/4)\"](https://www.kaggle.com/acchiko/whale-dolphin-repeatedly-cropped-images-3-4),\n- [\"Whale&Dolphin Repeatedly Cropped Images (4/4)\"](https://www.kaggle.com/acchiko/whale-dolphin-repeatedly-cropped-images-4-4).","metadata":{}},{"cell_type":"markdown","source":"NOTE1 : The author is a beginner of Kaggle/MachineLearning/Python/English. So the kernel may have several bugs/wrongs. I am happy to get your comments. Thank you in advance for your kind advice to make the kernel so NICE! and to make me NICE deep learning guy!! ","metadata":{}},{"cell_type":"markdown","source":"NOTE2 : Utility scripts for visualization of dataset for \"Happywhale - Whale and Dolphin Identification\" competition is defined in my other kernel [\"Utility Functions for Visualization of Dataset\"](https://www.kaggle.com/code/acchiko/utility-functions-for-visualization-of-dataset). The way to create/use utility scripts is summarized in my other kernel [\"How to Create Utility Scripts\"](https://www.kaggle.com/code/acchiko/utility-functions-for-visualization-of-dataset).","metadata":{}},{"cell_type":"markdown","source":"NOTE3 : Dataset for \"Happywhale - Whale and Dolphin Identification\" competition is visualized with [Plotly](https://plotly.com/python/) and [Matplotlib](https://matplotlib.org/) in my other kernel [\"Preview of Whale&Dolphin Dataset with Plotly/Matplotlib\"](https://www.kaggle.com/acchiko/preview-of-whale-dolphin-dataset-with-plotly-matpl). It may help us to get some insight into strategy of training, data augumentation, etc.","metadata":{}},{"cell_type":"markdown","source":"# <B>[2] Preparation of competition dataset</B>","metadata":{}},{"cell_type":"markdown","source":"# [2-1] Loading dataset","metadata":{}},{"cell_type":"markdown","source":"Dataset for \"Happywhale - Whale and Dolphin Identification\" competition is loaded by clicking the following items in the sidebar of kaggle notebook,","metadata":{}},{"cell_type":"markdown","source":"###  \"+ Add data\" -> \"Competition Data\" -> \"Add (Happywhale - Whale and Dolphin Identification)\".","metadata":{}},{"cell_type":"markdown","source":"If it succeeds, the dataset are loaded to the following path.","metadata":{}},{"cell_type":"code","source":"path_to_dir_happywhale_data = \"/kaggle/input/happy-whale-and-dolphin\"\n!ls {path_to_dir_happywhale_data}","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:09.085349Z","iopub.execute_input":"2022-04-10T10:14:09.086379Z","iopub.status.idle":"2022-04-10T10:14:09.877601Z","shell.execute_reply.started":"2022-04-10T10:14:09.086259Z","shell.execute_reply":"2022-04-10T10:14:09.876506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# [2-2] Showing contents of dataset ","metadata":{}},{"cell_type":"markdown","source":"Contents of metadata for train images is shown.","metadata":{}},{"cell_type":"code","source":"path_to_happywhale_train_metadata = \"%s/train.csv\" % path_to_dir_happywhale_data\n!head -5 {path_to_happywhale_train_metadata}","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:09.879723Z","iopub.execute_input":"2022-04-10T10:14:09.880012Z","iopub.status.idle":"2022-04-10T10:14:10.653976Z","shell.execute_reply.started":"2022-04-10T10:14:09.879980Z","shell.execute_reply":"2022-04-10T10:14:10.653187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"List of train/test images are shown.","metadata":{}},{"cell_type":"code","source":"path_to_dir_happywhale_train_images = \"%s/train_images\" % path_to_dir_happywhale_data\n!ls {path_to_dir_happywhale_train_images} | head -5","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:10.655282Z","iopub.execute_input":"2022-04-10T10:14:10.655530Z","iopub.status.idle":"2022-04-10T10:14:12.446304Z","shell.execute_reply.started":"2022-04-10T10:14:10.655500Z","shell.execute_reply":"2022-04-10T10:14:12.444974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path_to_dir_happywhale_test_images = \"%s/test_images\" % path_to_dir_happywhale_data\n!ls {path_to_dir_happywhale_test_images} | head -5","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:12.448862Z","iopub.execute_input":"2022-04-10T10:14:12.449140Z","iopub.status.idle":"2022-04-10T10:14:13.870002Z","shell.execute_reply.started":"2022-04-10T10:14:12.449104Z","shell.execute_reply":"2022-04-10T10:14:13.869235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# [2-3] Creation of metadata for test images","metadata":{}},{"cell_type":"markdown","source":"The metadata for train images exists, but the one for test images is not exists. So the dummy metadata for test images is created.","metadata":{}},{"cell_type":"code","source":"# Creates metadata for test images. \npath_to_happywhale_test_metadata = \"/kaggle/working/test.csv\"\n\n!echo \"image,species,individual_id\" > {path_to_happywhale_test_metadata}\n!ls {path_to_dir_happywhale_test_images} | sed \"s/.jpg/.jpg,unknown,unknown/g\" >> {path_to_happywhale_test_metadata}\n\n# Shows contents of created metadata.\n!head -5 {path_to_happywhale_test_metadata}","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:13.871649Z","iopub.execute_input":"2022-04-10T10:14:13.872109Z","iopub.status.idle":"2022-04-10T10:14:16.218614Z","shell.execute_reply.started":"2022-04-10T10:14:13.872075Z","shell.execute_reply":"2022-04-10T10:14:16.217575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <B>[3] Preparation of annotation data and cropped images</B>","metadata":{}},{"cell_type":"markdown","source":"# [3-1] Loading annotation data and cropped images","metadata":{}},{"cell_type":"markdown","source":"Annotation data is loaded by clicking the following items in the sidebar of kaggle notebook,","metadata":{}},{"cell_type":"markdown","source":"### \"+ Add data\" --> \"Competition Data\" --> \"Add (for the kernel you want to load.)\".","metadata":{}},{"cell_type":"markdown","source":" If it succeeds, metadata with annotation data and cropped images are loaded to the following paths.","metadata":{}},{"cell_type":"code","source":"# Shows list of loaded files.\npath_to_dirs_annotation = [\n    \"/kaggle/input/whale-dolphin-cropped-images-1-3\",\n    \"/kaggle/input/whale-dolphin-cropped-images-2-3\",\n    \"/kaggle/input/whale-dolphin-cropped-images-3-3\",\n]\n\nfor path_to_dir_annotation in path_to_dirs_annotation:\n    !echo \"{path_to_dir_annotation}:\"\n    !ls {path_to_dir_annotation}\n    !echo \"\"","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:16.222523Z","iopub.execute_input":"2022-04-10T10:14:16.222813Z","iopub.status.idle":"2022-04-10T10:14:22.962306Z","shell.execute_reply.started":"2022-04-10T10:14:16.222780Z","shell.execute_reply":"2022-04-10T10:14:22.961232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# [3-2] Showing contents of annotation data and cropped images","metadata":{}},{"cell_type":"markdown","source":"Contents of metadata with annotation data is shown.","metadata":{}},{"cell_type":"code","source":"# Shows contents of metadata with annotation data.\npath_to_train_metadata = \"%s/train_with_annotation.csv\" % path_to_dirs_annotation[0]\n!head -5 {path_to_train_metadata}","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:22.964014Z","iopub.execute_input":"2022-04-10T10:14:22.964302Z","iopub.status.idle":"2022-04-10T10:14:23.741870Z","shell.execute_reply.started":"2022-04-10T10:14:22.964267Z","shell.execute_reply":"2022-04-10T10:14:23.740982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"List of cropped images are shown.","metadata":{}},{"cell_type":"code","source":"# Shows list of cropped images.\npath_to_dirs_cropped_train_images = [\"%s/cropped/train_images\" % path_to_dir_annotation for path_to_dir_annotation in path_to_dirs_annotation]\n\nfor path_to_dir_cropped_train_images in path_to_dirs_cropped_train_images:\n    !echo \"{path_to_dir_cropped_train_images} :\"\n    !ls {path_to_dir_cropped_train_images} | cat -n | tail -5\n    !echo \"\"","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:23.744106Z","iopub.execute_input":"2022-04-10T10:14:23.744807Z","iopub.status.idle":"2022-04-10T10:14:30.969513Z","shell.execute_reply.started":"2022-04-10T10:14:23.744760Z","shell.execute_reply":"2022-04-10T10:14:30.968421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <B>[4] Creation of annotation data</B>","metadata":{}},{"cell_type":"markdown","source":"Annotation data for cropped images is created with the following steps (same procedure as shown in the above series of kernels),\n- Removing background of image,\n- Calculating bounding box,\n- Cropping bounding box,\n- Showing processed images.","metadata":{}},{"cell_type":"markdown","source":"Then, the annotation data is converted for original images.","metadata":{}},{"cell_type":"markdown","source":"It is difficult to process all images in a kernel because of several limitations of kaggle notebook. So it is done in the series of the following kernels,\n- [\"Whale&Dolphin Repeatedly Cropped Images (1/4)\"](https://www.kaggle.com/acchiko/whale-dolphin-repeatedly-cropped-images-1-4) (This kernel),\n- [\"Whale&Dolphin Repeatedly Cropped Images (2/4)\"](https://www.kaggle.com/acchiko/whale-dolphin-repeatedly-cropped-images-2-4),\n- [\"Whale&Dolphin Repeatedly Cropped Images (3/4)\"](https://www.kaggle.com/acchiko/whale-dolphin-repeatedly-cropped-images-3-4),\n- [\"Whale&Dolphin Repeatedly Cropped Images (4/4)\"](https://www.kaggle.com/acchiko/whale-dolphin-repeatedly-cropped-images-4-4).","metadata":{}},{"cell_type":"markdown","source":"# [4-1] Preparation","metadata":{}},{"cell_type":"markdown","source":"## [4-1-1] Creation of working directory","metadata":{}},{"cell_type":"markdown","source":"Working directories for saving processed images(background removed/cropped images) are created.","metadata":{}},{"cell_type":"code","source":"# Creates working directory for saving background removed images.\npath_to_dir_renobg_train_images = \"/kaggle/working/nobg/train_images\"\n!mkdir -p {path_to_dir_renobg_train_images}","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:30.974374Z","iopub.execute_input":"2022-04-10T10:14:30.974687Z","iopub.status.idle":"2022-04-10T10:14:31.725398Z","shell.execute_reply.started":"2022-04-10T10:14:30.974650Z","shell.execute_reply":"2022-04-10T10:14:31.724246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creates working directory for saving cropped images.\npath_to_dir_recropped_train_images = \"/kaggle/working/cropped/train_images\"\n!mkdir -p {path_to_dir_recropped_train_images}","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:31.730294Z","iopub.execute_input":"2022-04-10T10:14:31.730663Z","iopub.status.idle":"2022-04-10T10:14:32.478861Z","shell.execute_reply.started":"2022-04-10T10:14:31.730629Z","shell.execute_reply":"2022-04-10T10:14:32.477581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Shows created directories.\n!ls /kaggle/working/*","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:32.480448Z","iopub.execute_input":"2022-04-10T10:14:32.480709Z","iopub.status.idle":"2022-04-10T10:14:33.233786Z","shell.execute_reply.started":"2022-04-10T10:14:32.480676Z","shell.execute_reply":"2022-04-10T10:14:33.232685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [4-1-2] Loading utility scripts","metadata":{}},{"cell_type":"markdown","source":"Utility scripts for visualization of \"Happywhale - Whale and Dolphin Identification\" competition dataset, which is defined in the other kernel [\"Utility Functions for Visualization of Dataset\"](https://www.kaggle.com/code/acchiko/utility-functions-for-visualization-of-dataset), are loaded. The way to use/create utility scripts is summarized in the other kernel [\"How to Create Utility Scripts\"](https://www.kaggle.com/code/acchiko/how-to-use-create-utility-scripts).","metadata":{}},{"cell_type":"code","source":"import utility_functions_for_visualization_of_dataset as myutils","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:33.235801Z","iopub.execute_input":"2022-04-10T10:14:33.236322Z","iopub.status.idle":"2022-04-10T10:14:33.262864Z","shell.execute_reply.started":"2022-04-10T10:14:33.236283Z","shell.execute_reply":"2022-04-10T10:14:33.262217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [4-1-3] Defining class for processing annotation data and images","metadata":{}},{"cell_type":"markdown","source":"Class for processing annotation data and images is defined.","metadata":{}},{"cell_type":"code","source":"# Imports required libs.\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom PIL import Image, ImageDraw\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport os\nfrom tqdm import tqdm\nfrom sklearn.model_selection import KFold\nimport shutil","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:33.264378Z","iopub.execute_input":"2022-04-10T10:14:33.264618Z","iopub.status.idle":"2022-04-10T10:14:34.461289Z","shell.execute_reply.started":"2022-04-10T10:14:33.264587Z","shell.execute_reply":"2022-04-10T10:14:34.460320Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Defines class for processing metadata and images.\nclass WhaleAndDolphin():\n    def __init__(self, path_to_metadata, \\\n                 path_to_dir_images, \\\n                 path_to_dirs_cropped_images, \\\n                 path_to_dir_renobg_images, \\\n                 path_to_dir_recropped_images):\n        self._path_to_metadata = path_to_metadata\n        self._path_to_dir_images = path_to_dir_images\n        self._path_to_dirs_cropped_images = path_to_dirs_cropped_images\n        self._path_to_dir_renobg_images = path_to_dir_renobg_images\n        self._path_to_dir_recropped_images = path_to_dir_recropped_images\n        \n        # Loads metadata with annotation data to variable \"_metadata_all\"\n        self._metadata_all = pd.read_csv(path_to_metadata)\n        \n        # Adds several columns.\n        if not \"path_to_image\" in self._metadata_all:\n            path_to_images = \\\n                [\"%s/%s\" % (path_to_dir_images, row.image) \\\n                 for row in self._metadata_all.itertuples()]\n            self._metadata_all[\"path_to_image\"] = path_to_images\n            \n        if not \"annotations_xyxy\" in self._metadata_all:\n            annotations_xyxy = \\\n                [[] for row in self._metadata_all.itertuples()]\n            self._metadata_all[\"annotations_xyxy\"] = annotations_xyxy\n        else:\n            annotations_xyxy = \\\n                [eval(str) for str in self._metadata_all[\"annotations_xyxy\"]]\n            self._metadata_all[\"annotations_xyxy\"] = annotations_xyxy\n        \n        if not \"reannotations_xyxy\" in self._metadata_all:\n            reannotations_xyxy = \\\n                [[] for row in self._metadata_all.itertuples()]\n            self._metadata_all[\"reannotations_xyxy\"] = reannotations_xyxy\n        else:\n            reannotations_xyxy = \\\n                [eval(str) for str in self._metadata_all[\"reannotations_xyxy\"]]\n            self._metadata_all[\"reannotations_xyxy\"] = reannotations_xyxy\n            \n        if not \"path_to_nobg_image\" in self._metadata_all:\n            path_to_nobg_images = \\\n                [\"%s/%s\" % (path_to_dir_nobg_images, row.image.replace(\".jpg\", \".png\")) \\\n                 for row in self._metadata_all.itertuples()]\n            self._metadata_all[\"path_to_nobg_image\"] = path_to_nobg_images\n            \n        if not \"path_to_renobg_image\" in self._metadata_all:\n            path_to_renobg_images = \\\n                [\"%s/%s\" % (path_to_dir_renobg_images, row.image.replace(\".jpg\", \".png\")) \\\n                 for row in self._metadata_all.itertuples()]\n            self._metadata_all[\"path_to_renobg_image\"] = path_to_renobg_images\n            \n        if not \"path_to_cropped_image\" in self._metadata_all:\n            path_to_cropped_images = \\\n                [\"%s/%s\" % (path_to_dir_cropped_images, row.image.replace(\".jpg\", \".png\")) \\\n                 for row in self._metadata_all.itertuples()]\n            self._metadata_all[\"path_to_cropped_image\"] = path_to_cropped_images\n        else:\n            path_to_cropped_images = \\\n                [self.findPathToCroppedImage(path_to_dirs_cropped_images, row.image.replace(\".jpg\", \".png\")) \\\n                 for row in self._metadata_all.itertuples()]\n            self._metadata_all[\"path_to_cropped_image\"] = path_to_cropped_images\n            \n        if not \"path_to_recropped_image\" in self._metadata_all:\n            path_to_recropped_images = \\\n                [\"%s/%s\" % (path_to_dir_recropped_images, row.image.replace(\".jpg\", \".png\")) \\\n                 for row in self._metadata_all.itertuples()]\n            self._metadata_all[\"path_to_recropped_image\"] = path_to_recropped_images\n        \n        # Copies the metadata for processing it.\n        self._metadata = self._metadata_all.copy()\n        \n        self._all_species = self.getSpecies()\n        self._all_individual_ids = self.getIndividualIDs()\n        \n    def findPathToCroppedImage(self, path_to_dirs_cropped_images, file_name):\n        for path_to_dir_cropped_images in path_to_dirs_cropped_images:\n            path_to_image = \"%s/%s\" % (path_to_dir_cropped_images, file_name)\n            if os.path.exists(path_to_image):\n                return path_to_image\n\n        return None\n        \n    def resetMetadata(self, initialize=False):\n        if hasattr(self, \"_metadata_tmp\") and not initialize:\n            self._metadata = self._metadata_tmp.copy()\n        else:\n            self._metadata = self._metadata_all.copy()\n            \n    def saveMetadataTemporary(self):\n        self._metadata_tmp = self._metadata.copy()\n        \n    def filterMetadata(self, query=\"index > -1\"):\n        sliced_metadata = \\\n            self._metadata.query(query).reset_index(drop=True)\n        self._metadata = sliced_metadata.copy()\n        \n    def writeMetadata(self, path_to_metadata):\n        self._metadata.to_csv(path_to_metadata, index=False)\n        \n    def getMetadata(self):\n        return self._metadata\n    \n    def getMetadataSize(self):\n        return len(self._metadata)\n    \n    def getSpecies(self):\n        return self._metadata[\"species\"].unique()\n    \n    def getAllSpecies(self):\n        return self._all_species\n    \n    def _species2id(self, species):\n        return np.where(self._all_species == species)\n    \n    def getIndividualIDs(self):\n        return self._metadata[\"individual_id\"].unique()\n    \n    def showImagesTile(self, num_cols=4, draw_annotations=False):\n        metadata = self._metadata\n        titles = [row.image for row in metadata.itertuples()]\n        path_to_images = [row.path_to_image \\\n                          for row in metadata.itertuples()]\n        images = myutils.getImages(path_to_images)\n        if \"annotations_xyxy\" in metadata.columns and draw_annotations:\n            annotations_xyxy_for_images = [row.annotations_xyxy \\\n                                           for row in metadata.itertuples()]\n            texts_for_images = [[\"\" for _ in \\\n                                 range(len(row.annotations_xyxy))] \\\n                                 for row in metadata.itertuples()]\n            myutils.drawAnnotations( \\\n                images, \\\n                annotations_xyxy_for_images=annotations_xyxy_for_images, \\\n                texts_for_images=texts_for_images, \\\n                line_color=\"green\", line_width=3, text_color=\"green\" \\\n            )\n        myutils.showImagesTile(titles, images, num_cols=num_cols)\n        \n    def showProcessedImagesTile(self, num_cols=3, draw_annotations=False):\n        metadata = self._metadata\n        titles, path_to_images = [], []\n        for row in metadata.itertuples():\n            titles.append(\"%s (Org.)\" % row.image)\n            path_to_images.append(row.path_to_image)\n            \n            titles.append(\"%s (Cropped)\" % row.image.replace(\".jpg\", \".png\"))\n            path_to_images.append(row.path_to_cropped_image)\n            \n            titles.append(\"%s (Re-cropped)\" % row.image.replace(\".jpg\", \".png\"))\n            path_to_images.append(row.path_to_recropped_image)\n        \n        images = myutils.getImages(path_to_images)\n        if \"annotations_xyxy\" in metadata.columns and draw_annotations:\n            annotations_xyxy_for_images = [row.annotations_xyxy \\\n                                           for row in metadata.itertuples()]\n            texts_for_images = [[\"\" for _ in \\\n                                 range(len(row.annotations_xyxy))] \\\n                                 for row in metadata.itertuples()]\n            myutils.drawAnnotations( \\\n                images[::3], \\\n                annotations_xyxy_for_images=annotations_xyxy_for_images, \\\n                texts_for_images=texts_for_images, \\\n                line_color=\"green\", line_width=5, text_color=\"green\" \\\n            ) # For only org. images.\n            \n        if \"reannotations_xyxy\" in metadata.columns and draw_annotations:\n            annotations_xyxy_for_images = [row.reannotations_xyxy \\\n                                           for row in metadata.itertuples()]\n            texts_for_images = [[\"\" for _ in \\\n                                 range(len(row.reannotations_xyxy))] \\\n                                 for row in metadata.itertuples()]\n            myutils.drawAnnotations( \\\n                images[::3], \\\n                annotations_xyxy_for_images=annotations_xyxy_for_images, \\\n                texts_for_images=texts_for_images, \\\n                line_color=\"red\", line_width=5, text_color=\"red\" \\\n            ) # For only org. images.\n\n        myutils.showImagesTile(titles, images, num_cols=num_cols)\n        \n    def removeBackground(self, input_column=\"path_to_image\", \\\n                         output_column=\"path_to_nobg_image\"):\n        metadata = self._metadata\n        path_to_inputs = metadata[input_column].tolist()\n        path_to_outputs = metadata[output_column].tolist()\n        \n        for path_to_input, path_to_output in \\\n            zip(path_to_inputs, path_to_outputs):\n            !backgroundremover -i {path_to_input} -o {path_to_output}\n        \n    def calculateAnnotationsXyxy(self, input_column=\"path_to_nobg_image\", \\\n                                 output_column=\"annotations_xyxy\"):\n        batch_size = 100\n        num_batches = len(self._metadata) // batch_size + 1\n        \n        for i_batch in range(num_batches):\n            i_start = i_batch * batch_size\n            i_end = i_start + batch_size\n            metadata = self._metadata.iloc[i_start:i_end]\n            \n            path_to_nobg_images = metadata[input_column].tolist()\n            nobg_images = myutils.getImages(path_to_nobg_images)\n            class_ids = [self._species2id(row.species) for row \\\n                         in metadata.itertuples()]\n            \n            annotations_xyxy = []\n            for nobg_image, class_id in zip(nobg_images, class_ids):\n                _, _, _, a = nobg_image.split()\n                x_min, y_min, x_max, y_max = a.getbbox() # Bounding box of non-zero alpha region\n                confidence = 1.0 # Dummy\n                annotation_xyxy = myutils._annotationXyxy(class_id, \\\n                                                          x_min, y_min, \\\n                                                          x_max, y_max, \\\n                                                          confidence)\n                annotations_xyxy.append([annotation_xyxy])\n                \n            self._metadata[output_column].iloc[i_start:i_end] = \\\n                annotations_xyxy\n            \n    def _species2id(self, species):\n        return np.where(self._all_species == species)[0][0]\n    \n    def cropObject(self, input_column=\"path_to_image\", \\\n                   annotation_column=\"annotations_xyxy\", \\\n                   output_column=\"path_to_cropped_image\"):\n        batch_size = 100\n        num_batches = len(self._metadata) // batch_size + 1\n        \n        for i_batch in range(num_batches):\n            i_start = i_batch * batch_size\n            i_end = i_start + batch_size\n            metadata = self._metadata.iloc[i_start:i_end]\n            \n            path_to_inputs = metadata[input_column].tolist()\n            path_to_outputs = metadata[output_column].tolist()\n            annotations_xyxy = metadata[annotation_column].tolist()\n            \n            for path_to_input, path_to_output, annotations_xyxy in \\\n                zip(path_to_inputs, path_to_outputs, annotations_xyxy):\n                \n                image = Image.open(path_to_input)\n                annotation_xyxy = \\\n                    self._maxConfidenceAnnotation(annotations_xyxy)\n                x_min = annotation_xyxy[\"x_min\"]\n                y_min = annotation_xyxy[\"y_min\"]\n                x_max = annotation_xyxy[\"x_max\"]\n                y_max = annotation_xyxy[\"y_max\"]\n                image_cropped = image.crop((x_min, y_min, x_max, y_max))\n                image_cropped.save(path_to_output)\n            \n    def _maxConfidenceAnnotation(self, annotations_xyxy):\n        confidences = np.array([annotation_xyxy[\"confidence\"] \\\n                                for annotation_xyxy in annotations_xyxy])\n        index = np.argmax(confidences)\n        return annotations_xyxy[index]\n    \n    def convertAnnotation(self, \\\n                          reannotation_column=\"reannotations_xyxy\", \\\n                          annotation_column=\"annotations_xyxy\"):\n        metadata = self._metadata\n        reannotations_xyxy = metadata[reannotation_column].tolist()\n        annotations_xyxy = metadata[annotation_column].tolist()\n        \n        for _reannotations_xyxy, _annotations_xyxy in \\\n            zip(reannotations_xyxy, annotations_xyxy):\n            for reannotation_xyxy, annotation_xyxy in \\\n                zip(_reannotations_xyxy, _annotations_xyxy):\n                dx, dy = annotation_xyxy[\"x_min\"], annotation_xyxy[\"y_min\"]\n                reannotation_xyxy[\"x_min\"] = reannotation_xyxy[\"x_min\"] + dx\n                reannotation_xyxy[\"x_max\"] = reannotation_xyxy[\"x_max\"] + dx\n                reannotation_xyxy[\"y_min\"] = reannotation_xyxy[\"y_min\"] + dy\n                reannotation_xyxy[\"y_max\"] = reannotation_xyxy[\"y_max\"] + dy","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:34.463321Z","iopub.execute_input":"2022-04-10T10:14:34.463650Z","iopub.status.idle":"2022-04-10T10:14:34.557292Z","shell.execute_reply.started":"2022-04-10T10:14:34.463607Z","shell.execute_reply":"2022-04-10T10:14:34.556391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# [4-2] Removing background of image","metadata":{}},{"cell_type":"markdown","source":"Background of the above cropped images are removed with the command line tool [\"backgroundremover\"](https://github.com/nadermx/backgroundremover) to make easy to calculate bounding box.","metadata":{}},{"cell_type":"code","source":"!pip install backgroundremover","metadata":{"execution":{"iopub.status.busy":"2022-04-10T10:14:34.558557Z","iopub.execute_input":"2022-04-10T10:14:34.558790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Loads metadata for train images.\nwhale_and_dolphin = WhaleAndDolphin(\n    path_to_metadata=path_to_train_metadata,\n    path_to_dir_images=path_to_dir_happywhale_train_images,\n    path_to_dirs_cropped_images=path_to_dirs_cropped_train_images,\n    path_to_dir_renobg_images=path_to_dir_renobg_train_images,\n    path_to_dir_recropped_images=path_to_dir_recropped_train_images\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Narrows down number of images.\nnum_images_per_kernel = 3000\ni_start = 0 * num_images_per_kernel\ni_end = i_start + num_images_per_kernel\n\nwhale_and_dolphin.saveMetadataTemporary()\nwhale_and_dolphin.filterMetadata(\n    query=\"%d <= index < %d\" % (i_start, i_end)\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Removes background of cropped images.\nprint(\"Removing background ...\")\nwhale_and_dolphin.removeBackground(\n    input_column=\"path_to_cropped_image\",\n    output_column=\"path_to_renobg_image\"\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If it succeeds, processed images are saved in the path.","metadata":{}},{"cell_type":"code","source":"# Shows list of processed images.\n!echo \"Background removed images:\"\n!ls {path_to_dir_renobg_train_images} | cat -n | tail -5","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# [4-3] Calculating bounding box","metadata":{}},{"cell_type":"markdown","source":"Bounding box are calculated with [\"Image.getbbox()\" of pillow](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.getbbox) (\"Image.getbbox()\" is called in the method \"WhaleAndDolphin.calculateAnnotationsXyxy()\".).","metadata":{}},{"cell_type":"code","source":"# Calculates bounding box.\nprint(\"Calculates bounding box ...\")\nwhale_and_dolphin.calculateAnnotationsXyxy(\n    input_column=\"path_to_renobg_image\",\n    output_column=\"reannotations_xyxy\"\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If it succeeds, position of bounding box is saved as metadata \"reannotations_xyxy\".","metadata":{}},{"cell_type":"code","source":"# Shows processed metadata with annotation.\nmetadata = whale_and_dolphin.getMetadata()\nmetadata","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# [4-4] Cropping bounding box","metadata":{}},{"cell_type":"markdown","source":"Bounding box are cropped with [\"Image.crop()\" of pillow](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.crop) (\"Image.crop()\" is called in the method \"WhaleAndDolphin.cropObject()\".).","metadata":{}},{"cell_type":"code","source":"# Crops bounding box.\nprint(\"Crops bounding box ...\")\nwhale_and_dolphin.cropObject(\n    input_column=\"path_to_cropped_image\",\n    annotation_column=\"reannotations_xyxy\", \n    output_column=\"path_to_recropped_image\"\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If it succeeds, processed images are saved in the path.","metadata":{}},{"cell_type":"code","source":"# Shows list of processed images.\n!echo \"Cropped images:\"\n!ls {path_to_dir_recropped_train_images} | cat -n | tail -5\n!echo \"\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# [4-5] Converting bounding box","metadata":{}},{"cell_type":"markdown","source":"The created bounding box is for cropped images, so it is converted for original images.","metadata":{}},{"cell_type":"code","source":"# Converts coordinate system of bounding box for original images.\nwhale_and_dolphin.convertAnnotation(\n    reannotation_column=\"reannotations_xyxy\",\n    annotation_column=\"annotations_xyxy\"\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If it succeeds, converted position of bounding box is saved as metadata \"reannotations_xyxy\".","metadata":{}},{"cell_type":"code","source":"# Shows processed metadata with annotation.\nmetadata = whale_and_dolphin.getMetadata()\nmetadata","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, metadata with created annotation data is saved as csv file. The metadata is only for processed images (3000 images) in the kernel, so it must be merged later.","metadata":{}},{"cell_type":"code","source":"# Saves metadata as csv file.\npath_to_train_metadata_updated = \"/kaggle/working/train_with_annotation.csv\"\nwhale_and_dolphin.writeMetadata(path_to_train_metadata_updated)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Shows content of created csv file.\n!head -5 {path_to_train_metadata_updated}","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# [4-6] Showing processed annotation data and images","metadata":{}},{"cell_type":"markdown","source":"Processed images (Original/Cropped/Re-cropped images) are shown. Annotation data is drawn in original image. The green one is used for createing center image. The red one is used for creating right image.","metadata":{}},{"cell_type":"code","source":"# Shows processed images.\nnum_images_per_batch = 30\nmetadata_size = whale_and_dolphin.getMetadataSize()\nnum_batches = int(metadata_size / num_images_per_batch) + 1\nnum_cols = 3\n\nwhale_and_dolphin.saveMetadataTemporary()\nfor i in range(num_batches):\n    i_start = i * num_images_per_batch\n    i_end = i_start + num_images_per_batch\n    whale_and_dolphin.filterMetadata(\n        query=\"%d <= index < %d\" % (i_start, i_end)\n    )\n    whale_and_dolphin.showProcessedImagesTile( \\\n        num_cols=num_cols, \\\n        draw_annotations=True \\\n    )\n    whale_and_dolphin.resetMetadata()","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}