{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# <B><u>Annotation Data for Detecting Whale&Dolphin</u></B>","metadata":{}},{"cell_type":"markdown","source":"# <B>[1] Introduction</B>","metadata":{}},{"cell_type":"markdown","source":"There is no annotation data, which shows position of whale and dolphin, in \"Happywhale - Whale and Dolphin Identification\" competition dataset. It is difficult to create annotation data in a kernel because of limitations of kaggle notebook. So it is created in the series of the following kernels,\n- [\"Whale&Dolphin Background Removed Images (1/4)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-background-removed-images-1-4),\n- [\"Whale&Dolphin Background Removed Images (2/4)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-background-removed-images-2-4),\n- [\"Whale&Dolphin Background Removed Images (3/4)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-background-removed-images-3-4),\n- [\"Whale&Dolphin Background Removed Images (4/4)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-background-removed-images-4-4),\n- [\"Annotation Data for Detecting Whale&Dolphin\"](https://www.kaggle.com/code/acchiko/annotation-data-for-detecting-whale-dolphin) (This kernel),\n- [\"Whale&Dolphin Cropped Images (1/3)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-cropped-images-1-3),\n- [\"Whale&Dolphin Cropped Images (2/3)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-cropped-images-2-3),\n- [\"Whale&Dolphin Cropped Images (3/3)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-cropped-images-3-3).\n\nIf the kernels are loaded, created annotation data and cropped images can be available from the following path for example,\n- Annotation data : /kaggle/input/annotation-data-for-detecting-whale-dolphin/train_with_annotation.csv,\n- Background removed images : /kaggle/input/annotation-data-for-detecting-whale-dolphin/nobg/train_image/*.png,\n- Cropped images : /kaggle/input/whale-dolphin-cropped-images-3-3/cropped/train_image/*.png.","metadata":{}},{"cell_type":"markdown","source":"NOTE1 : The author is a beginner of Kaggle/MachineLearning/Python/English. So the kernel may have several bugs/wrongs. I am happy to get your comments. Thank you in advance for your kind advice to make the kernel so NICE! and to make me NICE deep learning guy!! ","metadata":{}},{"cell_type":"markdown","source":"NOTE2 : Utility scripts for visualization of dataset for \"Happywhale - Whale and Dolphin Identification\" competition is defined in my other kernel [\"Utility Functions for Visualization of Dataset\"](https://www.kaggle.com/code/acchiko/utility-functions-for-visualization-of-dataset). The way to create/use utility scripts is summarized in my other kernel [\"How to Create Utility Scripts\"](https://www.kaggle.com/code/acchiko/utility-functions-for-visualization-of-dataset).","metadata":{}},{"cell_type":"markdown","source":"NOTE3 : Dataset for \"Happywhale - Whale and Dolphin Identification\" competition is visualized with [Plotly](https://plotly.com/python/) and [Matplotlib](https://matplotlib.org/) in my other kernel [\"Preview of Whale&Dolphin Dataset with Plotly/Matplotlib\"](https://www.kaggle.com/acchiko/preview-of-whale-dolphin-dataset-with-plotly-matpl). It may help us to get some insight into strategy of training, data augumentation, etc.","metadata":{}},{"cell_type":"markdown","source":"# <B>[2] Preparation of dataset</B>","metadata":{}},{"cell_type":"markdown","source":"## [2-1] Loading dataset","metadata":{}},{"cell_type":"markdown","source":"Dataset for \"Happywhale - Whale and Dolphin Identification\" competition is loaded by clicking the following items in the sidebar of kaggle notebook,","metadata":{}},{"cell_type":"markdown","source":"###  \"+ Add data\" -> \"Competition Data\" -> \"Add (Happywhale - Whale and Dolphin Identification)\".","metadata":{}},{"cell_type":"markdown","source":"If it succeeds, the dataset are loaded to the following path.","metadata":{}},{"cell_type":"code","source":"path_to_dir_happywhale_data = \"/kaggle/input/happy-whale-and-dolphin\"\n!ls {path_to_dir_happywhale_data}","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:25:17.545546Z","iopub.execute_input":"2022-04-05T05:25:17.546096Z","iopub.status.idle":"2022-04-05T05:25:18.385211Z","shell.execute_reply.started":"2022-04-05T05:25:17.546056Z","shell.execute_reply":"2022-04-05T05:25:18.384348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [2-2] Showing contents of dataset ","metadata":{}},{"cell_type":"markdown","source":"Contents of metadata for train images is shown.","metadata":{}},{"cell_type":"code","source":"path_to_happywhale_train_metadata = \"%s/train.csv\" % path_to_dir_happywhale_data\n!head -5 {path_to_happywhale_train_metadata}","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:25:18.387764Z","iopub.execute_input":"2022-04-05T05:25:18.388136Z","iopub.status.idle":"2022-04-05T05:25:19.205234Z","shell.execute_reply.started":"2022-04-05T05:25:18.388087Z","shell.execute_reply":"2022-04-05T05:25:19.204472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"List of train/test images are shown.","metadata":{}},{"cell_type":"code","source":"path_to_dir_happywhale_train_images = \"%s/train_images\" % path_to_dir_happywhale_data\n!ls {path_to_dir_happywhale_train_images} | head -5","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:25:19.206785Z","iopub.execute_input":"2022-04-05T05:25:19.207055Z","iopub.status.idle":"2022-04-05T05:25:21.683069Z","shell.execute_reply.started":"2022-04-05T05:25:19.207020Z","shell.execute_reply":"2022-04-05T05:25:21.682077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path_to_dir_happywhale_test_images = \"%s/test_images\" % path_to_dir_happywhale_data\n!ls {path_to_dir_happywhale_test_images} | head -5","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:25:21.685871Z","iopub.execute_input":"2022-04-05T05:25:21.686530Z","iopub.status.idle":"2022-04-05T05:25:23.262517Z","shell.execute_reply.started":"2022-04-05T05:25:21.686475Z","shell.execute_reply":"2022-04-05T05:25:23.261432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [2-3] Creation of metadata for test images","metadata":{}},{"cell_type":"markdown","source":"The metadata for train images exists, but the one for test images is not exists. So the dummy metadata for test images is created.","metadata":{}},{"cell_type":"code","source":"# Creates metadata for test images. \npath_to_happywhale_test_metadata = \"/kaggle/working/test.csv\"\n\n!echo \"image,species,individual_id\" > {path_to_happywhale_test_metadata}\n!ls {path_to_dir_happywhale_test_images} | sed \"s/.jpg/.jpg,unknown,unknown/g\" >> {path_to_happywhale_test_metadata}\n\n# Shows contents of created metadata.\n!head -5 {path_to_happywhale_test_metadata}","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:25:23.264760Z","iopub.execute_input":"2022-04-05T05:25:23.265109Z","iopub.status.idle":"2022-04-05T05:25:25.764922Z","shell.execute_reply.started":"2022-04-05T05:25:23.265063Z","shell.execute_reply":"2022-04-05T05:25:25.763730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [2-4] Creation of working directories","metadata":{}},{"cell_type":"markdown","source":"Working directories for saving processed images(background removed/cropped images) are created.","metadata":{}},{"cell_type":"code","source":"# Creates working directory for saving background removed images.\npath_to_dir_nobg_train_images = \"/kaggle/working/nobg/train_images\"\n!mkdir -p {path_to_dir_nobg_train_images}","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:25:25.766622Z","iopub.execute_input":"2022-04-05T05:25:25.766846Z","iopub.status.idle":"2022-04-05T05:25:26.583742Z","shell.execute_reply.started":"2022-04-05T05:25:25.766819Z","shell.execute_reply":"2022-04-05T05:25:26.582733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creates working directory for saving cropped images.\npath_to_dir_cropped_train_images = \"/kaggle/working/cropped/train_images\"\n!mkdir -p {path_to_dir_cropped_train_images}","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:25:26.585453Z","iopub.execute_input":"2022-04-05T05:25:26.585754Z","iopub.status.idle":"2022-04-05T05:25:27.394726Z","shell.execute_reply.started":"2022-04-05T05:25:26.585722Z","shell.execute_reply":"2022-04-05T05:25:27.393691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Shows created directories.\n!ls /kaggle/working/*","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:25:27.396630Z","iopub.execute_input":"2022-04-05T05:25:27.396928Z","iopub.status.idle":"2022-04-05T05:25:28.202856Z","shell.execute_reply.started":"2022-04-05T05:25:27.396894Z","shell.execute_reply":"2022-04-05T05:25:28.202075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [2-5] Loading utility scripts","metadata":{}},{"cell_type":"markdown","source":"Utility scripts for visualization of \"Happywhale - Whale and Dolphin Identification\" competition dataset, which is defined in the other kernel [\"Utility Functions for Visualization of Dataset\"](https://www.kaggle.com/code/acchiko/utility-functions-for-visualization-of-dataset), are loaded. The way to use/create utility scripts is summarized in the other kernel [\"How to Create Utility Scripts\"](https://www.kaggle.com/code/acchiko/how-to-use-create-utility-scripts).","metadata":{}},{"cell_type":"code","source":"import utility_functions_for_visualization_of_dataset as myutils","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:25:28.204346Z","iopub.execute_input":"2022-04-05T05:25:28.204727Z","iopub.status.idle":"2022-04-05T05:25:28.209110Z","shell.execute_reply.started":"2022-04-05T05:25:28.204694Z","shell.execute_reply":"2022-04-05T05:25:28.208350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [2-6] Showing images","metadata":{}},{"cell_type":"markdown","source":"Train/Test images are shown. First, class for processing train/test images and metadata is defined.","metadata":{}},{"cell_type":"code","source":"# Imports required libs.\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom PIL import Image, ImageDraw\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport os\nfrom tqdm import tqdm","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:25:28.211584Z","iopub.execute_input":"2022-04-05T05:25:28.211848Z","iopub.status.idle":"2022-04-05T05:25:28.222139Z","shell.execute_reply.started":"2022-04-05T05:25:28.211816Z","shell.execute_reply":"2022-04-05T05:25:28.221545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Defines class for processing train/test images and metadata.\nclass WhaleAndDolphin():\n    def __init__(self, path_to_metadata, path_to_dir_images, \\\n                 path_to_dir_nobg_images, path_to_dir_cropped_images):\n        self._path_to_metadata = path_to_metadata\n        self._path_to_dir_images = path_to_dir_images\n        self._path_to_dir_nobg_images = path_to_dir_nobg_images\n        self._path_to_dir_cropped_images = path_to_dir_cropped_images\n        \n        # Loads metadata to variable \"_metadata_all\"\n        self._metadata_all = pd.read_csv(path_to_metadata)\n        \n        # Adds several colmuns.\n        path_to_images = \\\n            [\"%s/%s\" % (path_to_dir_images, row.image) \\\n             for row in self._metadata_all.itertuples()]\n        self._metadata_all[\"path_to_image\"] = path_to_images\n        \n        path_to_nobg_images = \\\n            [\"%s/%s\" % (path_to_dir_nobg_images, row.image.replace(\".jpg\", \".png\")) \\\n             for row in self._metadata_all.itertuples()]\n        self._metadata_all[\"path_to_nobg_image\"] = path_to_nobg_images\n        \n        path_to_cropped_images = \\\n            [\"%s/%s\" % (path_to_dir_cropped_images, row.image.replace(\".jpg\", \".png\")) \\\n             for row in self._metadata_all.itertuples()]\n        self._metadata_all[\"path_to_cropped_image\"] = path_to_cropped_images\n        \n        annotations_xyxy = \\\n            [[] for row in self._metadata_all.itertuples()]\n        self._metadata_all[\"annotations_xyxy\"] = annotations_xyxy\n        \n        # Copies the metadata for processing it.\n        self._metadata = self._metadata_all.copy()\n        \n        self._all_species = self.getSpecies()\n        self._all_individual_ids = self.getIndividualIDs()\n        \n    def resetMetadata(self, initialize=False):\n        if hasattr(self, \"_metadata_tmp\") and not initialize:\n            self._metadata = self._metadata_tmp.copy()\n        else:\n            self._metadata = self._metadata_all.copy()\n            \n    def saveMetadataTemporary(self):\n        self._metadata_tmp = self._metadata.copy()\n        \n    def filterMetadata(self, query=\"index > -1\"):\n        sliced_metadata = \\\n            self._metadata.query(query).reset_index(drop=True)\n        self._metadata = sliced_metadata.copy()\n        \n    def filterMetadataBackgroundRemovedImageExistence(self):\n        indices = []\n        for row in self._metadata.itertuples():\n            if not os.path.exists(row.path_to_nobg_image):\n                indices.append(row.Index)\n                \n        sliced_metadata = self._metadata.drop(index=indices).reset_index(drop=True)\n        self._metadata = sliced_metadata.copy()\n        \n    def writeMetadata(self, path_to_metadata):\n        self._metadata.to_csv(path_to_metadata, index=False)\n        \n    def getMetadata(self):\n        return self._metadata\n    \n    def getSpecies(self):\n        return self._metadata[\"species\"].unique()\n    \n    def _species2id(self, species):\n        return np.where(self._all_species == species)[0][0]\n    \n    def getIndividualIDs(self):\n        return self._metadata[\"individual_id\"].unique()\n    \n    def showImagesTile(self, num_cols=4, draw_annotations=False):\n        metadata = self._metadata\n        titles = [row.image for row in metadata.itertuples()]\n        path_to_images = [row.path_to_image \\\n                          for row in metadata.itertuples()]\n        images = myutils.getImages(path_to_images)\n        if \"annotations_xyxy\" in metadata.columns and draw_annotations:\n            annotations_xyxy_for_images = [row.annotations_xyxy \\\n                                           for row in metadata.itertuples()]\n            texts_for_images = [[\"\" for _ in \\\n                                 range(len(row.annotations_xyxy))] \\\n                                 for row in metadata.itertuples()]\n            myutils.drawAnnotations( \\\n                images, \\\n                annotations_xyxy_for_images=annotations_xyxy_for_images, \\\n                texts_for_images=texts_for_images, \\\n                line_color=\"green\", line_width=3, text_color=\"green\" \\\n            )\n        myutils.showImagesTile(titles, images, num_cols=num_cols)\n        \n    def showProcessedImagesTile(self, num_cols=3, draw_annotations=False):\n        metadata = self._metadata\n        titles, path_to_images = [], []\n        for row in metadata.itertuples():\n            titles.append(\"%s (Org.)\" % row.image)\n            path_to_images.append(row.path_to_image)\n            \n            titles.append(\"%s (BG. removed)\" % row.image)\n            path_to_images.append(row.path_to_nobg_image)\n            \n            titles.append(\"%s (Cropped)\" % row.image)\n            path_to_images.append(row.path_to_cropped_image)\n        \n        images = myutils.getImages(path_to_images)\n        if \"annotations_xyxy\" in metadata.columns and draw_annotations:\n            annotations_xyxy_for_images = [row.annotations_xyxy \\\n                                           for row in metadata.itertuples()]\n            texts_for_images = [[\"\" for _ in \\\n                                 range(len(row.annotations_xyxy))] \\\n                                 for row in metadata.itertuples()]\n            myutils.drawAnnotations( \\\n                images[::3], \\\n                annotations_xyxy_for_images=annotations_xyxy_for_images, \\\n                texts_for_images=texts_for_images, \\\n                line_color=\"red\", line_width=3, text_color=\"red\" \\\n            ) # For only org. images.\n        myutils.showImagesTile(titles, images, num_cols=num_cols)\n        \n    def showIndividualImagesTile(self, num_cols=4, \\\n                                 max_num_individual_images=4, \\\n                                 max_num_individuals=10, \\\n                                 draw_annotations=False):\n        self.saveMetadataTemporary()\n        \n        individual_ids = self.getIndividualIDs()\n        for individual_id in individual_ids[:max_num_individuals]:\n            print()\n            print(\"Individual ID : %s\" % individual_id)\n            self.filterMetadata(query=\"individual_id == \\\"%s\\\"\" % individual_id)\n            self.filterMetadata(query=\"index < %d\" % max_num_individual_images)\n            self.showImagesTile( \\\n                num_cols=num_cols, \\\n                draw_annotations=draw_annotations\n            )\n            self.resetMetadata()\n            \n    def removeBackground(self):\n        metadata = self._metadata\n        path_to_inputs = [row.path_to_image \\\n                          for row in metadata.itertuples()]\n        path_to_outputs = [row.path_to_nobg_image \\\n                           for row in metadata.itertuples()]\n        \n        for path_to_input, path_to_output in \\\n            zip(path_to_inputs, path_to_outputs):\n            !backgroundremover -i {path_to_input} -o {path_to_output}\n        \n    def calculateAnnotationsXyxy(self):\n        batch_size = 100\n        num_batches = len(self._metadata) // batch_size + 1\n        \n        for i_batch in range(num_batches):\n            i_start = i_batch * batch_size\n            i_end = i_start + batch_size\n            metadata = self._metadata.iloc[i_start:i_end]\n            \n            path_to_nobg_images = [row.path_to_nobg_image \\\n                                   for row in metadata.itertuples()]\n            nobg_images = myutils.getImages(path_to_nobg_images)\n            #fixed_images = [Image.eval(nobg_image, self._removeBugPixel) \\\n            #                for nobg_image in nobg_images]\n            class_ids = [self._species2id(row.species) for row \\\n                         in metadata.itertuples()]\n            \n            annotations_xyxy = []\n            for nobg_image, class_id in zip(nobg_images, class_ids):\n                _, _, _, a = nobg_image.split()\n                x_min, y_min, x_max, y_max = a.getbbox() # Bounding box of non-zero alpha region\n                confidence = 1.0 # Dummy\n                annotation_xyxy = myutils._annotationXyxy(class_id, \\\n                                                          x_min, y_min, \\\n                                                          x_max, y_max, \\\n                                                          confidence)\n                annotations_xyxy.append([annotation_xyxy])\n                \n            self._metadata[\"annotations_xyxy\"].iloc[i_start:i_end] = \\\n                annotations_xyxy\n    \n    #def _removeBugPixel(self, pixel_value):\n    #    if pixel_value == 1:\n    #        return 0\n    #    else:\n    #        return pixel_value\n        \n    def cropObject(self):\n        batch_size = 100\n        num_batches = len(self._metadata) // batch_size + 1\n        \n        for i_batch in range(num_batches):\n            i_start = i_batch * batch_size\n            i_end = i_start + batch_size\n            metadata = self._metadata.iloc[i_start:i_end]\n            \n            path_to_inputs = [row.path_to_image \\\n                              for row in metadata.itertuples()]\n            path_to_outputs = [row.path_to_cropped_image \\\n                               for row in metadata.itertuples()]\n            annotations_xyxy = [row.annotations_xyxy for row \\\n                                in metadata.itertuples()]\n            \n            for path_to_input, path_to_output, annotations_xyxy in \\\n                zip(path_to_inputs, path_to_outputs, annotations_xyxy):\n                \n                image = Image.open(path_to_input)\n                annotation_xyxy = \\\n                    self._maxConfidenceAnnotation(annotations_xyxy)\n                x_min = annotation_xyxy[\"x_min\"]\n                y_min = annotation_xyxy[\"y_min\"]\n                x_max = annotation_xyxy[\"x_max\"]\n                y_max = annotation_xyxy[\"y_max\"]\n                image_cropped = image.crop((x_min, y_min, x_max, y_max))\n                image_cropped.save(path_to_output)\n            \n    def _maxConfidenceAnnotation(self, annotations_xyxy):\n        confidences = np.array([annotation_xyxy[\"confidence\"] \\\n                                for annotation_xyxy in annotations_xyxy])\n        index = np.argmax(confidences)\n        return annotations_xyxy[index]","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:11:48.709122Z","iopub.execute_input":"2022-04-05T06:11:48.709690Z","iopub.status.idle":"2022-04-05T06:11:48.761648Z","shell.execute_reply.started":"2022-04-05T06:11:48.709653Z","shell.execute_reply":"2022-04-05T06:11:48.760230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some of train images are shown as example. Test images can be shown using the same class.","metadata":{}},{"cell_type":"code","source":"# Loads metadata for train images.\nwhale_and_dolphin = WhaleAndDolphin(\n    path_to_metadata=path_to_happywhale_train_metadata,\n    path_to_dir_images=path_to_dir_happywhale_train_images,\n    path_to_dir_nobg_images=path_to_dir_nobg_train_images,\n    path_to_dir_cropped_images=path_to_dir_cropped_train_images\n)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:11:52.507833Z","iopub.execute_input":"2022-04-05T06:11:52.508303Z","iopub.status.idle":"2022-04-05T06:11:53.129800Z","shell.execute_reply.started":"2022-04-05T06:11:52.508254Z","shell.execute_reply":"2022-04-05T06:11:53.128928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Shows train images for the first 3 individuals for each species.\nnum_cols = 4\nmax_num_individual_images = 4\nmax_num_individuals = 3\n\nall_species = whale_and_dolphin.getSpecies()\nfor i, species in enumerate(all_species):\n    whale_and_dolphin.filterMetadata(query=\"species == \\\"%s\\\"\" % species)\n    \n    print()\n    print(\"--------------------------------------------------\")\n    print()\n    print(\"   Images for species No.%02d %s\" % (i, species))\n    print()\n    print(\"--------------------------------------------------\")\n    whale_and_dolphin.showIndividualImagesTile( \\\n        num_cols=num_cols, \\\n        max_num_individual_images=max_num_individual_images, \\\n        max_num_individuals=max_num_individuals \\\n    )\n    print()\n    print()\n    \n    whale_and_dolphin.resetMetadata(initialize=True)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:25:28.884920Z","iopub.execute_input":"2022-04-05T05:25:28.885314Z","iopub.status.idle":"2022-04-05T05:28:27.136081Z","shell.execute_reply.started":"2022-04-05T05:25:28.885263Z","shell.execute_reply":"2022-04-05T05:28:27.135237Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <B>[3] Creation of annotation data</B>","metadata":{}},{"cell_type":"markdown","source":"Annotation data is created with the following steps,\n- Removing background of image,\n- Calculating bounding box,\n- Cropping bounding box,\n- Showing processed images.","metadata":{}},{"cell_type":"markdown","source":"## [3-1] Removing background of image","metadata":{}},{"cell_type":"markdown","source":"Background of image is removed with the command line tool [\"backgroundremover\"](https://github.com/nadermx/backgroundremover) to make easy to calculate bounding box. Sometimes background is not removed properly with the tool, but it works well for almost cases.","metadata":{}},{"cell_type":"markdown","source":"It is difficult to remove background with the tool in a kernel because of RAM utilization limitation of kaggle notebook (~4000 images can be processed in a kernel within the limitation.). So it is done in the series of the following kernels,\n- [\"Whale&Dolphin Background Removed Images (1/4)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-background-removed-images-1-4),\n- [\"Whale&Dolphin Background Removed Images (2/4)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-background-removed-images-2-4),\n- [\"Whale&Dolphin Background Removed Images (3/4)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-background-removed-images-3-4),\n- [\"Whale&Dolphin Background Removed Images (4/4)\"](https://www.kaggle.com/code/acchiko/whale-dolphin-background-removed-images-4-4).","metadata":{}},{"cell_type":"markdown","source":"First, the tool is installed (Background removed images in the other kernels will be used in the kernel, so the code is commented out.).","metadata":{}},{"cell_type":"code","source":"#!pip install backgroundremover","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:28:27.137249Z","iopub.execute_input":"2022-04-05T05:28:27.137991Z","iopub.status.idle":"2022-04-05T05:28:27.141684Z","shell.execute_reply.started":"2022-04-05T05:28:27.137954Z","shell.execute_reply":"2022-04-05T05:28:27.140702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, background is removed (Background removed images in the other kernels will be used in the kernel, so the code is commented out.).","metadata":{}},{"cell_type":"code","source":"# Loads metadata for train images.\nwhale_and_dolphin = WhaleAndDolphin(\n    path_to_metadata=path_to_happywhale_train_metadata,\n    path_to_dir_images=path_to_dir_happywhale_train_images,\n    path_to_dir_nobg_images=path_to_dir_nobg_train_images,\n    path_to_dir_cropped_images=path_to_dir_cropped_train_images\n)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:12:26.085819Z","iopub.execute_input":"2022-04-05T06:12:26.086473Z","iopub.status.idle":"2022-04-05T06:12:26.469742Z","shell.execute_reply.started":"2022-04-05T06:12:26.086420Z","shell.execute_reply":"2022-04-05T06:12:26.469060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Removes background. Limits number of processing images because of kaggle notebook limitation. Sets same number for each species as possible.\n#num_images_per_species = int(4000 / len(all_species))\n#i_start = 0\n#i_end = i_start + num_images_per_species\n\n#all_species = whale_and_dolphin.getSpecies()\n#whale_and_dolphin.saveMetadataTemporary()\n\n#for i, species in enumerate(all_species):\n#    whale_and_dolphin.filterMetadata(query=\"species == \\\"%s\\\"\" % species)\n#    whale_and_dolphin.filterMetadata(query=\"%d <= index < %d\" % (i_start, i_end))\n#    print()\n#    print(\"--------------------------------------------------\")\n#    print()\n#    print(\"   Processing images for species No.%02d %s\" % (i, species))\n#    print()\n#    print(\"--------------------------------------------------\")\n#    whale_and_dolphin.removeBackground()\n#    print()\n    \n#    whale_and_dolphin.resetMetadata()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:28:27.656513Z","iopub.execute_input":"2022-04-05T05:28:27.656820Z","iopub.status.idle":"2022-04-05T05:28:27.662677Z","shell.execute_reply.started":"2022-04-05T05:28:27.656778Z","shell.execute_reply":"2022-04-05T05:28:27.661783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If it succeeds, background removed images are saved in the following directory (Background removed images in the other kernels will be used in the kernel, so the code is commented out.).","metadata":{}},{"cell_type":"code","source":"#!ls {path_to_dir_nobg_train_images} | cat -n | tail -5","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:28:27.664107Z","iopub.execute_input":"2022-04-05T05:28:27.664542Z","iopub.status.idle":"2022-04-05T05:28:27.676691Z","shell.execute_reply.started":"2022-04-05T05:28:27.664496Z","shell.execute_reply":"2022-04-05T05:28:27.675903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Background removed images in the other kernels can be loaded with the same way as shown in the chapter \"[2-1] Loading dataset\". If it succeeds, the background removed images are loaded to the following path.","metadata":{}},{"cell_type":"code","source":"# Defines path to directories for background removed images.\npath_to_dir_nobg_train_images_part1 = \"/kaggle/input/whale-dolphin-background-removed-images-1-4/nobg/train_images\"\npath_to_dir_nobg_train_images_part2 = \"/kaggle/input/whale-dolphin-background-removed-images-2-4/nobg/train_images\"\npath_to_dir_nobg_train_images_part3 = \"/kaggle/input/whale-dolphin-background-removed-images-3-4/nobg/train_images\"\npath_to_dir_nobg_train_images_part4 = \"/kaggle/input/whale-dolphin-background-removed-images-4-4/nobg/train_images\"","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:28:27.677727Z","iopub.execute_input":"2022-04-05T05:28:27.678024Z","iopub.status.idle":"2022-04-05T05:28:27.687956Z","shell.execute_reply.started":"2022-04-05T05:28:27.677984Z","shell.execute_reply":"2022-04-05T05:28:27.687146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Shows list of background removed images.\n!echo \"{path_to_dir_nobg_train_images_part1}:\"\n!ls {path_to_dir_nobg_train_images_part1} | cat -n | tail -5\n!echo \"\"\n\n!echo \"{path_to_dir_nobg_train_images_part2}:\"\n!ls {path_to_dir_nobg_train_images_part2} | cat -n | tail -5\n!echo \"\"\n\n!echo \"{path_to_dir_nobg_train_images_part3}:\"\n!ls {path_to_dir_nobg_train_images_part3} | cat -n | tail -5\n!echo \"\"\n\n!echo \"{path_to_dir_nobg_train_images_part4}:\"\n!ls {path_to_dir_nobg_train_images_part4} | cat -n | tail -5\n!echo \"\"","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:28:27.688994Z","iopub.execute_input":"2022-04-05T05:28:27.689252Z","iopub.status.idle":"2022-04-05T05:28:37.629407Z","shell.execute_reply.started":"2022-04-05T05:28:27.689213Z","shell.execute_reply":"2022-04-05T05:28:37.628348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All the images are copied to the working directory.","metadata":{}},{"cell_type":"code","source":"# Copies background removed images into working directories.\n!cp  {path_to_dir_nobg_train_images_part1}/* {path_to_dir_nobg_train_images}\n!cp  {path_to_dir_nobg_train_images_part2}/* {path_to_dir_nobg_train_images}\n!cp  {path_to_dir_nobg_train_images_part3}/* {path_to_dir_nobg_train_images}\n!cp  {path_to_dir_nobg_train_images_part4}/* {path_to_dir_nobg_train_images}","metadata":{"execution":{"iopub.status.busy":"2022-04-05T05:28:37.631061Z","iopub.execute_input":"2022-04-05T05:28:37.631372Z","iopub.status.idle":"2022-04-05T05:31:00.224294Z","shell.execute_reply.started":"2022-04-05T05:28:37.631337Z","shell.execute_reply":"2022-04-05T05:31:00.222838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Shows list of copied background removed images.\n!echo \"{path_to_dir_nobg_train_images}:\"\n!ls {path_to_dir_nobg_train_images} | cat -n | tail -5","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:32:29.523341Z","iopub.execute_input":"2022-04-05T06:32:29.523674Z","iopub.status.idle":"2022-04-05T06:32:31.282142Z","shell.execute_reply.started":"2022-04-05T06:32:29.523643Z","shell.execute_reply":"2022-04-05T06:32:31.281090Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [3-2] Calculating bounding box","metadata":{}},{"cell_type":"markdown","source":"Bounding box is calculated with [\"Image.getbbox()\" of pillow](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.getbbox) (\"Image.getbbox()\" is called in the method \"WhaleAndDolphin.calculateAnnotationsXyxy()\".).","metadata":{}},{"cell_type":"code","source":"# Filters metadata, which has background removed image.\nwhale_and_dolphin.filterMetadataBackgroundRemovedImageExistence()\n\nmetadata = whale_and_dolphin.getMetadata()\nmetadata","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:32:39.629423Z","iopub.execute_input":"2022-04-05T06:32:39.629885Z","iopub.status.idle":"2022-04-05T06:32:39.745596Z","shell.execute_reply.started":"2022-04-05T06:32:39.629850Z","shell.execute_reply":"2022-04-05T06:32:39.744462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculates bounding box of object.\nwhale_and_dolphin.calculateAnnotationsXyxy()","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:12:43.695337Z","iopub.execute_input":"2022-04-05T06:12:43.695998Z","iopub.status.idle":"2022-04-05T06:32:19.435712Z","shell.execute_reply.started":"2022-04-05T06:12:43.695945Z","shell.execute_reply":"2022-04-05T06:32:19.434299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If it succeeds, annotation data for showing bounding box is created as follows.","metadata":{}},{"cell_type":"code","source":"# Shows metadata with calculated bounding box.\nmetadata = whale_and_dolphin.getMetadata()\nmetadata","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:32:19.437071Z","iopub.status.idle":"2022-04-05T06:32:19.437476Z","shell.execute_reply.started":"2022-04-05T06:32:19.437273Z","shell.execute_reply":"2022-04-05T06:32:19.437297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The metadata with created annotation data is saved as csv file in the following directory.","metadata":{}},{"cell_type":"code","source":"# Saves metadata as csv file.\npath_to_train_metadata = \"/kaggle/working/train_with_annotation.csv\"\nwhale_and_dolphin.writeMetadata(path_to_train_metadata)","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:32:19.438697Z","iopub.status.idle":"2022-04-05T06:32:19.439007Z","shell.execute_reply.started":"2022-04-05T06:32:19.438842Z","shell.execute_reply":"2022-04-05T06:32:19.438864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Shows created csv file.\n!head {path_to_train_metadata}","metadata":{"execution":{"iopub.status.busy":"2022-04-05T06:32:19.440650Z","iopub.status.idle":"2022-04-05T06:32:19.441189Z","shell.execute_reply.started":"2022-04-05T06:32:19.440998Z","shell.execute_reply":"2022-04-05T06:32:19.441020Z"},"trusted":true},"execution_count":null,"outputs":[]}]}