{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n

Great Barrier Reef EDA

\n
\n\n","metadata":{}},{"cell_type":"markdown","source":"#

Introduction

","metadata":{}},{"cell_type":"markdown","source":"Australia's stunningly beautiful Great Barrier Reef is the world’s largest coral reef and home to 1,500 species of fish, 400 species of corals, 130 species of sharks, rays, and a massive variety of other sea life.\n\nUnfortunately, the reef is under threat, in part because of the overpopulation of one particular starfish – the coral-eating crown-of-thorns starfish (or COTS for short). Scientists, tourism operators and reef managers established a large-scale intervention program to control COTS outbreaks to ecologically sustainable levels.Australia's stunningly beautiful Great Barrier Reef is the world’s largest coral reef and home to 1,500 species of fish, 400 species of corals, 130 species of sharks, rays, and a massive variety of other sea life.\n\nUnfortunately, the reef is under threat, in part because of the overpopulation of one particular starfish – the coral-eating crown-of-thorns starfish (or COTS for short). Scientists, tourism operators and reef managers established a large-scale intervention program to control COTS outbreaks to ecologically sustainable levels.\n
\nIn this notebook we will visualize the images of the coral reefs along with the bounding boxes present in the dataset. They look beautiful!","metadata":{}},{"cell_type":"markdown","source":"#

Install Required Libraries

","metadata":{}},{"cell_type":"code","source":"!pip install --upgrade wandb","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-29T14:05:02.101659Z","iopub.execute_input":"2021-11-29T14:05:02.102423Z","iopub.status.idle":"2021-11-29T14:05:17.589420Z","shell.execute_reply.started":"2021-11-29T14:05:02.102306Z","shell.execute_reply":"2021-11-29T14:05:17.588022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Import Required Libraries 📚

","metadata":{}},{"cell_type":"code","source":"import os\nimport cv2\nfrom PIL import Image\n\nimport numpy as np\nimport pandas as pd\n\nfrom tqdm import tqdm","metadata":{"execution":{"iopub.status.busy":"2021-11-29T14:05:17.593094Z","iopub.execute_input":"2021-11-29T14:05:17.593516Z","iopub.status.idle":"2021-11-29T14:05:17.841191Z","shell.execute_reply.started":"2021-11-29T14:05:17.593461Z","shell.execute_reply":"2021-11-29T14:05:17.840147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\"Weights\n\n Weights & Biases (W&B) is a set of machine learning tools that helps you build better models faster. Kaggle competitions require fast-paced model development and evaluation. There are a lot of components: exploring the training data, training different models, combining trained models in different combinations (ensembling), and so on.\n\n> ⏳ Lots of components = Lots of places to go wrong = Lots of time spent debugging\n\nWhy use W&B tables?\n\n* Well suited for Quick EDA!
\n* Let's you see the entire data
\n* Filter, sort and group data which can help answer some fundamental questions.
\n* Well suited to visualize model predictions and compare models
\n\nTo learn more about Weights and Biases Tables check out this colab notebook.","metadata":{}},{"cell_type":"code","source":"import wandb\n\ntry:\n from kaggle_secrets import UserSecretsClient\n user_secrets = UserSecretsClient()\n api_key = user_secrets.get_secret(\"wandb_api\")\n wandb.login(key=api_key)\n anony = None\nexcept:\n anony = \"must\"\n print('If you want to use your W&B account, go to Add-ons -> Secrets and provide your W&B access token. Use the Label name as wandb_api. \\nGet your W&B access token from here: https://wandb.ai/authorize')","metadata":{"execution":{"iopub.status.busy":"2021-11-29T14:05:17.842966Z","iopub.execute_input":"2021-11-29T14:05:17.843280Z","iopub.status.idle":"2021-11-29T14:05:19.883095Z","shell.execute_reply.started":"2021-11-29T14:05:17.843234Z","shell.execute_reply":"2021-11-29T14:05:19.881964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Read the Data 📖

","metadata":{}},{"cell_type":"markdown","source":"Data Structure\n
\n ","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"../input/tensorflow-great-barrier-reef/train.csv\")\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-29T14:05:19.884553Z","iopub.execute_input":"2021-11-29T14:05:19.884879Z","iopub.status.idle":"2021-11-29T14:05:19.966354Z","shell.execute_reply.started":"2021-11-29T14:05:19.884840Z","shell.execute_reply":"2021-11-29T14:05:19.965492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Processing

","metadata":{}},{"cell_type":"code","source":"# Convert String to List\ndf['annotations'] = df['annotations'].apply(eval)\n\n# Get the number of bounding boxes for each image\ndf['num_bboxes'] = df['annotations'].apply(lambda x: len(x))","metadata":{"execution":{"iopub.status.busy":"2021-11-29T14:05:19.969085Z","iopub.execute_input":"2021-11-29T14:05:19.969589Z","iopub.status.idle":"2021-11-29T14:05:20.238947Z","shell.execute_reply.started":"2021-11-29T14:05:19.969543Z","shell.execute_reply":"2021-11-29T14:05:20.237889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.groupby('video_id').count()","metadata":{"execution":{"iopub.status.busy":"2021-11-29T14:05:20.240797Z","iopub.execute_input":"2021-11-29T14:05:20.241171Z","iopub.status.idle":"2021-11-29T14:05:20.273378Z","shell.execute_reply.started":"2021-11-29T14:05:20.241131Z","shell.execute_reply":"2021-11-29T14:05:20.272333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will visualize only the images with bounding boxes","metadata":{}},{"cell_type":"code","source":"df_with_boxes = df[df.num_bboxes != 0].reset_index(drop=True)\ndf_with_boxes.shape","metadata":{"execution":{"iopub.status.busy":"2021-11-29T14:05:20.274950Z","iopub.execute_input":"2021-11-29T14:05:20.275875Z","iopub.status.idle":"2021-11-29T14:05:20.286408Z","shell.execute_reply.started":"2021-11-29T14:05:20.275820Z","shell.execute_reply":"2021-11-29T14:05:20.284991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Visualizations

","metadata":{}},{"cell_type":"code","source":"run = wandb.init(project='GreatBarrier',\n job_type='Visualization',\n name='Image Visualization',\n anonymous='must')","metadata":{"execution":{"iopub.status.busy":"2021-11-29T14:05:20.288089Z","iopub.execute_input":"2021-11-29T14:05:20.288771Z","iopub.status.idle":"2021-11-29T14:05:27.229170Z","shell.execute_reply.started":"2021-11-29T14:05:20.288715Z","shell.execute_reply":"2021-11-29T14:05:27.228448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preview_table = wandb.Table(columns=['Video Id', 'Image', 'Video Frame', 'Num Boxes'])\n\nfor i in tqdm(range(len(df_with_boxes))):\n row = df_with_boxes.loc[i]\n img = Image.open(f'../input/tensorflow-great-barrier-reef/train_images/video_{row.video_id}/{row.video_frame}.jpg')\n bboxes_list = []\n for annot in row.annotations:\n bbox = {\n \"position\": {\n \"minX\": annot['x'],\n \"maxX\": annot['x'] + annot['width'],\n \"minY\": annot['y'],\n \"maxY\": annot['y'] + annot['height']\n },\n \"class_id\": 1,\n \"box_caption\": \"starfish\",\n \"domain\": \"pixel\"\n }\n bboxes_list.append(bbox)\n \n image = wandb.Image(img,\n boxes = {\n \"ground_truth\": {\n \"box_data\": bboxes_list,\n \"class_labels\" : {1: 'starfish'}\n }\n },\n # Add extra dummy class to get red bounding boxes\n classes = [{\"id\": 0, \"name\": \"none\"}, {\"id\": 1, \"name\": \"starfish\"}]\n )\n preview_table.add_data(row.video_id,\n image,\n row.video_frame,\n len(bboxes_list))\n\nwandb.log({'Visualization': preview_table})\nrun.finish()","metadata":{"execution":{"iopub.status.busy":"2021-11-29T14:05:27.231536Z","iopub.execute_input":"2021-11-29T14:05:27.231888Z","iopub.status.idle":"2021-11-29T14:42:33.381318Z","shell.execute_reply.started":"2021-11-29T14:05:27.231843Z","shell.execute_reply":"2021-11-29T14:42:33.378960Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"View the Complete Table Here ⮕","metadata":{}},{"cell_type":"markdown","source":"![](https://i.imgur.com/gjcGnD7.gif)","metadata":{}},{"cell_type":"markdown","source":"Work in Progress 🚧","metadata":{}},{"cell_type":"markdown","source":"![Upvote!](https://img.shields.io/badge/Upvote-If%20you%20like%20my%20work-07b3c8?style=for-the-badge&logo=kaggle)","metadata":{}}]}