{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"First, I would like to extend my many thanks to the hosts of this competition to come up with something meaningful and impactful. Few months back I saw a Netflix documentary [Seaspiracy](https://www.youtube.com/watch?v=1Q5CXN7soQg) which showed the deep impact human created pollution has on coral reefs. I was shocked to know that we have lost JUST SO MUCH of one of the largest ocean species. I always wanted to contribute to this cause and this competition for this reason is cloe to my heart. \n\nThis kernel will explore the dataset. We will look at different sequences of the dataset using various Weights and Biases features. ","metadata":{}},{"cell_type":"markdown","source":"# Imports and Setup","metadata":{}},{"cell_type":"code","source":"# For video/gif creation\n!pip -qq install imageio pygifsicle\n# Install W&B\n!pip install -q --upgrade wandb","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-28T11:35:08.352229Z","iopub.execute_input":"2021-11-28T11:35:08.352892Z","iopub.status.idle":"2021-11-28T11:35:31.159172Z","shell.execute_reply.started":"2021-11-28T11:35:08.352789Z","shell.execute_reply":"2021-11-28T11:35:31.158156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Required by pygifsicle\n!apt-get install gifsicle","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-28T11:38:14.619141Z","iopub.execute_input":"2021-11-28T11:38:14.619468Z","iopub.status.idle":"2021-11-28T11:38:18.939911Z","shell.execute_reply.started":"2021-11-28T11:38:14.619423Z","shell.execute_reply":"2021-11-28T11:38:18.939045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Login \nimport wandb\nprint(wandb.__version__)\nwandb.login()","metadata":{"execution":{"iopub.status.busy":"2021-11-28T11:38:18.941736Z","iopub.execute_input":"2021-11-28T11:38:18.941991Z","iopub.status.idle":"2021-11-28T11:38:28.167365Z","shell.execute_reply.started":"2021-11-28T11:38:18.941956Z","shell.execute_reply":"2021-11-28T11:38:28.166494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport cv2\nimport ast\nimport numpy as np\nimport pandas as pd\nimport imageio as iio\nfrom tqdm import tqdm\nfrom pygifsicle import optimize\nimport matplotlib.pyplot as plt","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-11-28T11:38:29.487524Z","iopub.execute_input":"2021-11-28T11:38:29.487899Z","iopub.status.idle":"2021-11-28T11:38:29.830435Z","shell.execute_reply.started":"2021-11-28T11:38:29.487863Z","shell.execute_reply":"2021-11-28T11:38:29.829438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Dataset and Quick EDA","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('../input/tensorflow-great-barrier-reef/train.csv')\nprint(\"Number of training images: \", len(df))\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-28T11:38:29.887256Z","iopub.execute_input":"2021-11-28T11:38:29.888072Z","iopub.status.idle":"2021-11-28T11:38:29.961675Z","shell.execute_reply.started":"2021-11-28T11:38:29.888016Z","shell.execute_reply":"2021-11-28T11:38:29.960814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📍 Note: There are 23501 training images but the images are part of video sequences. Due to this fact there are images without any annotations to it. ","metadata":{}},{"cell_type":"code","source":"pd.DataFrame(df.annotations.value_counts())","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-28T11:38:30.522637Z","iopub.execute_input":"2021-11-28T11:38:30.522985Z","iopub.status.idle":"2021-11-28T11:38:30.547650Z","shell.execute_reply.started":"2021-11-28T11:38:30.522949Z","shell.execute_reply":"2021-11-28T11:38:30.546278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📍 Note: Out of 23501 images 18582 images have no annotations. This is something we need to take care while modeling. ","metadata":{}},{"cell_type":"code","source":"pd.DataFrame(df.sequence.value_counts())","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-11-28T11:38:31.073692Z","iopub.execute_input":"2021-11-28T11:38:31.074441Z","iopub.status.idle":"2021-11-28T11:38:31.085328Z","shell.execute_reply.started":"2021-11-28T11:38:31.074369Z","shell.execute_reply":"2021-11-28T11:38:31.084668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📍 Note: There are a total of 20 sequences. A sequence is gap-free subset of a given video. We can use this fact to create meaningful training and validation split. We however also need to consider the fact that only 4919 images are annotated. ","metadata":{}},{"cell_type":"markdown","source":"# Utilities","metadata":{}},{"cell_type":"code","source":"def get_frames_with_annotations(df_row: pd.Series) -> None:\n    \"\"\"\n    Get the frame (numpy.ndarray) and the associated annotations (bounding boxes).\n    \n    Arguments:\n        df_row (pd.Series): A row of the dataframe\n    \"\"\"\n    # Get frame path\n    frame_path = f\"{TRAIN_PATH}/video_{row.video_id}/{row.video_frame}.jpg\"\n    # Open the image with OpenCV\n    frame = cv2.imread(frame_path)\n    if frame is None:\n        return \n    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n    # Get annotations\n    annotations = ast.literal_eval(row.annotations)\n    \n    return frame, annotations","metadata":{"execution":{"iopub.status.busy":"2021-11-28T11:38:32.102268Z","iopub.execute_input":"2021-11-28T11:38:32.103029Z","iopub.status.idle":"2021-11-28T11:38:32.108756Z","shell.execute_reply.started":"2021-11-28T11:38:32.102990Z","shell.execute_reply":"2021-11-28T11:38:32.107909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def draw_bounding_boxes(frame: np.ndarray, \n                        annotations: list,\n                        color: tuple=(0, 0, 255),\n                        thickness: int= 2) -> None:\n    \"\"\"\n    Draw bounding box given an image and bounding box annotaions.\n    \n    Arguments:\n        frame (np.ndarray): Image \n        annotations (list): Bounding box coodinates in the format `x_min, y_min, width, height`.\n        color (tuple): Color of the bounding boxes drawn on the frame.\n        thickness (int): Thickness of the bounding boxees.\n    \"\"\"\n    frame = frame.copy()\n    \n    # Draw bounding boxes\n    for ant in annotations:\n        start_point = (ant['x'], ant['y'])\n        end_point = (ant['x']+ant['width'], ant['y']+ant['height'])\n\n        frame = cv2.rectangle(frame, start_point, end_point, color, thickness)\n        \n    return frame","metadata":{"execution":{"iopub.status.busy":"2021-11-28T11:38:32.817122Z","iopub.execute_input":"2021-11-28T11:38:32.817443Z","iopub.status.idle":"2021-11-28T11:38:32.824104Z","shell.execute_reply.started":"2021-11-28T11:38:32.817398Z","shell.execute_reply":"2021-11-28T11:38:32.823204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def wandb_bbox(image, bboxes, true_label, class_id_to_label, class_set):\n    \"\"\"\n    Create wandb.Image object. `class_set` is required to log bounding box onto W&B Tables.\n    \n    Arguments:\n    \n        image (np.ndarray): Image \n        bboxes (list): Bounding box coordinates as dictionary\n        true_label (int): Class id\n        class_id_to_label (dict): Dictionary mapping class id to class name\n        class_set (wandb.Classes): Needed to log image overlays onto W&B Tables. Might not be needed in future.\n    \"\"\"\n    all_boxes = []\n    for bbox in bboxes:\n        box_data = {\"position\": {\n                        \"minX\": bbox['x'],\n                        \"minY\": bbox['y'],\n                        \"maxX\": bbox['x']+bbox['width'],\n                        \"maxY\": bbox['y']+bbox['height']\n                    },\n                     \"class_id\" : int(true_label),\n                     \"box_caption\": class_id_to_label[true_label],\n                     \"domain\" : \"pixel\"}\n        all_boxes.append(box_data)\n    \n    return wandb.Image(image, boxes={\n        \"ground_truth\": {\n            \"box_data\": all_boxes,\n          \"class_labels\": class_id_to_label\n        }\n    }, classes=class_set)","metadata":{"execution":{"iopub.status.busy":"2021-11-28T11:38:33.510973Z","iopub.execute_input":"2021-11-28T11:38:33.511273Z","iopub.status.idle":"2021-11-28T11:38:33.517686Z","shell.execute_reply.started":"2021-11-28T11:38:33.511232Z","shell.execute_reply":"2021-11-28T11:38:33.517027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualize One Sequence","metadata":{}},{"cell_type":"code","source":"SEQ_NUMBER = 18048\nTRAIN_PATH = '../input/tensorflow-great-barrier-reef/train_images'","metadata":{"execution":{"iopub.status.busy":"2021-11-28T11:38:33.519312Z","iopub.execute_input":"2021-11-28T11:38:33.519990Z","iopub.status.idle":"2021-11-28T11:38:33.536434Z","shell.execute_reply.started":"2021-11-28T11:38:33.519952Z","shell.execute_reply":"2021-11-28T11:38:33.535290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get one sequence\nseq_df = df.loc[df.sequence == SEQ_NUMBER]\nseq_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-28T11:38:33.588460Z","iopub.execute_input":"2021-11-28T11:38:33.589313Z","iopub.status.idle":"2021-11-28T11:38:33.604912Z","shell.execute_reply.started":"2021-11-28T11:38:33.589267Z","shell.execute_reply":"2021-11-28T11:38:33.603559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"frames = []\n\n# Iterate over the sequence\nfor i in tqdm(range(len(seq_df))):\n    # Get the ith row\n    row = seq_df.iloc[i]\n    \n    # Get frame and annotations\n    frame, annotations = get_frames_with_annotations(row)\n    if frame is None:\n        continue\n        \n    # Draw bounding boxes\n    frame = draw_bounding_boxes(frame, annotations)\n    frames.append(frame)\n\nframes = np.array(frames)","metadata":{"execution":{"iopub.status.busy":"2021-11-28T11:38:34.102256Z","iopub.execute_input":"2021-11-28T11:38:34.102722Z","iopub.status.idle":"2021-11-28T11:38:37.047237Z","shell.execute_reply.started":"2021-11-28T11:38:34.102681Z","shell.execute_reply":"2021-11-28T11:38:37.046316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create gif\ngif_path = f\"sequence_{SEQ_NUMBER}.gif\"\n\nwith iio.get_writer(gif_path, mode='I') as writer:\n    for frame in frames:\n        writer.append_data(frame)\n        \noptimize(gif_path)\n\n# Log GIF as W&B Video\nrun = wandb.init(project='barrier_reef_viz')\nwandb.log({f\"{SEQ_NUMBER}\": wandb.Video(gif_path)})\nwandb.finish()\n\nrun","metadata":{"execution":{"iopub.status.busy":"2021-11-28T11:38:37.048820Z","iopub.execute_input":"2021-11-28T11:38:37.049071Z","iopub.status.idle":"2021-11-28T11:39:29.626560Z","shell.execute_reply.started":"2021-11-28T11:38:37.049042Z","shell.execute_reply":"2021-11-28T11:39:29.625683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [Check out the run page $\\rightarrow$](https://wandb.ai/ayush-thakur/barrier_reef_viz/runs/guwmlt1u?workspace=)","metadata":{}},{"cell_type":"markdown","source":"# Visualize Sequences with W&B Tables","metadata":{}},{"cell_type":"code","source":"unique_sequences = df.sequence.unique()\n\n# Define columns\ncolumns = [\"video_id\", \"video_frame\", \"sequence_frame_no\", \"is_annotation\"] # W&B Code 1\n\n# Setup a WandB Classes object. This will give additional metadata for visuals\n# Note that we need to pass class_set to wandb.Image. In future, we might not to do this extra step. \nclass_set = wandb.Classes([{'name': 'starfish', 'id': 0}]) # W&B Code 2\n\n# Iterate over each sequence\nfor seq in unique_sequences:\n    seq_df = df.loc[df.sequence == seq]\n    \n    # Log each sequence as separate W&B run\n    run = wandb.init(project='barrier_reef_viz', group='viz-tables', name=f\"table_{seq}\") # W&B Code 3\n    # Initialize W&B Tables\n    seq_table = wandb.Table(columns=columns) # W&B Code 4\n    \n    # Iterate over the sequence\n    for i in tqdm(range(len(seq_df))):\n        # Get the ith row\n        row = seq_df.iloc[i]\n        \n        # Get frame and annotations\n        frame, annotations = get_frames_with_annotations(row)\n        \n        if len(annotations)==0:\n            is_annotation = False\n        else:\n            is_annotation = True\n        \n        wandb_img = wandb_bbox(frame, annotations, 0, {0:'starfish'}, class_set)\n            \n        seq_table.add_data(row.video_id, \n                           wandb_img,\n                           row.sequence_frame,\n                           is_annotation) # W&B Code 5\n\n    wandb.log({\"tables_viz\": seq_table}) # W&B Code 6\n    \n    # Close W&B run\n    run.finish()","metadata":{"execution":{"iopub.status.busy":"2021-11-28T11:40:32.133082Z","iopub.execute_input":"2021-11-28T11:40:32.134079Z","iopub.status.idle":"2021-11-28T11:43:53.713562Z","shell.execute_reply.started":"2021-11-28T11:40:32.133998Z","shell.execute_reply":"2021-11-28T11:43:53.712815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Why log it to W&B?\n\nYou might wonder why take an extra step to log it onto W&B dashboard. That's a fair question and here are few reasons:\n* One can easily log all the sequences or any way the data (gif) here is generated for future reference. \n* You will probably visualize the dataset and start training models, logging the model prediction along with ground truth prediction like this will help document the experiments better. ","metadata":{}},{"cell_type":"markdown","source":"# WORK IN PROGRESS\n\nBetter ways to visualize the dataset in the context of model performance comparison. ","metadata":{}}]}