{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **EDA Solution Simplified in TensorFlow Barrier Reef Competition (Adapted from Baek Kyun Shin, Diego Gomez, and Aryan Lala)** #","metadata":{}},{"cell_type":"markdown","source":"## Introduction before Coding ##\nAfter we classified all the Crown-of-Thorns starfish in the previous notebook (https://www.kaggle.com/dinowun/the-help-protect-the-great-barrier-reef-code), the COTS has returned again, and they want payback! But theres another way to fix the COTS' payback plan, use the EDA solution. In this second notebook, we will go over my simplified walkthrough of the EDA Solution. ","metadata":{}},{"cell_type":"markdown","source":"## Load \"Train\" Data (and Imports) ##\nLike I said, before classifying the COTS again, we need to import a module, called pandas. Thus, we have to import the file path into the data_path variable and define the train variable that concatenates the file path of the competition data and the train.csv file and read it.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\nPATH_OF_DATA = '/kaggle/input/tensorflow-great-barrier-reef/'\ntrain_file = pd.read_csv(PATH_OF_DATA + 'train.csv')\ntrain_file","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:04:08.958157Z","iopub.execute_input":"2021-12-22T09:04:08.958700Z","iopub.status.idle":"2021-12-22T09:04:09.058889Z","shell.execute_reply.started":"2021-12-22T09:04:08.958606Z","shell.execute_reply":"2021-12-22T09:04:09.058307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## A Peek of the Train Data ##\nNow, after we read out the full path of the train.csv file, let's go and analyze the data inside. First analysis, we get the info of the train_file variable using the info function.","metadata":{}},{"cell_type":"code","source":"train_file.info()","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:04:09.060535Z","iopub.execute_input":"2021-12-22T09:04:09.060758Z","iopub.status.idle":"2021-12-22T09:04:09.089731Z","shell.execute_reply.started":"2021-12-22T09:04:09.060731Z","shell.execute_reply":"2021-12-22T09:04:09.088823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we run through the info() function of train_file, we see that the class of the train_file variable is a DataFrame object and there are 23501 entries according to RangeIndex. Also, there are 6 data columns: video_id (0), sequence (1), video_frame (2), sequence_frame (3), image_id (4), and last but not least, annotations (5). From each column, there are 4 int64 dtypes and 2 object dtypes. Over 1.1 MB of memory were used by the train_file variable. Second analysis, we'll find any signs of duplicated data over the train_file data.","metadata":{}},{"cell_type":"code","source":"train_file.duplicated().sum()","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:04:09.091097Z","iopub.execute_input":"2021-12-22T09:04:09.092325Z","iopub.status.idle":"2021-12-22T09:04:09.111899Z","shell.execute_reply.started":"2021-12-22T09:04:09.092274Z","shell.execute_reply":"2021-12-22T09:04:09.111181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Excellent!** There are no signs of duplicated data over the train_file data. Last analysis, we will make a summary of features known as feature summary.","metadata":{}},{"cell_type":"code","source":"def a_legit_resumetable(df):\n    '''This is a function to create summary but with features'''\n    print(f\"Shape: {df.shape}\")\n    one_summary = pd.DataFrame(df.dtypes, columns=['Data Type'])\n    one_summary = one_summary.reset_index()\n    one_summary = one_summary.rename(columns={'index': 'Features'})\n    one_summary['Num of Null Value'] = df.isnull().sum().values\n    one_summary['Num of Unique Value'] = df.nunique().values\n    one_summary['1st Value'] = df.iloc[0].values\n    one_summary['2nd Value'] = df.iloc[1].values\n    one_summary['3rd Value'] = df.iloc[2].values\n    return one_summary\n    ","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:04:09.113651Z","iopub.execute_input":"2021-12-22T09:04:09.114016Z","iopub.status.idle":"2021-12-22T09:04:09.122770Z","shell.execute_reply.started":"2021-12-22T09:04:09.113986Z","shell.execute_reply":"2021-12-22T09:04:09.121759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After running a_legit_resumetable function, we can now run the function again but this time with the train variable.","metadata":{}},{"cell_type":"code","source":"a_legit_resumetable(train_file)","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:04:09.124277Z","iopub.execute_input":"2021-12-22T09:04:09.124874Z","iopub.status.idle":"2021-12-22T09:04:09.167482Z","shell.execute_reply.started":"2021-12-22T09:04:09.124825Z","shell.execute_reply":"2021-12-22T09:04:09.166612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After running this line of code up there, we can see the shape of the train_file first, then we now observed that the one_summary variable is creating and reading out a csv file! There are 7 columns of labels: Features, Data Type, Num of Null Value, Num of Unique Value, 1st Value, 2nd Value, and 3rd Value. ","metadata":{}},{"cell_type":"markdown","source":"## Basic Engineering ##\nSince we setup'd our train_data, let's do some basic engineering. First, we have to create a function to downcast the train_file variable. Let's see how it went.","metadata":{}},{"cell_type":"code","source":"def downcast_time(df, verbose=True):\n    begin_mem = df.memory_usage().sum() / 1024**2\n    for col in df.columns:\n        dtype_name = df[col].dtype.name\n        if dtype_name == 'object':\n            pass\n        elif dtype_name == 'bool':\n            df[col] = df[col].astype('int8')\n        elif dtype_name.startswith('int') or (df[col].round() == df[col]).all():\n            df[col] = pd.to_numeric(df[col], downcast='integer')\n        else:\n            df[col] = pd.to_numeric(df[col], downcast='float')\n    end_of_mem = df.memory_usage().sum() / 1024**2\n    if verbose:\n        print('{:.1f}% Compressed'.format(100 * (begin_mem - end_of_mem) / begin_mem))\n        \n    return df","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:04:09.169160Z","iopub.execute_input":"2021-12-22T09:04:09.169647Z","iopub.status.idle":"2021-12-22T09:04:09.179842Z","shell.execute_reply.started":"2021-12-22T09:04:09.169603Z","shell.execute_reply":"2021-12-22T09:04:09.178864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Great! Now, we now define the train_file again with the downcast function with the train_file variable!","metadata":{}},{"cell_type":"code","source":"train_file = downcast_time(train_file)","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:04:09.181482Z","iopub.execute_input":"2021-12-22T09:04:09.182032Z","iopub.status.idle":"2021-12-22T09:04:09.204247Z","shell.execute_reply.started":"2021-12-22T09:04:09.181927Z","shell.execute_reply":"2021-12-22T09:04:09.203598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After we defined the train_file variable again, we can see that 41.7% of the train_file has been compressed.","metadata":{}},{"cell_type":"markdown","source":"## Feature Engineering ##\nNow that the train_file variable has been somewhat compressed, We can now start generating bounding boxes with annotations into the train_file variable! First of all, we have to import the ast module, then we convert the string of annotations to a list of annotations. Finally, we get the number of bounding boxes for each image.","metadata":{}},{"cell_type":"code","source":"# First, import the 'ast' module\nimport ast\n\n# Next, convert string to list type\ntrain_file['annotations'] = train_file['annotations'].apply(ast.literal_eval)\n\n# Finally, get the number of bounding boxes for each image\ntrain_file['num_bboxes'] = train_file['annotations'].apply(lambda i: len(i))","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:04:09.205577Z","iopub.execute_input":"2021-12-22T09:04:09.206002Z","iopub.status.idle":"2021-12-22T09:04:09.592017Z","shell.execute_reply.started":"2021-12-22T09:04:09.205969Z","shell.execute_reply":"2021-12-22T09:04:09.590675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Checking the Number of Frames with Bounding Boxes ##\nAfter the train_file variable obtained the number of bounding boxes, we can inspect the train_file variable by nesting the train_file variable with the 'num_bboxes' key and set it greater than 0.","metadata":{}},{"cell_type":"code","source":"train_file[train_file['num_bboxes'] > 0]","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:04:09.593701Z","iopub.execute_input":"2021-12-22T09:04:09.594055Z","iopub.status.idle":"2021-12-22T09:04:09.625144Z","shell.execute_reply.started":"2021-12-22T09:04:09.594010Z","shell.execute_reply":"2021-12-22T09:04:09.623966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After running this code, we see that all of the rows contained annotations of each images of the location of the COTS!","metadata":{}},{"cell_type":"markdown","source":"## Verification to Classify Whether There Is Corrupted Data ##\nBefore we plot frame images with bounding boxes that came from annotations, we have to verify if there are corrupted data in some images. So, let's take a run!","metadata":{}},{"cell_type":"code","source":"from os import listdir\nfrom PIL import Image\n\ndef verification(video_id):\n    path_of_file = PATH_OF_DATA + f'train_images/video_{video_id}/'\n    for filename in listdir(path_of_file):\n        if filename.endswith('.jpg'):\n            try:\n                image = Image.open(path_of_file + filename)\n                image.verify() # We need this line of code so that we can verify that it is an image\n            except (IOError, SyntaxError) as err:\n                print(\"There's a bad file there:\", filename) # This line prints out the names of corrupted files\n    print(f'Video {video_id} has all of the valid images. Results: Verified!')\n    \nfor video_id in range(3):\n    verification(video_id)","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:04:09.628932Z","iopub.execute_input":"2021-12-22T09:04:09.629639Z","iopub.status.idle":"2021-12-22T09:08:26.439772Z","shell.execute_reply.started":"2021-12-22T09:04:09.629577Z","shell.execute_reply":"2021-12-22T09:08:26.437951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Super! Video 0, 1, and 2 has all of the valid images, and none of the bad files were present in one of the folders. Now that there are no bad files in these folders, now lets get started on plotting frame images with bounding boxes!","metadata":{}},{"cell_type":"markdown","source":"## Plotting Frame Images with Bounding Boxes ##\nBefore we start plotting frame images with bounding boxes, we are going to load sequence of images with annotations. First, we are going to import numpy module as np and ImageDraw from the PIL (Python Image Library) module. Then, we will create two functions: one that fetch images, then fetch image lists.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nfrom PIL import ImageDraw\n\ndef fetch_image(df, video_id, frame_id):\n    # Let's first get a frame!\n    a_frame = df[(df['video_id'] == video_id) & (df['video_frame'] == frame_id)].iloc[0]\n    # Now, we will get bounding_boxes!\n    bounding_boxes = a_frame['annotations']\n    # Finally, let's open images!\n    img = Image.open(PATH_OF_DATA + f'train_images/video_{video_id}/{frame_id}.jpg')\n    \n    for a_box in bounding_boxes:\n        x0, y0, x1, y1 = (a_box['x'], a_box['y'], a_box['x'] + a_box['width'], a_box['y'] + a_box['height'])\n        drawing = ImageDraw.Draw(img)\n        drawing.rectangle((x0, y0, x1, y1), outline=180, width=5)\n    return img\n\ndef fetch_image_list(df, video_id, num_images, start_frame_idx):\n    image_list = [np.array(fetch_image(df, video_id, start_frame_idx + index)) for index in range(num_images)]\n    return image_list","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:08:26.442776Z","iopub.execute_input":"2021-12-22T09:08:26.443192Z","iopub.status.idle":"2021-12-22T09:08:26.459776Z","shell.execute_reply.started":"2021-12-22T09:08:26.443107Z","shell.execute_reply":"2021-12-22T09:08:26.459076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we are going to find the number of images in a list using the fetch_image_list function.","metadata":{}},{"cell_type":"code","source":"images = fetch_image_list(train_file, video_id=0, num_images=80, start_frame_idx=25)\nprint(f\"The number of images is: {len(images)}\")","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:08:26.460894Z","iopub.execute_input":"2021-12-22T09:08:26.461719Z","iopub.status.idle":"2021-12-22T09:08:29.351237Z","shell.execute_reply.started":"2021-12-22T09:08:26.461677Z","shell.execute_reply":"2021-12-22T09:08:29.350329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, there are 80 images in all after running the fetch_image_list function. With all being set, let's plot the images with bounding boxes! First, we are going to import the two matplotlib modules, then set up the grid and the figsize, and finally, plot every bounding boxes in each frames by 5.","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport matplotlib.gridspec as gridspec\n\nthe_grid = gridspec.GridSpec(4, 2)\nplt.figure(figsize=(18, 20))\n\nidx_list = [0, 5, 10, 15, 20, 25, 30, 35]\n\nfor i, idx in enumerate(idx_list):\n    ax = plt.subplot(the_grid[i])\n    plt.imshow(images[idx], interpolation='nearest')\n    ax.set_title(f'frame index {idx}')\n    plt.axis('off')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:08:29.352662Z","iopub.execute_input":"2021-12-22T09:08:29.352957Z","iopub.status.idle":"2021-12-22T09:08:31.104705Z","shell.execute_reply.started":"2021-12-22T09:08:29.352911Z","shell.execute_reply":"2021-12-22T09:08:31.103605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After we run this cell, we see that every 5 frames, there are bounding boxes plotted from the annotations in every index. Now we got every bounding boxes in every frame, we will now finally animate images with bounding boxes!","metadata":{}},{"cell_type":"markdown","source":"## Image Animation ##\nIn order to bind up all frame indexes, we are going to create an animated image or whatsoever call it a video. Another matplotlib module will be imported first with rc being set up, then the create_animation function will bind all frame indexes of images together in to one set thanks to the animate function inside, and lastly, set the frame interval to 130 and call the create_animation function.","metadata":{}},{"cell_type":"code","source":"from matplotlib import animation, rc\nrc('animation', html='jshtml')\n\ndef create_an_animation(imgs, frame_interval=130):\n    fig = plt.figure(figsize=(7, 4))\n    plt.axis('off')\n    img = plt.imshow(imgs[0])\n    \n    def animate(i):\n        img.set_array(imgs[i])\n        return [img]\n    \n    return animation.FuncAnimation(fig, animate, frames=len(imgs), interval=frame_interval)","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:08:31.106495Z","iopub.execute_input":"2021-12-22T09:08:31.107069Z","iopub.status.idle":"2021-12-22T09:08:31.116560Z","shell.execute_reply.started":"2021-12-22T09:08:31.107005Z","shell.execute_reply":"2021-12-22T09:08:31.115797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"frame_interval = 130 # Either set smaller number if wanted to play fast, else set bigger and slower\ncreate_an_animation(images, frame_interval=frame_interval)","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:08:31.117764Z","iopub.execute_input":"2021-12-22T09:08:31.118249Z","iopub.status.idle":"2021-12-22T09:08:43.112678Z","shell.execute_reply.started":"2021-12-22T09:08:31.118211Z","shell.execute_reply":"2021-12-22T09:08:43.111657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After all, we have done making a video that is made out of binded frame indexes and plotting a bounding box from the first image with annotations. But that's not all because we are going to submit predictions with inference over a TensorFlow COTS model.","metadata":{}},{"cell_type":"markdown","source":"## Loading the TensorFlow COTS Model to Run Inference ##\nNow that we had done the video framing of binded indexes with bounding boxes, let's load the TensorFlow COTS Model to run inference. First, let's import tensorflow, os, time, and sys modules including greatbarrierreef, and then let's set the model directory with the path of the model directory, clear session, load the saved model and finally, measure the end time with the end time variable. Thus, we have to set up the elapsed time variable and print them out to measure how long the TensorFlow COTS model load. ","metadata":{}},{"cell_type":"code","source":"import os\nimport sys\nimport tensorflow as tf\nimport time\n\n# Import the library that is used to submit the prediction result.\nINPUT_DIR = '../input/tensorflow-great-barrier-reef/'\nsys.path.insert(0, INPUT_DIR)\nimport greatbarrierreef\n\nMODEL_DIRECTORY = '../input/cots-detection-w-tensorflow-object-detection-api/cots_efficientdet_d0'\nstarting_time = time.time()\ntf.keras.backend.clear_session()\ndetect_fn_tf_odt = tf.saved_model.load(os.path.join(os.path.join(MODEL_DIRECTORY, 'output'), 'saved_model'))\nending_time = time.time()\ntotal_time = ending_time - starting_time\nprint(\"The total time is: \" + str(total_time) + \"s\")","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:08:43.114574Z","iopub.execute_input":"2021-12-22T09:08:43.114869Z","iopub.status.idle":"2021-12-22T09:09:24.373380Z","shell.execute_reply.started":"2021-12-22T09:08:43.114832Z","shell.execute_reply":"2021-12-22T09:09:24.372701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now after that, let's load images into a numpy array using a load_image_into_numpy_array function. Thus, we are going to set up detection by create a detect function (see code for further explanation).","metadata":{}},{"cell_type":"code","source":"def load_image_into_numpy_array(path):\n    \"\"\"This actually loads an image from a file to a numpy array.\n    \n    Here's how:\n    The function puts image into numpy array to feed into tensorflow graph.\n    Note that by convention, we put it into a numpy array with shape into a format of height, width, and channels (which equals to 3) for RGB.\n    \n    The Args:\n    Path: a file path (that can be local or on colosus)\n    \n    How it returns:\n    The path returns an uint8 numpy array with shape with a format containing img_height, img_width, and 3.\n    \"\"\"\n    img_data = tf.io.gfile.GFile(path, 'rb').read()\n    image = Image.open(io.BytesIO(img_data))\n    (im_width, im_height) = image.size\n    \n    return np.array(image.getdata()).reshape(\n    (im_height, im_width, 3)).astype(np.uint8)\n\ndef detect(image_np):\n    \"\"\"This function can detect the infamous COTS from a given numpy image.\"\"\"\n    \n    input_tensor = np.expand_dims(image_np, 0)\n    starting_time = time.time()\n    detections = detect_fn_tf_odt(input_tensor)\n    return detections","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:09:24.374564Z","iopub.execute_input":"2021-12-22T09:09:24.375338Z","iopub.status.idle":"2021-12-22T09:09:24.383083Z","shell.execute_reply.started":"2021-12-22T09:09:24.375300Z","shell.execute_reply":"2021-12-22T09:09:24.382056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## The Usage of The Provided Python Time-Series API To Create Submission file ##\nWithin this format of submission in a csv file way, we are going to start initialize the environment with the make_env function over greatbarrierreef in an env variable, then iterate which loops over the test set and sample submissions with the iter_test function with the iteration_test variable.","metadata":{}},{"cell_type":"code","source":"env = greatbarrierreef.make_env()  # First, initialize the environment\niteration_test = env.iter_test()  # Next, call iteration_test over an iterator which loops over the test set and sample submissions","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:09:24.384568Z","iopub.execute_input":"2021-12-22T09:09:24.384794Z","iopub.status.idle":"2021-12-22T09:09:24.405071Z","shell.execute_reply.started":"2021-12-22T09:09:24.384760Z","shell.execute_reply":"2021-12-22T09:09:24.404355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After we created our env, we will start by setting the DETECTION_THRESHOLD to 0.19, set up submission_dict as a dictionary containing id and prediction_string. Then, we will start looping iteration_test with image_np and sample_prediction_df over detection_boxes, detection_scores, and num_detections. And then after that, generate the submission data.","metadata":{}},{"cell_type":"code","source":"detection_threshold = 0.19 # Change it whatever you want...\n\nsubmission_dict = {\n    'id': [],\n    'prediction_string': [],\n}\n\nfor (image_np, sample_prediction_df) in iteration_test:\n    height, width, _ = image_np.shape\n    \n    # Run object detection using the TensorFlow model.\n    detections = detect(image_np)\n    \n    # Parse the detection result and generate a prediction string.\n    num_detections = detections['num_detections'][0].numpy().astype(np.int32)\n    predictions = []\n    for index in range(num_detections):\n        score = detections['detection_scores'][0][index].numpy()\n        if score < detection_threshold:\n            continue\n\n        bbox = detections['detection_boxes'][0][index].numpy()\n        y_min = int(bbox[0] * height)\n        x_min = int(bbox[1] * width)\n        y_max = int(bbox[2] * height)\n        x_max = int(bbox[3] * width)\n        \n        bbox_width = x_max - x_min\n        bbox_height = y_max - y_min\n        \n        predictions.append('{:.2f} {} {} {} {}'.format(score, x_min, y_min, bbox_width, bbox_height))\n    \n    # Generate the submission data.\n    prediction_str = ' '.join(predictions)\n    sample_prediction_df['annotations'] = prediction_str\n    env.predict(sample_prediction_df)\n\n    print('Predictions: ', prediction_str)","metadata":{"execution":{"iopub.status.busy":"2021-12-22T09:09:24.406534Z","iopub.execute_input":"2021-12-22T09:09:24.407724Z","iopub.status.idle":"2021-12-22T09:09:33.512544Z","shell.execute_reply.started":"2021-12-22T09:09:24.407681Z","shell.execute_reply":"2021-12-22T09:09:33.511602Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As a result, we have some predictions to observe on, and thus, a submission.csv file appeared just in sight! If we submit this file, I expect that this scored a prediction score of 0.420ish or so...","metadata":{}},{"cell_type":"markdown","source":"## Acknowledgements ##\nCredit to Aryan Lala and Diego Gomez for the EDA part and the prediction training part.\nTraining and Prediction Part: \nhttps://www.kaggle.com/aryanlala/object-detection-great-barrier-reef/notebook\nEDA Part:\nhttps://www.kaggle.com/diegoalejogm/great-barrier-reefs-eda-with-animations","metadata":{}}]}