{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"According to The Great Barrier Reef Foundation, COTS are an important part of a healthy ecosystem in a reasonable number on healthy coral reefs since they advance the coral diversity of the reefs. However, when they appear in mass, they can negatively impact the life of the coral reef. \nIt is the reason why we would like to develop a computer vision model to detect these COTS and save the reef from outbreaks. \n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-11-30T22:48:29.717956Z","iopub.execute_input":"2021-11-30T22:48:29.718291Z","iopub.status.idle":"2021-11-30T22:48:30.615881Z","shell.execute_reply.started":"2021-11-30T22:48:29.718256Z","shell.execute_reply":"2021-11-30T22:48:30.615257Z"}}},{"cell_type":"markdown","source":"Our EDA has pointed out some considerations to build our model. Thanks, [Diego Gomez](https://www.kaggle.com/diegoalejogm), for the [animations](https://www.kaggle.com/diegoalejogm/great-barrier-reefs-eda-with-animations). Thanks, [Prabhakaran D](https://www.kaggle.com/get2jawa),for the [basic and simple EDA](https://www.kaggle.com/get2jawa/great-barrier-reefs-basic-simple-eda), Thanks, [Shijin Yang](https://www.kaggle.com/sjyangkevin), for sharing your [EDA, Bouding Box Analysis & Annotated Videos](https://www.kaggle.com/sjyangkevin/eda-bouding-box-analysis-annotated-videos) notebook. ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns \nfrom matplotlib.image import imread\nimport ast\nimport os","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:26.475631Z","iopub.execute_input":"2021-12-07T02:59:26.476069Z","iopub.status.idle":"2021-12-07T02:59:27.507148Z","shell.execute_reply.started":"2021-12-07T02:59:26.476017Z","shell.execute_reply":"2021-12-07T02:59:27.506154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Load data\nmeta_df = pd.read_csv('../input/tensorflow-great-barrier-reef/train.csv')\nmeta_df_test = pd.read_csv('../input/tensorflow-great-barrier-reef/test.csv')","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:27.508714Z","iopub.execute_input":"2021-12-07T02:59:27.509314Z","iopub.status.idle":"2021-12-07T02:59:27.582983Z","shell.execute_reply.started":"2021-12-07T02:59:27.509275Z","shell.execute_reply":"2021-12-07T02:59:27.582031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"  **1. EDA of META-DATA**","metadata":{}},{"cell_type":"code","source":"meta_df.shape","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:27.584436Z","iopub.execute_input":"2021-12-07T02:59:27.584841Z","iopub.status.idle":"2021-12-07T02:59:27.59218Z","shell.execute_reply.started":"2021-12-07T02:59:27.584805Z","shell.execute_reply":"2021-12-07T02:59:27.591344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:27.594817Z","iopub.execute_input":"2021-12-07T02:59:27.595427Z","iopub.status.idle":"2021-12-07T02:59:27.613766Z","shell.execute_reply.started":"2021-12-07T02:59:27.595381Z","shell.execute_reply":"2021-12-07T02:59:27.612754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:27.615038Z","iopub.execute_input":"2021-12-07T02:59:27.615924Z","iopub.status.idle":"2021-12-07T02:59:27.633299Z","shell.execute_reply.started":"2021-12-07T02:59:27.615887Z","shell.execute_reply":"2021-12-07T02:59:27.632084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df.describe(include='all')\n#Could perform imputation here with NaN value","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:27.634625Z","iopub.execute_input":"2021-12-07T02:59:27.634959Z","iopub.status.idle":"2021-12-07T02:59:27.693966Z","shell.execute_reply.started":"2021-12-07T02:59:27.634842Z","shell.execute_reply":"2021-12-07T02:59:27.69307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df.info()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:27.695421Z","iopub.execute_input":"2021-12-07T02:59:27.695916Z","iopub.status.idle":"2021-12-07T02:59:27.716057Z","shell.execute_reply.started":"2021-12-07T02:59:27.695873Z","shell.execute_reply":"2021-12-07T02:59:27.714675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df_cat = meta_df.select_dtypes(include = 'object').copy()\nmeta_df_cat.nunique()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:27.718058Z","iopub.execute_input":"2021-12-07T02:59:27.718559Z","iopub.status.idle":"2021-12-07T02:59:27.73714Z","shell.execute_reply.started":"2021-12-07T02:59:27.718499Z","shell.execute_reply":"2021-12-07T02:59:27.736429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df.duplicated().sum()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:27.738363Z","iopub.execute_input":"2021-12-07T02:59:27.738792Z","iopub.status.idle":"2021-12-07T02:59:27.756481Z","shell.execute_reply.started":"2021-12-07T02:59:27.73876Z","shell.execute_reply":"2021-12-07T02:59:27.75582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:27.758852Z","iopub.execute_input":"2021-12-07T02:59:27.759238Z","iopub.status.idle":"2021-12-07T02:59:27.770504Z","shell.execute_reply.started":"2021-12-07T02:59:27.759207Z","shell.execute_reply":"2021-12-07T02:59:27.76944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are no null values.","metadata":{}},{"cell_type":"code","source":"from os import listdir\nfrom PIL import Image\n\n\"\"\"\nVerifying images are valid\n\"\"\"\ndef validated_images(video_id):\n    path = '../input/tensorflow-great-barrier-reef/train_images'.format(video_id)\n    print ('Verifying that video {} frames are valid...'.format(video_id))\n    for filename in listdir(path):\n        if '.jpg' in filename:\n            try:\n                image = Image.open(path+filename)\n                image.verify()\n            except (IOError, SyntaxError) as e:\n                print ('Bad file:', filename)\n            print ('Verfied! Video {} has all valid images'.format(video_id))\nfor video_id in range(3):\n    validated_images(video_id)","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:27.772415Z","iopub.execute_input":"2021-12-07T02:59:27.773614Z","iopub.status.idle":"2021-12-07T02:59:27.791733Z","shell.execute_reply.started":"2021-12-07T02:59:27.773454Z","shell.execute_reply":"2021-12-07T02:59:27.790631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sequence_counts = meta_df['sequence'].value_counts().sort_values().reset_index()\nsequence_counts.columns = [['sequence', 'num_frames']]\nprint (\"number of sequences:\", len(sequence_counts))\nsequence_counts.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:27.792721Z","iopub.execute_input":"2021-12-07T02:59:27.793262Z","iopub.status.idle":"2021-12-07T02:59:27.811586Z","shell.execute_reply.started":"2021-12-07T02:59:27.793221Z","shell.execute_reply":"2021-12-07T02:59:27.81056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_obj_wt_frame = meta_df[meta_df.annotations =='[]']['annotations'].count()\nprint (\"Number of frames without objects:\" , num_obj_wt_frame)","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:27.813228Z","iopub.execute_input":"2021-12-07T02:59:27.813766Z","iopub.status.idle":"2021-12-07T02:59:27.832369Z","shell.execute_reply.started":"2021-12-07T02:59:27.813719Z","shell.execute_reply":"2021-12-07T02:59:27.831449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_obj_with_frame = meta_df[meta_df.annotations != '[]']['annotations'].count()\nprint (\"Number of frames with objects:\",num_obj_with_frame)","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:27.833838Z","iopub.execute_input":"2021-12-07T02:59:27.834099Z","iopub.status.idle":"2021-12-07T02:59:27.845289Z","shell.execute_reply.started":"2021-12-07T02:59:27.834069Z","shell.execute_reply":"2021-12-07T02:59:27.844245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Showing what images can look like\nfolder = '../input/tensorflow-great-barrier-reef/train_images/video_0/'\nfor i in range(9):\n    plt.subplot(330 + 1 + i)\n    filename = folder + str(i) + '.jpg'\n    image = imread(filename)\n    plt.imshow(image)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:27.846498Z","iopub.execute_input":"2021-12-07T02:59:27.846898Z","iopub.status.idle":"2021-12-07T02:59:30.243338Z","shell.execute_reply.started":"2021-12-07T02:59:27.846867Z","shell.execute_reply":"2021-12-07T02:59:30.242632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df[meta_df.annotations != '[]'].head()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.244603Z","iopub.execute_input":"2021-12-07T02:59:30.245512Z","iopub.status.idle":"2021-12-07T02:59:30.260332Z","shell.execute_reply.started":"2021-12-07T02:59:30.24547Z","shell.execute_reply":"2021-12-07T02:59:30.259497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_obj_wt_frame = meta_df[meta_df.annotations =='[]']['annotations'].count()\nprint (\"Number of frames without objects:\" , num_obj_wt_frame)","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.261762Z","iopub.execute_input":"2021-12-07T02:59:30.263779Z","iopub.status.idle":"2021-12-07T02:59:30.274469Z","shell.execute_reply.started":"2021-12-07T02:59:30.263728Z","shell.execute_reply":"2021-12-07T02:59:30.273478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_obj_with_frame = meta_df[meta_df.annotations != '[]']['annotations'].count()\nprint (\"Number of frames with objects:\",num_obj_with_frame)","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.275795Z","iopub.execute_input":"2021-12-07T02:59:30.276738Z","iopub.status.idle":"2021-12-07T02:59:30.287465Z","shell.execute_reply.started":"2021-12-07T02:59:30.27669Z","shell.execute_reply":"2021-12-07T02:59:30.286615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('ratio of frames with objects:', num_obj_with_frame / len(meta_df))\n\nfig, axes = plt.subplots(1,1, figsize=(12, 6))\n\nsns.barplot(ax=axes, x=['Number of Frames with Objects', 'Number of Frames with No Objects'], y=[num_obj_with_frame, num_obj_wt_frame])\naxes.set_title(\"Distribution of Frames with/without Objects\")\naxes.set_xlabel(\"Frame Types\")\naxes.set_ylabel(\"Count\")\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.288964Z","iopub.execute_input":"2021-12-07T02:59:30.289916Z","iopub.status.idle":"2021-12-07T02:59:30.502798Z","shell.execute_reply.started":"2021-12-07T02:59:30.289859Z","shell.execute_reply":"2021-12-07T02:59:30.502094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most frames do not have objects in them.","metadata":{}},{"cell_type":"code","source":"frame_counts = meta_df['video_id'].value_counts().sort_values().to_frame()\nframe_counts.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.504021Z","iopub.execute_input":"2021-12-07T02:59:30.504278Z","iopub.status.idle":"2021-12-07T02:59:30.518011Z","shell.execute_reply.started":"2021-12-07T02:59:30.504239Z","shell.execute_reply":"2021-12-07T02:59:30.516697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df[meta_df.annotations.str.len()>2]","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.52245Z","iopub.execute_input":"2021-12-07T02:59:30.52343Z","iopub.status.idle":"2021-12-07T02:59:30.554378Z","shell.execute_reply.started":"2021-12-07T02:59:30.523355Z","shell.execute_reply":"2021-12-07T02:59:30.553148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df.annotations.map(len).value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.555769Z","iopub.execute_input":"2021-12-07T02:59:30.556645Z","iopub.status.idle":"2021-12-07T02:59:30.573667Z","shell.execute_reply.started":"2021-12-07T02:59:30.556603Z","shell.execute_reply":"2021-12-07T02:59:30.572781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df ['video_id'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.57476Z","iopub.execute_input":"2021-12-07T02:59:30.57565Z","iopub.status.idle":"2021-12-07T02:59:30.583118Z","shell.execute_reply.started":"2021-12-07T02:59:30.575601Z","shell.execute_reply":"2021-12-07T02:59:30.582271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df['sequence'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.584197Z","iopub.execute_input":"2021-12-07T02:59:30.584443Z","iopub.status.idle":"2021-12-07T02:59:30.598467Z","shell.execute_reply.started":"2021-12-07T02:59:30.584405Z","shell.execute_reply":"2021-12-07T02:59:30.597845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_theme(style='whitegrid')\nax=sns.countplot(x='video_id',data=meta_df)","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.599578Z","iopub.execute_input":"2021-12-07T02:59:30.600299Z","iopub.status.idle":"2021-12-07T02:59:30.827325Z","shell.execute_reply.started":"2021-12-07T02:59:30.600254Z","shell.execute_reply":"2021-12-07T02:59:30.825808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df_test.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.829066Z","iopub.execute_input":"2021-12-07T02:59:30.829324Z","iopub.status.idle":"2021-12-07T02:59:30.841018Z","shell.execute_reply.started":"2021-12-07T02:59:30.829295Z","shell.execute_reply":"2021-12-07T02:59:30.839711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df_test.shape","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.842471Z","iopub.execute_input":"2021-12-07T02:59:30.842879Z","iopub.status.idle":"2021-12-07T02:59:30.853919Z","shell.execute_reply.started":"2021-12-07T02:59:30.842837Z","shell.execute_reply":"2021-12-07T02:59:30.852944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**2. VISUALISTION OF IMAGES**","metadata":{}},{"cell_type":"code","source":"\nfrom os import listdir\nfrom PIL import Image\n\"\"\"\nVerifying images are valid\n\"\"\"\ndef validate_images(video_id):\n    path = '../input/tensorflow-great-barrier-reef/train_images'.format(video_id)\n    print ('Verifying that video {} frames are valid...'.format(video_id))\n    for filename in listdir(path):\n        if '.jpg'in filename:\n            try:\n                image = Image.open(path+filename)\n                image.verify()\n            except(IOError,SyntaxError) as e:\n                print ('Bad file:', filename)\n    print('Verified! Video {} has all valid images'.format(video_id))\nfor video_id in range(3):\n    validate_images(video_id)","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.857582Z","iopub.execute_input":"2021-12-07T02:59:30.85798Z","iopub.status.idle":"2021-12-07T02:59:30.871377Z","shell.execute_reply.started":"2021-12-07T02:59:30.857907Z","shell.execute_reply":"2021-12-07T02:59:30.870771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Images of COTS for which we have annotations as []\n\nfor i in range(3):\n    print (\"Video\" + str(i))\n    print (\"Frames with annotations:\" + str((meta_df[meta_df['video_id'] !=i] ['annotations'] != '[]').sum()))\n    print (\"Frames without annotations: \" +str((meta_df[meta_df['video_id']==i]['annotations']!= '[]').sum()))\n    print(\"-------\")","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.872513Z","iopub.execute_input":"2021-12-07T02:59:30.873175Z","iopub.status.idle":"2021-12-07T02:59:30.910661Z","shell.execute_reply.started":"2021-12-07T02:59:30.873135Z","shell.execute_reply":"2021-12-07T02:59:30.909632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#load sequence of images with annotations\nfrom PIL import Image, ImageDraw\nimport numpy as np\n\ndef fetch_image_list(df_tmp, video_id, num_images, start_frame_idx):\n    \n    '''\n    Load sequence of images with annotations\n    '''\n    def fetch_image(frame_id):\n        path_base = '../input/tensorflow-great-barrier-reef/train_images/video_{}/{}.jpg'\n        raw_image = Image.open(path_base.format(video_id, frame_id))\n\n        row_frame = df_tmp[(df_tmp.video_id == video_id) & (df_tmp.video_frame == frame_id)].iloc[0]\n        bounding_boxes = ast.literal_eval(row_frame.annotations)\n\n        for box in bounding_boxes:\n            draw = ImageDraw.Draw(raw_image)\n            x0, y0, x1, y1 = (box['x'], box['y'], box['x']+box['width'], box['y']+box['height'])\n            draw.rectangle( (x0, y0, x1, y1), outline=180, width=3)\n        return raw_image\n\n    return [np.array(fetch_image(start_frame_idx + index)) for index in range(num_images)]\n\nimages = fetch_image_list(meta_df, video_id = 0, num_images = 80, start_frame_idx = 25)\n\nprint(\"Num images: \", len(images))\nplt.imshow(images[0], interpolation='nearest')\nplt.axis('off')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:30.912567Z","iopub.execute_input":"2021-12-07T02:59:30.914101Z","iopub.status.idle":"2021-12-07T02:59:35.826157Z","shell.execute_reply.started":"2021-12-07T02:59:30.914048Z","shell.execute_reply":"2021-12-07T02:59:35.825269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from matplotlib import animation, rc\nrc('animation',html = 'jshtml')\n\ndef create_animation(ims):\n    fig = plt.figure(figsize=(9,9))\n    plt.axis('off')\n    im = plt.imshow(ims[0])\n    \n    def animate_func(i):\n        im.set_array(ims[i])\n        return [im]\n    return animation.FuncAnimation(fig,animate_func, frames=len(ims),\n                                  interval=1000/12)\n\ncreate_animation(images)","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:35.827448Z","iopub.execute_input":"2021-12-07T02:59:35.827797Z","iopub.status.idle":"2021-12-07T02:59:48.85484Z","shell.execute_reply.started":"2021-12-07T02:59:35.827749Z","shell.execute_reply":"2021-12-07T02:59:48.853784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Check the meta-data against image data ","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:48.856124Z","iopub.execute_input":"2021-12-07T02:59:48.856388Z","iopub.status.idle":"2021-12-07T02:59:48.860748Z","shell.execute_reply.started":"2021-12-07T02:59:48.856356Z","shell.execute_reply":"2021-12-07T02:59:48.859605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#loading images\nvideo_0 = os.listdir('../input/tensorflow-great-barrier-reef/train_images/video_0')\nvideo_1 = os.listdir('../input/tensorflow-great-barrier-reef/train_images/video_1')\nvideo_2 = os.listdir('../input/tensorflow-great-barrier-reef/train_images/video_2')","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:48.862079Z","iopub.execute_input":"2021-12-07T02:59:48.862428Z","iopub.status.idle":"2021-12-07T02:59:49.99962Z","shell.execute_reply.started":"2021-12-07T02:59:48.862397Z","shell.execute_reply":"2021-12-07T02:59:49.99885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Creating the image paths\nmeta_df['image_path'] = \"video_\"+ meta_df['video_id'].astype(str)+'/'+ meta_df['video_frame'].astype(str)+\".jpg\"\nmeta_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:50.002534Z","iopub.execute_input":"2021-12-07T02:59:50.002817Z","iopub.status.idle":"2021-12-07T02:59:50.063008Z","shell.execute_reply.started":"2021-12-07T02:59:50.002788Z","shell.execute_reply":"2021-12-07T02:59:50.062332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def show_image(meta_df, idx):\n    f_name = meta_df.iloc[idx]['image_path']\n    return PIL.Image.open(\"../input/tensorflow-great-barrier-reef/train_images/\"+f_name)","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:50.064121Z","iopub.execute_input":"2021-12-07T02:59:50.06477Z","iopub.status.idle":"2021-12-07T02:59:50.069584Z","shell.execute_reply.started":"2021-12-07T02:59:50.064733Z","shell.execute_reply":"2021-12-07T02:59:50.068433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Finding out whether all the image paths are existing regular files \nmeta_df['image_path'].apply(lambda f_name:os.path.isfile(\"../input/tensorflow-great-barrier-reef/train_images/\"+f_name)).all()","metadata":{"execution":{"iopub.status.busy":"2021-12-07T02:59:50.071023Z","iopub.execute_input":"2021-12-07T02:59:50.072006Z","iopub.status.idle":"2021-12-07T03:00:31.654854Z","shell.execute_reply.started":"2021-12-07T02:59:50.071956Z","shell.execute_reply":"2021-12-07T03:00:31.654005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import PIL\nshow_image(meta_df,0).resize((400,256))","metadata":{"execution":{"iopub.status.busy":"2021-12-07T03:00:31.6563Z","iopub.execute_input":"2021-12-07T03:00:31.65724Z","iopub.status.idle":"2021-12-07T03:00:31.738649Z","shell.execute_reply.started":"2021-12-07T03:00:31.657202Z","shell.execute_reply":"2021-12-07T03:00:31.737294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Check the npy files","metadata":{"execution":{"iopub.status.busy":"2021-12-07T03:00:31.740089Z","iopub.execute_input":"2021-12-07T03:00:31.740731Z","iopub.status.idle":"2021-12-07T03:00:31.745566Z","shell.execute_reply.started":"2021-12-07T03:00:31.740691Z","shell.execute_reply":"2021-12-07T03:00:31.74463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pixels = np.load(\"../input/tensorflow-great-barrier-reef/example_test.npy\")","metadata":{"execution":{"iopub.status.busy":"2021-12-07T03:00:31.747195Z","iopub.execute_input":"2021-12-07T03:00:31.747428Z","iopub.status.idle":"2021-12-07T03:00:32.132791Z","shell.execute_reply.started":"2021-12-07T03:00:31.747402Z","shell.execute_reply":"2021-12-07T03:00:32.131687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pixels.shape","metadata":{"execution":{"iopub.status.busy":"2021-12-07T03:00:32.13457Z","iopub.execute_input":"2021-12-07T03:00:32.13523Z","iopub.status.idle":"2021-12-07T03:00:32.142012Z","shell.execute_reply.started":"2021-12-07T03:00:32.135181Z","shell.execute_reply":"2021-12-07T03:00:32.1414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"PIL.Image.fromarray(pixels[0, :]).resize((400, 256))","metadata":{"execution":{"iopub.status.busy":"2021-12-07T03:00:32.142985Z","iopub.execute_input":"2021-12-07T03:00:32.14404Z","iopub.status.idle":"2021-12-07T03:00:32.229478Z","shell.execute_reply.started":"2021-12-07T03:00:32.143988Z","shell.execute_reply":"2021-12-07T03:00:32.228469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"PIL.Image.fromarray(pixels[2, :]).resize((400, 256))","metadata":{"execution":{"iopub.status.busy":"2021-12-07T03:00:32.231514Z","iopub.execute_input":"2021-12-07T03:00:32.232062Z","iopub.status.idle":"2021-12-07T03:00:32.302929Z","shell.execute_reply.started":"2021-12-07T03:00:32.232027Z","shell.execute_reply":"2021-12-07T03:00:32.301861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Evaluate annotation strings containing literals\nast.literal_eval(meta_df.iloc[16].annotations)","metadata":{"execution":{"iopub.status.busy":"2021-12-07T03:00:32.304815Z","iopub.execute_input":"2021-12-07T03:00:32.305064Z","iopub.status.idle":"2021-12-07T03:00:32.312961Z","shell.execute_reply.started":"2021-12-07T03:00:32.305036Z","shell.execute_reply":"2021-12-07T03:00:32.312069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Change the annotations into a list \nmeta_df ['annotations']= meta_df['annotations'].apply(eval)\nmeta_df[meta_df['annotations'].str.len()>1].iloc[0]['annotations']","metadata":{"execution":{"iopub.status.busy":"2021-12-07T03:00:32.314178Z","iopub.execute_input":"2021-12-07T03:00:32.31441Z","iopub.status.idle":"2021-12-07T03:00:32.545399Z","shell.execute_reply.started":"2021-12-07T03:00:32.314382Z","shell.execute_reply":"2021-12-07T03:00:32.544409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Distribution of boxes in frames\nmeta_df['n_objects'] = meta_df['annotations'].str.len()\nmeta_df.value_counts('n_objects').plot.bar(figsize=(10,5),alpha=0.5,rot=0,title='Distribution of boxes in frames');","metadata":{"execution":{"iopub.status.busy":"2021-12-07T03:00:32.546905Z","iopub.execute_input":"2021-12-07T03:00:32.547529Z","iopub.status.idle":"2021-12-07T03:00:32.970616Z","shell.execute_reply.started":"2021-12-07T03:00:32.547492Z","shell.execute_reply":"2021-12-07T03:00:32.968945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The hope for this competition is that we can find a better way to tame COTS outbreaks ethically and effectively. Maybe in the future we will not have to kill a massive number of COTS.","metadata":{}},{"cell_type":"markdown","source":"Please feel free to ask any question or give us any feedback on our work. Thank you. ","metadata":{}}]}