{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n \n

Tensorflow - Help Protect the Great Barrier Reef ๐ŸŒŸ๐Ÿ 

\n
\n\n
\n

Introduction

\n

Why this Competition โ“

\n

\n Australia's stunningly beautiful Great Barrier Reef is the worldโ€™s largest coral reef and home to 1,500 species of fish, 400 species of corals, 130 species of sharks, rays, and a massive variety of other sea life.Unfortunately, the reef is under threat, in part because of the overpopulation of one particular starfish โ€“ the coral-eating crown-of-thorns starfish (or COTS for short).\n

\n

Goal of Competition ๐Ÿฅ…

\n

\n The goal of this competition is to accurately identify starfish in real-time by building an object detection model trained on underwater videos of coral reefs.\n

\n
\n\n
\n

Sponsors ๐Ÿ’ฐ

\n

TensorFlow

\n

\n TensorFlow is a free and open-source software library for machine learning and artificial intelligence. It can be used across a range of tasks but has a particular focus on training and inference of deep neural networks.\n

\n

CSIRO

\n

\n The Commonwealth Scientific and Industrial Research Organisation is an Australian Government agency responsible for scientific research. CSIRO works with leading organisations around the world.\n

\n

GBRF

\n

\n The Great Barrier Reef Foundation is an Australian non-profit organisation established in 1999 to help protect and preserve the Great Barrier Reef. \n

\n
\n\n

More Details

\n

Check competition page for details

\n

Let's dive in โฌ‡๏ธ

\n\n
","metadata":{}},{"cell_type":"markdown","source":"

Install Libraries โฌ

","metadata":{}},{"cell_type":"code","source":"import torch\ncuda_version_major = int(torch.version.cuda.split('.')[0])\n\n!wget https://raw.githubusercontent.com/airctic/icevision/master/icevision_install.sh\n!bash icevision_install.sh cuda11\n!pip install torchtext==0.11.0 --upgrade\n\nimport IPython\nIPython.Application.instance().kernel.do_shutdown(True)","metadata":{"_kg_hide-output":true,"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Import Libraries ๐Ÿ“š

","metadata":{}},{"cell_type":"code","source":"import os\nimport shutil\n\nfrom kaggle_secrets import UserSecretsClient\nimport copy \nimport time\n\nimport pandas as pd\nfrom pandas_profiling import ProfileReport\n\nimport numpy as np\nimport ast\n\nimport cv2\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nsns.set()\n\nimport torch\nfrom icevision.all import *\n\nimport wandb\n\nfrom colorama import Fore, Back, Style\n\nfrom IPython.display import IFrame\nfrom tqdm import tqdm\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2021-12-13T18:39:58.770633Z","iopub.execute_input":"2021-12-13T18:39:58.771185Z","iopub.status.idle":"2021-12-13T18:40:56.344357Z","shell.execute_reply.started":"2021-12-13T18:39:58.771091Z","shell.execute_reply":"2021-12-13T18:40:56.343376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Initialize Constants ๐Ÿ”ฐ

","metadata":{}},{"cell_type":"code","source":"TRAIN_PATH = \"../input/tensorflow-great-barrier-reef/train_images\"\nHEIGHT, WIDTH = 720, 1280\nimage_size = 640\n\nvid0_path = \"../input/tensorflow-great-barrier-reef/train_images/video_0\"\nvid1_path = \"../input/tensorflow-great-barrier-reef/train_images/video_1\"\nvid2_path = \"../input/tensorflow-great-barrier-reef/train_images/video_2\"\nvid_paths = [vid0_path, vid1_path, vid2_path]\n\nvid0_ls = [os.path.join(vid0_path,f) for f in os.listdir(vid0_path)]\nvid0_ls = sorted(vid0_ls, key=lambda x: int(\"\".join([i for i in x if i.isdigit()])))\n\nvid1_ls = [os.path.join(vid1_path,f) for f in os.listdir(vid1_path)]\nvid1_ls = sorted(vid1_ls, key=lambda x: int(\"\".join([i for i in x if i.isdigit()])))\n\nvid2_ls = [os.path.join(vid2_path,f) for f in os.listdir(vid2_path)]\nvid2_ls = sorted(vid2_ls, key=lambda x: int(\"\".join([i for i in x if i.isdigit()])))\nfiles_ls = [vid0_ls, vid1_ls, vid2_ls]\n\ntrain_df = pd.read_csv(\"../input/tensorflow-great-barrier-reef/train.csv\")","metadata":{"execution":{"iopub.status.busy":"2021-12-13T18:40:56.346751Z","iopub.execute_input":"2021-12-13T18:40:56.347132Z","iopub.status.idle":"2021-12-13T18:40:56.685688Z","shell.execute_reply.started":"2021-12-13T18:40:56.347044Z","shell.execute_reply":"2021-12-13T18:40:56.684907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Data Preprocessing โš’๏ธ๐Ÿ”ฌ

","metadata":{}},{"cell_type":"code","source":"def get_oldpath(x):\n if x.video_id == 0:\n path = os.path.join(vid0_path,str(x.video_frame)+\".jpg\")\n elif x.video_id == 1:\n path = os.path.join(vid1_path,str(x.video_frame)+\".jpg\")\n else:\n path = os.path.join(vid2_path,str(x.video_frame)+\".jpg\")\n \n return path\n\ndef get_newpath(x):\n filename = f\"{x.video_id}_{x.video_frame}.jpg\"\n return os.path.join(\"./dataset\", filename)\n\ndef get_filename(x):\n return f\"{x.video_id}_{x.video_frame}.jpg\"","metadata":{"execution":{"iopub.status.busy":"2021-12-13T18:41:04.564208Z","iopub.execute_input":"2021-12-13T18:41:04.564610Z","iopub.status.idle":"2021-12-13T18:41:04.572016Z","shell.execute_reply.started":"2021-12-13T18:41:04.564553Z","shell.execute_reply":"2021-12-13T18:41:04.571269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = train_df[train_df.annotations != \"[]\"]\ntrain_df[\"annotations\"] = train_df[\"annotations\"].map(lambda x : ast.literal_eval(x))\n\ntrain_df[\"filepath\"] = train_df.apply(lambda x : get_oldpath(x), axis=1)\ntrain_df[\"newpath\"] = train_df.apply(lambda x : get_newpath(x), axis=1)\ntrain_df[\"filename\"] = train_df.apply(lambda x : get_filename(x), axis=1)\n\nos.makedirs(\"./dataset\",exist_ok=True)\n\nfor i in tqdm(range(len(train_df))):\n src = train_df.iloc[i][\"filepath\"]\n dst = train_df.iloc[i][\"newpath\"]\n shutil.copy(src,dst)\n \ntrain_df.head(2)","metadata":{"execution":{"iopub.status.busy":"2021-12-13T18:41:04.573829Z","iopub.execute_input":"2021-12-13T18:41:04.574540Z","iopub.status.idle":"2021-12-13T18:42:02.654456Z","shell.execute_reply.started":"2021-12-13T18:41:04.574496Z","shell.execute_reply":"2021-12-13T18:42:02.653609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = train_df\ndf = df.explode(\"annotations\")\n\ndf[\"width\"] = [WIDTH]*len(df)\ndf[\"height\"] = [HEIGHT]*len(df)\ndf[\"label\"] = [\"starfish\"]*len(df)\n\ndf[\"xmin\"] = df.apply(lambda x : x.annotations[\"x\"], axis=1)\ndf[\"ymin\"] = df.apply(lambda x : x.annotations[\"y\"], axis=1)\ndf[\"xmax\"] = df.apply(lambda x : x.annotations[\"x\"]+x.annotations[\"width\"], axis=1)\ndf[\"ymax\"] = df.apply(lambda x : x.annotations[\"y\"]+x.annotations[\"height\"], axis=1)\n\ndf.loc[df[\"xmax\"] > 1280, \"xmax\"] = 1280\ndf.loc[df[\"ymax\"] > 720, \"ymax\"] = 720\n\ndf = df.drop([\"video_id\",\"sequence\",\"video_frame\",\"sequence_frame\",\"image_id\",\"annotations\",\"filepath\",\"newpath\"], axis=1)\ndf = df.reset_index(drop=True)\ndf.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-12-13T18:42:02.656785Z","iopub.execute_input":"2021-12-13T18:42:02.657297Z","iopub.status.idle":"2021-12-13T18:42:03.930485Z","shell.execute_reply.started":"2021-12-13T18:42:02.657254Z","shell.execute_reply":"2021-12-13T18:42:03.929635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Data Visualization ๐Ÿ“Š๐Ÿ’น

","metadata":{}},{"cell_type":"markdown","source":"
\n

Weights and Biases

\n
\n Checkout Dashboard created for this notebook\n

\n Weights and Biases is a set of Machine Learning tools used for experiment tracking, dataset versioning, and collaborating on ML projects. Weights and Biases is useful in many applications such as\n

\n \n Go to offocial website for more tutorials and Documentation\n
","metadata":{}},{"cell_type":"code","source":"# Check https://www.kaggle.com/debarshichanda/pytorch-w-b-pawpularity-training for more details\ntry:\n user_secrets = UserSecretsClient()\n api_key = user_secrets.get_secret(\"wandb_api\")\n wandb.login(key=api_key)\n anony = None\nexcept:\n anony = \"must\"\n print('''If you want to use your W&B account, Follow these steps :\n -> go to Add-ons {Below name of notebook} -> Secrets -> Add a new Secret\n -> Label = wandb_api\n -> Value = W&B access token from https://wandb.ai/authorize \n ''')","metadata":{"execution":{"iopub.status.busy":"2021-12-13T18:42:03.932225Z","iopub.execute_input":"2021-12-13T18:42:03.932506Z","iopub.status.idle":"2021-12-13T18:42:04.553328Z","shell.execute_reply.started":"2021-12-13T18:42:03.932466Z","shell.execute_reply":"2021-12-13T18:42:04.552463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"WANDB_CONFIG = {\n \"project_name\" : \"Protect Great Barrier Reef\",\n \"group_name\" : \"IceVision Training\",\n \"job_type_data\" : \"Data Visualization\",\n \"job_type_train\" : \"Training\",\n \"anonymity\" : \"must\",\n \"artifact\" : \"training_data\"\n}\n\nrun = wandb.init(\n project = WANDB_CONFIG[\"project_name\"],\n group = WANDB_CONFIG[\"group_name\"],\n job_type = WANDB_CONFIG[\"job_type_data\"],\n anonymous= WANDB_CONFIG[\"anonymity\"]\n)","metadata":{"execution":{"iopub.status.busy":"2021-12-13T18:42:04.554874Z","iopub.execute_input":"2021-12-13T18:42:04.555191Z","iopub.status.idle":"2021-12-13T18:42:11.234452Z","shell.execute_reply.started":"2021-12-13T18:42:04.555109Z","shell.execute_reply":"2021-12-13T18:42:11.233631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wb_table = wandb.Table(columns = [\n \"filename\", \"Image\", \"xmin\", \"ymin\", \"xmax\", \"ymax\"\n])\n\nfor i in tqdm(range(len(df))[:100]):\n row = df.loc[i]\n impath = os.path.join(\"./dataset\",row[\"filename\"])\n im = PIL.Image.open(impath)\n bbox = [{\n \"position\": {\n \"minX\": int(row['xmin']),\n \"maxX\": int(row[\"xmax\"]),\n \"minY\": int(row[\"ymin\"]),\n \"maxY\": int(row[\"ymax\"])\n },\n \"class_id\": 1,\n \"box_caption\": \"starfish\",\n \"domain\": \"pixel\"\n }]\n \n image = wandb.Image(im,\n boxes = {\n \"ground_truth\": {\n \"box_data\": bbox,\n \"class_labels\" : {1: 'starfish'}\n }\n },\n classes = [{\"id\": 0, \"name\": \"background\"}, {\"id\": 1, \"name\": \"starfish\"}]\n )\n \n wb_table.add_data(row['filename'],\n image,\n row[\"xmin\"],\n row[\"ymin\"],\n row[\"xmax\"],\n row[\"ymax\"]\n )\n \nwandb.log({'Data Visualization': wb_table})\nrun.finish()","metadata":{"execution":{"iopub.status.busy":"2021-12-13T18:42:11.236013Z","iopub.execute_input":"2021-12-13T18:42:11.236285Z","iopub.status.idle":"2021-12-13T18:42:46.881467Z","shell.execute_reply.started":"2021-12-13T18:42:11.236246Z","shell.execute_reply":"2021-12-13T18:42:46.880693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"frame = IFrame(run.url, width=1280, height=720)\nframe","metadata":{"execution":{"iopub.status.busy":"2021-12-13T18:42:46.882872Z","iopub.execute_input":"2021-12-13T18:42:46.883150Z","iopub.status.idle":"2021-12-13T18:42:46.893171Z","shell.execute_reply.started":"2021-12-13T18:42:46.883111Z","shell.execute_reply":"2021-12-13T18:42:46.891770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

Pandas Profiling

\n
\n

\n Pandas profiling is an open source Python module with which we can quickly do an exploratory data analysis with just a few lines of code. Besides, if this is not enough to convince us to use this tool, it also generates interactive reports in web format that can be presented to any person, even if they donโ€™t know programming.\n

\n Go to offocial website for documentation\n
","metadata":{}},{"cell_type":"code","source":"train_report = ProfileReport(df,title=\"Metadata of Training images\")\ntrain_report.to_file(\"./train_metadata.html\")\ntrain_report","metadata":{"execution":{"iopub.status.busy":"2021-12-13T18:42:46.894592Z","iopub.execute_input":"2021-12-13T18:42:46.895349Z","iopub.status.idle":"2021-12-13T18:43:03.830035Z","shell.execute_reply.started":"2021-12-13T18:42:46.895300Z","shell.execute_reply":"2021-12-13T18:43:03.829328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

IceVision

\n
\n

\n IceVision is the first agnostic computer vision framework to offer a curated collection with hundreds of high-quality pre-trained models from torchvision, MMLabs, and soon Pytorch Image Models. It orchestrates the end-to-end deep learning workflow allowing to train networks with easy-to-use robust high-performance libraries such as Pytorch-Lightning and Fastai

\n IceVision Unique Features:\n \n Go to offocial website for documentation\n
","metadata":{}},{"cell_type":"markdown","source":"

Modelling ๐Ÿซ

","metadata":{}},{"cell_type":"markdown","source":"

Data Parser

","metadata":{}},{"cell_type":"code","source":"class PGBRParser(Parser):\n def __init__(self, template_record, data_dir, df):\n super().__init__(template_record=template_record)\n\n self.data_dir = data_dir\n self.df = df\n self.class_map = ClassMap(list(self.df['label'].unique()))\n\n def __iter__(self) -> Any:\n for o in self.df.itertuples():\n yield o\n\n def __len__(self) -> int:\n return len(self.df)\n\n def record_id(self, o) -> Hashable:\n return o.filename\n\n def parse_fields(self, o, record, is_new):\n if is_new:\n record.set_filepath(os.path.join(self.data_dir,o.filename))\n record.set_img_size(ImgSize(width=o.width, height=o.height))\n record.detection.set_class_map(self.class_map)\n\n record.detection.add_bboxes([BBox.from_xyxy(o.xmin, o.ymin, o.xmax, o.ymax)])\n record.detection.add_labels([o.label])","metadata":{"execution":{"iopub.status.busy":"2021-12-13T18:43:03.832195Z","iopub.execute_input":"2021-12-13T18:43:03.832425Z","iopub.status.idle":"2021-12-13T18:43:03.844327Z","shell.execute_reply.started":"2021-12-13T18:43:03.832395Z","shell.execute_reply":"2021-12-13T18:43:03.842461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Data Loader

","metadata":{}},{"cell_type":"code","source":"template_record = ObjectDetectionRecord()\nparser = PGBRParser(template_record, \"./dataset\", df)\n\ntrain_records, valid_records = parser.parse()\nprint(parser.class_map)\n\ntrain_tfms = tfms.A.Adapter([*tfms.A.aug_tfms(size=image_size, presize=512), tfms.A.Normalize()])\nvalid_tfms = tfms.A.Adapter([*tfms.A.resize_and_pad(image_size), tfms.A.Normalize()])\n\ntrain_ds = Dataset(train_records, train_tfms)\nvalid_ds = Dataset(valid_records, valid_tfms)\n\nsamples = [train_ds[0] for _ in range(3)]\nshow_samples(samples, ncols=3)","metadata":{"execution":{"iopub.status.busy":"2021-12-13T18:43:03.845883Z","iopub.execute_input":"2021-12-13T18:43:03.846546Z","iopub.status.idle":"2021-12-13T18:43:10.646633Z","shell.execute_reply.started":"2021-12-13T18:43:03.846493Z","shell.execute_reply":"2021-12-13T18:43:10.645737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_type = models.mmdet.vfnet\nbackbone = model_type.backbones.resnet50_fpn_mstrain_2x(pretrained=True)\nmodel = model_type.model(backbone=backbone(pretrained=True), num_classes=len(parser.class_map)) \n\ntrain_dl = model_type.train_dl(train_ds, batch_size=16, num_workers=4, shuffle=True)\nvalid_dl = model_type.valid_dl(valid_ds, batch_size=16, num_workers=4, shuffle=False)","metadata":{"_kg_hide-output":true,"scrolled":true,"execution":{"iopub.status.busy":"2021-12-13T18:43:38.653545Z","iopub.execute_input":"2021-12-13T18:43:38.653882Z","iopub.status.idle":"2021-12-13T18:43:57.948924Z","shell.execute_reply.started":"2021-12-13T18:43:38.653845Z","shell.execute_reply":"2021-12-13T18:43:57.947912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Metrics

","metadata":{}},{"cell_type":"code","source":"metrics = [COCOMetric(metric_type=COCOMetricType.bbox)]","metadata":{"execution":{"iopub.status.busy":"2021-12-13T18:44:03.327353Z","iopub.execute_input":"2021-12-13T18:44:03.327652Z","iopub.status.idle":"2021-12-13T18:44:03.332340Z","shell.execute_reply.started":"2021-12-13T18:44:03.327617Z","shell.execute_reply":"2021-12-13T18:44:03.331397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Training

","metadata":{}},{"cell_type":"code","source":"learn = model_type.fastai.learner(dls=[train_dl, valid_dl], model=model, metrics=metrics)\nlearn.fine_tune(20, 0.005, freeze_epochs=1)","metadata":{"execution":{"iopub.status.busy":"2021-12-13T18:44:08.046361Z","iopub.execute_input":"2021-12-13T18:44:08.046663Z","iopub.status.idle":"2021-12-13T21:22:38.480701Z","shell.execute_reply.started":"2021-12-13T18:44:08.046612Z","shell.execute_reply":"2021-12-13T21:22:38.479671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Inference ๐Ÿ”ฎ

","metadata":{}},{"cell_type":"code","source":"infer_dl = model_type.infer_dl([valid_ds[0],valid_ds[4],valid_ds[7],valid_ds[9]], batch_size=4, shuffle=False)\npreds = model_type.predict_from_dl(model, infer_dl, keep_images=True)\nshow_preds(preds=preds)","metadata":{"execution":{"iopub.status.busy":"2021-12-13T21:22:38.483107Z","iopub.execute_input":"2021-12-13T21:22:38.483730Z","iopub.status.idle":"2021-12-13T21:22:40.844405Z","shell.execute_reply.started":"2021-12-13T21:22:38.483663Z","shell.execute_reply":"2021-12-13T21:22:40.843606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"shutil.make_archive('./checkpoints',\n 'zip',\n './',\n 'checkpoints')\n\nshutil.rmtree(\"./checkpoints\")\nshutil.rmtree(\"./dataset\")\nshutil.rmtree(\"./wandb\")","metadata":{"execution":{"iopub.status.busy":"2021-12-13T21:22:40.848691Z","iopub.execute_input":"2021-12-13T21:22:40.849147Z","iopub.status.idle":"2021-12-13T21:22:57.503833Z","shell.execute_reply.started":"2021-12-13T21:22:40.849110Z","shell.execute_reply":"2021-12-13T21:22:57.502789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Download File ","metadata":{}},{"cell_type":"markdown","source":"
\n

If you come this far, you could've got some insights from this notebook. An upvote would be very helpful :). Kindly comment if there are any doubts or mistakes

\n\n
","metadata":{}}]}