{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# AI4NTD KK2.0 P1.5 STH & SCHm Notebook","metadata":{"_uuid":"dee50805-409c-4cf2-b5a1-8b9c457eb658","_cell_guid":"a0940e67-b5bd-4f4e-8722-6485fe6d99fb","trusted":true}},{"cell_type":"markdown","source":"This notebook contains code to train a Soil-Transmitted Helminth (STH) and Schistosoma mansoni (SCH) detection model to serve as a baseline model for our dataset [AI4NTD P1.5 STH SCH](https://www.kaggle.com/peterkward/ai4ntd-p1-5).\n\nWe provide examples on how to explore the dataset by getting summaries and plotting images.\n\nWe show how the [TensorFlow Object Detection API](https://github.com/tensorflow/models/tree/master/research/object_detection) can be used to apply transfer learning on an [EfficientDet-D0](https://arxiv.org/abs/1911.09070) pretrained model.\n\nFinally we evaluate the performance of a selection of models from the TensorFlow model zoo.\n\nThis notebook was based on the [COTS detection w/ TensorFlow Object Detection API](https://www.kaggle.com/khanhlvg/cots-detection-w-tensorflow-object-detection-api) by Kahn and Ryan Kolbrook.","metadata":{"_uuid":"22ee0484-3fb8-478c-90fa-065fc338f7ce","_cell_guid":"c993d333-68c4-4f1b-ae87-c7f86c74322f","trusted":true}},{"cell_type":"markdown","source":"## Notebook overview for new users\n\nThis notebook takes about 20 minutes to run from start to end. \n\nBegin by saving a copy of the notebook with the **Copy & Edit** button (top right). This will copy the notebook and the linked data to your **'Your Work'** space.\n\nThis notebook contains **'Code'** cells and **'Markdown'** cells. **'Code'** cells can be executed one at a time by pressing the play symbol or **'Ctrl' + 'Enter'**, or all cells by using **'Run All'** or **'Ctrl' + 'Alt' + 'Shift' + 'Enter'** to run all cells. \n\nNotice on the right hand side, under the **Data** menu, you can find the input data used in this demo notebook [AI4NTD P1.5 STH SCH](https://www.kaggle.com/peterkward/ai4ntd-p1-5) \n\nThe input data includes:\n- Three TFRECORDS (train, eval, test) containing the images and metadata used to generate the detection model\n- A 'egg_label_map.pbtxt' which specifies the mapping of eggs classes to ids as used in the TFRECORDS\n- A 'pipeline.config' which specifies the configuration for training the model\n- RFCN Resnet model used in the [paper](paper link) developed in TF 1.13\n- Faster RCNN Resnet 50/100 models also trained on the provided dataset\n\nFiles in the input directory are read-only.\n\nOther files that are downloaded, generated or used in this notebook will be available in your */kaggle/working* directory.\n\nThe **'Settings'** menu on the right hand side allows you to configure settings applied the the notebook session. Check that these are configure before proceeding. \n- Language: Python\n- Environment: Pin to Original Environment\n- Accelerator: GPU *(You need to verify your account before you can enable this)*\n- Internet: Must be enabled (Ticked) *(You need to verify your account before you can enable this)*\n\nPressing **\"Save Version\"** then **'Save & Run All'** will step through all **'Code'** cells sequentially and save all output data into the */kaggle/working* directory. This is recommended as a first step to ensure the notebook is working from start to end. By default this notebook uses parameters TRAINING_STEPS, WARMUP_STEPS and BATCH_SIZE all equal to 1. Once you have tested the notebook you will need to update these to the default values for meaningful results. Note, that using the default values to achieve meaningful results, the training time will take about 48hours which is not possible with the free resources available on Kaggle.","metadata":{"_uuid":"36acd5b5-ee17-49d2-9fbf-9df8e6bceaa0","_cell_guid":"ce71d58a-7b50-48bf-b8e2-dc69880d13fc","trusted":true}},{"cell_type":"markdown","source":"### Summary\n\nPress **'Copy & Edit**' <br />\nSelect **'Run All'** <br />\nWait for the notebook to run through all the cells.\nWhile the cells are executing, browse through the notebook and follow progress as indicated by loading signs on each cell. After each cell has been evaluated, the output will be printed below the cell. Running the notebook will take about 25mins.\nOccasionally, the notebook fails to run the whole way through - this is due to the memory limitations of the Kaggle notebook and the size and quantity of our training dataset. If this occurs, you will notice a red banner with a memory warning, try perform a 'Factory Reset' and run all the cells again. ","metadata":{"_uuid":"6809919e-b74b-4dba-9daf-38805b753003","_cell_guid":"240a9917-aead-4320-a5c8-81d16a4234dc","trusted":true}},{"cell_type":"markdown","source":"# TensorFlow Object Detection API setup\n\nThe notebook session already includes many useful libraries. Code used in the notebook is for Python 3.\n\nWe will begin by downloading and setting up additional libraries for TensorFlow Object Detection (TF OD API).","metadata":{"_uuid":"428f4da0-a7bc-4afa-9b6b-bb7f2e440e01","_cell_guid":"55950dd6-f555-4137-9cfa-a7804b63b7cd","trusted":true}},{"cell_type":"code","source":"# Download TensorFlow model libraries\n!git clone https://github.com/tensorflow/models\n# Check out a certain commit to ensure that future changes in the TF OD API codebase won't affect this notebook.\n!cd models && git checkout ac8d06519","metadata":{"_uuid":"c3cfa762-502b-4dbc-89f1-034b8386d90a","_cell_guid":"a3a8e0c7-5059-46be-a977-fc0ff1427d2f","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:08:07.206018Z","iopub.execute_input":"2022-02-22T15:08:07.206823Z","iopub.status.idle":"2022-02-22T15:08:30.997014Z","shell.execute_reply.started":"2022-02-22T15:08:07.206718Z","shell.execute_reply":"2022-02-22T15:08:30.996161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%bash\n\ncd /kaggle/working/models/research\n# Compile protos.\nprotoc object_detection/protos/*.proto --python_out=.\n# Install TensorFlow Object Detection API with fixes.\nwget https://storage.googleapis.com/odml-dataset/others/setup.py\npip install -q --user .\n\n# You can optionally test the Object Dectection API is working correctly\n#python object_detection/builders/model_builder_tf2_test.py","metadata":{"_uuid":"ed8c24aa-1547-491b-a291-1283f4ad3f33","_cell_guid":"b5fc63a9-02df-4e48-a780-0d03b68fb65c","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:08:30.999265Z","iopub.execute_input":"2022-02-22T15:08:30.999637Z","iopub.status.idle":"2022-02-22T15:09:30.202706Z","shell.execute_reply.started":"2022-02-22T15:08:30.999596Z","shell.execute_reply":"2022-02-22T15:09:30.201756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Import dependencies","metadata":{"_uuid":"d9822d3e-a505-4541-aa54-7bd3797e8c5a","_cell_guid":"831a8b7a-aa19-4258-b8e4-ab2fe82937c8","trusted":true}},{"cell_type":"code","source":"import contextlib2\nimport io\nimport IPython\nimport json\nimport numpy as np\nimport os\nimport pathlib\nimport pandas as pd\nimport sys\nimport tensorflow as tf\nimport time\n\nfrom PIL import Image, ImageDraw\n\nimport glob\nfrom matplotlib import pyplot as plt\nimport matplotlib.patches as patches\n%matplotlib inline\nimport random\n\n\n# TF does not release GPU memory properly in the notebook so we need to clear between tf gpu operations\n#!pip install numba\nfrom numba import cuda\nimport gc","metadata":{"_uuid":"312009ff-6885-49c0-a38a-3368958091e3","_cell_guid":"27433868-f7b7-4639-96ec-5fae90ae766e","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:38:30.034875Z","iopub.execute_input":"2022-02-22T15:38:30.035564Z","iopub.status.idle":"2022-02-22T15:38:32.314035Z","shell.execute_reply.started":"2022-02-22T15:38:30.035518Z","shell.execute_reply":"2022-02-22T15:38:32.313221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check that the correct TF version is being used and that the GPU has been found. <br /> There may be some warnings, but the output should look like:\n> 2.6.0 <br /> True <br />  [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]","metadata":{"_uuid":"623ed329-608a-4dbe-865a-a10bb24438e2","_cell_guid":"70a9b88d-9200-4bc7-be9d-0895270575c1","trusted":true}},{"cell_type":"code","source":"# The notebook is supposed to run with TF 2.6.0\nprint(tf.__version__)\n# Check that the gpu is found\nprint(tf.test.is_gpu_available())\nprint(tf.config.list_physical_devices('GPU'))","metadata":{"_uuid":"2b4142b0-c28c-437d-8780-55a38e59a30a","_cell_guid":"352d7622-bd5a-48cc-873a-1f3985bc91d8","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:09:34.723867Z","iopub.execute_input":"2022-02-22T15:09:34.724405Z","iopub.status.idle":"2022-02-22T15:09:36.684847Z","shell.execute_reply.started":"2022-02-22T15:09:34.724364Z","shell.execute_reply":"2022-02-22T15:09:36.68412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Explore the data sets\nHere we will load the datasets, evaluate how many images and object annotations are present, then plot some image to check if the annotations look correct.\nImages will be plotted below the **'Code'** cell for the corresponding dataset. \nNote: Some of the cells have been converted to Markdown as the Kaggle environment does not provide enough RAM to unpack the whole dataset into memory at once.\n\nFirst we will define functions to help load and plot the records.","metadata":{"_uuid":"733bddde-29c9-4870-ba6b-d84f1166f12e","_cell_guid":"4ebea88f-5bf1-46aa-bea4-c433e4a98607","trusted":true}},{"cell_type":"code","source":"def get_images_and_objects(tfrecord_file, features):\n    tf_dataset = tf.data.TFRecordDataset(tfrecord_file)\n    imgs=[]\n    bbs=[]\n    labels=[]\n    for data in tf_dataset:#.take(num_records_to_plot):\n        example = tf.train.Example()\n        example.ParseFromString(data.numpy())\n        record = tf.io.parse_single_example(data, feature_description)\n\n        image = record['image/encoded']\n        object_texts = record['image/object/class/text']\n        object_labels = record['image/object/class/label']\n        width = record['image/width'].numpy()\n        height = record['image/height'].numpy()    \n        xmins = record['image/object/bbox/xmin']\n        xmaxs = record['image/object/bbox/xmax']\n        ymins = record['image/object/bbox/ymin']\n        ymaxs = record['image/object/bbox/ymax']\n        img_bbs = []\n        img_labels = []\n\n        for i in range(0,xmins.shape[0]):\n            # Convert Tensors to TensorProtos and then to numpy arrays\n            xmin = tf.make_ndarray(tf.make_tensor_proto(xmins.values[i]))*width\n            ymin = tf.make_ndarray(tf.make_tensor_proto(ymins.values[i]))*height\n            xmax = tf.make_ndarray(tf.make_tensor_proto(xmaxs.values[i]))*width\n            ymax = tf.make_ndarray(tf.make_tensor_proto(ymaxs.values[i]))*height\n            \n            label = tf.make_ndarray(tf.make_tensor_proto(object_labels.values[i]))\n            img_labels.append(label)\n            \n            rect = (xmin, ymin, xmax, ymax)\n            img_bbs.append(rect)\n\n        # Convert image from raw bytes to numpy array\n        image_decoded = tf.image.decode_image(image)\n        image_decoded_np = image_decoded.numpy()\n\n        imgs.append(image_decoded_np)\n        bbs.append(img_bbs)\n        labels.append(img_labels)\n        \n    return imgs, bbs, labels","metadata":{"execution":{"iopub.status.busy":"2022-02-22T15:38:35.50367Z","iopub.execute_input":"2022-02-22T15:38:35.50438Z","iopub.status.idle":"2022-02-22T15:38:35.520258Z","shell.execute_reply.started":"2022-02-22T15:38:35.504335Z","shell.execute_reply":"2022-02-22T15:38:35.51793Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot some of the training images\nimport random\n%matplotlib inline\n\ndef object_label(object_id):\n    return {\n        0: '???',\n        1: 'ASC.',\n        2: 'TRI.',\n        3: 'HKW.',\n        4: 'SCH.',\n    }.get(object_id, 0)\n\ndef object_color(object_id):\n    return {\n        0: 'black',\n        1: 'red',\n        2: 'blue',\n        3: 'yellow',\n        4: 'green',\n    }.get(object_id, 0)\n\ndef plot_images(images, bounding_boxes, labels, rows=3, columns=3, sizex=40, sizey=40):\n    fig, ax = plt.subplots(figsize=(sizex, sizey))\n\n    for i in range(1, columns*rows +1):\n        image_no = random.randint(0,len(images)-1)\n        image = images[image_no]\n        fig.add_subplot(rows, columns, i)\n        for index, bbs in enumerate(bounding_boxes[image_no]):\n            object_id = int(labels[image_no][index])\n            #bbs = (xmin, ymin, xmax, ymax)\n            #rect = ((xmin, ymin), (xmax - xmin), (ymax - ymin))\n            rect = patches.Rectangle((bbs[0],bbs[1]),(bbs[2]-bbs[0]),(bbs[3]-bbs[1]), linewidth=4, edgecolor=object_color(object_id), facecolor='none')\n            plt.gca().add_patch(rect)\n            plt.gca().annotate(object_label(object_id), xy=rect.get_xy(), fontsize=40, color=object_color(object_id))\n        plt.imshow(image)\n\n","metadata":{"execution":{"iopub.status.busy":"2022-02-22T15:44:46.987232Z","iopub.execute_input":"2022-02-22T15:44:46.98795Z","iopub.status.idle":"2022-02-22T15:44:47.006482Z","shell.execute_reply.started":"2022-02-22T15:44:46.987906Z","shell.execute_reply":"2022-02-22T15:44:47.004802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we read list the records available and define the feature descriptions for each record in the TFRECORD.","metadata":{}},{"cell_type":"code","source":"filenames = glob.glob('/kaggle/input/ai4ntd-p1-5/*.record')\nprint(filenames)\n\n\n# the feature descriptions / metadata available for every record in the TFRECORD\nfeature_description = {\n    'image/width': tf.io.FixedLenFeature([], tf.int64, default_value=0),\n    'image/height': tf.io.FixedLenFeature([], tf.int64, default_value=0),\n    'image/encoded': tf.io.FixedLenFeature([], tf.string, default_value=''),\n    'image/object/class/text': tf.io.VarLenFeature(tf.string),\n    'image/object/class/label': tf.io.VarLenFeature(tf.int64),\n    'image/object/bbox/xmin': tf.io.VarLenFeature(tf.float32),\n    'image/object/bbox/xmax': tf.io.VarLenFeature(tf.float32),\n    'image/object/bbox/ymin': tf.io.VarLenFeature(tf.float32),\n    'image/object/bbox/ymax': tf.io.VarLenFeature(tf.float32),\n    'image/object/bbox/xmin': tf.io.VarLenFeature(tf.float32)\n}","metadata":{"_uuid":"8387bad6-acab-44f6-8119-24442af8c73a","_cell_guid":"63ca0b78-951c-453a-9b68-5a7df720cd4c","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:38:38.618275Z","iopub.execute_input":"2022-02-22T15:38:38.619376Z","iopub.status.idle":"2022-02-22T15:38:38.631212Z","shell.execute_reply.started":"2022-02-22T15:38:38.619323Z","shell.execute_reply":"2022-02-22T15:38:38.630354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train Set summary\nThe train and eval records are rather large when unpacked, so requires a lot of memory. We leave this one commented out for testing.\nIf you would like to use them on another machine we recommend you have 64GB of RAM available. If you would like to use a machine having less memory, we recommend that you read out the images from the TFRECORD and reduce the size of the images, before recompiling the TFRECORD.\n\nConvert the following cell from Markdown to Code if you want to run it. Note that the Kaggle work books don't have enough memory to load the train set.","metadata":{}},{"cell_type":"markdown","source":"\"\"\" Convert this cell to Code to explore the train record\"\"\"\ntrain_record = glob.glob('/kaggle/input/ai4ntd-p1-5/train.record')\ntrain_images, train_objects, train_labels =  get_images_and_objects(train_record, feature_description)\nprint(f\"There are {train_images} images and {train_objects} eggs in the train set\")\nplot_images(train_images, train_objects, train_labels)","metadata":{"execution":{"iopub.status.busy":"2022-02-21T20:04:31.474322Z","iopub.execute_input":"2022-02-21T20:04:31.474961Z"}}},{"cell_type":"markdown","source":"### Eval Set summary\nConvert the following cell from Markdown to Code if you want to run it. Note that the Kaggle work book doesn't include enough memory to load the entire eval set.","metadata":{}},{"cell_type":"markdown","source":"eval_record = glob.glob('/kaggle/input/ai4ntd-p1-5/eval.record')\neval_images, eval_objects, eval_labels =  get_images_and_objects(eval_record, feature_description)\nprint(f\"There are {eval_images} and {eval_objects} eggs images in the eval set\")\nplot_images(eval_images, eval_objects, eval_labels)","metadata":{"execution":{"iopub.status.busy":"2022-02-21T20:03:49.488453Z","iopub.status.idle":"2022-02-21T20:03:49.488743Z","shell.execute_reply.started":"2022-02-21T20:03:49.488592Z","shell.execute_reply":"2022-02-21T20:03:49.488611Z"}}},{"cell_type":"markdown","source":"### Test Set summary","metadata":{}},{"cell_type":"code","source":"# Read the test record in\ntest_record = glob.glob('/kaggle/input/ai4ntd-p1-5/test.record')\n# Extract the images, objects and labels from the record\ntest_images, test_objects, test_labels =  get_images_and_objects(test_record, feature_description)\neggs = sum( [ len(object) for object in test_objects])\nprint(f\"There are {len(test_images)} images and  {eggs} eggs in the test set\")\nplot_images(test_images, test_objects, test_labels)","metadata":{"execution":{"iopub.status.busy":"2022-02-22T15:44:51.258832Z","iopub.execute_input":"2022-02-22T15:44:51.259674Z","iopub.status.idle":"2022-02-22T15:45:23.081828Z","shell.execute_reply.started":"2022-02-22T15:44:51.259634Z","shell.execute_reply":"2022-02-22T15:45:23.080294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can re-run the plotting function to generate more examples e.g.","metadata":{}},{"cell_type":"code","source":"plot_images(test_images, test_objects, test_labels, rows=2, columns=2, sizex=40, sizey=30)","metadata":{"execution":{"iopub.status.busy":"2022-02-22T15:51:13.414453Z","iopub.execute_input":"2022-02-22T15:51:13.415135Z","iopub.status.idle":"2022-02-22T15:51:17.388861Z","shell.execute_reply.started":"2022-02-22T15:51:13.415092Z","shell.execute_reply":"2022-02-22T15:51:17.387743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_images(test_images, test_objects, test_labels,  rows=1, columns=1, sizex=40, sizey=30)","metadata":{"execution":{"iopub.status.busy":"2022-02-22T15:50:59.291467Z","iopub.execute_input":"2022-02-22T15:50:59.292361Z","iopub.status.idle":"2022-02-22T15:51:03.74272Z","shell.execute_reply.started":"2022-02-22T15:50:59.292316Z","shell.execute_reply":"2022-02-22T15:51:03.741551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This Kaggle notebook is rather limited on memory so we'll clean up before proceeding\ndel test_images, test_objects, test_labels\ngc.collect()\n\n# TF does not release GPU memory properly in the notebook so we need to clear between tf gpu operations\n# Using tf sess does not solve this either\ncuda.select_device(0)\ncuda.close()","metadata":{"execution":{"iopub.status.busy":"2022-02-22T15:10:31.172148Z","iopub.execute_input":"2022-02-22T15:10:31.172511Z","iopub.status.idle":"2022-02-22T15:10:31.582268Z","shell.execute_reply.started":"2022-02-22T15:10:31.172471Z","shell.execute_reply":"2022-02-22T15:10:31.581471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train an object detection model","metadata":{"_uuid":"83aeb1e6-f385-430c-be70-2325dc4ebfa5","_cell_guid":"0f7e8616-4fee-48db-b088-75697f2f0582","trusted":true}},{"cell_type":"markdown","source":"### Setup class label mapping\n\nWe could import the available 'egg_label_map.pbtxt', but instead we will generate a new one that you could configure otherwise, e.g. you could leave out classes that are not desired or extend to new parasites.","metadata":{"_uuid":"6f15f5c6-abe4-4234-9170-24b4b770758e","_cell_guid":"6bd84ad8-e88b-4404-8005-335022546897","trusted":true}},{"cell_type":"code","source":"# Create a label map to map between label index and human-readable label name.\n!mkdir /kaggle/working/dataset\nlabel_map_str = \"\"\"item {\n  name: \"Ascaris\"\n  id: 1\n}\nitem {\n  name: \"Trichuris\"\n  id: 2\n}\nitem {\n  name: \"Hookworm\"\n  id: 3\n}\nitem {\n  name: \"Schistosoma\"\n  id: 4\n}\"\"\"\n\n\nwith open('/kaggle/working/dataset/label_map.pbtxt', 'w') as f:\n    f.write(label_map_str)\n\n!more /kaggle/working/dataset/label_map.pbtxt","metadata":{"_uuid":"69abcc78-c641-48fc-b71a-e0d4f65925f6","_cell_guid":"8d03e774-98af-4a41-89f2-0c6d6d057c95","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:10:31.583734Z","iopub.execute_input":"2022-02-22T15:10:31.58409Z","iopub.status.idle":"2022-02-22T15:10:33.151342Z","shell.execute_reply.started":"2022-02-22T15:10:31.584052Z","shell.execute_reply":"2022-02-22T15:10:33.150363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Download pretrain model to be used for transfer learning\n\nWe'll use [TensorFlow Object Detection API](https://github.com/tensorflow/models/tree/master/research/object_detection) and an EfficientDet-D0 base model and apply transfer learning to train a STH&SCH detection model. \nTransfer learning allows us to take a model which has been pretrained with another dataset and apply the some of this knowledge in another domain space. For example, some layers in the network that identify shapes, patterns, gradients, textures may be useful in any domain. Transfer learning reduces the time taken to relearn these skills / update model parameters.\n\nEfficientDet-D0 is the smallest model in the EfficientDet model family and we pick it to reduce training time for demonstration purposes. You can probably increase accuracy by switch to using a larger EfficientDet model. Refer to the [TF Model Zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md) for a full list of available models. If changing model, you may need update parts of the pipeline.config to suit.","metadata":{"_uuid":"86d6a56f-a1ff-4a99-ab3d-6a4daa7281ec","_cell_guid":"8b4c9f2b-8ada-4961-9cb2-666f66379a95","trusted":true}},{"cell_type":"code","source":"# Download the pretrained EfficientDet-D0 checkpoint\n!wget http://download.tensorflow.org/models/object_detection/tf2/20200711/efficientdet_d0_coco17_tpu-32.tar.gz\n!tar -xvzf efficientdet_d0_coco17_tpu-32.tar.gz","metadata":{"_uuid":"f92d6558-aa5c-46bc-a57a-9c9642d08d03","_cell_guid":"05112c48-3601-4961-b350-09226822e8e3","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:10:33.153447Z","iopub.execute_input":"2022-02-22T15:10:33.153766Z","iopub.status.idle":"2022-02-22T15:10:35.80471Z","shell.execute_reply.started":"2022-02-22T15:10:33.153722Z","shell.execute_reply":"2022-02-22T15:10:35.803845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Setup the pipeline.config\nA configured pipeline.config is available in the input data directory, however we will setup one here that can be adjusted between runs.","metadata":{"_uuid":"abf0c0b1-b555-417a-a561-db8f15fdcad7","_cell_guid":"5b7782db-d6fb-4dd5-b9e2-4a5e2726a4d0","trusted":true}},{"cell_type":"code","source":"# This sets up a pipeline.config that uses the efficientdet_d0_coco17_tpu-32 pipeline.config as a template.\n# The 'fine_tune_checkpoint', 'batch_size', 'label_map_path', 'train_record' and 'eval_record' can be configured in following steps.\n\nfrom string import Template\n\nconfig_file_template = \"\"\"\n# SSD with EfficientNet-b0 + BiFPN feature extractor,\n# shared box predictor and focal loss (a.k.a EfficientDet-d0).\n# See EfficientDet, Tan et al, https://arxiv.org/abs/1911.09070\n# See Lin et al, https://arxiv.org/abs/1708.02002\n# Initialized from an EfficientDet-D0 checkpoint.\n#\n# Train on GPU\n\nmodel {\n  ssd {\n    inplace_batchnorm_update: true\n    freeze_batchnorm: false\n    num_classes: 1\n    add_background_class: false\n    box_coder {\n      faster_rcnn_box_coder {\n        y_scale: 10.0\n        x_scale: 10.0\n        height_scale: 5.0\n        width_scale: 5.0\n      }\n    }\n    matcher {\n      argmax_matcher {\n        matched_threshold: 0.5\n        unmatched_threshold: 0.5\n        ignore_thresholds: false\n        negatives_lower_than_unmatched: true\n        force_match_for_each_row: true\n        use_matmul_gather: true\n      }\n    }\n    similarity_calculator {\n      iou_similarity {\n      }\n    }\n    encode_background_as_zeros: true\n    anchor_generator {\n      multiscale_anchor_generator {\n        min_level: 3\n        max_level: 7\n        anchor_scale: 4.0\n        aspect_ratios: [1.0, 2.0, 0.5]\n        scales_per_octave: 3\n      }\n    }\n    image_resizer {\n      keep_aspect_ratio_resizer {\n        min_dimension: 512\n        max_dimension: 512\n        pad_to_max_dimension: true\n        }\n    }\n    box_predictor {\n      weight_shared_convolutional_box_predictor {\n        depth: 64\n        class_prediction_bias_init: -4.6\n        conv_hyperparams {\n          force_use_bias: true\n          activation: SWISH\n          regularizer {\n            l2_regularizer {\n              weight: 0.00004\n            }\n          }\n          initializer {\n            random_normal_initializer {\n              stddev: 0.01\n              mean: 0.0\n            }\n          }\n          batch_norm {\n            scale: true\n            decay: 0.99\n            epsilon: 0.001\n          }\n        }\n        num_layers_before_predictor: 3\n        kernel_size: 3\n        use_depthwise: true\n      }\n    }\n    feature_extractor {\n      type: 'ssd_efficientnet-b0_bifpn_keras'\n      bifpn {\n        min_level: 3\n        max_level: 7\n        num_iterations: 3\n        num_filters: 64\n      }\n      conv_hyperparams {\n        force_use_bias: true\n        activation: SWISH\n        regularizer {\n          l2_regularizer {\n            weight: 0.00004\n          }\n        }\n        initializer {\n          truncated_normal_initializer {\n            stddev: 0.03\n            mean: 0.0\n          }\n        }\n        batch_norm {\n          scale: true,\n          decay: 0.99,\n          epsilon: 0.001,\n        }\n      }\n    }\n    loss {\n      classification_loss {\n        weighted_sigmoid_focal {\n          alpha: 0.25\n          gamma: 1.5\n        }\n      }\n      localization_loss {\n        weighted_smooth_l1 {\n        }\n      }\n      classification_weight: 1.0\n      localization_weight: 1.0\n    }\n    normalize_loss_by_num_matches: true\n    normalize_loc_loss_by_codesize: true\n    post_processing {\n      batch_non_max_suppression {\n        score_threshold: 1e-8\n        iou_threshold: 0.5\n        max_detections_per_class: 100\n        max_total_detections: 100\n      }\n      score_converter: SIGMOID\n    }\n  }\n}\n\ntrain_config: {\n  fine_tune_checkpoint: $fine_tune_checkpoint\n  fine_tune_checkpoint_version: V2\n  fine_tune_checkpoint_type: \"detection\"\n  batch_size: $batch_size\n  sync_replicas: false\n  startup_delay_steps: 0\n  replicas_to_aggregate: 1\n  use_bfloat16: false\n  num_steps: $training_steps\n  data_augmentation_options {\n    random_horizontal_flip {\n    }\n  }\n  data_augmentation_options {\n    random_scale_crop_and_pad_to_square {\n      output_size: 512\n      scale_min: 0.5\n      scale_max: 2.0\n    }\n  }\n  optimizer {\n    momentum_optimizer: {\n      learning_rate: {\n        cosine_decay_learning_rate {\n          learning_rate_base: $learning_rate_base\n          total_steps: $training_steps\n          warmup_learning_rate: $warmup_learning_rate\n          warmup_steps: $warmup_steps\n        }\n      }\n      momentum_optimizer_value: 0.9\n    }\n    use_moving_average: false\n  }\n  max_number_of_boxes: 100\n  unpad_groundtruth_tensors: false\n}\n\ntrain_input_reader: {\n  label_map_path: $label_map_path\n  tf_record_input_reader {\n    input_path: $train_record\n  }\n}\n\neval_config: {\n  metrics_set: \"coco_detection_metrics\"\n  use_moving_averages: false\n  batch_size: 1;\n}\n\neval_input_reader: {\n  label_map_path: $label_map_path\n  shuffle: false\n  num_epochs: 1\n  tf_record_input_reader {\n    input_path: $eval_record\n  }\n}\n\"\"\"","metadata":{"_uuid":"0c3ea96e-450d-4762-8436-5ca4a587df46","_cell_guid":"3ce39aa4-c71f-4b9d-abe2-7407f82485b2","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:10:35.810205Z","iopub.execute_input":"2022-02-22T15:10:35.810438Z","iopub.status.idle":"2022-02-22T15:10:35.822857Z","shell.execute_reply.started":"2022-02-22T15:10:35.810409Z","shell.execute_reply":"2022-02-22T15:10:35.822017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define the parameters to be updated in the training pipeline\nTRAIN_RECORD = '\"/kaggle/input/ai4ntd-p1-5/train.record\"'\nEVAL_RECORD = '\"/kaggle/input/ai4ntd-p1-5/eval.record\"'\nLABEL_MAP_PATH = '\"/kaggle/working/dataset/label_map.pbtxt\"'\nFINE_TUNE_CHECKPOINT = '\"/kaggle/working/efficientdet_d0_coco17_tpu-32/checkpoint/ckpt-0\"' # we start with the pretrained model\n\n# Learning rate and batch size will need to be tuned. \n# Batch size determines the number of images used per batch before the model parameters are tuned/updated\n# We use a batch size of 1 here due to GPU memory limitations of the free kaggle environment.\n# The default batch size for 'efficientdet_d0_coco17_tpu-32' is 128.\nBATCH_SIZE = 1 \n# Depending on the batch size the total number of training steps and the learning rate will need to be adjusted.\n# A small batch size may generally required a lower learning rate and take longer for convergence.\n# A larger batch size may generally allow for a larger learning rate and converge faster.\n# By using a warm up step parameter the learning rate can differ for the specified number of steps.\n# i.e. The learning rate will increase linearly from WARMUP_LEARNING_RATE to LEARNING_RATE_BASE over WARMUP_STEPS\n# This can help prevent early over-fitting from clustered data, particularly if the batch_size is 1.\nTRAINING_STEPS = 100 # the number of steps we train for, default = 400000\nLEARNING_RATE_BASE = 5e-3\nWARMUP_STEPS = 100 # # the number of warmup setups with the WARMUP_LEARNING_RATE, default = 2000\nWARMUP_LEARNING_RATE = 5e-4","metadata":{"_uuid":"a9df7a94-7c04-46b0-8eb2-c484a72dedc1","_cell_guid":"b1cbbdd7-5784-4689-9046-736ea5a33d97","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:10:35.825847Z","iopub.execute_input":"2022-02-22T15:10:35.826746Z","iopub.status.idle":"2022-02-22T15:10:35.833906Z","shell.execute_reply.started":"2022-02-22T15:10:35.826707Z","shell.execute_reply":"2022-02-22T15:10:35.833007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Update the pipeline.config with the specified parameters\npipeline = Template(config_file_template).substitute(\n    fine_tune_checkpoint=FINE_TUNE_CHECKPOINT,\n    label_map_path=LABEL_MAP_PATH,\n    train_record=TRAIN_RECORD,\n    eval_record=EVAL_RECORD,\n    training_steps=TRAINING_STEPS,\n    warmup_steps=WARMUP_STEPS,\n    batch_size=BATCH_SIZE,\n    learning_rate_base = LEARNING_RATE_BASE,\n    warmup_learning_rate = WARMUP_LEARNING_RATE)\n\nPIPELINE_CONFIG_PATH = '/kaggle/working/dataset/pipeline.config'\nwith open(PIPELINE_CONFIG_PATH, 'w') as f:\n    f.write(pipeline)","metadata":{"_uuid":"087fff56-9cb3-4d2c-83ad-33dfad17d1b2","_cell_guid":"8f2f42d6-6293-4e9c-aeb2-cacc36b3c9fc","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:10:35.83536Z","iopub.execute_input":"2022-02-22T15:10:35.835698Z","iopub.status.idle":"2022-02-22T15:10:35.845225Z","shell.execute_reply.started":"2022-02-22T15:10:35.835661Z","shell.execute_reply":"2022-02-22T15:10:35.844405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Setup Tensorboard to monitor training progress","metadata":{"_uuid":"d71a9842-68d0-4c5f-ad97-4f589747bd60","_cell_guid":"f0f3b726-0f1d-4822-9835-afa6048c24bc","trusted":true}},{"cell_type":"code","source":"# Specify and setup the directory for the new model data \nMODEL_DIR='/kaggle/working/sthsch_efficientdet_d0'\n!mkdir {MODEL_DIR}","metadata":{"_uuid":"1aaafce5-d37e-4e99-b130-fe6f2654cb7b","_cell_guid":"e2d262df-1d75-4e6d-bfca-47584a16ebb8","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:10:35.848425Z","iopub.execute_input":"2022-02-22T15:10:35.849375Z","iopub.status.idle":"2022-02-22T15:10:36.621929Z","shell.execute_reply.started":"2022-02-22T15:10:35.849336Z","shell.execute_reply":"2022-02-22T15:10:36.620919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Begin the model training\n\nRunning the following cell with start the model training process. <br />\nAll data will be saved to the newly specified model directory and visible under your Output data folder.\n\nWe need to run training and evaluation in parallel so we set them up in separate processes. For this some additional libraries and dependencies are need.","metadata":{"_uuid":"eed69c62-560a-4755-9e17-5b8baff13950","_cell_guid":"6ac54051-8646-48bd-8c20-45477a29f9bb","trusted":true}},{"cell_type":"code","source":"!python /kaggle/working/models/research/object_detection/model_main_tf2.py \\\n    --pipeline_config_path={PIPELINE_CONFIG_PATH} \\\n    --model_dir={MODEL_DIR} \\\n    --alsologtostderr {MODEL_DIR}/train.log","metadata":{"_uuid":"a432e4cc-a641-4e6f-b3fe-170c66ee6bfd","_cell_guid":"5aca35c9-2c65-4db5-9fde-f3f9afcd66ad","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:10:36.624081Z","iopub.execute_input":"2022-02-22T15:10:36.624327Z","iopub.status.idle":"2022-02-22T15:13:29.076621Z","shell.execute_reply.started":"2022-02-22T15:10:36.624299Z","shell.execute_reply":"2022-02-22T15:13:29.064785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Evaluate model\n","metadata":{}},{"cell_type":"code","source":"# Load the exported model and evaluate\n!python /kaggle/working/models/research/object_detection/model_main_tf2.py \\\n    --pipeline_config_path={PIPELINE_CONFIG_PATH} \\\n    --model_dir={MODEL_DIR} \\\n    --checkpoint_dir={MODEL_DIR} \\\n    --eval_timeout=0 \\\n    --alsologtostderr # &> {MODEL_DIR}/eval.log ","metadata":{"_uuid":"63000a43-ace5-47dd-bb8e-d115167b9aa1","_cell_guid":"c43ab7ab-aef8-4fbd-b450-a6751db1e9bd","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:13:29.084509Z","iopub.execute_input":"2022-02-22T15:13:29.084997Z","iopub.status.idle":"2022-02-22T15:18:55.610968Z","shell.execute_reply.started":"2022-02-22T15:13:29.08496Z","shell.execute_reply":"2022-02-22T15:18:55.610065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# View the training and evaluation progress on the tensorboard. Both training and eval\n# will have finished now but it is still good to see the progress.\n# In this notebook we can't run the training, evaluation and tenosorboard in parallel easily\n# If you run this offline you should move these steps to separate kernels and extend the \n# --eval_timeout to something greater than the training time for the batch size e.g. 3600 seconds\n\n#NOTE: Tensorboard has been disabled in notebooks by Kaggle, uncomment following lines to use otherwise\n#from tensorboard import notebook\n#%load_ext tensorboard\n#%tensorboard --logdir {MODEL_DIR}","metadata":{"_uuid":"1c2b82d6-326e-48af-8c89-d083fbad0c42","_cell_guid":"b76de1bb-8a19-42af-a2d8-923b9e785995","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:18:55.616306Z","iopub.execute_input":"2022-02-22T15:18:55.618443Z","iopub.status.idle":"2022-02-22T15:18:55.625096Z","shell.execute_reply.started":"2022-02-22T15:18:55.618396Z","shell.execute_reply":"2022-02-22T15:18:55.624388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Export a model for inference \n\nIn TF2 model evaluation in the object detection","metadata":{"_uuid":"48275d15-1e1c-4a52-a092-c367dfc71546","_cell_guid":"1cb7bef3-e42f-41b9-88f0-c5efdc1e8542","trusted":true}},{"cell_type":"code","source":"# Export the model\n!python /kaggle/working/models/research/object_detection/exporter_main_v2.py \\\n    --input_type image_tensor \\\n    --pipeline_config_path={PIPELINE_CONFIG_PATH} \\\n    --trained_checkpoint_dir={MODEL_DIR} \\\n    --output_directory={MODEL_DIR}/output","metadata":{"execution":{"iopub.status.busy":"2022-02-22T15:18:55.628583Z","iopub.execute_input":"2022-02-22T15:18:55.629906Z","iopub.status.idle":"2022-02-22T15:21:00.601059Z","shell.execute_reply.started":"2022-02-22T15:18:55.629864Z","shell.execute_reply":"2022-02-22T15:21:00.600113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Evaluation\n\nWe will use the exported model on the test data set and generate a confusion matrix.\n\nIf the steps have not been updated then the model will not be performant at all. But to save your GPU time, we evaluate our pre-trained models for STH & SCH below.","metadata":{"_uuid":"cad28e95-fe33-4fa0-900f-ddf2a458e080","_cell_guid":"04ccbe58-a2ec-4acd-b62d-a16f4961ea96","trusted":true}},{"cell_type":"code","source":"# Download a library that we will use to generate the confusion matrix\n# Here we use a script by Santiago Valdarrama https://github.com/svpino/tf_object_detection_cm\n!wget https://raw.githubusercontent.com/svpino/tf_object_detection_cm/master/confusion_matrix_tf2.py\n\n# Read in the python script\nwith open('confusion_matrix_tf2.py', 'r') as file :\n    cm_script_tf2 = file.read()\n\n# We need to make a correction to the script with the IoU calculation\n# Our bounding boxes are recorded as floats, normalised between 0 to 1, so the '+1' correction introduces an error\n# If we were to load images and maintain bounding boxes with pixel reference the '+1' term would be necessary\ncm_script_tf2 = cm_script_tf2.replace('intersection = max(0, xb - xa + 1) * max(0, yb - ya + 1)', 'intersection = max(0, xb - xa) * max(0, yb - ya)')\ncm_script_tf2 = cm_script_tf2.replace('boxAArea = (g_xmax - g_xmin + 1) * (g_ymax - g_ymin + 1)', 'boxAArea = (g_xmax - g_xmin) * (g_ymax - g_ymin)')\ncm_script_tf2 = cm_script_tf2.replace('boxBArea = (d_xmax - d_xmin + 1) * (d_ymax - d_ymin + 1)', 'boxBArea = (d_xmax - d_xmin) * (d_ymax - d_ymin)')    \n\n# Write the file out again\nwith open('confusion_matrix_tf2.py', 'w') as file:\n    file.write(cm_script_tf2)","metadata":{"_uuid":"9845fdd7-7ce2-40c4-9475-79d4087cc65f","_cell_guid":"09974692-7152-4533-8934-6cc1a37597cf","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:21:00.603348Z","iopub.execute_input":"2022-02-22T15:21:00.603645Z","iopub.status.idle":"2022-02-22T15:21:01.734219Z","shell.execute_reply.started":"2022-02-22T15:21:00.603606Z","shell.execute_reply":"2022-02-22T15:21:01.733294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!mkdir -p {MODEL_DIR}/output/confusion_matrix/images/\n!python /kaggle/working/confusion_matrix_tf2.py \\\n  --input_tfrecord_path /kaggle/input/ai4ntd-p1-5/test.record \\\n  --output_path {MODEL_DIR}/output/confusion_matrix/cm_output.csv \\\n  --inference_graph {MODEL_DIR}/output/saved_model \\\n  --class_labels {LABEL_MAP_PATH} #\\\n  #--draw_option True \\\n  #--draw_save_path {MODEL_DIR}/output/confusion_matrix/images/","metadata":{"_uuid":"9cd40de7-12de-45e5-a8c7-271d94fd576e","_cell_guid":"3de7f3de-40ae-4053-85b8-82b6e8e70d94","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-22T15:21:01.736151Z","iopub.execute_input":"2022-02-22T15:21:01.736443Z","iopub.status.idle":"2022-02-22T15:23:10.538294Z","shell.execute_reply.started":"2022-02-22T15:21:01.736405Z","shell.execute_reply":"2022-02-22T15:23:10.537372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The confusion matrix and images will be saved in your '**Output'** data folder *'/kaggle/working/confusion_matrix/'*.\n\nNotice the results above are poor!\nThis is probably because the default values for batch size and training steps of 1 was used. You will want to increase this to improve the performance. To do so you can download this notebook for use in a cloud machine or local environment.\n\nBelow we show the performance of our pretrained STH and SCH models which were trained for up to 100,000 steps.","metadata":{"_uuid":"16439101-8a89-4cb7-a7f1-e332dbbf7789","_cell_guid":"09bfa912-fad5-4410-ad57-0559e55fe770","trusted":true}},{"cell_type":"markdown","source":"### Lets repeat this evaluation now with the models we trained offline","metadata":{}},{"cell_type":"markdown","source":"Our earlier research was done with TF1.13 so we will need some different libraries to use the model ","metadata":{"execution":{"iopub.status.busy":"2022-02-11T15:19:41.222266Z","iopub.execute_input":"2022-02-11T15:19:41.222596Z","iopub.status.idle":"2022-02-11T15:19:41.921876Z","shell.execute_reply.started":"2022-02-11T15:19:41.222558Z","shell.execute_reply":"2022-02-11T15:19:41.920985Z"}}},{"cell_type":"code","source":"# First we generate a tfrecord containing the detections from the trained models\n!mkdir -p '/kaggle/working/rfcn_resnet101_sthsch_1/'\n!mkdir -p '/kaggle/working/rfcn_resnet101_sthsch_2/'\n!mkdir -p '/kaggle/working/faster_rcnn_resnet50_sthsch_1/'\n!mkdir -p '/kaggle/working/faster_rcnn_resnet101_sthsch_2/'\n!mkdir -p '/kaggle/working/faster_rcnn_resnet101_sthsch_3/'\n\n!python /kaggle/working/models/research/object_detection/inference/infer_detections.py \\\n    --input_tfrecord_paths='/kaggle/input/ai4ntd-p1-5/test.record' \\\n    --output_tfrecord_path='/kaggle/working/rfcn_resnet101_sthsch_1/detections.tfrecord' \\\n    --inference_graph='/kaggle/input/ai4ntd-p1-5/rfcn_resnet101_sthsch_1/frozen_inference_graph.pb'\n\n!python /kaggle/working/models/research/object_detection/inference/infer_detections.py \\\n    --input_tfrecord_paths='/kaggle/input/ai4ntd-p1-5/test.record' \\\n    --output_tfrecord_path='/kaggle/working/rfcn_resnet101_sthsch_2/detections.tfrecord' \\\n    --inference_graph='/kaggle/input/ai4ntd-p1-5/rfcn_resnet101_sthsch_2/frozen_inference_graph.pb'\n\n!python /kaggle/working/models/research/object_detection/inference/infer_detections.py \\\n    --input_tfrecord_paths='/kaggle/input/ai4ntd-p1-5/test.record' \\\n    --output_tfrecord_path='/kaggle/working/faster_rcnn_resnet50_sthsch_1/detections.tfrecord' \\\n    --inference_graph='/kaggle/input/ai4ntd-p1-5/faster_rcnn_resnet50_sthsch_1/frozen_inference_graph.pb'\n\n!python /kaggle/working/models/research/object_detection/inference/infer_detections.py \\\n    --input_tfrecord_paths='/kaggle/input/ai4ntd-p1-5/test.record' \\\n    --output_tfrecord_path='/kaggle/working/faster_rcnn_resnet101_sthsch_2/detections.tfrecord' \\\n    --inference_graph='/kaggle/input/ai4ntd-p1-5/faster_rcnn_resnet101_sthsch_2/frozen_inference_graph.pb'\n\n!python /kaggle/working/models/research/object_detection/inference/infer_detections.py \\\n    --input_tfrecord_paths='/kaggle/input/ai4ntd-p1-5/test.record' \\\n    --output_tfrecord_path='/kaggle/working/faster_rcnn_resnet101_sthsch_3/detections.tfrecord' \\\n    --inference_graph='/kaggle/input/ai4ntd-p1-5/faster_rcnn_resnet101_sthsch_3/frozen_inference_graph.pb'\n","metadata":{"execution":{"iopub.status.busy":"2022-02-22T15:23:10.540413Z","iopub.execute_input":"2022-02-22T15:23:10.540683Z","iopub.status.idle":"2022-02-22T15:29:43.715244Z","shell.execute_reply.started":"2022-02-22T15:23:10.540646Z","shell.execute_reply":"2022-02-22T15:29:43.713945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Because our models were generated with TF1 they don't load properly with TF2 CM script,\n# instead we download and update the TF1 confusion matrix script\n# Here we use a script by Santiago Valdarrama https://github.com/svpino/tf_object_detection_cm\n\n%cd /kaggle/working/\n# Download a library that we will use to generate the confusion matrix, note this one is for TF1\n!wget https://raw.githubusercontent.com/svpino/tf_object_detection_cm/master/confusion_matrix.py\n\n# Read in the python script\nwith open('confusion_matrix.py', 'r') as file :\n    cm_script = file.read()\n    \n# Because we are actually using TF2 we need to update the script so that we ensure we use tf1 compatibility\ncm_script = cm_script.replace('import tensorflow as tf', 'import tensorflow.compat.v1 as tf')\n\n# We again need to make the correction to the script in the IoU calculation\n# Our bounding boxes are recorded as floats, normalised between 0 to 1, so the '+1' for pixel correction introduces an error\n# If we were to load images and maintain bounding boxes with pixel references the '+1' term would be necessary\ncm_script = cm_script.replace('intersection = max(0, xb - xa + 1) * max(0, yb - ya + 1)', 'intersection = max(0, xb - xa) * max(0, yb - ya)')\ncm_script = cm_script.replace('boxAArea = (g_xmax - g_xmin + 1) * (g_ymax - g_ymin + 1)', 'boxAArea = (g_xmax - g_xmin) * (g_ymax - g_ymin)')\ncm_script = cm_script.replace('boxBArea = (d_xmax - d_xmin + 1) * (d_ymax - d_ymin + 1)', 'boxBArea = (d_xmax - d_xmin) * (d_ymax - d_ymin)')    \n    \n# Write the file out again\nwith open('confusion_matrix_tf1.py', 'w') as file:\n    file.write(cm_script)","metadata":{"execution":{"iopub.status.busy":"2022-02-22T15:29:43.722742Z","iopub.execute_input":"2022-02-22T15:29:43.723253Z","iopub.status.idle":"2022-02-22T15:29:45.247754Z","shell.execute_reply.started":"2022-02-22T15:29:43.723207Z","shell.execute_reply":"2022-02-22T15:29:45.246868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!mkdir -p '/kaggle/working/confusion_matrix/'\n\n!python /kaggle/working/confusion_matrix_tf1.py \\\n--detections_record '/kaggle/working/rfcn_resnet101_sthsch_1/detections.tfrecord' \\\n--label_map '/kaggle/input/ai4ntd-p1-5/egg_label_map.pbtxt'\\\n--output_path '/kaggle/working/confusion_matrix/cm_output_rfcn_resnet101_sthsch_1.csv'\n\n!python /kaggle/working/confusion_matrix_tf1.py \\\n--detections_record '/kaggle/working/rfcn_resnet101_sthsch_2/detections.tfrecord' \\\n--label_map '/kaggle/input/ai4ntd-p1-5/egg_label_map.pbtxt'\\\n--output_path '/kaggle/working/confusion_matrix/cm_output_rfcn_resnet101_sthsch_2.csv'\n\n!python /kaggle/working/confusion_matrix_tf1.py \\\n--detections_record '/kaggle/working/faster_rcnn_resnet50_sthsch_1/detections.tfrecord' \\\n--label_map '/kaggle/input/ai4ntd-p1-5/egg_label_map.pbtxt'\\\n--output_path '/kaggle/working/confusion_matrix/cm_output_faster_rcnn_resnet50_sthsch_1.csv'\n\n!python /kaggle/working/confusion_matrix_tf1.py \\\n--detections_record '/kaggle/working/faster_rcnn_resnet101_sthsch_2/detections.tfrecord' \\\n--label_map '/kaggle/input/ai4ntd-p1-5/egg_label_map.pbtxt'\\\n--output_path '/kaggle/working/confusion_matrix/cm_output_faster_rcnn_resnet101_sthsch_2.csv'\n\n!python /kaggle/working/confusion_matrix_tf1.py \\\n--detections_record '/kaggle/working/faster_rcnn_resnet101_sthsch_3/detections.tfrecord' \\\n--label_map '/kaggle/input/ai4ntd-p1-5/egg_label_map.pbtxt'\\\n--output_path '/kaggle/working/confusion_matrix/cm_output_faster_rcnn_resnet101_sthsch_3.csv'","metadata":{"execution":{"iopub.status.busy":"2022-02-22T15:29:45.249582Z","iopub.execute_input":"2022-02-22T15:29:45.250446Z","iopub.status.idle":"2022-02-22T15:30:28.475126Z","shell.execute_reply.started":"2022-02-22T15:29:45.250412Z","shell.execute_reply":"2022-02-22T15:30:28.474085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%cd /kaggle/working/confusion_matrix/\n\ndf1 = pd.read_csv(\"cm_output_rfcn_resnet101_sthsch_1.csv\")\ndf_av = pd.DataFrame([[df1[\"precision_@0.5IOU\"].mean(), df1[\"recall_@0.5IOU\"].mean()]], columns=[\"precision\", \"recall\"], index=['rfcn_resnet101_sthsch_1'])\n\ndf2 = pd.read_csv(\"cm_output_rfcn_resnet101_sthsch_2.csv\")\ndf_av2 = pd.DataFrame([[df2[\"precision_@0.5IOU\"].mean(), df2[\"recall_@0.5IOU\"].mean()]], columns=[\"precision\", \"recall\"], index=['rfcn_resnet101_sthsch_2'])\ndf_av = df_av.append(df_av2)\n\ndf3 = pd.read_csv(\"cm_output_faster_rcnn_resnet50_sthsch_1.csv\")\ndf_av3 = pd.DataFrame([[df3[\"precision_@0.5IOU\"].mean(), df3[\"recall_@0.5IOU\"].mean()]], columns=[\"precision\", \"recall\"], index=['faster_rcnn_resnet50_sthsch_1'])\ndf_av = df_av.append(df_av3)\n\ndf4 = pd.read_csv(\"cm_output_faster_rcnn_resnet101_sthsch_2.csv\")\ndf_av4 = pd.DataFrame([[df4[\"precision_@0.5IOU\"].mean(), df4[\"recall_@0.5IOU\"].mean()]], columns=[\"precision\", \"recall\"], index=['faster_rcnn_resnet101_sthsch_2'])\ndf_av = df_av.append(df_av4)\n\ndf5 = pd.read_csv(\"cm_output_faster_rcnn_resnet101_sthsch_3.csv\")\ndf_av5 = pd.DataFrame([[df5[\"precision_@0.5IOU\"].mean(), df5[\"recall_@0.5IOU\"].mean()]], columns=[\"precision\", \"recall\"], index=['faster_rcnn_resnet101_sthsch_3'])\ndf_av = df_av.append(df_av5)\n\nprint(df_av)\n\ndf_av.plot.scatter(x=\"precision\", y=\"recall\")\ni=0\nfor x,y in zip(df_av.precision,df_av.recall):\n    label = df_av.index[i]\n    i+=1\n    plt.annotate(label,\n                 (x,y),\n                 textcoords=\"offset points\",\n                 xytext=(10,-5),\n                 ha='left')\n\n\nplt.xlabel('Precision')\nplt.ylabel('Recall')   \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-22T15:30:28.477394Z","iopub.execute_input":"2022-02-22T15:30:28.477688Z","iopub.status.idle":"2022-02-22T15:30:28.820836Z","shell.execute_reply.started":"2022-02-22T15:30:28.477648Z","shell.execute_reply":"2022-02-22T15:30:28.820111Z"},"trusted":true},"execution_count":null,"outputs":[]}]}