{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Hello fellow Kagglers,\n\nThe Sartorius cell instance segmentation competition introduced me to Mask-RCNN. As many of you, I used the Tensorflow 2.0 compatible [Mask-RCNN](https://github.com/leekunhee/Mask_RCNN/tree/tensorflow2.0) by leekunhee. Although easy to use, this library is CPU heavy, resulting in extremely long training times on Kaggle GPU notebook with 2 CPU cores. Over the past weeks I worked on improving the training efficiency which resulted in a massive reduction in training time, accomplished by 2 key improvements:\n\n1. Caching computed training samples\n2. Replacing Resnet with the modern [EfficientNetV2](https://github.com/google/automl/tree/master/efficientnetv2) models\n\nThis notebook demonstrates the training process of a Mask-RCNN model with an EfficientNetV2 model as backbone and a caching mechanism. Many improvements are possible, such as increasing the number of epochs or training/finetuning with a larger batch size. The fast training time should allow to experiment with different configurations.\n\nThe EfficientNetV2 models are efficient in terms of parameters, FLOPS and training time. The Mask-RCNN configuration used in this notebook has over 3 times less parameters than the conventional ResNet50 configuration. This allows for faster training with larger batch sizes and prevents overfitting, a key problem with just 660 training samples.\n\nThe caching mechanism will cache each sample using the lightning fast [LZ4](https://en.wikipedia.org/wiki/LZ4_(compression_algorithm)) compression algorithm. After each epoch the number of cache hits increases, improving training time. After around 10 epochs most samples are cached and a training epoch with the complete training dataset takes less than 3 minutes!\n\nPossible drawbacks are the restriction to just horizontal/vertical flips as data augmentation and not being able to load pretrained COCO weights. The EfficientNetV2 models could of course be pretrained on COCO, pretrained weights might be added in the future.\n\nThe inference notebook can be found [here](https://www.kaggle.com/markwijkhuizen/sartorius-mask-rcnn-efficientnetv2-inference).\n\nThe dataset containing the Mask-RCNN EfficientNetV2 model with caching can be found [here](https://www.kaggle.com/markwijkhuizen/maskrcnn-tf-2-efficientnetv2-caching).\n\n**V2 Updates**\n\n- Increased prediction mask size from 28x28 to 56x56. Added functionality that automatically adds N (Conv2D, BatchNorm, ReLu and Conv2DTranspose) layers for a mask of size 28Nx28N in the configuration.\n- Increased FPN_CLASSIF_FC_LAYERS_SIZE from 128 to the original 1024 as this improves performance.\n- Decreased DETECTION_MIN_CONFIDENCE from 0.70 -> 0.50\n- Multi class prediction, meaning each instance is classified as \"astro\", \"cort\" or \"shsy5y\" and not just \"cell\". This is particularly import for improving the inference process, but more on this in the [inference notebook](https://www.kaggle.com/markwijkhuizen/sartorius-mask-rcnn-efficientnetv2-inference).\n- Loading EfficientNetV2 pretrained weights on ImageNet21K","metadata":{}},{"cell_type":"code","source":"# Library to silence Tensorflow Logs\n!pip install -q silence-tensorflow\nimport silence_tensorflow.auto","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:35:57.565198Z","iopub.execute_input":"2022-11-10T20:35:57.565536Z","iopub.status.idle":"2022-11-10T20:36:15.284339Z","shell.execute_reply.started":"2022-11-10T20:35:57.565450Z","shell.execute_reply":"2022-11-10T20:36:15.283404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Add Folders to path, required to import them\nimport sys\nsys.path.append('../input/maskrcnn-tf-2-efficientnetv2-caching/Instance_Segmentation/efficientnetv2')\nsys.path.append('../input/maskrcnn-tf-2-efficientnetv2-caching/Instance_Segmentation/Mask_RCNN')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:36:15.286111Z","iopub.execute_input":"2022-11-10T20:36:15.286396Z","iopub.status.idle":"2022-11-10T20:36:15.291088Z","shell.execute_reply.started":"2022-11-10T20:36:15.286367Z","shell.execute_reply":"2022-11-10T20:36:15.290255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Install LZ4 Compression/Decompression Library\n!pip install -q ../input/maskrcnn-tf-2-efficientnetv2-caching/lz4-3.1.3-cp37-cp37m-manylinux1_x86_64.whl","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:36:15.292382Z","iopub.execute_input":"2022-11-10T20:36:15.292816Z","iopub.status.idle":"2022-11-10T20:36:22.455183Z","shell.execute_reply.started":"2022-11-10T20:36:15.292780Z","shell.execute_reply":"2022-11-10T20:36:22.454154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport mrcnn.utils as utils\nimport mrcnn.model as modellib\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\n\nimport os\nimport sys\nimport json\nimport time\nimport skimage\nimport imageio\nimport glob\nimport imgaug\nimport multiprocessing\n\nfrom PIL import Image, ImageDraw\nfrom tqdm.notebook import tqdm\nfrom sklearn.model_selection import KFold\nfrom PIL import Image, ImageEnhance\nfrom mrcnn.config import Config\nfrom mrcnn import visualize\n\n# ignore warnings to make outputs clearer\nimport warnings\nwarnings.filterwarnings('ignore')\n\nprint(f'Python Version: {sys.version}')\nprint(f'Tensorflow Version: {tf.__version__}')\nprint(f'Tensorflow Keras Version: {tf.keras.__version__}')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:36:22.457769Z","iopub.execute_input":"2022-11-10T20:36:22.458349Z","iopub.status.idle":"2022-11-10T20:36:24.656044Z","shell.execute_reply.started":"2022-11-10T20:36:22.458306Z","shell.execute_reply":"2022-11-10T20:36:24.655292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:36:24.657240Z","iopub.execute_input":"2022-11-10T20:36:24.657495Z","iopub.status.idle":"2022-11-10T20:36:24.847018Z","shell.execute_reply.started":"2022-11-10T20:36:24.657461Z","shell.execute_reply":"2022-11-10T20:36:24.846220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv('../input/sartorius-cell-instance-segmentation/train.csv')\n\n# Unique Image IDs\nid_unique = train['id'].unique()\n\n# Original Image File Path\ndef get_file_path(image_id):\n    return f'/kaggle/input/sartorius-cell-instance-segmentation/train/{image_id}.png'\n\ntrain['file_path'] = train['id'].apply(get_file_path)\n\n# Unique Cell Names\nCELL_NAMES = np.sort(train['cell_type'].unique())\nprint(f'CELL_NAMES: {CELL_NAMES}')\n\n# Cell Type to Label Dictionary\nCELL_NAMES_DICT = dict([(v, k) for k, v in enumerate(CELL_NAMES)])\nprint(f'CELL_NAMES_DICT: {CELL_NAMES_DICT}')\n\n# Add Cell Type Label to train, \" + 1\" becaue label 0 is reserved for background\ntrain['cell_type_label'] = train['cell_type'].apply(CELL_NAMES_DICT.get) + 1\n\n# Image Id to Cell Type Label Dictionary\nID2CELL_LABEL = dict(\n    [(k, v) for k, v in train[['id', 'cell_type_label']].itertuples(name=None, index=False)]\n)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:36:24.848484Z","iopub.execute_input":"2022-11-10T20:36:24.849041Z","iopub.status.idle":"2022-11-10T20:36:25.686293Z","shell.execute_reply.started":"2022-11-10T20:36:24.849002Z","shell.execute_reply":"2022-11-10T20:36:25.684773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# path to COCO-dataset weights\nCOCO_MODEL_PATH = '../input/maskrcnn-tf-2-efficientnetv2-caching/mask_rcnn_coco.h5'\n\n# Download COCO trained weights from Releases if needed\nif not os.path.exists(COCO_MODEL_PATH):\n    utils.download_trained_weights(COCO_MODEL_PATH)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:36:25.687450Z","iopub.execute_input":"2022-11-10T20:36:25.687713Z","iopub.status.idle":"2022-11-10T20:36:25.695075Z","shell.execute_reply.started":"2022-11-10T20:36:25.687678Z","shell.execute_reply":"2022-11-10T20:36:25.694311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training Configuration","metadata":{}},{"cell_type":"code","source":"# Original Image Dimensions\nHEIGHT = 520\nWIDTH = 704\nSHAPE = (HEIGHT, WIDTH)\n\n# Target Image Dimensions which are divisable by 64 as required by the MASK-RCNN model\nHEIGHT_TARGET = 576\nWIDTH_TARGET = 704\nSHAPE_TARGET = (HEIGHT_TARGET, WIDTH_TARGET)\n\nBATCH_SIZE = 1\nN_SAMPLES = train['id'].nunique()\n\n# Debug mode for fast experementing with 50 samples\nDEBUG = False\nDEBUG_SIZE = 50","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:36:25.696383Z","iopub.execute_input":"2022-11-10T20:36:25.696663Z","iopub.status.idle":"2022-11-10T20:36:25.712742Z","shell.execute_reply.started":"2022-11-10T20:36:25.696630Z","shell.execute_reply":"2022-11-10T20:36:25.711916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"EPOCHS_ALL = 10 if DEBUG else 20","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:36:25.715775Z","iopub.execute_input":"2022-11-10T20:36:25.715976Z","iopub.status.idle":"2022-11-10T20:36:25.721692Z","shell.execute_reply.started":"2022-11-10T20:36:25.715953Z","shell.execute_reply":"2022-11-10T20:36:25.720819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Configuration\n\n1. Class classification size is reduced from 1024 to 128 to reduce model size, which is done to reduce overfitting. The COCO dataset contains 80 object classes, whereas this training method only contains a single \"cell\" class, requiring less parameters.\n2. Disable multithreading for data generators by setting workers to 0. Surprisingly, multithreading is slower than running the data generator on a single core.","metadata":{}},{"cell_type":"code","source":"class CellConfig(Config):\n    \"\"\"Configuration for training on the cigarette butts dataset.\n    Derives from the base Config class and overrides values specific\n    to the cigarette butts dataset.\n    \"\"\"\n    \n    NAME = \"cell\"\n\n    # Set batch size to 1.\n    GPU_COUNT = 1\n    IMAGES_PER_GPU = BATCH_SIZE\n    STEPS_PER_EPOCH = int(DEBUG_SIZE / BATCH_SIZE)  if DEBUG else int(N_SAMPLES / BATCH_SIZE)\n    \n    # Number of Classes\n    NUM_CLASSES = 1 + len(CELL_NAMES)\n\n    # Image Dimensions\n    IMAGE_MIN_DIM = HEIGHT_TARGET\n    IMAGE_MAX_DIM = WIDTH_TARGET\n    IMAGE_SHAPE = [HEIGHT_TARGET, WIDTH_TARGET, 3]\n    IMAGE_RESIZE_MODE = 'none'\n    \n    BACKBONE = 'efficientnetv2-b0'\n\n    # Training Structure\n    FPN_CLASSIF_FC_LAYERS_SIZE = 1024\n    RPN_ANCHOR_SCALES = (32, 64, 128, 256, 512)\n    # Regions of Interest\n    PRE_NMS_LIMIT = 6000\n    # Non Max Supression\n    POST_NMS_ROIS_TRAINING = 2000\n    POST_NMS_ROIS_INFERENCE = 2000\n    # Instances\n    MAX_GT_INSTANCES = 790\n    TRAIN_ROIS_PER_IMAGE = 200\n    DETECTION_MAX_INSTANCES = 200\n    \n    # Thresholds\n    RPN_NMS_THRESHOLD = 0.70        # IoU Threshold for RPN proposals and GT\n    DETECTION_MIN_CONFIDENCE = 0.50 # Non-Background Confidence Threshold\n    DETECTION_NMS_THRESHOLD = 0.30  # IoU Threshold for ROI and GT\n    ROI_POSITIVE_RATIO = 0.33\n    \n    # Prediction Mask Shape\n    MASK_SHAPE = (56, 56)\n    # Size of mask groundtruth\n    USE_MINI_MASK = True\n    MINI_MASK_SHAPE = (112, 112)\n    \n    # DO NOT train Batch Normalization because of small batch size\n    # There are too few samples to correctly train the normalization\n    TRAIN_BN = False\n    \n    # Learning Rate\n    LEARNING_RATE = 0.004\n    WEIGHT_DECAY = 0.0\n    N_WARMUP_STEPS = 2\n    LR_SCHEDULE = True\n    \n    # Dataloader Queue Size (was set to 100 but resulted in OOM error)\n    MAX_QUEUE_SIZE = 10\n    \n    # Cache Items\n    CACHE = True\n    \n    # Debug mode will disable model checkpoints\n    DEBUG = False\n    \n    # Do not use multithreading as this slows down the dataloader!\n    WORKERS = 0\n    \n    # Losses\n    LOSS_WEIGHTS = {\n        'rpn_class_loss': 1.0,    # is the class of the bbox correct? / RPN anchor classifier loss (Forground/Background)\n        'rpn_bbox_loss': 1.0,     # is the size of the bbox correct? / RPN bounding box loss graph (bbox of generic object)\n        'mrcnn_class_loss': 1.0,  # loss for the classifier head of Mask R-CNN (Background / specific class)\n        'mrcnn_bbox_loss': 1.0,   # is the size of the bounding box correct or not? / loss for Mask R-CNN bounding box refinement\n        'mrcnn_mask_loss': 1.0,   # is the class correct? is the pixel correctly assign to the class? / mask binary cross-entropy loss for the masks head\n    }\n    \nconfig = CellConfig()\nconfig.display()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:36:25.725701Z","iopub.execute_input":"2022-11-10T20:36:25.725902Z","iopub.status.idle":"2022-11-10T20:36:25.747292Z","shell.execute_reply.started":"2022-11-10T20:36:25.725872Z","shell.execute_reply":"2022-11-10T20:36:25.746436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# RLE Decode","metadata":{}},{"cell_type":"code","source":"# ref: https://www.kaggle.com/paulorzp/run-length-encode-and-decode\ndef rle_decode_by_image_id(image_id):\n    rows = train.loc[train['id'] == image_id]\n    \n    # Image Shape\n    mask = np.full(shape=[len(rows), np.prod(SHAPE)], fill_value=0, dtype=np.uint8)\n    \n    for idx, (_, row) in enumerate(rows.iterrows()):\n        s = row['annotation'].split()\n        starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n        starts -= 1\n        ends = starts + lengths\n        for lo, hi in zip(starts, ends):\n            mask[idx, lo:hi] = True\n    \n    mask = mask.reshape([len(rows), *SHAPE])\n    mask = np.moveaxis(mask, 0, 2)\n    \n    return mask","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:36:25.750216Z","iopub.execute_input":"2022-11-10T20:36:25.750647Z","iopub.status.idle":"2022-11-10T20:36:25.760664Z","shell.execute_reply.started":"2022-11-10T20:36:25.750616Z","shell.execute_reply":"2022-11-10T20:36:25.759889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Training Dataset","metadata":{}},{"cell_type":"code","source":"# Function to pad images and masks\ndef pad_image(image, constant_values):\n    pad_h = (HEIGHT_TARGET - HEIGHT) // 2\n    pad_w = (WIDTH_TARGET - WIDTH) // 2\n    \n    if len(image.shape) == 3:\n        return np.pad(image, ((pad_h, pad_h), (pad_w, pad_w), (0,0)), constant_values=constant_values)\n    else:\n        return np.pad(image, ((pad_h, pad_h), (pad_w, pad_w)), constant_values=constant_values)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:36:25.762519Z","iopub.execute_input":"2022-11-10T20:36:25.763043Z","iopub.status.idle":"2022-11-10T20:36:25.771724Z","shell.execute_reply.started":"2022-11-10T20:36:25.762985Z","shell.execute_reply":"2022-11-10T20:36:25.770611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!rm -rf ./train ./test\n!mkdir ./train ./test","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:36:25.773263Z","iopub.execute_input":"2022-11-10T20:36:25.773592Z","iopub.status.idle":"2022-11-10T20:36:28.003576Z","shell.execute_reply.started":"2022-11-10T20:36:25.773516Z","shell.execute_reply":"2022-11-10T20:36:28.002573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"id_unique = train['id'].unique()\nif DEBUG:\n    id_unique = id_unique[:DEBUG_SIZE]\n\nimage_id2file_path = train.groupby('id')[['id', 'file_path']].head(1)\nimage_id2file_path = image_id2file_path.set_index('id').squeeze().to_dict()\n\n# Create padded training samples with enhanced contrast\nfor image_id in tqdm(id_unique):\n    # Read Original Image\n    image = imageio.imread(image_id2file_path[image_id])\n    # Pad Image\n    image = pad_image(image, 128)\n    \n    # Save image in working directory, required for Mask-RCNN\n    imageio.imwrite(f'./train/{image_id}.png', image)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:36:28.006330Z","iopub.execute_input":"2022-11-10T20:36:28.010133Z","iopub.status.idle":"2022-11-10T20:37:26.784251Z","shell.execute_reply.started":"2022-11-10T20:36:28.010092Z","shell.execute_reply":"2022-11-10T20:37:26.783482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset","metadata":{}},{"cell_type":"code","source":"class CellDataset(utils.Dataset):\n\n    def load_data(self, image_ids, form, image_group):\n   \n        for i, name in enumerate(CELL_NAMES):\n            self.add_class('cell', 1 + i, name)\n       \n        # Add the image using the base method from utils.Dataset\n        for vertical_flip in [True, False]:\n            for horizontal_flip in [True, False]:\n                for image in tqdm(image_ids):\n                    self.add_image('cell', \n                           image_id=image, \n                           path=(f'./{image_group}/{image}.png'), \n                           label = ID2CELL_LABEL[image],\n                           height=512, width=512,\n                          vertical_flip=vertical_flip, horizontal_flip=horizontal_flip,\n                      )\n            \n            \n    def load_mask(self, image_id):\n        \"\"\" Load instance masks for the given image.\n        MaskRCNN expects masks in the form of a bitmap [height, width, instances].\n        Args:\n            image_id: The id of the image to load masks for\n        Returns:\n            masks: A bool array of shape [height, width, instance count] with\n                one mask per instance.\n            class_ids: a 1D array of class IDs of the instance masks.\n        \"\"\"\n    \n        info = self.image_info[image_id]\n        image_id = info['id']\n    \n        # Get masks by image_id\n        masks = rle_decode_by_image_id(image_id)\n        masks = pad_image(masks, 0)\n\n        # Get label\n        _, _, size = masks.shape\n        label = info['label']\n        class_ids = np.full(size, label, dtype=np.int32)\n        \n        return masks, class_ids","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:37:26.785670Z","iopub.execute_input":"2022-11-10T20:37:26.786120Z","iopub.status.idle":"2022-11-10T20:37:26.795484Z","shell.execute_reply.started":"2022-11-10T20:37:26.786080Z","shell.execute_reply":"2022-11-10T20:37:26.794743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create Training Dataset\ndataset_train = CellDataset()\ndataset_train.load_data(id_unique, 'png', 'train')\ndataset_train.prepare()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:37:26.797041Z","iopub.execute_input":"2022-11-10T20:37:26.797378Z","iopub.status.idle":"2022-11-10T20:37:26.959793Z","shell.execute_reply.started":"2022-11-10T20:37:26.797341Z","shell.execute_reply":"2022-11-10T20:37:26.959121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot Training Samples with Target Masks, note the instance classes!\ndataset = dataset_train\nimage_ids = np.random.choice(dataset.image_ids, 5)\nfor image_id in image_ids:\n    image = dataset.load_image(image_id)\n    mask, class_ids = dataset.load_mask(image_id)\n    visualize.display_top_masks(image, mask, class_ids, dataset.class_names)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:37:26.961114Z","iopub.execute_input":"2022-11-10T20:37:26.961539Z","iopub.status.idle":"2022-11-10T20:37:33.728259Z","shell.execute_reply.started":"2022-11-10T20:37:26.961504Z","shell.execute_reply":"2022-11-10T20:37:33.727474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model","metadata":{}},{"cell_type":"code","source":"# Create model in training mode\n!mkdir 'model_checkpoints'\nmodel = modellib.MaskRCNN(mode=\"training\", config=config, model_dir='model_checkpoints')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:37:33.729604Z","iopub.execute_input":"2022-11-10T20:37:33.730380Z","iopub.status.idle":"2022-11-10T20:37:47.839019Z","shell.execute_reply.started":"2022-11-10T20:37:33.730341Z","shell.execute_reply":"2022-11-10T20:37:47.838138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"init_with = \"coco\"\nexclude = [\"mrcnn_class_logits\", \"mrcnn_bbox_fc\", \"mrcnn_bbox\", \"mrcnn_mask\"]\nif 'efficientnetv2-b' in config.BACKBONE:\n    exclude += [\n        \"fpn_c5p5\", \"fpn_c4p4\", \"fpn_c3p3\", \"fpn_c2p2\",\n    ]\n    \n# Excluce FC layer if it is not the original size\nif config.FPN_CLASSIF_FC_LAYERS_SIZE != 1024:\n    print(f'Excluding FC layer')\n    exclude += [\n        \"mrcnn_class_conv1\", \"mrcnn_class_bn1\", \"mrcnn_class_conv2\", \"mrcnn_class_bn2\",\n    ]\n    \n# using coco weights\nmodel.load_weights(\n    COCO_MODEL_PATH,\n    by_name=True,\n    exclude=exclude,\n)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:37:47.840429Z","iopub.execute_input":"2022-11-10T20:37:47.840736Z","iopub.status.idle":"2022-11-10T20:37:54.807168Z","shell.execute_reply.started":"2022-11-10T20:37:47.840697Z","shell.execute_reply":"2022-11-10T20:37:54.806386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load EfficientNetV2 Weights Pretrained on Imagenet21K \nmodel.keras_model.layers[1].load_weights('/kaggle/input/maskrcnn-tf-2-efficientnetv2-caching/Instance_Segmentation/efficientnetv2_model_checkpoints/efficientnetv2-b0-imagenet21k.h5')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:37:54.808710Z","iopub.execute_input":"2022-11-10T20:37:54.809007Z","iopub.status.idle":"2022-11-10T20:37:56.321067Z","shell.execute_reply.started":"2022-11-10T20:37:54.808968Z","shell.execute_reply":"2022-11-10T20:37:56.320055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Summary","metadata":{}},{"cell_type":"code","source":"# Added functionality, plot model summary\nmodel.show_summary()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:37:56.322775Z","iopub.execute_input":"2022-11-10T20:37:56.323069Z","iopub.status.idle":"2022-11-10T20:37:59.173205Z","shell.execute_reply.started":"2022-11-10T20:37:56.323031Z","shell.execute_reply":"2022-11-10T20:37:59.172086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show Mask-RCNN Architecture\nplt.figure(figsize=(25, 10))\nplt.title('Mask-RCNN Model Architecture')\nplt.imshow(imageio.imread('./model.png'))\nplt.axis(False)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:37:59.175658Z","iopub.execute_input":"2022-11-10T20:37:59.176069Z","iopub.status.idle":"2022-11-10T20:38:03.807256Z","shell.execute_reply.started":"2022-11-10T20:37:59.176015Z","shell.execute_reply":"2022-11-10T20:38:03.806498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Learning Rate Scheduler","metadata":{}},{"cell_type":"code","source":"# Added functionality, show learning rate schedule\nmodel.plot_lr_schedule(EPOCHS_ALL)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:38:03.808743Z","iopub.execute_input":"2022-11-10T20:38:03.809285Z","iopub.status.idle":"2022-11-10T20:38:04.350214Z","shell.execute_reply.started":"2022-11-10T20:38:03.809244Z","shell.execute_reply":"2022-11-10T20:38:04.349445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training Whole Model","metadata":{}},{"cell_type":"markdown","source":"here the actual training happens. Training will be increasingly fast as the training samples are cached and cache hits increase.","metadata":{}},{"cell_type":"code","source":"start_train = time.time()\nhistory = model.train(\n    dataset_train, None, \n    learning_rate=config.LEARNING_RATE,\n    epochs=EPOCHS_ALL, \n    layers=\"all\",\n    augmentation=None,\n)\n\nend_train = time.time()\nminutes = round((end_train - start_train) / 60, 2)\nprint(f'Training took {minutes} minutes')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T20:38:04.351699Z","iopub.execute_input":"2022-11-10T20:38:04.352177Z","iopub.status.idle":"2022-11-10T22:35:33.970602Z","shell.execute_reply.started":"2022-11-10T20:38:04.352138Z","shell.execute_reply":"2022-11-10T22:35:33.969544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training History","metadata":{}},{"cell_type":"code","source":"# Plots a metric\ndef plot_history_metric(metric, f_best=np.argmax):\n    values = history.history[metric]\n    plt.figure(figsize=(15, 8))\n    N_EPOCHS = len(values)\n    # Epoch Ticks\n    if N_EPOCHS <= 20:\n        x = np.arange(1, N_EPOCHS + 1)\n    else:\n        x = [1, 5] + [10 + 5 * idx for idx in range((N_EPOCHS - 10) // 5 + 1)]\n    x_ticks = np.arange(1, N_EPOCHS+1)\n        \n    # summarize history for accuracy\n    plt.plot(x_ticks, values, label='train')\n    argmin = f_best(values)\n    plt.scatter(argmin + 1, values[argmin], color='red', s=75, marker='o', label='train_best')\n    \n    plt.title(f'Model {metric}', fontsize=24, pad=10)\n    plt.ylabel(metric, fontsize=20, labelpad=10)\n    plt.xlabel('epoch', fontsize=20, labelpad=10)\n    plt.tick_params(axis='x', labelsize=8)\n    plt.xticks(x, fontsize=16) # set tick step to 1 and let x axis start at 1\n    plt.yticks(fontsize=16)\n    plt.legend(prop={'size': 18})\n    plt.grid()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T22:35:33.972482Z","iopub.execute_input":"2022-11-10T22:35:33.972931Z","iopub.status.idle":"2022-11-10T22:35:33.982868Z","shell.execute_reply.started":"2022-11-10T22:35:33.972889Z","shell.execute_reply":"2022-11-10T22:35:33.982011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Mean loss\nplot_history_metric('loss', f_best=np.argmin)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T22:35:33.984327Z","iopub.execute_input":"2022-11-10T22:35:33.985040Z","iopub.status.idle":"2022-11-10T22:35:34.394188Z","shell.execute_reply.started":"2022-11-10T22:35:33.984995Z","shell.execute_reply":"2022-11-10T22:35:34.393438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Region Proposal Network Foreground / Background Classifier\nplot_history_metric('rpn_class_loss', f_best=np.argmin)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T22:35:34.395706Z","iopub.execute_input":"2022-11-10T22:35:34.396199Z","iopub.status.idle":"2022-11-10T22:35:34.812086Z","shell.execute_reply.started":"2022-11-10T22:35:34.396161Z","shell.execute_reply":"2022-11-10T22:35:34.811424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Region Proposal Network Bounding Box Loss\nplot_history_metric('rpn_bbox_loss', f_best=np.argmin)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T22:35:34.816838Z","iopub.execute_input":"2022-11-10T22:35:34.817054Z","iopub.status.idle":"2022-11-10T22:35:35.227322Z","shell.execute_reply.started":"2022-11-10T22:35:34.817028Z","shell.execute_reply":"2022-11-10T22:35:35.226486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Mask RCNN Head Class Classifier Background / specific class\nplot_history_metric('mrcnn_class_loss', f_best=np.argmin)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T22:35:35.228796Z","iopub.execute_input":"2022-11-10T22:35:35.229064Z","iopub.status.idle":"2022-11-10T22:35:35.631241Z","shell.execute_reply.started":"2022-11-10T22:35:35.229031Z","shell.execute_reply":"2022-11-10T22:35:35.630543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Mask RCNN Head Bounding Box Loss\nplot_history_metric('mrcnn_bbox_loss', f_best=np.argmin)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T22:35:35.632646Z","iopub.execute_input":"2022-11-10T22:35:35.633072Z","iopub.status.idle":"2022-11-10T22:35:36.039253Z","shell.execute_reply.started":"2022-11-10T22:35:35.633035Z","shell.execute_reply":"2022-11-10T22:35:36.038534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Mask RCNN Head Object Mask Binary Cross Entropy Loss\nplot_history_metric('mrcnn_mask_loss', f_best=np.argmin)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T22:35:36.040698Z","iopub.execute_input":"2022-11-10T22:35:36.041747Z","iopub.status.idle":"2022-11-10T22:35:36.489446Z","shell.execute_reply.started":"2022-11-10T22:35:36.041704Z","shell.execute_reply":"2022-11-10T22:35:36.488673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Inference","metadata":{}},{"cell_type":"code","source":"class InferenceConfig(CellConfig):\n    IMAGES_PER_GPU = 1\n    DETECTION_MAX_INSTANCES = 200\n    DETECTION_MIN_CONFIDENCE = 0.70\n    USE_MINI_MASK = False\n    \n\ninference_config = InferenceConfig()\ninference_config.display()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T22:35:36.490688Z","iopub.execute_input":"2022-11-10T22:35:36.491323Z","iopub.status.idle":"2022-11-10T22:35:36.511049Z","shell.execute_reply.started":"2022-11-10T22:35:36.491271Z","shell.execute_reply":"2022-11-10T22:35:36.510292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Recreate the model in inference mode\nmodel = modellib.MaskRCNN(mode=\"inference\", config=inference_config, model_dir='model_checkpoints')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T22:35:36.514135Z","iopub.execute_input":"2022-11-10T22:35:36.514386Z","iopub.status.idle":"2022-11-10T22:35:44.325215Z","shell.execute_reply.started":"2022-11-10T22:35:36.514357Z","shell.execute_reply":"2022-11-10T22:35:44.324339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Set EfficientNetV2 head untrainable\nif 'efficientnetv2-b' in  inference_config.BACKBONE:\n    model.keras_model.layers[1].layers[-1].trainable = False","metadata":{"execution":{"iopub.status.busy":"2022-11-10T22:35:44.326671Z","iopub.execute_input":"2022-11-10T22:35:44.326978Z","iopub.status.idle":"2022-11-10T22:35:44.335164Z","shell.execute_reply.started":"2022-11-10T22:35:44.326934Z","shell.execute_reply":"2022-11-10T22:35:44.334358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Find last epoch\nmodel_path = model.find_last()\n\n# Load trained weights (fill in path to trained weights here)\nprint(\"Loading weights from \", model_path)\nmodel.load_weights(model_path, by_name=True)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T22:35:44.337496Z","iopub.execute_input":"2022-11-10T22:35:44.338106Z","iopub.status.idle":"2022-11-10T22:35:46.947336Z","shell.execute_reply.started":"2022-11-10T22:35:44.338067Z","shell.execute_reply":"2022-11-10T22:35:46.946505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualize Train Predictions","metadata":{}},{"cell_type":"code","source":"for file_path in glob.glob('./train/*.png')[:10]:\n    img = skimage.io.imread(file_path)\n    img = np.expand_dims(img, axis=2)\n    img = np.concatenate((img, img, img), axis=2)\n    results = model.detect([img], verbose=1)\n    r = results[0]\n    \n    # Image Id\n    image_id = file_path.split('/')[-1].split('.')[0]\n    print(f'image_id: {image_id}')\n    \n    mask = rle_decode_by_image_id(image_id)\n    mask = np.sum(mask, axis=2)\n    plt.figure(figsize=(16,16))\n    plt.imshow(mask)\n    plt.show()\n    \n    visualize.display_instances(\n        img,\n        r['rois'],\n        r['masks'],\n        r['class_ids'], \n        ['BG'] + CELL_NAMES.tolist(),\n        r['scores'],\n        figsize=(16,16)\n    )","metadata":{"execution":{"iopub.status.busy":"2022-11-10T22:35:46.948687Z","iopub.execute_input":"2022-11-10T22:35:46.948995Z","iopub.status.idle":"2022-11-10T22:37:26.061277Z","shell.execute_reply.started":"2022-11-10T22:35:46.948955Z","shell.execute_reply":"2022-11-10T22:37:26.060530Z"},"trusted":true},"execution_count":null,"outputs":[]}]}