{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction \n","metadata":{}},{"cell_type":"markdown","source":"# References and Resources\n* Pytorch RCNN : [ https://www.kaggle.com/julian3833/sartorius-starter-torch-mask-r-cnn-lb-0-273 ]\n* TF Efficientdet : [ https://www.kaggle.com/dschettler8845/train-sartorius-segmentation-eda-effdet-tf ]\n* PyTorch Detectron : P1 [ https://www.kaggle.com/slawekbiel/positive-score-with-detectron-1-3-input-data/ ], P2 [ https://www.kaggle.com/slawekbiel/positive-score-with-detectron-2-3-training/notebook ]\n* Mask RCNN Paper : [  https://arxiv.org/pdf/1703.06870v3.pdf ]\n* Run Length Encoding : [ https://en.wikipedia.org/wiki/Run-length_encoding#Example ]\n* Open CV [ https://learnopencv.com/getting-started-with-opencv/ ]\n* Pytorch UNET : [ https://www.kaggle.com/ebinan92/unet-with-deep-watershed-transform-dwt-train ]\n* Keras UNET   :  [ https://www.kaggle.com/ammarnassanalhajali/sartorius-segmentation-keras-u-net-training ]\n* Pytorch Detectron [  ]https://www.kaggle.com/ammarnassanalhajali/sartorius-segmentation-detectron2-training ]\n* Guide to using detectron2 : [ https://www.analyticsvidhya.com/blog/2021/08/your-guide-to-object-detection-with-detectron2-in-pytorch/ ]\n* captum ai segmentation tutorial : [ https://captum.ai/tutorials/Segmentation_Interpret ]\n* [ https://www.tensorflow.org/tutorials/images/segmentation ] \n* TF UNET [ www.kaggle.com/arunamenon/cell-instance-segmentation-unet-eda ]\n* [ https://medium.com/@ngocson2vn/a-gentle-explanation-of-backpropagation-in-convolutional-neural-network-cnn-1a70abff508b ]\n* PDF - NEURONAL CELL TYPES : [https://www.cell.com/current-biology/pdf/S0960-9822(04)00440-3.pdf] \n* Youtube :UNET FOR SEGMENTATION - Digital Sreeni: [https://www.youtube.com/watch?v=lOZDTDOlqfk]\n* Youtube : CELL IMAGE SEGMENTATION - MIT           : [https://www.youtube.com/watch?v=lOZDTDOlqfk]\n* [https://spark-in.me/post/playing-with-dwt-and-ds-bowl-2018] ","metadata":{}},{"cell_type":"markdown","source":"# **Installing detectron**\n\n* Detectron Starter [ https://www.analyticsvidhya.com/blog/2021/08/your-guide-to-object-detection-with-detectron2-in-pytorch/ ]\n\n","metadata":{}},{"cell_type":"code","source":"!pip install 'git+https://github.com/facebookresearch/detectron2.git' -q\n!pip install pycocotools -q","metadata":{"execution":{"iopub.status.busy":"2021-12-14T10:19:03.229860Z","iopub.execute_input":"2021-12-14T10:19:03.230277Z","iopub.status.idle":"2021-12-14T10:22:57.643709Z","shell.execute_reply.started":"2021-12-14T10:19:03.230229Z","shell.execute_reply":"2021-12-14T10:22:57.642371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"# pytorch\nimport torch, torchvision\nprint(torch.__version__)\nif  torch.cuda.is_available():\n    print('GPU \\n')\n    !nvidia-smi","metadata":{"execution":{"iopub.status.busy":"2021-12-14T10:22:57.647686Z","iopub.execute_input":"2021-12-14T10:22:57.648039Z","iopub.status.idle":"2021-12-14T10:22:58.473794Z","shell.execute_reply.started":"2021-12-14T10:22:57.647995Z","shell.execute_reply":"2021-12-14T10:22:58.472708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# general libraries\nimport pandas as pd\nimport numpy as np\nimport pandas as pd \nimport os, json, cv2, random\nfrom tqdm import tqdm_notebook as tqdm \nimport matplotlib.pyplot as plt\n# import skimage.io as io\nfrom pathlib import Path\n\n\n# for data preprocessing and augmentation\nimport albumentations as A\nfrom albumentations.pytorch.transforms import ToTensorV2\n\n\n#coco \nfrom pycocotools.coco import COCO\nfrom pycocotools import mask as maskUtils\nfrom joblib import Parallel, delayed\n\n# detectron2\nimport pycocotools.mask as mask_util\nfrom detectron2.structures import BoxMode\nfrom detectron2 import model_zoo                 # pretrained models are available in model zoo\nfrom detectron2.config import get_cfg\nfrom detectron2.engine import DefaultPredictor, DefaultTrainer, launch\nfrom detectron2.structures import BoxMode\nfrom detectron2.utils.visualizer import ColorMode\nfrom detectron2.utils.logger import setup_logger\nfrom detectron2.utils.visualizer import Visualizer\n\n\n# data loading and utils\nfrom detectron2.data import DatasetCatalog, MetadataCatalog, build_detection_test_loader, build_detection_train_loader\nfrom detectron2.data.datasets import register_coco_instances\nfrom detectron2.data import detection_utils as utils\nimport detectron2.data.transforms as T\nfrom detectron2.evaluation import COCOEvaluator, inference_on_dataset\nfrom detectron2.evaluation.evaluator import DatasetEvaluator\n\n\n# filter warinings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n\n# Initialize the detectron2 logger and set its verbosity level to \"DEBUG\".\nsetup_logger()","metadata":{"execution":{"iopub.status.busy":"2021-12-14T10:23:26.091051Z","iopub.execute_input":"2021-12-14T10:23:26.092174Z","iopub.status.idle":"2021-12-14T10:23:26.110935Z","shell.execute_reply.started":"2021-12-14T10:23:26.092121Z","shell.execute_reply":"2021-12-14T10:23:26.109646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Sartorius Cell Instance Segmentation : Data Summary .**\n\n* We have 606 Images in the Train set, and 3 Images in the public test set. Images are in PNG format.\n* The annotations and paths for training images are given in the train.csv.\n* The submission format is as given in sample_submission.csv.\n* There are about 1900+ Images present without annotations, which can be used for semi-supervised learning at path '../input/sartorius-cell-instance-segmentation/train_semi_supervised'.\n* There are also Images from LIVECell dataset with annotations, which could be included in training at path '../input/sartorius-cell-instance-segmentation/LIVECell_dataset_2021'.\n","metadata":{}},{"cell_type":"code","source":"#directory paths \n\nlivecell_ds = '../input/sartorius-cell-instance-segmentation/LIVECell_dataset_2021'\ntrain_dir = '../input/sartorius-cell-instance-segmentation/train'\ntest_dir  =  '../input/sartorius-cell-instance-segmentation/test'\n\n\n#csv files \nsample_sub = pd.read_csv('../input/sartorius-cell-instance-segmentation/sample_submission.csv')\ntrain  = pd.read_csv('../input/sartorius-cell-instance-segmentation/train.csv')\n\n\n#checking the csv \ntrain.head(2)","metadata":{"execution":{"iopub.status.busy":"2021-12-14T10:23:29.265062Z","iopub.execute_input":"2021-12-14T10:23:29.265382Z","iopub.status.idle":"2021-12-14T10:23:29.849020Z","shell.execute_reply.started":"2021-12-14T10:23:29.265350Z","shell.execute_reply":"2021-12-14T10:23:29.848065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preparing the data in coco format.","metadata":{}},{"cell_type":"markdown","source":"**Getting the data ready in COCO format (Just for demonstration purpose,using a pre-prepared dataset from [ https://www.kaggle.com/slawekbiel/positive-score-with-detectron-2-3-training/notebook ].)**","metadata":{}},{"cell_type":"code","source":"#from : https://www.kaggle.com/coldfir3/efficient-coco-dataset-generator?scriptVersionId=79100851\n\ndef rle2mask(rle, img_w, img_h):\n    '''convert run length encoding to a image mask'''\n    ## transforming the string into an array of shape (2, N)\n    array = np.fromiter(rle.split(), dtype = np.uint)\n    array = array.reshape((-1,2)).T\n    array[0] = array[0] - 1\n    \n    ## decompressing the rle encoding (ie, turning [3, 1, 10, 2] into [3, 4, 10, 11, 12])\n    # for faster mask construction\n    starts, lenghts = array\n    mask_decompressed = np.concatenate([np.arange(s, s + l, dtype = np.uint) for s, l in zip(starts, lenghts)])\n\n    ## Building the binary mask\n    msk_img = np.zeros(img_w * img_h, dtype = np.uint8)\n    msk_img[mask_decompressed] = 1\n    msk_img = msk_img.reshape((img_h, img_w))\n    msk_img = np.asfortranarray(msk_img) ## This is important so pycocotools can handle this object\n    \n    return msk_img\n\n\ndef annotate(idx, row, cat_ids):\n    \n    mask = rle2mask(row['annotation'], row['width'], row['height']) # Binary mask\n    c_rle = maskUtils.encode(mask) # Encoding it back to rle (coco format)\n    c_rle['counts'] = c_rle['counts'].decode('utf-8') # converting from binary to utf-8\n    area = maskUtils.area(c_rle).item() # calculating the area\n    bbox = maskUtils.toBbox(c_rle).astype(int).tolist() # calculating the bboxes\n    annotation = {\n        'segmentation': c_rle,\n        'bbox': bbox,\n        'area': area,\n        'image_id':row['id'], \n        'category_id':cat_ids[row['cell_type']], \n        'iscrowd':0, \n        'id':idx\n    }\n    return annotation\n\ndef coco_structure(df, workers = 6):\n    \n    ## Building the header\n    cat_ids = {name:idx+1 for idx, name in enumerate(df.cell_type.unique())}    \n    \n    cats =[{'name':name, 'id':idx} for name,idx in cat_ids.items()]\n    \n    images = [{'id':idx, 'width':row.width, 'height':row.height, 'file_name':f'train/{idx}.png'} for idx,row in df.groupby('id').agg('first').iterrows()]\n    \n    # Building the annotations\n    annotations = Parallel(n_jobs=workers)(delayed(annotate)(idx, row, cat_ids) for idx, row in tqdm(df.iterrows(), total = len(df)))\n        \n    return {'categories':cats, 'images':images, 'annotations':annotations}\n\n\n\n\n#checking the code on first 1000 rows in train csv \ncoco_annotations = coco_structure(train.iloc[:1000])\n\nwith open('annotations_train_sample.json', 'w', encoding='utf-8') as f:\n    json.dump(coco_annotations, f, ensure_ascii=True, indent=4)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-14T10:25:48.300204Z","iopub.execute_input":"2021-12-14T10:25:48.300587Z","iopub.status.idle":"2021-12-14T10:25:51.175413Z","shell.execute_reply.started":"2021-12-14T10:25:48.300553Z","shell.execute_reply":"2021-12-14T10:25:51.174343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# look at the generated format\n\nprint('Categories in the data',coco_annotations['categories'])\nprint('_'*50,'---','_'*50)\nprint('Images dict example', coco_annotations['images'][0])\nprint('_'*50,'---','_'*50)\nprint('Annotations dict example',coco_annotations['annotations'][0])","metadata":{"execution":{"iopub.status.busy":"2021-12-14T10:32:23.172562Z","iopub.execute_input":"2021-12-14T10:32:23.172869Z","iopub.status.idle":"2021-12-14T10:32:23.183169Z","shell.execute_reply.started":"2021-12-14T10:32:23.172836Z","shell.execute_reply":"2021-12-14T10:32:23.182102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading data ","metadata":{}},{"cell_type":"code","source":"# laoding data\n\n\n# path to the data preprocessed by : https://www.kaggle.com/slawekbiel\ndataDir=Path('../input/sartorius-cell-instance-segmentation/')\n\n\n#get configuration \ncfg = get_cfg()\ncfg.INPUT.MASK_FORMAT='bitmask'\n\n#resgister coco instance\n#train\nregister_coco_instances('sartorius_train',{}, '../input/sartorius-cell-instance-segmentation-coco/annotations_train.json', dataDir)\n#validation \nregister_coco_instances('sartorius_val',{},'../input/sartorius-cell-instance-segmentation-coco/annotations_val.json', dataDir)\n#all the train data \nregister_coco_instances('sartorius_train_full',{},'../input/sartorius-cell-instance-segmentation-coco/annotations_all.json', dataDir)\n","metadata":{"execution":{"iopub.status.busy":"2021-12-14T10:34:54.695444Z","iopub.execute_input":"2021-12-14T10:34:54.695759Z","iopub.status.idle":"2021-12-14T10:34:58.955098Z","shell.execute_reply.started":"2021-12-14T10:34:54.695725Z","shell.execute_reply":"2021-12-14T10:34:58.954033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get train data \nmetadata = MetadataCatalog.get('sartorius_train')\ntrain_ds = DatasetCatalog.get('sartorius_train')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get all train data (all 606 Images)\nall_metadata = MetadataCatalog.get('sartorius_train_full')\nall_dat = DatasetCatalog.get('sartorius_train_full')","metadata":{"execution":{"iopub.status.busy":"2021-12-14T10:35:05.437092Z","iopub.execute_input":"2021-12-14T10:35:05.437537Z","iopub.status.idle":"2021-12-14T10:35:10.208701Z","shell.execute_reply.started":"2021-12-14T10:35:05.437495Z","shell.execute_reply":"2021-12-14T10:35:10.207678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Visualizing some samples**","metadata":{}},{"cell_type":"code","source":"#select image \nch = train_ds[5]\nimg = cv2.imread(ch[\"file_name\"])\n\n#get visual\nvisualizer = Visualizer(img[:, :, ::-1], metadata=metadata)\nout = visualizer.draw_dataset_dict(ch)\n\n#plot \nplt.figure(figsize = (20,15))\nplt.imshow(out.get_image()[:, :, ::-1])\nplt.axis('off')\nplt.title('sample image')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-14T10:36:45.170410Z","iopub.execute_input":"2021-12-14T10:36:45.171093Z","iopub.status.idle":"2021-12-14T10:36:46.831432Z","shell.execute_reply.started":"2021-12-14T10:36:45.171055Z","shell.execute_reply":"2021-12-14T10:36:46.830492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ch = train_ds[10]\nimg = cv2.imread(ch[\"file_name\"])\n\n\nvisualizer = Visualizer(img[:, :, ::-1], metadata=metadata)\nout = visualizer.draw_dataset_dict(ch)\n\nplt.figure(figsize = (20,15))\nplt.imshow(out.get_image()[:, :, ::-1])\nplt.axis('off')\nplt.title('sample image')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-14T10:48:27.381464Z","iopub.execute_input":"2021-12-14T10:48:27.381780Z","iopub.status.idle":"2021-12-14T10:48:32.054145Z","shell.execute_reply.started":"2021-12-14T10:48:27.381748Z","shell.execute_reply":"2021-12-14T10:48:32.053009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ch = train_ds[100]\nimg = cv2.imread(ch[\"file_name\"])\n\n\nvisualizer = Visualizer(img[:, :, ::-1], metadata=metadata)\nout = visualizer.draw_dataset_dict(ch)\n\nplt.figure(figsize = (20,15))\nplt.imshow(out.get_image()[:, :, ::-1])\nplt.axis('off')\nplt.title('sample image')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:15:48.976753Z","iopub.execute_input":"2021-12-14T12:15:48.977071Z","iopub.status.idle":"2021-12-14T12:15:49.817686Z","shell.execute_reply.started":"2021-12-14T12:15:48.977038Z","shell.execute_reply":"2021-12-14T12:15:49.813734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Metrics and Training Helper functions","metadata":{}},{"cell_type":"code","source":"# Taken from https://www.kaggle.com/theoviel/competition-metric-map-iou\n\n\ndef precision_at(threshold, iou):\n    matches = iou > threshold\n    true_positives = np.sum(matches, axis=1) == 1  # Correct objects\n    false_positives = np.sum(matches, axis=0) == 0  # Missed objects\n    false_negatives = np.sum(matches, axis=1) == 0  # Extra objects\n    return np.sum(true_positives), np.sum(false_positives), np.sum(false_negatives)\n\ndef score(pred, targ):\n    pred_masks = pred['instances'].pred_masks.cpu().numpy()\n    enc_preds = [mask_util.encode(np.asarray(p, order='F')) for p in pred_masks]\n    enc_targs = list(map(lambda x:x['segmentation'], targ))\n    ious = mask_util.iou(enc_preds, enc_targs, [0]*len(enc_targs))\n    prec = []\n    for t in np.arange(0.5, 1.0, 0.05):\n        tp, fp, fn = precision_at(t, ious)\n        p = tp / (tp + fp + fn)\n        prec.append(p)\n    return np.mean(prec)\n\nclass MAPIOUEvaluator(DatasetEvaluator):\n    def __init__(self, dataset_name):\n        dataset_dicts = DatasetCatalog.get(dataset_name)\n        self.annotations_cache = {item['image_id']:item['annotations'] for item in dataset_dicts}\n            \n    def reset(self):\n        self.scores = []\n\n    def process(self, inputs, outputs):\n        for inp, out in zip(inputs, outputs):\n            if len(out['instances']) == 0:\n                self.scores.append(0)    \n            else:\n                targ = self.annotations_cache[inp['image_id']]\n                self.scores.append(score(out, targ))\n\n    def evaluate(self):\n        return {\"MaP IoU\": np.mean(self.scores)}\n\nclass Trainer(DefaultTrainer):\n    @classmethod\n    def build_evaluator(cls, cfg, dataset_name, output_folder=None):\n        return MAPIOUEvaluator(dataset_name)\n    ","metadata":{"execution":{"iopub.status.busy":"2021-12-14T10:47:44.177558Z","iopub.execute_input":"2021-12-14T10:47:44.178471Z","iopub.status.idle":"2021-12-14T10:47:44.193766Z","shell.execute_reply.started":"2021-12-14T10:47:44.178434Z","shell.execute_reply":"2021-12-14T10:47:44.192669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Setting Config for the model","metadata":{}},{"cell_type":"code","source":"#getting Mask RCNN architecture -get config file for RCC_resnet50\ncfg.merge_from_file(model_zoo.get_config_file(\"COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml\"))\n\n\n#set training and test data dir\n\n#using all the data for training\ncfg.DATASETS.TRAIN = (\"sartorius_train_full\",)\n\n#validation data(inc in training)\ncfg.DATASETS.TEST = (\"sartorius_val\",)\n\n\n#num workers\ncfg.DATALOADER.NUM_WORKERS = 2\n\n# loading pretrained weights\ncfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(\"COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml\")  # Let training initialize from model zoo\n\n#default\ncfg.SOLVER.IMS_PER_BATCH = 2\n\n#learning rate for base model\ncfg.SOLVER.BASE_LR = 0.00075\n\n# number of iterations\ncfg.SOLVER.MAX_ITER = 2000\n\n#Region of Interest batch size\ncfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128   \n\n#number of output classes\ncfg.MODEL.ROI_HEADS.NUM_CLASSES = 3\n\n#threshold\ncfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = .5\n\n#from : https://www.kaggle.com/ammarnassanalhajali/sartorius-segmentation-detectron2-training\ncfg.SOLVER.WARMUP_ITERS = 10 #How many iterations to go from 0 to reach base LR\ncfg.SOLVER.STEPS = (500, 1000) #At which point to change the LR 0.25,0.5\ncfg.TEST.EVAL_PERIOD = 250\ncfg.SOLVER.CHECKPOINT_PERIOD=250\n\n\ncfg.TEST.EVAL_PERIOD = len(DatasetCatalog.get('sartorius_train_full')) // cfg.SOLVER.IMS_PER_BATCH  # Once per epoch","metadata":{"execution":{"iopub.status.busy":"2021-12-14T10:50:21.191302Z","iopub.execute_input":"2021-12-14T10:50:21.191638Z","iopub.status.idle":"2021-12-14T10:50:24.862009Z","shell.execute_reply.started":"2021-12-14T10:50:21.191588Z","shell.execute_reply":"2021-12-14T10:50:24.860952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#training\n\n#make a directory to store model outputs\nos.makedirs(cfg.OUTPUT_DIR, exist_ok=True)\n\n#intanciate trainer with configs from pervious cells\ntrainer = Trainer(cfg) \ntrainer.resume_or_load(resume=True)\n\n#start train\ntrainer.train()","metadata":{"execution":{"iopub.status.busy":"2021-12-14T10:51:05.249030Z","iopub.execute_input":"2021-12-14T10:51:05.249583Z","iopub.status.idle":"2021-12-14T11:53:22.733510Z","shell.execute_reply.started":"2021-12-14T10:51:05.249546Z","shell.execute_reply":"2021-12-14T11:53:22.732270Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Plotting History**","metadata":{}},{"cell_type":"code","source":"metrics_df.columns","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:11:15.319070Z","iopub.execute_input":"2021-12-14T12:11:15.319753Z","iopub.status.idle":"2021-12-14T12:11:15.329813Z","shell.execute_reply.started":"2021-12-14T12:11:15.319715Z","shell.execute_reply":"2021-12-14T12:11:15.328440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"metrics_df = pd.read_json(\"./output/metrics.json\",\n                          orient=\"records\",\n                          lines=True).sort_values(\"iteration\")\n\n#get values \nx1 = metrics_df.iteration\nl_cls=metrics_df.loss_cls\nl_msk=metrics_df.loss_mask\nl_bb=metrics_df.loss_box_reg\n\n\nplt.subplots(figsize=(16,8))\nplt.plot(x1,l_cls,color='b',label='Class_loss')\nplt.plot(x1,l_msk,color='k',label='Mask_loss')\nplt.plot(x1,l_bb,color='g',label='BBox_loss')\n\nplt.title('Train Summary')\nplt.legend()\nplt.xlabel('Iterations')\nplt.ylabel('Loss')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:13:54.000061Z","iopub.execute_input":"2021-12-14T12:13:54.000693Z","iopub.status.idle":"2021-12-14T12:13:54.438268Z","shell.execute_reply.started":"2021-12-14T12:13:54.000655Z","shell.execute_reply":"2021-12-14T12:13:54.436703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# PLotting predicted Images against Original Images","metadata":{}},{"cell_type":"code","source":"cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, \"model_final.pth\")  # path to the model we just trained\ncfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # set a custom testing threshold\n\n\n#predictor\npredictor = DefaultPredictor(cfg)\n\n#get tvalidataion data for eval\ndataset_dicts = DatasetCatalog.get('sartorius_val')\n\n\n\noutp,outt = [],[]  # predicted, original\n\n#selecting 5 sampeles to evaluate on \nfor d in random.sample(dataset_dicts, 6):    \n    img = cv2.imread(d[\"file_name\"])\n    outputs = predictor(img)  # format is documented at https://detectron2.readthedocs.io/tutorials/models.html#model-output-format\n    \n    #visualize\n    v = Visualizer(img[:, :, ::-1],\n                   metadata = MetadataCatalog.get('sartorius_train'), \n                   instance_mode=ColorMode.IMAGE_BW   # remove the colors of unsegmented pixels. This option is only available for segmentation models\n    )\n    \n    \n    out_pred = v.draw_instance_predictions(outputs[\"instances\"].to(\"cpu\"))\n    visualizer = Visualizer(img[:, :, ::-1])\n    out_target = visualizer.draw_dataset_dict(d)\n    outp.append(out_pred)\n    outt.append(out_target)\n","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:19:23.938711Z","iopub.execute_input":"2021-12-14T12:19:23.939397Z","iopub.status.idle":"2021-12-14T12:19:33.621752Z","shell.execute_reply.started":"2021-12-14T12:19:23.939346Z","shell.execute_reply":"2021-12-14T12:19:33.620631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.subplots(6,2,figsize=(30,60))\n\nn=1\nfor i in range(len(outp)):\n    plt.subplot(6,2,n)\n    plt.imshow(outp[i].get_image())\n    plt.axis('off')\n    plt.title('Predicted(Thresh=0.5)')\n    n+=1\n    \n    plt.subplot(6,2,n)\n    plt.imshow(outt[i].get_image())\n    plt.axis('off')\n    plt.title('Original')\n    if n==12:\n        break\n    n+=1\n    \n    \nplt.tight_layout()\nplt.show()\n    ","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:19:33.623342Z","iopub.execute_input":"2021-12-14T12:19:33.623740Z","iopub.status.idle":"2021-12-14T12:19:42.414520Z","shell.execute_reply.started":"2021-12-14T12:19:33.623683Z","shell.execute_reply":"2021-12-14T12:19:42.413308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Visualizing with a different prediction threshold**","metadata":{}},{"cell_type":"markdown","source":"**Thresh = 0.35**","metadata":{}},{"cell_type":"code","source":"# changing pred thresh\ncfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.35\n\n\n#predictor\npredictor = DefaultPredictor(cfg)\n\n\noutp,outt = [],[]  # predicted, original\n\n#selecting 5 sampeles to evaluate on \nfor d in random.sample(dataset_dicts, 10):    \n    img = cv2.imread(d[\"file_name\"])\n    outputs = predictor(img)  # format is documented at https://detectron2.readthedocs.io/tutorials/models.html#model-output-format\n    \n    #visualize\n    v = Visualizer(img[:, :, ::-1],\n                   metadata = MetadataCatalog.get('sartorius_train'), \n                   instance_mode=ColorMode.IMAGE_BW   # remove the colors of unsegmented pixels. This option is only available for segmentation models\n    )\n    \n    \n    out_pred = v.draw_instance_predictions(outputs[\"instances\"].to(\"cpu\"))\n    visualizer = Visualizer(img[:, :, ::-1])\n    out_target = visualizer.draw_dataset_dict(d)\n    outp.append(out_pred)\n    outt.append(out_target)\n","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:20:44.079388Z","iopub.execute_input":"2021-12-14T12:20:44.079717Z","iopub.status.idle":"2021-12-14T12:20:56.358052Z","shell.execute_reply.started":"2021-12-14T12:20:44.079684Z","shell.execute_reply":"2021-12-14T12:20:56.357034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.subplots(6,2,figsize=(30,60))\n\nn=1\nfor i in range(len(outp)):\n    plt.subplot(6,2,n)\n    plt.imshow(outp[i].get_image())\n    plt.axis('off')\n    plt.title('Predicted(Thresh=0.35)')\n    n+=1\n    \n    plt.subplot(6,2,n)\n    plt.imshow(outt[i].get_image())\n    plt.axis('off')\n    plt.title('Original')\n    if n==12:\n        break\n    n+=1\n    \n    \nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:20:56.360355Z","iopub.execute_input":"2021-12-14T12:20:56.360759Z","iopub.status.idle":"2021-12-14T12:21:04.243668Z","shell.execute_reply.started":"2021-12-14T12:20:56.360677Z","shell.execute_reply":"2021-12-14T12:21:04.242258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Thresh = 0.25**","metadata":{}},{"cell_type":"code","source":"# changing pred thresh\ncfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.25\n\n\n#predictor\npredictor = DefaultPredictor(cfg)\n\n\noutp,outt = [],[]  # predicted, original\n\n#selecting 5 sampeles to evaluate on \nfor d in random.sample(dataset_dicts, 10):    \n    img = cv2.imread(d[\"file_name\"])\n    outputs = predictor(img)  # format is documented at https://detectron2.readthedocs.io/tutorials/models.html#model-output-format\n    \n    #visualize\n    v = Visualizer(img[:, :, ::-1],\n                   metadata = MetadataCatalog.get('sartorius_train'), \n                   instance_mode=ColorMode.IMAGE_BW   # remove the colors of unsegmented pixels. This option is only available for segmentation models\n    )\n    \n    \n    out_pred = v.draw_instance_predictions(outputs[\"instances\"].to(\"cpu\"))\n    visualizer = Visualizer(img[:, :, ::-1])\n    out_target = visualizer.draw_dataset_dict(d)\n    outp.append(out_pred)\n    outt.append(out_target)","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:28:34.065485Z","iopub.execute_input":"2021-12-14T12:28:34.065806Z","iopub.status.idle":"2021-12-14T12:28:46.480136Z","shell.execute_reply.started":"2021-12-14T12:28:34.065757Z","shell.execute_reply":"2021-12-14T12:28:46.478978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.subplots(6,2,figsize=(30,60))\n\nn=1\nfor i in range(len(outp)):\n    plt.subplot(6,2,n)\n    plt.imshow(outp[i].get_image())\n    plt.axis('off')\n    plt.title('Predicted(Thresh=0.25)')\n    n+=1\n    \n    plt.subplot(6,2,n)\n    plt.imshow(outt[i].get_image())\n    plt.axis('off')\n    plt.title('Original')\n    if n==12:\n        break\n    n+=1\n    \n    \nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:28:46.482596Z","iopub.execute_input":"2021-12-14T12:28:46.482896Z","iopub.status.idle":"2021-12-14T12:28:54.433695Z","shell.execute_reply.started":"2021-12-14T12:28:46.482855Z","shell.execute_reply":"2021-12-14T12:28:54.432522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Thresh = 0.15**","metadata":{}},{"cell_type":"code","source":"# changing pred thresh\ncfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.15\n\n\n#predictor\npredictor = DefaultPredictor(cfg)\n\n\noutp,outt = [],[]  # predicted, original\n\n#selecting 5 sampeles to evaluate on \nfor d in random.sample(dataset_dicts, 10):    \n    img = cv2.imread(d[\"file_name\"])\n    outputs = predictor(img)  # format is documented at https://detectron2.readthedocs.io/tutorials/models.html#model-output-format\n    \n    #visualize\n    v = Visualizer(img[:, :, ::-1],\n                   metadata = MetadataCatalog.get('sartorius_train'), \n                   instance_mode=ColorMode.IMAGE_BW   # remove the colors of unsegmented pixels. This option is only available for segmentation models\n    )\n    \n    \n    out_pred = v.draw_instance_predictions(outputs[\"instances\"].to(\"cpu\"))\n    visualizer = Visualizer(img[:, :, ::-1])\n    out_target = visualizer.draw_dataset_dict(d)\n    outp.append(out_pred)\n    outt.append(out_target)","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:26:33.897188Z","iopub.execute_input":"2021-12-14T12:26:33.897617Z","iopub.status.idle":"2021-12-14T12:26:48.607115Z","shell.execute_reply.started":"2021-12-14T12:26:33.897555Z","shell.execute_reply":"2021-12-14T12:26:48.606151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.subplots(6,2,figsize=(30,60))\n\nn=1\nfor i in range(len(outp)):\n    plt.subplot(6,2,n)\n    plt.imshow(outp[i].get_image())\n    plt.axis('off')\n    plt.title('Predicted(Thresh=0.15)')\n    n+=1\n    \n    plt.subplot(6,2,n)\n    plt.imshow(outt[i].get_image())\n    plt.axis('off')\n    plt.title('Original')\n    if n==12:\n        break\n    n+=1\n    \n    \nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:27:08.858943Z","iopub.execute_input":"2021-12-14T12:27:08.859282Z","iopub.status.idle":"2021-12-14T12:27:16.061475Z","shell.execute_reply.started":"2021-12-14T12:27:08.859241Z","shell.execute_reply":"2021-12-14T12:27:16.060007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Inference in a Different Notebook**\n\n# Thank You!","metadata":{}}]}