{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"training：https://www.kaggle.com/code/huangzeyuzheng/hubmap-2023-yolov7-baseline/notebook?scriptVersionId=137858694/","metadata":{}},{"cell_type":"markdown","source":" version1/2:baseline of inference for test sample proveded
\n version3：inference with dilation for test sample proveded
\n version4:inference with erode for test sample proveded
\n version5: failed
\n version6：inference with dilation for test sample proveded, add detail commends
\n version7:baseline inference for custom test set, without any pre- or post-processing
\n version8:inference for custom test set with **dilation** postprocessing, no pre-processing
\n version9:inference for custom test set with **erode** postprocessing, no pre-processing
\n version10:inference for custom test set with **imopen** postprocessing, no pre-processing
\n version11:inference for custom test set with **imclose** postprocessing, no pre-processing
\n version13-17:inference using provided test sample on 2 custom trained models with/without dilation respectively
\n ### Pre-process:\n version12:inference for custom test set based on the model trained by blood_vessel only
\n \n ### train v4\n v18-23:inference based on train v4 model
\n v18:baseline inference on trainv4 on custom test set
\n v19: inference with dilation on trainv4 on custom test set
\n v20:inferece with dilation on train v4 on test sample
\n v21:inference with imopen on train v4 on custom test set
\n v22:inference with imclose on train v4 on custom test set
\n v23:inference with erode on train v4 on custom test set,**failed, but saved the output already**
\n \n ### train v5\n v24:mistake
\n v25: inference with **imclose** on trainv5 with **test sample**
\n v26:mistake
\n v27: **baseline** inference on trainv5 with **test sample**
\n v28: inference with **dilation** on trainv5 with **test sample**
\n v29: **baseline** inference on trainv5 with **testset**
\n v30: inference with **dilation** on trainv5 with **testset**
\n v31: inference with **erode** on trainv5 with **testset**
\n v33: inference with **imclose** on trainv5 with **testset**
\n v34: inference with **imopen** on trainv5 with **testset**
\n v35: inference with **erode** on trainv5 with **test sample**
\n v36: inference with **imopen** on trainv5 with **test sample**
\n \n ### train v6\n v40: **baseline** inference on trainv6 with **testset**
\n v41: inference with **dilation** on trainv6 with **testset**
\n v42: inference with **erode** on trainv6 with **testset**
\n v43: inference with **imclose** on trainv6 with **testset**
\n v44: inference with **imopen** on trainv6 with **testset**
\n v45: **baseline** inference on trainv6 with **test sample**
\n v46: inference with **dilation** on trainv6 with **test sample**
\n v47: inference with **erode** on trainv6 with **test sample**
\n v48: inference with **imclose** on trainv6 with **test sample**
\n v49: inference with **imopen** on trainv6 with **test sample**
\n \n ### train v7\n v50: **baseline** inference on trainv7 with **testset**
\n v51: inference with **dilation** on trainv7 with **testset**
\n v52: inference with **erode** on trainv7 with **testset**
\n v53: inference with **imclose** on trainv7 with **testset**
\n v54: inference with **imopen** on trainv7 with **testset**
\n v55: **baseline** inference on trainv7 with **test sample**
\n v56: inference with **dilation** on trainv7 with **test sample**
\n v57: inference with **erode** on trainv7 with **test sample**
\n v58: inference with **imclose** on trainv7 with **test sample**
\n v59: inference with **imopen** on trainv7 with **test sample**
\n \n ### train v8\n v60: **baseline** inference on trainv8 with **testset**
\n v61: inference with **dilation** on trainv8 with **testset**
\n v62: inference with **erode** on trainv8 with **testset**
\n v63: inference with **imclose** on trainv8 with **testset**
\n v64: inference with **imopen** on trainv8 with **testset**
\n v65: **baseline** inference on trainv8 with **test sample**
\n v66: inference with **dilation** on trainv8 with **test sample**
\n v67: inference with **erode** on trainv8 with **test sample**
\n v68: inference with **imclose** on trainv8 with **test sample**
\n v69: inference with **imopen** on trainv8 with **test sample**
\n \n ### train v9\n v70: **baseline** inference on trainv9 with **testset**
\n v71: inference with **dilation** on trainv9 with **testset**
\n v72: inference with **erode** on trainv9 with **testset**
\n v73: inference with **imclose** on trainv9 with **testset**
\n v74: inference with **imopen** on trainv9 with **testset**
\n v75: **baseline** inference on trainv9 with **test sample**
\n v76: inference with **dilation** on trainv9 with **test sample**
\n v77: inference with **erode** on trainv9 with **test sample**
\n v78: inference with **imclose** on trainv9 with **test sample**
\n v79: inference with **imopen** on trainv9 with **test sample**
\n \n ### train v10\n v80: **baseline** inference on trainv10 with **testset**
\n v81: inference with **dilation** on trainv10 with **testset**
\n v82: inference with **erode** on trainv10 with **testset**
\n v83: inference with **imclose** on trainv10 with **testset**
\n v84: inference with **imopen** on trainv10 with **testset**
\n v85: **baseline** inference on trainv10 with **test sample**
\n v86: inference with **dilation** on trainv10 with **test sample**
\n v87: inference with **erode** on trainv10 with **test sample**
\n v88: inference with **imclose** on trainv10 with **test sample**
\n v89: inference with **imopen** on trainv10 with **test sample**
\n \n ### train v11\n v90: **baseline** inference on trainv11 with **testset**
\n v91: inference with **dilation** on trainv11 with **testset**
\n v92: inference with **erode** on trainv11 with **testset**
\n v93: inference with **imclose** on trainv11 with **testset**
\n v94: inference with **imopen** on trainv11 with **testset**
\n v95: **baseline** inference on trainv11 with **test sample**
\n v96: inference with **dilation** on trainv11 with **test sample**
\n v97: inference with **erode** on trainv11 with **test sample**
\n v98: inference with **imclose** on trainv11 with **test sample**
\n v99: inference with **imopen** on trainv11 with **test sample**
","metadata":{}},{"cell_type":"markdown","source":"### Install pycocotools","metadata":{}},{"cell_type":"code","source":"!cp -r /kaggle/input/pycocotools/ /kaggle/working/pycocotools\n!pip install /kaggle/working/pycocotools/pycocotools-2.0.6 --no-index --find-links=/kaggle/working/pycocotools/ ","metadata":{"execution":{"iopub.status.busy":"2023-08-02T11:12:53.583902Z","iopub.execute_input":"2023-08-02T11:12:53.584627Z","iopub.status.idle":"2023-08-02T11:13:27.556635Z","shell.execute_reply.started":"2023-08-02T11:12:53.584591Z","shell.execute_reply":"2023-08-02T11:13:27.555460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### librabries","metadata":{}},{"cell_type":"code","source":"#load training output and modle\nimport sys \nsys.path.append('/kaggle/input/hubmap-2023-yolov7-baseline')\nsys.path.append('/kaggle/input/hubmap-2023-yolov7-baseline/yolov7/seg')\n\nimport torch\nprint('torch',torch.__version__)\n\nimport pandas as pd\nimport numpy as np\nfrom glob import glob\n\n%matplotlib inline \nimport matplotlib\nimport matplotlib.pyplot as plt\n\nimport json\n\nimport base64\nfrom pycocotools import _mask as coco_mask\nimport typing as t\nimport zlib\nimport cv2\n\nprint('numpy',np.__version__)\n#import important modules/functions from source code to reproduce inference\nfrom yolov7.seg.models.common import DetectMultiBackend\nfrom yolov7.seg.utils.general import non_max_suppression\nfrom yolov7.seg.utils.segment.general import process_mask, scale_masks ##import for process mask\nfrom yolov7.seg.utils.torch_utils import select_device\n\nprint('IMPORT OK!!!')","metadata":{"execution":{"iopub.status.busy":"2023-08-02T11:14:38.790107Z","iopub.execute_input":"2023-08-02T11:14:38.790506Z","iopub.status.idle":"2023-08-02T11:14:42.956006Z","shell.execute_reply.started":"2023-08-02T11:14:38.790475Z","shell.execute_reply":"2023-08-02T11:14:42.955044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Inference","metadata":{}},{"cell_type":"code","source":"# Create a yaml file as expected by YOLOv7, used to define the classes of whole dataset\n#It also can define the path of train/validation/test folder/file/list\nyaml_text = \"\"\"\n# class names\nnames: \n 0: blood_vessel\n 1: glomerulus\n 2: unsure\n\"\"\"\nwith open('/kaggle/working/hubmap-coco.yaml', 'w') as text_file:\n text_file.write(yaml_text)","metadata":{"execution":{"iopub.status.busy":"2023-08-02T11:14:47.339752Z","iopub.execute_input":"2023-08-02T11:14:47.340343Z","iopub.status.idle":"2023-08-02T11:14:47.346293Z","shell.execute_reply.started":"2023-08-02T11:14:47.340310Z","shell.execute_reply":"2023-08-02T11:14:47.345260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#function used for encode mask into MS COCO RLE encoding\ndef encode_binary_mask(mask: np.ndarray) -> t.Text:\n \"\"\"Converts a binary mask into OID challenge encoding ascii text.\"\"\"\n\n # check input mask --\n if mask.dtype != np.bool:\n raise ValueError(\n \"encode_binary_mask expects a binary mask, received dtype == %s\" %\n mask.dtype)\n\n mask = np.squeeze(mask)\n if len(mask.shape) != 2:\n raise ValueError(\n \"encode_binary_mask expects a 2d mask, received shape == %s\" %\n mask.shape)\n\n # convert input mask to expected COCO API input --\n mask_to_encode = mask.reshape(mask.shape[0], mask.shape[1], 1)\n mask_to_encode = mask_to_encode.astype(np.uint8)\n mask_to_encode = np.asfortranarray(mask_to_encode)\n\n # RLE encode mask --\n encoded_mask = coco_mask.encode(mask_to_encode)[0][\"counts\"]\n\n # compress and base64 encoding --\n binary_str = zlib.compress(encoded_mask, zlib.Z_BEST_COMPRESSION)\n base64_str = base64.b64encode(binary_str)\n return base64_str","metadata":{"execution":{"iopub.status.busy":"2023-08-02T11:14:49.730461Z","iopub.execute_input":"2023-08-02T11:14:49.731243Z","iopub.status.idle":"2023-08-02T11:14:49.740050Z","shell.execute_reply.started":"2023-08-02T11:14:49.731202Z","shell.execute_reply":"2023-08-02T11:14:49.739170Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#load json data and convert it into a list of dicts\n#dicts contain IDs, class name and polygon annotation of segmentation masks\n\n# with open('/kaggle/input/hubmap-hacking-the-human-vasculature/polygons.jsonl','r') as json_file:\n# json_list=list(json_file)\n \n# tiles_dicts=[]\n# for json_str in json_list:\n# tiles_dicts.append(json.loads(json_str))","metadata":{"execution":{"iopub.status.busy":"2023-08-01T11:59:16.140589Z","iopub.execute_input":"2023-08-01T11:59:16.142895Z","iopub.status.idle":"2023-08-01T11:59:16.147241Z","shell.execute_reply.started":"2023-08-01T11:59:16.142847Z","shell.execute_reply":"2023-08-01T11:59:16.145991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#a dict contains multiple dicts\n# dict_of_tiles={}\n# for tile in tiles_dicts:\n# dict_of_tiles[tile['id']]=tile['annotations']","metadata":{"execution":{"iopub.status.busy":"2023-08-01T11:59:16.722904Z","iopub.execute_input":"2023-08-01T11:59:16.723598Z","iopub.status.idle":"2023-08-01T11:59:16.728449Z","shell.execute_reply.started":"2023-08-01T11:59:16.723542Z","shell.execute_reply":"2023-08-01T11:59:16.727408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# len(dict_of_tiles)","metadata":{"execution":{"iopub.status.busy":"2023-08-01T11:59:17.265374Z","iopub.execute_input":"2023-08-01T11:59:17.265765Z","iopub.status.idle":"2023-08-01T11:59:17.271863Z","shell.execute_reply.started":"2023-08-01T11:59:17.265734Z","shell.execute_reply":"2023-08-01T11:59:17.270677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#post_processing:dilation\ndef dilate_prediction(out_mask):\n for i in range(len(out_mask)):\n kernel=np.ones(shape=(3,3), dtype=np.uint8)\n out_mask[i]=cv2.dilate(out_mask[i],kernel,3)\n \n return out_mask\n","metadata":{"execution":{"iopub.status.busy":"2023-08-02T11:14:53.433859Z","iopub.execute_input":"2023-08-02T11:14:53.434314Z","iopub.status.idle":"2023-08-02T11:14:53.440568Z","shell.execute_reply.started":"2023-08-02T11:14:53.434277Z","shell.execute_reply":"2023-08-02T11:14:53.439638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#post_processing:erode\ndef erode_prediction(out_mask):\n for i in range(len(out_mask)):\n kernel=np.ones(shape=(3,3), dtype=np.uint8)\n out_mask[i]=cv2.erode(out_mask[i],kernel,3)\n \n return out_mask","metadata":{"execution":{"iopub.status.busy":"2023-08-02T11:14:54.325794Z","iopub.execute_input":"2023-08-02T11:14:54.326178Z","iopub.status.idle":"2023-08-02T11:14:54.332050Z","shell.execute_reply.started":"2023-08-02T11:14:54.326147Z","shell.execute_reply":"2023-08-02T11:14:54.331046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##post_processing:imopen(erode followed by dilation)\ndef imopen_prediction(out_mask):\n for i in range(len(out_mask)):\n kernel=np.ones(shape=(3,3), dtype=np.uint8)\n out_mask[i] = cv2.morphologyEx(out_mask[i], cv2.MORPH_OPEN, kernel)\n \n return out_mask","metadata":{"execution":{"iopub.status.busy":"2023-08-02T11:14:55.077942Z","iopub.execute_input":"2023-08-02T11:14:55.078857Z","iopub.status.idle":"2023-08-02T11:14:55.085161Z","shell.execute_reply.started":"2023-08-02T11:14:55.078813Z","shell.execute_reply":"2023-08-02T11:14:55.084202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##post_processing:imclose(erode followed by dilation)\ndef imclose_prediction(out_mask):\n for i in range(len(out_mask)):\n kernel=np.ones(shape=(3,3), dtype=np.uint8)\n out_mask[i] = cv2.morphologyEx(out_mask[i], cv2.MORPH_CLOSE, kernel)\n return out_mask","metadata":{"execution":{"iopub.status.busy":"2023-08-02T11:14:56.234010Z","iopub.execute_input":"2023-08-02T11:14:56.234366Z","iopub.status.idle":"2023-08-02T11:14:56.240207Z","shell.execute_reply.started":"2023-08-02T11:14:56.234339Z","shell.execute_reply":"2023-08-02T11:14:56.239135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#prediction\ndef prediction(model,image_file, device):\n img0=cv2.imread(image_file) #read image file as BGR format\n \n #resize and pad image while meeting stride-multiple constraints\n im=img0.transpose((2,0,1))[::-1]#HWC to CHW, BGR to RGB\n im=np.ascontiguousarray(im) #contiguous\n \n im=torch.from_numpy(im).to(device)#put image array on cuda\n im=im.float()\n im/=255 #[0,255] to [0,1.0]\n im=im[None] #expand for batch\n \n #inference\n pred,out=model(im, augment=False, visualize=False)#make prediction\n #print(pred.type()+'\\n')\n #print(out.type())\n proto=out[1]#prototype\n # print(pred.shape)：torch.Size([1, 16128, 40])--> 16128=1x3x64x64+1x3x32x32+1x3x16x16\n #print(proto.shape):torch.Size([1, 32, 128, 128]) CHW\n #proto[i].shape:torch.Size([32, 128, 128]) 32prototypes for each image\n \n #NMS\n pred_nms=non_max_suppression(pred,conf_thres=0.001, iou_thres=0.6, classes=0, agnostic=False, max_det=1000,nm=32)#nm:number of masks\n #print(len(pred_nms)) =number of testset\n #since only 1 image now, only 1 detection, and the detection size is torch.Size([315, 38])\n \n out_mask=[]\n out_conf=[]\n for i, det in enumerate(pred_nms): #per image\n if len(det):\n #print(i,proto[i].shape):torch.Size([32, 128, 128])\n #print(im.shape[2:]):torch.Size([512, 512])\n masks=process_mask(proto[i],det[:,6:],det[:,:4],im.shape[2:], upsample=True)#HWC, this step is for produce the masks for each instance\n confs=det[:,4]\n clasf=det[:,5]\n \n out_conf=confs.data.cpu().numpy()\n \n for mask, confidence, classfication in zip(masks, confs, clasf):\n binary_mask=mask.cpu().numpy()\n out_mask.append(binary_mask)\n \n return pred,out,pred_nms,out_mask,out_conf\n \n ","metadata":{"execution":{"iopub.status.busy":"2023-08-02T11:14:57.578323Z","iopub.execute_input":"2023-08-02T11:14:57.578688Z","iopub.status.idle":"2023-08-02T11:14:57.592352Z","shell.execute_reply.started":"2023-08-02T11:14:57.578659Z","shell.execute_reply":"2023-08-02T11:14:57.590845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#load pretrained model\n#best_model='/kaggle/input/hubmap-2023-yolov7-baseline/yolov7-fine-tune_EX3_trainv9/trainv11_2class_with_unsure/weights/best.pt'\nbest_model='/kaggle/input/pre-trained-weights/training_v4_best.pt'\ndata_yaml='/kaggle/working/hubmap-coco.yaml'\n\n#test_dir='/kaggle/input/hubmap-2023-yolov7-baseline/COCO/test/'\ntest_dir='/kaggle/input/hubmap-hacking-the-human-vasculature/test/'\nimage_id = glob(f'{test_dir}/*.tif')\nimage_id = [f.split('/')[-1][:-4] for f in image_id]\nprint(len(image_id))\n\ndevice = select_device('0') \nmodel = DetectMultiBackend(best_model, device=device, dnn=False, data=data_yaml, fp16=False)\nmodel.warmup(imgsz=(1,3,512,512)) \nprint('model.training', model.model.training)","metadata":{"execution":{"iopub.status.busy":"2023-08-02T11:49:14.907799Z","iopub.execute_input":"2023-08-02T11:49:14.908208Z","iopub.status.idle":"2023-08-02T11:49:16.543199Z","shell.execute_reply.started":"2023-08-02T11:49:14.908177Z","shell.execute_reply":"2023-08-02T11:49:16.542157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission=[]\n\nfor t, id in enumerate(image_id):\n print(t,id)\n prediction_string=''\n \n image_file=test_dir+f'{id}.tif'\n \n pred, out,pred_nms,out_mask,out_conf=prediction(model,image_file,device)\n \n #dilation\n #out_mask=dilate_prediction(out_mask)\n \n #erode\n #out_mask=erode_prediction(out_mask)\n \n #imopen\n #out_mask=imopen_prediction(out_mask)\n \n #imclose\n out_mask=imclose_prediction(out_mask)\n \n all=np.zeros((512,512), dtype=np.uint8)\n if len(out_mask)>0:\n num_mask=len(out_mask)\n for i in range(num_mask):\n m=out_mask[i]>0 #out_mask[i]>0 is the position with the polygons, every m is 512x512\n #print(m.shape)\n all[m]=1\n \n e=encode_binary_mask(m)\n if i==0:\n prediction_string=f'0 {out_conf[i]} {e.decode(\"utf-8\")}'\n else:\n prediction_string+=f' 0 {out_conf[i]} {e.decode(\"utf-8\")}'\n \n if t==0:\n print(\"prediction string\")\n print(prediction_string[:200])\n \n image=cv2.imread(image_file)\n out_mask=all/(all.max()+0.001)\n plt.imshow(image)\n plt.imshow(out_mask)\n plt.show()\n \n submission.append({\n 'id':id,\n 'height':512,\n 'width':512,\n 'prediction_string':prediction_string,\n }) \n \n\nsubmission_df = pd.DataFrame(submission)\nsubmission_df.to_csv('submission.csv',index=False)\nprint(submission_df)\n","metadata":{"execution":{"iopub.status.busy":"2023-08-02T11:52:54.799065Z","iopub.execute_input":"2023-08-02T11:52:54.799446Z","iopub.status.idle":"2023-08-02T11:52:56.011804Z","shell.execute_reply.started":"2023-08-02T11:52:54.799419Z","shell.execute_reply":"2023-08-02T11:52:56.010829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# pred.shape[2]-32-5#num_class=3","metadata":{"execution":{"iopub.status.busy":"2023-07-21T15:39:08.702640Z","iopub.execute_input":"2023-07-21T15:39:08.703680Z","iopub.status.idle":"2023-07-21T15:39:08.711194Z","shell.execute_reply.started":"2023-07-21T15:39:08.703632Z","shell.execute_reply":"2023-07-21T15:39:08.710033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# proto=out[1]\n# proto[0].shape","metadata":{"execution":{"iopub.status.busy":"2023-07-21T15:39:09.998255Z","iopub.execute_input":"2023-07-21T15:39:09.998649Z","iopub.status.idle":"2023-07-21T15:39:10.005880Z","shell.execute_reply.started":"2023-07-21T15:39:09.998616Z","shell.execute_reply":"2023-07-21T15:39:10.004868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# for i, det in enumerate(pred_nms): \n# print(i,det.shape)#ttorch.Size([315, 38])","metadata":{"execution":{"iopub.status.busy":"2023-07-21T15:39:10.743281Z","iopub.execute_input":"2023-07-21T15:39:10.744533Z","iopub.status.idle":"2023-07-21T15:39:10.750688Z","shell.execute_reply.started":"2023-07-21T15:39:10.744493Z","shell.execute_reply":"2023-07-21T15:39:10.749549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fig = plt.figure(figsize=(25,10))\n# for i in range(32):\n# pro=proto[:,i,:,:].squeeze(0).cpu().detach().numpy()\n# #print(i)\n# plt.subplot(4,8,i+1)\n# plt.imshow(pro)","metadata":{"execution":{"iopub.status.busy":"2023-07-21T15:39:53.012604Z","iopub.execute_input":"2023-07-21T15:39:53.012983Z","iopub.status.idle":"2023-07-21T15:40:00.821265Z","shell.execute_reply.started":"2023-07-21T15:39:53.012951Z","shell.execute_reply":"2023-07-21T15:40:00.820196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}