Double U-Net consists of a combination of two intertwined U-Nets. Pre-trained VGG-19 is used as the encoder sub-network in the first U-Net which is combined with a custom decoder block. The second U-Net has custom encoder and decoder sub-networks to capture more semantic information efficiently. Atrous Spatial Pyramid Pooling (ASPP) is employed to capture contextual information within the network. Several Skip connections are also employed between the two U-Nets. Output concatenations and multiplications are performed in several stages. Squeeze and excite blocks are used to reduce redundant information and improve contextual information capture.
\n\nThis variation of U-net is based on the paper, [A Deep Convolutional Neural Network for Medical Image Segmentation paper by Debesh Jha et al](https://arxiv.org/pdf/2006.04868.pdf)\n\nI have added several intuitive & operational optimizations including the conversion of Upscaling layers to Convolutional 2D Tranpose layers so that TPU for Keras is supported, also allowing upscaling filers to learn feature groups. Additionally, Convlutional Layers have been added to the output to combine the output feature maps from the two U-Nets.
\n\nDetailed implemenation details are present in the model building section.\n\nSuperior performance is exibited by this network in several medical segmentation datasets, covering various imaging modalities such as colonoscopy, dermoscopy & and microscopy. Experiments on the 2015 MICCAI sub-challenge on automatic polyp detection dataset, the CVC-ClinicDB, the 2018 Data Science Bowl challenge, and the Lesion boundary segmen-tation datasets demonstrate that the DoubleU-Net outperforms U-Net and the baseline models.
\n\nI have created an Augmented Image Dataset which consists of all 512x512 tiles with gloms and Tfrecords with original and augmented images split into validation and train sets. These have been uploaded and are available as public datasets.
\n\n**Augmented images 512x512 tiled. Augmentations done only for images with glomerules. Validation set was not augmented to avoid leakage.**\n\nhttps://www.kaggle.com/sreevishnudamodaran/hubmap-512x512-augmented\n\n\n**TFRecord dataset with actual and augmented images grouped into 90-130MB records.**\n\nhttps://www.kaggle.com/sreevishnudamodaran/hubmap-512x512-tfrecords-with-aug\n\n\n\n\n\n\n[](https://www.kaggle.com/sreevishnudamodaran)\n\n\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"## Import Libraries","metadata":{}},{"cell_type":"code","source":"%matplotlib inline\n\nimport json\nimport os\nimport glob\nimport re\nimport datetime\nimport os.path as osp\nfrom path import Path\nimport collections\nimport sys\nimport uuid\nimport random\nimport warnings\nfrom itertools import chain\nfrom tqdm import tqdm\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom matplotlib import rcParams\nsns.set(rc={\"font.size\":9,\"axes.titlesize\":15,\"axes.labelsize\":9,\n \"axes.titlepad\":2, \"axes.labelpad\":9, \"legend.fontsize\":7,\n \"legend.title_fontsize\":7, 'axes.grid' : False,\n 'figure.titlesize':35})\n\n# from skimage import measure\nfrom PIL import Image\nimport cv2\n# from skimage.io import imread, imshow, imread_collection, concatenate_images\n# from skimage.transform import resize\n# from skimage.morphology import label\n\nimport tensorflow as tf\nfrom tensorflow.keras.models import Model, load_model\nfrom tensorflow.keras.layers import Input\nfrom tensorflow.keras.layers import Conv2D, Conv2DTranspose\nfrom tensorflow.keras.layers import MaxPooling2D, UpSampling2D, Conv2DTranspose\nfrom tensorflow.keras.layers import concatenate\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\nfrom tensorflow.keras import backend as K\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.optimizers import Adam\n\nfrom keras.engine.topology import Layer\nfrom keras.utils.generic_utils import get_custom_objects\nfrom kaggle_datasets import KaggleDatasets\nfrom kaggle_secrets import UserSecretsClient\n\nfrom sklearn.model_selection import train_test_split\nfrom tensorflow.keras.losses import binary_crossentropy\nimport plotly\nimport plotly.graph_objs as go\nimport numpy as np # So we can use random numbers in examples\n","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Read Images and Masks\n\nUsing the [HuBMAP: 512x512 full size tiles](https://www.kaggle.com/xhlulu/hubmap-512x512-full-size-tiles) by xhlulu as the base before augmentation","metadata":{}},{"cell_type":"code","source":"# image_paths = glob.glob(\"/kaggle/input/hubmap-1024x1024/train/*.png\")\n# mask_paths = glob.glob(\"/kaggle/input/hubmap-1024x1024/masks/*.png\")\n# len(image_paths)\n\nimage_paths = glob.glob(\"/kaggle/input/hubmap-512x512-full-size-tiles/train/*.png\")\nmask_paths = glob.glob(\"/kaggle/input/hubmap-512x512-full-size-tiles/masks/*.png\")\nlen(image_paths)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Doing augmentation in parts due to harddisk limitations and the large dataset size\n\nimage_paths = image_paths[:11607]\nmask_paths = mask_paths[:11607]\nlen(image_paths)\n# image_paths = image_paths[11608:]\n# mask_paths = mask_paths[11608:]\n# len(image_paths)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Helper Functions","metadata":{}},{"cell_type":"code","source":"def read_single(img_path, msk_path):\n \"\"\" Read the image and mask from the given path. \"\"\"\n image = cv2.imread(img_path, cv2.IMREAD_COLOR)\n mask = cv2.imread(msk_path, cv2.IMREAD_GRAYSCALE)\n return image, mask\n\ndef read_data(image_paths, mask_paths, gloms_only=False):\n images = []\n masks = []\n\n for img_path, msk_path in tqdm(zip(image_paths, mask_paths), total=len(image_paths)):\n\n image, mask = read_single(img_path, msk_path)\n mask_density = np.count_nonzero(mask) \n if gloms_only:\n if(mask_density>0):\n images.append(image)\n masks.append(mask)\n else:\n images.append(image)\n masks.append(mask)\n\n images = np.array(images)\n masks = np.array(masks)\n print('images shape:', images.shape)\n print('masks shape:', masks.shape)\n return images, masks","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from google.cloud import storage\nstorage_client = storage.Client(project='placesproject-284409')\n\ndef create_bucket(dataset_name):\n \"\"\"Creates a new bucket. https://cloud.google.com/storage/docs/ \"\"\"\n bucket = storage_client.create_bucket(dataset_name)\n print('Bucket {} created'.format(bucket.name))\n\ndef upload_blob(bucket_name, source_file_name, destination_blob_name):\n \"\"\"Uploads a file to the bucket. https://cloud.google.com/storage/docs/ \"\"\"\n bucket = storage_client.get_bucket(bucket_name)\n blob = bucket.blob(destination_blob_name)\n blob.upload_from_filename(source_file_name)\n# print('File {} uploaded to {}.'.format(\n# source_file_name,\n# destination_blob_name))\n \ndef list_blobs(bucket_name):\n \"\"\"Lists all the blobs in the bucket. https://cloud.google.com/storage/docs/\"\"\"\n blob_list = []\n blobs = storage_client.list_blobs(bucket_name)\n for blob in blobs:\n blob_list.append(blob.name)\n #print(blob_list)\n return blob_list\n \ndef download_to_kaggle(bucket_name,destination_directory,file_name):\n \"\"\"Takes the data from your GCS Bucket and puts it into the working directory of your Kaggle notebook\"\"\"\n os.makedirs(destination_directory, exist_ok = True)\n full_file_path = os.path.join(destination_directory, file_name)\n blobs = storage_client.list_blobs(bucket_name)\n for blob in blobs:\n blob.download_to_filename(full_file_path)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Set your own project id here\nPROJECT_ID = 'placesproject-284409'\nfrom google.cloud import storage\nstorage_client = storage.Client(project=PROJECT_ID)\nuser_secrets = UserSecretsClient()\nuser_credential = user_secrets.get_gcloud_credential()\nuser_secrets.set_tensorflow_credential(user_credential)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"lowband_density_values = []\nmask_density_values = []\n\nfor img_path, msk_path in tqdm(zip(image_paths, mask_paths), total=len(image_paths)):\n image, mask = read_single(img_path, msk_path)\n img_hist = np.histogram(image)\n #print(\"img_hist\", img_hist)\n lowband_density = np.sum(img_hist[0][0:4])\n mask_density = np.count_nonzero(mask)\n #print(\"lowband_density\", lowband_density)\n #print(\"highband_density\", highband_density)\n #print(\"mask_density\", mask_density)\n lowband_density_values.append(lowband_density)\n mask_density_values.append(mask_density)\ntrain_helper_df = pd.DataFrame(data=list(zip(image_paths, mask_paths, lowband_density_values,\n mask_density_values)),\n columns=['image_path','mask_path', 'lowband_density', 'mask_density'])\ntrain_helper_df.astype(dtype={'image_path':'object','mask_path':'object',\n 'lowband_density':'int64', 'mask_density':'int64'})","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# bucket_name = 'hubmap_512x512_with_aug'\n# # train_helper_df.to_csv('./hubmap_dataset_helper.csv')\n# # upload_blob(bucket_name, './hubmap_dataset_helper.csv', 'hubmap_dataset_helper.csv')\n\n# download_to_kaggle(bucket_name, '/kaggle/working', 'hubmap_dataset_helper.csv')\n# train_helper_df = pd.read_csv('hubmap_dataset_helper.csv')","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_helper_df.sample(5)","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Selecting Images with Tissues","metadata":{}},{"cell_type":"code","source":"images_tissue = train_helper_df[train_helper_df.lowband_density>100].image_path\nmasks_tissue = train_helper_df[train_helper_df.lowband_density>100].mask_path\nimages_tissue.shape","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Visualize Samples","metadata":{}},{"cell_type":"code","source":"images, masks = read_data(images_tissue[1200:1218], masks_tissue[1200:1218])","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 6\nmax_cols = 6\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,18))\nfig.suptitle('Sample Images', y=0.93)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(images[:plot_count], masks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n #sns.distplot(img_array.flatten(), ax=ax[1]);\n ax[row_masks, col].imshow(mas)\n","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Data Preparation\n## Augmentation\n\nAugmentation is done only on images with gloms as the dataset is already huge.\n\nValidation samples are split and kept aside and it is not used for augmentation to avoid leakage of train data to val data","metadata":{}},{"cell_type":"code","source":"image_90_per_tissues, image_val_files, mask_90_per_tissues, mask_val_files = train_test_split(images_tissue, masks_tissue, test_size=0.30, random_state=17)\nprint(\"Split Counts\\n\\tImage_90_per_files:\\t{0}\\n\\tMask_90_per_files:\\t{2}\\n\\tVal Images:\\t\\t{1}\\n\\tVal Masks:\\t\\t{3}\\n\"\n .format(len(image_90_per_tissues), len(image_val_files), len(mask_90_per_tissues), len(mask_val_files)))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from albumentations import (\nCLAHE,\nElasticTransform,\nGridDistortion,\nOpticalDistortion,\nHorizontalFlip,\nRandomBrightnessContrast,\nRandomGamma,\nHueSaturationValue,\nRGBShift,\nMedianBlur,\nGaussianBlur,\nGaussNoise,\nChannelShuffle,\nCoarseDropout\n)\n\ndef augment_data(image_paths, mask_paths): \n\n if not os.path.exists('hubmap_512x512_augmented/images_aug2'):\n os.makedirs('hubmap_512x512_augmented/images_aug2')\n if not os.path.exists('hubmap_512x512_augmented/masks_aug2'):\n os.makedirs('hubmap_512x512_augmented/masks_aug2')\n\n for image, mask in tqdm(zip(image_paths, mask_paths), total=len(image_paths)):\n images_aug = []\n masks_aug = []\n image_name = Path(image).stem\n mask_name = Path(mask).stem\n\n x, y = read_single(image, mask)\n mask_density = np.count_nonzero(y)\n\n ## Augmenting only images with Gloms\n if(mask_density>0):\n\n try:\n h, w, c = x.shape\n except Exception as e:\n image = image[:-1]\n x, y = read_single(image, mask)\n h, w, c = x.shape\n\n aug = CLAHE(clip_limit=1.0, tile_grid_size=(8, 8), always_apply=False, p=1)\n augmented = aug(image=x, mask=y)\n x0 = augmented['image']\n y0 = augmented['mask']\n\n ## ElasticTransform\n aug = ElasticTransform(p=1, alpha=120, sigma=512*0.05, alpha_affine=512*0.03)\n augmented = aug(image=x, mask=y)\n x1 = augmented['image']\n y1 = augmented['mask']\n\n ## Grid Distortion\n aug = GridDistortion(p=1)\n augmented = aug(image=x, mask=y)\n x2 = augmented['image']\n y2 = augmented['mask']\n\n ## Optical Distortion\n aug = OpticalDistortion(p=1, distort_limit=2, shift_limit=0.5)\n augmented = aug(image=x, mask=y)\n x3 = augmented['image']\n y3 = augmented['mask']\n\n ## Horizontal Flip\n aug = HorizontalFlip(p=1)\n augmented = aug(image=x, mask=y)\n x4 = augmented['image']\n y4 = augmented['mask']\n\n ## Random Brightness and Contrast\n aug = RandomBrightnessContrast(p=1)\n augmented = aug(image=x, mask=y)\n x5 = augmented['image']\n y5 = augmented['mask']\n\n aug = RandomGamma(p=1)\n augmented = aug(image=x, mask=y)\n x6 = augmented['image']\n y6 = augmented['mask']\n\n aug = HueSaturationValue(p=1)\n augmented = aug(image=x, mask=y)\n x7 = augmented['image']\n y7 = augmented['mask']\n\n aug = RGBShift(p=1)\n augmented = aug(image=x, mask=y)\n x8 = augmented['image']\n y8 = augmented['mask']\n\n aug = MedianBlur(p=1, blur_limit=5)\n augmented = aug(image=x, mask=y)\n x9 = augmented['image']\n y9 = augmented['mask']\n\n aug = GaussianBlur(p=1, blur_limit=3)\n augmented = aug(image=x, mask=y)\n x10 = augmented['image']\n y10 = augmented['mask']\n\n aug = GaussNoise(p=1)\n augmented = aug(image=x, mask=y)\n x11 = augmented['image']\n y11 = augmented['mask']\n\n aug = ChannelShuffle(p=1)\n augmented = aug(image=x, mask=y)\n x12 = augmented['image']\n y12 = augmented['mask']\n\n aug = CoarseDropout(p=1, max_holes=8, max_height=32, max_width=32)\n augmented = aug(image=x, mask=y)\n x13 = augmented['image']\n y13 = augmented['mask']\n\n images_aug.extend([\n x0, x1, x2, x3, x4, x5, x6,\n x7, x8, x9, x10, x11, x12,\n x13])\n\n masks_aug.extend([\n y0, y1, y2, y3, y4, y5, y6,\n y7, y8, y9, y10, y11, y12,\n y13])\n\n idx = 0\n for i, m in zip(images_aug, masks_aug):\n tmp_image_name = f\"{image_name}_{idx}.png\"\n tmp_mask_name = f\"{mask_name}_{idx}.png\"\n\n image_path = os.path.join(\"hubmap_512x512_augmented/images_aug2/\", tmp_image_name)\n mask_path = os.path.join(\"hubmap_512x512_augmented/masks_aug2/\", tmp_mask_name)\n\n cv2.imwrite(image_path, i)\n cv2.imwrite(mask_path, m)\n\n idx += 1\n\n return images_aug, masks_aug\n\nimages_aug, masks_aug = augment_data(image_90_per_tissues, mask_90_per_tissues)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Load Augmented Dataset\nLoading the augmented images from the registered dataset.","metadata":{}},{"cell_type":"code","source":"aug_img_paths = glob.glob(\"/kaggle/input/hubmap-512x512-augmented/images_aug/*.png\")\naug_msk_paths = glob.glob(\"/kaggle/input/hubmap-512x512-augmented/masks_aug/*.png\")\naug_img_paths2 = glob.glob(\"/kaggle/input/hubmap-512x512-augmented/images_aug2/*.png\")\naug_msk_paths2 = glob.glob(\"/kaggle/input/hubmap-512x512-augmented/masks_aug2/*.png\")\n\naug_img_paths.extend(aug_img_paths2)\naug_msk_paths.extend(aug_msk_paths2)\nprint(\"Number of Augmented Images\", len(aug_img_paths))\nprint(\"Number of Augmented Masks\", len(aug_msk_paths))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Visualize Augmented","metadata":{}},{"cell_type":"code","source":"aug_img_paths = aug_img_paths[-100:]\naug_msk_paths = aug_msk_paths[-100:]\naug_imgs, aug_msks = read_data(aug_img_paths, aug_msk_paths)","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 10\nmax_cols = 4\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,32))\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sel_img_paths = [img_path for img_path in aug_img_paths if '_0.png' in img_path]\nsel_msk_paths = [msk_path for msk_path in aug_msk_paths if '_0.png' in msk_path]\naug_imgs, aug_msks = read_data(sel_img_paths, sel_msk_paths)","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 2\nmax_cols = 2\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,9))\nfig.suptitle(\"CLAHE\", y=0.95)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sel_img_paths = [img_path for img_path in aug_img_paths if '_1.png' in img_path]\nsel_msk_paths = [msk_path for msk_path in aug_msk_paths if '_1.png' in msk_path]\naug_imgs, aug_msks = read_data(sel_img_paths, sel_msk_paths)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 2\nmax_cols = 2\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,9))\nfig.suptitle(\"ElasticTransform\", y=0.95)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sel_img_paths = [img_path for img_path in aug_img_paths if '_2.png' in img_path]\nsel_msk_paths = [msk_path for msk_path in aug_msk_paths if '_2.png' in msk_path]\naug_imgs, aug_msks = read_data(sel_img_paths, sel_msk_paths)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 2\nmax_cols = 2\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,9))\nfig.suptitle(\"GridDistortion\", y=0.95)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sel_img_paths = [img_path for img_path in aug_img_paths if '_3.png' in img_path]\nsel_msk_paths = [msk_path for msk_path in aug_msk_paths if '_3.png' in msk_path]\naug_imgs, aug_msks = read_data(sel_img_paths, sel_msk_paths)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 2\nmax_cols = 2\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,9))\nfig.suptitle(\"OpticalDistortion\", y=0.95)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sel_img_paths = [img_path for img_path in aug_img_paths if '_4.png' in img_path]\nsel_msk_paths = [msk_path for msk_path in aug_msk_paths if '_4.png' in msk_path]\naug_imgs, aug_msks = read_data(sel_img_paths, sel_msk_paths)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 2\nmax_cols = 2\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,9))\nfig.suptitle(\"HorizontalFlip\", y=0.95)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sel_img_paths = [img_path for img_path in aug_img_paths if '_5.png' in img_path]\nsel_msk_paths = [msk_path for msk_path in aug_msk_paths if '_5.png' in msk_path]\naug_imgs, aug_msks = read_data(sel_img_paths, sel_msk_paths)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 2\nmax_cols = 2\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,9))\nfig.suptitle(\"RandomBrightnessContrast\", y=0.95)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sel_img_paths = [img_path for img_path in aug_img_paths if '_6.png' in img_path]\nsel_msk_paths = [msk_path for msk_path in aug_msk_paths if '_6.png' in msk_path]\naug_imgs, aug_msks = read_data(sel_img_paths, sel_msk_paths)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 2\nmax_cols = 2\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,9))\nfig.suptitle(\"RandomGamma\", y=0.95)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sel_img_paths = [img_path for img_path in aug_img_paths if '_7.png' in img_path]\nsel_msk_paths = [msk_path for msk_path in aug_msk_paths if '_7.png' in msk_path]\naug_imgs, aug_msks = read_data(sel_img_paths, sel_msk_paths)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 2\nmax_cols = 2\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,9))\nfig.suptitle(\"HueSaturationValue\", y=0.95)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sel_img_paths = [img_path for img_path in aug_img_paths if '_8.png' in img_path]\nsel_msk_paths = [msk_path for msk_path in aug_msk_paths if '_8.png' in msk_path]\naug_imgs, aug_msks = read_data(sel_img_paths, sel_msk_paths)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 2\nmax_cols = 2\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,9))\nfig.suptitle(\"RGBShift\", y=0.95)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sel_img_paths = [img_path for img_path in aug_img_paths if '_9.png' in img_path]\nsel_msk_paths = [msk_path for msk_path in aug_msk_paths if '_9.png' in msk_path]\naug_imgs, aug_msks = read_data(sel_img_paths, sel_msk_paths)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 2\nmax_cols = 2\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,9))\nfig.suptitle(\"MedianBlur\", y=0.95)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sel_img_paths = [img_path for img_path in aug_img_paths if '_10.png' in img_path]\nsel_msk_paths = [msk_path for msk_path in aug_msk_paths if '_10.png' in msk_path]\naug_imgs, aug_msks = read_data(sel_img_paths, sel_msk_paths)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 2\nmax_cols = 2\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,9))\nfig.suptitle(\"GaussianBlur\", y=0.95)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sel_img_paths = [img_path for img_path in aug_img_paths if '_11.png' in img_path]\nsel_msk_paths = [msk_path for msk_path in aug_msk_paths if '_11.png' in msk_path]\naug_imgs, aug_msks = read_data(sel_img_paths, sel_msk_paths)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 2\nmax_cols = 2\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,9))\nfig.suptitle(\"GaussNoise\", y=0.95)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sel_img_paths = [img_path for img_path in aug_img_paths if '_12.png' in img_path]\nsel_msk_paths = [msk_path for msk_path in aug_msk_paths if '_12.png' in msk_path]\naug_imgs, aug_msks = read_data(sel_img_paths, sel_msk_paths)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 2\nmax_cols = 2\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,9))\nfig.suptitle(\"ChannelShuffle\", y=0.95)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sel_img_paths = [img_path for img_path in aug_img_paths if '_13.png' in img_path]\nsel_msk_paths = [msk_path for msk_path in aug_msk_paths if '_13.png' in msk_path]\naug_imgs, aug_msks = read_data(sel_img_paths, sel_msk_paths)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_rows = 2\nmax_cols = 2\nfig, ax = plt.subplots(max_rows, max_cols, figsize=(20,9))\nfig.suptitle(\"CoarseDropout\", y=0.95)\nplot_count = (max_rows*max_cols)//2\nfor idx, (img, mas) in enumerate(zip(aug_imgs[:plot_count], aug_msks[:plot_count])):\n row = (idx//max_cols)*2\n row_masks = row+1\n col = idx % max_cols\n ax[row, col].imshow(img)\n ax[row_masks, col].imshow(mas)","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n## Merge Augmented Data","metadata":{}},{"cell_type":"code","source":"images_tissue = train_helper_df[train_helper_df.lowband_density>100].image_path\nmasks_tissue = train_helper_df[train_helper_df.lowband_density>100].mask_path\n\nprint(\"Number of images with tissue:\", images_tissue.shape)\nprint(\"Number of masks with tissue:\", masks_tissue.shape)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"aug_img_paths = glob.glob(\"/kaggle/input/hubmap-512x512-augmented/images_aug/*.png\")\naug_msk_paths = glob.glob(\"/kaggle/input/hubmap-512x512-augmented/masks_aug/*.png\")\naug_img_paths2 = glob.glob(\"/kaggle/input/hubmap-512x512-augmented/images_aug2/*.png\")\naug_msk_paths2 = glob.glob(\"/kaggle/input/hubmap-512x512-augmented/masks_aug2/*.png\")\n\naug_img_paths.extend(aug_img_paths2)\naug_msk_paths.extend(aug_msk_paths2)\n\nprint(\"Number of Augmented Images\", len(aug_img_paths))\nprint(\"Number of Augmented Masks\", len(aug_msk_paths))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"val_files = pd.read_csv('../input/hubmap-512x512-augmented/validation_files.csv')\nval_files.sample(3)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"image_val_files = val_files['image_val_files'].tolist()\nmask_val_files = val_files['mask_val_files'].tolist()\nprint(\"Total Val Image Count:\", len(image_val_files))\nprint(\"Total Val Mask Count:\", len(mask_val_files))\nimage_val_files[:2]","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Note: Including Augmented Images only in the Train Set and not in the Validation Set to prevent leakage of validation data into training data","metadata":{}},{"cell_type":"code","source":"# image_90_per_tissues, image_val_files, mask_90_per_tissues, mask_val_files = train_test_split(images_tissue, masks_tissue, test_size=0.35, random_state=13)\n# print(\"Split Counts\\n\\tImage_90_per_files:\\t{0}\\n\\tMask_90_per_files:\\t{2}\\n\\tVal Images:\\t\\t{1}\\n\\tVal Masks:\\t\\t{3}\\n\"\n# .format(len(image_90_per_tissues), len(image_val_files), len(mask_90_per_tissues), len(mask_val_files)))\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"image_90_per_tissues = image_90_per_tissues.tolist()\nimage_90_per_tissues.extend(aug_img_paths)\nmask_90_per_tissues = mask_90_per_tissues.tolist()\nmask_90_per_tissues.extend(aug_msk_paths)\nprint(\"Total Train Image Count:\", len(image_90_per_tissues))\nprint(\"Total Train Mask Count:\", len(mask_90_per_tissues))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"image_45_per_files1, image_45_per_files2, mask_45_per_files1, mask_45_per_files2 = train_test_split(image_90_per_tissues, mask_90_per_tissues, test_size=0.5, random_state=13)\nprint(\"Split Counts\\n\\timage_45_per_files1:\\t{0}\\n\\tmask_45_per_files1:\\t{2}\\n\\timage_45_per_files2:\\t{1}\\n\\tmask_45_per_files2:\\t{3}\\n\"\n .format(len(image_45_per_files1), len(image_45_per_files2), len(mask_45_per_files1), len(mask_45_per_files2)))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Build TfRecords","metadata":{}},{"cell_type":"code","source":"def _bytes_feature(value):\n \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n if isinstance(value, type(tf.constant(0))):\n value = value.numpy() # BytesList won't unpack a string from an EagerTensor.\n return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n\ndef _float_feature(value):\n \"\"\"Returns a float_list from a float / double.\"\"\"\n return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))\n\ndef _int64_feature(value):\n \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def image_example(image, mask):\n image_shape = image.shape \n img_bytes = image.tostring()\n mask_bytes = mask.tostring()\n feature = {\n 'height': _int64_feature(image_shape[0]),\n 'width': _int64_feature(image_shape[1]),\n 'num_channels': _int64_feature(image_shape[2]),\n 'img_bytes': _bytes_feature(img_bytes),\n 'mask' : _bytes_feature(mask_bytes),\n }\n return tf.train.Example(features=tf.train.Features(feature=feature))\n\ndef create_tfrecord(image, mask, output_path):\n opts = tf.io.TFRecordOptions(compression_type=\"GZIP\")\n with tf.io.TFRecordWriter(output_path, opts) as writer:\n tf_example = image_example(image, mask)\n writer.write(tf_example.SerializeToString())\n writer.close()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def write_dataset(img_files, msk_files, records_per_part, prefix):\n opts = tf.io.TFRecordOptions(compression_type=\"GZIP\")\n part_num = 0\n num_records = 0\n output_path = prefix+'_part{}.tfrecords'.format(part_num)\n writer = tf.io.TFRecordWriter(output_path, opts)\n \n for img_file, msk_file in tqdm(zip(img_files, msk_files), total=len(img_files), position=0, leave=True):\n image, mask = read_single(img_file, msk_file)\n assert image.shape == (512, 512, 3), print(\"Wrong image.shape\", image.shape)\n assert mask.shape == (512, 512), print(\"mask.shape\", mask.shape)\n #print(\"image.shape\", image.shape)\n mask = np.expand_dims(mask, axis=-1)\n tf_example = image_example(image, mask)\n writer.write(tf_example.SerializeToString())\n num_records += 1 \n if(num_records == records_per_part - 1):\n # close current file and open new one\n print(\"wrote part #{}\".format(part_num))\n writer.close()\n part_num += 1\n output_path = prefix+'_part{}.tfrecords'.format(part_num)\n writer = tf.io.TFRecordWriter(output_path, opts)\n num_records = 0\n writer.close()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"bucket_name = 'hubmap_train_job_2' \ntry:\n create_bucket(bucket_name) \nexcept:\n pass","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if not os.path.exists('train_tfrecords'):\n os.makedirs('train_tfrecords')\n\nprint(\"Writing Train Dataset\")\nwrite_dataset(image_45_per_files1, mask_45_per_files1, 256, '/kaggle/working/train_tfrecords/train1')","metadata":{"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!ls -lah ./train_tfrecords | wc -l","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"files = glob.glob(\"/kaggle/working/train_tfrecords/*.tfrecords\")\nfor file in files:\n file_name = os.path.join('train', os.path.basename(Path(file)))\n print(file_name)\n upload_blob(bucket_name, file, file_name)","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!rm -R ./train_tfrecords","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if not os.path.exists('train_tfrecords'):\n os.makedirs('train_tfrecords')\n \nwrite_dataset(image_45_per_files2, mask_45_per_files2, 256, '/kaggle/working/train_tfrecords/train2')","metadata":{"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!ls -lah ./train_tfrecords","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"files = glob.glob(\"/kaggle/working/train_tfrecords/*.tfrecords\")\nfor file in files:\n file_name = os.path.join('train', os.path.basename(Path(file)))\n print(file_name)\n upload_blob(bucket_name, file, file_name)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!rm -R ./train_tfrecords","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if not os.path.exists('val_tfrecords'):\n os.makedirs('val_tfrecords')\n \nprint(\"Writing Validation Dataset\")\nwrite_dataset(image_val_files, mask_val_files, 256, '/kaggle/working/val_tfrecords/val')","metadata":{"_kg_hide-output":true,"_kg_hide-input":false,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!ls -lah ./val_tfrecords | wc -l","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"files = glob.glob(\"/kaggle/working/val_tfrecords/*.tfrecords\")\nfor file in files:\n file_name = os.path.join('val', os.path.basename(Path(file)))\n print(file_name)\n upload_blob(bucket_name, file, file_name)","metadata":{"_kg_hide-input":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!rm -R ./val_tfrecords","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Intialize and Get TPU Ready","metadata":{}},{"cell_type":"code","source":"ACCELERATOR_TYPE = 'TPU'\n\nif ACCELERATOR_TYPE == 'TPU':\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n strategy = tf.distribute.MirroredStrategy()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"user_secrets = UserSecretsClient()\nuser_credential = user_secrets.get_gcloud_credential()\nuser_secrets.set_tensorflow_credential(user_credential)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## View TFRecord Samples","metadata":{}},{"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n\nimage_feature_description = {\n 'height': tf.io.FixedLenFeature([], tf.int64),\n 'width': tf.io.FixedLenFeature([], tf.int64),\n 'num_channels': tf.io.FixedLenFeature([], tf.int64),\n 'img_bytes': tf.io.FixedLenFeature([], tf.string),\n 'mask': tf.io.FixedLenFeature([], tf.string),\n}\n\ndef _parse_image_and_masks_function(example_proto):\n single_example = tf.io.parse_single_example(example_proto, image_feature_description)\n img_bytes = tf.io.decode_raw(single_example['img_bytes'], out_type='uint8')\n img_array = tf.reshape(img_bytes, (512, 512, 3))\n mask_bytes = tf.io.decode_raw(single_example['mask'], out_type='bool')\n mask = tf.reshape(mask_bytes, (512, 512, 1))\n \n ## normalize images array and cast image and mask to float32\n# img_array = tf.cast(img_array, tf.float32) / 255.0\n# mask = tf.cast(mask, tf.float32)\n return img_array, mask\n\ndef read_dataset(storage_file_path):\n encoded_image_dataset = tf.data.TFRecordDataset(storage_file_path, compression_type=\"GZIP\")\n parsed_image_dataset = encoded_image_dataset.map(_parse_image_and_masks_function)\n return parsed_image_dataset","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Get the credential from the Cloud SDK\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nuser_credential = user_secrets.get_gcloud_credential()\n\n# Set the credentials\nuser_secrets.set_tensorflow_credential(user_credential)\n\n# Use a familiar call to get the GCS path of the dataset\nfrom kaggle_datasets import KaggleDatasets\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('hubmap-512x512-tfrecords-with-aug')\nGCS_DS_PATH","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_tf_gcs = GCS_DS_PATH+'/train/*.tfrecords'\nval_tf_gcs = GCS_DS_PATH+'/val/*.tfrecords'\ntrain_tf_files = tf.io.gfile.glob(train_tf_gcs)\nval_tf_files = tf.io.gfile.glob(val_tf_gcs)\nprint(val_tf_files[:3])\nprint(\"Train TFrecord Files:\", len(train_tf_files))\nprint(\"Val TFrecord Files:\", len(val_tf_files))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_dataset = read_dataset(train_tf_files[15])\nvalidation_dataset = read_dataset(val_tf_files[15])\n\ntrain_image = []\ntrain_mask =[]\nfor image, mask in train_dataset.take(5):\n train_image, train_mask = image, mask\ntrain_mask = np.squeeze(train_mask)\n \ntest_image = []\ntest_mask =[]\nfor image, mask in validation_dataset.take(5):\n test_image, test_mask = image, mask\ntest_mask = np.squeeze(test_mask)\n \nfig, ax = plt.subplots(2,2,figsize=(20,10))\nax[0][0].imshow(train_image)\nax[0][1].imshow(train_mask)\nax[1][0].imshow(test_image)\nax[1][1].imshow(test_mask)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Model Building\n\nThe model consists of a VGG19 pretrined sub-network as an encoder, trained on imagenet and a custom decoder sub-network which forms the First U-net Network (NETWORK1). The Second U-net (NETWORK2) consists of a element-wise image-mask multiplier, custom encoder blocks and custom decoder blocks.\n\nVGG and custom encoder blocker encodes the information contained in the input image. Each encoder block in the NETWORK2 performs two 3×3 convolution operation, each followed by a batch normalization. The batch normalization reduces the internal co-variant shift and also regularizes the model. A Rectified Linear Unit (ReLU) activation function is applied, which introduces non-linearity into the model. This is followed by a squeeze-and- excitation block, which enhances the quality of the feature maps and max-pooling with a 2×2 window and stride 2 to reduce the spatial dimension of the feature maps.\n\nAtrous SpatialPyramid Pooling (ASPP) is used in both the sub-networks between the encoder and the decoder to extract high-resolution feature maps that lead to superior performance.\n\nDecoder blocks uses Conv2DTranspose layers which learns a number of filters for performing the upsizing specified with the appropriate kernel_size. The decoder in the NETWORK1,uses only skip connection from the first encoder but, in the decoder of NETWORK2, uses skip connection from both the encoders, which maintains the spatial resolution and enhances the quality of the output feature maps. Squeeze-and-excite blocks are used in the decoder blocks of NETWORK1 and NETWORK2 which helps in reducing redundant information.\n\nThe output masks from both the networks are concatenated, then a final conv layer is used to combine both the masks to get the final output mask. \n\nThe intermediate concatenation and the multiplications of the input image with the output of NETWORK1 and then again the concatenation with the output of NETWORK2 improves the performance of the network and this is the intuitive basis and motivation behind this architecture as described in the original paper.\n\n\n\n","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.layers import *\nfrom tensorflow.keras.applications import *\nfrom tensorflow.keras.callbacks import *\nfrom tensorflow.keras.optimizers import Adam, Nadam\nfrom tensorflow.keras.metrics import *\nfrom tensorflow.keras.losses import binary_crossentropy\n\nnp.random.seed(13)\ntf.random.set_seed(13)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def squeeze_excite_block(inputs, ratio=8):\n init = inputs\n channel_axis = -1\n filters = init.shape[channel_axis]\n se_shape = (1, 1, filters)\n\n se = GlobalAveragePooling2D()(init)\n se = Reshape(se_shape)(se)\n se = Dense(filters // ratio, activation='relu', kernel_initializer='he_normal', use_bias=False)(se)\n se = Dense(filters, activation='sigmoid', kernel_initializer='he_normal', use_bias=False)(se)\n\n x = Multiply()([init, se])\n return x\n\ndef conv_block(inputs, filters):\n x = inputs\n\n x = Conv2D(filters, (3, 3), padding=\"same\")(x)\n x = BatchNormalization()(x)\n x = Activation('relu')(x)\n\n x = Conv2D(filters, (3, 3), padding=\"same\")(x)\n x = BatchNormalization()(x)\n x = Activation('relu')(x)\n\n x = squeeze_excite_block(x)\n\n return x\n\ndef encoder1(inputs):\n skip_connections = []\n\n model = VGG19(include_top=False, weights='imagenet', input_tensor=inputs)\n names = [\"block1_conv2\", \"block2_conv2\", \"block3_conv4\", \"block4_conv4\"]\n for name in names:\n skip_connections.append(model.get_layer(name).output)\n\n output = model.get_layer(\"block5_conv4\").output\n return output, skip_connections\n\ndef decoder1(inputs, skip_connections):\n num_filters = [256, 128, 64, 32]\n skip_connections.reverse()\n x = inputs\n shape = x.shape\n\n for i, f in enumerate(num_filters):\n x = Conv2DTranspose(shape[3], (2, 2), activation=\"relu\", strides=(2, 2))(x)\n x = Concatenate()([x, skip_connections[i]])\n x = conv_block(x, f)\n\n return x\n\ndef encoder2(inputs):\n num_filters = [32, 64, 128, 256]\n skip_connections = []\n x = inputs\n\n for i, f in enumerate(num_filters):\n x = conv_block(x, f)\n skip_connections.append(x)\n x = MaxPool2D((2, 2))(x)\n\n return x, skip_connections\n\ndef decoder2(inputs, skip_1, skip_2):\n num_filters = [256, 128, 64, 32]\n skip_2.reverse()\n x = inputs\n shape = x.shape\n\n for i, f in enumerate(num_filters):\n x = Conv2DTranspose(shape[3], (2, 2), activation=\"relu\", strides=(2, 2))(x)\n x = Concatenate()([x, skip_1[i], skip_2[i]])\n x = conv_block(x, f)\n\n return x\n\ndef output_block(inputs):\n x = Conv2D(1, (1, 1), padding=\"same\")(inputs)\n x = Activation('sigmoid')(x)\n return x\n\ndef ASPP(x, filter):\n shape = x.shape\n\n y1 = AveragePooling2D(pool_size=(shape[1], shape[2]))(x)\n y1 = Conv2D(filter, 1, padding=\"same\")(y1)\n y1 = BatchNormalization()(y1)\n y1 = Activation(\"relu\")(y1)\n shape2 = y1.shape\n \n y1 = Conv2DTranspose(shape2[3], (8,8), activation=\"relu\", strides=(shape[1], shape[2]))(y1)\n \n\n y2 = Conv2D(filter, 1, dilation_rate=1, padding=\"same\", use_bias=False)(x)\n y2 = BatchNormalization()(y2)\n y2 = Activation(\"relu\")(y2)\n\n y3 = Conv2D(filter, 3, dilation_rate=6, padding=\"same\", use_bias=False)(x)\n y3 = BatchNormalization()(y3)\n y3 = Activation(\"relu\")(y3)\n\n y4 = Conv2D(filter, 3, dilation_rate=12, padding=\"same\", use_bias=False)(x)\n y4 = BatchNormalization()(y4)\n y4 = Activation(\"relu\")(y4)\n\n y5 = Conv2D(filter, 3, dilation_rate=18, padding=\"same\", use_bias=False)(x)\n y5 = BatchNormalization()(y5)\n y5 = Activation(\"relu\")(y5)\n\n y = Concatenate()([y1, y2, y3, y4, y5])\n\n y = Conv2D(filter, 1, dilation_rate=1, padding=\"same\", use_bias=False)(y)\n y = BatchNormalization()(y)\n y = Activation(\"relu\")(y)\n\n return y\n\ndef build_model():\n inputs = Input((256, 256, 3))\n x, skip_1 = encoder1(inputs)\n x = ASPP(x, 64)\n x = decoder1(x, skip_1)\n outputs1 = output_block(x)\n\n x = inputs * outputs1\n\n x, skip_2 = encoder2(x)\n x = ASPP(x, 64)\n x = decoder2(x, skip_1, skip_2)\n outputs2 = output_block(x)\n outputs = Concatenate()([outputs1, outputs2])\n \n combine_output = Conv2D(1, (64, 64), activation=\"sigmoid\", padding=\"same\")(outputs)\n\n model = Model(inputs, combine_output)\n return model","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model = build_model()\nmodel.summary(line_length=150)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Define Metrics","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n def dice_coeff(y_true, y_pred):\n # add epsilon to avoid a divide by 0 error in case a slice has no pixels set\n # we only care about relative value, not absolute so this alteration doesn't matter\n _epsilon = 10 ** -7\n intersections = tf.reduce_sum(y_true * y_pred)\n unions = tf.reduce_sum(y_true + y_pred)\n dice_scores = (2.0 * intersections + _epsilon) / (unions + _epsilon)\n return dice_scores\n\n def dice_loss(y_true, y_pred):\n loss = 1 - dice_coeff(y_true, y_pred)\n return loss\n \n def iou(y_true, y_pred):\n def f(y_true, y_pred):\n intersection = (y_true * y_pred).sum()\n union = y_true.sum() + y_pred.sum() - intersection\n x = (intersection + smooth) / (union + smooth)\n x = x.astype(np.float32)\n return x\n return tf.numpy_function(f, [y_true, y_pred], tf.float32)\n \n def bce_dice_loss(y_true, y_pred):\n return binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)\n\n# def focal_loss(y_true, y_pred):\n# alpha=0.25\n# gamma=2\n# def focal_loss_with_logits(logits, targets, alpha, gamma, y_pred):\n# weight_a = alpha * (1 - y_pred) ** gamma * targets\n# weight_b = (1 - alpha) * y_pred ** gamma * (1 - targets)\n# return (tf.math.log1p(tf.exp(-tf.abs(logits))) + tf.nn.relu(-logits)) * (weight_a + weight_b) + logits * weight_b\n\n# y_pred = tf.clip_by_value(y_pred, tf.keras.backend.epsilon(), 1 - tf.keras.backend.epsilon())\n# logits = tf.math.log(y_pred / (1 - y_pred))\n# loss = focal_loss_with_logits(logits=logits, targets=y_true, alpha=alpha, gamma=gamma, y_pred=y_pred)\n# # or reduce_sum and/or axis=-1\n# return tf.reduce_mean(loss)\n \n def tversky(y_true, y_pred, smooth=1, alpha=0.7):\n y_true_pos = tf.reshape(y_true,[-1])\n y_pred_pos = tf.reshape(y_pred,[-1])\n true_pos = tf.reduce_sum(y_true_pos * y_pred_pos)\n false_neg = tf.reduce_sum(y_true_pos * (1 - y_pred_pos))\n false_pos = tf.reduce_sum((1 - y_true_pos) * y_pred_pos)\n return (true_pos + smooth) / (true_pos + alpha * false_neg + (1 - alpha) * false_pos + smooth)\n\n def tversky_loss(y_true, y_pred):\n return 1 - tversky(y_true, y_pred)\n\n def focal_tversky_loss(y_true, y_pred, gamma=0.75):\n tv = tversky(y_true, y_pred)\n return K.pow((1 - tv), gamma)\n\n\n get_custom_objects().update({\"dice\": dice_loss})","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Functions to Load Records","metadata":{}},{"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\nimage_feature_description = {\n 'height': tf.io.FixedLenFeature([], tf.int64),\n 'width': tf.io.FixedLenFeature([], tf.int64),\n 'num_channels': tf.io.FixedLenFeature([], tf.int64),\n 'img_bytes': tf.io.FixedLenFeature([], tf.string),\n 'mask': tf.io.FixedLenFeature([], tf.string),\n}\n\ndef _parse_image_function(example_proto):\n single_example = tf.io.parse_single_example(example_proto, image_feature_description)\n image = tf.reshape( tf.io.decode_raw(single_example['img_bytes'],out_type='uint8'), (512, 512, 3))\n mask = tf.reshape(tf.io.decode_raw(single_example['mask'],out_type='bool'),(512, 512, 1))\n ## normalize images array and cast image and mask to float32\n image = tf.cast(image, tf.float32) / 255.0\n mask = tf.cast(mask, tf.float32)\n return image, mask\n\ndef load_dataset(filenames, ordered=False):\n ignore_order = tf.data.Options()\n if not ordered:\n ignore_order.experimental_deterministic = False\n dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO, compression_type=\"GZIP\")\n dataset = dataset.with_options(ignore_order)\n dataset = dataset.map(_parse_image_function, num_parallel_calls=AUTO)\n return dataset\n\ndef get_training_dataset():\n dataset = load_dataset(train_tf_files)\n #dataset = dataset.repeat()\n dataset = dataset.shuffle(20000)\n dataset = dataset.batch(32, drop_remainder=True)\n dataset = dataset.prefetch(AUTO)\n return dataset\ndef get_val_dataset():\n dataset = load_dataset(val_tf_files)\n #dataset = dataset.repeat()\n dataset = dataset.shuffle(5000)\n dataset = dataset.batch(32, drop_remainder=True)\n dataset = dataset.prefetch(AUTO)\n return dataset","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Model Training","metadata":{}},{"cell_type":"code","source":"if not os.path.exists('/kaggle/working/train_job'):\n os.makedirs('/kaggle/working/train_job')","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n Training Samples : 47703\n\tValidation Samples : 5829\n \n\n","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n metrics = [\n dice_coeff,\n# iou,\n bce_dice_loss,\n# focal_loss,\n Recall(),\n Precision(),\n tversky_loss,\n focal_tversky_loss\n ]\n \n callbacks = [\n ModelCheckpoint('/kaggle/working/hubmap-model-1.h5', verbose=1),\n ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=10),\n CSVLogger(\"/kaggle/working/train_job/data.csv\"),\n # TensorBoard(),\n EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=False)\n ]\n \n model = build_model()\n model.compile(optimizer = Adam(lr = 1e-3), loss = 'dice', metrics=metrics)\n train_dataset = get_training_dataset()\n validation_dataset = get_val_dataset()\n \n train_steps = round((47703//32)*0.70)\n validation_steps = round((5829//32)*0.70)\n\n model.fit(train_dataset, epochs=20, steps_per_epoch=train_steps,\n validation_data=validation_dataset, validation_steps=validation_steps,\n callbacks=callbacks)\n\n model.save_weights(\"/kaggle/working/train_job/hubmap_model_1.h5\")\n","metadata":{"_kg_hide-output":false,"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_steps = round((47703//32)*0.70)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_steps","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}