{"metadata":{"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":61446,"databundleVersionId":6962461,"sourceType":"competition"},{"sourceId":7108687,"sourceType":"datasetVersion","datasetId":4098519}],"dockerImageVersionId":30587,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"papermill":{"default_parameters":{},"duration":3684.058212,"end_time":"2023-12-02T02:58:35.403491","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2023-12-02T01:57:11.345279","version":"2.4.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# UCI CS 184A - Final Project (Fall 2023)\n## [SenNet + HOA - Hacking the Human Vasculature in 3D](https://www.kaggle.com/competitions/blood-vessel-segmentation)\n\n**Team Number:**\n- Canvas Group 35\n- Presentation 2023/12/04 Group 3\n\n**Team Members:**\n- HyunJun Park, 60978255, hyunjup4@uci.edu\n- Rohan Gupta, 31375533, rohang5@uci.edu\n\n\n## Resources\n- [Kaggle Competition Page](https://www.kaggle.com/competitions/blood-vessel-segmentation)  \n- [Project Proposal](https://docs.google.com/document/d/1QiBUwSoZnzDt6wrNvFiOegdJ5592CiDX/edit?usp=sharing&ouid=109514509803758103520&rtpof=true&sd=true)\n- [Project Presentation](https://docs.google.com/presentation/d/1veo1HxeDaX2ksUPXF3ImG_mYxrIeA2ue3EE8Fw5i5Ec/edit?usp=sharing)\n- [Project Report](https://docs.google.com/document/d/1ioLjvBVFu2mRsARxts7K_MP2OyynlN2u/edit?usp=sharing&ouid=109514509803758103520&rtpof=true&sd=true)\n- [GitHub Repository](https://github.com/Nitro1231/SenNet-HOA-Hacking-the-Human-Vasculature-in-3D)\n\n\n## Disclaimer\nThe idea of Attention U-Net Architecture was inspired by [SenNet+HOA | Seg. | PyTorch: Attention-Gated UNet](https://www.kaggle.com/code/aniketkolte04/sennet-hoa-seg-pytorch-attention-gated-unet) by [Aniket Patil](https://www.kaggle.com/aniketkolte04)","metadata":{}},{"cell_type":"markdown","source":"## Import","metadata":{"papermill":{"duration":0.007554,"end_time":"2023-12-02T01:57:14.792656","exception":false,"start_time":"2023-12-02T01:57:14.785102","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os\nimport cv2\nimport numpy as np\nimport pandas as pd\nimport albumentations as A\nimport matplotlib.image as mpimg\nimport matplotlib.pyplot as plt\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom typing import Callable\nfrom tqdm.notebook import tqdm\nfrom torch.utils.data import Dataset, DataLoader, random_split","metadata":{"papermill":{"duration":6.495571,"end_time":"2023-12-02T01:57:21.296105","exception":false,"start_time":"2023-12-02T01:57:14.800534","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-12-07T03:28:52.515204Z","iopub.execute_input":"2023-12-07T03:28:52.515447Z","iopub.status.idle":"2023-12-07T03:28:57.872244Z","shell.execute_reply.started":"2023-12-07T03:28:52.515425Z","shell.execute_reply":"2023-12-07T03:28:57.871366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Pre-setup","metadata":{"papermill":{"duration":0.008577,"end_time":"2023-12-02T01:57:21.313297","exception":false,"start_time":"2023-12-02T01:57:21.304720","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Download and Unzip Data","metadata":{}},{"cell_type":"code","source":"# !kaggle competitions download -c blood-vessel-segmentation\n# !unzip blood-vessel-segmentation.zip","metadata":{"execution":{"iopub.status.busy":"2023-12-07T03:28:57.873969Z","iopub.execute_input":"2023-12-07T03:28:57.874362Z","iopub.status.idle":"2023-12-07T03:28:57.879204Z","shell.execute_reply.started":"2023-12-07T03:28:57.874333Z","shell.execute_reply":"2023-12-07T03:28:57.877922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Defining Constant Variables","metadata":{"papermill":{"duration":0.008029,"end_time":"2023-12-02T01:57:21.329270","exception":false,"start_time":"2023-12-02T01:57:21.321241","status":"completed"},"tags":[]}},{"cell_type":"code","source":"TRAIN = True\nKAGGLE = True\n\nif KAGGLE:\n    CSV_SAVE = '/kaggle/working/model_performance.csv'\n    CSV_LOAD = '/kaggle/input/previous_data/model_performance.csv'\n    MODEL_SAVE = '/kaggle/working/kidney_trained_model.pth'\n    MODEL_LOAD = '/kaggle/input/previous_data/kidney_trained_model.pth'\n    SUBMIT_FILE = '/kaggle/working/submission.csv'\n    DATASET_PATH = '/kaggle/input/blood-vessel-segmentation/'\nelse:\n    CSV_SAVE = './previous_data/model_performance.csv'\n    CSV_LOAD = './previous_data/model_performance.csv'\n    MODEL_SAVE = './previous_data/kidney_trained_model.pth'\n    MODEL_LOAD = './previous_data/kidney_trained_model.pth'\n    SUBMIT_FILE = './previous_data/submission.csv'\n    DATASET_PATH = './blood-vessel-segmentation/'\n\nSPLIT_RATIO = 0.7","metadata":{"papermill":{"duration":0.015781,"end_time":"2023-12-02T01:57:21.353189","exception":false,"start_time":"2023-12-02T01:57:21.337408","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-12-07T03:28:57.880409Z","iopub.execute_input":"2023-12-07T03:28:57.881025Z","iopub.status.idle":"2023-12-07T03:28:57.893558Z","shell.execute_reply.started":"2023-12-07T03:28:57.881000Z","shell.execute_reply":"2023-12-07T03:28:57.892722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Setting torch backend accelerator","metadata":{"papermill":{"duration":0.00782,"end_time":"2023-12-02T01:57:21.368623","exception":false,"start_time":"2023-12-02T01:57:21.360803","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if torch.cuda.is_available():\n    device = torch.device('cuda')\nelif torch.backends.mps.is_available():\n    device = torch.device('mps')\nelse:\n    device = torch.device('cpu')\nprint('Current backend accelerator:', device)","metadata":{"papermill":{"duration":0.079937,"end_time":"2023-12-02T01:57:21.456410","exception":false,"start_time":"2023-12-02T01:57:21.376473","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-12-07T03:28:57.896035Z","iopub.execute_input":"2023-12-07T03:28:57.896347Z","iopub.status.idle":"2023-12-07T03:28:57.930397Z","shell.execute_reply.started":"2023-12-07T03:28:57.896316Z","shell.execute_reply":"2023-12-07T03:28:57.929549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Preparation","metadata":{"papermill":{"duration":0.008178,"end_time":"2023-12-02T01:57:21.472586","exception":false,"start_time":"2023-12-02T01:57:21.464408","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Define Image Preprocessor","metadata":{"papermill":{"duration":0.007755,"end_time":"2023-12-02T01:57:21.488428","exception":false,"start_time":"2023-12-02T01:57:21.480673","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def preprocess_image(path: str) -> torch.Tensor:\n    '''\n    Preprocesses an image for model input.\n\n    Args:\n        path (str): The file path to the image.\n\n    Returns:\n        torch.Tensor: A preprocessed image tensor suitable for model input.\n    '''\n\n    # Reads the image from the given path.\n    img = cv2.imread(path, cv2.IMREAD_UNCHANGED)\n    \n    # Ensures the image has three channels by replicating the single channel if needed.\n    img = np.tile(img[..., None], [1, 1, 3])\n    \n    # Normalizes the image by dividing by its maximum value.\n    img = img.astype('float32')\n    mx = np.max(img)\n    if mx: \n        img /= mx\n\n    # Transposes the image to match the (channels, height, width)\n    img = np.transpose(img, (2, 0, 1))\n\n    return torch.tensor(img)\n\n\ndef preprocess_mask(path: str) -> torch.Tensor:\n    '''\n    Preprocesses a mask image for model input.\n\n    Args:\n        path (str): The file path to the mask image.\n\n    Returns:\n        torch.Tensor: A preprocessed mask tensor suitable for model input.\n    '''\n\n    # Reads the mask image from the given path.\n    msk = cv2.imread(path, cv2.IMREAD_UNCHANGED)\n    \n    # Converts the mask to float32 for precision and normalize.\n    msk = msk.astype('float32')\n    \n    # Normalizes the mask by dividing by 255.\n    msk /= 255.0\n\n    return torch.tensor(msk)","metadata":{"papermill":{"duration":0.017485,"end_time":"2023-12-02T01:57:21.513574","exception":false,"start_time":"2023-12-02T01:57:21.496089","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-12-07T03:28:57.931875Z","iopub.execute_input":"2023-12-07T03:28:57.932496Z","iopub.status.idle":"2023-12-07T03:28:57.950676Z","shell.execute_reply.started":"2023-12-07T03:28:57.932463Z","shell.execute_reply":"2023-12-07T03:28:57.949828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def augment_image(image: torch.Tensor, mask: torch.Tensor = None) -> tuple:\n    '''\n    Applies augmentation transformations to an image and optionally to a mask.\n\n    Args:\n        image (torch.Tensor): The input image as a PyTorch tensor.\n        mask (torch.Tensor, optional): The corresponding mask as a PyTorch tensor. Defaults to None.\n\n    Returns:\n        tuple: A tuple containing the augmented image and, if provided, the augmented mask.\n               If no mask is provided, the second element of the tuple is None.\n    '''\n\n    # Defines a composition of augmentations including resizing, flipping,\n    # shifting, scaling, rotating, cropping, and varying brightness.\n    transform = A.Compose([\n        A.Resize(256, 256, interpolation=cv2.INTER_NEAREST),\n        A.HorizontalFlip(p=0.5),\n        A.VerticalFlip(p=0.5),\n        A.ShiftScaleRotate(scale_limit=0.5, rotate_limit=0, shift_limit=0.1, p=1, border_mode=0),\n        A.RandomCrop(height=256, width=256, always_apply=True),\n        A.RandomBrightness(p=1),\n        A.OneOf(\n            [\n                A.Blur(blur_limit=3, p=1),\n                A.MotionBlur(blur_limit=3, p=1),\n            ],\n            p=0.9,\n        ),\n    ])\n\n    # Convert PyTorch tensor to numpy array for augmentation\n    image_np = image.permute(1, 2, 0).numpy()\n\n    # Apply augmentation to image and optionally to mask\n    if mask is not None:\n        mask_np = mask.numpy()\n        augmented = transform(image=image_np, mask=mask_np)\n        augmented_mask = augmented['mask']\n        augmented_mask = torch.tensor(augmented_mask, dtype=torch.float32)\n    else:\n        augmented = transform(image=image_np)\n\n    # Convert the augmented image back to PyTorch tensor\n    augmented_image = augmented['image']\n    augmented_image = torch.tensor(augmented_image, dtype=torch.float32).permute(2, 0, 1)\n\n    # Return the augmented image and mask\n    if mask is not None:\n        return augmented_image, augmented_mask\n    else:\n        return augmented_image, None","metadata":{"papermill":{"duration":0.01881,"end_time":"2023-12-02T01:57:21.555540","exception":false,"start_time":"2023-12-02T01:57:21.536730","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-12-07T03:28:57.951769Z","iopub.execute_input":"2023-12-07T03:28:57.952039Z","iopub.status.idle":"2023-12-07T03:28:57.962649Z","shell.execute_reply.started":"2023-12-07T03:28:57.952009Z","shell.execute_reply":"2023-12-07T03:28:57.961808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Define Dataset","metadata":{"papermill":{"duration":0.007711,"end_time":"2023-12-02T01:57:21.571265","exception":false,"start_time":"2023-12-02T01:57:21.563554","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class KidneyDataset(Dataset):\n    '''\n    A PyTorch Dataset subclass for loading and transforming kidney images and their corresponding labels.\n\n    Args:\n        dataset (str): Name of the dataset folder. For example, 'kidney_1_dense', 'kidney_2', etc.\n        transform (Callable, optional): A function or callable object for applying transformations to the images and masks.\n        mode (str, optional): Mode of the dataset. Can be 'train' or 'test'. Defaults to 'train'.\n\n    Attributes:\n        dataset (str): Name of the dataset.\n        transform (Callable): Transformation function for images and masks.\n        mode (str): Mode of the dataset.\n        image_files (list): List of paths to the image files.\n        label_files (list, optional): List of paths to the label files, only used in 'train' mode.\n    '''\n\n    def __init__(self, dataset: str, transform: Callable = None, mode: str = 'train') -> None:\n        self.dataset = dataset\n        self.transform = transform\n        self.mode = mode\n\n        # Define the path for images\n        images_path = os.path.join(DATASET_PATH, dataset, 'images')\n        self.image_files = sorted([os.path.join(images_path, f) for f in os.listdir(images_path) if f.endswith('.tif')])\n        \n        # Load labels if in 'train' mode\n        if mode == 'train':\n            labels_path = os.path.join(DATASET_PATH, dataset, 'labels')\n            self.label_files = sorted([os.path.join(labels_path, f) for f in os.listdir(labels_path) if f.endswith('.tif')])\n\n    def __len__(self) -> int:\n        # Return the total number of images in the dataset\n        return len(self.image_files)\n\n    def __getitem__(self, idx: int) -> tuple:\n        '''\n        Retrieves an image (and its corresponding mask, if in 'train' mode) by index.\n\n        Args:\n            idx (int): The index of the item.\n\n        Returns:\n            tuple: Depending on the mode, returns a tuple (image, mask, image_file_path) in 'train' mode or (image, image_file_path) otherwise.\n        '''\n\n        # Load and preprocess the image\n        image = preprocess_image(self.image_files[idx])\n\n        # Load and preprocess the mask if in 'train' mode\n        mask = preprocess_mask(self.label_files[idx]) if self.mode == 'train' else None\n    \n        # Apply transformations, if provided\n        if self.transform:\n            image, mask = self.transform(image, mask)\n        \n        # Return the appropriate data\n        if self.mode == 'train':\n            return image, mask, self.image_files[idx]\n        else:\n            return image, self.image_files[idx]","metadata":{"papermill":{"duration":0.019685,"end_time":"2023-12-02T01:57:21.598690","exception":false,"start_time":"2023-12-02T01:57:21.579005","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-12-07T03:28:57.964059Z","iopub.execute_input":"2023-12-07T03:28:57.964661Z","iopub.status.idle":"2023-12-07T03:28:57.975479Z","shell.execute_reply.started":"2023-12-07T03:28:57.964636Z","shell.execute_reply":"2023-12-07T03:28:57.974587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load and Split Dataset","metadata":{"papermill":{"duration":0.007892,"end_time":"2023-12-02T01:57:21.614496","exception":false,"start_time":"2023-12-02T01:57:21.606604","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Load Dataset\ndataset = KidneyDataset(dataset='train/kidney_1_dense', transform=augment_image)\ndataset += KidneyDataset(dataset='train/kidney_2', transform=augment_image)\ndataset += KidneyDataset(dataset='train/kidney_1_voi', transform=augment_image)\ndataset += KidneyDataset(dataset='train/kidney_3_sparse', transform=augment_image)\n\n# Split Dataset into trainning and testing\ntrain_size = int(SPLIT_RATIO * len(dataset))\ntest_size = len(dataset) - train_size\ntrain_dataset, test_dataset = random_split(dataset, [train_size, test_size])\n\ntrain_dataloader = DataLoader(train_dataset, batch_size=8, shuffle=True)\nval_dataloader = DataLoader(test_dataset, batch_size=8, shuffle=False)","metadata":{"papermill":{"duration":0.261208,"end_time":"2023-12-02T01:57:21.883695","exception":false,"start_time":"2023-12-02T01:57:21.622487","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-12-07T03:28:57.976661Z","iopub.execute_input":"2023-12-07T03:28:57.977306Z","iopub.status.idle":"2023-12-07T03:29:00.405010Z","shell.execute_reply.started":"2023-12-07T03:28:57.977281Z","shell.execute_reply":"2023-12-07T03:29:00.404205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Example Images (Original and Processed)","metadata":{}},{"cell_type":"code","source":"for images, masks, paths in train_dataloader:\n    for image, mask, path in zip(images, masks, paths):\n        image = image.permute((1, 2, 0)).numpy() * 255.0\n        image = image.astype('uint8')\n        mask = (mask*255).numpy().astype('uint8')\n        \n        # print(path)\n        plt.figure(figsize=(10, 10))\n\n        plt.subplot(2, 2, 1)\n        plt.imshow(mpimg.imread(path))\n        plt.title('Original Image')\n\n        plt.subplot(2, 2, 2)\n        plt.imshow(image, cmap='gray')\n        plt.title('Processed Image')\n\n        plt.subplot(2, 2, 3)\n        plt.imshow(mpimg.imread(path.replace('images', 'labels')))\n        plt.title('Original Mask')\n        \n        plt.subplot(2, 2, 4)\n        plt.imshow(mask, cmap='gray')\n        plt.title('Processed Mask')\n\n        plt.tight_layout()\n        plt.show()\n    break","metadata":{"papermill":{"duration":5.500796,"end_time":"2023-12-02T01:57:27.416497","exception":false,"start_time":"2023-12-02T01:57:21.915701","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-12-07T03:29:00.406102Z","iopub.execute_input":"2023-12-07T03:29:00.406371Z","iopub.status.idle":"2023-12-07T03:29:11.257730Z","shell.execute_reply.started":"2023-12-07T03:29:00.406348Z","shell.execute_reply":"2023-12-07T03:29:11.256788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model: Attention U-Net","metadata":{"papermill":{"duration":0.017218,"end_time":"2023-12-02T01:57:27.454617","exception":false,"start_time":"2023-12-02T01:57:27.437399","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class ConvBlock(nn.Module):\n    '''\n    A convolutional block consisting of two convolutional layers, each followed by batch normalization and a ReLU activation function.\n\n    Args:\n        in_channels (int): Number of channels in the input tensor.\n        out_channels (int): Number of channels produced by the convolutional layers.\n    \n    Attributes:\n        conv (nn.Sequential): Sequential container of convolutional layers, batch normalization, and ReLU activation.\n    '''\n\n    def __init__(self, in_channels: int, out_channels: int) -> None:\n        super(ConvBlock, self).__init__()\n        self.conv = nn.Sequential(\n            nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=True),\n            nn.BatchNorm2d(out_channels),\n            nn.ReLU(inplace=True),\n            nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=True),\n            nn.BatchNorm2d(out_channels),\n            nn.ReLU(inplace=True)\n        )\n\n    def forward(self, x: torch.tensor) -> torch.tensor:\n        x = self.conv(x)\n        return x\n\nclass UpConv(nn.Module):\n    '''\n    An upsampling block that increases the spatial dimensions of the input tensor using bilinear upsampling, followed by a convolutional layer.\n\n    Args:\n        in_channels (int): Number of channels in the input tensor.\n        out_channels (int): Number of channels produced by the convolutional layer.\n\n    Attributes:\n        up (nn.Sequential): Sequential container for upsampling and convolutional layer.\n    '''\n\n    def __init__(self, in_channels: int, out_channels: int) -> None:\n        super(UpConv, self).__init__()\n        self.up = nn.Sequential(\n            nn.Upsample(scale_factor=2),\n            nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=True),\n            nn.BatchNorm2d(out_channels),\n            nn.ReLU(inplace=True)\n        )\n\n    def forward(self, x: torch.tensor) -> torch.tensor:\n        x = self.up(x)\n        return x\n\nclass AttentionBlock(nn.Module):\n    '''\n    An attention block that computes attention coefficients to enhance feature representation from skip connections in U-Net architecture.\n\n    Args:\n        prev_layer (int): Number of feature maps in the gating signal from the previous layer.\n        encoder_layer (int): Number of feature maps in the corresponding encoder layer transferred via the skip connection.\n        coef (int): Number of learnable multi-dimensional attention coefficients.\n    \n    Attributes:\n        gate (nn.Sequential): Sequential container for processing the gating signal.\n        trans (nn.Sequential): Sequential container for processing the feature maps from the skip connection.\n        psi (nn.Sequential): Sequential container for computing the final attention coefficients.\n        relu (nn.ReLU): ReLU activation function.\n    '''\n\n    def __init__(self, prev_layer: int, encoder_layer: int, coef: int) -> None:\n        super(AttentionBlock, self).__init__()\n        self.gate = nn.Sequential(\n            nn.Conv2d(prev_layer, coef, kernel_size=1, stride=1, padding=0, bias=True),\n            nn.BatchNorm2d(coef)\n        )\n        self.trans = nn.Sequential(\n            nn.Conv2d(encoder_layer, coef, kernel_size=1, stride=1, padding=0, bias=True),\n            nn.BatchNorm2d(coef)\n        )\n        self.psi = nn.Sequential(\n            nn.Conv2d(coef, 1, kernel_size=1, stride=1, padding=0, bias=True),\n            nn.BatchNorm2d(1),\n            nn.Sigmoid()\n        )\n        self.relu = nn.ReLU(inplace=True)\n\n    def forward(self, prev_gate: torch.Tensor, skip_connection: torch.Tensor) -> torch.tensor:\n        '''\n        Args:\n            gate (torch.Tensor): Signal from previous layer.\n            skip_connection (torch.Tensor): Activation from corresponding encoder layer.\n        \n        Return:\n            (torch.Tensor): Output activations.\n        '''\n\n        g = self.gate(prev_gate)\n        x = self.trans(skip_connection)\n        psi = self.relu(g + x)\n        psi = self.psi(psi)\n        out = skip_connection * psi\n        return out\n\nclass AttentionUNet(nn.Module):\n    '''\n    An Attention U-Net architecture for biomedical image segmentation, incorporating attention blocks to focus on important features.\n\n    Args:\n        input_ch (int, optional): Number of channels in the input image. Default is 3 for RGB images.\n        output_ch (int, optional): Number of channels in the output tensor. Default is 1 for binary segmentation.\n\n    Attributes:\n        MaxPool (nn.MaxPool2d): Max pooling layer to reduce spatial dimensions.\n        Conv1, Conv2, Conv3, Conv4, Conv5, Conv6 (ConvBlock): Convolutional blocks for the encoder part of U-Net.\n        Up6, Up5, Up4, Up3, Up2 (UpConv): Upsampling blocks for the decoder part of U-Net.\n        Att6, Att5, Att4, Att3, Att2 (AttentionBlock): Attention blocks to refine skip connections.\n        UpConv6, UpConv5, UpConv4, UpConv3, UpConv2 (ConvBlock): Convolutional blocks for merging upsampled features and skip connections.\n        Conv (nn.Conv2d): Final convolutional layer to produce the segmentation map.\n    '''\n\n    def __init__(self, input_ch: int=3, output_ch: int=1) -> None:\n        super(AttentionUNet, self).__init__()\n        self.MaxPool = nn.MaxPool2d(kernel_size=2, stride=2)\n\n        # Encoder\n        self.Conv1 = ConvBlock(input_ch, 32)\n        self.Conv2 = ConvBlock(32, 64)\n        self.Conv3 = ConvBlock(64, 128)\n        self.Conv4 = ConvBlock(128, 256)\n        self.Conv5 = ConvBlock(256, 512)\n        self.Conv6 = ConvBlock(512, 1024)\n\n        # Decoder\n        self.Up6 = UpConv(1024, 512)\n        self.Att6 = AttentionBlock(prev_layer=512, encoder_layer=512, coef=256)\n        self.UpConv6 = ConvBlock(1024, 512)\n\n        self.Up5 = UpConv(512, 256)\n        self.Att5 = AttentionBlock(prev_layer=256, encoder_layer=256, coef=128)\n        self.UpConv5 = ConvBlock(512, 256)\n\n        self.Up4 = UpConv(256, 128)\n        self.Att4 = AttentionBlock(prev_layer=128, encoder_layer=128, coef=64)\n        self.UpConv4 = ConvBlock(256, 128)\n\n        self.Up3 = UpConv(128, 64)\n        self.Att3 = AttentionBlock(prev_layer=64, encoder_layer=64, coef=32)\n        self.UpConv3 = ConvBlock(128, 64)\n        \n        self.Up2 = UpConv(64, 32)\n        self.Att2 = AttentionBlock(prev_layer=32, encoder_layer=32, coef=16)\n        self.UpConv2 = ConvBlock(64, 32)\n\n        # Output layer\n        self.Conv = nn.Conv2d(32, output_ch, kernel_size=1, stride=1, padding=0)\n\n    def forward(self, x: torch.tensor) -> torch.tensor:\n        '''\n        e: Encoder layers.\n        d: Decoder layers.\n        s: Skip-connections from encoder layers to decoder layers.\n        '''\n\n        # Encoder pathway\n        e1 = self.Conv1(x)\n        e2 = self.Conv2(self.MaxPool(e1))\n        e3 = self.Conv3(self.MaxPool(e2))\n        e4 = self.Conv4(self.MaxPool(e3))\n        e5 = self.Conv5(self.MaxPool(e4))\n        e6 = self.Conv6(self.MaxPool(e5))\n\n\n        # Decoder pathway\n        d6 = self.Up6(e6)\n        s5 = self.Att6(prev_gate=d6, skip_connection=e5)\n        d6 = torch.cat((s5, d6), dim=1) \n        d6 = self.UpConv6(d6)\n        \n        d5 = self.Up5(d6)\n        s4 = self.Att5(prev_gate=d5, skip_connection=e4)\n        d5 = torch.cat((s4, d5), dim=1) \n        d5 = self.UpConv5(d5)\n\n        d4 = self.Up4(d5)\n        s3 = self.Att4(prev_gate=d4, skip_connection=e3)\n        d4 = torch.cat((s3, d4), dim=1)\n        d4 = self.UpConv4(d4)\n\n        d3 = self.Up3(d4)\n        s2 = self.Att3(prev_gate=d3, skip_connection=e2)\n        d3 = torch.cat((s2, d3), dim=1)\n        d3 = self.UpConv3(d3)\n\n        d2 = self.Up2(d3)\n        s1 = self.Att2(prev_gate=d2, skip_connection=e1)\n        d2 = torch.cat((s1, d2), dim=1)\n        d2 = self.UpConv2(d2)\n\n        # Output layer\n        out = self.Conv(d2)\n        return out","metadata":{"papermill":{"duration":0.046673,"end_time":"2023-12-02T01:57:27.553370","exception":false,"start_time":"2023-12-02T01:57:27.506697","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-12-07T03:29:11.260825Z","iopub.execute_input":"2023-12-07T03:29:11.261116Z","iopub.status.idle":"2023-12-07T03:29:11.292128Z","shell.execute_reply.started":"2023-12-07T03:29:11.261091Z","shell.execute_reply":"2023-12-07T03:29:11.291318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training and Evaluating","metadata":{"papermill":{"duration":0.014331,"end_time":"2023-12-02T01:57:27.726869","exception":false,"start_time":"2023-12-02T01:57:27.712538","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Define Evaulator","metadata":{"papermill":{"duration":0.01382,"end_time":"2023-12-02T01:57:27.581817","exception":false,"start_time":"2023-12-02T01:57:27.567997","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def dice_coeff(prediction: torch.Tensor, target: torch.Tensor) -> float:\n    '''\n    Calculates the Dice coefficient, a measure of overlap between two samples.\n\n    Args:\n        prediction (torch.Tensor): The prediction tensor from the model.\n        target (torch.Tensor): The ground truth tensor.\n\n    Returns:\n        float: The Dice coefficient, ranging from 0 (no overlap) to 1 (perfect overlap).\n    '''\n\n    # Convert prediction to a binary mask\n    mask = np.zeros_like(prediction)\n    mask[prediction >= 0.5] = 1\n\n    # Calculate intersection and union\n    inter = np.sum(mask * target)\n    union = np.sum(mask) + np.sum(target)\n\n    # Small constant to avoid division by zero\n    epsilon = 1e-6\n\n    # Calculate and return the Dice coefficient\n    result = 2 * inter / (union + epsilon)\n    return float(np.mean(result))","metadata":{"papermill":{"duration":0.023002,"end_time":"2023-12-02T01:57:27.618986","exception":false,"start_time":"2023-12-02T01:57:27.595984","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-12-07T03:29:11.293094Z","iopub.execute_input":"2023-12-07T03:29:11.293364Z","iopub.status.idle":"2023-12-07T03:29:11.304917Z","shell.execute_reply.started":"2023-12-07T03:29:11.293341Z","shell.execute_reply":"2023-12-07T03:29:11.304118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Define Loss (Focal Loss)","metadata":{}},{"cell_type":"code","source":"class FocalLoss(nn.modules.loss._WeightedLoss):\n    '''\n    Focal Loss, a variant of Cross-Entropy Loss, which adds a factor to down-weight easy examples and focus training on hard negatives.\n\n    Args:\n        gamma (float, optional): Focusing parameter to adjust the rate at which easy examples are down-weighted.\n                                 When gamma is 0, Focal Loss is equivalent to Binary Cross-Entropy Loss.\n                                 The higher the value of gamma, the more focus on hard negatives.\n        size_average (bool, optional): By default, the losses are averaged over each loss element in the batch.\n                                       If the field size_average is set to False, the losses are instead summed.\n        ignore_index (int, optional): Specifies a target value that is ignored and does not contribute to the input gradient.\n        reduce (bool, optional): Deprecated (use reduction instead). By default, the losses are averaged or summed over observations for each minibatch.\n        balance_param (float, optional): Balancing parameter to balance the positive and negative examples.\n\n    Methods:\n        forward(input, target): Compute the loss given input predictions and true targets.\n    '''\n\n    def __init__(self, gamma: float = 0, size_average: bool = None, ignore_index: int = -100, reduce: bool = None, balance_param: float = 1.0) -> None:\n        super(FocalLoss, self).__init__(size_average)\n        self.gamma = gamma\n        self.size_average = size_average\n        self.ignore_index = ignore_index\n        self.balance_param = balance_param\n\n    def forward(self, input: torch.Tensor, target: torch.Tensor) -> torch.Tensor:\n        '''\n        Forward pass to compute the focal loss.\n\n        Args:\n            input (torch.Tensor): Predictions output by the model (before activation).\n            target (torch.Tensor): Ground truth labels.\n\n        Returns:\n            torch.Tensor: Computed focal loss.\n        '''\n\n        # Ensure the input and target shapes are consistent\n        assert len(input.shape) == len(target.shape)\n        assert input.size(0) == target.size(0)\n        assert input.size(1) == target.size(1)\n\n        # Calculate binary cross-entropy loss with logits\n        logpt = - F.binary_cross_entropy_with_logits(input, target)\n\n        # Calculate the probability of the correct class\n        pt = torch.exp(logpt)\n\n        # Apply the focal loss formula\n        focal_loss = -((1 - pt) ** self.gamma) * logpt\n\n        # Apply the balance parameter\n        balanced_focal_loss = self.balance_param * focal_loss\n\n        return balanced_focal_loss","metadata":{"papermill":{"duration":0.025844,"end_time":"2023-12-02T01:57:27.658792","exception":false,"start_time":"2023-12-02T01:57:27.632948","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-12-07T03:29:11.305935Z","iopub.execute_input":"2023-12-07T03:29:11.306192Z","iopub.status.idle":"2023-12-07T03:29:11.317272Z","shell.execute_reply.started":"2023-12-07T03:29:11.306169Z","shell.execute_reply":"2023-12-07T03:29:11.316572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Define `train_and_eval` function.","metadata":{}},{"cell_type":"code","source":"def train_and_eval(\n    model: torch.nn.Module, \n    train_data: torch.utils.data.DataLoader, \n    val_data: torch.utils.data.DataLoader, \n    optimizer: torch.optim.Optimizer, \n    criterion, \n    num_epochs: int = 25, \n    save_csv: bool = True\n) -> tuple:\n    '''\n    Trains and evaluates a PyTorch model.\n\n    Args:\n        model (torch.nn.Module): The neural network model to train and test.\n        train_data (torch.utils.data.DataLoader): DataLoader for the training data.\n        val_data (torch.utils.data.DataLoader): DataLoader for the validation data.\n        optimizer (torch.optim.Optimizer): The optimizer for training the model.\n        criterion: The loss function used for training.\n        num_epochs (int, optional): The number of training epochs. Defaults to 25.\n        save_csv (bool, optional): If True, saves the training and validation metrics to a CSV file. Defaults to True.\n\n    Returns:\n        tuple: A tuple containing model, lists of training losses, validation losses, training Dice coefficients, and validation Dice coefficients.\n    '''\n\n    # Initialize metrics\n    training_loss, test_loss, training_dice_coeff, test_dice_coeff = [], [], [], []\n\n    # Transfer model to the specified device\n    model.to(device)\n\n    # Training and Evaluation loop\n    for epoch in tqdm(range(1, num_epochs + 1)):\n        \n        # Training phase\n        model.train()\n        batch_train_loss, batch_train_dice_coeff = 0.0, 0.0\n        for images, masks, _ in train_data:\n            images, masks = images.to(device), masks.to(device)\n            masks = masks.unsqueeze(1)\n\n            optimizer.zero_grad()\n            outputs = model(images)\n            loss = criterion(outputs, masks)\n\n            # Calculate predictions and true values for dice coefficient\n            y_pred = outputs.data.cpu().numpy().ravel()\n            y_true = masks.data.cpu().numpy().ravel()\n\n            batch_train_loss += loss.item() * images.size(0)\n            batch_train_dice_coeff += dice_coeff(y_pred, y_true)\n\n            loss.backward()\n            optimizer.step()\n        \n        # Record average training loss and dice coefficient for the epoch\n        training_loss.append(batch_train_loss / len(train_data))\n        training_dice_coeff.append(batch_train_dice_coeff / len(train_data))\n\n        # Evaluation phase\n        model.eval()\n        batch_test_loss, batch_test_dice_coeff = 0.0, 0.0\n        with torch.no_grad():\n            for images, masks, _ in val_data:\n                images, masks = images.to(device), masks.to(device)\n                masks = masks.unsqueeze(1)\n\n                outputs = model(images)\n                loss = criterion(outputs, masks)\n\n                y_pred = outputs.data.cpu().numpy().ravel()\n                y_true = masks.data.cpu().numpy().ravel()\n\n                batch_test_loss += loss.item() * images.size(0)\n                batch_test_dice_coeff += dice_coeff(y_pred, y_true)\n        \n        # Record average test loss and dice coefficient for the epoch\n        test_loss.append(batch_test_loss / len(val_data))\n        test_dice_coeff.append(batch_test_dice_coeff / len(val_data))\n\n        print(f'Epoch [{epoch}/{num_epochs}]')\n        print('\\tTrain Loss:', batch_train_loss / len(train_data))\n        print('\\tTest Loss:', batch_test_loss / len(val_data))\n        print('\\tTrain Dice Coefficient:', batch_train_dice_coeff / len(train_data))\n        print('\\tTest Dice Coefficient:', batch_test_dice_coeff / len(val_data))\n\n    results = {\n        'Epoch': list(range(1, num_epochs + 1)),\n        'Train Loss': training_loss,\n        'Test Loss': test_loss,\n        'Train Dice Coefficient': training_dice_coeff,\n        'Test Dice Coefficient': test_dice_coeff,\n    }\n    df = pd.DataFrame(results)\n\n    if save_csv: # Save to CSV file\n        df.to_csv(CSV_SAVE, index=False)\n\n    return model, df","metadata":{"papermill":{"duration":0.032974,"end_time":"2023-12-02T01:57:27.774348","exception":false,"start_time":"2023-12-02T01:57:27.741374","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-12-07T03:29:11.318481Z","iopub.execute_input":"2023-12-07T03:29:11.319124Z","iopub.status.idle":"2023-12-07T03:29:11.335811Z","shell.execute_reply.started":"2023-12-07T03:29:11.319090Z","shell.execute_reply":"2023-12-07T03:29:11.335066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Define Training parameters","metadata":{}},{"cell_type":"code","source":"model = AttentionUNet()\nmodel.to(device)\n\nepochs = 25\nlearning_rate = 1e-4\ncriterion = FocalLoss(gamma=2)\noptimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)","metadata":{"papermill":{"duration":3664.300214,"end_time":"2023-12-02T02:58:32.089077","exception":false,"start_time":"2023-12-02T01:57:27.788863","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-12-07T08:29:49.651955Z","iopub.execute_input":"2023-12-07T08:29:49.652328Z","iopub.status.idle":"2023-12-07T08:29:50.003683Z","shell.execute_reply.started":"2023-12-07T08:29:49.652294Z","shell.execute_reply":"2023-12-07T08:29:50.002479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train or Load model","metadata":{}},{"cell_type":"code","source":"if TRAIN:\n    model, performance = train_and_eval(model, train_dataloader, val_dataloader, optimizer, criterion, epochs, True)\n    torch.save(model.state_dict(), MODEL_SAVE)\nelse:\n    model.load_state_dict(torch.load(MODEL_LOAD))\n    performance = pd.read_csv(CSV_LOAD)","metadata":{"execution":{"iopub.status.busy":"2023-12-07T03:29:14.311186Z","iopub.execute_input":"2023-12-07T03:29:14.311465Z","iopub.status.idle":"2023-12-07T07:05:14.510927Z","shell.execute_reply.started":"2023-12-07T03:29:14.311441Z","shell.execute_reply":"2023-12-07T07:05:14.509139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Performance","metadata":{}},{"cell_type":"markdown","source":"### Performance Measures","metadata":{}},{"cell_type":"code","source":"display(performance)\n\nfigure, axis = plt.subplots(1, 2, figsize=(12, 5))\naxis[0].plot(performance['Epoch'], performance['Train Loss'], label='Train Loss', lw=2)\naxis[0].plot(performance['Epoch'], performance['Test Loss'], label='Test Loss', lw=2)\naxis[0].legend()\naxis[0].set_ylabel('Loss')\naxis[0].set_xlabel('Epochs')\naxis[0].set_title('Loss for each Epochs')\n\naxis[1].semilogy(performance['Epoch'], performance['Train Loss'], label='Train Loss', lw=2)\naxis[1].semilogy(performance['Epoch'], performance['Test Loss'], label='Test Loss', lw=2)\naxis[1].legend()\naxis[1].set_ylabel('Loss')\naxis[1].set_xlabel('Epochs')\naxis[1].set_title('Loss for each Epochs (Log Scale)')\nplt.show()\n\nplt.plot(performance['Epoch'], performance['Train Dice Coefficient'], label='Train Dice Coefficient', lw=2)\nplt.plot(performance['Epoch'], performance['Test Dice Coefficient'], label='Test Dice Coefficient', lw=2)\nplt.legend()\nplt.ylabel('Dice Coefficient')\nplt.xlabel('Epochs')\nplt.title(f'Dice Coefficient for each Epochs')\nplt.show()\n\nprint(f'Min Train Loss: {np.min(performance[\"Train Loss\"]):.4f} (Epoch: {performance[\"Epoch\"][np.argmin(performance[\"Train Loss\"])]})')\nprint(f'Min Test Loss: {np.min(performance[\"Test Loss\"]):.4f} (Epoch: {performance[\"Epoch\"][np.argmin(performance[\"Test Loss\"])]})')\nprint(f'Best Train Dice Coefficient: {np.max(performance[\"Train Dice Coefficient\"]):.4f} (Epoch: {performance[\"Epoch\"][np.argmax(performance[\"Train Dice Coefficient\"])]})')\nprint(f'Best Test Dice Coefficient: {np.max(performance[\"Test Dice Coefficient\"]):.4f} (Epoch: {performance[\"Epoch\"][np.argmax(performance[\"Test Dice Coefficient\"])]})')","metadata":{"papermill":{"duration":0.323679,"end_time":"2023-12-02T02:58:32.700546","exception":false,"start_time":"2023-12-02T02:58:32.376867","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-12-07T07:05:14.515475Z","iopub.execute_input":"2023-12-07T07:05:14.515886Z","iopub.status.idle":"2023-12-07T07:05:16.062973Z","shell.execute_reply.started":"2023-12-07T07:05:14.515850Z","shell.execute_reply":"2023-12-07T07:05:16.061745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Examples","metadata":{}},{"cell_type":"code","source":"model.eval()\nfor images, masks, paths in val_dataloader:\n    images = images.to(device)\n    outputs = model(images)\n    preds = torch.sigmoid(outputs).cpu()\n    \n    for path, image, mask, pred in zip(paths, images, masks, preds):\n        pred_image = pred.detach().numpy().ravel() > 0.5\n\n        image = image.cpu().permute((1, 2, 0)).numpy()*255.0\n        image = image.astype('uint8')\n        mask = (mask.cpu()*255).numpy().astype('uint8')\n        pred = (pred*255).detach().numpy().astype('uint8')\n\n        plt.figure(figsize=(10, 10))\n\n        plt.subplot(2, 3, 1)\n        plt.imshow(mpimg.imread(path))\n        plt.title('Original Image')\n\n        plt.subplot(2, 3, 2)\n        plt.imshow(image, cmap='gray')\n        plt.title('Processed Image')\n\n        plt.subplot(2, 3, 4)\n        plt.imshow(mpimg.imread(path.replace('images', 'labels')))\n        plt.title('Original Mask')\n        \n        plt.subplot(2, 3, 5)\n        plt.imshow(mask, cmap='gray')\n        plt.title('Processed Mask')\n        \n        plt.subplot(2, 3, 6)\n        plt.imshow(mask, cmap='gray')\n        plt.title('Predicted Mask')\n\n        plt.tight_layout()\n        plt.show()\n    break","metadata":{"execution":{"iopub.status.busy":"2023-12-07T07:05:16.064690Z","iopub.execute_input":"2023-12-07T07:05:16.065152Z","iopub.status.idle":"2023-12-07T07:05:27.859858Z","shell.execute_reply.started":"2023-12-07T07:05:16.065110Z","shell.execute_reply":"2023-12-07T07:05:27.859004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Kaggle Submission","metadata":{}},{"cell_type":"markdown","source":"### Define the RLE encoding function","metadata":{}},{"cell_type":"code","source":"def rle_encode(mask: np.ndarray) -> str:\n    '''\n    Encodes a binary mask to run-length encoding (RLE).\n    ref.: https://www.kaggle.com/stainsby/fast-tested-rle\n\n    Args:\n        mask (np.ndarray): A 2D numpy array representing the binary mask image to be encoded.\n\n    Returns:\n        str: The run-length encoded string of the mask.\n    '''\n\n    # Flatten the mask and add zero padding at the start and end\n    pixel = mask.flatten()\n    pixel = np.concatenate([[0], pixel, [0]])\n\n    # Find the indices of changes in the mask value\n    run = np.where(pixel[1:] != pixel[:-1])[0] + 1\n\n    # Compute the lengths of runs of 1's\n    run[1::2] -= run[::2]\n\n    # Convert the run lengths to a string format\n    rle = ' '.join(str(r) for r in run)\n\n    # Handle the special case of an all-zero mask\n    if rle == '':\n        rle = '1 0'\n    return rle","metadata":{"execution":{"iopub.status.busy":"2023-12-07T07:05:27.861061Z","iopub.execute_input":"2023-12-07T07:05:27.861350Z","iopub.status.idle":"2023-12-07T07:05:27.868890Z","shell.execute_reply.started":"2023-12-07T07:05:27.861324Z","shell.execute_reply":"2023-12-07T07:05:27.867945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load Submission Dataset","metadata":{}},{"cell_type":"code","source":"test_dataset = KidneyDataset(dataset='test/kidney_5', transform=augment_image, mode='test')\ntest_dataset += KidneyDataset(dataset='test/kidney_6', transform=augment_image, mode='test')\ntest_dataloader = DataLoader(test_dataset, batch_size=8, shuffle=False)","metadata":{"execution":{"iopub.status.busy":"2023-12-07T07:05:27.869956Z","iopub.execute_input":"2023-12-07T07:05:27.870224Z","iopub.status.idle":"2023-12-07T07:05:27.888567Z","shell.execute_reply.started":"2023-12-07T07:05:27.870200Z","shell.execute_reply":"2023-12-07T07:05:27.887722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Run Model and Save Submission Data","metadata":{}},{"cell_type":"code","source":"submission_data = []\n\nmodel.to(device)\nmodel.eval()\nwith torch.no_grad():\n    for images, path in tqdm(test_dataloader):\n        images = images.to(device)\n        outputs = model(images)\n        predictions = torch.sigmoid(outputs)\n\n        for j in range(predictions.shape[0]):\n            rle_mask = rle_encode(predictions[j].cpu().numpy().ravel() > 0.5)\n            id_text = path[j].split('test/')[1].replace('/images/', '_').split('.')[0]\n            submission_data.append({'id': id_text, 'rle': rle_mask})\n\nsubmission_df = pd.DataFrame(submission_data)\nsubmission_df.to_csv(SUBMIT_FILE, index=False)\ndisplay(submission_df)","metadata":{"execution":{"iopub.status.busy":"2023-12-07T07:05:27.889657Z","iopub.execute_input":"2023-12-07T07:05:27.889958Z","iopub.status.idle":"2023-12-07T07:05:28.573355Z","shell.execute_reply.started":"2023-12-07T07:05:27.889934Z","shell.execute_reply":"2023-12-07T07:05:28.572490Z"},"trusted":true},"execution_count":null,"outputs":[]}]}