{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":61446,"databundleVersionId":6962461,"sourceType":"competition"}],"dockerImageVersionId":30587,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Competition goal <a class=\"anchor\" id=\"goal\"></a>\n\nOur task is to create segmentation masks for the kidney datasets in the test set.","metadata":{}},{"cell_type":"markdown","source":"*Work in progress*","metadata":{}},{"cell_type":"markdown","source":"### Table of contents\n\n1. [Competition goal](#goal)\n2. [Prepare to start](#prepare)\n3. [Exploring the training data](#eda)\n4. [Ideas for validation](#validation)\n5. [Setting up a baseline](#baseline)","metadata":{}},{"cell_type":"markdown","source":"# Prepare to start <a class=\"anchor\" id=\"prepare\"></a>\n\n* [Importing packages](#packages)\n* [Helper methods](#helpers)","metadata":{}},{"cell_type":"markdown","source":"## Importing packages <a class=\"anchor\" id=\"packages\"></a>","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\nimport cv2\n\nfrom sklearn.model_selection import StratifiedKFold\n\nfrom os import listdir\n\nfrom PIL import Image\nfrom glob import glob\n\nimport torch\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision.models import resnet50, ResNet50_Weights\n\nimport pytorch_lightning as pl\nfrom torch import nn, optim","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-11-23T12:38:48.491750Z","iopub.execute_input":"2023-11-23T12:38:48.491999Z","iopub.status.idle":"2023-11-23T12:38:58.947658Z","shell.execute_reply.started":"2023-11-23T12:38:48.491975Z","shell.execute_reply":"2023-11-23T12:38:58.946775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helper methods <a class=\"anchor\" id=\"helpers\"></a>","metadata":{}},{"cell_type":"markdown","source":"We need the rle_encoding that we can obtain from here:","metadata":{}},{"cell_type":"markdown","source":"https://www.kaggle.com/code/paulorzp/run-length-encode-and-decode/script","metadata":{}},{"cell_type":"code","source":"def rle_encode(img):\n    '''\n    img: numpy array, 1 - mask, 0 - background\n    Returns run length as string formated\n    '''\n    pixels = img.flatten()\n    pixels = np.concatenate([[0], pixels, [0]])\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n    return ' '.join(str(x) for x in runs)\n \ndef rle_decode(mask_rle, shape):\n    '''\n    mask_rle: run-length as string formated (start length)\n    shape: (height,width) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n\n    '''\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape)\n\n\ndef show_kidney_dataset(images, masks, filenames):\n    fig, ax = plt.subplots(len(images), 2 ,figsize=(8, 5*len(images)))\n    \n    for n in range(len(images)):\n        if len(images) == 1:\n            ax[0].imshow(images[n], cmap=\"rainbow\")\n            ax[0].set_title(f\"Image: {filenames[n]}\")\n            ax[1].imshow(masks[n])\n            ax[1].set_title(f\"Mask: {filenames[n]}\")\n        else:    \n            ax[n, 0].imshow(images[n], cmap=\"rainbow\")\n            ax[n, 0].set_title(f\"Image: {filenames[n]}\")\n            ax[n, 1].imshow(masks[n])\n            ax[n, 1].set_title(f\"Mask: {filenames[n]}\")\n    plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-11-23T12:38:58.949450Z","iopub.execute_input":"2023-11-23T12:38:58.949957Z","iopub.status.idle":"2023-11-23T12:38:58.963757Z","shell.execute_reply.started":"2023-11-23T12:38:58.949926Z","shell.execute_reply":"2023-11-23T12:38:58.962757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Exploring the training data <a class=\"anchor\" id=\"eda\"></a>\n\nThe train images consists of TIFF scans from several kidney datasets:\n\n> * **kidney_1_dense** - The whole of a right kidney at 50um resolution. The entire 3D arterial vascular tree has been densely segmented, down to two generations from the glomeruli (i.e. the capillary bed). Uses beamline BM05.\n> * **kidney_1_voi** - A high-resolution subset of kidney_1, at 5.2um resolution.\n> * **kidney_2** - The whole of a kidney from another donor, at 50um resolution. Sparsely segmented (about 65%).\n> * **kidney_3_dense** - A portion (500 slices) of a kidney at 50.16um resolution using BM05. Densely segmented. Note that we provide all of the images for kidney_3 in the kidney_3_sparse/images folder. This dataset accordingly has only a labels folder.\n> * **kidney_3_sparse** - The remainder of the segmentation masks for kidney_3. Sparsely segmented (about 85%).\n\nThat does sound challenging! The datasets vary in resolutions, potentially involve different beamlines, and some may only be partially segmented. Let's delve into the specifics to understand the extent of these differences!\n\n* [Kidney Iterator](#iterator)\n* [Kindey 1 dense](#kidney1dense)\n* [Kidney 1 voi](#kidney1voi)\n* [Kidney 2](#kidney2)\n* [Kidney 3 sparse](#kidney3sparse)\n* [Summary of findings](#eda_summary)","metadata":{}},{"cell_type":"markdown","source":"## Kidney iterator <a class=\"anchor\" id=\"iterator\"></a>\n\nEffectively exploring numerous images without overwhelming the notebook can pose a challenge, especially when comparing multiple datasets. To address this, let's establish a dataset iterator. This will allow us to inspect the data and gain insights into the distinctive characteristics of each dataset:","metadata":{}},{"cell_type":"code","source":"class KidneyIterator:\n    \n    def __init__(self, basepath, batch_size=1):\n        self.basepath = basepath\n        self.batch_size = batch_size\n        self.files = sorted(listdir(f\"{basepath}images/\"))\n        self.length = len(self.files)\n    \n    def get_loader(self):\n        filenames = []\n        images = []\n        masks = []\n        idx = 0\n        \n        for path in self.files:\n            img = Image.open(f\"{self.basepath}images/{path}\")\n            mask = Image.open(f\"{self.basepath}labels/{path}\")\n            \n            img = np.array(img)\n            mask = np.array(mask)\n            \n            images.append(img)\n            masks.append(mask)\n            filenames.append(f\"{path}\")\n            \n            idx += 1\n            if idx == self.batch_size:\n                yield images, masks, filenames\n                filenames = []\n                images = []\n                masks = []\n                idx = 0\n        if idx > 0:\n            yield images, masks, filenames","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2023-11-23T12:38:58.964936Z","iopub.execute_input":"2023-11-23T12:38:58.965332Z","iopub.status.idle":"2023-11-23T12:38:58.977749Z","shell.execute_reply.started":"2023-11-23T12:38:58.965294Z","shell.execute_reply":"2023-11-23T12:38:58.976836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By specifying the path, you can now easily browse through the dataset to get an impression. ","metadata":{}},{"cell_type":"markdown","source":"## Kidney 1 dense <a class=\"anchor\" id=\"kidney1dense\"></a>","metadata":{}},{"cell_type":"code","source":"path = \"/kaggle/input/blood-vessel-segmentation/train/kidney_1_dense/\"\nimage_loader = KidneyIterator(path)\niterator = image_loader.get_loader()\nprint(image_loader.length)","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:38:58.980406Z","iopub.execute_input":"2023-11-23T12:38:58.980707Z","iopub.status.idle":"2023-11-23T12:38:59.192418Z","shell.execute_reply.started":"2023-11-23T12:38:58.980665Z","shell.execute_reply":"2023-11-23T12:38:59.191385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Simply click the following lines of code successively multiple times to navigate through the dataset:","metadata":{}},{"cell_type":"code","source":"images, masks, filenames = next(iterator)\nshow_kidney_dataset(images, masks, filenames)\nprint(images[0].shape)\nprint(images[0].min())\nprint(images[0].max())","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:38:59.193590Z","iopub.execute_input":"2023-11-23T12:38:59.193900Z","iopub.status.idle":"2023-11-23T12:38:59.982344Z","shell.execute_reply.started":"2023-11-23T12:38:59.193875Z","shell.execute_reply":"2023-11-23T12:38:59.981356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* At least initially, many masks are empty, and the corresponding images feature circular-shaped, flocculent background signals.\n* The image signals appear to be unnormalized. The relationship between background and tissue signals varies from image to image, occasionally exhibiting abrupt fluctuations, such as a very high tissue intensity in a few images.\n* It takes approximately 80 images before the first signals emerge in the mask, and they are exceptionally small.","metadata":{}},{"cell_type":"markdown","source":"## Kidney 1 voi <a class=\"anchor\" id=\"kidney1voi\"></a>","metadata":{}},{"cell_type":"code","source":"path = \"/kaggle/input/blood-vessel-segmentation/train/kidney_1_voi/\"\nimage_loader = KidneyIterator(path)\niterator = image_loader.get_loader()\nprint(image_loader.length)","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:38:59.983585Z","iopub.execute_input":"2023-11-23T12:38:59.983890Z","iopub.status.idle":"2023-11-23T12:39:00.094077Z","shell.execute_reply.started":"2023-11-23T12:38:59.983864Z","shell.execute_reply":"2023-11-23T12:39:00.093073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"images, masks, filenames = next(iterator)\nshow_kidney_dataset(images, masks, filenames)\nprint(images[0].shape)\nprint(images[0].min())\nprint(images[0].max())","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:00.095113Z","iopub.execute_input":"2023-11-23T12:39:00.095385Z","iopub.status.idle":"2023-11-23T12:39:01.094528Z","shell.execute_reply.started":"2023-11-23T12:39:00.095361Z","shell.execute_reply":"2023-11-23T12:39:01.093600Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* This dataset is a **high resolution subset of the Kidney1 dense** dataset. \n* The shape of masks and images differ from the dense dataset of Kidney1.\n* The range of values is sometimes larger as in the Kindey1 dense dataset. Again, we can observe fluctuations in the min/max intensity range. \n* If present - we can clearly see larger segments of the vasculature in the masks.\n* I'm not sure yet, but it looks like the order that we have seen in the Kidney1 dense dataset is destroyed. Do we still browse through slices in z-direction?\n* The background signals are less floculent and one can see something like a single surrounding circle. And the boundary of this circle we can sometimes explore artifacts. (For example tif 0026 at the bottom right)","metadata":{}},{"cell_type":"markdown","source":"## Kidney 2 <a class=\"anchor\" id=\"kidney2\"></a>\n\nFor this kidney dataset the beamline was not mentioned in the dataset description. Can we be sure that the same was used? I don't think so. The resolution is similar to the Kidney1 dense dataset but the kidney originates from another donor.","metadata":{}},{"cell_type":"code","source":"path = \"/kaggle/input/blood-vessel-segmentation/train/kidney_2/\"\nimage_loader = KidneyIterator(path, batch_size=1)\niterator = image_loader.get_loader()\nprint(image_loader.length)","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:01.095544Z","iopub.execute_input":"2023-11-23T12:39:01.095871Z","iopub.status.idle":"2023-11-23T12:39:01.201424Z","shell.execute_reply.started":"2023-11-23T12:39:01.095832Z","shell.execute_reply":"2023-11-23T12:39:01.200480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"images, masks, filenames = next(iterator)\nshow_kidney_dataset(images, masks, filenames)\nprint(images[0].shape)\nprint(images[0].min())\nprint(images[0].max())","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:01.202665Z","iopub.execute_input":"2023-11-23T12:39:01.203029Z","iopub.status.idle":"2023-11-23T12:39:01.872577Z","shell.execute_reply.started":"2023-11-23T12:39:01.202996Z","shell.execute_reply":"2023-11-23T12:39:01.871748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* The image shape differs again from the other datasets.\n* The value range of the images is significantly larger than for Kidney 1 dense. Is this a sign for a different beamline?\n* The background is again floculent and the \"surrounding circle\" is larger compared to Kidney 1 dense.\n* Large intensity fluctulations between images are present as well. ","metadata":{}},{"cell_type":"markdown","source":"## Kidney 3 - sparse <a class=\"anchor\" id=\"kidney3sparse\"></a>","metadata":{}},{"cell_type":"code","source":"path = \"/kaggle/input/blood-vessel-segmentation/train/kidney_3_sparse/\"\nimage_loader = KidneyIterator(path, batch_size=1)\niterator = image_loader.get_loader()\nprint(image_loader.length)","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:01.877149Z","iopub.execute_input":"2023-11-23T12:39:01.877485Z","iopub.status.idle":"2023-11-23T12:39:02.067291Z","shell.execute_reply.started":"2023-11-23T12:39:01.877457Z","shell.execute_reply":"2023-11-23T12:39:02.066315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"images, masks, filenames = next(iterator)\nshow_kidney_dataset(images, masks, filenames)\nprint(images[0].shape)\nprint(images[0].min())\nprint(images[0].max())","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:02.068384Z","iopub.execute_input":"2023-11-23T12:39:02.068686Z","iopub.status.idle":"2023-11-23T12:39:02.931554Z","shell.execute_reply.started":"2023-11-23T12:39:02.068646Z","shell.execute_reply":"2023-11-23T12:39:02.930683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* A dataset of similar resolution and intensity values range like Kidney1 dense with beamline BM05 \n* The background looks less floculent compared to Kindey1 dense and the like in the Kidney 1 voi dataset we can observe artifacts at the boundary of the \"surrounding circle\".","metadata":{}},{"cell_type":"markdown","source":"## Test images\n\n> Contains the TIFF scans for the test set. These scans may or may not use a different beamline or resolution from the scans used in the training set. The names of the datasets are kidney_5 and kidney_6.","metadata":{}},{"cell_type":"markdown","source":"## Summary of findings <a class=\"anchor\" id=\"eda_summary\"></a>\n\nThis competition data is difficult in several aspects:\n* The datasets differ in resolution (Kidney1 voi vs. the others) and probably in the used beamline (maybe Kidney 2 used a different one).\n* The image shapes differ and sometimes also the way background signals are present or surrounded. \n* Furthermore the datasets are of different size. \n* We are asked to make predictions for test data that was generated with probably different beamlines or resolutions we have seen so far. \n    \nIn my opinion this competition will be challenging in finding a good validation strategy. ","metadata":{}},{"cell_type":"markdown","source":"# Ideas for validation <a class=\"anchor\" id=\"validation\"></a>\n\nMaybe it's easier to start simple and to focus on the dense datasets (without kidney_1_voi). This way we can try to find out whether our model is able to generalize well to images all close to a resolution of 50um.","metadata":{}},{"cell_type":"code","source":"files = sorted(glob(f\"/kaggle/input/blood-vessel-segmentation/train/kidney_1_dense/images/*.tif\"))\nmasks = sorted(glob(f\"/kaggle/input/blood-vessel-segmentation/train/kidney_1_dense/labels/*.tif\"))\ndf = pd.DataFrame(files, columns=[\"img_path\"])\ndf.loc[:, \"mask_path\"] = masks\ndf.loc[:, \"dataset\"] = \"kidney1_dense\"\n\ntrain = df.copy(deep=True)","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:02.932842Z","iopub.execute_input":"2023-11-23T12:39:02.933191Z","iopub.status.idle":"2023-11-23T12:39:03.189843Z","shell.execute_reply.started":"2023-11-23T12:39:02.933163Z","shell.execute_reply":"2023-11-23T12:39:03.188738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for dataset in [\"kidney_2\", \"kidney_3_sparse\"]:\n    files = sorted(glob(f\"/kaggle/input/blood-vessel-segmentation/train/{dataset}/images/*.tif\"))\n    masks = sorted(glob(f\"/kaggle/input/blood-vessel-segmentation/train/{dataset}/labels/*.tif\"))\n    df = pd.DataFrame(files, columns=[\"img_path\"])\n    df.loc[:, \"mask_path\"] = masks\n    df.loc[:, \"dataset\"] = dataset\n    \n    train = pd.concat([train, df], axis=0)","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.191393Z","iopub.execute_input":"2023-11-23T12:39:03.192114Z","iopub.status.idle":"2023-11-23T12:39:03.693457Z","shell.execute_reply.started":"2023-11-23T12:39:03.192079Z","shell.execute_reply":"2023-11-23T12:39:03.692445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape[0]","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.694555Z","iopub.execute_input":"2023-11-23T12:39:03.694843Z","iopub.status.idle":"2023-11-23T12:39:03.700961Z","shell.execute_reply.started":"2023-11-23T12:39:03.694818Z","shell.execute_reply":"2023-11-23T12:39:03.700010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.dataset.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.702189Z","iopub.execute_input":"2023-11-23T12:39:03.702507Z","iopub.status.idle":"2023-11-23T12:39:03.719683Z","shell.execute_reply.started":"2023-11-23T12:39:03.702484Z","shell.execute_reply":"2023-11-23T12:39:03.718782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To begin experimenting, I would likely employ the kidney2 dataset for validation, as it appears to originate from a different beamline. By selecting this dataset, one might discover strategies to adapt to datasets from as-yet-unknown beamlines. Alternatively, one could consider utilizing each of these kidneys once as a validation fold, implementing a 3-fold grouped KFold validation strategy.","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\nencoder = LabelEncoder()\ntrain[\"fold\"] = encoder.fit_transform(train.dataset.values)\ntrain.fold.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.720910Z","iopub.execute_input":"2023-11-23T12:39:03.721314Z","iopub.status.idle":"2023-11-23T12:39:03.733900Z","shell.execute_reply.started":"2023-11-23T12:39:03.721283Z","shell.execute_reply":"2023-11-23T12:39:03.733002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.reset_index(drop=True)\ntrain.to_csv(\"train.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.734982Z","iopub.execute_input":"2023-11-23T12:39:03.735289Z","iopub.status.idle":"2023-11-23T12:39:03.794878Z","shell.execute_reply.started":"2023-11-23T12:39:03.735251Z","shell.execute_reply":"2023-11-23T12:39:03.794132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Setting up a baseline <a class=\"anchor\" id=\"baseline\"></a>","metadata":{}},{"cell_type":"markdown","source":"## Dataset","metadata":{}},{"cell_type":"code","source":"class MyDataset(Dataset):\n    \n    def __init__(self, df, transform=None, mask_transform=None):\n        self.df = df\n        self.transform = transform\n        self.mask_transform = mask_transform\n    \n    def __len__(self):\n        return self.df.shape[0]\n    \n    def __getitem__(self, idx):\n        image = cv2.imread(self.df.loc[idx, \"img_path\"], cv2.IMREAD_GRAYSCALE)\n        mask = cv2.imread(self.df.loc[idx, \"mask_path\"], cv2.IMREAD_UNCHANGED)\n        \n        if self.transform:\n            augmented = self.transform(image=image)\n            image = augmented[\"image\"]\n            \n        if self.mask_transform:  \n            augmented = self.transform(image=mask)\n            mask = augmented[\"image\"]\n            \n        return image, mask","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.796099Z","iopub.execute_input":"2023-11-23T12:39:03.796743Z","iopub.status.idle":"2023-11-23T12:39:03.804829Z","shell.execute_reply.started":"2023-11-23T12:39:03.796679Z","shell.execute_reply":"2023-11-23T12:39:03.803741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.groupby(\"fold\").dataset.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.805976Z","iopub.execute_input":"2023-11-23T12:39:03.806308Z","iopub.status.idle":"2023-11-23T12:39:03.832890Z","shell.execute_reply.started":"2023-11-23T12:39:03.806281Z","shell.execute_reply":"2023-11-23T12:39:03.832088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataset = MyDataset(train[train.fold!=1])","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.834314Z","iopub.execute_input":"2023-11-23T12:39:03.834786Z","iopub.status.idle":"2023-11-23T12:39:03.839902Z","shell.execute_reply.started":"2023-11-23T12:39:03.834755Z","shell.execute_reply":"2023-11-23T12:39:03.839032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image, mask = train_dataset.__getitem__(0)\nprint(image.shape)\nprint(mask.shape)","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.841138Z","iopub.execute_input":"2023-11-23T12:39:03.841481Z","iopub.status.idle":"2023-11-23T12:39:03.874803Z","shell.execute_reply.started":"2023-11-23T12:39:03.841447Z","shell.execute_reply":"2023-11-23T12:39:03.873940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Searching for good augmentations\n\n* We already know that the image shape depends on the dataset. We would need to adjust the image size to a common shape for processing the images in batches. Unfortunately, just resizing the image can be a source of bias: By resizing we add distortion to the image and the way this distortion happens depends on the image shape of the individual datasets. As a consequence we would have a specific kind of distortion for each dataset. \n* We could try to break that by using GridTransform or ElasticTransform. \n* Another idea to solve this issue could be to use random crops close to the center of the image. \n\nInstead of guessing what could be the best augmentation to remove this potential bias, let's play around with different augmentations ans strategies by predicting the dataset. That sounds a bit crazy, but let's see what happens! :-)","metadata":{}},{"cell_type":"markdown","source":"### Removing the dataset dependency\n\n* For this experimental part we need a model to predict the dataset and we need a dataset that returns images and the source of the dataset as class label\n* For this task we have 3 classes (I dropped kidney_1_voi for simplicity)\n* Furthermore we need a set of augmentations we want to play with\n* Besides that we need a validation split with all datasets being present in both - the validation and training datasets","metadata":{}},{"cell_type":"code","source":"class DatasetPredictorDataset(Dataset):\n    \n    def __init__(self, df, transform=None):\n        self.df = df\n        self.transform = transform\n    \n    def __len__(self):\n        return self.df.shape[0]\n    \n    def __getitem__(self, item):\n        idx = self.df.index.values[item]\n        image = cv2.imread(self.df.loc[idx, \"img_path\"], cv2.IMREAD_GRAYSCALE)\n        label = self.df.loc[idx, \"fold\"] # in our case fold equals the dataset \n        # as we wanted to split datasets into different folds later when training a segmentation model\n        \n        if self.transform:\n            augmented = self.transform(image=image)\n            image = augmented[\"image\"]\n        \n        return image.to(torch.float32), label","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.875805Z","iopub.execute_input":"2023-11-23T12:39:03.876094Z","iopub.status.idle":"2023-11-23T12:39:03.882958Z","shell.execute_reply.started":"2023-11-23T12:39:03.876065Z","shell.execute_reply":"2023-11-23T12:39:03.881944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class DatasetPredictor(pl.LightningModule):\n\n    def __init__(self, backbone, lr, num_classes):\n        super().__init__()\n        self.backbone = backbone\n        self.loss_module = nn.functional.cross_entropy\n        self.lr = lr\n        self.num_classes = num_classes\n        \n        self.features = nn.Sequential(*list(backbone.children())[1:-1])\n        self.features[0] = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n        \n        num_filters = self.backbone.fc.in_features\n        self.classifier = nn.Linear(num_filters, self.num_classes)\n        self.all_preds = []\n        \n    def forward(self, x):\n        x = self.features(x)\n        x = x.squeeze(-1).squeeze(-1)\n        logits = self.classifier(x)\n        return logits\n        \n    def training_step(self, batch, batch_idx):\n        x, y = batch\n        logits = self(x)\n        loss = self.loss_module(input=logits, target=y)\n        self.log(\"train/loss\", loss, prog_bar=True, on_step=True, on_epoch=True)\n        return {\"loss\": loss}\n    \n    def validation_step(self, batch, batch_idx):\n        x, y = batch\n        logits = self(x)\n        loss = self.loss_module(logits, y)\n        \n        # Calculate accuracy\n        probabilities = torch.nn.functional.softmax(logits, dim=1)\n        _, predicted = torch.max(probabilities, 1)\n        accuracy = torch.sum(predicted == y).item() / y.size(0)\n        \n        self.log(\"val/loss\", loss, prog_bar=True, on_step=False, on_epoch=True)\n        self.log('val_accuracy', accuracy, prog_bar=True)\n        return {\"loss\": loss}\n    \n    def predict_step(self, x):\n        logits = self(x)\n        probabilities = torch.nn.functional.softmax(logits, dim=1)\n        self.all_preds.extend(list(torch.argmax(probabilities,1).cpu().detach().numpy()))\n        \n    def configure_optimizers(self):\n        optimizer = optim.AdamW(self.parameters(), lr=self.lr)\n        return optimizer\n    ","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.884194Z","iopub.execute_input":"2023-11-23T12:39:03.884478Z","iopub.status.idle":"2023-11-23T12:39:03.899456Z","shell.execute_reply.started":"2023-11-23T12:39:03.884455Z","shell.execute_reply":"2023-11-23T12:39:03.898525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_exp1():\n    \n    transform ={\n        \"train\": A.Compose([\n            A.Resize(324, 324),\n            ToTensorV2()\n        ]),\n        \"val\": A.Compose([\n            A.Resize(324, 324),\n            ToTensorV2()\n        ])\n    }\n    \n    return transform\n","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.900613Z","iopub.execute_input":"2023-11-23T12:39:03.901367Z","iopub.status.idle":"2023-11-23T12:39:03.911640Z","shell.execute_reply.started":"2023-11-23T12:39:03.901342Z","shell.execute_reply":"2023-11-23T12:39:03.910887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kf = StratifiedKFold(n_splits=3)\ntrain[\"aug_fold\"] = None\nfor fold, (train_idx, val_idx) in enumerate(kf.split(train.index.values, y=train.dataset.values)):\n    train.loc[val_idx, \"aug_fold\"] = fold","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.912729Z","iopub.execute_input":"2023-11-23T12:39:03.913256Z","iopub.status.idle":"2023-11-23T12:39:03.933146Z","shell.execute_reply.started":"2023-11-23T12:39:03.913231Z","shell.execute_reply":"2023-11-23T12:39:03.932340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.groupby(\"aug_fold\").dataset.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.934142Z","iopub.execute_input":"2023-11-23T12:39:03.934429Z","iopub.status.idle":"2023-11-23T12:39:03.948747Z","shell.execute_reply.started":"2023-11-23T12:39:03.934398Z","shell.execute_reply":"2023-11-23T12:39:03.947487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = train[train.aug_fold != 0].copy()\nval_df = train[train.aug_fold == 0].copy()","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.949751Z","iopub.execute_input":"2023-11-23T12:39:03.950009Z","iopub.status.idle":"2023-11-23T12:39:03.961452Z","shell.execute_reply.started":"2023-11-23T12:39:03.949987Z","shell.execute_reply":"2023-11-23T12:39:03.960124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"backbone = resnet50(weights=ResNet50_Weights.DEFAULT)\nmodel = DatasetPredictor(backbone=backbone, lr=1e-6, num_classes=train.dataset.nunique())\n\ntrain_dataset = DatasetPredictorDataset(train_df, transform=get_exp1()['train'])\nval_dataset = DatasetPredictorDataset(val_df, transform=get_exp1()['val'])\n\ntrain_loader = DataLoader(\n    train_dataset,\n    shuffle=True,\n    drop_last=True,\n    batch_size=16,\n    num_workers=3\n)\nval_loader = DataLoader(\n    val_dataset,\n    batch_size=16,\n    num_workers=3\n)","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:03.966715Z","iopub.execute_input":"2023-11-23T12:39:03.967084Z","iopub.status.idle":"2023-11-23T12:39:04.997003Z","shell.execute_reply.started":"2023-11-23T12:39:03.967058Z","shell.execute_reply":"2023-11-23T12:39:04.996087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trainer = pl.Trainer(\n    max_epochs=1\n)\n\ntrainer.fit(\n    model=model,\n    train_dataloaders=train_loader,\n    val_dataloaders=val_loader,\n)","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:39:04.998276Z","iopub.execute_input":"2023-11-23T12:39:04.998613Z","iopub.status.idle":"2023-11-23T12:42:37.884798Z","shell.execute_reply.started":"2023-11-23T12:39:04.998582Z","shell.execute_reply":"2023-11-23T12:42:37.883849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class DatasetPredictor_PredictDataset(Dataset):\n    \n    def __init__(self, df, transform=None):\n        self.df = df\n        self.transform = transform\n    \n    def __len__(self):\n        return self.df.shape[0]\n    \n    def __getitem__(self, item):\n        idx = self.df.index.values[item]\n        image = cv2.imread(self.df.loc[idx, \"img_path\"], cv2.IMREAD_GRAYSCALE)\n        \n        if self.transform:\n            augmented = self.transform(image=image)\n            image = augmented[\"image\"]\n        \n        return image.to(torch.float32)","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:42:37.886285Z","iopub.execute_input":"2023-11-23T12:42:37.886588Z","iopub.status.idle":"2023-11-23T12:42:37.894887Z","shell.execute_reply.started":"2023-11-23T12:42:37.886558Z","shell.execute_reply":"2023-11-23T12:42:37.893993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predict_dataset = DatasetPredictor_PredictDataset(val_df, transform=get_exp1()['val'])\npred_dataloader = DataLoader(predict_dataset, batch_size=16)\nmodel.eval()\n\n_ = trainer.predict(model, pred_dataloader)\npreds = model.all_preds","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:42:37.896492Z","iopub.execute_input":"2023-11-23T12:42:37.896845Z","iopub.status.idle":"2023-11-23T12:43:01.504876Z","shell.execute_reply.started":"2023-11-23T12:42:37.896810Z","shell.execute_reply":"2023-11-23T12:43:01.503920Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_df[\"aug_pred\"] = preds\nval_df.groupby(\"dataset\").aug_pred.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-11-23T12:43:01.506386Z","iopub.execute_input":"2023-11-23T12:43:01.507396Z","iopub.status.idle":"2023-11-23T12:43:01.531016Z","shell.execute_reply.started":"2023-11-23T12:43:01.507359Z","shell.execute_reply":"2023-11-23T12:43:01.530109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* Indeed, we can easily predict the dataset source with Resize being our only augmentation so far. \n* It will be fun to explore which augmentation technique might help to get rid of this dataset source dependency.","metadata":{}}]}