{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Exploratory data analysis\nThis notebook performs basic exploratory analysis and visualization of the HuBMAP competition dataset. The first two sections deal with the training and test dataset. The third section focuses on the analysis of the distribution of patients over the training and test datasets, and its possible effects. This notebook does not include analysis of patient-specific information such as age, ethnicity, etc.\n1. [Training dataset](#section-one)<br>\n    1.1 [Image resolutions](#section-one-one)<br>\n    1.2 [RGB and HSV color spaces](#section-one-two)<br>\n    1.3 [Glomeruli count](#section-one-three)<br>\n    1.4 [Glomeruli size distribution](#section-one-four)<br>\n    1.5 [Anatomical structures](#section-one-five)<br>\n    1.6 [Mask visualization](#section-one-six)<br>\n2. [Test datatset](#section-two)<br>\n    2.1 [Image resolutions](#section-two-one)<br>\n    2.2 [Anatomical structures](#section-two-two)<br>\n    2.3 [Mask visualization](#section-two-three)<br>\n3. [Patient images](#section-three)<br>","metadata":{}},{"cell_type":"code","source":"import cv2\nimport glob\nimport json\nimport numpy as np\nimport os\nimport pandas as pd\nimport tifffile as tiff\nfrom matplotlib import colors\nfrom matplotlib import pyplot as plt\nfrom matplotlib.lines import Line2D\nfrom matplotlib_venn import venn2_unweighted","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:39:39.269745Z","iopub.execute_input":"2022-06-02T07:39:39.270331Z","iopub.status.idle":"2022-06-02T07:39:40.061838Z","shell.execute_reply.started":"2022-06-02T07:39:39.270234Z","shell.execute_reply":"2022-06-02T07:39:40.058614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Get file names of training and test images.","metadata":{}},{"cell_type":"code","source":"train_images = glob.glob('/kaggle/input/hubmap-kidney-segmentation/train/*.tiff')\ntest_images = glob.glob('/kaggle/input/hubmap-kidney-segmentation/test/*.tiff')\n\ntrain_images = list(map(lambda x: os.path.basename(x), train_images))\ntest_images = list(map(lambda x: os.path.basename(x), test_images))","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:39:40.064858Z","iopub.execute_input":"2022-06-02T07:39:40.065794Z","iopub.status.idle":"2022-06-02T07:39:40.100808Z","shell.execute_reply.started":"2022-06-02T07:39:40.065711Z","shell.execute_reply":"2022-06-02T07:39:40.099884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Read dataset information file which contains general information about images and patients, calculate the total number of pixels for each image and store it in a new column `pixels_total`.","metadata":{}},{"cell_type":"code","source":"pd.set_option('display.max_colwidth', None)\ndataset_information = pd.read_csv('/kaggle/input/hubmap-kidney-segmentation/HuBMAP-20-dataset_information.csv')\ndataset_information['pixels_total'] = dataset_information.width_pixels * dataset_information.height_pixels","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:39:40.102428Z","iopub.execute_input":"2022-06-02T07:39:40.103063Z","iopub.status.idle":"2022-06-02T07:39:40.150398Z","shell.execute_reply.started":"2022-06-02T07:39:40.103015Z","shell.execute_reply":"2022-06-02T07:39:40.149273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Split the dataset information file into training and test partitions.","metadata":{}},{"cell_type":"code","source":"train_dataset_information = dataset_information[dataset_information['image_file'].isin(train_images)].reset_index(drop=True)\ntest_dataset_information = dataset_information[dataset_information['image_file'].isin(test_images)].reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:39:40.152096Z","iopub.execute_input":"2022-06-02T07:39:40.152579Z","iopub.status.idle":"2022-06-02T07:39:40.165996Z","shell.execute_reply.started":"2022-06-02T07:39:40.152536Z","shell.execute_reply":"2022-06-02T07:39:40.165060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Training dataset<a id=\"section-one\"></a>\nThe following section deals exclusively with data from the __training__ dataset. The training dataset is comprised of the following data:\n- 15 images in TIFF format\n- 15 glomeruli masks in JSON format (alternatively `train.csv` file with RLE-encoded masks)\n- 15 anatomical masks in JSON format\n\n## 1.1 Image resolutions<a id=\"section-one-one\"></a>\nThis table shows image resolutions in descending order. As can be seen in the table, the images are quite large. ","metadata":{}},{"cell_type":"code","source":"train_dataset_information.sort_values('pixels_total', ascending=False)[['image_file', 'width_pixels','height_pixels']].reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:39:40.171073Z","iopub.execute_input":"2022-06-02T07:39:40.171591Z","iopub.status.idle":"2022-06-02T07:39:40.201103Z","shell.execute_reply.started":"2022-06-02T07:39:40.171530Z","shell.execute_reply":"2022-06-02T07:39:40.199826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Image resolutions as a bar chart.","metadata":{}},{"cell_type":"code","source":"train_dataset_information.plot.bar(x='image_file', y='pixels_total', rot=90)","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:39:40.204349Z","iopub.execute_input":"2022-06-02T07:39:40.204725Z","iopub.status.idle":"2022-06-02T07:39:40.560710Z","shell.execute_reply.started":"2022-06-02T07:39:40.204692Z","shell.execute_reply":"2022-06-02T07:39:40.559697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.2 RGB and HSV color spaces<a id=\"section-one-two\"></a>\nThe following section shows RGB and HSV color spaces of `aaa6a05cc.tiff`. HSV color space is often used in segmentation tasks and can be useful to separate tissue pixels from background pixels.","metadata":{}},{"cell_type":"code","source":"# open and resize image\nimage = cv2.imread('/kaggle/input/hubmap-kidney-segmentation/train/aaa6a05cc.tiff')\nimage_resize = cv2.resize(image,(image.shape[1]//10,image.shape[0]//10), interpolation = cv2.INTER_CUBIC)","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:39:40.562244Z","iopub.execute_input":"2022-06-02T07:39:40.562557Z","iopub.status.idle":"2022-06-02T07:39:45.655296Z","shell.execute_reply.started":"2022-06-02T07:39:40.562530Z","shell.execute_reply":"2022-06-02T07:39:45.654159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"RGB scatter plot of `aaa6a05cc.tiff`.","metadata":{}},{"cell_type":"code","source":"# calculate colors\npixel_colors = image_resize.reshape((np.shape(image_resize)[0]*np.shape(image_resize)[1], 3))\nnorm = colors.Normalize(vmin=-1.,vmax=1.)\nnorm.autoscale(pixel_colors)\npixel_colors = norm(pixel_colors).tolist()\n\n# split channels\nb, g, r = cv2.split(image_resize)\n\n# scatter plot\nfig = plt.figure()\naxis = fig.add_subplot(1, 1, 1, projection='3d')\naxis.scatter(r.flatten(), g.flatten(), b.flatten(), facecolors=pixel_colors, marker='.')\naxis.set_xlabel('Red')\naxis.set_ylabel('Green')\naxis.set_zlabel('Blue')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:39:45.656707Z","iopub.execute_input":"2022-06-02T07:39:45.657298Z","iopub.status.idle":"2022-06-02T07:40:33.290409Z","shell.execute_reply.started":"2022-06-02T07:39:45.657206Z","shell.execute_reply":"2022-06-02T07:40:33.289056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"HSV scatter plot of `aaa6a05cc.tiff`.","metadata":{}},{"cell_type":"code","source":"# convert to hsv\nhsv_image = cv2.cvtColor(image_resize, cv2.COLOR_BGR2HSV)\nh, s, v = cv2.split(hsv_image)\n\n# scatter plot\nfig = plt.figure()\naxis = fig.add_subplot(1, 1, 1, projection='3d')\naxis.scatter(s.flatten(), h.flatten(), v.flatten(), facecolors=pixel_colors, marker='.')\naxis.set_xlabel('Saturation')\naxis.set_ylabel('Hue')\naxis.set_zlabel('Value')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:40:33.292006Z","iopub.execute_input":"2022-06-02T07:40:33.292617Z","iopub.status.idle":"2022-06-02T07:41:17.965694Z","shell.execute_reply.started":"2022-06-02T07:40:33.292554Z","shell.execute_reply":"2022-06-02T07:41:17.964498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.3 Glomeruli count<a id=\"section-one-three\"></a>\nThis section provides information about the number of glomeruli masks per image in descending order.","metadata":{}},{"cell_type":"code","source":"train_glom_seg_files = train_dataset_information['glomerulus_segmentation_file'].to_list()\ntrain_glomeruli_dict = {}\n\nfor file_name in train_glom_seg_files:\n    file_id = file_name[:9]\n    with open(f'/kaggle/input/hubmap-kidney-segmentation/train/{file_name}') as json_file:\n        data = json.load(json_file)\n        train_glomeruli_dict[file_id] = 0\n        for entry in data:\n            if entry['type'] == 'Feature' and entry['properties']['classification']['name'] == 'glomerulus':\n                    train_glomeruli_dict[file_id] += 1\n            else:\n                raise Exception(f\"Unexpected json format: {entry['type']}, {entry['properties']['classification']['name']}\")\n                \ntrain_nr_glom = pd.DataFrame(list(train_glomeruli_dict.items()), columns=['file_id', 'nr_glomeruli'])\ntrain_nr_glom.sort_values('nr_glomeruli', ascending=False).reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:41:17.967746Z","iopub.execute_input":"2022-06-02T07:41:17.968272Z","iopub.status.idle":"2022-06-02T07:41:18.893698Z","shell.execute_reply.started":"2022-06-02T07:41:17.968223Z","shell.execute_reply":"2022-06-02T07:41:18.892505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Total number of glomeruli in the training dataset.","metadata":{}},{"cell_type":"code","source":"train_nr_glom['nr_glomeruli'].sum()","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:41:18.895054Z","iopub.execute_input":"2022-06-02T07:41:18.895394Z","iopub.status.idle":"2022-06-02T07:41:18.904782Z","shell.execute_reply.started":"2022-06-02T07:41:18.895362Z","shell.execute_reply":"2022-06-02T07:41:18.903295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.4 Glomeruli size distribution<a id=\"section-one-four\"></a>\nIt is also possible to calculate sizes of glomeruli masks. The following section provides basic information about the distrubution of glomeruli mask sizes.","metadata":{}},{"cell_type":"code","source":"train_glom_seg_files = train_dataset_information['glomerulus_segmentation_file'].to_list()\ntrain_glomeruli_polys_dict = {}\n\nfor file_name in train_glom_seg_files:\n    with open(f'/kaggle/input/hubmap-kidney-segmentation/train/{file_name}') as json_file:\n        data = json.load(json_file)\n        train_glomeruli_polys_dict[file_name] = []\n        for entry in data:\n            if entry['type'] == 'Feature' and entry['properties']['classification']['name'] == 'glomerulus':\n                geom = np.array(entry['geometry']['coordinates']).astype(np.float32)\n                x,y,w,h = cv2.boundingRect(geom.squeeze(axis=0))\n                train_glomeruli_polys_dict[file_name].append((h,w)) # height, width!\n            else:\n                raise Exception(f\"Unexpected json format: {entry['type']}, {entry['properties']['classification']['name']}\")\n                \ntrain_res_glom = pd.DataFrame(list(train_glomeruli_polys_dict.items()), columns=['glomerulus_segmentation_file', 'glomeruli_height_width'])\ntrain_res_glom = train_res_glom.explode('glomeruli_height_width')\ntrain_res_glom['height'], train_res_glom['width'] = zip(*train_res_glom['glomeruli_height_width'])\ntrain_res_glom = train_res_glom.drop(columns=['glomeruli_height_width'])\ntrain_res_glom.describe()","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:41:18.906535Z","iopub.execute_input":"2022-06-02T07:41:18.906977Z","iopub.status.idle":"2022-06-02T07:41:20.069779Z","shell.execute_reply.started":"2022-06-02T07:41:18.906940Z","shell.execute_reply":"2022-06-02T07:41:20.068591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.5 Anatomical structures<a id=\"section-one-five\"></a>\nIn addition to glomeruli masks and images itself, the challenge dataset contains segmentaion masks of some antomical structures that can be used for training. Each image contains a different set of anatomical masks. This table shows all available anatomical masks for each image from the training dataset.\n\n__Note:__ To our knowledge, the private test dataset doesn't provide segmentation masks for anatomical structures. Therefore the use of anatomical structures is limited.","metadata":{}},{"cell_type":"code","source":"def list_structures(seg_files, folder):\n    train_anatomical_dict = {}\n    folder_path = os.path.join('/kaggle/input/hubmap-kidney-segmentation/', folder)\n\n    for file_name in seg_files:\n        file_id = file_name[:9]\n        with open(os.path.join(folder_path, file_name)) as json_file:\n            data = json.load(json_file)\n            train_anatomical_dict[file_id] = []\n            for entry in data:\n                if entry['type'] == 'Feature':\n                    train_anatomical_dict[file_id].append(entry['properties']['classification']['name'])\n                else:\n                    raise Exception(f\"Unexpected json format: {entry['type']}, {entry['properties']['classification']['name']}\")\n\n    return pd.DataFrame(list(train_anatomical_dict.items()), columns=['file_id', 'anatomical_structure'])","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:41:20.072080Z","iopub.execute_input":"2022-06-02T07:41:20.072629Z","iopub.status.idle":"2022-06-02T07:41:20.082703Z","shell.execute_reply.started":"2022-06-02T07:41:20.072560Z","shell.execute_reply":"2022-06-02T07:41:20.081287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_anatomical_seg_files = train_dataset_information['anatomical_structures_segmention_file'].to_list()\ntrain_anatomical = list_structures(train_anatomical_seg_files, 'train')\ntrain_anatomical","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:41:20.084811Z","iopub.execute_input":"2022-06-02T07:41:20.085301Z","iopub.status.idle":"2022-06-02T07:41:20.203616Z","shell.execute_reply.started":"2022-06-02T07:41:20.085246Z","shell.execute_reply":"2022-06-02T07:41:20.202339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"List of all unique anatomical structures that can be found in the training dataset. ","metadata":{}},{"cell_type":"code","source":"pd.DataFrame(train_anatomical.explode('anatomical_structure')['anatomical_structure'].unique(), columns=['unique_structures'])","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:41:20.205764Z","iopub.execute_input":"2022-06-02T07:41:20.206559Z","iopub.status.idle":"2022-06-02T07:41:20.223680Z","shell.execute_reply.started":"2022-06-02T07:41:20.206520Z","shell.execute_reply":"2022-06-02T07:41:20.222457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.6 Mask visualization<a id=\"section-one-six\"></a>\nThis section provides visualization of glomeruli and anatomical masks. The function `make_grid` (borrowed from https://www.kaggle.com/leighplt/pytorch-fcn-resnet50) creates a grid for the sliding window operation.","metadata":{}},{"cell_type":"code","source":"def make_grid(shape, window=256, min_overlap=32):\n    \"\"\"\n    source: https://www.kaggle.com/leighplt/pytorch-fcn-resnet50\n    \n    function to generate a grid layout for sliding window\n    :param shape: height and width of the image\n    :param window: size of the window\n    :param min_overlap: minimal window overlap\n    :return: array of window coordinates (x1,x2,y1,y2)\n    \"\"\"\n    x, y = shape\n    nx = x // (window - min_overlap) + 1\n    x1 = np.linspace(0, x, num=nx, endpoint=False, dtype=np.int64)\n    x1[-1] = x - window\n    x2 = (x1 + window).clip(0, x)\n    ny = y // (window - min_overlap) + 1\n    y1 = np.linspace(0, y, num=ny, endpoint=False, dtype=np.int64)\n    y1[-1] = y - window\n    y2 = (y1 + window).clip(0, y)\n    slices = np.zeros((nx,ny, 4), dtype=np.int64)\n    \n    for i in range(nx):\n        for j in range(ny):\n            slices[i,j] = x1[i], x2[i], y1[j], y2[j]    \n    return slices.reshape(nx*ny,4)","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:41:20.225129Z","iopub.execute_input":"2022-06-02T07:41:20.226080Z","iopub.status.idle":"2022-06-02T07:41:20.236307Z","shell.execute_reply.started":"2022-06-02T07:41:20.225984Z","shell.execute_reply":"2022-06-02T07:41:20.235303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following visualization is inspired by https://www.kaggle.com/mpware/masks-quick-eda-updated-data. The visualization contains anatomical structures and glomeruli which are shown as small circles. Since the provided images are large, it is necessary to split the images into smaller frames using sliding window operation in order to use them as input for a machine learning algorithm. The perpendicular lines represent the way the images will be split into frames. In this example, the frame size is 1024x1024 pixels with 256 pixels overlap. As can be seen in the images, glomeruli are mostly located in the cortex. However, there are also some glomeruli masks outside the cortex.","metadata":{}},{"cell_type":"code","source":"def plot_masks(dataset_information, folder, frame_size, frame_overlap, plot_glom):\n    folder_path = os.path.join(\"/kaggle/input/hubmap-kidney-segmentation\", folder)\n    \n    for i in range(len(dataset_information)):\n\n        # create new figure\n        plt.figure(figsize=(32, 30))\n        obj_line_thickness = 60\n\n        # find metadata row for json file\n        image_metadata = dataset_information.iloc[i]\n\n        # open image file\n        image = tiff.imread(os.path.join(folder_path, image_metadata['image_file']))\n        print(image_metadata['image_file'])\n\n        # reshape image if necessary\n        if len(image.shape) == 5:\n            image = image.squeeze()\n        if image.shape[0] == 3:\n            image = image.transpose(1, 2, 0)\n\n        # create a copy of the image\n        image = image.copy()\n\n        # draw sliding window boxes\n        frame_grid = make_grid((image.shape[1], image.shape[0]), frame_size, frame_overlap)\n\n        for frame in frame_grid:\n            x1, y1 = frame[0], frame[2]\n            x2, y2 = frame[1], frame[3]\n            image = cv2.rectangle(image, (x1, y1), (x2, y2), color=(255,255,255), thickness=16)\n\n        # draw glomeruli polygons\n        if plot_glom:\n            # open glomeruli json file\n            read_glom_seg_file = open(os.path.join(folder_path, image_metadata['glomerulus_segmentation_file']), 'r')\n            glom_seg_data = json.load(read_glom_seg_file)\n            \n            for k in range(len(glom_seg_data)):\n                glom_poly = np.array(glom_seg_data[k]['geometry']['coordinates']).astype(np.int32) # get coordinates of glomeruli\n                cv2.polylines(image, glom_poly, True,(255,0,0), thickness=obj_line_thickness)\n\n        # open anatomical json file\n        read_anatomical_seg_file = open(os.path.join(folder_path, image_metadata['anatomical_structures_segmention_file']), 'r')\n        anatomical_seg_data = json.load(read_anatomical_seg_file)\n\n        # scan anatomical json file and draw lines\n        for n in range(len(anatomical_seg_data)):\n            obj_name = anatomical_seg_data[n]['properties']['classification']['name']\n            obj_coords = anatomical_seg_data[n]['geometry']['coordinates']\n\n            if (obj_name == 'Cortex'): # draw line around cortex\n                cv2.polylines(image, np.expand_dims(np.array(obj_coords[0]).astype(np.int32), axis=0), True, (0,0,255), thickness=obj_line_thickness)\n            elif (obj_name == 'Medulla'): # draw line around medulla\n                cv2.polylines(image, np.array(obj_coords).astype(np.int32), True, (0,255,0), thickness=obj_line_thickness)\n            elif (obj_name == 'Inner medulla'): # draw line around inner medulla\n                cv2.polylines(image, np.array(obj_coords).astype(np.int32), True, (255,255,0), thickness=obj_line_thickness)\n            elif (obj_name == 'Outer Medulla'): # draw line around outer medulla\n                cv2.polylines(image, np.array(obj_coords).astype(np.int32), True, (0,255,255), thickness=obj_line_thickness)\n            elif (obj_name == 'Outer Stripe'): # draw line around outer stripe\n                cv2.polylines(image, np.array(obj_coords).astype(np.int32), True, (255,0,255), thickness=obj_line_thickness)\n            else:\n                raise Exception(f'Unknown anatomical object: {obj_name}')\n\n        # down-scale the image\n        image_resize = cv2.resize(image,(image.shape[1]//10,image.shape[0]//10), interpolation = cv2.INTER_CUBIC)\n\n        # add legend and view the image\n        custom_lines = [Line2D([0], [0], color=(0.,0.,1.), lw=4),\n                    Line2D([0], [0], color=(0.,1.,0.), lw=4),\n                    Line2D([0], [0], color=(1.,1.,0.), lw=4),\n                    Line2D([0], [0], color=(0.,1.,1.), lw=4),\n                    Line2D([0], [0], color=(1.,0.,1.), lw=4),]\n\n        plt.legend(custom_lines, ['Cortex', 'Medulla', 'Inner medulla', 'Outer Medulla', 'Outer Stripe'])\n        plt.axis('off')\n        plt.title(image_metadata['image_file'])\n        plt.imshow(image_resize)\n        plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:41:20.237808Z","iopub.execute_input":"2022-06-02T07:41:20.238431Z","iopub.status.idle":"2022-06-02T07:41:20.265477Z","shell.execute_reply.started":"2022-06-02T07:41:20.238378Z","shell.execute_reply":"2022-06-02T07:41:20.264562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_masks(train_dataset_information, 'train', 1024, 256, True)","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:41:20.267713Z","iopub.execute_input":"2022-06-02T07:41:20.268348Z","iopub.status.idle":"2022-06-02T07:48:45.888457Z","shell.execute_reply.started":"2022-06-02T07:41:20.268300Z","shell.execute_reply":"2022-06-02T07:48:45.886787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Test dataset<a id=\"section-two\"></a>\nThe following section deals exclusively with data from the __test__ dataset. The test dataset is comprised of the following data:\n- 5 images in TIFF format\n- 5 anatomical masks in JSON format\n\n## 2.1 Image resolutions<a id=\"section-two-one\"></a>\nThis table shows image resolutions in descending order.","metadata":{}},{"cell_type":"code","source":"test_dataset_information.sort_values('pixels_total', ascending=False)[['image_file', 'width_pixels','height_pixels']]","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:48:45.890958Z","iopub.execute_input":"2022-06-02T07:48:45.891430Z","iopub.status.idle":"2022-06-02T07:48:45.907225Z","shell.execute_reply.started":"2022-06-02T07:48:45.891394Z","shell.execute_reply":"2022-06-02T07:48:45.905903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Image resolutions as a bar chart.","metadata":{}},{"cell_type":"code","source":"test_dataset_information.plot.bar(x='image_file', y='pixels_total')","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:48:45.909346Z","iopub.execute_input":"2022-06-02T07:48:45.909956Z","iopub.status.idle":"2022-06-02T07:48:46.141648Z","shell.execute_reply.started":"2022-06-02T07:48:45.909885Z","shell.execute_reply":"2022-06-02T07:48:46.140029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2 Anatomical structures<a id=\"section-two-two\"></a>\nThis table shows all available anatomical masks for each image from the test dataset.","metadata":{}},{"cell_type":"code","source":"test_anatomical_seg_files = test_dataset_information['anatomical_structures_segmention_file'].to_list()\ntest_anatomical = list_structures(test_anatomical_seg_files, 'test')","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:48:46.143928Z","iopub.execute_input":"2022-06-02T07:48:46.144305Z","iopub.status.idle":"2022-06-02T07:48:46.179617Z","shell.execute_reply.started":"2022-06-02T07:48:46.144268Z","shell.execute_reply":"2022-06-02T07:48:46.177960Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"List of all unique anatomical structures in the test dataset. ","metadata":{}},{"cell_type":"code","source":"pd.DataFrame(test_anatomical.explode('anatomical_structure')['anatomical_structure'].unique(), columns=['unique_structures'])","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:48:46.181641Z","iopub.execute_input":"2022-06-02T07:48:46.182561Z","iopub.status.idle":"2022-06-02T07:48:46.203343Z","shell.execute_reply.started":"2022-06-02T07:48:46.182523Z","shell.execute_reply":"2022-06-02T07:48:46.202282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.3 Mask visualization<a id=\"section-two-three\"></a>\nAs already mentioned, this visualization is inspired by https://www.kaggle.com/mpware/masks-quick-eda-updated-data. The visualization shows anatomical structures. The perpendicular lines represent the way the images will be split into frames. In this example, the frame size is 1024x1024 pixels with 256 pixels overlap.","metadata":{}},{"cell_type":"code","source":"plot_masks(test_dataset_information, 'test', 1024, 256, False)","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:48:46.205108Z","iopub.execute_input":"2022-06-02T07:48:46.205752Z","iopub.status.idle":"2022-06-02T07:51:13.027184Z","shell.execute_reply.started":"2022-06-02T07:48:46.205701Z","shell.execute_reply":"2022-06-02T07:51:13.026177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Patient images<a id=\"section-three\"></a>\nSome patients from the training dataset have more than one image, as shown in the table below. This must be considered when partitioning training data into training, validation and test sets in order to avoid data leakage.","metadata":{}},{"cell_type":"code","source":"train_dataset_information.groupby('patient_number').agg(list)['image_file'].to_frame()","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:51:13.028489Z","iopub.execute_input":"2022-06-02T07:51:13.029002Z","iopub.status.idle":"2022-06-02T07:51:13.116835Z","shell.execute_reply.started":"2022-06-02T07:51:13.028968Z","shell.execute_reply":"2022-06-02T07:51:13.115524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Patients from the test dataset have only one image each.","metadata":{}},{"cell_type":"code","source":"test_dataset_information.groupby('patient_number').agg(list)['image_file'].to_frame()","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:51:13.118259Z","iopub.execute_input":"2022-06-02T07:51:13.118599Z","iopub.status.idle":"2022-06-02T07:51:13.197945Z","shell.execute_reply.started":"2022-06-02T07:51:13.118566Z","shell.execute_reply":"2022-06-02T07:51:13.196512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Furthermore, there are patients that are both in training __and__ test dataset, as shown below.","metadata":{}},{"cell_type":"code","source":"train_patients = set(train_dataset_information['patient_number'].unique().tolist())\ntest_patients = set(test_dataset_information['patient_number'].unique().tolist())\n\ntrain_patients.intersection(test_patients)","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:51:13.199537Z","iopub.execute_input":"2022-06-02T07:51:13.199987Z","iopub.status.idle":"2022-06-02T07:51:13.209363Z","shell.execute_reply.started":"2022-06-02T07:51:13.199945Z","shell.execute_reply":"2022-06-02T07:51:13.208265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Patients that exclusively belong to the  __training dataset__.","metadata":{}},{"cell_type":"code","source":"train_excl_pat = train_patients - test_patients\ntrain_excl_pat","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:51:13.210893Z","iopub.execute_input":"2022-06-02T07:51:13.211295Z","iopub.status.idle":"2022-06-02T07:51:13.222841Z","shell.execute_reply.started":"2022-06-02T07:51:13.211257Z","shell.execute_reply":"2022-06-02T07:51:13.221677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Patients that exclusively belong to the __test dataset__.","metadata":{}},{"cell_type":"code","source":"test_excl_pat = test_patients - train_patients\ntest_excl_pat","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:51:13.224337Z","iopub.execute_input":"2022-06-02T07:51:13.224736Z","iopub.status.idle":"2022-06-02T07:51:13.234725Z","shell.execute_reply.started":"2022-06-02T07:51:13.224684Z","shell.execute_reply":"2022-06-02T07:51:13.233592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Venn diagram representation of patient distribution over the training and test datasets.","metadata":{}},{"cell_type":"code","source":"venn = venn2_unweighted([train_patients, test_patients], ('Training', 'Test'))\nvenn.get_label_by_id('10').set_text('\\n'.join(sorted(map(str, train_patients - test_patients))))\nvenn.get_label_by_id('11').set_text('\\n'.join(sorted(map(str, train_patients.intersection(test_patients)))))\nvenn.get_label_by_id('01').set_text('\\n'.join(sorted(map(str, test_patients - train_patients))))","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:51:13.236611Z","iopub.execute_input":"2022-06-02T07:51:13.237079Z","iopub.status.idle":"2022-06-02T07:51:13.352639Z","shell.execute_reply.started":"2022-06-02T07:51:13.237036Z","shell.execute_reply":"2022-06-02T07:51:13.351646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As mentioned before, it is necessary to prevent data leakage when splitting the training data into training, validation and test partitions. Therefore the test partition isn't allowed to contain images of patients whose images were used during the training of a machine learning model. This means that only patients from the left side of the Venn diagram can be selected for the test partition. Moreover, due to the limited training dataset, it can be beneficial to select images of different patients for the test partition rather than having multiple images from the same patient. Based on these statements, the requirements for test partition can be defined as follows:\n- Patient belongs exclusively to the training dataset\n- Patient has one image at most\n\nThe following section shows possible candidates for the test partition.","metadata":{}},{"cell_type":"code","source":"patient_images = train_dataset_information.groupby('patient_number').agg(list)['image_file'].to_frame().reset_index()\npatient_images = patient_images[patient_images['patient_number'].isin(train_excl_pat)]\npatient_images = patient_images[patient_images['image_file'].map(len) < 2]\npatient_images = patient_images['image_file'].apply(lambda x: x[0]).to_frame().reset_index(drop=True)\npatient_images","metadata":{"execution":{"iopub.status.busy":"2022-06-02T07:51:13.354147Z","iopub.execute_input":"2022-06-02T07:51:13.354466Z","iopub.status.idle":"2022-06-02T07:51:13.430145Z","shell.execute_reply.started":"2022-06-02T07:51:13.354433Z","shell.execute_reply":"2022-06-02T07:51:13.428920Z"},"trusted":true},"execution_count":null,"outputs":[]}]}