{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Overview"},{"metadata":{},"cell_type":"markdown","source":"I have tried to make this EDA as concise and to the point as possible. Instead of making the notebook lengthy and wasting the readers time, I have included only the important points.\n\nFor more exploration and understanding of the data, I recommend the readers to copy and edit this notebook. As there is no better exploration available that what is done yourself ;)"},{"metadata":{},"cell_type":"markdown","source":"# Please upvote the notebook if you find the content useful. This will motivate me to create more of these content :)"},{"metadata":{},"cell_type":"markdown","source":"# Before proceeding with the EDA, I will advice the readers to get a little bit of domain knowledge.\n\nFollow the link for a concised and consolidated domain knowledge of this competition\n\nhttps://www.kaggle.com/prvnkmr/better-understanding-of-the-problem-statement"},{"metadata":{},"cell_type":"markdown","source":"# Libraries import"},{"metadata":{"trusted":true},"cell_type":"code","source":"import PIL\nimport gc\nimport os\nimport random\nimport tifffile\nimport cv2\nimport json\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nPATH = \"../input/hubmap-kidney-segmentation/\"\nCFG = {\n        'PATH' : \"../input/hubmap-kidney-segmentation/\",\n        'PATH_TRAIN' : PATH + \"train/\",\n        'PATH_TEST' : PATH + \"test/\",\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Utility Functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"os.makedirs('../output')\noutput_dir = '../output'\n\ndef resize_im(im_name, scale_percent):\n    \n    image_path = os.path.join(CFG['PATH_TRAIN'], im_name+'.tiff')\n    im_read = tifffile.imread(image_path)\n    width = int(im_read.shape[1] * scale_percent / 100)\n    height = int(im_read.shape[0] * scale_percent / 100)\n    dim = (width, height)\n    print('File name: {}, original size: {}, resized to: {}'.format(im_name, \n                                                                    (im_read.shape[0], im_read.shape[1]), \n                                                                    (width, height)))\n    resized = cv2.resize(im_read, dim, interpolation=cv2.INTER_AREA)\n    image_path = os.path.join(output_dir, ('r_' + im_name + '.tiff'))\n    tifffile.imwrite(image_path, resized)\n\ndef rle2mask(mask_rle, shape):\n    \n    '''\n    mask_rle: run-length as string formated (start length)\n    shape: (width,height) of array to return\n    Returns numpy array, 1 - mask, 0 - background\n    '''\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape).T\n\ndef resize_mask(im_name, scale_percent):\n    \n    im_read = tifffile.imread(os.path.join(CFG['PATH_TRAIN'], im_name+'.tiff'))\n    mask_rle = df_train[df_train[\"id\"] == im_name][\"encoding\"].values[0]\n    mask = rle2mask(df_train[df_train[\"id\"] == im_name][\"encoding\"].values[0], (im_read.shape[1], im_read.shape[0]))*255\n    width = int(im_read.shape[1] * scale_percent / 100)\n    height = int(im_read.shape[0] * scale_percent / 100)\n    dim = (width, height)\n    print('File name: {}, original size: {}, resized to: {}'.format(im_name, \n                                                                (im_read.shape[0], im_read.shape[1]), \n                                                                (width, height)))\n    resized = cv2.resize(mask, dim, interpolation=cv2.INTER_AREA)\n    image_path = os.path.join(output_dir, ('r_' + im_name+'_m.tiff'))\n    tifffile.imwrite(image_path, resized)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train.csv\n\nAs we know, it contains the training data for the segmentation models.\nColumn info below:"},{"metadata":{},"cell_type":"markdown","source":"1. **id**       - id of each image\n2. **encoding** - RLE encoded segmentation masks"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(CFG['PATH'] + 'train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(len(df_train)):\n\n    print(len(df_train['encoding'][i]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Important points about train.csv:\n\n1. 8 images in training data\n2. RLE Encoding length of each image is huge but very small as compared to the total pixel size of the images. Which is why instead of segmented image, only RLE encoding of the segmentation masks are to be returned."},{"metadata":{},"cell_type":"markdown","source":"# HuBMAP-20-dataset_information.csv\n\nColumns of csv are as follows:\n\n**image_file** - name of image file in .tiff format\n\n**width_pixels** - image pixel width\n\n**height_pixels** - image pixel height\n\n**anatomical_structures_segmention_file** - name of .json file, storing segments(polygons) of kidney parts(cortex/medulla)\n\n**glomerulus_segmentation_file** - name of .json file, storing segments(polygons) of glomerulus cells\n\n**patient_number** - patient number\n\n**race** - race of patient\n\n**sex** - patient gender\n\n**ethnicity** ethnicity of patient\n\n**age** - patient age\n\n**weight_kilograms** - weight of patient in kg\n\n**height_centimeters** - height of patient in cm\n\n**bmi_kg/m^2** - body mass index(weight_kilograms / height_centimeters^2)\n\n**laterality** - laterality of kidney(left / right)\n\n**percent_cortex** percent of cortex(outer part of the kidney)\n\n**percent_medulla** percent of medulla(inner part of the kidney)"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_info = pd.read_csv(PATH + 'HuBMAP-20-dataset_information.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_info.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**We will only require the column no 0 to 4 for our computations. Though I am not saying that the rest of the columns are entirely useless**"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_info.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(df_info)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Imp points about HuBMAP-20-dataset_information.csv\n\n* 13 records in the file : 8 for training and 5 for testing\n* Contains the names of the images and the pixel sizes\n* Contains polygon co-ordinates of glomerulus cells and cortex"},{"metadata":{},"cell_type":"markdown","source":"# Visualizing the images"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_info[['image_file', 'width_pixels', 'height_pixels']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Selecting the images to be visualized**"},{"metadata":{"trusted":true},"cell_type":"code","source":"im_list = [\n    df_train['id'][0],\n    df_train['id'][1],\n    df_train['id'][2],\n    df_train['id'][3],\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"im_list","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Resizing of the mask and the images to make the operations a bit more faster**"},{"metadata":{"trusted":true},"cell_type":"code","source":"for im in im_list:\n    resize_im(im, 5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for im in im_list:\n    resize_mask(im, 5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Resized Images:**"},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(output_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Function used for visualization of the images**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_image(image_id):\n    \n    fig, ax = plt.subplots(nrows=2, ncols=1, figsize=(16, 32))\n    image_path = os.path.join(output_dir, 'r_{}.tiff'.format(image_id))\n    mask_path = os.path.join(output_dir, 'r_{}_m.tiff'.format(image_id))\n    \n    image = tifffile.imread(image_path)\n    mask = tifffile.imread(mask_path)\n    \n    if len(mask.shape)==2:    \n        hybr = image[:, :, 0] + mask[:, :]/2\n    else:\n        hybr = image[:, :, 0] + mask[:,: , 0]/2\n    \n    ax[0].imshow(image)\n    ax[0].axis('off')\n    ax[0].set_title('Real Image')\n    \n    ax[1].imshow(hybr)\n    ax[1].axis('off')\n    ax[1].set_title('Masks')\n    \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Visualing the images from im_list**"},{"metadata":{"trusted":true},"cell_type":"code","source":"show_image(im_list[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_image(im_list[2])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_image(im_list[1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 'Real image' is the image without the mask on. And when put on the mask, it reveals the glomeruli FTUs inside the cell."},{"metadata":{},"cell_type":"markdown","source":"# Afte this I will advise the readers to play around with the mask. Based on its 'size' and 'number' inside a cell"},{"metadata":{},"cell_type":"markdown","source":"I have taken some help from these two notebooks. Please upvote them as well !!\n\nhttps://www.kaggle.com/yuriikochurovskyi/hubmap-image-eda-step-by-step-beginner-friendly\n\nhttps://www.kaggle.com/kiruganko/hubmap-eda#Images"},{"metadata":{},"cell_type":"markdown","source":"# Please upvote if you liked the content :)"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}