{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#       Human Protein Atlas - Single Cell Classification","metadata":{}},{"cell_type":"markdown","source":"### Introduction:","metadata":{}},{"cell_type":"markdown","source":"    Human body consists of trillions of cells but also not all humans have the same kind of cells. Location of protein is very important in cells and hence dissimilarity in location of protein can breed cellular heterogeneity. For cellular processes/operations protein plays a crucial role, Collection of proteins come together at some discrete location to perform some task and outcome of this task is based on which kind of protein are present. From this different subcellular dispensation of one protein can give rise to great functional differences, finding such differences and figuring out why and how they occur, is important for understanding how cells function, how diseases develop, and ultimately how to develop better treatments for those diseases. \n \n    This is a supervised multi-label classification problem. Given images of cells from the microscopes and given labels of protein location assigned together for all the cells in the image. In this notebook We have developed a model which is capable of segmenting and classifying each individual cell with precise labels.","metadata":{}},{"cell_type":"code","source":"!pip install ../input/hpa-library-install/iterative-stratification-master/iterative-stratification-master\n!pip install ../input/hpa-library-install/pytorch_zoo-master/pytorch_zoo-master\n!pip install ../input/hpa-library-install/HPA-Cell-Segmentation-master/HPA-Cell-Segmentation-master","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# importing required libraries for basic operations\nimport pandas as pd # for dataset processing \nimport numpy as np # for mathemetical processes\nimport pickle # for files read and write operations\nimport os # for system related operations\nimport zipfile # for zip files read and write operations\nfrom tqdm import tqdm # for displaying progress bar\nimport cv2 # for image processing\nfrom PIL import Image # for image processing\n\n# importing required libraries for plotting\nimport matplotlib.pyplot as plt \nimport plotly.graph_objects as go\nimport plotly.express as px\n\n# importing required libraries for model creation \nfrom fastai.vision.all import * # deep learning library\n\nfrom iterstrat.ml_stratifiers import MultilabelStratifiedKFold # library for sampling dataset\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data:","metadata":{}},{"cell_type":"markdown","source":"    We have train data for training and test data for testing our model. In training data we have a directory containing all the images for training purpose and a csv file containing labels for all images.","metadata":{}},{"cell_type":"markdown","source":"### Train Data:","metadata":{}},{"cell_type":"code","source":"# loading train.csv file into a dataframe\ntrain_df = pd.read_csv('../input/hpa-single-cell-image-classification/train.csv')\nprint('(rows, columns)',train_df.shape)\ntrain_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train Data Description","metadata":{}},{"cell_type":"markdown","source":"    There are 2 columns and 21806 records in train.csv file. \n \n1. **ID**:- Contains Image files name \n2. **Label**:- Contains corresponding labels for each image file. There total 19 labels from 0-18. Following are their means","metadata":{}},{"cell_type":"code","source":"pd.read_csv('../input/additional-data/Labels.csv').style.hide_index()","metadata":{"hideCode":true,"hidePrompt":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Images for training","metadata":{}},{"cell_type":"code","source":"len(os.listdir('../input/hpa-single-cell-image-classification/train'))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are total 87224 images are given for training purpose. But notice here that in train.csv files there are only 21806 records instead of 87224. This is becuase the images are provided in 4 different channels, it means there are 4 images belonging to a single image. All images have following 4 channels: \n \n1. Red (Microtubules)\n2. Green (Protein of interest)\n3. Blue (Nucleus)\n4. Yellow (Endoplasmic reticulum)","metadata":{}},{"cell_type":"markdown","source":"According to https://biologydictionary.net/","metadata":{}},{"cell_type":"markdown","source":"**Microtubules**: Microtubules are microscopic hollow tubes made of the proteins alpha and beta tubulin that are part of a cell’s cytoskeleton, a network of protein filaments that extends throughout the cell, gives the cell shape, and keeps its organelles in place. Microtubules are the largest structures in the cytoskeleton at about 24 nanometers thick. They have roles in cell movement, cell division, and transporting materials within cells.","metadata":{}},{"cell_type":"markdown","source":"**Nucleus**: The cell nucleus is a large organelle in eukaryotic organisms which protects the majority of the DNA within each cell. The nucleus also produces the necessary precursors for protein synthesis. The DNA housed within the cell nucleus contains the information necessary for the creation of the majority of the proteins needed to keep a cell functional. While some DNA is stored in other organelles, such as mitochondria, the majority of an organism’s DNA is located in the cell nucleus. The DNA housed in the cell nucleus is extremely valuable, and as such the cell nucleus has a variety of important structures to help maintain, process, and protect the DNA.","metadata":{}},{"cell_type":"markdown","source":"**Endoplasmic reticulum**: The endoplasmic reticulum (ER) is a large organelle made of membranous sheets and tubules that begin near the nucleus and extend across the cell. The endoplasmic reticulum creates, packages, and secretes many of the products created by a cell. Ribosomes, which create proteins, line a portion of the endoplasmic reticulum.","metadata":{}},{"cell_type":"markdown","source":"**Protein of interest**: The information about Protein of interest is not disclosed much but these are marked in the images for which the scientist are researching.","metadata":{}},{"cell_type":"markdown","source":"Let's take out the first record from our training data and visualize the image and it's four channels ","metadata":{}},{"cell_type":"code","source":"train_df.iloc[0] # first record in training data","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Following are the four channels for above image id in the training Image directory')\nprint('\\n')\nID = '5c27f04c-bb99-11e8-b2b9-ac1f6b6435d0'\nfor i in os.listdir('../input/hpa-single-cell-image-classification/train'):\n if(ID in i):\n print(i)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's visualize these 4 channels","metadata":{}},{"cell_type":"code","source":"#create custom color maps\ncdict1 = {'red': ((0.0, 0.0, 0.0),\n (1.0, 0.0, 0.0)),\n\n 'green': ((0.0, 0.0, 0.0),\n (0.75, 1.0, 1.0),\n (1.0, 1.0, 1.0)),\n\n 'blue': ((0.0, 0.0, 0.0),\n (1.0, 0.0, 0.0))}\n\ncdict2 = {'red': ((0.0, 0.0, 0.0),\n (0.75, 1.0, 1.0),\n (1.0, 1.0, 1.0)),\n\n 'green': ((0.0, 0.0, 0.0),\n (1.0, 0.0, 0.0)),\n\n 'blue': ((0.0, 0.0, 0.0),\n (1.0, 0.0, 0.0))}\n\ncdict3 = {'red': ((0.0, 0.0, 0.0),\n (1.0, 0.0, 0.0)),\n\n 'green': ((0.0, 0.0, 0.0),\n (1.0, 0.0, 0.0)),\n\n 'blue': ((0.0, 0.0, 0.0),\n (0.75, 1.0, 1.0),\n (1.0, 1.0, 1.0))}\n\ncdict4 = {'red': ((0.0, 0.0, 0.0),\n (0.75, 1.0, 1.0),\n (1.0, 1.0, 1.0)),\n\n 'green': ((0.0, 0.0, 0.0),\n (0.75, 1.0, 1.0),\n (1.0, 1.0, 1.0)),\n\n 'blue': ((0.0, 0.0, 0.0),\n (1.0, 0.0, 0.0))}\n\n\nnewcmap = matplotlib.colors.LinearSegmentedColormap('greens', cdict1)\nplt.register_cmap('greens', newcmap)\n\nnewcmap = matplotlib.colors.LinearSegmentedColormap('greens', cdict2)\nplt.register_cmap('reds', newcmap)\n\nnewcmap = matplotlib.colors.LinearSegmentedColormap('greens', cdict3)\nplt.register_cmap('blues', newcmap)\n\nnewcmap = matplotlib.colors.LinearSegmentedColormap('greens', cdict4)\nplt.register_cmap('yellows', newcmap)\n\ngreen = cv2.imread('../input/hpa-single-cell-image-classification/train/5c27f04c-bb99-11e8-b2b9-ac1f6b6435d0_green.png', 0)\nred = cv2.imread('../input/hpa-single-cell-image-classification/train/5c27f04c-bb99-11e8-b2b9-ac1f6b6435d0_red.png', 0)\nblue = cv2.imread('../input/hpa-single-cell-image-classification/train/5c27f04c-bb99-11e8-b2b9-ac1f6b6435d0_blue.png', 0)\nyellow = cv2.imread('../input/hpa-single-cell-image-classification/train/5c27f04c-bb99-11e8-b2b9-ac1f6b6435d0_yellow.png', 0)\n\n#display each channel separately\nfig, ax = plt.subplots(nrows = 2, ncols=2, figsize=(15, 15))\nax[0, 0].imshow(green, cmap=\"greens\")\nax[0, 0].set_title(\"Protein of interest (Green)\", fontsize=18)\nax[0, 1].imshow(red, cmap=\"reds\")\nax[0, 1].set_title(\"Microtubules (Red)\", fontsize=18)\nax[1, 0].imshow(blue, cmap=\"blues\")\nax[1, 0].set_title(\"Nucleus (Blue)\", fontsize=18)\nax[1, 1].imshow(yellow, cmap=\"yellows\")\nax[1, 1].set_title(\"Endoplasmic reticulum (Yellow)\", fontsize=18)\nfor i in range(2):\n for j in range(2):\n ax[i, j].set_xticklabels([])\n ax[i, j].set_yticklabels([])\n ax[i, j].tick_params(left=False, bottom=False)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's visualize the RGB image for image id 5c27f04c-bb99-11e8-b2b9-ac1f6b6435d0 with it's labels","metadata":{}},{"cell_type":"code","source":"# mearging RGB channels to produce RGB image\nplt.figure(figsize=(7,7))\ngreen = cv2.imread('../input/hpa-single-cell-image-classification/train/5c27f04c-bb99-11e8-b2b9-ac1f6b6435d0_green.png', cv2.IMREAD_UNCHANGED)\nred = cv2.imread('../input/hpa-single-cell-image-classification/train/5c27f04c-bb99-11e8-b2b9-ac1f6b6435d0_red.png', cv2.IMREAD_UNCHANGED)\nblue = cv2.imread('../input/hpa-single-cell-image-classification/train/5c27f04c-bb99-11e8-b2b9-ac1f6b6435d0_blue.png', cv2.IMREAD_UNCHANGED)\nimg = cv2.merge((red, green, blue))\ncv2.imwrite('first_image.tif', img)\nplt.imshow(img)\nplt.xticks([])\nplt.yticks([])\nplt.title(\"RGB Image\", fontsize=18)\nplt.xlabel('Labels 8:Intermediate filaments \\n5:Nuclear bodies\\n0:Nucleoplasm ', fontsize=18)\nplt.tick_params(left=False, bottom=False)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's visualize each label individually. We know what each lalel represents here but if we visualize them, we will able to undestand them more clearly.","metadata":{}},{"cell_type":"markdown","source":"### Train Data Preprocessing","metadata":{}},{"cell_type":"markdown","source":"    Since each images have many cells with different labels so to visualize each label individually we will need to crop each cell with it's corresponding label from the given images and them visualize them.","metadata":{}},{"cell_type":"markdown","source":"These are the steps to crop individual cells from images: \n1. Image segmentation (the segmentation process will segment individual cells in the given images and then label it with corresponding classes).\n2. Cropping process (the cropping process will crop individual cells from all images after segmentation).","metadata":{}},{"cell_type":"markdown","source":"**Processing train.csv data to perform image segmentation**","metadata":{}},{"cell_type":"code","source":"class_labels = [str(i) for i in range(19)] # list of class labels [0-18]\n\ntrain_label = train_df.Label # labels in training data\n\n# one-hot encoding of class labels in training data\nfor x in class_labels: \n train_df[x] = train_df['Label'].apply(lambda r: int(x in r.split('|')))\n \ntrain_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now there are 19 more binary columns. To understand them lets take image of first record, it has labels 8,5,0 so for this image the values for columns '8', '5', '0' are 1 and other (0-19 except 0,5,8) are 0. Same with other images also. \n \n1 mean True. \n0 mean False.","metadata":{}},{"cell_type":"markdown","source":"**Checking class labels imbalancies**","metadata":{}},{"cell_type":"code","source":"len(train_df.Label.unique())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are total 432 unique values","metadata":{}},{"cell_type":"code","source":"max_label_counts = 0\n\nfor i in train_df.Label.unique():\n if(max_label_counts <= len(i.split('|'))):\n max_label_counts = len(i.split('|'))\n \nmax_label_counts","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The maximum number of labels for an Image in our training data are 5","metadata":{}},{"cell_type":"code","source":"min_label_counts = 1\n\nfor i in train_df.Label.unique():\n if(max_label_counts >= len(i.split('|'))):\n max_label_counts = len(i.split('|'))\n \nmin_label_counts","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The minimum number of class labels for an Image in our training data is 1","metadata":{}},{"cell_type":"markdown","source":"**Class labels imbalancies Visualization**","metadata":{}},{"cell_type":"code","source":"# Adding string lables in training csv data for better visualization\nLabels = {0: \"Nucleoplasm\", 1: \"Nuclear membrane\", 2: \"Nucleoli\", 3: \"Nucleoli fibrillar center\" , 4: \"Nuclear speckles\",\n 5: \"Nuclear bodies\", 6: \"Endoplasmic reticulum\", 7: \"Golgi apparatus\", 8: \"Intermediate filaments\",\n 9: \"Actin filaments\", 10: \"Microtubules\", 11: \"Mitotic spindle\", 12: \"Centrosome\", 13: \"Plasma membrane\",\n 14: \"Mitochondria\", 15: \"Aggresome\", 16: \"Cytosol\", 17: \"Vesicles and punctate cytosolic patterns\", \n 18: \"Negative\"}\n\n# Map the Individual labels to String_label\ntrain_df[\"string_label\"] = train_df.Label.apply(lambda x: \"|\".join([Labels[int(i)] for i in x.split(\"|\")]))\ntrain_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"label_counts = Counter([c for sublist in train_df.string_label.str.split(\"|\").to_list() for c in sublist])\nfig = px.bar(x=label_counts.keys(), y=label_counts.values(), opacity=0.85, \n color=label_counts.keys(),\n labels={\n \"y\":\"Number of Occurences Within The Dataset\", \n \"x\":\"Label Name\", \n \"color\":\"Label Name\"\n },\n title=\"Number of Occurences For Each Label Within The Dataset\")\nfig.update_layout(legend_title=None,\n xaxis_title=\"Label Names\",\n yaxis_title=\"Number of Occurences Within The Dataset\")\nfig.update_xaxes(categoryorder=\"total descending\")\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the above graph\n\n1. We can see that the training data highly imbalanced. \n\n2. We can see that most common protein structures belong to coarse grained cellular components Nucleoplasm. \n \n3. In contrast small or thin components like the mitotic spindle, microtubles, and vesicles are very seldom in our train data. In addition, rare organelles like Aggresome's and Negative also have very little representation in the dataset. For these classes the prediction will be very difficult as we have only a few examples that may not cover all variation normally present within these biological structures will be captured. So, we will struggle to make accurate predictions on the minor classes.","metadata":{}},{"cell_type":"markdown","source":"Let's compare the number of occurance of each label indiviaully with total occurance","metadata":{}},{"cell_type":"code","source":"unique_counts = {}\nfor label in class_labels:\n unique_counts[label] = len(train_df[train_df.Label == label])\n\nfull_counts = {}\nfor label in class_labels:\n count = 0\n for row_label in train_df['Label']:\n if label in row_label.split('|'): count += 1\n full_counts[label] = count\n \ncounts = list(zip(full_counts.keys(), full_counts.values(), unique_counts.values()))\ncounts = np.array(sorted(counts, key=lambda x:-x[1]))\ncounts = pd.DataFrame(counts, columns=['label', 'Total Count', 'Individual Count'])\ncounts.label = [int(i) for i in counts.label]\ncounts = counts.sort_values(by='label')\ncounts.set_index('label').T","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Sampling","metadata":{}},{"cell_type":"markdown","source":"To deal with class imbalancies, we will perform downsampling here.. \n \nSteps: \n1. We will first choose 500 records randomly which have single label for each label.\n2. There are some class labels which don't have 500 records so for those labels we will select remaining records from the records which have more than one label.\n3. If still some lable don't have 500 records we will leave them as it is.","metadata":{}},{"cell_type":"code","source":"train_dfs_0 = train_df[train_df['Label'] == '0'].sample(n=500, random_state=42).reset_index(drop=True)\ntrain_dfs_1u = train_df[train_df['Label'] == '1'].sample(n=221, random_state=42).reset_index(drop=True)\ntrain_dfs_1 = train_df[train_df['1'] == 1].sample(n=500-221, random_state=42).reset_index(drop=True)\ntrain_dfs_2 = train_df[train_df['Label'] == '2'].sample(n=500, random_state=42).reset_index(drop=True)\ntrain_dfs_3 = train_df[train_df['Label'] == '3'].sample(n=500, random_state=42).reset_index(drop=True)\ntrain_dfs_4 = train_df[train_df['Label'] == '4'].sample(n=500, random_state=42).reset_index(drop=True)\ntrain_dfs_5 = train_df[train_df['Label'] == '5'].sample(n=500, random_state=42).reset_index(drop=True)\ntrain_dfs_6u = train_df[train_df['Label'] == '6'].sample(n=476, random_state=42).reset_index(drop=True)\ntrain_dfs_6 = train_df[train_df['6'] == 1].sample(n=500-476, random_state=42).reset_index(drop=True)\ntrain_dfs_7 = train_df[train_df['Label'] == '7'].sample(n=500, random_state=42).reset_index(drop=True)\ntrain_dfs_8 = train_df[train_df['Label'] == '8'].sample(n=500, random_state=42).reset_index(drop=True)\ntrain_dfs_9u = train_df[train_df['Label'] == '9'].sample(n=294, random_state=42).reset_index(drop=True)\ntrain_dfs_9 = train_df[train_df['9'] == 1].sample(n=500-294, random_state=42).reset_index(drop=True)\ntrain_dfs_10u = train_df[train_df['Label'] == '10'].sample(n=404, random_state=42).reset_index(drop=True)\ntrain_dfs_10 = train_df[train_df['10'] == 1].sample(n=500-404, random_state=42).reset_index(drop=True)\ntrain_dfs_11u = train_df[train_df['Label'] == '11'].sample(n=1, random_state=42).reset_index(drop=True)\ntrain_dfs_11 = train_df[train_df['11'] == 1].reset_index(drop=True)\ntrain_dfs_12 = train_df[train_df['Label'] == '12'].sample(n=500, random_state=42).reset_index(drop=True)\ntrain_dfs_13 = train_df[train_df['Label'] == '13'].sample(n=500, random_state=42).reset_index(drop=True)\ntrain_dfs_14 = train_df[train_df['Label'] == '14'].sample(n=500, random_state=42).reset_index(drop=True)\ntrain_dfs_15u = train_df[train_df['Label'] == '15'].sample(n=82, random_state=42).reset_index(drop=True)\ntrain_dfs_15 = train_df[train_df['15'] == 1].reset_index(drop=True)\ntrain_dfs_16 = train_df[train_df['Label'] == '16'].sample(n=500, random_state=42).reset_index(drop=True)\ntrain_dfs_17u = train_df[train_df['Label'] == '17'].sample(n=274, random_state=42).reset_index(drop=True)\ntrain_dfs_17 = train_df[train_df['17'] == 1].sample(n=500-274, random_state=42).reset_index(drop=True)\ntrain_dfs_18 = train_df[train_df['18'] == 1].reset_index(drop=True)\ntrain_dfs_ = [train_dfs_0, train_dfs_1u, train_dfs_1, train_dfs_2, train_dfs_3, train_dfs_4, train_dfs_5, train_dfs_6u,\n train_dfs_6, train_dfs_7, train_dfs_8, train_dfs_9u, train_dfs_9, train_dfs_10u, train_dfs_10, train_dfs_11u, \n train_dfs_11, train_dfs_12, train_dfs_13, train_dfs_14, train_dfs_15u, train_dfs_15, train_dfs_16,\n train_dfs_17u, train_dfs_17, train_dfs_18]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dfs = pd.concat(train_dfs_, ignore_index=True)\ntrain_dfs.drop_duplicates(inplace=True, ignore_index=True)\nlen(train_dfs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After sampling there are total 8081 records but we have now balanced dataset which is good for training our model.","metadata":{}},{"cell_type":"code","source":"train_dfs.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's compare the number of occurance of each label indiviaully with total occurance in our sampled dataset","metadata":{}},{"cell_type":"code","source":"unique_counts = {}\nfor label in class_labels:\n unique_counts[label] = len(train_dfs[train_dfs.Label == label])\n\nfull_counts = {}\nfor label in class_labels:\n count = 0\n for row_label in train_dfs['Label']:\n if label in row_label.split('|'): count += 1\n full_counts[label] = count\n \ncounts = list(zip(full_counts.keys(), full_counts.values(), unique_counts.values()))\ncounts = np.array(sorted(counts, key=lambda x:-x[1]))\ncounts = pd.DataFrame(counts, columns=['label', 'Total Count', 'Individual Count'])\ncounts.label = [int(i) for i in counts.label]\ncounts = counts.sort_values(by='label')\ncounts.set_index('label').T","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we have balaced data set so the next is we have to perform segmentation and masking to crop individual cells from given images to visualize it and for training purpose also.","metadata":{}},{"cell_type":"markdown","source":"#### Image Segmantation","metadata":{}},{"cell_type":"markdown","source":"For segmentation we will use a pretrained model which is specific to cell segementation, read more about it from https://github.com/CellProfiling/HPA-Cell-Segmentation/","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\n\nimport hpacellseg.cellsegmentator as cellsegmentator\nfrom hpacellseg.utils import label_cell, label_nuclei\n\nfrom sklearn.preprocessing import MultiLabelBinarizer\nfrom array import array\n\nNUC_MODEL = \"../input/processed-hpa-data/nuclei-model.pth\"\nCELL_MODEL = \"../input/processed-hpa-data/cell-model.pth\"\n\nsegmentator = cellsegmentator.CellSegmentator(\n NUC_MODEL,\n CELL_MODEL,\n scale_factor=0.25,\n padding=False,\n multi_channel_model=True,\n)\n\nnuc_segmentations = segmentator.pred_nuclei([img])\n\nf, ax = plt.subplots(1, 2, figsize=(10,10))\nax[0].imshow(img)\nax[0].set_title('Original Nucleis', size=20)\nax[1].imshow(nuc_segmentations[0])\nax[1].set_title('Segmented Nucleis', size=20)\nplt.show()\n\n# Cell segmentation\ninter_step = [[i] for i in [red, green, blue]]\ncell_segmentations = segmentator.pred_cells(inter_step)\n\nf, ax = plt.subplots(1, 2, figsize=(10,10))\nax[0].imshow(cv2.merge((red, green, blue)))\nax[0].set_title('Original Cells', size=20)\nax[1].imshow(cell_segmentations[0])\nax[1].set_title('Segmented Cells', size=20)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's apply masking on above two segmented images so that we can visualize each cell individually","metadata":{}},{"cell_type":"code","source":"nuclei_mask = label_nuclei(nuc_segmentations[0])\n# Cell masks\ncell_nuclei_mask, cell_mask = label_cell(nuc_segmentations[0], cell_segmentations[0])\n# Plotting\nf, ax = plt.subplots(1, 3, figsize=(16,16))\nax[0].imshow(nuclei_mask)\nax[0].set_title('Nuclei Mask', size=20)\nax[1].imshow(cell_nuclei_mask)\nax[1].set_title('Cell Nuclei Mask', size=20)\nax[2].imshow(cell_mask)\nax[2].set_title('Cell Mask', size=20)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualizing each cell individually.","metadata":{}},{"cell_type":"code","source":"# Unique vector of cell_mask numbers\nnumbers = set(np.ravel(cell_mask))\nnumbers.remove(0)\n\nfig = plt.figure(figsize=(20,len(numbers)))\nindex = 1\n\nax = fig.add_subplot(len(numbers)//5+1, 5, index)\nax.set_title(\"Complete Cell Mask\", size=16)\nplt.imshow(cell_mask)\n\nindex += 1\nfor number in numbers:\n isolated_cell = np.where(cell_mask==number, cell_mask, 0)\n ax = fig.add_subplot(len(numbers)//5+1, 5, index)\n ax.set_title(f\"Segment {number}\", size=16)\n plt.imshow(isolated_cell)\n index += 1","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's segment all training images and crop invidual cell from all of them with their corresponding labels and create a new image dataset to train our model and visualizing each label individually as well.","metadata":{}},{"cell_type":"code","source":"# function to crop segmentaed cells from given images\n\ndef get_cropped_cell(img, msk):\n bmask = msk.astype(int)[...,None]\n masked_img = img * bmask\n true_points = np.argwhere(bmask)\n top_left = true_points.min(axis=0)\n bottom_right = true_points.max(axis=0)\n cropped_arr = masked_img[top_left[0]:bottom_right[0]+1,top_left[1]:bottom_right[1]+1]\n return cropped_arr","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# return height and widht of given images\n\ndef get_stats(cropped_cell):\n x = (cropped_cell/255.0).reshape(-1,3).mean(0)\n x2 = ((cropped_cell/255.0)**2).reshape(-1,3).mean(0)\n return x, x2","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ROOT = '../input/hpa-single-cell-image-classification'\ndef read_img(image_id, color, train_or_test='train', image_size=None):\n filename = f'{ROOT}/{train_or_test}/{image_id}_{color}.png'\n assert os.path.exists(filename), f'not found {filename}'\n img = cv2.imread(filename, cv2.IMREAD_UNCHANGED)\n if image_size is not None:\n img = cv2.resize(img, (image_size, image_size))\n if img.max() > 255:\n img_max = img.max()\n img = (img/255).astype('uint8')\n return img","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The below code cell will take around 8 hours to complete. I have already run it and saved the output. SO I will directly use the save data.","metadata":{}},{"cell_type":"code","source":"\"\"\"\nx_tot,x2_tot = [],[]\nlbls = []\nnum_files = len(train_dfs)\nall_cells = []\ncell_mask_dir = '../input/hpa-mask/hpa_cell_mask'\ntrain_or_test = 'train'\n\nwith zipfile.ZipFile('cells.zip', 'w') as img_out:\n\n for idx in tqdm(1606, 2000):\n image_id = train_dfs.iloc[idx].ID\n labels = train_dfs.iloc[idx].Label\n cell_mask = np.load(f'{cell_mask_dir}/{image_id}.npz')['arr_0']\n red = read_img(image_id, \"red\", train_or_test, None)\n green = read_img(image_id, \"green\", train_or_test, None)\n blue = read_img(image_id, \"blue\", train_or_test, None)\n #yellow = read_img(image_id, \"yellow\", train_or_test, image_size)\n stacked_image = np.transpose(np.array([blue, green, red]), (1,2,0))\n\n for cell in range(1, np.max(cell_mask) + 1):\n bmask = cell_mask == cell\n cropped_cell = get_cropped_cell(stacked_image, bmask)\n fname = f'{image_id}_{cell}.jpg'\n im = cv2.imencode('.jpg', cropped_cell)[1]\n img_out.writestr(fname, im)\n x, x2 = get_stats(cropped_cell)\n x_tot.append(x)\n x2_tot.append(x2)\n all_cells.append({\n 'image_id': image_id, c\n 'r_mean': x[0],\n 'g_mean': x[1],\n 'b_mean': x[2],\n 'cell_id': cell,\n 'image_labels': labels,\n 'size1': cropped_cell.shape[0],\n 'size2': cropped_cell.shape[1],\n })\n\n#image stats\nimg_avr = np.array(x_tot).mean(0)\nimg_std = np.sqrt(np.array(x2_tot).mean(0) - img_avr**2)\ncell_train_df = pd.DataFrame(all_cells)\ncell_train_df.to_csv('cell_train_df.csv', index=False)\nprint('mean:',img_avr, ', std:', img_std)\n\"\"\"\n\"\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# loading processed train csv data\ntrain_df = pd.read_csv('../input/processed-hpa-data/cell_train_df.csv')\ntrain_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since now we have individual images for each cell so lets visualize images for each label","metadata":{}},{"cell_type":"code","source":"# Choosing one-one image id belonging to each label\n\nimage_label_0 = train_df['image_id'][train_df['image_labels']=='0'].iloc[0]\nimage_label_1 = train_df['image_id'][train_df['image_labels']=='1'].iloc[0]\nimage_label_2 = train_df['image_id'][train_df['image_labels']=='2'].iloc[0]\nimage_label_3 = train_df['image_id'][train_df['image_labels']=='3'].iloc[0]\nimage_label_4 = train_df['image_id'][train_df['image_labels']=='4'].iloc[0]\nimage_label_5 = train_df['image_id'][train_df['image_labels']=='5'].iloc[0]\nimage_label_6 = train_df['image_id'][train_df['image_labels']=='6'].iloc[0]\nimage_label_7 = train_df['image_id'][train_df['image_labels']=='7'].iloc[0]\nimage_label_8 = train_df['image_id'][train_df['image_labels']=='8'].iloc[0]\nimage_label_9 = train_df['image_id'][train_df['image_labels']=='9'].iloc[0]\nimage_label_10 = train_df['image_id'][train_df['image_labels']=='10'].iloc[0]\nimage_label_11 = train_df['image_id'][train_df['image_labels']=='11'].iloc[0]\nimage_label_12 = train_df['image_id'][train_df['image_labels']=='12'].iloc[0]\nimage_label_13 = train_df['image_id'][train_df['image_labels']=='13'].iloc[0]\nimage_label_14 = train_df['image_id'][train_df['image_labels']=='14'].iloc[0]\nimage_label_15 = train_df['image_id'][train_df['image_labels']=='15'].iloc[0]\nimage_label_16 = train_df['image_id'][train_df['image_labels']=='16'].iloc[0]\nimage_label_17 = train_df['image_id'][train_df['image_labels']=='17'].iloc[0]\nimage_label_18 = train_df['image_id'][train_df['image_labels']=='18'].iloc[0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.image as mpimg\nimage_ids = [image_label_0, image_label_1 ,image_label_2, image_label_3,image_label_4 ,image_label_5,image_label_6,\n image_label_7, image_label_8, image_label_9, image_label_10, image_label_11, image_label_12,\n image_label_13, image_label_14, image_label_15, image_label_16, image_label_17, image_label_18]\n\nplt.figure(figsize=(20,25))\nfor i, label in Labels.items():\n plt.subplot(6,5,int(i)+1)\n img = mpimg.imread('../input/processed-hpa-data/cells/cells/'+image_ids[int(i)]+'_1.jpg')\n imgplot = plt.imshow(img)\n plt.title(label, fontsize=15)\n plt.xticks([])\n plt.yticks([])\n\nplt.suptitle('Individual Cells with labels',fontsize=20)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Test Data","metadata":{}},{"cell_type":"markdown","source":"    In test data there only images for which the labels will predict.","metadata":{}},{"cell_type":"markdown","source":"### Model Creation and Training","metadata":{}},{"cell_type":"markdown","source":"We will use fastai tool to create our model. We need images of individual cells as an input to the classification model. Due to system limitation we will not use all images for model training instead we have created a sample balanced dataset to train the model. We use RGB channels only, which has proven to work well in the previous HPA challenge. We saved the extracted cells as RGB jpg images already so that I can feed them easily into my classifier.\n \nWe will first train three models using three different pretrained models resnet, densenet and unet and after training we will compare which one will be best.\n \n1. **Resnet**: A residual neural network (ResNet) is an artificial neural network (ANN) of a kind that builds on constructs known from pyramidal cells in the cerebral cortex. Residual neural networks do this by utilizing skip connections, or shortcuts to jump over some layers. Typical ResNet models are implemented with double- or triple- layer skips that contain nonlinearities (ReLU) and batch normalization in between.An additional weight matrix may be used to learn the skip weights; these models are known as HighwayNets. \n\n2. **EfficientNet**: EfficientNet is a convolutional neural network architecture and scaling method that uniformly scales all dimensions of depth/width/resolution using a compound coefficient. Unlike conventional practice that arbitrary scales these factors, the EfficientNet scaling method uniformly scales network width, depth, and resolution with a set of fixed scaling coefficients.\n\n3. **Densenet**: DenseNet is one of the new discoveries in neural networks for visual object recognition. DenseNet is quite similar to ResNet with some fundamental differences. ResNet uses an additive method (+) that merges the previous layer (identity) with the future layer, whereas DenseNet concatenates (.) the output of the previous layer with the future layer. ","metadata":{}},{"cell_type":"code","source":"labels = [str(i) for i in range(19)]\nfor x in labels: \n train_df[x] = train_df['image_labels'].apply(lambda r: int(x in r.split('|')))\ntrain_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# test data for checking the performace of our model after trainining\n\ntest_performance_df_0 = train_df[train_df['image_labels'] == '0'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_1 = train_df[train_df['image_labels'] == '1'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_2 = train_df[train_df['image_labels'] == '2'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_3 = train_df[train_df['image_labels'] == '3'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_4 = train_df[train_df['image_labels'] == '4'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_5 = train_df[train_df['image_labels'] == '5'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_6 = train_df[train_df['image_labels'] == '6'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_7 = train_df[train_df['image_labels'] == '7'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_8 = train_df[train_df['image_labels'] == '8'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_9 = train_df[train_df['image_labels'] == '9'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_10 = train_df[train_df['image_labels'] == '10'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_11 = train_df[train_df['image_labels'] == '10|11'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_11['image_labels'] = ['11']*25\ntest_performance_df_12 = train_df[train_df['image_labels'] == '12'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_13 = train_df[train_df['image_labels'] == '13'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_14 = train_df[train_df['image_labels'] == '14'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_15 = train_df[train_df['image_labels'] == '15'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_16 = train_df[train_df['image_labels'] == '16'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_17 = train_df[train_df['image_labels'] == '17'].sample(n=25, random_state=42).reset_index(drop=True)\ntest_performance_df_18 = train_df[train_df['image_labels'] == '18'].sample(n=25, random_state=42).reset_index(drop=True)\n\ntest_performance_df_ = [test_performance_df_0, test_performance_df_1, test_performance_df_2, test_performance_df_3, test_performance_df_4, test_performance_df_5,\n test_performance_df_6, test_performance_df_7, test_performance_df_8, test_performance_df_9, test_performance_df_10, test_performance_df_11,\n test_performance_df_12, test_performance_df_13, test_performance_df_14, test_performance_df_15, test_performance_df_16, test_performance_df_17,\n test_performance_df_18]\n\ntest_performance_df = pd.concat(test_performance_df_, ignore_index=True)\ntest_performance_df.drop_duplicates(inplace=True, ignore_index=True)\n\ntest_performance_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 20% sampled dataset \ntrain_dfs = train_df.sample(frac=0.20, random_state=42)\ntrain_dfs = train_dfs.reset_index(drop=True)\nlen(train_dfs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's compare the number of occurance of each label indiviaully with total occurance in our sampled dataset","metadata":{}},{"cell_type":"code","source":"unique_counts = {}\nfor lbl in labels:\n unique_counts[lbl] = len(train_dfs[train_dfs.image_labels == lbl])\n\nfull_counts = {}\nfor lbl in labels:\n count = 0\n for row_label in train_dfs['image_labels']:\n if lbl in row_label.split('|'): count += 1\n full_counts[lbl] = count\n \ncounts = list(zip(full_counts.keys(), full_counts.values(), unique_counts.values()))\ncounts = np.array(sorted(counts, key=lambda x:-x[1]))\ncounts = pd.DataFrame(counts, columns=['label', 'Total Count', 'Individual Count'])\ncounts.set_index('label').T","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Class label 11 has 0 individual counts, this may create zero division error. To handle this we will add 10 records for label 11 from records label '10|11'.","metadata":{}},{"cell_type":"code","source":"df_11 = train_df[train_df.image_labels.str.contains('11')][0:10]\ndf_11['image_labels'] = ['11']*10\ntrain_dfs = pd.concat((train_dfs, df_11))\n\nunique_counts = {}\nfor lbl in labels:\n unique_counts[lbl] = len(train_dfs[train_dfs.image_labels == lbl])\n\nfull_counts = {}\nfor lbl in labels:\n count = 0\n for row_label in train_dfs['image_labels']:\n if lbl in row_label.split('|'): count += 1\n full_counts[lbl] = count\n \ncounts = list(zip(full_counts.keys(), full_counts.values(), unique_counts.values()))\ncounts = np.array(sorted(counts, key=lambda x:-x[1]))\ncounts = pd.DataFrame(counts, columns=['label', 'Total Count', 'Individual Count'])\ncounts.set_index('label').T","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now Label 11 has 10 individual counts.","metadata":{}},{"cell_type":"markdown","source":"Spliting the sampled training data into training data and validation data using stratify method to balance the class labels on both train and validation data.","metadata":{}},{"cell_type":"code","source":"nfold = 5\n\ny = train_dfs[labels].values\nX = train_dfs[['image_id', 'cell_id']].values\n\ntrain_dfs['fold'] = np.nan\n\nfrom iterstrat.ml_stratifiers import MultilabelStratifiedKFold\nmskf = MultilabelStratifiedKFold(n_splits=nfold, shuffle=True, random_state=None)\nfor i, (_, test_index) in enumerate(mskf.split(X, y)):\n train_dfs.iloc[test_index, -1] = i\n \ntrain_dfs['fold'] = train_dfs['fold'].astype('int')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dfs['is_valid'] = False\ntrain_dfs['is_valid'][train_dfs['fold'] == 0] = True","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dfs.is_valid.value_counts()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now there are 29125 records in Train data and 7284 records in validation data.","metadata":{}},{"cell_type":"code","source":"# defining function to return image for given image id\ndef get_x(r): \n return '../input/processed-hpa-data/cells/cells/'+(r['image_id']+'_'+str(r['cell_id'])+'.jpg')\n\n# defining function to return label for \ndef get_y(r): \n return r['image_labels'].split('|')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# sample_stats = (image_array_mean, image_array_std) # one image have 3 channels # mean and std of all cell images.\nsample_stats = ([0.07290461, 0.04505656, 0.07713918] , [0.1727259 , 0.10327134, 0.14257778])\nitem_tfms = RandomResizedCrop(224, min_scale=0.75, ratio=(1.,1.))\nbatch_tfms = [*aug_transforms(flip_vert=True, size=128, max_warp=0), Normalize.from_stats(*sample_stats)]\nbs=128","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# code to create batch dataset\ndblock = DataBlock(blocks=(ImageBlock, MultiCategoryBlock(vocab=labels)),\n splitter=ColSplitter(col='is_valid'),\n get_x=get_x,\n get_y=get_y,\n item_tfms=item_tfms,\n batch_tfms=batch_tfms,\n )\ndls = dblock.dataloaders(train_dfs, bs=bs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dls.show_batch(nrows=3, ncols=3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Training with Resnet**","metadata":{}},{"cell_type":"code","source":"# copying pretrained models \nif not os.path.exists('/root/.cache/torch/hub/checkpoints/'):\n os.makedirs('/root/.cache/torch/hub/checkpoints/')\n\n!cp ../input/models/resnet50-19c8e357.pth /root/.cache/torch/hub/checkpoints/\n!cp ../input/models/densenet121-a639ec97.pth /root/.cache/torch/hub/checkpoints/\n!cp ../input/models/adv-efficientnet-b7-4652b6dd.pth /root/.cache/torch/hub/checkpoints/","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#creating our cnn model\n\nres_learn = cnn_learner(dls, resnet50, metrics=[accuracy_multi, PrecisionMulti()]).to_fp16()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"torch.cuda.empty_cache()\nres_learn.lr_find()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The suggested value of learning parameter is around 0.03","metadata":{}},{"cell_type":"code","source":"lr=3e-2 # learning parameter\ntorch.cuda.empty_cache() # empty GPU cache memory\nres_learn.fine_tune(5,base_lr=lr) # starting training with 5 epochs","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After epoch 7 the difference between train and valid loss started increasing but both are decreasing so to prevent our model from being overfit 10 epochs are enough and if we want to increase the accuracy further so we should increase the fraction of training data. Currently we have trained our on 20% of total training data due to system limitations.","metadata":{}},{"cell_type":"code","source":"res_learn.recorder.plot_loss() # plotting train and validation loss","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"res_learn.save('hpa_resnet50_model') # saving our model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Training with Efficient net","metadata":{}},{"cell_type":"code","source":"# locating and downloading the pretrained Efficient net model\n# We will use transfer learning method with Efficientnet-B7 to train our learner\n\n\npackage_path = '../input/efficientnet-pytorch/EfficientNet-PyTorch/EfficientNet-PyTorch-master'\nsys.path.append(package_path)\n\n%cd /kaggle/input/efficientnet-pytorch/EfficientNet-PyTorch/EfficientNet-PyTorch-master\nfrom efficientnet_pytorch import EfficientNet\n%cd -\n\ndef get_learner(lr=1e-3):\n # Optimization funciton and parameters\n opt_func = partial(Adam, lr=lr, wd=0.01, eps=1e-8) \n model = EfficientNet.from_pretrained(\"efficientnet-b7\", advprop=True)\n # Set output layer\n model._fc = nn.Linear(2560, dls.c)\n # Group model, dataloader and metrics\n learn = Learner(\n dls, model, opt_func=opt_func,\n metrics=[accuracy_multi, PrecisionMulti()]\n ).to_fp16()\n return learn","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Initialize lerner\neffi_learn=get_learner()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Finding best value for learning parameter\n\n\ntorch.cuda.empty_cache()\neffi_learn.lr_find()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Training our model \n\nlr = 1e-3\ntorch.cuda.empty_cache()\neffi_learn.fine_tune(5,base_lr=lr)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"effi_learn.recorder.plot_loss()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"effi_learn.save('hpa_effi-b7_model') # saving our model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Traininig with densenet","metadata":{}},{"cell_type":"code","source":"dens_learn = cnn_learner(dls, models.densenet121, metrics=[accuracy_multi, PrecisionMulti()]).to_fp16()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ntorch.cuda.empty_cache() # empty GPU cache memory\ndens_learn.lr_find() # finding best value for learning parameter to train our cnn model\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nlr=3e-2 # learning parameter\ntorch.cuda.empty_cache() # empty GPU cache memory\ndens_learn.fine_tune(5,base_lr=lr) # starting training with 5 epochs\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dens_learn.recorder.plot_loss()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dens_learn.save('hpa_densenet121_model') # saving our model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Models Comparison","metadata":{}},{"cell_type":"markdown","source":"We have trained 3 models based on resnet, efficientnet and densenet to find out which one will be better. We have trained each model for 5 epochs- \n \nAfter 5 epochs the losses for \nResnet are :- Train Loss = 0.132128 and Validation Loss = 0.132426 \nEfficient net are :- Train Loss = 0.095444 and Validation Loss = 0.121629 \nDensenet are :- Train Loss = 0.127250 and Validation Loss = 0.130601 \n \nThe difference between Train Loss and Validation Loss is less in Resnet based model so it means it's a balanced model not an overfitted or underfitted model. So we will use Resnet based model to preform prediction on test images.","metadata":{}},{"cell_type":"markdown","source":"##### Let's train resnet model for 10 epochs to get more better results. Since we training our model on 20% of the data due to system limitations so it's performance will be less. To increase the performance of the model train it on 100% data.","metadata":{}},{"cell_type":"code","source":"res_learn = cnn_learner(dls, resnet50, metrics=[accuracy_multi, PrecisionMulti(), RocAucMulti()]).to_fp16()\nlr=3e-2 # learning parameter\ntorch.cuda.empty_cache() # empty GPU cache memory\nres_learn.fine_tune(10,base_lr=lr) # starting training with 5 epochs","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"res_learn.recorder.plot_loss() # plotting train and validation loss","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model Prediction on Test Data","metadata":{}},{"cell_type":"code","source":"# loading sample_submission.csv into a dataframe \ntest_df = pd.read_csv('../input/hpa-single-cell-image-classification/sample_submission.csv')\ntest_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In sample submission we have Image files name for which we have to perform prediction. By looking at the sample submission, we realize that we need to predict a string for each test image which can be generate as below. \n\n1. Segment each single cell contained in the image.\n2. Predict their class labels confidence.\n3. Then generate a string by doing encoding of segmented cells.\n\nThe structure of the prediction string is as\n\nImageID,ImageWidth,ImageHeight,PredictionString\n\n1. ImageAID,ImageAWidth,ImageAHeight,LabelA1 ConfidenceA1 EncodedMaskA1 LabelA2 ConfidenceA2 EncodedMaskA2 ...\n\n2. ImageBID,ImageBWidth,ImageBHeight,LabelB1 ConfidenceB1 EncodedMaskB1 LabelB2 ConfidenceB2 EncodedMaskB2 …\n\nSample real values could be..\n\nID,ImageWidth,ImageHeight,PredictionString\n1. 721568e01a744247,1118,1600,0 0.637833 eNqLi8xJM7BOTjS08DT2NfI38DfyM/Q3NMAJgJJ+RkBs7JecF5tnAADw+Q9I\n2. 7b018c5e3a20daba,1600,1066,16 0.85117 eNqLiYrLN7DNCjDMMIj0N/Iz9DcwBEIDfyN/QyA2AAsBRfxMPcKTA1MMADVADIo=\n\nBelow is the code to encode the segmentation mask provided by the organiser..","metadata":{}},{"cell_type":"markdown","source":"Next two cells will create encoded string for each segmented cells in images and the whole process will take around 1 hours to complete for all images. I have already run it so I will use the saved data.","metadata":{}},{"cell_type":"code","source":"\"\"\"\nimport base64\nimport numpy as np\nfrom pycocotools import _mask as coco_mask\nimport typing as t\nimport zlib\n\n\ndef encode_binary_mask(mask: np.ndarray) -> t.Text:\n\n # check input mask --\n if mask.dtype != np.bool:\n raise ValueError(\n \"encode_binary_mask expects a binary mask, received dtype == %s\" %\n mask.dtype)\n\n mask = np.squeeze(mask)\n if len(mask.shape) != 2:\n raise ValueError(\n \"encode_binary_mask expects a 2d mask, received shape == %s\" %\n mask.shape)\n\n # convert input mask to expected COCO API input --\n mask_to_encode = mask.reshape(mask.shape[0], mask.shape[1], 1)\n mask_to_encode = mask_to_encode.astype(np.uint8)\n mask_to_encode = np.asfortranarray(mask_to_encode)\n\n # RLE encode mask --\n encoded_mask = coco_mask.encode(mask_to_encode)[0][\"counts\"]\n\n # compress and base64 encoding --\n binary_str = zlib.compress(encoded_mask, zlib.Z_BEST_COMPRESSION)\n base64_str = base64.b64encode(binary_str)\n return base64_str.decode('ascii')\n \n\"\"\"\n''","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Performing test image segmentation and generating encoding strings..","metadata":{}},{"cell_type":"code","source":"\"\"\"\nx_tot,x2_tot = [],[]\nlbls = []\nnum_files = len(test_df)\nall_cells = []\ntrain_or_test = 'test'\ncell_mask_dir = 'F:/HPA/work/cell_masks'\n\nwith zipfile.ZipFile('F:/HPA/test_cells.zip', 'w') as img_out:\n\n for idx in tqdm(range(num_files)):\n image_id = test_df.iloc[idx].ID\n cell_mask = np.load(f'{cell_mask_dir}/{image_id}.npz')['arr_0']\n red = read_img(image_id, \"red\", train_or_test, None)\n green = read_img(image_id, \"green\", train_or_test, None)\n blue = read_img(image_id, \"blue\", train_or_test, None)\n #yellow = read_img(image_id, \"yellow\", train_or_test, image_size)\n stacked_image = np.transpose(np.array([blue, green, red]), (1,2,0))\n\n for j in range(1, np.max(cell_mask) + 1):\n bmask = (cell_mask == j)\n enc = encode_binary_mask(bmask)\n cropped_cell = get_cropped_cell(stacked_image, bmask)\n fname = f'{image_id}_{j}.jpg'\n im = cv2.imencode('.jpg', cropped_cell)[1]\n img_out.writestr(fname, im)\n x, x2 = get_stats(cropped_cell)\n x_tot.append(x)\n x2_tot.append(x2)\n all_cells.append({\n 'image_id': image_id,\n 'fname': fname,\n 'r_mean': x[0],\n 'g_mean': x[1],\n 'b_mean': x[2],\n 'cell_id': j,\n 'size1': cropped_cell.shape[0],\n 'size2': cropped_cell.shape[1],\n 'enc': enc,\n })\n\n#image stats\nimg_avr = np.array(x_tot).mean(0)\nimg_std = np.sqrt(np.array(x2_tot).mean(0) - img_avr**2)\ncell_test_df = pd.DataFrame(all_cells)\ncell_test_df.to_csv('F:/HPA/cell_test_df.csv', index=False)\nprint('mean:',img_avr, ', std:', img_std)\n\"\"\"\n\"\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# loading saved data for encoded cell segment \ncell_test_df = pd.read_csv('../input/processed-hpa-data/cell_test_df.csv')\ncell_test_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# code to create batch dataset\ntest_dl = res_learn.dls.test_dl(cell_test_df)\ntorch.cuda.empty_cache()\ntest_dl.show_batch()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Predecting labels for cells presents in all test images..","metadata":{}},{"cell_type":"code","source":"# performing prediction\npreds, _ = res_learn.get_preds(dl=test_dl) ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# saving prediction in a file\nwith open('preds.pickle', 'wb') as handle:\n pickle.dump(preds, handle)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cls_prds = torch.argmax(preds, dim=-1)\nlen(cls_prds), cls_prds","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Creating submission file..","metadata":{}},{"cell_type":"code","source":"sample_submission = pd.read_csv('../input/hpa-single-cell-image-classification/sample_submission.csv')\nsample_submission.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# combining predicted labels and encoded string\ncell_test_df['cls'] = cls_prds\ncell_test_df['pred'] = cell_test_df[['cls', 'enc']].apply(lambda r: str(r[0]) + ' 1 ' + r[1], axis=1)\ncell_test_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Grouping Cells records into their Image records from where segmented cells were cropped.\nsubm = cell_test_df.groupby(['image_id'])['pred'].apply(lambda x: ' '.join(x)).reset_index()\nsubm.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = pd.merge(sample_submission,subm,how=\"left\",left_on='ID',right_on='image_id')\nsub.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def isNaN(num):\n return num != num\n\nfor i, row in sub.iterrows():\n if isNaN(row['pred']): continue\n sub.PredictionString.loc[i] = row['pred']","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = sub[sample_submission.columns]\nsub.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model Performance on Test data or unseen data","metadata":{}},{"cell_type":"markdown","source":"To check the performance of the model we need labeled and unseen data. So we will use 25 records for each label from training data which we have already seperated from train data before training.","metadata":{}},{"cell_type":"code","source":"test_performance_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_performance_df.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# code to create batch dataset\ntest_dl = res_learn.dls.test_dl(test_performance_df)\ntorch.cuda.empty_cache()\ntest_dl.show_batch()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# performing predictions\npredictions, _ = res_learn.get_preds(dl=test_dl)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(predictions)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Converting predicted probabilites into class labels\ncls_predictions = torch.argmax(predictions, dim=-1)\nlen(cls_predictions), cls_predictions","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cls_predictions = np.array(cls_predictions)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import accuracy_score, confusion_matrix","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculating accuracy score\ntrue_label = [int(i) for i in test_performance_df['image_labels']]\naccuracy_score(true_label, cls_predictions)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have got 53.26% accuracy which is less. This is because we have trained our model only on 20% of given training data due system limitation. To increase the prediction accuracy train the model on whole training dataset.","metadata":{}},{"cell_type":"code","source":"# Plotting confusion matrix to check the prediction accuracy for each class\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nplt.figure(figsize=(15,10))\ncfm = confusion_matrix(true_label, cls_predictions)\nsns.heatmap(cfm, annot=True)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Let's interpret the confusion matrix** \n1. For class label '0' \nOut of 25 reocrds 7 records are predicted correctly. Means 28% accuracy for class '0'.\n2. For class label '1' \nOut of 25 reocrds 19 records are predicted correctly. Means 76% accuracy for class '1'.\n3. For class label '2' \nOut of 25 reocrds 13 records are predicted correctly. Means 52% accuracy for class '2'. \n \nand so on....","metadata":{}},{"cell_type":"markdown","source":"#### NOTE:- we have trained our model only on 20% of given training data due system limitation. To increase the prediction accuracy train the model on whole training dataset.","metadata":{}}]}