{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Context\n\n* For my initial submission, I prepared the [infernce kernel](https://www.kaggle.com/ayuraj/hpa-inference) with the assumption that each cell-level labes are same as image-level label. This gave me an LB score of 0.029.\n* Thus, I trained an \"image-level\" multi-label classifier using this [kernel](https://www.kaggle.com/ayuraj/hpa-multi-label-classification-with-tf-and-w-b). I used this [kernel](https://www.kaggle.com/ayuraj/rgb-512x512-dataset-creation-versioning-with-w-b) to prepare 256x256 and 512x512 resolution RGB images by ignoring the Endoplasmic Reticulum (Y channel). \n* I used the segmentation tool provided by the competition hosts to get cell instance segmentation mask for each image. I built this [kernel](https://www.kaggle.com/ayuraj/hpa-segmentation-mask-visualization-with-w-b) to visualize segmentation masks using W&B.\n* The idea with my initial submission was to prepare a pipeline for submission based on future improvements. \n\n# About this Kernel\n\nObviously due to weak supervision image-level labels can not be blindly assigned as cell-level labels. Thus I decided to create a **cell-level** dataset of images which have only one image-level label. I will then create a dataset of images which have two image-lebel label (will talk about it later). \n\n* I am using the provided segmentation tool to get cell-level masks. The tool gives cell-level mask along with nucleus-level mask. \n* I am discarding the cells whose nucleus is bordering the edge of the image. I am also discarding the cells whose nucleus could not be segmented.\n* For each cell I am assigning it the image-level label. In this case since the whole image is associated to a single label, this assumption holds to some extent (we will see that after submission). \n* I also think that this dataset can be used in a weakly supervised classifier training (will explore that soon and share with you all). \n\n❗ **Note:** I have used this [kernel](https://www.kaggle.com/ayuraj/dataset-versioning-for-single-label-dataset) to create a `.csv` file of image_ids along with labels with only only image-level labels. You will notice that I have created 4 splits of this CSV file. This is so that I can use 9 hr limit of Kaggle kernel to create this cell-level dataset. \n\n## Why use Weights and Biases's Dataset and Visualization (DSViz) tool?\n\nThe dataset creation process in Kaggle can be made more lively with the use of DSViz. W&B's DSViz is a powerful new feature (still in early phase) on top of W&B Artifacts to let you visualize and query datasets and model evaluations at the example level. Check out the official doc [here](https://docs.wandb.ai/datasets-and-predictions).\n\nIt's just a few lines of code that can enable you to visualize and query your image dataset. In my opinion this makes the dataset creation pipeline more interactive. One can easily check if the generated images are correct. \n\n![img](https://i.imgur.com/3p9R90J.gif)\n\n# Datasets\n\nI have processed 3 out of 4 splits. Here's the links to to the dataset and respective W&B DSViz page:\n* [HPA: Single Label RGB+Protein Dataset (0)](https://www.kaggle.com/ayuraj/singlelabel128x128split0) | [DSViz Page](https://wandb.ai/ayush-thakur/hpa/artifacts/dataset/single_label_split_0/34b460a9c6ea265a4e95/files/dataset.table.json)\n* [HPA: Single Label RGB+Protein Dataset (1)](https://www.kaggle.com/ayuraj/singlelabel128x128split1) | [DSViz Page](https://wandb.ai/ayush-thakur/hpa/artifacts/dataset/single_label_split_1/b0f0ffa4613ed7ec454e/files/dataset.table.json)\n* [HPA: Single Label RGB+Protein Dataset (2)](https://www.kaggle.com/ayuraj/singlelabel128x128split2) | [DSViz Page](https://wandb.ai/ayush-thakur/hpa/artifacts/dataset/single_label_split_2/584d4f0d963478a01a6e/files/dataset.table.json)\n\n## Upcoming:\n\n* I will combine all the images from the dataset in a single Kaggle Dataset."},{"metadata":{},"cell_type":"markdown","source":"# Imports and Setups"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%%capture\n\n# Install W&B for ease of life.\n!pip install wandb --upgrade\n\n# Install HPA Cell Segmentation tool.\n!pip install https://github.com/CellProfiling/HPA-Cell-Segmentation/archive/master.zip","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# General imports.\nimport os\nimport re\nimport cv2\nimport glob\nimport imageio\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom PIL import Image\nfrom tqdm.notebook import tqdm\nimport matplotlib.pyplot as plt\nfrom skimage.transform import resize\nfrom sklearn.model_selection import train_test_split\n\n%matplotlib inline\n\n# HPA Segmentation tool related imports\nimport hpacellseg.cellsegmentator as cellsegmentator\nfrom hpacellseg.utils import label_cell, label_nuclei","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# W&B import and login\nimport wandb\nfrom kaggle_secrets import UserSecretsClient\n\nuser_secrets = UserSecretsClient()\nwandb_api = user_secrets.get_secret(\"wandb_api\")\n\nwandb.login(key=wandb_api)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Hyperparameters\nWORKING_DIR_PATH = '../input/hpa-single-cell-image-classification/'\nIMAGE_HEIGHT = 128\nIMAGE_WIDTH = 128\n\n# The original csv file of image_ids with only one image-level label takes more than 9 hrs to process. \n# To be able to use Kaggle's kernel to generate the dataset I made 4 splits of the original csv.\n# The SPLIT number is the index of the split used.\nSPLIT = 3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Labels\n# Ref: https://www.kaggle.com/divyanshuusingh/eda-image-segmentation\nlabel_names= {\n0: \"Nucleoplasm\",\n1: \"Nuclear membrane\",\n2: \"Nucleoli\",\n3: \"Nucleoli fibrillar center\",\n4: \"Nuclear speckles\",\n5: \"Nuclear bodies\",\n6: \"Endoplasmic reticulum\",\n7: \"Golgi apparatus\",\n8: \"Intermediate filaments\",\n9: \"Actin filaments\",\n10: \"Microtubules\",\n11: \"Mitotic spindle\",\n12: \"Centrosome\",\n13: \"Plasma membrane\",\n14: \"Mitochondria\",\n15: \"Aggresome\",\n16: \"Cytosol\",\n17: \"Vesicles and punctate cytosolic patterns\",\n18: \"Negative\"\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image-Level Labels Analysis\n\nRefer to this Kernel: https://www.kaggle.com/ayuraj/dataset-versioning-for-single-label-dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Downloading csv files.\nrun = wandb.init(entity='ayush-thakur', project='hpa', job_type='consume_single_label_dataset')\nartifact = run.use_artifact('ayush-thakur/hpa/single_label_split:v0', type='dataset')\nartifact_dir = artifact.download()\nrun.finish()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(artifact_dir)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv(artifact_dir+f'/train_single_{SPLIT}.csv')\nprint(f'Total number of images: {len(df)}')\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train Dataset Paths"},{"metadata":{"trusted":true},"cell_type":"code","source":"red_images = sorted(glob.glob(WORKING_DIR_PATH+'train/*_red.png'))\ngreen_images = sorted(glob.glob(WORKING_DIR_PATH+'train/*_green.png'))\nblue_images = sorted(glob.glob(WORKING_DIR_PATH+'train/*_blue.png'))\nyellow_images = sorted(glob.glob(WORKING_DIR_PATH+'train/*_yellow.png'))\n\nprint(len(red_images), len(green_images), len(blue_images), len(yellow_images))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mt = [] # red\ner = [] # yellow\nnu = [] # blue\npr = [] # green\n\n# Test if the image ids are aligned properly\nfor r, g, b, y in zip(red_images, green_images, blue_images, yellow_images):\n    if re.findall(r'[^\\/]+(?=\\_.)', r)[0] == re.findall(r'[^\\/]+(?=\\_.)', g)[0] == re.findall(r'[^\\/]+(?=\\_.)', b)[0] == re.findall(r'[^\\/]+(?=\\_.)', y)[0]:\n        if re.findall(r'[^\\/]+(?=\\_.)', r)[0] in df.ID.values:\n            mt.append(r)\n            pr.append(g)\n            nu.append(b)\n            er.append(y)\n    else:\n        print(r)\n        \nprint(len(mt), len(pr), len(nu), len(er))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Segment Cell and Extract Cell Level Images"},{"metadata":{"trusted":true},"cell_type":"code","source":"NUC_MODEL = \"./nuclei-model.pth\"\nCELL_MODEL = \"./cell-model.pth\"\n\nsegmentator = cellsegmentator.CellSegmentator(\n    NUC_MODEL,\n    CELL_MODEL,\n    scale_factor=0.25,\n    device=\"cuda\",\n    padding=True,\n    multi_channel_model=True,\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualize - understand the process"},{"metadata":{"trusted":true},"cell_type":"code","source":"i = 3\n# Get red, blue and yellow channel images. \nmicrotubule = np.array(Image.open(mt[i]))\nendoplasmicrec = np.array(Image.open(er[i]))\nnuclei = np.array(Image.open(nu[i]))\nprotein = np.array(Image.open(pr[i]))\n# Stack the channels to form image.\nimage = np.dstack((microtubule, endoplasmicrec, nuclei))\n\nplt.figure(figsize=(5, 5));\nplt.imshow(image);\nplt.title(f'{df.Label.values[i]}');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(5, 5));\nplt.imshow(protein, cmap='gray');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# For nuclei segmentation only blue channel is required.\nnuc_segmentation = segmentator.pred_nuclei([nu[i]])\n# For full cells all the three reference(except green) channels are required.\ncell_segmentation = segmentator.pred_cells([[mt[i]], [er[i]], [nu[i]]])\n# get cell mask\nnuclei_mask, cell_mask = label_cell(nuc_segmentation[0], cell_segmentation[0])\n\nplt.figure(figsize=(5,5))\nplt.imshow(cell_mask);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# number of cells\ncells = np.unique(cell_mask)\nprint(f\"Number of cells: {len(cells)}\")\n\n# visualize one cell\nmask = np.where(cell_mask==1, 1,0).astype('uint8')\nplt.figure(figsize=(5,5))\nplt.imshow(mask);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# number of cells\nnucleies = np.unique(nuclei_mask)\nprint(f\"Number of cells: {len(nucleies)}\")\n\n# visualize one cell\nn_mask = np.where(nuclei_mask==1, 1,0).astype('uint8')\nplt.figure(figsize=(5,5))\nplt.imshow(n_mask);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get contour using mask for one cell.\ncnts, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)\n\n# visualize contour\nim = image.copy()\nim = cv2.drawContours(im, cnts, -1, (255, 0, 255), 20)\nplt.figure(figsize=(5,5))\nplt.imshow(im);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get contour using mask for one nuclei.\nn_cnts, _ = cv2.findContours(n_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)\n\n# visualize contour\nim = image.copy()\nim = cv2.drawContours(im, n_cnts, -1, (255, 0, 255), 20)\nplt.figure(figsize=(5,5))\nplt.imshow(im);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check if the contour for nuclie is touching the image boundary. \ndef is_border_nuclei(contour_points):\n    unique_points = np.unique(contour_points)\n    # basically if any point is 0 that means its touching the edge of the image.\n    if 0 in unique_points:\n        return True\n    return False\n\nis_border_nuclei(n_cnts[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Mask input image with binary mask\ncell = cv2.bitwise_and(image, image, mask=mask)\n# Color background white\ncell[mask==0] = 255\n\n# Visualize\nplt.figure(figsize=(5,5))\nplt.imshow(cell);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get bounding box covering the contour. \nx,y,w,h = cv2.boundingRect(cnts[0])\n\na = cell[y:y+h, x:x+w]\na = cv2.resize(a, (IMAGE_HEIGHT, IMAGE_WIDTH), cv2.INTER_AREA)\nplt.imshow(a);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# mask protein image \ncell_protein = cv2.bitwise_and(protein, protein, mask=mask)\n\n# Visualize\nplt.figure(figsize=(5,5))\nplt.imshow(cell_protein, cmap='gray');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"a_protein = cell_protein[y:y+h, x:x+w]\na_protein = cv2.resize(a_protein, (IMAGE_HEIGHT, IMAGE_WIDTH), cv2.INTER_AREA)\n\nplt.imshow(a_protein, cmap='gray');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataset Creation Loop"},{"metadata":{"trusted":true},"cell_type":"code","source":"SAVE_DIR = f'/kaggle/tmp/hpa_single_label_128x128_split_{SPLIT}/'\n\nos.makedirs(SAVE_DIR+'rgb', exist_ok=True)\nos.makedirs(SAVE_DIR+'protein', exist_ok=True)\n\n!ls /kaggle/tmp/hpa_single_label_128x128_split_3/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Intialize W&B run\nrun = wandb.init(entity='ayush-thakur', project='hpa', job_type=f'dataset-creation-single-label_split_{SPLIT}')\n# Using `single_label.csv` file\nartifact_csv = run.use_artifact('ayush-thakur/hpa/single_label_split:v0', type='dataset')\n\n# Create W&B Table\ntable = wandb.Table(['image_id', 'image_cell', 'image_protein', 'label'])\n\nWAND_LOG_AT = 150\nNUM_IMGS = 0\n\nfor i in tqdm(range(len(df))):\n    # Get image ID\n    image_id = re.findall(r'[^\\/]+(?=\\_.)', mt[i])[0]\n    \n    # LOAD IMAGES\n    \n    # Get all channel images.\n    microtubule = np.array(Image.open(mt[i]))\n    endoplasmicrec = np.array(Image.open(er[i]))\n    nuclei = np.array(Image.open(nu[i]))\n    protein = np.array(Image.open(pr[i]))\n    \n    # Stack the channels to form image. Using Red, Yellow and Blue.\n    image = np.dstack((microtubule, endoplasmicrec, nuclei))\n\n    # PERFORM SEGMENTATION\n    \n    # For nuclei segmentation only blue channel is required.\n    nuc_segmentation = segmentator.pred_nuclei([nu[i]])\n    # For full cells all the three reference(except green) channels are required.\n    cell_segmentation = segmentator.pred_cells([[mt[i]], [er[i]], [nu[i]]])\n    # get cell mask\n    nuclei_mask, cell_mask = label_cell(nuc_segmentation[0], cell_segmentation[0])\n\n    # GET IMDIVIDUAL CELLS\n    \n    # Count the number of cells.\n    cells = np.unique(cell_mask)\n    nuclei = np.unique(nuclei_mask)\n\n    cell_count = 0\n    for cell_index in cells[1:]:\n        # Get cell and nucleus mask for one cell.\n        single_cell_mask = np.where(cell_mask==cell_index, 1,0).astype('uint8')\n        if cell_index in nuclei:\n            nucleus_mask = np.where(nuclei_mask==cell_index, 1,0).astype('uint8')\n        else:\n            continue\n        \n        # get contour for cell and nucleus\n        cell_cnts, _ = cv2.findContours(single_cell_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)\n        nucleus_cnts, _ = cv2.findContours(nucleus_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)\n\n        # Check if the nucleus is touching the boundary of image.\n        if not is_border_nuclei(nucleus_cnts[0]): # If not touching the boundary \n            # Mask the cell to be cropped.\n            cell = cv2.bitwise_and(image, image, mask=single_cell_mask)\n            # Mask the protein to be cropped.\n            cell_protein = cv2.bitwise_and(protein, protein, mask=nucleus_mask)\n            \n            # Get bounding box covering the contour. \n            x,y,w,h = cv2.boundingRect(cell_cnts[0])\n            # Crop cell\n            cell = cell[y:y+h, x:x+w]\n            # Resize cell\n            cell = cv2.resize(cell, (IMAGE_HEIGHT, IMAGE_WIDTH), cv2.INTER_AREA)\n            \n            # Crop protein\n            cell_protein = cell_protein[y:y+h, x:x+w]\n            # Resize protein\n            cell_protein = cv2.resize(cell_protein, (IMAGE_HEIGHT, IMAGE_WIDTH), cv2.INTER_AREA)\n            \n            # Save images\n            cv2.imwrite(SAVE_DIR+'rgb/'+image_id+f'_{cell_count}.png', cell)\n            cv2.imwrite(SAVE_DIR+'protein/'+image_id+f'_{cell_count}.png', cell_protein)\n            \n            # Logging every 200th generated image.\n            if NUM_IMGS%WAND_LOG_AT==0:\n                table.add_data(\n                    image_id+f'_{cell_count}',\n                    wandb.Image(cell), \n                    wandb.Image(cell_protein),\n                    df.loc[df['ID']==image_id]['Label'].values[0]\n                )\n            \n            cell_count+=1\n            NUM_IMGS+=1\n       \n# LOG AS ARTIFACT\n\n# Put the table in an artifact and save it\ndataset_artifact = wandb.Artifact(f'single_label_split_{SPLIT}', type='dataset')\ndataset_artifact.add(table, 'dataset')\nrun.log_artifact(dataset_artifact)\n\n# Close W&B run\nrun.finish()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Copy Kaggle API token to ~/.kaggle\n! mkdir -p /root/.kaggle/\n! cp ../input/apitoken/kaggle.json /root/.kaggle/kaggle.json\n# Initialize dataset creation\n! kaggle datasets init -p /kaggle/tmp/hpa_single_label_128x128_split_3/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls /kaggle/tmp/hpa_single_label_128x128_split_3/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%bash\necho \"{\n  \\\"title\\\": \\\"HPA: Single Label RGB+Protein Dataset (3)\\\",\n  \\\"id\\\": \\\"ayuraj/SingleLabel128x128Split3\\\",\n  \\\"licenses\\\": [\n    {\n      \\\"name\\\": \\\"CC0-1.0\\\"\n    }\n  ]\n}\" > /kaggle/tmp/hpa_single_label_128x128_split_3/dataset-metadata.json","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!kaggle datasets create -p /kaggle/tmp/hpa_single_label_128x128_split_3/ -u --dir-mode tar\n# !kaggle datasets version -p /kaggle/tmp/hpa_single_label_128x128_split_0 -m \"add rgb images\"  --dir-mode tar","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}