{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Downloading HPA public data"},{"metadata":{},"cell_type":"markdown","source":"## Update \n\n**2021/02/26**\n\nI indicated which images are in trainset in the column `in_trainset`. Please remove them so you don't have to download twice.\n\nI have also added a column `Label_idx`. As I mentioned in [Single Cell Patterns](https://www.kaggle.com/lnhtrang/single-cell-patterns) notebook, we merged some classes for this competition. The function to convert from label names to indexes is also added if you want to double check."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import io\nimport os\nimport requests\nimport pathlib\nimport gzip\nimport imageio\nimport pandas as pd\n\n\ndef tif_gzip_to_png(tif_path):\n    '''Function to convert .tif.gz to .png and put it in the same folder\n    Eg. for working in local work station\n    '''\n    png_path = pathlib.Path(tif_path.replace('.tif.gz','.png'))\n    tf = gzip.open(tif_path).read()\n    img = imageio.imread(tf, 'tiff')\n    imageio.imwrite(png_path, img)\n    \ndef download_and_convert_tifgzip_to_png(url, target_path):    \n    '''Function to convert .tif.gz to .png and put it in the same folder\n    Eg. in Kaggle notebook\n    '''\n    r = requests.get(url)\n    f = io.BytesIO(r.content)\n    tf = gzip.open(f).read()\n    img = imageio.imread(tf, 'tiff')\n    imageio.imwrite(target_path, img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# All label names in the public HPA and their corresponding index. \nall_locations = dict({\n    \"Nucleoplasm\": 0,\n    \"Nuclear membrane\": 1,\n    \"Nucleoli\": 2,\n    \"Nucleoli fibrillar center\": 3,\n    \"Nuclear speckles\": 4,\n    \"Nuclear bodies\": 5,\n    \"Endoplasmic reticulum\": 6,\n    \"Golgi apparatus\": 7,\n    \"Intermediate filaments\": 8,\n    \"Actin filaments\": 9,\n    \"Focal adhesion sites\": 9,\n    \"Microtubules\": 10,\n    \"Mitotic spindle\": 11,\n    \"Centrosome\": 12,\n    \"Centriolar satellite\": 12,\n    \"Plasma membrane\": 13,\n    \"Cell Junctions\": 13,\n    \"Mitochondria\": 14,\n    \"Aggresome\": 15,\n    \"Cytosol\": 16,\n    \"Vesicles\": 17,\n    \"Peroxisomes\": 17,\n    \"Endosomes\": 17,\n    \"Lysosomes\": 17,\n    \"Lipid droplets\": 17,\n    \"Cytoplasmic bodies\": 17,\n    \"No staining\": 18\n})\n\n\ndef add_label_idx(df, all_locations):\n    '''Function to convert label name to index\n    '''\n    df[\"Label_idx\"] = None\n    for i, row in df.iterrows():\n        labels = row.Label.split(',')\n        idx = []\n        for l in labels:\n            if l in all_locations.keys():\n                idx.append(str(all_locations[l]))\n        if len(idx)>0:\n            df.loc[i,\"Label_idx\"] = \"|\".join(idx)\n            \n        print(df.loc[i,\"Label\"], df.loc[i,\"Label_idx\"])\n    return df\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"public_hpa_df = pd.read_csv('../input/publichpa-withcellline/kaggle_2021.tsv')\n# Remove all images overlapping with Training set\npublic_hpa_df = public_hpa_df[public_hpa_df.in_trainset == False]\n\n# Remove all images with only labels that are not in this competition\npublic_hpa_df = public_hpa_df[~public_hpa_df.Label_idx.isna()]\n\ncolors = ['blue', 'red', 'green', 'yellow']\ncelllines = ['A-431', 'A549', 'EFO-21', 'HAP1', 'HEK 293', 'HUVEC TERT2', 'HaCaT', 'HeLa', 'PC-3', 'RH-30', 'RPTEC TERT1', 'SH-SY5Y', 'SK-MEL-30', 'SiHa', 'U-2 OS', 'U-251 MG', 'hTCEpi']\npublic_hpa_df_17 = public_hpa_df[public_hpa_df.Cellline.isin(celllines)]\nlen(public_hpa_df), len(public_hpa_df_17)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"public_hpa_df.tail()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"! Warning: These are raw, full size images. Each channel is approximately 8MB. With ~~82495~~ 77668 images * 4 channels, everything amounts to around ~~2.6TB~~ 2.4TB of data.\n\nWe have 17 cell lines in the training set and test set. So if you want to download extra public data for training and downloading is too slow because of big size, you probably want to consider:\n- Downloading just 17 cell lines (67k images * 4 channels, instead of 77.6k images)\n- Sampling according to label (eg. You have a lot of Nucleoplasm and Cytosol in the training set already so maybe you just need more rare labels)\n- Using jpeg images. These were purposely created for visualization on HPA website. They are much smaller in size (you will loose some info compared to tif file, but maybe that's enough for your model. Your call). To download jpeg, simply change `.tif.gz` to `.jpg` in the url. For example: https://images.proteinatlas.org/10005/921_B9_1_blue.jpg\t"},{"metadata":{"trusted":true},"cell_type":"code","source":"save_dir = os.path.join(os.getcwd(),'publichpa')\nif not os.path.exists(save_dir):\n    os.makedirs(save_dir)\n    \nfor i, row in public_hpa_df[0:5].iterrows():\n    try:\n        img = row.Image\n        for color in colors:\n            img_url = f'{img}_{color}.tif.gz'\n            save_path = os.path.join(save_dir,  f'{os.path.basename(img)}_{color}.png')\n            download_and_convert_tifgzip_to_png(img_url, save_path)\n            print(f'Downloaded {img_url} as {save_path}')    \n    except:\n        print(f'failed to download: {img}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Using HPA segmentation tool\n\nFrom https://github.com/CellProfiling/HPA-Cell-Segmentation\n\nBelow is a demonstration of installing and using this segmentation tool on some of the downloaded images above."},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install https://github.com/CellProfiling/HPA-Cell-Segmentation/archive/master.zip","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import hpacellseg.cellsegmentator as cellsegmentator\nfrom hpacellseg.utils import label_cell, label_nuclei\nimport glob\n\nmt = glob.glob(save_dir + '/' + '*_red.png')\ner = [f.replace('red', 'yellow') for f in mt]\nnu = [f.replace('red', 'blue') for f in mt]\nimages = [mt, er, nu]\n\n\nNUC_MODEL = \"./nuclei-model.pth\"\nCELL_MODEL = \"./cell-model.pth\"\nsegmentator = cellsegmentator.CellSegmentator(\n    NUC_MODEL,\n    CELL_MODEL,\n    scale_factor=0.25,\n    device=\"cuda\",\n    padding=False,\n    multi_channel_model=True,\n)\n\n# For nuclei\nnuc_segmentations = segmentator.pred_nuclei(images[2])\n\n# For full cells\ncell_segmentations = segmentator.pred_cells(images)\n\n# post-processing\nfor i, pred in enumerate(cell_segmentations):\n    nuclei_mask, cell_mask = label_cell(nuc_segmentations[i], cell_segmentations[i])\n    FOVname = os.path.basename(mt[i]).replace('red','predictedmask')\n    imageio.imwrite(os.path.join(save_dir,FOVname), cell_mask) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualizing the segmentation masks we just predicted above\n\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nfig, ax = plt.subplots(1,3, figsize=(20,50))\nfor i in range(3):\n    microtubule = plt.imread(mt[i])    \n    endoplasmicrec = plt.imread(er[i])    \n    nuclei = plt.imread(nu[i])\n    mask = plt.imread(mt[i].replace('red','predictedmask'))\n    img = np.dstack((microtubule, endoplasmicrec, nuclei))\n    ax[i].imshow(img)\n    ax[i].imshow(mask, alpha=0.5)\n    ax[i].axis('off')\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}