{"cells":[{"metadata":{"_uuid":"2d6da9673d8622bb6edbadb79b98b77263fc1d70"},"cell_type":"markdown","source":"# Human Protein Atlas Competition\n\nUpdate to the correlation matrix in Allunia's analysis"},{"metadata":{"_uuid":"cce6944f02a874a8773859db189ab119393822ac"},"cell_type":"markdown","source":"## Loading packages and data"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom PIL import Image\nfrom scipy.misc import imread\n\nimport tensorflow as tf\nsns.set()\n\nimport os\nprint(os.listdir(\"../input\"))\n\nimport warnings\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\nwarnings.filterwarnings(\"ignore\", category=UserWarning)\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_labels = pd.read_csv(\"../input/human-protein-atlas-image-classification/train.csv\")\ntrain_labels.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"882bba3d7f68f209051fea3899f046bd96dc2915"},"cell_type":"markdown","source":"## Helper code"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"beaac70d0392fab7c5b1d49649a8c994310320e4"},"cell_type":"code","source":"label_names = {\n    0:  \"Nucleoplasm\",  \n    1:  \"Nuclear membrane\",   \n    2:  \"Nucleoli\",   \n    3:  \"Nucleoli fibrillar center\",   \n    4:  \"Nuclear speckles\",\n    5:  \"Nuclear bodies\",   \n    6:  \"Endoplasmic reticulum\",   \n    7:  \"Golgi apparatus\",   \n    8:  \"Peroxisomes\",   \n    9:  \"Endosomes\",   \n    10:  \"Lysosomes\",   \n    11:  \"Intermediate filaments\",   \n    12:  \"Actin filaments\",   \n    13:  \"Focal adhesion sites\",   \n    14:  \"Microtubules\",   \n    15:  \"Microtubule ends\",   \n    16:  \"Cytokinetic bridge\",   \n    17:  \"Mitotic spindle\",   \n    18:  \"Microtubule organizing center\",   \n    19:  \"Centrosome\",   \n    20:  \"Lipid droplets\",   \n    21:  \"Plasma membrane\",   \n    22:  \"Cell junctions\",   \n    23:  \"Mitochondria\",   \n    24:  \"Aggresome\",   \n    25:  \"Cytosol\",   \n    26:  \"Cytoplasmic bodies\",   \n    27:  \"Rods & rings\"\n}\n\nreverse_train_labels = dict((v,k) for k,v in label_names.items())\n\ndef fill_targets(row):\n    row.Target = np.array(row.Target.split(\" \")).astype(np.int)\n    for num in row.Target:\n        name = label_names[int(num)]\n        row.loc[name] = 1\n    return row","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"707280dec8e4b10dc743b5473ba8d1c4492141ff"},"cell_type":"markdown","source":"## Which proteins occur most often in images?"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"13a1f4fd6b594cd4a225f6d79d966096d1c800ae"},"cell_type":"code","source":"for key in label_names.keys():\n    train_labels[label_names[key]] = 0","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"3662893c91d0f74fe666300c214f67cfb03060c7","scrolled":false},"cell_type":"code","source":"train_labels = train_labels.apply(fill_targets, axis=1)\ntrain_labels.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c81f42c28ef8c3e01433e574880fc2955d5da4f5"},"cell_type":"code","source":"# path to directory where training images are held\ntrain_path = '../input/human-protein-atlas-image-classification/train/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8dee0a26a65ef38060d824bc5ad62cbc09907ee1"},"cell_type":"code","source":"for color in ['red', 'green', 'blue', 'yellow']:\n    train_labels[f'{color}_filename'] = \\\n        (train_labels['Id']\n         .transform(\n             lambda id: f'{id}_{color}.png'\n         )\n        )\n    train_labels[f'{color}_path'] = \\\n        (train_labels[f'{color}_filename']\n         .transform(\n             lambda path: os.path.join(train_path, path)\n         )\n        )\n#     Can't load all into memory at once, unfortunately\n#     train_labels[f'{color}_img'] = (train_labels[f'{color}_path']\n#                                     .transform(\n#                                         lambda path: Image.open(path)\n#                                     )\n#                                    )\ntrain_labels.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cb957f819453857d62a96a236febabcfb9ce5a22"},"cell_type":"markdown","source":"## Which targets are correlated?\n\nLet's see if we find some correlations between our targets. This way we may already see that some proteins often come together."},{"metadata":{"trusted":true,"_uuid":"b3c0f5a77e34296416136ca2800b8623f94885d8","_kg_hide-input":true,"scrolled":true},"cell_type":"code","source":"plt.figure(figsize=(15,15))\n# Allunia's notebook had an additional slice for train_labels.number_of_targets>1,\n# but then we miss the fact that Endosomes and Lysosomes are not fully correlated\n# because some images are only tagged \"Lysosome\"\nlabels = [label for label in label_names.values()]\nsns.heatmap(train_labels[labels].corr(),\n            cmap=\"RdYlBu\",\n            vmin=-1,\n            vmax=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c8083f38bf943a2d38c86544cf322ffe721d8a19"},"cell_type":"markdown","source":"This is slightly different to Allunia's original heatmap (shown below), which had an additional slice for train_labels.number_of_targets>1. This means it looks like Endosomes and Lysosomes are fully correlated, (because the images which are only tagged \"Lysosome\" are removed)."},{"metadata":{"trusted":true,"_uuid":"93a8640961cbd33c4c43fcdd691d5312e41ee422"},"cell_type":"code","source":"plt.figure(figsize=(15,15))\ntrain_labels[\"number_of_targets\"] = train_labels.drop([\"Id\", \"Target\"],axis=1).sum(axis=1)\nsns.heatmap(train_labels[train_labels.number_of_targets>1][labels].corr(),\n            cmap=\"RdYlBu\",\n            vmin=-1,\n            vmax=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"86c06ed4c0dbf02c605e447a5ecc40373b35771a","_kg_hide-input":true},"cell_type":"code","source":"def find_counts(special_target, labels):\n    counts = labels[labels[special_target] == 1][[label for label in label_names.values()]].sum(axis=0)\n    counts = counts[counts > 0]\n    counts = counts.sort_values()\n    return counts","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7873bc72f3093f2d173e8b1aa47b8e241d5bec39"},"cell_type":"markdown","source":"When we check which proteins are present in images tagged \"Lysosymes\", it looks like Endosomes and Lysosymes are fully correlated:"},{"metadata":{"trusted":true,"_uuid":"e2af724b721a15ebd294771629fb270c9ad9f0ae","_kg_hide-input":true,"scrolled":true},"cell_type":"code","source":"lyso_counts = find_counts(\"Lysosomes\", train_labels)\n\nplt.figure(figsize=(10,3))\nsns.barplot(x=lyso_counts.index.values, y=lyso_counts.values, palette=\"Blues\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"510ada389acc39d6eab30fa9ab89944d3dafe9b8"},"cell_type":"markdown","source":"However, when we find \"Endosomes\" instead of \"Lysosymes\" we see the full picture:"},{"metadata":{"trusted":true,"_uuid":"3e5150faef19f3a28bc84447887ed7d5f9ca90c6"},"cell_type":"code","source":"endo_counts = find_counts(\"Endosomes\", train_labels)\n\nplt.figure(figsize=(10,3))\nsns.barplot(x=endo_counts.index.values, y=endo_counts.values, palette=\"Blues\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d9c5f6135c3412814c9ba33fc9b42d29ab86c0f0"},"cell_type":"markdown","source":"So **every image with lysosomes also contains endosomes**, but **every image with endosomes does not contain lysosomes**. For example, the following image is tagged with \"endosomes\" but not \"lysosomes\":"},{"metadata":{"trusted":true,"_uuid":"e683689598742ae0494f6004c3c8baf6b21ce0b3"},"cell_type":"code","source":"mask = (train_labels[\"Endosomes\"]==1) & (train_labels[\"Lysosomes\"]==0)\nexample_index = train_labels[mask].index[0]\nr = np.array(Image.open(train_labels.loc[example_index, \"red_path\"]))\ng = np.array(Image.open(train_labels.loc[example_index, \"green_path\"]))\nb = np.array(Image.open(train_labels.loc[example_index, \"blue_path\"]))\ny = np.array(Image.open(train_labels.loc[example_index, \"yellow_path\"]))\nfig, ax = plt.subplots(1,4)\n\nax[0].imshow(r, cmap=plt.cm.Reds)\nax[0].set_axis_off()\n\nax[1].imshow(g, cmap=plt.cm.Greens)\nax[1].set_axis_off()\n\nax[2].imshow(b, cmap=plt.cm.Blues)\nax[2].set_axis_off()\n\nax[3].imshow(y, cmap=plt.cm.Oranges)\nax[3].set_axis_off()\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}