{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#  **Projet - Reconnaissance d’Image**","metadata":{}},{"cell_type":"markdown","source":"# Table of contents <a id='0.1'></a>\n\n1. [Introduction](#0)\n1. [L'objectif](#1)\n2. [Import Packages](#2)\n3. [Analyse des données](#3)\n5. [Analyse et benchmark des modèles existants](#4)\n5. [Choix du modèle](#5)\n6. [Paramétrage et architecture du modèle](#6)\n6. [Analyse du résultat du modèle choisi](#7)\n","metadata":{}},{"cell_type":"markdown","source":"# 1. <a id='1'>Introduction📒</a>\n[Table of contents](#0.1)\n\n","metadata":{}},{"cell_type":"markdown","source":"**Type de problèmatique : Instance classification**\n\n**Le choix du dataset** : [Human Protein Atlas - Single Cell Classification](https://www.kaggle.com/c/hpa-single-cell-image-classification/data) \n\nLe competition vise à résoudre le défi de la classification des images unicellulaires qui nous aidera à caractériser l'hétérogénéité unicellulaire dans la collection d'images en générant des annotations plus précises des localisations subcellulaires pour des milliers de protéines humaines dans des cellules individuelles.\n\n\n* C'est un problème de classification multi label. Nous avons des images de cellules microscopiques et des labels correspondantes à l'emplacement des protéines attribuées pour chaque cellule de l'image.\n\n* Il y a au total 19 labels différentes présentes dans l'ensemble de données (18 labels pour des emplacements spécifiques et le label 18 pour un signal négatif/non spécifique).\n\n* Pour chaque échantillon, nous avons 4 fichiers image. Chaque fichier représente un filtre différent sur les modèles de protéines subcellulaires représentés par l'échantillon. Les couleurs sont le rouge pour les channels microtubulaires, le bleu pour les channels noyaux, le jaune pour les channels du réticulum endoplasmique (ER) et le vert pour la protéine d'intérêt.","metadata":{}},{"cell_type":"markdown","source":"# 2. <a id='1'>L'objectif 📃</a>\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"markdown","source":"**Que prévoyons-nous?**\n\nOn predit les étiquettes de localisation des organites protéiques pour chaque cellule de l'image.","metadata":{}},{"cell_type":"markdown","source":"# 3. <a id='2'>Import Packages📚</a>\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\nfrom PIL import Image\nimport os\nfrom tqdm.notebook import tqdm\nimport seaborn as sns","metadata":{"execution":{"iopub.status.busy":"2021-05-25T09:07:55.879959Z","iopub.execute_input":"2021-05-25T09:07:55.880429Z","iopub.status.idle":"2021-05-25T09:07:56.255545Z","shell.execute_reply.started":"2021-05-25T09:07:55.880321Z","shell.execute_reply":"2021-05-25T09:07:56.253967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. <a id='5'>Analyse🧬</a>\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"code","source":"train_labels = pd.read_csv(\"../input/hpa-single-cell-image-classification/train.csv\")\ndisplay(train_labels.head())\nprint(f\"df.shape: {train_labels.shape}\")","metadata":{"execution":{"iopub.status.busy":"2021-05-25T09:07:56.796495Z","iopub.execute_input":"2021-05-25T09:07:56.7969Z","iopub.status.idle":"2021-05-25T09:07:56.846771Z","shell.execute_reply.started":"2021-05-25T09:07:56.796863Z","shell.execute_reply":"2021-05-25T09:07:56.845449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'Valeurs manquantes dans train_labels.csv :\\n{train_labels.isnull().sum()}')","metadata":{"execution":{"iopub.status.busy":"2021-05-25T09:07:57.307593Z","iopub.execute_input":"2021-05-25T09:07:57.308Z","iopub.status.idle":"2021-05-25T09:07:57.32228Z","shell.execute_reply.started":"2021-05-25T09:07:57.307956Z","shell.execute_reply":"2021-05-25T09:07:57.320561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Valeurs uniques dans chaque colonne de train_labels.csv')\nfor col in train_labels:\n    print(f'{col}: {train_labels[col].nunique()}')","metadata":{"execution":{"iopub.status.busy":"2021-05-25T09:07:58.072505Z","iopub.execute_input":"2021-05-25T09:07:58.072912Z","iopub.status.idle":"2021-05-25T09:07:58.095942Z","shell.execute_reply.started":"2021-05-25T09:07:58.072876Z","shell.execute_reply":"2021-05-25T09:07:58.094808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**📌 Observations**\n\nNous avons les features suivantes dans train.csv :\n\n   * ID - Le nom de fichier de base de l'échantillon. Tous les échantillons se composent de quatre fichiers - bleu, vert, rouge et jaune.\n   * Label - Ceci représente les labels attribuées à chaque échantillon.\n   \nOn a 21806 lignes et 2 colonnes.\n\nIl n'y a pas des valeurs manquantes dans train.","metadata":{"execution":{"iopub.status.busy":"2021-05-22T13:02:35.948558Z","iopub.execute_input":"2021-05-22T13:02:35.949018Z","iopub.status.idle":"2021-05-22T13:02:35.955488Z","shell.execute_reply.started":"2021-05-22T13:02:35.94898Z","shell.execute_reply":"2021-05-22T13:02:35.954245Z"}}},{"cell_type":"code","source":"def get_image(ddir, filename):\n    r = Image.open(f'{ddir}/{filename}_red.png')\n    g = Image.open(f'{ddir}/{filename}_green.png')\n    b = Image.open(f'{ddir}/{filename}_blue.png')\n    y = Image.open(f'{ddir}/{filename}_yellow.png')\n    return r, g, b, y\n\n\ndef display_image(image, ax):\n    [a.axis('off') for a in ax]\n    r, g, b, y = image\n    ax[0].imshow(r,cmap='Reds')\n    ax[0].set_title('Microtubules')\n    ax[1].imshow(g,cmap='Greens')\n    ax[1].set_title('Protein of Interest')\n    ax[2].imshow(b,cmap='Blues')\n    ax[2].set_title('Nucleus')\n    ax[3].imshow(y,cmap='Oranges') \n    ax[3].set_title('Endoplasmic Reticulum')\n    return ax","metadata":{"execution":{"iopub.status.busy":"2021-05-25T09:07:59.438275Z","iopub.execute_input":"2021-05-25T09:07:59.438697Z","iopub.status.idle":"2021-05-25T09:07:59.448035Z","shell.execute_reply.started":"2021-05-25T09:07:59.438658Z","shell.execute_reply":"2021-05-25T09:07:59.446755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"filename = train_labels.ID.sample(1, random_state=9473).values[0]\nimgs = get_image('../input/hpa-single-cell-image-classification/train', filename)\n\nfig, ax = plt.subplots(figsize=(15,5),nrows=1, ncols=4)\ndisplay_image(imgs, ax);","metadata":{"execution":{"iopub.status.busy":"2021-05-25T09:08:00.025486Z","iopub.execute_input":"2021-05-25T09:08:00.026607Z","iopub.status.idle":"2021-05-25T09:08:02.308027Z","shell.execute_reply.started":"2021-05-25T09:08:00.026558Z","shell.execute_reply":"2021-05-25T09:08:02.30657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Labels","metadata":{}},{"cell_type":"code","source":"train_labels[\"Label\"] = train_labels[\"Label\"].str.split(\"|\")\n\n# class labels\nclass_labels = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18']\n\n# binarizing each label/class\nfor label in tqdm(class_labels):\n    train_labels[label] = train_labels['Label'].map(lambda result: 1 if label in result else 0)\n\n# rename column\ntrain_labels.columns = ['ID', 'Label', 'Nucleoplasm', 'Nuclear membrane', 'Nucleoli', 'Nucleoli fibrillar center',\n                    'Nuclear speckles', 'Nuclear bodies', 'Endoplasmic reticulum', 'Golgi apparatus', 'Intermediate filaments',\n                    'Actin filaments', 'Microtubules', 'Mitotic spindle', 'Centrosome', 'Plasma membrane', 'Mitochondria',\n                    'Aggresome', 'Cytosol', 'Vesicles and punctate cytosolic patterns', 'Negative']\n\ntrain_labels","metadata":{"execution":{"iopub.status.busy":"2021-05-25T09:08:05.385084Z","iopub.execute_input":"2021-05-25T09:08:05.385711Z","iopub.status.idle":"2021-05-25T09:08:05.926705Z","shell.execute_reply.started":"2021-05-25T09:08:05.385658Z","shell.execute_reply":"2021-05-25T09:08:05.925596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels['Length'] = train_labels['Label'].str.len()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels['labels_string'] = train_labels['Label'].apply(lambda x: ','.join(map(str, x)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df=train_labels[train_labels['Length']==2]\nl=df.labels_string.value_counts()\nl","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df=df.drop(['ID', 'Label','Length','labels_string'],axis=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from matplotlib import pyplot as plt\n\nplt.figure(figsize=(17,30))\nwith sns.axes_style(\"whitegrid\"):\n    aa = sns.barplot(y=df.index.values, x=df.values, palette='mako')\n    plt.title(\"Label Distribution\")\n    plt.savefig('Label Distribution')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(14,12))\nclass_counts = train_labels.sum().drop(['ID', 'Label','Length','labels_string']).sort_values(ascending=False)\nwith sns.axes_style(\"whitegrid\"):\n    aa = sns.barplot(y=class_counts.index.values, x=class_counts.values, palette='mako')\n    plt.title(\"Label Distribution\")\nplt.savefig('Label Distribution')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for column in class_counts.keys():\n    print(f\"The class {column} has {train_labels[column].sum()} samples\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**📌 Observations**\n\n* Le nucléoplasme a le plus d'occurrence autour de 8797.\n* Les négatifs sont au moins seulement 34 échantillons avec un emplacement non spécifié.\n* La plupart des labels semblent avoir eu lieu moins de 2000.\n","metadata":{}},{"cell_type":"code","source":"from matplotlib import pyplot as plt\n\nlabel_per_image = train_labels.drop(['ID', 'Label'], axis=1).sum(axis=1)\n\nplt.figure(figsize=(16,10))\nwith sns.axes_style(\"whitegrid\"):\n    ax = sns.countplot(label_per_image, palette='Pastel2')\n    for p in ax.patches:\n        height = p.get_height()\n        ax.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(height/len(label_per_image)*100),\n                ha=\"center\", fontsize=12)\n    plt.title(\"Label Per Sample/Image\", fontsize=16)\n    plt.savefig('Label Per Sample/Image')","metadata":{"execution":{"iopub.status.busy":"2021-05-25T09:09:06.705301Z","iopub.execute_input":"2021-05-25T09:09:06.705677Z","iopub.status.idle":"2021-05-25T09:09:06.996572Z","shell.execute_reply.started":"2021-05-25T09:09:06.705644Z","shell.execute_reply":"2021-05-25T09:09:06.994699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**📌 Observations**\n\n* 48% des échantillons n'ont qu'une seule label, 40% ont 2 labels par image.\n* 10% des échantillons ont 3 labels.\n* Un très petit nombre d'échantillons semble avoir plus de 3 labels.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}