{"nbformat": 4, "nbformat_minor": 0, "metadata": {"language_info": {"mimetype": "text/x-python", "file_extension": ".py", "codemirror_mode": {"name": "ipython", "version": 3}, "name": "python", "pygments_lexer": "ipython3", "nbconvert_exporter": "python", "version": "3.6.0"}, "kernelspec": {"language": "python", "name": "python3", "display_name": "Python 3"}}, "cells": [{"cell_type": "markdown", "outputs": [], "source": "# Cervix EDA\n\nIn this competition we have a multi-class classification problem with **three** classes. We are asked, given an image, to identify the cervix type.\n\nFrom the data description:\n\n*In this competition, you will develop algorithms to correctly classify cervix types based on cervical images. These different types of cervix in our data set are all considered normal (not cancerous), but since the transformation zones aren't always visible, some of the patients require further testing while some don't. This decision is very important for the healthcare provider and critical for the patient. Identifying the transformation zones is not an easy task for the healthcare providers, therefore, an algorithm-aided decision will significantly improve the quality and efficiency of cervical cancer screening for these patients.*\n\nThe submission format is asking for a probability for each of the three different cervix types.\n\nIn this notebook we will be looking at:\n\n* basic dataset stats like number of samples per class, image sizes\n* different embeddings of RGB image space\n* pairwise distances and a clustermap of images in RGB space\n* (linear) model selection with basic multi class evaluation metrics.\n\n**If you like this kernel, please give an upvote, thanks! :)**", "execution_count": null, "metadata": {"_uuid": "9f86fbe5eeeefaf998ee495b15eca9220f2ba5bf", "_active": false, "_cell_guid": "419ad209-85bf-c4ef-d3d7-d8fa2b6be0ea"}}, {"cell_type": "code", "outputs": [], "source": "import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom skimage.io import imread, imshow\nimport cv2\n\n%matplotlib inline\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input/train\"]).decode(\"utf8\"))", "metadata": {"_uuid": "1f939161c9be15313457765d4b712c7eb29308af", "_execution_state": "idle", "_active": false, "trusted": false, "_cell_guid": "6f6e8717-2620-3fbc-dff4-7093bde04be9"}, "execution_count": 1}, {"cell_type": "markdown", "outputs": [], "source": "We are given training images for each of cervix types. Lets first count them for each class.", "execution_count": null, "metadata": {"_uuid": "c73f79b963328a575d4e81023dbcf3e62b742672", "_active": false, "_cell_guid": "a640fa62-669c-a08c-aa2b-20c4c713bcdd"}}, {"cell_type": "code", "outputs": [], "source": "from glob import glob\nbasepath = '../input/train/'\n\nall_cervix_images = []\n\nfor path in sorted(glob(basepath + \"*\")):\n    cervix_type = path.split(\"/\")[-1]\n    cervix_images = sorted(glob(basepath + cervix_type + \"/*\"))\n    all_cervix_images = all_cervix_images + cervix_images\n\nall_cervix_images = pd.DataFrame({'imagepath': all_cervix_images})\nall_cervix_images['filetype'] = all_cervix_images.apply(lambda row: row.imagepath.split(\".\")[-1], axis=1)\nall_cervix_images['type'] = all_cervix_images.apply(lambda row: row.imagepath.split(\"/\")[-2], axis=1)\nall_cervix_images.head()", "metadata": {"_uuid": "8f0b41fa4cb7a79c1d1aafd0fa99eccb3f385b6e", "_execution_state": "idle", "_active": false, "trusted": false, "_cell_guid": "df96bdda-f3bf-84f9-0abd-ee6161018846"}, "execution_count": 2}, {"cell_type": "markdown", "outputs": [], "source": "## Image types\n\nNow that we have the data in a handy dataframe we can do a few aggregations on the data. Let us first see how many images there are for each cervix type and which file types they have.\n\nAll files are in JPG format and Type 2 is the most common one with a little bit more than 50% in the training data in total, Type 1 on the other hand has a little bit less than 20% in the training data.", "execution_count": null, "metadata": {"_uuid": "5cee36a58d3ce589ee04a326179be47e487561b5", "_active": false, "_cell_guid": "172accdb-dc4d-17da-fa39-0740c32a54f9"}}, {"cell_type": "code", "outputs": [], "source": "print('We have a total of {} images in the whole dataset'.format(all_cervix_images.shape[0]))\ntype_aggregation = all_cervix_images.groupby(['type', 'filetype']).agg('count')\ntype_aggregation_p = type_aggregation.apply(lambda row: 1.0*row['imagepath']/all_cervix_images.shape[0], axis=1)\n\nfig, axes = plt.subplots(nrows=2, ncols=1, figsize=(10, 8))\n\ntype_aggregation.plot.barh(ax=axes[0])\naxes[0].set_xlabel(\"image count\")\ntype_aggregation_p.plot.barh(ax=axes[1])\naxes[1].set_xlabel(\"training size fraction\")", "metadata": {"_uuid": "fbd1d0d565ddc39040bc41aaac93d0718a6d794f", "_execution_state": "idle", "_active": false, "trusted": false, "_cell_guid": "1c42f54f-45fb-7a2e-6c89-ba7c7a6a48a3"}, "execution_count": null}, {"cell_type": "markdown", "outputs": [], "source": "Now, lets read the files for each type to get an idea about how the images look like.\n\nThe images seem to vary alot in they formats, the first two samples have only a circular area with the actual image, the last sample has the image in a rectangle.", "execution_count": null, "metadata": {"_uuid": "3d2524ebf161801f77d9d018ff66e6378796b0cf", "_active": false, "_cell_guid": "4b3581d5-c9a3-49e4-427c-08ef8ede1053"}}, {"cell_type": "code", "outputs": [], "source": "fig = plt.figure(figsize=(12,8))\n\ni = 1\nfor t in all_cervix_images['type'].unique():\n    ax = fig.add_subplot(1,3,i)\n    i+=1\n    f = all_cervix_images[all_cervix_images['type'] == t]['imagepath'].values[0]\n    plt.imshow(plt.imread(f))\n    plt.title('sample for cervix {}'.format(t))", "metadata": {"_uuid": "22ec67009bfb109a4429c3be3f38ba0119cd48b3", "_execution_state": "busy", "_active": false, "trusted": false, "_cell_guid": "0b500f12-8d54-4dab-a9d1-8bc2fb2de755"}, "execution_count": null}, {"cell_type": "markdown", "outputs": [], "source": "### Additional images", "execution_count": null, "metadata": {"_uuid": "876d4e7bdfbcf36a203b0771f803847b7d49af1c", "_active": false, "_cell_guid": "314a36cd-4c6d-9480-c983-4dcdd7d25d58"}}, {"cell_type": "code", "outputs": [], "source": "print(check_output([\"ls\", \"../input/additional\"]).decode(\"utf8\"))", "metadata": {"_uuid": "6e55194f3216d470947e92492f933ede17900466", "_execution_state": "busy", "_active": false, "trusted": false, "_cell_guid": "2e416358-e7db-f3ce-caba-5c00ff8fcd79"}, "execution_count": null}, {"cell_type": "code", "outputs": [], "source": "basepath = '../input/additional/'\n\nall_cervix_images_a = []\n\nfor path in sorted(glob(basepath + \"*\")):\n    cervix_type = path.split(\"/\")[-1]\n    cervix_images = sorted(glob(basepath + cervix_type + \"/*\"))\n    all_cervix_images_a = all_cervix_images_a + cervix_images\n\nall_cervix_images_a = pd.DataFrame({'imagepath': all_cervix_images_a})\nall_cervix_images_a['filetype'] = all_cervix_images_a.apply(lambda row: row.imagepath.split(\".\")[-1], axis=1)\nall_cervix_images_a['type'] = all_cervix_images_a.apply(lambda row: row.imagepath.split(\"/\")[-2], axis=1)\nall_cervix_images_a.head()", "metadata": {"_uuid": "15d5752a0228ad7545947742891ce26dc7aa8626", "_execution_state": "busy", "_active": false, "trusted": false, "_cell_guid": "8ff1cd5e-bca7-6893-9243-1b5a453c9a88"}, "execution_count": null}, {"cell_type": "code", "outputs": [], "source": "print('We have a total of {} images in the whole dataset'.format(all_cervix_images_a.shape[0]))\ntype_aggregation = all_cervix_images_a.groupby(['type', 'filetype']).agg('count')\ntype_aggregation_p = type_aggregation.apply(lambda row: 1.0*row['imagepath']/all_cervix_images_a.shape[0], axis=1)\n\nfig, axes = plt.subplots(nrows=2, ncols=1, figsize=(10, 8))\n\ntype_aggregation.plot.barh(ax=axes[0])\naxes[0].set_xlabel(\"image count\")\ntype_aggregation_p.plot.barh(ax=axes[1])\naxes[1].set_xlabel(\"training size fraction\")", "metadata": {"_uuid": "39cb7107f81a29867ae1dbf13aa5b3cb47d8b847", "_execution_state": "busy", "_active": false, "trusted": false, "_cell_guid": "699a7598-a61d-1ace-1d83-48ea5e00ae4e"}, "execution_count": null}, {"cell_type": "code", "outputs": [], "source": "fig = plt.figure(figsize=(12,8))\n\ni = 1\nfor t in all_cervix_images_a['type'].unique():\n    ax = fig.add_subplot(1,3,i)\n    i+=1\n    f = all_cervix_images_a[all_cervix_images_a['type'] == t]['imagepath'].values[0]\n    plt.imshow(plt.imread(f))\n    plt.title('sample for cervix {}'.format(t))", "metadata": {"_uuid": "27785d5c53b7c08a300465d618ef63ce371cad4c", "_execution_state": "busy", "_active": false, "trusted": false, "_cell_guid": "18dc9c71-96e7-1881-fb18-f49c70a7c66f"}, "execution_count": null}, {"cell_type": "markdown", "outputs": [], "source": "### All images", "execution_count": null, "metadata": {"_uuid": "451d3a2fd1866940dd47d311d6b03a9879188a9a", "_active": false, "_cell_guid": "96bdc992-d7ee-c480-3db2-6836dc6ff344"}}, {"cell_type": "code", "outputs": [], "source": "all_cervix_images_ = pd.concat( [all_cervix_images, all_cervix_images_a], join='outer' )\nprint(all_cervix_images_)", "metadata": {"trusted": false, "_execution_state": "busy", "_active": false, "_cell_guid": "3ed33384-d62a-1d5c-38cf-bcd928a82991", "collapsed": false, "_uuid": "4acde09db1c09046aeb8583d824dbc149e54e833"}, "execution_count": null}, {"cell_type": "code", "outputs": [], "source": "print('We have a total of {} images in the whole dataset'.format(all_cervix_images_.shape[0]))\ntype_aggregation = all_cervix_images_.groupby(['type', 'filetype']).agg('count')\ntype_aggregation_p = type_aggregation.apply(lambda row: 1.0*row['imagepath']/all_cervix_images_a.shape[0], axis=1)\n\nfig, axes = plt.subplots(nrows=2, ncols=1, figsize=(10, 8))\n\ntype_aggregation.plot.barh(ax=axes[0])\naxes[0].set_xlabel(\"image count\")\ntype_aggregation_p.plot.barh(ax=axes[1])\naxes[1].set_xlabel(\"training size fraction\")", "metadata": {"_uuid": "d68ca77d035fab221e9abc3ae9f517e987e7d9ef", "_execution_state": "busy", "_active": true, "trusted": false, "_cell_guid": "bf6cf862-f75c-1956-958b-6602d22d2d35"}, "execution_count": null}, {"cell_type": "code", "outputs": [], "source": "fig = plt.figure(figsize=(12,8))\n\ni = 1\nfor t in all_cervix_images_['type'].unique():\n    ax = fig.add_subplot(1,3,i)\n    i+=1\n    f = all_cervix_images_[all_cervix_images_['type'] == t]['imagepath'].values[0]\n    plt.imshow(plt.imread(f))\n    plt.title('sample for cervix {}'.format(t))", "metadata": {"_uuid": "9389f705bddb446d4d39538aadde7fc48dce8541", "_execution_state": "busy", "_active": false, "trusted": false, "_cell_guid": "4322cea3-8ffe-bc2f-ba8c-25f4f17ecef6"}, "execution_count": null}]}