{"cells":[{"metadata":{},"cell_type":"markdown","source":"# RANZCR CLiP - Catheter and Line Position Challenge - Exploratory Data Analysis\n\nQuick Exploratory Data Analysis for [RANZCR CLiP - Catheter and Line Position Challenge](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification) challenge    \n\nIn this competition, you’ll detect the presence and position of catheters and lines on chest x-rays. Use machine learning to train and test your model on 40,000 images to categorize a tube that is poorly placed."},{"metadata":{},"cell_type":"markdown","source":"![](https://storage.googleapis.com/kaggle-competitions/kaggle/23870/logos/header.png?t=2020-12-01-04-28-05)"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"top\"></a>\n\n<div class=\"list-group\" id=\"list-tab\" role=\"tablist\">\n<h3 class=\"list-group-item list-group-item-action active\" data-toggle=\"list\" style='color:white; background:#6E848D; border:0' role=\"tab\" aria-controls=\"home\"><center>Quick Navigation</center></h3>\n\n* [Overview](#1)\n* [Annotations](#2)\n    \n    \n    \n* [ETT - Abnormal](#4)\n* [ETT - Borderline](#5)\n* [ETT - Normal](#6)\n* [NGT - Abnormal](#7)\n* [NGT - Borderline](#8)\n* [NGT - Incompletely Imaged](#9)\n* [NGT - Normal](#10)\n* [CVC - Abnormal](#11)\n* [CVC - Borderline](#12)\n* [CVC - Normal](#13)\n* [Swan Ganz Catheter Present](#14)\n    \n\n* [Venn Diagrams](#50)\n    \n    \n* [Submission](#100)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"!pip install -q -U pip\n!pip install -q -U seaborn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"1\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>Overview<center><h2>"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport ast\nimport random\n\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport matplotlib.pyplot as plt\nfrom matplotlib_venn import venn2, venn3\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"BASE_DIR = \"../input/ranzcr-clip-catheter-line-classification/\"\nprint(os.listdir(BASE_DIR))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**train.csv** contains image IDs, binary labels, and patient IDs."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(os.path.join(BASE_DIR, \"train.csv\"), index_col=0)\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**StudyInstanceUID** - unique ID for each image   \n**ETT - Abnormal** - endotracheal tube placement abnormal   \n**ETT - Borderline** - endotracheal tube placement borderline abnormal   \n**ETT - Normal** - endotracheal tube placement normal   \n**NGT - Abnormal** - nasogastric tube placement abnormal   \n**NGT - Borderline** - nasogastric tube placement borderline abnormal   \n**NGT - Incompletely Imaged** - nasogastric tube placement inconclusive due to imaging   \n**NGT - Normal** - nasogastric tube placement borderline normal   \n**CVC - Abnormal** - central venous catheter placement abnormal   \n**CVC - Borderline** - central venous catheter placement borderline abnormal   \n**CVC - Normal** - central venous catheter placement normal   \n**Swan Ganz Catheter Present**   \n**PatientID** - unique ID for each patient in the dataset   "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(8, 8))\ndf_tmp = df_train.iloc[:, :-1].sum()\nsns.barplot(x=df_tmp.values, y=df_tmp.index)\nplt.xticks(fontsize=14)\nplt.yticks(fontsize=14)\nplt.xlabel(\"Number of images\", fontsize=15)\nplt.title(\"Distribution of labels\", fontsize=16);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of unique patients: \", df_train[\"PatientID\"].unique().shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\ndf_tmp = df_train[\"PatientID\"].value_counts()\nsns.countplot(x=df_tmp.values)\nplt.xticks(fontsize=12, rotation=90)\nplt.yticks(fontsize=14)\nplt.xlabel(\"Number of observations\", fontsize=15)\nplt.ylabel(\"Number of patients\", fontsize=15)\nplt.title(\"Distribution of observations by PatientID\", fontsize=16);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>Annotations<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"**train_annotations.csv** these are segmentation annotations for training samples that have them. They are included solely as additional information for competitors."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_annot = pd.read_csv(os.path.join(BASE_DIR, \"train_annotations.csv\"))\ndf_annot.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_image_with_annotations(row_ind):\n    row = df_annot.iloc[row_ind]\n    image_path = os.path.join(BASE_DIR, \"train\", row[\"StudyInstanceUID\"] + \".jpg\")\n    label = row[\"label\"]\n    data = np.array(ast.literal_eval(row[\"data\"]))\n    \n    plt.figure(figsize=(10, 5))\n    image = cv2.imread(image_path)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    plt.subplot(1, 2, 1)\n    plt.imshow(image)\n    plt.subplot(1, 2, 2)\n    plt.imshow(image)\n    plt.scatter(data[:, 0], data[:, 1])\n    \n    plt.suptitle(label, fontsize=15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_image_with_annotations(8)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# from https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/203353#1117642\n\ndef visualize_annotations(file_id):\n    plt.figure(figsize=(8, 8))\n    \n    image = cv2.imread(os.path.join(BASE_DIR, \"train\", file_id + \".jpg\"))\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    plt.imshow(image)\n    \n    df_patient = df_annot.loc[df_annot[\"StudyInstanceUID\"] == file_id]\n    \n    if df_patient.shape[0]:        \n        labels = df_patient[\"label\"].values.tolist()\n        lines = df_patient[\"data\"].apply(ast.literal_eval).values.tolist()\n\n        for line, label in zip(lines, labels):         \n            line = np.asarray(line)\n            plt.scatter(line[:, 0], line[:, 1], s=40, label=label)\n        \n        plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0, prop={'size': 20})\n        \n    plt.tick_params(axis=\"x\", labelsize=15)\n    plt.tick_params(axis=\"y\", labelsize=15)\n    \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_ids = [\n    \"1.2.826.0.1.3680043.8.498.83331936392921199432218327504041001669\",\n    \"1.2.826.0.1.3680043.8.498.11693509889426445054876979814173446281\",\n    \"1.2.826.0.1.3680043.8.498.15159015355212130418020059688126994534\",\n    \"1.2.826.0.1.3680043.8.498.92067938763801985117661596637576203997\",\n]\n\nfor image_id in image_ids:\n    visualize_annotations(image_id)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def visualize_batch(image_ids):\n    plt.figure(figsize=(16, 10))\n    \n    for ind, image_id in enumerate(image_ids):\n        plt.subplot(2, 3, ind + 1)\n        image = cv2.imread(os.path.join(BASE_DIR, \"train\", f\"{image_id}.jpg\"))\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n        plt.imshow(image)\n        plt.axis(\"off\")\n    \n    plt.show()\n\n    \ndef plot_statistics(df, col):\n    plt.figure(figsize=(16, 2))\n    sns.countplot(y=df[col])\n    \n    plt.xticks(fontsize=12)\n    plt.yticks(fontsize=12)\n    plt.xlabel(\"Number of observations\", fontsize=15)\n    plt.ylabel(col, fontsize=15)\n    plt.title(f\"Distribution of {col}\", fontsize=16);\n    \n    plt.show()\n#     \n#     \n#     \n    \n\ndef process_class(col_name):\n    plot_statistics(df_train, col_name)\n    tmp_df = df_train[df_train[col_name] == 1]\n    visualize_batch(random.sample(tmp_df.index.tolist(), 6))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"4\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>ETT - Abnormal<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"endotracheal tube placement abnormal"},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"ETT - Abnormal\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_annotations(\"1.2.826.0.1.3680043.8.498.93345761486297843389996628528592497280\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"5\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>ETT - Borderline<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"endotracheal tube placement borderline abnormal"},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"ETT - Borderline\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"6\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>ETT - Normal<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"endotracheal tube placement normal"},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"ETT - Normal\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"7\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>NGT - Abnormal<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"nasogastric tube placement abnormal"},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"NGT - Abnormal\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"8\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>NGT - Borderline<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"nasogastric tube placement borderline abnormal"},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"NGT - Borderline\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"9\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>NGT - Incompletely Imaged<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"nasogastric tube placement inconclusive due to imaging"},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"NGT - Incompletely Imaged\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"10\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>NGT - Normal<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"nasogastric tube placement borderline normal"},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"NGT - Normal\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"11\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>CVC - Abnormal<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"central venous catheter placement abnormal"},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"CVC - Abnormal\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"12\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>CVC - Borderline<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"central venous catheter placement borderline abnormal"},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"CVC - Borderline\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"13\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>CVC - Normal<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"central venous catheter placement normal"},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"CVC - Normal\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"14\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>Swan Ganz Catheter Present<center><h2>"},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"Swan Ganz Catheter Present\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"50\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>Venn Diagrams<center><h2>"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_venn2(col_1, col_2):\n    plt.figure(figsize=(6, 6))\n    \n    area_10 = df_train[col_1].sum()\n    area_01 = df_train[col_2].sum()\n    area_11 = df_train[(df_train[col_1] == 1) & (df_train[col_2] == 1)].shape[0]\n\n    venn2(\n        subsets=(area_10, area_01, area_11), \n        set_labels=(col_1, col_2),\n        alpha=0.5,\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_venn2(\"ETT - Abnormal\", \"NGT - Abnormal\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_venn2(\"ETT - Abnormal\", \"CVC - Abnormal\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_venn2(\"NGT - Abnormal\", \"CVC - Abnormal\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_venn3(col_1, col_2, col_3):\n    plt.figure(figsize=(6, 6))\n    \n    area_100 = df_train[col_1].sum()\n    area_010 = df_train[col_2].sum()\n    area_110 = df_train[(df_train[col_1] == 1) & (df_train[col_2] == 1)].shape[0]\n    area_001 = df_train[col_3].sum()\n    area_101 = df_train[(df_train[col_1] == 1) & (df_train[col_3] == 1)].shape[0]\n    area_011 = df_train[(df_train[col_2] == 1) & (df_train[col_3] == 1)].shape[0]\n    area_111 = df_train[(df_train[col_1] == 1) & (df_train[col_2] == 1) & (df_train[col_3] == 1)].shape[0]\n\n#     print(area_100, area_010, area_110, area_001, area_101, area_011, area_111)\n\n    venn3(\n        subsets=(area_100, area_010, area_110, area_001, area_101, area_011, area_111), \n        set_labels=(col_1, col_2, col_3), \n        alpha=0.5\n    );","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_venn3(\n    \"ETT - Abnormal\",\n    \"NGT - Abnormal\",\n    \"CVC - Abnormal\",\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_venn3(\n    \"ETT - Normal\",\n    \"NGT - Normal\",\n    \"CVC - Normal\",\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_venn3(\n    \"ETT - Borderline\",\n    \"NGT - Borderline\",\n    \"CVC - Borderline\",\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"100\"></a>\n<h2 style='background:#6E848D; border:0; color:white'><center>Submission<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"**sample_submission.csv** - a sample submission file in the correct format"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submission = pd.read_csv(os.path.join(BASE_DIR, \"sample_submission.csv\"), index_col=0)\ndf_submission","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submission.to_csv(\"submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}