{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Quick Exploratory Data Analysis for RANZCR CLiP - Catheter and Line Position Challenge challenge**\n\nIn this competition, you’ll detect the presence and position of catheters and lines on chest x-rays. Use machine learning to train and test your model on 40,000 images to categorize a tube that is poorly placed."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!pip install -q -U pip\n!pip install -q -U seaborn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Importing Libraries**"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport ast\nimport random\n\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport matplotlib.pyplot as plt\nfrom matplotlib_venn import venn2, venn3\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BASE_DIR = \"../input/ranzcr-clip-catheter-line-classification/\"\nprint(os.listdir(BASE_DIR))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(os.path.join(BASE_DIR, \"train.csv\"), index_col=0)\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**StudyInstanceUID** - unique ID for each image <br>\n**ETT** - Abnormal - endotracheal tube placement abnormal<br>\n**ETT** - Borderline - endotracheal tube placement borderline abnormal<br>\n**ETT** - Normal - endotracheal tube placement normal<br>\n**NGT** - Abnormal - nasogastric tube placement abnormal<br>\n**NGT** - Borderline - nasogastric tube placement borderline abnormal<br>\n**NGT** - Incompletely Imaged - nasogastric tube placement inconclusive due to imaging<br>\n**NGT** - Normal - nasogastric tube placement borderline normal<br>\n**CVC** - Abnormal - central venous catheter placement abnormal<br>\n**CVC** - Borderline - central venous catheter placement borderline abnormal<br>\n**CVC** - Normal - central venous catheter placement normal<br>\nSwan Ganz Catheter Present<br>\n**PatientID** - unique ID for each patient in the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(8, 8))\ndf_tmp = df_train.iloc[:, :-1].sum()\nsns.barplot(x=df_tmp.values, y=df_tmp.index)\nplt.xticks(fontsize=14)\nplt.yticks(fontsize=14)\nplt.xlabel(\"Number of images\", fontsize=15)\nplt.title(\"Distribution of labels\", fontsize=16);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of unique patients: \", df_train[\"PatientID\"].unique().shape[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\ndf_tmp = df_train[\"PatientID\"].value_counts()\nsns.countplot(x=df_tmp.values)\nplt.xticks(fontsize=12, rotation=90)\nplt.yticks(fontsize=14)\nplt.xlabel(\"Number of observations\", fontsize=15)\nplt.ylabel(\"Number of patients\", fontsize=15)\nplt.title(\"Distribution of observations by PatientID\", fontsize=16);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Annotations**"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_annot = pd.read_csv(os.path.join(BASE_DIR, \"train_annotations.csv\"))\ndf_annot.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_image_with_annotations(row_ind):\n    row = df_annot.iloc[row_ind]\n    image_path = os.path.join(BASE_DIR, \"train\", row[\"StudyInstanceUID\"] + \".jpg\")\n    label = row[\"label\"]\n    data = np.array(ast.literal_eval(row[\"data\"]))\n    \n    plt.figure(figsize=(10, 5))\n    image = cv2.imread(image_path)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    plt.subplot(1, 2, 1)\n    plt.imshow(image)\n    plt.subplot(1, 2, 2)\n    plt.imshow(image)\n    plt.scatter(data[:, 0], data[:, 1])\n    \n    plt.suptitle(label, fontsize=15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_image_with_annotations(8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef visualize_annotations(file_id):\n    plt.figure(figsize=(8, 8))\n    \n    image = cv2.imread(os.path.join(BASE_DIR, \"train\", file_id + \".jpg\"))\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    plt.imshow(image)\n    \n    df_patient = df_annot.loc[df_annot[\"StudyInstanceUID\"] == file_id]\n    \n    if df_patient.shape[0]:        \n        labels = df_patient[\"label\"].values.tolist()\n        lines = df_patient[\"data\"].apply(ast.literal_eval).values.tolist()\n\n        for line, label in zip(lines, labels):         \n            line = np.asarray(line)\n            plt.scatter(line[:, 0], line[:, 1], s=40, label=label)\n        \n        plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0, prop={'size': 20})\n        \n    plt.tick_params(axis=\"x\", labelsize=15)\n    plt.tick_params(axis=\"y\", labelsize=15)\n    \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_ids = [\n    \"1.2.826.0.1.3680043.8.498.83331936392921199432218327504041001669\",\n    \"1.2.826.0.1.3680043.8.498.11693509889426445054876979814173446281\",\n    \"1.2.826.0.1.3680043.8.498.15159015355212130418020059688126994534\",\n    \"1.2.826.0.1.3680043.8.498.92067938763801985117661596637576203997\",\n]\n\nfor image_id in image_ids:\n    visualize_annotations(image_id)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **ETT Abnormal**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_batch(image_ids):\n    plt.figure(figsize=(16, 10))\n    \n    for ind, image_id in enumerate(image_ids):\n        plt.subplot(2, 3, ind + 1)\n        image = cv2.imread(os.path.join(BASE_DIR, \"train\", f\"{image_id}.jpg\"))\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n        plt.imshow(image)\n        plt.axis(\"off\")\n    \n    plt.show()\n\n    \ndef plot_statistics(df, col):\n    plt.figure(figsize=(16, 2))\n    sns.countplot(y=df[col])\n    \n    plt.xticks(fontsize=12)\n    plt.yticks(fontsize=12)\n    plt.xlabel(\"Number of observations\", fontsize=15)\n    plt.ylabel(col, fontsize=15)\n    plt.title(f\"Distribution of {col}\", fontsize=16);\n    \n    plt.show()\n    \ndef process_class(col_name):\n    plot_statistics(df_train, col_name)\n    tmp_df = df_train[df_train[col_name] == 1]\n    visualize_batch(random.sample(tmp_df.index.tolist(), 6))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"ETT - Abnormal\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_annotations(\"1.2.826.0.1.3680043.8.498.93345761486297843389996628528592497280\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"ETT - Borderline\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"ETT - Normal\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Abnormal**"},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"NGT - Abnormal\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Borderline**"},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"NGT - Borderline\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Incomplete Images**"},{"metadata":{"trusted":true},"cell_type":"code","source":"process_class(\"NGT - Incompletely Imaged\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Venn Diagrams**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_venn2(col_1, col_2):\n    plt.figure(figsize=(6, 6))\n    \n    area_10 = df_train[col_1].sum()\n    area_01 = df_train[col_2].sum()\n    area_11 = df_train[(df_train[col_1] == 1) & (df_train[col_2] == 1)].shape[0]\n\n    venn2(\n        subsets=(area_10, area_01, area_11), \n        set_labels=(col_1, col_2),\n        alpha=0.5,\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_venn2(\"ETT - Abnormal\", \"NGT - Abnormal\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_venn2(\"ETT - Abnormal\", \"CVC - Abnormal\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_venn3(col_1, col_2, col_3):\n    plt.figure(figsize=(6, 6))\n    \n    area_100 = df_train[col_1].sum()\n    area_010 = df_train[col_2].sum()\n    area_110 = df_train[(df_train[col_1] == 1) & (df_train[col_2] == 1)].shape[0]\n    area_001 = df_train[col_3].sum()\n    area_101 = df_train[(df_train[col_1] == 1) & (df_train[col_3] == 1)].shape[0]\n    area_011 = df_train[(df_train[col_2] == 1) & (df_train[col_3] == 1)].shape[0]\n    area_111 = df_train[(df_train[col_1] == 1) & (df_train[col_2] == 1) & (df_train[col_3] == 1)].shape[0]\n\n#     print(area_100, area_010, area_110, area_001, area_101, area_011, area_111)\n\n    venn3(\n        subsets=(area_100, area_010, area_110, area_001, area_101, area_011, area_111), \n        set_labels=(col_1, col_2, col_3), \n        alpha=0.5\n    );","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_venn3(\n    \"ETT - Abnormal\",\n    \"NGT - Abnormal\",\n    \"CVC - Abnormal\",\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_venn3(\n    \"ETT - Normal\",\n    \"NGT - Normal\",\n    \"CVC - Normal\",\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_venn3(\n    \"ETT - Borderline\",\n    \"NGT - Borderline\",\n    \"CVC - Borderline\",\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submission = pd.read_csv(os.path.join(BASE_DIR, \"sample_submission.csv\"), index_col=0)\ndf_submission","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submission.to_csv(\"submission.csv\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}