{"cells":[{"metadata":{},"cell_type":"markdown","source":"# <center> RANZCR Catheter and Line Position Challenge </center>\n## <center> Exploratory Data Analysis </center>"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"import os\nimport ast\n\nimport numpy as np\nimport pandas as pd\n\nimport cv2\n\nimport re\nregex = re.compile('\\d+')\n\nfrom matplotlib import pyplot as plt\n%matplotlib inline\n%config InlineBackend.figure_format = 'svg'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"BASE_DIR = '../input/ranzcr-clip-catheter-line-classification'\n\ndf_train = pd.read_csv(os.path.join(BASE_DIR, 'train.csv'))\ndf_annotations = pd.read_csv(os.path.join(BASE_DIR, 'train_annotations.csv'))\n\ndf_annotations['image_path'] = [os.path.join(BASE_DIR, 'train', x + '.jpg') for x in df_annotations['StudyInstanceUID']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Number of Samples\n\n- The number of samples (images) is ~30k, which is very small than the imagenet sample size (14M). So, transfer learning will be extremely beneficial in this competition."},{"metadata":{"trusted":true,"_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"df_train.shape[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Number of Targets"},{"metadata":{"trusted":true,"_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"targets = df_train.columns[1:-1]\nprint(list(targets))\nprint('\\nNumber of Targets: %d' % len(targets))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The target list includes 4 types of tubes that are seen in the X-ray images. Since the goal of the competition is to predict whether these tubes are placed normally, abnormally, or borderline, we better know where these tubes are inserted in the body and where they are directed to.\n\n`ETT:` Endotracheal Tube - A flexible plastic tube that is **placed through the mouth into the trachea (windpipe)** to help a patient breathe. The endotracheal tube is then connected to a ventilator, which delivers oxygen to the lungs.\n\n`NGT:` Nasogastric Tube - A flexible tube of rubber or plastic that is **passed through the nose, down through the esophagus, and into the stomach**. It can be used to either remove substances from or add them to the stomach. \n\n`CVC:` Central Venous Catheter - A thin, flexible tube that is **inserted into a vein, usually below the right collarbone (shoulders), and guided (threaded) into a large vein above the right side of the heart** called the superior vena cava.\n\n`Swan Ganz Catheter`: A thin tube that is usually **inserted in the neck or groin and directed through the veins and into the right side of the heart**."},{"metadata":{},"cell_type":"markdown","source":"## Target Co-occurance\n\n- The target co-occurance is not 1 for all samples as seen below, so this is a **MULTI-LABEL** classification problem.\n- 24 samples have no targets!"},{"metadata":{"trusted":true,"_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"df_train.head()\ndf_train[targets].sum(1).value_counts(sort=True).plot.bar()\nplt.xlabel('Co-occurance')\nplt.ylabel('Frequency')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"df_train[targets].sum(1).value_counts(sort=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Target Mean\n\n- There is a huge target imbalance.\n- The most frequent target is CVC-Normal, which is observed in 71% of the data. "},{"metadata":{"trusted":true,"_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"df_train[targets].mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Patient Frequency\n\n- There are 3255 unique patients. One particular patient is present as much as 172 times in the data. It may be a good idea to do cross validation by grouping the patients."},{"metadata":{"trusted":true,"_kg_hide-input":true,"scrolled":false},"cell_type":"code","source":"df_train['PatientID'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Are the label conditions mutually exclusive?"},{"metadata":{},"cell_type":"markdown","source":"- ETT can have **only** 1 of the 3 three conditions: Abnormal, Borderline, Normal, or None"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train[targets[:3]].value_counts().rename('Counts').reset_index()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- NGT can have 2 conditions at the same time but not 3! The cases with 2 conditions are only 25."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train[targets[4:7]].value_counts().rename('Counts').reset_index()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- CVC can have all combinations of the Abnormal, Borderline, and Normal."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train[targets[7:10]].value_counts().rename('Counts').reset_index()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Annotations\n\n- Annotation lengths are not same for all the images. They vary between 4 and 152. The distribution of the annodation lengths can be seen figure below."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_annotations['length'] = [len(regex.findall(df_annotations.loc[i, 'data'])) for i in range(len(df_annotations))]\nprint('Minimum annotation length:', df_annotations['length'].min())\nprint('Maximum annotation length:', df_annotations['length'].max())","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"ax = df_annotations['length'].hist(bins=70, grid=False)\nax.set_xlabel('Annotation Length')\nax.set_ylabel('Frequency')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Number of annotated tubes for a given image\n\n- Majority of the images with tube annotations have 1 annotated tube. However, tube annotations are observed as much as 6!\n- There are also images without the tube annotation data."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_annotations['StudyInstanceUID'].value_counts().value_counts().plot.bar()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_row(row=5):\n    colors = [(228, 26, 28), (55, 126, 184), (152, 78, 163), (255, 255, 51), (166, 86, 40), (247, 129, 191), (153, 153, 153)]\n    studyid = df_annotations.loc[row, 'StudyInstanceUID']\n    img = cv2.imread(df_annotations.loc[row, 'image_path'], 1)\n    \n    data = df_annotations[df_annotations['StudyInstanceUID'] == studyid]\n    print(data['label'])\n    for index, row in data.reset_index().iterrows():\n        pts = np.array(ast.literal_eval(row['data']), np.int32)\n        if row['label'][:3] == 'NGT':\n            color_index = 0\n        elif row['label'][:3] == 'CVC':\n            color_index = 1\n        elif row['label'][:3] == 'ETT':\n            color_index = 3\n        else:\n            color_index = 4\n        for i, pt in enumerate(pts):\n            if i < len(pts)-1:\n                cv2.line(img,tuple(pts[i]), tuple(pts[i+1]), colors[color_index], 30)\n    plt.imshow(img)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Some Example Images with Annotations"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_row(35)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_row(42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_row(310)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_row(325)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}