{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Table Of Contents\n* [Intro](#section-one)\n    - [Check File Size](#chapter-one)\n    - [Check train file](#chapter-two)\n    - [Definitions of Variables](#chapter-three)\n    - [Data Distribution of Each Variable](#chapter-four)\n        + [Quick Visualization](#ch04_sub_chapter-one)\n    - [Background Knowledge](#chapter-five)\n        + [Endotracheal Tube](#ch05_sub_chapter-one)\n        + [Nasogastric Tube](#ch05_sub_chapter-two)\n        + [Central venous catheter](#ch05_sub_chapter-three)\n        + [Swan Ganz Catheter Presen](#ch05_sub_chapter-four)\n    - [Check train annotation file](#chapter-six)\n    - [Visualization of X-rays image](#chapter-seven)\n    \n"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"section-one\"></a>\n# Intro\n- Thanks to [RANZCR/resnext50_32x4d starter [training]](https://www.kaggle.com/yasufuminakama/ranzcr-resnext50-32x4d-starter-training)\n    + Please visit here and upvote"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\nimport pandas as pd\n\nfrom matplotlib import pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"chapter-one\"></a>\n## Check File Size\n- Check Each Size of Dataset Folder in this competition\n    + train_records = 4.5GB\n    + test_tfrecords = 0.5MB\n    + train (image data) = 6.5GB\n    + test (image data) = 0.8MB"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\ndef get_folder_size(file_directory):\n  # file_list = os.listdir(file_directory)\n  dir_sizes = {}\n  for r, d, f in os.walk(file_directory, False):\n      size = sum(os.path.getsize(os.path.join(r,f)) for f in f+d)\n      size += sum(dir_sizes[os.path.join(r,d)] for d in d)\n      dir_sizes[r] = size\n      print(\"{} is {} MB\".format(r, round(size/2**20), 2))      \n  \nbase_dir = '../input/ranzcr-clip-catheter-line-classification'\nget_folder_size(base_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"chapter-two\"></a>\n## Check train file\n- Let's descirbe train"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/ranzcr-clip-catheter-line-classification/train.csv', index_col = 0)\ntest = pd.read_csv('../input/ranzcr-clip-catheter-line-classification/sample_submission.csv', index_col = 0)\ndisplay(train.head())\ndisplay(test.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"chapter-three\"></a>\n## Definitions of Variables \n- What's inside data?\n    + StudyInstanceUID - unique ID for each image\n    + ETT - Abnormal - endotracheal tube placement abnormal\n    + ETT - Borderline - endotracheal tube placement borderline abnormal\n    + ETT - Normal - endotracheal tube placement normal\n    + NGT - Abnormal - nasogastric tube placement abnormal\n    + NGT - Borderline - nasogastric tube placement borderline abnormal\n    + NGT - Incompletely Imaged - nasogastric tube placement inconclusive due to imaging\n    + NGT - Normal - nasogastric tube placement borderline normal\n    + CVC - Abnormal - central venous catheter placement abnormal\n    + CVC - Borderline - central venous catheter placement borderline abnormal\n    + CVC - Normal - central venous catheter placement normal\n    + Swan Ganz Catheter Present(??)\n    + PatientID - unique ID for each patient in the dataset\n"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"chapter-four\"></a>\n### Data Distribution of Each Variable\n- why two calculations are different?\n    + When inserting catheters and lines into patients, some patients needs them to put on multiple positions. \n    + Let's see PatientID - bf4c6da3c\n- But, you realize that three groups - ETT, NGT, CVC counted seperately. "},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Total Rows of Train Data is\", len(train))\nprint(\"Total Count of Each Variable in Train Data is\", train.iloc[:, :-1].sum().sum())\n\nvar_cal_tmp = train.iloc[:, :-1].sum()\nprint(var_cal_tmp)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.iloc[1].to_frame().T","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"ch04_sub_chapter-one\"></a>\n### Quick Visualization\n- In general, CVC outnumbered other group. "},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10, 6))\nsns.barplot(x = var_cal_tmp.values, y = var_cal_tmp.index, ax=ax)\nax.tick_params(axis=\"x\", labelsize=14)\nax.tick_params(axis=\"y\", labelsize=14)\nax.set_xlabel(\"Number of Images\", fontsize=15)\nax.set_title(\"Distribution of Labels\", fontsize=15)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- The number of Patients are smaller than total data. \n- It means some patients are frequently checked, depending upon patients"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of Unique Patients: \", train[\"PatientID\"].unique().shape[0])\nprint(\"Number of Total Data: \", len(train[\"PatientID\"]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tmp = train['PatientID'].value_counts()\nprint(tmp)\nfig, ax = plt.subplots(figsize=(24, 6))\nsns.countplot(x = tmp.values, ax=ax)\nax.tick_params(axis=\"x\", labelsize=10)\nax.tick_params(axis=\"y\", labelsize=14)\nax.set_xlabel(\"Number of Images\", fontsize=15)\nax.set_title(\"Distribution of Labels\", fontsize=15)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Now, we need to see the distribution of data in each variable. "},{"metadata":{"trusted":true},"cell_type":"code","source":"target_cols = ['ETT - Abnormal', 'ETT - Borderline', 'ETT - Normal', 'NGT - Abnormal', \n               'NGT - Borderline', 'NGT - Incompletely Imaged', 'NGT - Normal', 'CVC - Abnormal',\n               'CVC - Borderline', 'CVC - Normal', 'Swan Ganz Catheter Present']\n\nfig, ax = plt.subplots(4, 3, figsize=(16, 10))\nfor i, col in enumerate(train[target_cols].columns[0:]):\n  print(i, col)\n  if i <= 2:\n    ax[0, i].hist(train[col].values)\n    ax[0, i].set_title(f'target: {col}')\n  elif i <= 5:\n    ax[1, i-3].hist(train[col].values)\n    ax[1, i-3].set_title(f'target: {col}')\n  elif i <= 8:\n    ax[2, i-6].hist(train[col].values)\n    ax[2, i-6].set_title(f'target: {col}')\n  else:\n    ax[3, i-9].hist(train[col].values)\n    ax[3, i-9].set_title(f'target: {col}')\n\nfig.tight_layout()\nfig.subplots_adjust(top=0.95)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- How to interpret the graph?\n    + CVC group is the top most amongst groups\n    + In each group, Normal is the top most.\n- This datasets are typically imbalanced, and multi-classification problem is revealed.\n"},{"metadata":{},"cell_type":"markdown","source":"<a id = \"chapter-five\"></a>\n## Background Knowledge\n- Since my major is far from this medical area, it difficults to figure what to classify from images. \n- So, need some videos to understand the processing. \n- Thanks to [RANZCR CLiP: Visualize and Understand Dataset](https://www.kaggle.com/nayuts/ranzcr-clip-visualize-and-understand-dataset)\n    + Please visit here and upvote\n"},{"metadata":{},"cell_type":"markdown","source":"<a id = \"ch05_sub_chapter-one\"></a>\n### Endotracheal Tube¶\n- It's so called ETT in this dataset. "},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import YouTubeVideo\nYouTubeVideo('FtJr7i7ENMY')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"ch05_sub_chapter-two\"></a>\n### Nasogastric Tube\n- It's so called NTT in this dataset. "},{"metadata":{"trusted":true},"cell_type":"code","source":"YouTubeVideo('Abf3Gd6AaZQ')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"ch05_sub_chapter-three\"></a>\n### Central venous catheter\n- It's so called CVC in this dataset. "},{"metadata":{"trusted":true},"cell_type":"code","source":"YouTubeVideo('mTBrCMn86cU')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"ch05_sub_chapter-four\"></a>\n### Swan Ganz Catheter Present\n- It's Swan Ganz Catheter Present"},{"metadata":{"trusted":true},"cell_type":"code","source":"YouTubeVideo('YkN30T6ig30')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = chapter-six></a>\n## Check train annotation file\n- What's Inside train_annotations file?\n    + The main purpose is said that 'These are segmentation annotations for training samples that have them. They are included solely as additional information for competitors.'\n- Let's look at data\n    "},{"metadata":{"trusted":true},"cell_type":"code","source":"annot = pd.read_csv(\"../input/ranzcr-clip-catheter-line-classification/train_annotations.csv\")\nannot.head(30)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"chapter-seven\"></a>\n## Visualization of X-rays image\n- combined train + train_annotations, let's draw sample image\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"from PIL import Image, ImageDraw\n\ndef train_base_chest_plot(row_ind, base_dir):\n    row = annot.loc[row_ind]\n    train_img = Image.open(base_dir + row['StudyInstanceUID'] + '.jpg')\n    uid = row['StudyInstanceUID']\n    label = row['label']\n    fig, ax = plt.subplots(figsize=(15, 6))\n    ax.imshow(train_img)\n    plt.title(f\"train: {label}\")\n\nbase_dir = '../input/ranzcr-clip-catheter-line-classification/train/'\ntrain_base_chest_plot(1, base_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- But, what we need is to draw tube. Thus, we need to use column 'data' in this plot. Let's do this. "},{"metadata":{"trusted":true},"cell_type":"code","source":"import ast \nimport numpy as np\n\ndef train_base_tube_plot(row_ind, base_dir):\n    row = annot.loc[row_ind]\n    train_img = Image.open(base_dir + row['StudyInstanceUID'] + '.jpg')\n    uid = row['StudyInstanceUID']\n    label = row['label']\n    data = np.array(ast.literal_eval(row['data']))\n    fig, ax = plt.subplots(figsize=(15, 6))\n    ax.imshow(train_img)\n    ax.plot(data[:, 0], data[:, 1], color = 'b', linewidth=2, marker='o')\n    plt.title(f\"train: {label}\")\n\nbase_dir = '../input/ranzcr-clip-catheter-line-classification/train/'\ntrain_base_tube_plot(1, base_dir)\ntrain_base_tube_plot(2, base_dir)\ntrain_base_tube_plot(25, base_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Well, still difficult to figure out what the difference between normal and abnormal is. So, Droped to draw more. "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}