{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Hello\n\nConsidering **[this topic](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205243)** by @hengck23 training your models with annotations can be a bit tricky yet powerfull approach potentially boosting your solution by 1-2%. So I've decided to do some quick EDA (which I'm honestly not so good at) and preparation for this training pipeline by simply putting these colourfull annotations from `train_annotations.csv` on training images with the help of **OpenCV** library and saving the results in `.jpg` and `.tfrec` format.\n\nMy code is mostly ambiguous, sorry for that. I'm sure there's much fancier implementation for things done here.\n\n## Contents\n1. [Quick EDA](#EDA)\n2. [Saving annotated jpegs](#Jpegs)\n3. [Going even further: TFRecords](#Tfrecords)"},{"metadata":{},"cell_type":"markdown","source":"### Imports"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"%matplotlib inline\nfrom tqdm.notebook import tqdm\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nimport pandas as pd\nimport numpy as np\nimport cv2\nimport ast\nimport os","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IMG_SIZE = 600\n\ntry: \n    os.mkdir('./train')\nexcept FileExistsError:\n    pass\n\ntry:\n    os.mkdir('./train_tfrecords')\nexcept FileExistsError:\n    pass\n\nCLASSES = [\n    'ETT - Abnormal',\n    'ETT - Borderline',\n    'ETT - Normal',\n    'NGT - Abnormal',\n    'NGT - Borderline',\n    'NGT - Incompletely Imaged',\n    'NGT - Normal',\n    'CVC - Abnormal',\n    'CVC - Borderline',\n    'CVC - Normal',\n    'Swan Ganz Catheter Present']\n\nroot = '../input/ranzcr-clip-catheter-line-classification'\n\nmain_df = pd.read_csv(\n    os.path.join(root, 'train.csv'),\n    index_col='StudyInstanceUID').drop('PatientID', axis=1)\nanno_df = pd.read_csv(\n    os.path.join(root, 'train_annotations.csv'), \n    index_col='StudyInstanceUID')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='EDA'></a>\n## 1. Quick EDA\nHere we quickly look through annotation distributions across classes."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=[12, 6])\n\nmain_df.apply(pd.Series.value_counts).loc[1].plot.barh(label='not annotated', color='red')\nanno_df['label'].value_counts().reindex(CLASSES).plot.barh(label='annotated', color='blue')\n\nlegend = plt.legend()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_kg_hide-input":true},"cell_type":"code","source":"print(f'Found annotations for {anno_df.index.nunique()}/{len(main_df)} samples, from which:')\n\nmain_df_count = main_df.reindex(anno_df.index.drop_duplicates())\n\n\ndef _parse(raw):\n    parsed = np.zeros((len(CLASSES),))\n    for i in range(len(CLASSES)):\n        if raw == CLASSES[i]:\n            parsed[i] = 1\n            return parsed\n        \n\nindex=main_df_count.index\nanno_df_count = pd.DataFrame(\n    columns=CLASSES, \n    index=index,\n    data=np.zeros((len(index), len(CLASSES))))\n\nfor index in main_df_count.index:\n    \n    annotations = anno_df.loc[index]\n    \n    if isinstance(annotations, pd.DataFrame):\n        for label, _ in anno_df.loc[index].values:\n            anno_df_count.loc[index] += _parse(label)\n    else:\n        anno_df_count.loc[index] += _parse(annotations['label'])  \n        \nprint(f'  - {((anno_df_count < main_df_count).sum(axis=1) != 0).sum()} are incomplete')\nprint(f'  - {((anno_df_count != main_df_count).sum(axis=1) == 0).sum()} are complete')\nprint(f'  - {((anno_df_count > main_df_count).sum(axis=1) != 0).sum()} are overcomplete')\n\nprint('Saving filtered DataFrame.')\n\nmain_df = pd.read_csv(\n    os.path.join(root, 'train.csv'), \n    index_col='StudyInstanceUID').reindex(anno_df.index.drop_duplicates())\nmain_df = main_df[(main_df_count == anno_df_count).all(axis=1)]\nmain_df.to_csv('train_annotated.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you see, `9095` training images have annotations, and most of them are complete, i.e. only `24` images miss some of annotations corresponding for each of the catheters inserted. Another interesting thing to point out is that at least `1349` cases has **multiple catheters of each class**. Lastly, we notice is that `24 + 7723 + 1349 = 9096`. It seems that one image has the both missing and extra annotaions. \n\nWe save only `7723` fully-annotated samples with no extra annotations for further training. However, it's up to you to decide which images should be filtered out.\n\n\n### How do annotated images look like?\n\nHere we visualize some fully annotated samples (with a minimum of 4 catheters inserted). Note that each of 11 classes has its own unique colour (colours are extracted from `matplotlib` `gist_rainbow` colormap). For now we just plot annotations over the images without affecting them."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"colors = [\n    (1.0000, 0.0000, 0.1600),\n    (1.0000, 0.3678, 0.0000),\n    (1.0000, 0.9189, 0.0000),\n    (0.5511, 1.0000, 0.0000),\n    (0.0000, 1.0000, 0.0000),\n    (0.0000, 1.0000, 0.5482),\n    (0.0000, 0.9239, 1.0000),\n    (0.0000, 0.3698, 1.0000),\n    (0.1630, 0.0000, 1.0000),\n    (0.7172, 0.0000, 1.0000),\n    (1.0000, 0.0000, 0.7500)]\n\ncmap = {key: color for key, color in zip(CLASSES, colors)}\n\ndef _parse_annotation(raw):\n    annotation = ast.literal_eval(raw)\n    annotation = np.array(annotation, dtype=np.int32)\n    return annotation","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=[16, 8])\n\nfor i, index in enumerate(main_df[main_df.sum(axis=1) >= 4].index[:6]):\n    plt.subplot(2, 3, i + 1)\n    \n    img = cv2.imread(os.path.join(root, 'train', index + '.jpg'), 1)\n    \n    annotations = anno_df.loc[index]\n    \n    if isinstance(annotations, pd.DataFrame):\n        for target, annotation in anno_df.loc[index].values:\n            annotation = _parse_annotation(annotation)\n            plt.plot(annotation[:, 0], annotation[:, 1], label=target, color=cmap[target])\n    else:\n        target = annotations['label']\n        annotation = _parse_annotation(annotations['data'])\n        plt.plot(annotation[:, 0], annotation[:, 1], label=target, color=cmap[target])\n    \n    plt.imshow(img)\n\n    plt.legend(loc='lower right')\n    plt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='Jpegs'></a>\n## 2. Saving annotated jpegs\n\nNow we manually insert annotations inside the images with **OpenCV** `polylines` method and save the output in `.jpg` format.\n### Annotation functions"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"colors = [\n    (255, 0, 40),\n    (255, 93, 0),\n    (255, 234, 0),\n    (140, 255, 0),\n    (0, 255, 0),\n    (0, 255, 139),\n    (0, 235, 255),\n    (0, 94, 255),\n    (41, 0, 255),\n    (182, 0, 255),\n    (255, 0, 191)]\n\ncmap = {key: color for key, color in zip(CLASSES, colors)}\n\n\ndef _parse_annotation(raw):\n    annotation = ast.literal_eval(raw)\n    annotation = np.array(annotation, dtype=np.int32)\n    annotation = np.expand_dims(annotation, axis=0)\n    return annotation\n\n\ndef annotate(img, target, annotation):\n    annotation = _parse_annotation(annotation)\n    cv2.polylines(img, annotation, False, cmap[target], 10)\n    return img","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Run annotation"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"for index in tqdm(main_df.index, total=len(main_df)):\n    \n    img = cv2.imread(os.path.join(root, 'train', index + '.jpg'), 1)\n\n    annotations = anno_df.loc[index]\n    \n    if isinstance(annotations, pd.DataFrame):\n        for target, annotation in anno_df.loc[index].values:\n            img = annotate(img, target, annotation)\n    else:\n        img = annotate(img, annotations['label'], annotations['data'])\n        \n    img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))\n    cv2.imwrite(os.path.join('./train', index + '.jpg'), img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='Tfrecords'></a>\n## 3. Going even further: TFRecords\nIf you are a **TensorFlow** user, working with `.tfrec` files can significantly boost up performance. Here we serialize our previous results and save the output in `.tfrec` format.\n### Serialization functions"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def _serialize_image(path):\n    image = tf.io.read_file(path)\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = tf.image.resize(image, [IMG_SIZE, IMG_SIZE])\n    image = tf.cast(image, tf.uint8)\n    return tf.image.encode_jpeg(image).numpy()\n\n\ndef _serialize_sample(uid, image, proba):\n    feature = {\n        'image': tf.train.Feature(bytes_list=tf.train.BytesList(value=[image])),\n        'StudyInstanceUID': tf.train.Feature(bytes_list=tf.train.BytesList(value=[uid])),\n        'ETT - Abnormal': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[0]])),\n        'ETT - Borderline': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[1]])),\n        'ETT - Normal': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[2]])),\n        'NGT - Abnormal': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[3]])),\n        'NGT - Borderline': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[4]])),\n        'NGT - Incompletely Imaged': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[5]])),\n        'NGT - Normal': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[6]])),\n        'CVC - Abnormal': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[7]])),\n        'CVC - Borderline': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[8]])),\n        'CVC - Normal': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[9]])),\n        'Swan Ganz Catheter Present':  tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[10]]))}\n    sample = tf.train.Example(features=tf.train.Features(feature=feature))\n    return sample.SerializeToString()\n\n\ndef serialize_fold(fold, name):\n    samples = []\n    \n    for uid, proba in fold.iterrows():\n        samples.append(_serialize_sample(\n            uid.encode(), \n            _serialize_image(os.path.join('./train', uid + '.jpg')), \n            proba))\n    \n    with tf.io.TFRecordWriter(name + '.tfrec') as writer:\n        [writer.write(x) for x in samples]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Run serialization"},{"metadata":{"trusted":true},"cell_type":"code","source":"n_files = 16\n\nfor i, fold in tqdm(enumerate(np.array_split(main_df, n_files)), total=n_files):\n    serialize_fold(fold, name='./train_tfrecords/%.2i-%.3i' % (i, len(fold)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!zip -rm -qq train train\n!zip -rm -qq train_tfrecords train_tfrecords","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}