{"cells":[{"metadata":{},"cell_type":"markdown","source":"Forked from [RANZCR CLiP - GroupKFold with TFRecords](https://www.kaggle.com/nickuzmenkov/ranzcr-clip-groupkfold-with-tfrecords)\n\nThis version is for creating tfrecords with annotated images."},{"metadata":{},"cell_type":"markdown","source":"### Hello!\n\nAs mentioned numerous times on the competition forum, the most proper way to organize folds is as follows:\n* all folds must share nearly same number of samples\n* label-wise distributions must be kept close to those in the entire dataset, as there are some extremely rare cases (e.g. `ETT - Abnormal`)\n* no `PatientID` can appear in different folds to prevent data leaks\n\nI've found two solutions so far: one by @underwearfitting **[here](https://www.kaggle.com/underwearfitting/how-to-properly-split-folds)** and another by @virilo **[here](https://www.kaggle.com/virilo/ranzcr-clip-stratified-kfold-to-team-up-v3)**.\n\nIf you are a **TensorFlow** user, then having this splits the easiest way to start your efficient data workflow is `tf.data.Dataset.from_tensor_slices` which makes it just as easy as feeding a dataframe into the network but results in longer (really longer) runtime.\n\nOn the other hand, serializing this dataset to TFRecords can be done in just 10 minutes without any acceleration. However, this step would save you up to few hours on TPU when training an ensemble or a large model.\n\nIn this short notebook I will take splits made by @virilo and re-serialize the original dataset to TFRecords of 600x600 image size grouped into 5 folds. You can then adjust the `IMG_SIZE` and replace the placeholder at the end of the notebook by your training pipeline."},{"metadata":{},"cell_type":"markdown","source":"### Imports"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import KFold\nfrom tqdm.notebook import tqdm\nimport tensorflow as tf\nimport pandas as pd\nimport numpy as np\nimport os\nimport IPython.display as display\nimport matplotlib.pyplot as plt\nimport ast\nimport cv2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"STRATEGY = tf.distribute.get_strategy()    \nBATCH_SIZE = 16\nIMG_SIZE = 900\nSEED = 42\n    \nCOLOR_MAP = {'ETT - Abnormal': (255, 0, 0),\n             'ETT - Borderline': (0, 255, 0),\n             'ETT - Normal': (0, 0, 255),\n             'NGT - Abnormal': (255, 255, 0),\n             'NGT - Borderline': (255, 0, 255),\n             'NGT - Incompletely Imaged': (0, 255, 255),\n             'NGT - Normal': (128, 0, 0),\n             'CVC - Abnormal': (0, 128, 0),\n             'CVC - Borderline': (0, 0, 128),\n             'CVC - Normal': (128, 128, 0),\n             'Swan Ganz Catheter Present': (128, 0, 128),\n            }\n\ntrain_annotations = pd.read_csv('../input/ranzcr-clip-catheter-line-classification/train_annotations.csv')\nannotation_thickness = 1\nprint('Using tensorflow %s' % tf.__version__)\nclahe = cv2.createCLAHE(clipLimit=8, tileGridSize=(8, 8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"file_name = '1.2.826.0.1.3680043.8.498.83331936392921199432218327504041001669'\nquery_string = f\"StudyInstanceUID == '{file_name}'\"\ndf = train_annotations.query(query_string)\n\nimage_path = f'../input/ranzcr-clip-catheter-line-classification/train/{file_name}.jpg'\nimage = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)\n\nmask = image > 0\nimage = image[np.ix_(mask.any(1), mask.any(0))]\n\n# image = 255-image\nimage = clahe.apply(image)\nimage = cv2.cvtColor(image, cv2.COLOR_GRAY2RGB)\nfor i, row in df.iterrows():\n    label = row[\"label\"]\n    data = np.array(ast.literal_eval(row[\"data\"]))\n    prev_d = None\n    for d in data:\n        if prev_d is not None:\n            image = cv2.line(image, (d[0], d[1]), (prev_d[0], prev_d[1]), COLOR_MAP[label], thickness=annotation_thickness) \n        prev_d = d\n\nplt.imshow(image)\nplt.grid(False)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Serialization functions"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def _serialize_image(path, df):\n    image = cv2.imread(path, cv2.IMREAD_GRAYSCALE)\n    \n    mask = image > 0\n    image = image[np.ix_(mask.any(1), mask.any(0))]\n\n#     image = 255-image\n    image = clahe.apply(image)\n    image = cv2.cvtColor(image, cv2.COLOR_GRAY2RGB)\n    image_orig = tf.identity(image)\n    for i, row in df.iterrows():\n        label = row[\"label\"]\n        data = np.array(ast.literal_eval(row[\"data\"]))\n        prev_d = None\n        for d in data:\n            if prev_d is not None:\n                image = cv2.line(image, (d[0], d[1]), (prev_d[0], prev_d[1]), COLOR_MAP[label], thickness=annotation_thickness) \n            prev_d = d\n            \n#     target_size = max(image.shape)\n    target_size = min(image.shape[:2])\n    image = tf.image.resize_with_crop_or_pad(image, target_height=target_size, target_width=target_size)\n    image = tf.image.resize(image, [IMG_SIZE, IMG_SIZE])\n    image = tf.cast(image, tf.uint8)\n    \n    image_orig = tf.image.resize_with_crop_or_pad(image_orig, target_height=target_size, target_width=target_size)\n    image_orig = tf.image.resize(image_orig, [IMG_SIZE, IMG_SIZE])\n    image_orig = tf.cast(image_orig, tf.uint8)\n    \n    return tf.image.encode_jpeg(image).numpy(), tf.image.encode_jpeg(image_orig).numpy()\n\n\ndef _serialize_sample(uid, image, image_orig, proba):\n    feature = {\n        'image': tf.train.Feature(bytes_list=tf.train.BytesList(value=[image_orig])),\n        'image_annotation': tf.train.Feature(bytes_list=tf.train.BytesList(value=[image])),\n        'StudyInstanceUID': tf.train.Feature(bytes_list=tf.train.BytesList(value=[uid])),\n        'ETT - Abnormal': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[0]])),\n        'ETT - Borderline': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[1]])),\n        'ETT - Normal': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[2]])),\n        'NGT - Abnormal': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[3]])),\n        'NGT - Borderline': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[4]])),\n        'NGT - Incompletely Imaged': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[5]])),\n        'NGT - Normal': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[6]])),\n        'CVC - Abnormal': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[7]])),\n        'CVC - Borderline': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[8]])),\n        'CVC - Normal': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[9]])),\n        'Swan Ganz Catheter Present':  tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[10]])),\n        'is_annotate': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[11]])),\n        'width': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[12]])),\n        'height': tf.train.Feature(int64_list=tf.train.Int64List(value=[proba[13]]))}\n    sample = tf.train.Example(features=tf.train.Features(feature=feature))\n    return sample.SerializeToString()\n\n\ndef serialize_fold(fold, name):\n    samples = []\n    \n    for uid, (_, p0, p1, p2, p3, p4, p5, p6, p7, p8, p9, p10, p11, p12, p13) in fold.iterrows():\n        \n        query_string = f\"StudyInstanceUID == '{uid}'\"\n        df = train_annotations.query(query_string)   \n        if len(df) > 0:\n            samples.append(_serialize_sample(\n                uid.encode(), \n                *_serialize_image(os.path.join(f'../input/ranzcr-clip-catheter-line-classification/train/{uid}.jpg'), df), \n                [p0, p1, p2, p3, p4, p5, p6, p7, p8, p9, p10, p11, p12, p13]))\n    \n    with tf.io.TFRecordWriter(name + '.tfrec') as writer:\n        [writer.write(x) for x in samples]\n        print(len(samples))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Here we check, again:\n1. Whether folds have close number of samples\n2. Whether no patient occures in multiple folds\n3. Whether label-wise distributions across folds is close to those of the entire dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv('../input/ranzcr-train-fix/train.more.fix.csv', \n                 index_col='StudyInstanceUID')\ndf_fold = pd.read_csv('../input/ranzcr-clip-stratified-kfold-to-team-up-v3/stratified_5_folds.csv', \n                      index_col='StudyInstanceUID')\ndf_fold = df_fold.reindex(df.index)\n\nfor i in range(5):\n    print(f'Fold %.i. Number of samples: %.i' % (i, len(df[df_fold['fold'] == i])))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Number of patients occuring in multiple folds: %.i' % len(set.intersection(\n    set(df[df_fold['fold'] == 0]['PatientID']),\n    set(df[df_fold['fold'] == 1]['PatientID']),\n    set(df[df_fold['fold'] == 2]['PatientID']),\n    set(df[df_fold['fold'] == 3]['PatientID']),\n    set(df[df_fold['fold'] == 4]['PatientID']))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df.drop('PatientID', axis=1)\n\nvalue_counts = lambda x: pd.Series.value_counts(x, normalize=True)\n\ndf_occurence = pd.DataFrame({\n    'origin': df.apply(value_counts).loc[0],\n    'fold_0': df[df_fold['fold'] == 0].apply(value_counts).loc[0],\n    'fold_1': df[df_fold['fold'] == 1].apply(value_counts).loc[0],\n    'fold_2': df[df_fold['fold'] == 2].apply(value_counts).loc[0],\n    'fold_3': df[df_fold['fold'] == 3].apply(value_counts).loc[0],\n    'fold_4': df[df_fold['fold'] == 4].apply(value_counts).loc[0]})\n\nbar = df_occurence.plot.barh(figsize=[14, 14], colormap='plasma')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Run serialization"},{"metadata":{"trusted":true},"cell_type":"code","source":"n_subfolds = 5\n\nfor i in tqdm(range(5)):    \n    for j, fold in enumerate(np.array_split(df[df_fold['fold'] == i].sample(frac=1), n_subfolds)):\n        serialize_fold(fold, name=f\"{i}_{j}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data workflow functions"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# feature_map = {\n#     'image': tf.io.FixedLenFeature([], tf.string),\n#     'StudyInstanceUID': tf.io.FixedLenFeature([], tf.string),  \n#     'ETT - Abnormal': tf.io.FixedLenFeature([], tf.int64),\n#     'ETT - Borderline': tf.io.FixedLenFeature([], tf.int64),\n#     'ETT - Normal': tf.io.FixedLenFeature([], tf.int64),\n#     'NGT - Abnormal': tf.io.FixedLenFeature([], tf.int64),\n#     'NGT - Borderline': tf.io.FixedLenFeature([], tf.int64),\n#     'NGT - Incompletely Imaged': tf.io.FixedLenFeature([], tf.int64),\n#     'NGT - Normal': tf.io.FixedLenFeature([], tf.int64),\n#     'CVC - Abnormal': tf.io.FixedLenFeature([], tf.int64),\n#     'CVC - Borderline': tf.io.FixedLenFeature([], tf.int64),\n#     'CVC - Normal': tf.io.FixedLenFeature([], tf.int64),\n#     'Swan Ganz Catheter Present': tf.io.FixedLenFeature([], tf.int64)}\n\n\n# def count_data_items(filenames):\n#     return np.sum([int(x[:-6].split('-')[-1]) for x in filenames])\n\n\n# def decode_image(image_data):\n#     image = tf.image.decode_jpeg(image_data, channels=1)\n#     image = tf.reshape(image, [IMG_SIZE, IMG_SIZE, 1])\n#     return image\n\n\n# def scale_image(image, target):\n#     image = tf.cast(image, tf.float32) / 255.\n#     return image, target\n\n\n# def read_tfrecord(example):\n#     example = tf.io.parse_single_example(example, feature_map)\n#     image = decode_image(example['image'])\n#     target = [\n#         example['ETT - Abnormal'],\n#         example['ETT - Borderline'],\n#         example['ETT - Normal'],\n#         example['NGT - Abnormal'],\n#         example['NGT - Borderline'],\n#         example['NGT - Incompletely Imaged'],\n#         example['NGT - Normal'],\n#         example['CVC - Abnormal'],\n#         example['CVC - Borderline'],\n#         example['CVC - Normal'],\n#         example['Swan Ganz Catheter Present']]\n#     return image, target\n\n\n# def data_augment(image, target):\n#     image = tf.image.random_flip_left_right(image, seed=SEED)\n#     image = tf.image.random_flip_up_down(image, seed=SEED)\n#     return image, target\n\n\n# def get_dataset(filenames, shuffled=False, repeated=False, \n#                 cached=False, augmented=False, distributed=True):\n#     auto = tf.data.experimental.AUTOTUNE\n#     dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=auto)\n#     dataset = dataset.map(read_tfrecord, num_parallel_calls=auto)\n#     if augmented:\n#         dataset = dataset.map(data_augment, num_parallel_calls=auto)\n#     dataset = dataset.map(scale_image, num_parallel_calls=auto)\n#     if shuffled:\n#         dataset = dataset.shuffle(2048, seed=SEED)\n#     if repeated:\n#         dataset = dataset.repeat()\n#     dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)\n#     if cached:\n#         dataset = dataset.cache()\n#     dataset = dataset.prefetch(auto)\n#     if distributed:\n#         dataset = STRATEGY.experimental_distribute_dataset(dataset)\n#     return dataset\n\n\n# def get_model():\n#     model = tf.keras.models.Sequential([\n#         tf.keras.applications.EfficientNetB0(\n#             include_top=False,\n#             input_shape=(None, None, 1),\n#             weights=None,\n#             pooling='avg'),\n#         tf.keras.layers.Dense(11, activation='sigmoid')\n#     ])\n#     model.compile(\n#         optimizer='adam',\n#         loss='binary_crossentropy',\n#         metrics=tf.keras.metrics.AUC(multi_label=True))\n\n#     return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train placeholder\nJust making shure that new TFRecords are readable.\n\nPlace your training pipeline here if you train your model in this notebook."},{"metadata":{"trusted":true},"cell_type":"code","source":"# kfold = KFold(n_splits=5, shuffle=True, random_state=SEED)\n# folders = os.listdir('./')\n\n# for i, (train_index, val_index) in enumerate(kfold.split(folders)):\n    \n#     if i > 0:\n#         break\n\n#     tf.keras.backend.clear_session()\n    \n#     train_filenames = []\n#     for j in train_index:\n#         train_filenames += tf.io.gfile.glob(os.path.join('./', folders[j], '*.tfrec'))\n#     np.random.shuffle(train_filenames)\n    \n#     val_filenames = []\n#     for j in val_index:\n#         val_filenames += tf.io.gfile.glob(os.path.join('./', folders[j], '*.tfrec'))\n#     np.random.shuffle(val_filenames)\n    \n#     train_filenames = train_filenames[:1]\n#     val_filenames = val_filenames[:1]\n        \n#     train_dataset = get_dataset(train_filenames, shuffled=True, augmented=True, repeated=True)\n#     val_dataset = get_dataset(val_filenames, shuffled=False, cached=True)\n\n#     steps_per_epoch = count_data_items(train_filenames) // BATCH_SIZE\n#     validation_steps = count_data_items(val_filenames) // BATCH_SIZE\n    \n#     with STRATEGY.scope():\n#         model = get_model()\n        \n#         model.compile(\n#             loss='binary_crossentropy',\n#             optimizer='adam',\n#             metrics=[tf.keras.metrics.AUC(multi_label=True)])\n    \n#     history = model.fit(\n#         train_dataset,\n#         steps_per_epoch=steps_per_epoch,\n#         epochs=1,\n#         validation_data=val_dataset,\n#         validation_steps=validation_steps,\n#         verbose=2)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}