{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This series of notebooks is a Tensorflow implementation of RANZCR 1st Place Solution by [@haqishen](https://www.kaggle.com/haqishen). The original notebooks are the following three:\n\n* [RANZCR 1st Place Soluiton Seg Model (small ver.)](https://www.kaggle.com/haqishen/ranzcr-1st-place-soluiton-seg-model-small-ver)\n* [RANZCR 1st Place Soluiton Cls Model (small ver.)](https://www.kaggle.com/haqishen/ranzcr-1st-place-soluiton-cls-model-small-ver)\n* [RANZCR 1st Place Soluiton Inference (small ver.)](https://www.kaggle.com/haqishen/ranzcr-1st-place-soluiton-inference-small-ver)\n\nThe discussion of the solution is [here](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/226633).\n\nMy implementation consists of the following five notebooks. They run on Kaggle CPU, GPU and TPU.\n\n* RANZCR 1st Place Solution by TF (1) Make Masks (this one)\n* [RANZCR 1st Place Solution by TF (2) Seg Model](https://www.kaggle.com/tt195361/ranzcr-1st-place-solution-by-tf-2-seg-model)\n* [RANZCR 1st Place Solution by TF (3) Gen Masks](https://www.kaggle.com/tt195361/ranzcr-1st-place-solution-by-tf-3-gen-masks)\n* [RANZCR 1st Place Solution by TF (4) Cls Model](https://www.kaggle.com/tt195361/ranzcr-1st-place-solution-by-tf-4-cls-model)\n* [RANZCR 1st Place Solution by TF (5) Inference](https://www.kaggle.com/tt195361/ranzcr-1st-place-solution-by-tf-5-inference)\n\nI made two more notebooks as compared to the original. The additional ones are for making TFRecord files. They are used to train the segmentation and classification models. Tensorflow can process TFRecord files faster than other types, for example JPEG. For TPU, [here](https://www.kaggle.com/docs/tpu) is useful information.\n\nI made the following four Datasets:\n\n* [RANZCR segmentation masks](https://www.kaggle.com/tt195361/ranzcr-segmentation-masks) -- contains TFRecord files made by this notebook, used to train the segmentation model.\n* [RANZCR 1st Place Solution by TF Train Data](https://www.kaggle.com/tt195361/ranzcr-1st-place-solution-by-tf-train-data) -- contains TFRecord files made by the third notebook, used to train the classification model.\n* [RANZCR 1st Place Solution by TF Models](https://www.kaggle.com/tt195361/ranzcr-1st-place-solution-by-tf-models) -- contains model weights of the segmentation and classification models, used by the inference model for submission.\n* [Segmentation Models Keras](https://www.kaggle.com/tt195361/segmentation-models-keras) -- Python \\*.whl files to install [Segmentation Models](https://github.com/qubvel/segmentation_models) without Internet, which is not allowed for submission notebook.\n\nThe followings are the points of these notebooks:\n\n* Segmentation model: at first, I tried to convert the original PyTorch model to Keras by torch.onnx.export() and [onnx2keras](https://github.com/nerox8664/onnx2keras). I made a [version](https://www.kaggle.com/tt195361/ranzcr-1st-place-solution-by-tf-2-seg-model?scriptVersionId=57986327), then trained. But it did not went well, accuracy was not improved. So, I changed to use a model in [Segmentation Models](https://github.com/qubvel/segmentation_models).\n* 5 channel inputs for classification model: The original one uses 3 channels for image and 2 channels for mask, so total input channels are 5. It is not easy for Keras models to use the pre-trained weights and change the number of input channels at the same time. \n* Data augmentation: the original notebooks use [Albumentations](https://github.com/albumentations-team/albumentations). To train Tensorflow model on CPU/GPU, it's possible to use it as described [here](https://albumentations.ai/docs/examples/tensorflow-example/). However, on TPU, all pipeline needs to be Tensorflow, so I made a similar one by Tensorflow.\n\nThe public/private score of the final result is 0.97015 and 0.97090. The original author says that it is sufficient to achieve LB score 0.972.  So, my score is lower than he expected. I used EfficientNetB5 as the base of the segmentation model. I trained only 1 fold for it. For classification, I selected EfficientNetB4 and ran 5 folds.\n\n----\nIn this first notebook, let's make TFRecords for images, masks, and fold information.\nTo draw lines and circles for masks, CV2 is used as in the original one. It's not straight forward by Tensorflow.\n\nBy using the TFRecord files made in this notebook, I built a dataset [RANZCR segmentation masks](https://www.kaggle.com/tt195361/ranzcr-segmentation-masks). It is used by the segmentation model in the next [notebook](https://www.kaggle.com/tt195361/ranzcr-1st-place-solution-by-tf-2-seg-model).","metadata":{}},{"cell_type":"code","source":"DEBUG = False","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# libraries\nimport os\nimport numpy as np\nimport pandas as pd\nimport ast\nimport cv2\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\n\nprint(tf.__version__)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Config","metadata":{}},{"cell_type":"code","source":"data_dir = '../input/ranzcr-clip-catheter-line-classification'\nimage_size = 1024\nimage_folder = 'train'\njpeg_quality = 100","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train_v2\n\ntrain_v2.csv has 2 additional columns, \"fold\" and \"w_anno\".","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv('../input/ranzcr-fold/train_v2.csv')\n\n# If DEBUG == True, use only 500 samples.\ndf_train = pd.concat([\n    df_train.query('fold == 0').sample(100),\n    df_train.query('fold == 1').sample(100),\n    df_train.query('fold == 2').sample(100),\n    df_train.query('fold == 3').sample(100),\n    df_train.query('fold == 4').sample(100),\n]) if DEBUG else df_train\n\ndf_train.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"df_train['fold'] shows the fold the row belongs to. There are 5 folds and they look evenly distributed for both all the rows and rows with \"w_anno == True\".","metadata":{}},{"cell_type":"code","source":"df_train['fold'].value_counts().sort_index()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.query('w_anno == True')['fold'].value_counts().sort_index()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['w_anno'].value_counts().sort_index()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Annotations and Masks","metadata":{}},{"cell_type":"code","source":"df_train_anno = pd.read_csv(os.path.join(data_dir, 'train_annotations.csv'))\n\nprint(df_train_anno.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_image(StudyInstanceUID):\n    path = os.path.join(data_dir, image_folder, StudyInstanceUID + '.jpg')\n    image = cv2.imread(path)[:, :, ::-1]\n    return image","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def make_mask(image, StudyInstanceUID):\n    df_this = df_train_anno.query(\n        f'StudyInstanceUID == \"{StudyInstanceUID}\"')\n    # Use 3 channels for encoding as PNG.\n    mask = np.zeros(\n        (image.shape[0], image.shape[1], 3)).astype(np.uint8)\n    for _, anno in df_this.iterrows():\n        anno_this = np.array(ast.literal_eval(anno[\"data\"]))\n        mask1 = mask[:, :, 0].copy()\n        mask1 = cv2.polylines(\n            mask1, np.int32([anno_this]), isClosed=False,\n            color=1, thickness=15, lineType=16)\n        mask[:, :, 0] = mask1\n        mask2 = mask[:, :, 1].copy()\n        mask2 = cv2.circle(\n            mask2, (anno_this[0][0], anno_this[0][1]),\n            radius=15, color=1, thickness=25)\n        mask2 = cv2.circle(\n            mask2, (anno_this[-1][0], anno_this[-1][1]),\n            radius=15, color=1, thickness=25)\n        mask[:, :, 1] = mask2\n\n    mask = (mask > 0.5).astype(np.uint8)\n    return mask","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Make TFRecord","metadata":{}},{"cell_type":"code","source":"def convert_image_mask(image, mask):\n    image = cv2.resize(image, (image_size, image_size))\n    image = tf.constant(image, dtype=tf.uint8)\n    image = tf.image.encode_jpeg(image, quality=jpeg_quality)\n    \n    mask = cv2.resize(mask, (image_size, image_size))\n    mask = tf.constant(mask, dtype=tf.uint8)\n    mask = tf.io.encode_png(mask)\n    \n    return image, mask","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def _bytes_feature(value):\n    \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n    if isinstance(value, type(tf.constant(0))):\n        # BytesList won't unpack a string from an EagerTensor.\n        value = value.numpy() \n    elif isinstance(value, str):\n        # string needs to be encoded to bytes.\n        value = value.encode('utf-8')\n    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n\ndef _int64_feature(value):\n    \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def serialize_example(image, mask, fold):\n    feature = {\n        'image': _bytes_feature(image),\n        'mask': _bytes_feature(mask),\n        'fold': _int64_feature(fold),\n    }\n    \n    example_proto = tf.train.Example(\n        features=tf.train.Features(feature=feature))\n    return example_proto.SerializeToString()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train_w_anno = df_train.query('w_anno==True')\nn_df_train_w_anno = len(df_train_w_anno)\n\nprint(df_train_w_anno.shape)\nprint(n_df_train_w_anno)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define a generator to provide a DataFrame row one by one.\ndef get_next_row():\n    for _, row in df_train_w_anno.iterrows():\n        yield row","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_tfrec = 15\nn_anno_per_tfrec = (n_df_train_w_anno + n_tfrec - 1) // n_tfrec\n\nn_anno_per_tfrec","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"remaining_item_count = len(df_train_w_anno)\ntfrec_i = 0\ngnr = get_next_row()\nwhile 0 < remaining_item_count:\n    tfrec_item_count = min(n_anno_per_tfrec, remaining_item_count)\n    tfrec_file_name = \"{0:02d}-{1:03d}.tfrec\".format(tfrec_i, tfrec_item_count)\n    print(\"Writing {0}...\".format(tfrec_file_name))\n    with tf.io.TFRecordWriter(tfrec_file_name) as writer:\n        for tfrec_item_i in range(tfrec_item_count):\n            if tfrec_item_i % 100 == 0:\n                print(tfrec_item_i, \", \", end='')\n            row = next(gnr)\n            image = read_image(row.StudyInstanceUID)\n            mask = make_mask(image, row.StudyInstanceUID)\n            image, mask = convert_image_mask(image, mask)\n            example = serialize_example(image, mask, row.fold)\n            writer.write(example)\n    print()\n    remaining_item_count -= tfrec_item_count\n    tfrec_i += 1","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! ls -l","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Verify TFRecords","metadata":{}},{"cell_type":"code","source":"def decode_image(image_bytes):\n    image = tf.image.decode_jpeg(image_bytes, channels=3)\n    return image\n\ndef decode_mask(mask_bytes):\n    mask = tf.io.decode_png(mask_bytes, channels=3)\n    return mask\n\ndef read_tfrecord(example):\n    TFREC_FORMAT = {\n        'image': tf.io.FixedLenFeature([], tf.string),\n        'mask': tf.io.FixedLenFeature([], tf.string),\n        'fold': tf.io.FixedLenFeature([], tf.int64),\n    }\n    \n    example = tf.io.parse_single_example(example, TFREC_FORMAT)\n    image = decode_image(example['image'])\n    mask = decode_mask(example['mask'])\n    fold = example['fold']\n    return image, mask, fold\n\ndef load_dataset(filenames):\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=None)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=None)\n    return dataset","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tfrec_file_names = tf.io.gfile.glob('*.tfrec')\nds = load_dataset(tfrec_file_names)\n\nprint(ds)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pylab import rcParams\nrcParams['figure.figsize'] = 20,10\n\nf, axarr = plt.subplots(1,5)\nmasks = []\nds_iter = iter(ds)\nfor p in range(5):\n    img, mask, _ = next(ds_iter)\n    axarr[p].imshow(img)\n    masks.append(mask)\n\nf, axarr = plt.subplots(1,5)\nfor p in range(5):\n    axarr[p].imshow(masks[p][ : , : , 0])\n\nf, axarr = plt.subplots(1,5)\nfor p in range(5):\n    axarr[p].imshow(masks[p][ : , : , 1])","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}