{"cells":[{"metadata":{},"cell_type":"markdown","source":"# How To Create TFRecords\nIn this notebook, we learn how to create TFRecords to train TensorFlow models. We will create TFRecords from the Kaggle dataset of 512x512x3 jpegs [here][1]. This dataset contains the Melanoma Classification competition data (train 30,000 and test 10,000 ) and an additional 30,000 external images. It was published by [Alex Shonenkov][2]\n\nThere is a discussion post about these TFRecords [here][3] and Alex discusses where these images came from [here][4]\n\n[1]: https://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg\n[2]: https://www.kaggle.com/shonenkov\n[3]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/156245\n[4]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155859","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Load Meta Data","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport re\nimport math\n\nimport numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt\n\nimport tensorflow as tf\nimport cv2 as cv\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BASE_PATH = '../input/siim-isic-melanoma-classification/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_IMAGE_PATH = os.path.join(BASE_PATH, 'jpeg', 'train')\nTEST_IMAGE_PATH  = os.path.join(BASE_PATH, 'jpeg', 'test')\n\nTRAIN_IMAGE_LIST = os.listdir(TRAIN_IMAGE_PATH)\nTEST_IMAGE_LIST  = os.listdir(TEST_IMAGE_PATH)\n\nprint('There are %i train images and %i test images' % (len(TRAIN_IMAGE_LIST), len(TEST_IMAGE_LIST)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(os.path.join(BASE_PATH, 'train.csv'))\ndf_train.rename({'image_id': 'image_name'}, axis=1, inplace=True)\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = pd.read_csv(os.path.join(BASE_PATH, 'test.csv'))\ndf_test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Label Encode Meta Data\n\nIt is more efficient to store this meta data as integers instead of strings. We will impute the Age NaNs to Age mean. Then all other NaNs will be convert to `-1` and the other strings will be converted to `0, 1, 2, 3, ...` in the order they appear in the printed lists below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_combined = pd.concat([df_train[df_test.columns], df_test[df_test.columns]], ignore_index=True, axis=0).reset_index(drop=True) # Combine test and train to encode together","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Label Encode all the strings\nlabels_categorical = ['patient_id','sex','anatom_site_general_challenge'] \nfor label in labels_categorical:\n    df_combined[label], mp = df_combined[label].factorize()\n    print(mp)\n\n# Mean Encode the Age NaN valies\nprint('Imputing Age NaN count =', df_combined.age_approx.isnull().sum())\ndf_combined.age_approx.fillna(df_combined.age_approx.mean(), inplace=True)\ndf_combined['age_approx'] = df_combined.age_approx.astype('int')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Rewrite encoded data to original dataframes\nlabels_categorical = labels_categorical + ['age_approx']\ndf_train[labels_categorical] = df_combined.loc[ : df_train.shape[0] - 1, labels_categorical].values\ndf_test[labels_categorical]  = df_combined.loc[df_train.shape[0] : , labels_categorical].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Label encode the site of the image\ndf_train.diagnosis, mp = df_train.diagnosis.factorize()\nprint(mp)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = df_train.drop(['benign_malignant'], axis=1)\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Some Helper Code","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def _bytes_feature(value):\n    \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n    if isinstance(value, type(tf.constant(0))):\n        value = value.numpy() # BytesList won't unpack a string from an EagerTensor.\n    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n\ndef _float_feature(value):\n    \"\"\"Returns a float_list from a float / double.\"\"\"\n    return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))\n\ndef _int64_feature(value):\n    \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def hair_remove(image):\n    # convert image to grayScale\n    grayScale = cv.cvtColor(image, cv.COLOR_RGB2GRAY)\n    # kernel for morphologyEx\n    kernel = cv.getStructuringElement(1,(17,17))\n    # apply MORPH_BLACKHAT to grayScale image\n    blackhat = cv.morphologyEx(grayScale, cv.MORPH_BLACKHAT, kernel)\n    # apply thresholding to blackhat\n    _,threshold = cv.threshold(blackhat,10,255,cv.THRESH_BINARY)\n    # inpaint with original image and threshold image\n    final_image = cv.inpaint(image,threshold,1,cv.INPAINT_TELEA)\n    \n    return final_image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Write TFRecords - Train\n\nAll the code below comes from TensorFlow's docs [here][1]\n\n[1]: https://www.tensorflow.org/tutorials/load_data/tfrecord","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def serialize_train(image, image_name, patient_id, sex, age, site, diagnosis, target):\n    feature = {\n      'image': _bytes_feature(image),\n      'image_name': _bytes_feature(image_name),\n      'patient_id': _int64_feature(patient_id),\n      'sex': _int64_feature(sex),\n      'age_approx': _int64_feature(age),\n      'anatom_site_general_challenge': _int64_feature(site),\n      'diagnosis': _int64_feature(diagnosis),\n      'target': _int64_feature(target)\n    }\n    example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n    return example_proto.SerializeToString()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 2000\nBATCH_COUNT = len(TRAIN_IMAGE_LIST) // BATCH_SIZE + int(len(TRAIN_IMAGE_LIST) % BATCH_SIZE != 0)\n\nfor j in range(BATCH_COUNT):\n    print('\\nWriting TFRecord %i of %i...' % (j, BATCH_COUNT))\n    CURRENT_BATCH_SIZE = min(BATCH_SIZE, len(TRAIN_IMAGE_LIST) - j * BATCH_SIZE)\n    with tf.io.TFRecordWriter('train%.2i-%i.tfrec'%(j, CURRENT_BATCH_SIZE)) as writer:\n        for k in range(CURRENT_BATCH_SIZE):\n            img = cv.imread(os.path.join(TRAIN_IMAGE_PATH, TRAIN_IMAGE_LIST[BATCH_SIZE * j + k]))\n            img = cv.resize(img, (256, 256), interpolation = cv.INTER_AREA)\n            img = hair_remove(img)\n            img = cv.imencode('.jpg', img, (cv.IMWRITE_JPEG_QUALITY, 94))[1].tostring()\n            name = TRAIN_IMAGE_LIST[BATCH_SIZE * j + k].split('.')[0]\n            row = df_train.loc[df_train.image_name == name]\n            example = serialize_train(\n                img, \n                str.encode(name),\n                row.patient_id.values[0],\n                row.sex.values[0],\n                row.age_approx.values[0],\n                row.anatom_site_general_challenge.values[0],\n                row.diagnosis.values[0],\n                row.target.values[0]\n            )\n            writer.write(example)\n            if (k % 100 == 0): print(k // 100, end=' ')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir('.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Write TFRecords - Test","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def serialize_test(image, image_name, patient_id, sex, age, site): \n    feature = {\n      'image': _bytes_feature(image),\n      'image_name': _bytes_feature(image_name),\n      'patient_id': _int64_feature(patient_id),\n      'sex': _int64_feature(sex),\n      'age_approx': _int64_feature(age),\n      'anatom_site_general_challenge': _int64_feature(site),\n    }\n    example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n    return example_proto.SerializeToString()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 500\nBATCH_COUNT = len(TEST_IMAGE_LIST) // BATCH_SIZE + int(len(TEST_IMAGE_LIST) % BATCH_SIZE != 0)\n\nfor j in range(BATCH_COUNT):\n    print(); print('Writing TFRecord %i of %i...' % (j, BATCH_COUNT))\n    CURRENT_BATCH_SIZE = min(BATCH_SIZE,len(TEST_IMAGE_LIST) - j * BATCH_SIZE)\n    with tf.io.TFRecordWriter('test%.2i-%i.tfrec' % (j, CURRENT_BATCH_SIZE)) as writer:\n        for k in range(CURRENT_BATCH_SIZE):\n            img = cv.imread(os.path.join(TEST_IMAGE_PATH, TEST_IMAGE_LIST[BATCH_SIZE * j + k]))\n            img = cv.resize(img, (256, 256), interpolation = cv.INTER_AREA)\n            img = hair_remove(img)\n            img = cv.imencode('.jpg', img, (cv.IMWRITE_JPEG_QUALITY, 94))[1].tostring()\n            name = TEST_IMAGE_LIST[BATCH_SIZE * j + k].split('.')[0]\n            row = df_test.loc[df_test.image_name == name]\n            example = serialize_test(\n                img,\n                str.encode(name),\n                row.patient_id.values[0],\n                row.sex.values[0],\n                row.age_approx.values[0],\n                row.anatom_site_general_challenge.values[0]\n            )\n            writer.write(example)\n            if (k % 100 == 0): print(k // 100, end=' ')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir('.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Verify TFRecords","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\ncfg = dict(\n    read_size = 512,\n    crop_size = 500, \n    net_size  = 448, \n)\n\ndef read_labeled_tfrecord(example, return_image_name):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n        'patient_id'                   : tf.io.FixedLenFeature([], tf.int64),\n        'sex'                          : tf.io.FixedLenFeature([], tf.int64),\n        'age_approx'                   : tf.io.FixedLenFeature([], tf.int64),\n        'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n        'diagnosis'                    : tf.io.FixedLenFeature([], tf.int64),\n        'target'                       : tf.io.FixedLenFeature([], tf.int64)\n    }           \n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['sex'], example['age_approx'], example['anatom_site_general_challenge'], example['image_name'] if return_image_name else example['target']\n\n\ndef read_unlabeled_tfrecord(example, return_image_name):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n        'patient_id'                   : tf.io.FixedLenFeature([], tf.int64),\n        'sex'                          : tf.io.FixedLenFeature([], tf.int64),\n        'age_approx'                   : tf.io.FixedLenFeature([], tf.int64),\n        'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64)\n    }\n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['sex'], example['age_approx'], example['anatom_site_general_challenge'], example['image_name'] if return_image_name else 0\n\n \ndef prepare_image(img, augment=True):    \n    img = tf.image.decode_jpeg(img, channels=3)\n    img = tf.image.resize(img, [cfg['read_size'], cfg['read_size']])\n    img = tf.cast(img, tf.float32) / 255.0\n    img = tf.image.central_crop(img, cfg['crop_size'] / cfg['read_size'])                               \n    img = tf.image.resize(img, [cfg['net_size'], cfg['net_size']])\n    img = tf.reshape(img, [cfg['net_size'], cfg['net_size'], 3])\n    return img\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)\n\n\ndef get_dataset(files, labeled=True, return_image_names=True):\n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO).cache()\n    \n    if labeled:\n        ds = ds.map(lambda example: read_labeled_tfrecord(example, return_image_names), num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), num_parallel_calls=AUTO)\n    \n    ds = ds.map(lambda img, sex, age, site, label: tuple([tuple([prepare_image(img), sex, age, site]), label]), \n            num_parallel_calls=AUTO)\n    ds = ds.batch(32)\n    ds = ds.prefetch(AUTO)\n    return ds","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"TRAIN_RECORDS = [ file for file in os.listdir('.') if 'train' in file ]\nTEST_RECORDS = [ file for file in os.listdir('.') if 'test' in file ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(5, 2, figsize=(10, 25))\n\nds = get_dataset(TRAIN_RECORDS, labeled=True).unbatch().take(5)\nfor idx, item in enumerate(ds):\n    ax[idx][0].imshow(item[0][0])\n    original = plt.imread(os.path.join(BASE_PATH, 'jpeg', 'train', item[1].numpy().decode(\"utf-8\") + '.jpg'))\n    ax[idx][1].imshow(original)\n    print('Sex: %s, Age: %s, Site: %s'%(item[0][1], item[0][2], item[0][3]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(5, 2, figsize=(10, 25))\n\nds = get_dataset(TEST_RECORDS, labeled=False).unbatch().take(5)\nfor idx, item in enumerate(ds):\n    ax[idx][0].imshow(item[0][0])\n    original = plt.imread(os.path.join(BASE_PATH, 'jpeg', 'test', item[1].numpy().decode(\"utf-8\") + '.jpg'))\n    ax[idx][1].imshow(original)\n    print('Sex: %s, Age: %s, Site: %s'%(item[0][1], item[0][2], item[0][3]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ds = get_dataset(TEST_RECORDS, labeled=False, return_image_names=True)\nimage_names = np.array([img_name.numpy().decode(\"utf-8\") for img, img_name in iter(ds.unbatch())])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(image_names.shape)\nprint(np.unique(image_names).shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Everything Works, yay!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# !tar czf test.tar.gz test*.tfrec","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# !tar czf train.tar.gz train*.tfrec","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}