{"cells":[{"metadata":{},"cell_type":"markdown","source":"# ***Disclaimer:*** \nHello Kagglers! I am a Solution Architect with the Google Cloud Platform. I am a coach for this competition, the focus of my contributions is on helping users to leverage GCP components (GCS, TPUs, BigQueryetc..) in order to solve large problems. My ideas and contributions represent my own opinion, and are not representative of an official recommendation by Google. Also, I try to develop notebooks quickly in order to help users early in competitions. There may be better ways to solving particular problems, I welcome comments and suggestions. Use my contributions at your own risk, I don't garantee that they will help on winning any competition, but I am hoping to learn by collaborating with everyone.\n"},{"metadata":{},"cell_type":"markdown","source":"# Objective:\n\nThe objective of this notebook is to provide an example of how to transform the HubMAP Hacking the Kidney competition dataset into a form that can readily used to train models leveraging accelerators. The images in this competition have very high resolution, averaging 30,000 x 30,000 pixels, and this presents a difficult challenge in memory management. It is just not possible to read them all in memory in the Kaggle environment, and it is also not possible to build a model using the whole image as input. This notebooks provides some tips for reading the competitions images and masks, and proposes a strategy to deal with the large sizes. \nThe strategy adopted in this Notebook is to tile the images in 512X512 tiles, and then transforming the tiles into TFRecords such that we can later use them as input to train models using GPU or TPU accelerators. \n\nThis Notebook takes a long time to run because it processes all the competition files and the resulting, compressed dataset is 18.1G, almost exceeding the Kaggle VM limit. I was able to process all files and then uploaded the results to a Kaggle daset that I have made public:\n--> [Link to the TFRecord Dataset Produced by this Notebook.](https://www.kaggle.com/marcosnovaes/hubmap-tfrecord-512)\n\nI have also developed a Notebook that explains how to use the TFRecord Dataset: [https://www.kaggle.com/marcosnovaes/hubmap-looking-at-tfrecords/](https://www.kaggle.com/marcosnovaes/hubmap-looking-at-tfrecords/)\n\nIf you want to use the dataset without change you don't need to run the Notebook -- but do read through it because it provides a lot of insight on how the read the images, masks and convert them to TFRecords. I will be using this dataset on my subsequent notebooks. You can also easily costumize this Notebook if you want to produce tiles od different sizes (I used 512x512) or if you want to include more metadata for each tile. "},{"metadata":{},"cell_type":"markdown","source":"# Reading the Images\nSome of the images are in TIFF format, some are in BigTIFF. I used the tiffile library and it seems to read the images with no problem."},{"metadata":{"trusted":true},"cell_type":"code","source":"pip install tifffile","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Libs used in this Notebook"},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\n\nimport cv2\nimport json\nimport os\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport pandas as pd\n\n#from imread import imread, imsave\n\nimport shutil\n\nimport tensorflow as tf\n\nimport glob\nimport tifffile\nimport gc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at the input data. Find and read the competition train.csv file."},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls /kaggle/input/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls -l /kaggle/input/hubmap-kidney-segmentation/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"basepath = '/kaggle/input/hubmap-kidney-segmentation/'\ntrain_df = pd.read_csv(basepath + \"train.csv\")\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The id corresponds to the images provided. For each image, you are provided a .tiff image file and the Run Length Encoding Mask. \n\nBut notice that the masks are also provided as a .json file with polygon definitions. I used this option instead in this Notebook. "},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls -l /kaggle/input/hubmap-kidney-segmentation/train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The next cell reads all tiffs and prints their shapes. It turns out that some TIFF images are channel first (number \"3\" first) and others channel last (number \"3\" last). When reading them, we must check if the \"3\" is first and swap the axis as needed. This loop will take a long time as each image is read just so we can tell its shape. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# verify that we can read all images\ndef verify_read(file_list):\n    for file_name in file_list:\n        baseimage = tifffile.imread(file_name)\n        #baseimage = tif.series[0].asarray()\n        print('img id = {}, shape = {}'.format(file_name,baseimage.shape))\n        gc.collect()\n        \nfile_list = glob.glob('/kaggle/input/hubmap-kidney-segmentation/train/*.tiff')\nverify_read(file_list)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Reading and Showing a sample image and mask\nThe next cells show the code that can read the first image in the csv file. \n\nIMPORTANT: Note that in the case of a \"channel first\" TIFF (number \"3\" first) we need to swap the axis of the numpy array as noted below. You CANNOT use \"reshape\" instead, that will scramble the channels."},{"metadata":{"trusted":true},"cell_type":"code","source":"#select an image to investigate\nworking_image_index = 0\nworking_image_id = train_df['id'][working_image_index]\nworking_image_id\nworking_image_path = '/kaggle/input/hubmap-kidney-segmentation/train/'+working_image_id+'.tiff'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here is the code that takes care of the difference in shapes. \nIMPORTANT: Notice that you need to use the numpy.swapaxes function to change the shape, using \"reshape\" will scramble the channels."},{"metadata":{"trusted":true},"cell_type":"code","source":"\nbaseimage = tifffile.imread(working_image_path)\nprint ('original image shape',baseimage.shape)\nbaseimage = np.squeeze(baseimage)\nif( baseimage.shape[0] == 3):\n    baseimage = baseimage.swapaxes(0,1)\n    baseimage = baseimage.swapaxes(1,2)\n    print ('swaped shape',baseimage.shape)\n\nplt.figure()\n\nplt.imshow(baseimage)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The masks are provided in the csv files in RLE format, but we are also provided json files that describe the mask as polygons. I will be using the json files:"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# read json mask\nworking_image_json_mask = '/kaggle/input/hubmap-kidney-segmentation/train/'+working_image_id+'.json'\n\nread_file = open(working_image_json_mask, \"r\") \nmask_data = json.load(read_file)\nmask_data[0]\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The following function converts the polygons into a numpy boolean mask with the same shape as the image."},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_mask(mask_file, mask_shape):\n    read_file = open(mask_file, \"r\") \n    mask_data = json.load(read_file)\n    polys = []\n    for index in range(mask_data.__len__()):\n        geom = np.array(mask_data[index]['geometry']['coordinates'])\n        polys.append(geom)\n\n    mask = np.zeros(mask_shape)\n    cv2.fillPoly(mask, polys, 1)\n    mask = mask.astype(bool)\n    return mask\n   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mask_shape = (baseimage.shape[0], baseimage.shape[1])\nmask = read_mask(working_image_json_mask, mask_shape)\nplt.imshow(mask)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"baseimage.dtype","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mask.dtype","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, now we know how to read each image and mask, and that their types are uint8 and bool respectively. But they have very large dimensions, we would not be able to train a ML model at these dimensions. So, in the next section I takes the approach of tiling up the image and working with tiles."},{"metadata":{},"cell_type":"markdown","source":"# Tiling the Large Images into 512x512 tiles\nHere are some useful functions that use the numpy slicing capability to select specifc tiles of the image. \n\nNOTE: The numpy arrays have dimensions [height, width, channels]. This Notebook will tile the image using offsets for the height index and width index. So:\n- a Tile with coordinate [0,0] represents the first tile on the top left corner.  \n- a Tile with coordinate [1,0] represents a tile with height offset = 1*Tile Size, in this case it starts at numpy coordinates [512,0], which means it is the tile below [0,0]\n- a Tile with coordinate [0,1] represents a tile with wodth offset = 1*Tile Size, in this case it starts at numpy coordinates [0,512], which means it is the tile to the right of [0,0]"},{"metadata":{"trusted":true},"cell_type":"code","source":"#explore a few tiles\ndef show_tile_and_mask(baseimage, mask, tile_size, tile_col_pos, tile_row_pos):\n    start_col = tile_col_pos*tile_size\n    end_col = start_col + tile_size\n    start_row = tile_row_pos * tile_size\n    end_row = start_row + tile_size\n    tile_image = baseimage[start_col:end_col, start_row:end_row,:]\n    tile_mask = mask[start_col:end_col, start_row:end_row]\n    fig, ax = plt.subplots(1,2,figsize=(20,3))\n    ax[0].imshow(tile_image)\n    ax[1].imshow(tile_mask)\n    \ndef get_tile(baseimage, tile_size, tile_col_pos, tile_row_pos):\n    start_col = tile_col_pos*tile_size\n    end_col = start_col + tile_size\n    start_row = tile_row_pos * tile_size\n    end_row = start_row + tile_size\n    tile_image = baseimage[start_col:end_col, start_row:end_row,:]\n    return tile_image\n\ndef get_tile_mask(baseimage, tile_size, tile_col_pos, tile_row_pos):\n    start_col = tile_col_pos*tile_size\n    end_col = start_col + tile_size\n    start_row = tile_row_pos * tile_size\n    end_row = start_row + tile_size\n    tile_image = baseimage[start_col:end_col, start_row:end_row]\n    return tile_image\n\ndef show_tile_dist(tile):\n    fig, ax = plt.subplots(1,2,figsize=(20,3))\n    #ax[0].set_title(\"Tile ID = {} Xpos = {} Ypos = {}\".format(img_mtd['tile_id'], img_mtd['tile_col_pos'],img_mtd['tile_row_pos']))\n    ax[0].imshow(tile)\n    ax[1].set_title(\"Pixelarray distribution\");\n    sns.distplot(tile.flatten(), ax=ax[1]);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As can be noticed in the sample image displayed, there is a black border and then a lot of white surrounding the tissue. If we select [0,0] we expect to see a black tile. If we move a little to the right and down, we are then in the white zone. So let's try the values [0,0] and [5,5] and we should be a black and a white tile respectively."},{"metadata":{"trusted":true},"cell_type":"code","source":"tile_size = 512\ntile = get_tile(baseimage, tile_size, 0, 0)\nshow_tile_dist(tile)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Black as predicted. As we explore the tiles, I also calculate the tile histogram. If we observe the histogram we will notice that it will provide a useful way to filter black and white tiles later. The numpy.histogram function divides the color spectrum in 10 bins and shows how many pixels call within each bin. We can notice that black and white fall into the higher end of the spectrum. Black tiles have 0 pixels in the lower end, while \"white\" (actually \"dirty gray\") has only about 20 pixels in that region. We then see that tiles with some actual tissue have a more even distribution. Let's call this metric \"lowpass energy\". It turns out that if we later select lowpass energy > 100 we are garanteed to have actual tissue in the slide, and we can discard anything with < 100. "},{"metadata":{"trusted":true},"cell_type":"code","source":"img_hist = np.histogram(tile)\nprint('histogram = {}'.format(img_hist[0]))\nprint('histogram_lowpass = {}'.format(np.sum(img_hist[0][0:4])))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And here is the white one ([5,5]"},{"metadata":{"trusted":true},"cell_type":"code","source":"tile = get_tile(baseimage, tile_size, 5, 5)\nshow_tile_dist(tile)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_hist = np.histogram(tile)\nprint('histogram = {}'.format(img_hist[0]))\nprint('histogram_lowpass = {}'.format(np.sum(img_hist[0][0:4])))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's try to find a glomerulus. If we look back at the polygon dump above, it shows that the first glom starts at pixel [10503, 4384]. If we divide both indexes by 512, we expect to find a glom in tile [8,20]"},{"metadata":{"trusted":true},"cell_type":"code","source":"tile = get_tile(baseimage, tile_size, 8, 20)\nshow_tile_dist(tile)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_hist = np.histogram(tile)\nprint('histogram = {}'.format(img_hist[0]))\nprint('histogram_lowpass = {}'.format(np.sum(img_hist[0][0:4])))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_tile_and_mask(baseimage, mask, tile_size, 8, 20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Bingo!!! We found our first glom. Let's now derive a metric for masks, so that in the future we can easily find tiles with gloms. This metric will be used when we want to filter the training dataset to make sure it includes a certain number of tiles with gloms. Simply counting the number of \"TRUE\" pixels in the mask is a great metric that indicate the tile contains a glom."},{"metadata":{"trusted":true},"cell_type":"code","source":"tile_mask = get_tile_mask(mask, tile_size, 8, 20)\nmask_density = np.count_nonzero(tile_mask)\nmask_density","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's move down the image by incrementing the height offset to [9,20], which should be the tile below [8,20]"},{"metadata":{"trusted":true},"cell_type":"code","source":"show_tile_and_mask(baseimage, mask, tile_size, 9, 20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tile_mask = get_tile_mask(mask, tile_size, 9, 20)\nmask_density = np.count_nonzero(tile_mask)\nmask_density","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, the glom ends in that tile, and there are fewer TRUE pixels. Going further down we find a cortex tile with no gloms."},{"metadata":{"trusted":true},"cell_type":"code","source":"show_tile_and_mask(baseimage, mask, tile_size, 10, 20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tile_mask = get_tile_mask(mask, tile_size, 10, 20)\nmask_density = np.count_nonzero(tile_mask)\nmask_density","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Transforming the Tiles into a TFRecord Dataset\nWe are now ready to read all the images (one at a time or we will run out of memory!) and then writing each tile to a TFRecord file. Kaggle has a limit of 50 upper level directories, so we will create one dir for each image. We will also build a pandas dataframe that has the metadata for each tile, including the lowpass energy and mask density metrics that we derived above. \n\nUsing the TFRecord format for storing data should be easy, but unfortunately it requires data serialization which complicates it a little bit. This is done using [protocol buffers](https://developers.google.com/protocol-buffers/) and that is a bit of a learning curve. But in ML you only need to understand the [TFExample](https://www.tensorflow.org/api_docs/python/tf/train/Example) format. In this Notebook I provide a little template code for dealing with TFExamples that can be quickly customized for any type of data. This template is explained in detail in [this tutorial](https://www.tensorflow.org/tutorials/load_data/tfrecord); but you don't need to read all this, in this Notebook I provide an example specific for image data that you can quickly customize.\n\nFor serialization using TFExample, we have to make any data fit into either one of 3 types:\n* bytes_feature\n* float_feature\n* int_64_feature\n\nIn this Notebook and image and mask are passed as bytes_features and the other metadata as int_64. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Utilities serialize data into a TFRecord\ndef _bytes_feature(value):\n  \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n  if isinstance(value, type(tf.constant(0))):\n    value = value.numpy() # BytesList won't unpack a string from an EagerTensor.\n  return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n\ndef _float_feature(value):\n  \"\"\"Returns a float_list from a float / double.\"\"\"\n  return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))\n\ndef _int64_feature(value):\n  \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n  return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def image_example(image_index, image, mask, tile_id, tile_col_pos, tile_row_pos):\n    image_shape = image.shape\n    \n    img_bytes = image.tostring()\n\n    mask_bytes = mask.tostring()\n    \n    feature = {\n        'img_index': _int64_feature(image_index),\n        'height': _int64_feature(image_shape[0]),\n        'width': _int64_feature(image_shape[1]),\n        'num_channels': _int64_feature(image_shape[2]),\n        'img_bytes': _bytes_feature(img_bytes),\n        'mask' : _bytes_feature(mask_bytes),\n        'tile_id':  _int64_feature(tile_id),\n        'tile_col_pos': _int64_feature(tile_col_pos),\n        'tile_row_pos': _int64_feature(tile_row_pos),\n        \n    }\n    return tf.train.Example(features=tf.train.Features(feature=feature))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This function writes a tile to storage, notice the GZIP compression -- this makes possible for all the tiles to be stored locally without exceeding the HD allowance of the Kaggle machine."},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_tfrecord( image_index, image, mask, tile_id, tile_col_pos, tile_row_pos, output_path):\n    opts = tf.io.TFRecordOptions(compression_type=\"GZIP\")\n    with tf.io.TFRecordWriter(output_path, opts) as writer:\n        tf_example = image_example(image_index, image, mask, tile_id, tile_col_pos, tile_row_pos)\n        writer.write(tf_example.SerializeToString())\n    writer.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here is the function that takes an image, slices into tiles, calculates tile metadata and commits to storage. It also builds a pandas dataframe with the metadata for all tiles. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def write_tfrecord_tiles( image_index, image_id, image, mask, tile_size, output_path ):\n    output_dir = output_path+image_id\n    if os.path.exists(output_dir):\n        shutil.rmtree(output_dir)\n    os.mkdir(output_dir)\n    \n    image_cols = image.shape[0]\n    image_rows = image.shape[1]\n    tile_cols = image_cols // tile_size\n    tile_rows = image_rows // tile_size\n    tileID = 0\n    \n    # create a pandas dataframe to store metadata for each tile\n    tile_df = pd.DataFrame(columns = ['img_index', 'img_id','tile_id', 'tile_rel_path','tile_col_num', 'tile_row_num', 'lowband_density', 'mask_density'])\n    \n    # create one directory for each row of images\n    for col_number in range(tile_cols):\n        print('col_offset{} '.format(col_number),end='')\n        dir_path = output_dir+'/col{}'.format(col_number)\n        # create directory\n        if os.path.exists(dir_path):\n            shutil.rmtree(dir_path)\n        os.mkdir(dir_path)\n        for row_number in range(tile_rows):\n            #print(\"row{}\".format(row_number),end='')\n            dataset_file_path = dir_path+'/col{}_row{}.tfrecords'.format(col_number,row_number)\n            relative_path = image_id+'/col{}_row{}.tfrecords'.format(col_number,row_number)\n            lower_col_range = col_number * tile_size\n            higher_col_range = lower_col_range + tile_size\n            lower_row_range = row_number * tile_size\n            higher_row_range = lower_row_range + tile_size\n            image_tile = image[lower_col_range:higher_col_range, lower_row_range:higher_row_range, :]\n            tile_mask = mask[lower_col_range:higher_col_range, lower_row_range:higher_row_range]\n            num_records = create_tfrecord( image_index, image_tile, tile_mask, tileID, col_number, row_number, dataset_file_path)\n            # populate the metadata for this tile\n            img_hist = np.histogram(image_tile)\n            lowband_density = np.sum(img_hist[0][0:4])\n            mask_density = np.count_nonzero(tile_mask)\n            tile_df = tile_df.append({'img_index':image_index, 'img_id':image_id, 'tile_id': tileID, 'tile_rel_path':relative_path, \n                           'tile_col_num':col_number, 'tile_row_num':row_number,'lowband_density':lowband_density, 'mask_density':mask_density},ignore_index=True)\n            tileID += 1\n    return tile_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This inline code will each image and mask in the train set, swap axes when needed, loading the image and mask into numpy arrays and then invoking the above function for each image/mask pair. This will take a long time..."},{"metadata":{"trusted":true},"cell_type":"code","source":"output_dir = '/kaggle/working/train/'\nif os.path.exists(output_dir):\n    shutil.rmtree(output_dir)\nos.mkdir(output_dir)\n\ntile_size = 512\nnum_images = train_df.shape[0]\nfor image_index in range(num_images):\n    image_id = train_df['id'][image_index]\n    image_path = '/kaggle/input/hubmap-kidney-segmentation/train/'+image_id+'.tiff'\n    image_json_mask = '/kaggle/input/hubmap-kidney-segmentation/train/'+image_id+'.json'\n\n    baseimage = tifffile.imread(image_path)\n    print ('original image shape',baseimage.shape)\n    baseimage = np.squeeze(baseimage)\n    if( baseimage.shape[0] == 3):\n        baseimage = baseimage.swapaxes(0,1)\n        baseimage = baseimage.swapaxes(1,2)\n        print ('swaped shape',baseimage.shape)\n    \n    # read json mask\n    mask_shape = (baseimage.shape[0], baseimage.shape[1])\n    mask = read_mask(image_json_mask, mask_shape)\n    \n    print('writing tiles for image {}'.format(image_id))\n    tile_df = write_tfrecord_tiles( image_index, image_id, baseimage, mask, tile_size, output_dir )\n    \n    #write the dataframe\n    print('writing tile metadata for image {}'.format(image_id))\n    df_path = output_dir+image_id+'_tiles.csv'\n    tile_df.to_csv(df_path)\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Verify all train images were written to file."},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls /kaggle/working/train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the same fashion, the code below scans the test directory and converts all the images there too. The test images will be needed for evaluating the loss during training. \n\nWARNING: I ran out of memory trying to write both train and tests images in one session. To build the complete dataset I built one at the time."},{"metadata":{"trusted":true},"cell_type":"code","source":"#convert all the test images\nimg_file_list = glob.glob('/kaggle/input/hubmap-kidney-segmentation/test/*.tiff')\nmask_file_list = glob.glob('/kaggle/input/hubmap-kidney-segmentation/test/*.json')\n\noutput_dir = '/kaggle/working/test/'\nif os.path.exists(output_dir):\n    shutil.rmtree(output_dir)\nos.mkdir(output_dir)\n\ntile_size = 512\nnum_images = img_file_list.__len__()\nfor image_index in range(num_images):\n    file_name = img_file_list[image_index]\n    prefix = file_name.split('.')\n    parts = prefix[0].split('/')\n    image_id = parts[-1]\n    image_path = img_file_list[image_index]\n    image_json_mask = mask_file_list[image_index]\n\n    baseimage = tifffile.imread(image_path)\n    print ('original image shape',baseimage.shape)\n    baseimage = np.squeeze(baseimage)\n    if( baseimage.shape[0] == 3):\n        baseimage = baseimage.swapaxes(0,1)\n        baseimage = baseimage.swapaxes(1,2)\n        print ('swaped shape',baseimage.shape)\n    \n    # read json mask\n    mask_shape = (baseimage.shape[0], baseimage.shape[1])\n    mask = read_mask(image_json_mask, mask_shape)\n    \n    print('writing tiles for image {}'.format(image_id))\n    tile_df = write_tfrecord_tiles( image_index, image_id, baseimage, mask, tile_size, output_dir )\n    \n    #write the dataframe\n    print('writing tile metadata for image {}'.format(image_id))\n    df_path = output_dir+image_id+'_tiles.csv'\n    tile_df.to_csv(df_path)\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Verify all images were written"},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls /kaggle/working/test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Verifying the TFRecord correctness\nLet's now read back a sample tile to verify correctness. We introduce here the read TFRecord functions that follow the model described in [this tutorial](https://www.tensorflow.org/tutorials/load_data/tfrecord)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# read back a record to make sure it the decoding works\n# Create a dictionary describing the features.\nimage_feature_description = {\n    'img_index': tf.io.FixedLenFeature([], tf.int64),\n    'height': tf.io.FixedLenFeature([], tf.int64),\n    'width': tf.io.FixedLenFeature([], tf.int64),\n    'num_channels': tf.io.FixedLenFeature([], tf.int64),\n    'img_bytes': tf.io.FixedLenFeature([], tf.string),\n    'mask': tf.io.FixedLenFeature([], tf.string),\n    'tile_id': tf.io.FixedLenFeature([], tf.int64),\n    'tile_col_pos': tf.io.FixedLenFeature([], tf.int64),\n    'tile_row_pos': tf.io.FixedLenFeature([], tf.int64),\n}\n\ndef _parse_image_function(example_proto):\n  # Parse the input tf.Example proto using the dictionary above.\n    single_example = tf.io.parse_single_example(example_proto, image_feature_description)\n    img_index = single_example['img_index']\n    img_height = single_example['height']\n    img_width = single_example['width']\n    num_channels = single_example['num_channels']\n    \n    img_bytes =  tf.io.decode_raw(single_example['img_bytes'],out_type='uint8')\n   \n    img_array = tf.reshape( img_bytes, (img_height, img_width, num_channels))\n   \n    mask_bytes =  tf.io.decode_raw(single_example['mask'],out_type='bool')\n    \n    mask = tf.reshape(mask_bytes, (img_height,img_width))\n    mtd = dict()\n    mtd['img_index'] = single_example['img_index']\n    mtd['width'] = single_example['width']\n    mtd['height'] = single_example['height']\n    mtd['tile_id'] = single_example['tile_id']\n    mtd['tile_col_pos'] = single_example['tile_col_pos']\n    mtd['tile_row_pos'] = single_example['tile_row_pos']\n    struct = {\n        'img_array': img_array,\n        'mask': mask,\n        'mtd': mtd\n    } \n    return struct\n\ndef read_tf_dataset(storage_file_path):\n    encoded_image_dataset = tf.data.TFRecordDataset(storage_file_path, compression_type=\"GZIP\")\n    parsed_image_dataset = encoded_image_dataset.map(_parse_image_function)\n    return parsed_image_dataset\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's read the tile with col offset 8 and row offset 20 (i.e. [8,20]) which we previously noticed to contain a glom."},{"metadata":{"trusted":true},"cell_type":"code","source":"working_image_index = 0\nworking_image_id = train_df['id'][working_image_index]\nworking_image_id\nds_path = '/kaggle/working/train/'+working_image_id+'/col8/col8_row20.tfrecords'\nds = read_tf_dataset(ds_path)\n\nfor struct in ds.as_numpy_iterator():\n    #struct = g_dataset.get_next()\n    img_mtd = struct[\"mtd\"]\n    img_array  = struct[\"img_array\"]\n    img_mask = struct[\"mask\"]\n \n    fig, ax = plt.subplots(1,2,figsize=(20,3))\n    ax[0].set_title(\"Tile ID = {} Xpos = {} Ypos = {}\".format(img_mtd['tile_id'], img_mtd['tile_col_pos'],img_mtd['tile_row_pos']))\n    ax[0].imshow(img_array)\n    #ax[1].set_title(\"Pixelarray distribution\");\n    #sns.distplot(img_array.flatten(), ax=ax[1]);\n    ax[1].imshow(img_mask)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Voila!! Let's also verify that the tile metadata file is correct."},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls -l /kaggle/working/train/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"working_image_index = 0\nworking_image_id = train_df['id'][working_image_index]\ndata_path = '/kaggle/working/train/'+working_image_id+'_tiles.csv'\ntiles_df = pd.read_csv(data_path)\ntiles_df.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}