{"cells":[{"metadata":{},"cell_type":"markdown","source":"# ***Disclaimer:*** \nHello Kagglers! I am a Solution Architect with the Google Cloud Platform. I am a coach for this competition, the focus of my contributions is on helping users to leverage GCP components (GCS, TPUs, BigQueryetc..) in order to solve large problems. My ideas and contributions represent my own opinion, and are not representative of an official recommendation by Google. Also, I try to develop notebooks quickly in order to help users early in competitions. There may be better ways to solving particular problems, I welcome comments and suggestions. Use my contributions at your own risk, I don't garantee that they will help on winning any competition, but I am hoping to learn by collaborating with everyone."},{"metadata":{},"cell_type":"markdown","source":"# Objective\n\nThe main objective of this Notebook is to show how to use a Dataset that we previously created using this Notebook:\n[\"HubMAP: Read data and build TFRecords\"](https://www.kaggle.com/marcosnovaes/hubmap-read-data-and-build-tfrecords)\n\nI made this Dataset Public, it is called [\"hubmap-train-test\"](https://www.kaggle.com/marcosnovaes/hubmap-tfrecord-512).\n\nThis Dataset was built by tiling each image into 512x512 tiles and then converting them to TFRecord files. \n\nThis Notebook shows how to load the TFRecords directly from Google Cloud Storage (GCS). This is the way in which you can pass the Dataset to a TPU accelerator, as will be shown in the next Notebook.\n\nNOTICE: In order for this Notebook to work, you need to link this Notebook to a GCP project using \"Add Ons-->Google Cloud SDK\", because I am reading directly from GCS to illustrate how it works. But you can modify the notebook to run from the local dataset as well.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\n\n#import cv2\nimport json\nimport os\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport pandas as pd\n\n#import shutil\n\nimport tensorflow as tf\n\nimport glob\n#import tifffile\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\nfrom kaggle_secrets import UserSecretsClient","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After importing the Kaggle Dataset  [\"hubmap-train-test\"](https://www.kaggle.com/marcosnovaes/hubmap-tfrecord-512), you should see the following in the input folder.\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!ls /kaggle/input","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is the local name of the dataset: \"hubmap-tfrecord-512\" . We will use this name later to find out its location in GCS. \n\nLet's inspect the dataset structure perusing the local file system."},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls /kaggle/input/hubmap-tfrecord-512/train/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls -l /kaggle/input/hubmap-tfrecord-512/train/2f6ecfcdf/col0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import glob \n\nfile_list = glob.glob('/kaggle/input/hubmap-tfrecord-512/train/*.csv')\nfile_list","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The CSV files contain the metadata of each tile, such as its tile row and col number that are used to calculate the height and width offsets."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\n\ntrain_df = pd.read_csv(file_list[0])\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"list_img_id = train_df.loc[0]['img_id']\nlist_img_id","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In order to read TFRecord using the TF library, we must grant tensorflow user credentials as below.\n\nNOTICE: In order for this code to work, you need to link this Notebook to a GCP project using \"Add Ons-->Google Cloud SDK\""},{"metadata":{"trusted":true},"cell_type":"code","source":"user_secrets = UserSecretsClient()\nuser_credential = user_secrets.get_gcloud_credential()\nuser_secrets.set_tensorflow_credential(user_credential)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's find out the bucket name where the dataset is stored. This information will allow us to access the dataset directly from GCS, which is the way the TPU accesses it. "},{"metadata":{"trusted":true},"cell_type":"code","source":"#from kaggle_datasets import KaggleDatasets\nGCS_PATH = KaggleDatasets().get_gcs_path('hubmap-tfrecord-512')\nGCS_PATH","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# strip the first 5 chars, that is the \"gs://\" prefix\nbucket_name = GCS_PATH[5:]\nbucket_name\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The function below will build a complete GCS file path for a given tile identified by img_name, tile_row and tile_col."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_tile_gcs_path( bucket_name, subdir,img_id, tile_row, tile_col):\n    return 'gs://{}/{}/{}/col{}/col{}_row{}.tfrecords'.format(bucket_name, subdir, img_id, tile_row,tile_row, tile_col)\n\ndef get_tile_local_path( subdir,img_id, tile_row, tile_col):\n    return '/kaggle/input/hubmap-tfrecord-512/{}/{}/col{}/col{}_row{}.tfrecords'.format(subdir, img_id, tile_row,tile_row, tile_col)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_tile_path = get_tile_gcs_path( bucket_name ,'train','2f6ecfcdf',8,20)\nsample_tile_path","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"local_tile_path = get_tile_local_path( 'train','2f6ecfcdf',8,20)\nlocal_tile_path","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# read back a record to make sure it the decoding works\n# Create a dictionary describing the features.\nimage_feature_description = {\n    'img_index': tf.io.FixedLenFeature([], tf.int64),\n    'height': tf.io.FixedLenFeature([], tf.int64),\n    'width': tf.io.FixedLenFeature([], tf.int64),\n    'num_channels': tf.io.FixedLenFeature([], tf.int64),\n    'img_bytes': tf.io.FixedLenFeature([], tf.string),\n    'mask': tf.io.FixedLenFeature([], tf.string),\n    'tile_id': tf.io.FixedLenFeature([], tf.int64),\n    'tile_col_pos': tf.io.FixedLenFeature([], tf.int64),\n    'tile_row_pos': tf.io.FixedLenFeature([], tf.int64),\n}\n\ndef _parse_image_function(example_proto):\n  # Parse the input tf.Example proto using the dictionary above.\n    single_example = tf.io.parse_single_example(example_proto, image_feature_description)\n    img_index = single_example['img_index']\n    img_height = single_example['height']\n    img_width = single_example['width']\n    num_channels = single_example['num_channels']\n    \n    img_bytes =  tf.io.decode_raw(single_example['img_bytes'],out_type='uint8')\n   \n    img_array = tf.reshape( img_bytes, (img_height, img_width, num_channels))\n   \n    mask_bytes =  tf.io.decode_raw(single_example['mask'],out_type='bool')\n    \n    mask = tf.reshape(mask_bytes, (img_height,img_width))\n    mtd = dict()\n    mtd['img_index'] = single_example['img_index']\n    mtd['width'] = single_example['width']\n    mtd['height'] = single_example['height']\n    mtd['tile_id'] = single_example['tile_id']\n    mtd['tile_col_pos'] = single_example['tile_col_pos']\n    mtd['tile_row_pos'] = single_example['tile_row_pos']\n    struct = {\n        'img_array': img_array,\n        'mask': mask,\n        'mtd': mtd\n    } \n    return struct\n\ndef read_tf_dataset(storage_file_path):\n    encoded_image_dataset = tf.data.TFRecordDataset(storage_file_path, compression_type=\"GZIP\")\n    parsed_image_dataset = encoded_image_dataset.map(_parse_image_function)\n    return parsed_image_dataset\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_tile_path","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nds = read_tf_dataset(sample_tile_path)\n\nfor struct in ds.as_numpy_iterator():\n    #struct = g_dataset.get_next()\n    img_mtd = struct[\"mtd\"]\n    img_array  = struct[\"img_array\"]\n    img_mask = struct[\"mask\"]\n \n    fig, ax = plt.subplots(1,2,figsize=(20,3))\n    ax[0].set_title(\"Tile ID = {} Xpos = {} Ypos = {}\".format(img_mtd['tile_id'], img_mtd['tile_col_pos'],img_mtd['tile_row_pos']))\n    ax[0].imshow(img_array)\n    #ax[1].set_title(\"Pixelarray distribution\");\n    #sns.distplot(img_array.flatten(), ax=ax[1]);\n    ax[1].imshow(img_mask)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, the dataset provides all the competition images as 512 x 512 tiles. The CSV files provide useful metadata for each tile. This meatadata can be used to build datasets of specific distributions, for example 50% with gloms and 50% no gloms, and so forth. As an example, here is how to build a dataset of all gloms in the first image:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# build a dataset of all image tiles from image 0 that have gloms in them\n#for csv_file in file_list:\ncsv_file = file_list[0]\ntiles_df = pd.read_csv(csv_file)\ngloms_df = tiles_df.loc[tiles_df[\"mask_density\"]  > 0]\n\ngloms_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gloms_df.__len__()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have filtered out only the tilest with gloms. Now build a list of tile paths from the data in the dataframe, and that list is basically our dataset. When you build a dataset using tf.data.TFRecordDataset you only need to pass a list of files. You can pass local file paths or paths to files that reside in GCS, which are URIs starting with \"gs://\". This allows for very efficient data transfer and this will be the way in which a TPU will be able to read your files, as illustrated in a future notebook. We build this list as below:"},{"metadata":{"trusted":true},"cell_type":"code","source":"tile_path_array = []\nnum_tiles = gloms_df.__len__()\nfor index in range(num_tiles) :\n    dataset_name = 'train'\n    img_id = gloms_df.iloc[index]['img_id']\n    col_offset = gloms_df.iloc[index]['tile_col_num']\n    row_offset = gloms_df.iloc[index]['tile_row_num']\n    tile_path = get_tile_gcs_path( bucket_name ,dataset_name,img_id,col_offset,row_offset)\n    tile_path_array.append(tile_path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we will build a dataset using only the first five 5 names, just for testing. Notice that the TFRecords will be read directly from GCS."},{"metadata":{"trusted":true},"cell_type":"code","source":"# check the first 5 paths to see if they are correct\ntile_path_array[0:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# reat the dataset passing all the file names\nds = read_tf_dataset(tile_path_array[0:5])\n#read 5 images and masks using the numpy iterator\nglom_mtd = []\nglom_tiles = []\nglom_masks = []\n\nfor struct in ds.as_numpy_iterator():\n    img_mtd = struct[\"mtd\"]\n    glom_mtd.append(img_mtd)\n    img_array  = struct[\"img_array\"]\n    glom_tiles.append(img_array)\n    img_mask = struct[\"mask\"]\n    glom_masks.append(img_mask)\n    \n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's plot the 5 tiles and corresponding masks, to verify that indeed we built a dataset with only tiles that contain gloms. "},{"metadata":{"trusted":true},"cell_type":"code","source":"#plot the 5 glom tiles from the dataset\nfig, ax = plt.subplots(5,2,figsize=(10,20))\nax[0][0].set_title(\"Tile ID = {} Xpos = {} Ypos = {}\".format(glom_mtd[0]['tile_id'], glom_mtd[0]['tile_col_pos'],glom_mtd[0]['tile_row_pos']))\nax[0][0].imshow(glom_tiles[0])\nax[0][1].imshow(glom_masks[0])\n\nax[1][0].set_title(\"Tile ID = {} Xpos = {} Ypos = {}\".format(glom_mtd[1]['tile_id'], glom_mtd[1]['tile_col_pos'],glom_mtd[1]['tile_row_pos']))\nax[1][0].imshow(glom_tiles[1])\nax[1][1].imshow(glom_masks[1])\n\nax[2][0].set_title(\"Tile ID = {} Xpos = {} Ypos = {}\".format(glom_mtd[2]['tile_id'], glom_mtd[2]['tile_col_pos'],glom_mtd[2]['tile_row_pos']))\nax[2][0].imshow(glom_tiles[2])\nax[2][1].imshow(glom_masks[2])\n\nax[3][0].set_title(\"Tile ID = {} Xpos = {} Ypos = {}\".format(glom_mtd[3]['tile_id'], glom_mtd[3]['tile_col_pos'],glom_mtd[3]['tile_row_pos']))\nax[3][0].imshow(glom_tiles[3])\nax[3][1].imshow(glom_masks[3])\n\nax[4][0].set_title(\"Tile ID = {} Xpos = {} Ypos = {}\".format(glom_mtd[4]['tile_id'], glom_mtd[4]['tile_col_pos'],glom_mtd[4]['tile_row_pos']))\nax[4][0].imshow(glom_tiles[4])\nax[4][1].imshow(glom_masks[4])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, now you can build a file list of GCS file paths using the tile metadata in the CSV files. You can build any filter you want, for example, avoid pure black or white tiles that have the lowband_density < 100, for example. In my next notebook I will be building a balanced set with gloms and no gloms to feed a UNET model using TPUs.\n\nIn order to save all the hard work done here, let's generate dataframes that contain the local and gcs file paths for all tiles and include them with the daset. That way the paths for each tile will be readily available for either local or remote access. I will then update the dataset with the CSVs that contain the files generated below."},{"metadata":{"trusted":true},"cell_type":"code","source":"# create a uber lits of all tiles for all images. Include local and cloud paths for each tile\ndef create_dataset_file_info( dataset_name):\n    file_list = glob.glob('/kaggle/input/hubmap-tfrecord-512/{}/*.csv'.format(dataset_name))\n    uber_tile_df = pd.DataFrame(columns = ['img_id','tile_id','tile_row_index', 'tile_col_index', 'lowband_density', 'mask_density','local_path','gcs_path'])\n    for file_name in file_list:\n            tile_df = pd.read_csv(file_name)\n            num_tiles = tile_df.__len__()\n            for index in range(num_tiles):\n                img_id = tile_df.iloc[index]['img_id']\n                tile_id = tile_df.iloc[index]['tile_id']\n                tile_col_num = tile_df.iloc[index]['tile_col_num']\n                tile_row_num = tile_df.iloc[index]['tile_row_num']\n                lowband_density = tile_df.iloc[index]['lowband_density']\n                mask_density = tile_df.iloc[index]['mask_density']           \n                local_tile_path = get_tile_local_path(dataset_name, img_id, tile_col_num, tile_row_num )\n                gcs_tile_path = get_tile_gcs_path(bucket_name, dataset_name, img_id, tile_col_num, tile_row_num )\n                uber_tile_df = uber_tile_df.append({'img_id':img_id, 'tile_id': tile_id, 'tile_row_index':tile_col_num, 'tile_col_index':tile_row_num,\n                                                     'lowband_density':lowband_density, 'mask_density':mask_density,\n                                                     'local_path':local_tile_path, 'gcs_path':gcs_tile_path},ignore_index=True)\n    #write the dataframe\n    print('writing tile metadata for dataset {}'.format(dataset_name))\n    output_file_name = '/kaggle/working/'+dataset_name+'_all_tiles.csv'\n    uber_tile_df.to_csv(output_file_name)\n    return uber_tile_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"uber_tile_df = create_dataset_file_info('test')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"uber_tile_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"uber_train_tile_df = create_dataset_file_info('train')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"uber_train_tile_df.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}