{"cells":[{"metadata":{"papermill":{"duration":0.007,"end_time":"2021-03-07T11:07:42.067198","exception":false,"start_time":"2021-03-07T11:07:42.060198","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Building Data Pipeline with TensorFlow\n\nIn this notebook I will tell and show by example using TensorFlow why building pipeline is important for creating an effective neural network training cycle."},{"metadata":{"execution":{"iopub.execute_input":"2021-03-07T11:07:42.083105Z","iopub.status.busy":"2021-03-07T11:07:42.082437Z","iopub.status.idle":"2021-03-07T11:07:48.895276Z","shell.execute_reply":"2021-03-07T11:07:48.895759Z"},"papermill":{"duration":6.822897,"end_time":"2021-03-07T11:07:48.896177","exception":false,"start_time":"2021-03-07T11:07:42.07328","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport tensorflow as tf\nfrom sklearn.preprocessing import MultiLabelBinarizer","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-03-07T11:07:48.911629Z","iopub.status.busy":"2021-03-07T11:07:48.910893Z","iopub.status.idle":"2021-03-07T11:07:48.916525Z","shell.execute_reply":"2021-03-07T11:07:48.916007Z"},"papermill":{"duration":0.014194,"end_time":"2021-03-07T11:07:48.916667","exception":false,"start_time":"2021-03-07T11:07:48.902473","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"TEST_PATH = \"../input/hpa-single-cell-image-classification/test\"\nTRAIN_PATH = \"../input/hpa-single-cell-image-classification/train\"\nTRAIN_CSV = \"../input/hpa-single-cell-image-classification/train.csv\"\nN_CLASSES = 19\nSIZE = (512, 512)\nBATCH_SIZE = 32","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-03-07T11:07:48.935848Z","iopub.status.busy":"2021-03-07T11:07:48.935207Z","iopub.status.idle":"2021-03-07T11:07:48.938342Z","shell.execute_reply":"2021-03-07T11:07:48.93768Z"},"papermill":{"duration":0.015709,"end_time":"2021-03-07T11:07:48.938482","exception":false,"start_time":"2021-03-07T11:07:48.922773","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"def parse_label(raw_label):\n    '''Parse label to indicator vector'''\n    \n    label = list(map(int, raw_label.split('|')))\n    label = mlb.transform([label])\n    return np.squeeze(label).astype(np.int8)\n\ndef set_full_path(path):\n    '''Wrap function, that return full of file'''\n    def f(filename):\n        return path + \"/\" + filename\n    return f","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-03-07T11:07:48.957729Z","iopub.status.busy":"2021-03-07T11:07:48.957052Z","iopub.status.idle":"2021-03-07T11:07:50.842169Z","shell.execute_reply":"2021-03-07T11:07:50.841505Z"},"papermill":{"duration":1.897647,"end_time":"2021-03-07T11:07:50.84233","exception":false,"start_time":"2021-03-07T11:07:48.944683","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"# Load data\ndata = pd.read_csv(TRAIN_CSV)\n\n# Create MultiLabelBinarizer to transform label into multilabel vector\nmlb = MultiLabelBinarizer()\nmlb.fit([range(N_CLASSES)])\n\n# apply parse_label function to 'Label' column\ndata['Label'] = data['Label'].apply(parse_label)\ndata['ID'] = data['ID'].apply(set_full_path(TRAIN_PATH))\n\n# Get filenames and labels\nfilenames, labels = data['ID'], data['Label']\nlabels = np.array(labels.values.tolist()).astype(np.int8)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.005937,"end_time":"2021-03-07T11:07:50.85465","exception":false,"start_time":"2021-03-07T11:07:50.848713","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Creating Data Pipeline with tf.data\n\n---\n\nWhy we need to create data pipeline?\n\nThe process of training our NN on one batch can be divided by two parts:\n1. Prepare batch (read data from directory, do data augmentation, etc.)\n2. Feed batch to NN\n\nIn order to achieve a high speed of NN training, we need to prevent our GPU/TPU from [data starvation](http://https://en.wikipedia.org/wiki/Starvation_(computer_science)). In other words, the GPU / TPU should't stand idle. Without datapipeline our GPU/TPU will be waiting for next batch after backpropagation process is complete.\n\n---\n\nWithout pipelining, the CPU and the GPU/TPU sit idle much of the time:\n\n\n\n![Fig 1: Sequential execution frequently leaves the GPU idle](https://supportkb.dell.com/img/ka02R000000hGN0QAM/ka02R000000hGN0QAM_en_US_1.jpeg)\nWith pipelining, idle time diminishes significantly:\n\n\n\n\n![Fig 2: Pipelining overlaps CPU and GPU utilization, maximizing GPU utilization](https://supportkb.dell.com/img/ka02R000000hGN0QAM/ka02R000000hGN0QAM_en_US_2.jpeg)\n\n\nResource: [Optimization Techniques...](https://www.dell.com/support/kbdoc/ru-ua/000124384/optimization-techniques-for-training-chexnet-on-dell-c4140-with-nvidia-v100-gpus)\n\n---\n\nFor this purpose in TensorFlow exist <code>tf.data module</code>\n\nWe apply the following steps for training:\n\n1. Create <code>dataset</code> from <code>filenames</code> and <code>labels</code>\n```\ndataset = tf.data.Dataset.from_tensor_slices((filenames, labels))\n```\n2. Shuffle instances. Gradient descent works better when instances in the training set are independent and identically distributed\n```\ndataset = dataset.shuffle(len(filenames))\n```\n3. Parse images from labels. I use <code>num_parallel_calls=tf.data.AUTOTUNE</code> to tune the value dynamically at runtime\n```\ndataset = dataset.map(parse_function, num_parallel_calls=tf.data.AUTOTUNE)\n```\n4. Use data augmentation for the images.\n```\ndataset = dataset.map(train_preprocess, num_parallel_calls=tf.data.AUTOTUNE)\n```\n5. Batch the images\n```\ndataset = dataset.batch(BATCH_SIZE)\n```\n6. Prefetch one batch. In some cases you want to prefetch more than 1 batch if the duration of the preprocessing varies a lot.\n```\ndataset = dataset.prefetch(1)\n```"},{"metadata":{"execution":{"iopub.execute_input":"2021-03-07T11:07:50.879049Z","iopub.status.busy":"2021-03-07T11:07:50.878056Z","iopub.status.idle":"2021-03-07T11:07:50.881163Z","shell.execute_reply":"2021-03-07T11:07:50.880532Z"},"papermill":{"duration":0.020578,"end_time":"2021-03-07T11:07:50.881426","exception":false,"start_time":"2021-03-07T11:07:50.860848","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"# it is not necessary to wrap this function by tf.function\n@tf.function\ndef parse_function(filename, label):\n    '''\n    Load 512x512x3 Tensor and convert label:\n        - read content of 3 files (red, blue, green channels)\n        - decode and resize them using png format\n        - stack 3 channels\n        - convert values to float32\n        \n        - convert label to Tensor \n    '''    \n    red = tf.io.read_file(filename + \"_red.png\")\n    blue = tf.io.read_file(filename + \"_blue.png\")\n    green = tf.io.read_file(filename + \"_green.png\")\n    \n    red = tf.io.decode_png(red, channels=1)\n    blue = tf.io.decode_png(blue, channels=1)\n    green = tf.io.decode_png(green, channels=1)\n    \n    red = tf.image.resize(red, [*SIZE])\n    blue = tf.image.resize(blue, [*SIZE])\n    green = tf.image.resize(green, [*SIZE])\n    \n    image = tf.stack([red, green, blue], axis=-1)\n    \n    image = tf.squeeze(image)\n    image = tf.image.convert_image_dtype(image, tf.float32)\n    label = tf.convert_to_tensor(label, dtype=tf.int8)\n    \n    return image, label\n\n@tf.function\ndef train_preprocess(image, label):\n    '''\n    Augmnet data:\n        - random flip left/right\n        - random flip up/down\n    '''\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n\n    return image, label","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-03-07T11:07:50.919791Z","iopub.status.busy":"2021-03-07T11:07:50.919096Z","iopub.status.idle":"2021-03-07T11:07:51.247666Z","shell.execute_reply":"2021-03-07T11:07:51.246859Z"},"papermill":{"duration":0.35947,"end_time":"2021-03-07T11:07:51.247817","exception":false,"start_time":"2021-03-07T11:07:50.888347","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"dataset = tf.data.Dataset.from_tensor_slices((filenames, labels))\ndataset = dataset.shuffle(len(filenames))\ndataset = dataset.map(parse_function, num_parallel_calls=tf.data.AUTOTUNE)\ndataset = dataset.map(train_preprocess, num_parallel_calls=tf.data.AUTOTUNE)\ndataset = dataset.batch(BATCH_SIZE)\ndataset = dataset.prefetch(1)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}