{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction","metadata":{}},{"cell_type":"markdown","source":"This notebook is an effort to understand TPU and get better at computer vision tasks. This is my first effort in performing tasks in TPU. Following is how I approached this competition - ","metadata":{}},{"cell_type":"markdown","source":"* V1: Simple Resnet-50 with freezing of parameters. (F-score: **0.00025**)\n* V2: EfficientNet-B4 with non freezing of layers.(From now on layers are never freezed) Also added ramp up learning rate. (F-score: **0.87193**)\n* V3 & V4: Has errors\n* V5: EfficientNet-B7 with class weights for imbalance. (Model learns very slowly) F-score: **0.88873**\n* V6: Added the validation data into training data (F-score: **0.95352**)\n* V7: Used Denset201. It overfits a little bit (F-score: **0.94371**)\n* V8: Added random-blackout augmentation with EfficientNet-B7 (F-score: **0.95465**)\n* V9: Ensemble model of EfficientNet-B7 and DenseNet201. Added external dataset from Oxford (F-score: **0.96329**)\n* V10: Using K-fold cross validation. Did not run it because training takes long time.\n* V11: Same as version 9 but without external dataset and more epochs. (F-score: **0.96140**)\n* V12: Same as version 11 but including external dataset","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"!pip install -q efficientnet","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:29:57.198555Z","iopub.execute_input":"2021-06-30T07:29:57.198929Z","iopub.status.idle":"2021-06-30T07:30:04.96932Z","shell.execute_reply.started":"2021-06-30T07:29:57.198847Z","shell.execute_reply":"2021-06-30T07:30:04.96828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import math, re, os\nimport tensorflow as tf\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.backend as K\nimport efficientnet.tfkeras as efn\nfrom tensorflow import keras\nfrom functools import partial\nprint(\"Tensorflow version \" + tf.__version__)\n\nfrom collections import Counter\nimport gc","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:04.971052Z","iopub.execute_input":"2021-06-30T07:30:04.971409Z","iopub.status.idle":"2021-06-30T07:30:10.738425Z","shell.execute_reply.started":"2021-06-30T07:30:04.971369Z","shell.execute_reply":"2021-06-30T07:30:10.737389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Distribution Strategy","metadata":{}},{"cell_type":"markdown","source":"A TPU has eight different cores and each of these cores acts as its own accelerator. (A TPU is sort of like having eight GPUs in one machine.) We tell TensorFlow how to make use of all these cores at once through a distribution strategy. The following cell creates the distribution strategy that we'll later apply to our model.\n\nWe'll use the distribution strategy when we create our neural network model. Then, TensorFlow will distribute the training among the eight TPU cores by creating eight different replicas of the model, one for each core.","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:10.740512Z","iopub.execute_input":"2021-06-30T07:30:10.741103Z","iopub.status.idle":"2021-06-30T07:30:10.754666Z","shell.execute_reply.started":"2021-06-30T07:30:10.741044Z","shell.execute_reply":"2021-06-30T07:30:10.753747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from numpy.random import seed\nseed(1)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:10.758193Z","iopub.execute_input":"2021-06-30T07:30:10.758459Z","iopub.status.idle":"2021-06-30T07:30:10.762895Z","shell.execute_reply.started":"2021-06-30T07:30:10.758433Z","shell.execute_reply":"2021-06-30T07:30:10.761815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading the Data","metadata":{}},{"cell_type":"code","source":"AUTOTUNE = tf.data.experimental.AUTOTUNE\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('tpu-getting-started')\nGCS_DS_PATH_EXT = KaggleDatasets().get_gcs_path('oxford-102')\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nHEIGHT = 512 \nWIDTH = 512\nCHANNELS = 3\nIMAGE_SIZE = [HEIGHT, WIDTH]\nEPOCHS = 20","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:10.764746Z","iopub.execute_input":"2021-06-30T07:30:10.765341Z","iopub.status.idle":"2021-06-30T07:30:11.627382Z","shell.execute_reply.started":"2021-06-30T07:30:10.765301Z","shell.execute_reply":"2021-06-30T07:30:11.626588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## GCS Path\nWhen used with TPUs, datasets need to be stored in a Google Cloud Storage bucket. You can use data from any public GCS bucket by giving its path just like you would data from '/kaggle/input'. The following will retrieve the GCS path for this competition's dataset.","metadata":{}},{"cell_type":"code","source":"GCS_PATH = GCS_DS_PATH + '/tfrecords-jpeg-512x512'\nGCS_PATH_EXT = GCS_DS_PATH_EXT + '/tfrecords-jpeg-512x512'\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') \n\nOXFORD_FILES = tf.io.gfile.glob(GCS_PATH_EXT + '/*.tfrec')\n\nTRAINING_FILENAMES = TRAINING_FILENAMES + OXFORD_FILES\n\nSKIP_VALIDATION = True\nif SKIP_VALIDATION:\n    TRAINING_FILENAMES = TRAINING_FILENAMES + VALIDATION_FILENAMES\n\nCLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 102\n\n","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:11.630443Z","iopub.execute_input":"2021-06-30T07:30:11.630721Z","iopub.status.idle":"2021-06-30T07:30:12.493468Z","shell.execute_reply.started":"2021-06-30T07:30:11.630692Z","shell.execute_reply":"2021-06-30T07:30:12.49262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We convert the image from tfRecord to tensor so that we can- \n* Change the image size\n* Normalize the pixels between 0-1","metadata":{}},{"cell_type":"code","source":"def decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:12.496462Z","iopub.execute_input":"2021-06-30T07:30:12.496732Z","iopub.status.idle":"2021-06-30T07:30:12.503039Z","shell.execute_reply.started":"2021-06-30T07:30:12.496705Z","shell.execute_reply":"2021-06-30T07:30:12.502229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Understanding TFRecord","metadata":{}},{"cell_type":"markdown","source":"Our data is structured data- (image, label). A single 'Example' represents a single instance in the dataset. Each Example has 'Features' described as a dictionary of feature names and values. A value can be either a BytesList, a FloatList, or an Int64List, each wrapped as a single Feature. Heres how we encode the data-","metadata":{}},{"cell_type":"code","source":"from tensorflow.train import BytesList, FloatList, Int64List\nfrom tensorflow.train import Example, Features, Feature\n\n# The Data\nimage = tf.constant([ # this could also be a numpy array\n    [0, 1, 2],\n    [3, 4, 5],\n    [6, 7, 8],\n])\nlabel = 0\nclass_name = \"Class A\"\n\n\n# Wrap with Feature as a BytesList, FloatList, or Int64List\nimage_feature = Feature(\n    bytes_list=BytesList(value=[\n        tf.io.serialize_tensor(image).numpy(),\n    ])\n)\nlabel_feature = Feature(\n    int64_list=Int64List(value=[label]),\n)\nclass_name_feature = Feature(\n    bytes_list=BytesList(value=[\n        class_name.encode()\n    ])\n)\n\n\n# Create a Features dictionary\nfeatures = Features(feature={\n    'image': image_feature,\n    'label': label_feature,\n    'class_name': class_name_feature,\n})\n\n# Wrap with Example\nexample = Example(features=features)\n\nprint(example)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:12.506982Z","iopub.execute_input":"2021-06-30T07:30:12.507255Z","iopub.status.idle":"2021-06-30T07:30:14.214371Z","shell.execute_reply.started":"2021-06-30T07:30:12.507222Z","shell.execute_reply":"2021-06-30T07:30:14.212741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All the data is stored as attributes of Example instance. Once everything is encoded as an Example, you can serialize it with the SerializeToString method. \nSerialization is the process of converting an object into a stream of bytes to store the object or transmit it to memory, a database, or a file. Its main purpose is to save the state of an object in order to be able to recreate it when needed.","metadata":{}},{"cell_type":"code","source":"example_bytes = example.SerializeToString()\nprint(example_bytes)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:14.216334Z","iopub.execute_input":"2021-06-30T07:30:14.216694Z","iopub.status.idle":"2021-06-30T07:30:14.221775Z","shell.execute_reply.started":"2021-06-30T07:30:14.216655Z","shell.execute_reply":"2021-06-30T07:30:14.220732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we already have Serialized data. To decode it, we need to tell tf what kind of data it has to expect. Passing the info to 'tf.io.parse_single_example' gives us the 'Example' as mentioned above which has image in Bytes format. So we have to decode the image back to tensor. ","metadata":{}},{"cell_type":"code","source":"\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:14.223162Z","iopub.execute_input":"2021-06-30T07:30:14.223763Z","iopub.status.idle":"2021-06-30T07:30:14.233385Z","shell.execute_reply.started":"2021-06-30T07:30:14.223702Z","shell.execute_reply":"2021-06-30T07:30:14.232558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One advantage of TPU is that we can run multiple files across TPU at once. Thus once the data gets in, we want to use it immediately and avoid creating any data streaming bottlenecks","metadata":{}},{"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTOTUNE) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTOTUNE)\n    return dataset","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:14.234699Z","iopub.execute_input":"2021-06-30T07:30:14.235177Z","iopub.status.idle":"2021-06-30T07:30:14.242592Z","shell.execute_reply.started":"2021-06-30T07:30:14.23514Z","shell.execute_reply":"2021-06-30T07:30:14.241542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Augmentations","metadata":{}},{"cell_type":"code","source":"# data augmentation @cdeotte kernel: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\ndef transform_rotation(image, height, rotation):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated\n    DIM = height\n    XDIM = DIM%2 #fix for size 331\n    \n    rotation = rotation * tf.random.uniform([1],dtype='float32')\n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    \n    # ROTATION MATRIX\n    c1 = tf.math.cos(rotation)\n    s1 = tf.math.sin(rotation)\n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    rotation_matrix = tf.reshape(tf.concat([c1,s1,zero, -s1,c1,zero, zero,zero,one],axis=0),[3,3])\n\n    # LIST DESTINATION PIXEL INDICES\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(rotation_matrix,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES \n    idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM,DIM,3])\n\ndef transform_shear(image, height, shear):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly sheared\n    DIM = height\n    XDIM = DIM%2 #fix for size 331\n    \n    shear = shear * tf.random.uniform([1],dtype='float32')\n    shear = math.pi * shear / 180.\n        \n    # SHEAR MATRIX\n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)\n    shear_matrix = tf.reshape(tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3])    \n\n    # LIST DESTINATION PIXEL INDICES\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(shear_matrix,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES \n    idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM,DIM,3])\n\ndef transform_shift(image, height, h_shift, w_shift):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly shifted\n    DIM = height\n    XDIM = DIM%2 #fix for size 331\n    \n    height_shift = h_shift * tf.random.uniform([1],dtype='float32') \n    width_shift = w_shift * tf.random.uniform([1],dtype='float32') \n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n        \n    # SHIFT MATRIX\n    shift_matrix = tf.reshape(tf.concat([one,zero,height_shift, zero,one,width_shift, zero,zero,one],axis=0),[3,3])\n\n    # LIST DESTINATION PIXEL INDICES\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(shift_matrix,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES \n    idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM,DIM,3])","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:14.244144Z","iopub.execute_input":"2021-06-30T07:30:14.244567Z","iopub.status.idle":"2021-06-30T07:30:14.270143Z","shell.execute_reply.started":"2021-06-30T07:30:14.244518Z","shell.execute_reply":"2021-06-30T07:30:14.269105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def random_blockout(img, sl=0.1, sh=0.2, rl=0.4):\n\n    h, w, c = tf.shape(img)[0], tf.shape(img)[1], 3\n    origin_area = tf.cast(h*w, tf.float32)\n\n    e_size_l = tf.cast(tf.round(tf.sqrt(origin_area * sl * rl)), tf.int32)\n    e_size_h = tf.cast(tf.round(tf.sqrt(origin_area * sh / rl)), tf.int32)\n\n    e_height_h = tf.minimum(e_size_h, h)\n    e_width_h = tf.minimum(e_size_h, w)\n\n    erase_height = tf.random.uniform(shape=[], minval=e_size_l, maxval=e_height_h, dtype=tf.int32)\n    erase_width = tf.random.uniform(shape=[], minval=e_size_l, maxval=e_width_h, dtype=tf.int32)\n\n    erase_area = tf.zeros(shape=[erase_height, erase_width, c])\n    erase_area = tf.cast(erase_area, tf.uint8)\n\n    pad_h = h - erase_height\n    pad_top = tf.random.uniform(shape=[], minval=0, maxval=pad_h, dtype=tf.int32)\n    pad_bottom = pad_h - pad_top\n\n    pad_w = w - erase_width\n    pad_left = tf.random.uniform(shape=[], minval=0, maxval=pad_w, dtype=tf.int32)\n    pad_right = pad_w - pad_left\n\n    erase_mask = tf.pad([erase_area], [[0,0],[pad_top, pad_bottom], [pad_left, pad_right], [0,0]], constant_values=1)\n    erase_mask = tf.squeeze(erase_mask, axis=0)\n    erased_img = tf.multiply(tf.cast(img,tf.float32), tf.cast(erase_mask, tf.float32))\n\n    return tf.cast(erased_img, img.dtype)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:14.271594Z","iopub.execute_input":"2021-06-30T07:30:14.272139Z","iopub.status.idle":"2021-06-30T07:30:14.284809Z","shell.execute_reply.started":"2021-06-30T07:30:14.272093Z","shell.execute_reply":"2021-06-30T07:30:14.283828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def data_augment(image, label):\n    seed = (0,0)\n    p_rotation = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_spatial = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_pixel = tf.random.uniform([], 0, 1.0, dtype=tf.float32)    \n    p_shear = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_shift = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_blackout = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    \n    # Blackout\n    if p_blackout >=0.8:\n        image = random_blockout(image)\n    \n    # Flips\n    if p_spatial >= .2:\n        image = tf.image.stateless_random_flip_left_right(image, seed)\n        image = tf.image.stateless_random_flip_up_down(image, seed)\n        \n    # Rotates\n    if p_rotate > .75:\n        image = tf.image.rot90(image, k=3) # rotate 270º\n    elif p_rotate > .5:\n        image = tf.image.rot90(image, k=2) # rotate 180º\n    elif p_rotate > .25:\n        image = tf.image.rot90(image, k=1) # rotate 90º\n    \n    if p_rotation >= .3: # Rotation\n        image = transform_rotation(image, height=HEIGHT, rotation=45.)\n    if p_shift >= .3: # Shift\n        image = transform_shift(image, height=HEIGHT, h_shift=15., w_shift=15.)\n    if p_shear >= .3: # Shear\n        image = transform_shear(image, height=HEIGHT, shear=20.)\n        \n    # Crops\n    if p_crop > .3:\n        crop_size = tf.random.uniform([], int(HEIGHT*.7), HEIGHT, dtype=tf.int32)\n        image = tf.image.random_crop(image, size=[crop_size, crop_size, CHANNELS])\n    elif p_crop > .7:\n        if p_crop > .9:\n            image = tf.image.central_crop(image, central_fraction=.7)\n        elif p_crop > .8:\n            image = tf.image.central_crop(image, central_fraction=.8)\n        else:\n            image = tf.image.central_crop(image, central_fraction=.9)\n            \n    image = tf.image.resize(image, size=[HEIGHT, WIDTH])\n        \n    # Pixel-level transforms\n    if p_pixel >= .2:\n        if p_pixel >= .7:\n            image = tf.image.stateless_random_saturation(image, lower=0.2, upper=0.8, seed=seed)\n        elif p_pixel >= .5:\n            image = tf.image.stateless_random_contrast(image, lower=.3, upper=0.7, seed=seed)\n        elif p_pixel >= .3:\n            image = tf.image.stateless_random_brightness(image, max_delta=.7, seed=seed)\n        else:\n            image = tf.image.adjust_gamma(image, gamma=.6)\n\n    return image, label","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:14.286456Z","iopub.execute_input":"2021-06-30T07:30:14.286858Z","iopub.status.idle":"2021-06-30T07:30:14.304514Z","shell.execute_reply.started":"2021-06-30T07:30:14.286822Z","shell.execute_reply":"2021-06-30T07:30:14.30369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_training_dataset(): \n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True) \n    dataset = dataset.map(data_augment, num_parallel_calls=AUTOTUNE)  \n    dataset = dataset.repeat()          # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTOTUNE)   # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:14.305751Z","iopub.execute_input":"2021-06-30T07:30:14.306104Z","iopub.status.idle":"2021-06-30T07:30:14.316901Z","shell.execute_reply.started":"2021-06-30T07:30:14.306069Z","shell.execute_reply":"2021-06-30T07:30:14.316021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_validation_dataset( ordered=False): \n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered) \n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTOTUNE)\n    return dataset","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:14.318341Z","iopub.execute_input":"2021-06-30T07:30:14.318708Z","iopub.status.idle":"2021-06-30T07:30:14.328726Z","shell.execute_reply.started":"2021-06-30T07:30:14.318667Z","shell.execute_reply":"2021-06-30T07:30:14.327812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTOTUNE)\n    return dataset","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:14.330106Z","iopub.execute_input":"2021-06-30T07:30:14.330478Z","iopub.status.idle":"2021-06-30T07:30:14.338087Z","shell.execute_reply.started":"2021-06-30T07:30:14.330439Z","shell.execute_reply":"2021-06-30T07:30:14.33721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:14.339662Z","iopub.execute_input":"2021-06-30T07:30:14.340071Z","iopub.status.idle":"2021-06-30T07:30:14.3463Z","shell.execute_reply.started":"2021-06-30T07:30:14.340032Z","shell.execute_reply":"2021-06-30T07:30:14.34497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"NUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = (1 - SKIP_VALIDATION) * count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\n\nprint('Dataset: {} training images, {} validation images, {} (unlabeled) test images'.format(\n    NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:14.347974Z","iopub.execute_input":"2021-06-30T07:30:14.34848Z","iopub.status.idle":"2021-06-30T07:30:14.357107Z","shell.execute_reply.started":"2021-06-30T07:30:14.348438Z","shell.execute_reply":"2021-06-30T07:30:14.355809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualizing Dataset","metadata":{}},{"cell_type":"code","source":"from matplotlib import pyplot as plt\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case,\n                                     # these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is\n    # the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square\n    # or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n\n\ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:14.358813Z","iopub.execute_input":"2021-06-30T07:30:14.359231Z","iopub.status.idle":"2021-06-30T07:30:14.378823Z","shell.execute_reply.started":"2021-06-30T07:30:14.359194Z","shell.execute_reply":"2021-06-30T07:30:14.378067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds_iter = iter(get_training_dataset().unbatch().batch(20))","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:14.380925Z","iopub.execute_input":"2021-06-30T07:30:14.381513Z","iopub.status.idle":"2021-06-30T07:30:16.572772Z","shell.execute_reply.started":"2021-06-30T07:30:14.381472Z","shell.execute_reply":"2021-06-30T07:30:16.571741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"one_batch = next(ds_iter)\ndisplay_batch_of_images(one_batch)","metadata":{"execution":{"iopub.status.busy":"2021-06-30T07:30:16.574289Z","iopub.execute_input":"2021-06-30T07:30:16.574607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Defining Model ","metadata":{}},{"cell_type":"code","source":"# Learning rate schedule for TPU, GPU and CPU.\n# Using an LR ramp up because fine-tuning a pre-trained model.\n# Starting with a high LR would break the pre-trained weights.\n\nLR_START = 0.00001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = .8\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n\nrng = [i for i in range(25 if EPOCHS<25 else EPOCHS)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#gc.enable()\n\n#def get_training_dataset_raw():\n#    dataset = load_dataset(TRAINING_FILENAMES, labeled = True, ordered = False)\n#    return dataset\n\n#raw_training_dataset = get_training_dataset_raw()\n\n#label_counter = Counter()\n#for images, labels in raw_training_dataset:\n#    label_counter.update([labels.numpy()])\n\n#del raw_training_dataset    \n\n#mean=0\n#for i in label_counter.values():\n#    mean = mean + i\n#mean = float(round(mean/len(label_counter)))\n\n#def get_weight_for_class(class_id):\n#    counting = label_counter[class_id]\n#    weight = mean / counting\n#    return weight\n\n#weight_per_class = {class_id: get_weight_for_class(class_id) for class_id in range(104)}","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model-1","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    pretrained_model = efn.EfficientNetB7(\n        weights='noisy-student',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    pretrained_model.trainable = True\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        pretrained_model,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        #tf.keras.layers.BatchNormalization(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.compile(\n    optimizer=tf.keras.optimizers.Adam(lr=0.0001),\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy'],\n)\n\nmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataset = get_training_dataset()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"STEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nVALID_STEPS = NUM_VALIDATION_IMAGES // BATCH_SIZE\n\nhistory = model.fit(train_dataset, \n                    steps_per_epoch=STEPS_PER_EPOCH, \n                    epochs=EPOCHS,\n                    validation_data=None if SKIP_VALIDATION else get_validation_dataset(),\n                    validation_steps=None if SKIP_VALIDATION else VALID_STEPS,\n                    callbacks=[lr_callback])\n                    #class_weight = weight_per_class)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if not SKIP_VALIDATION:\n    history_frame = pd.DataFrame(history.history)\n    history_frame.loc[:, ['loss', 'val_loss']].plot()\n    history_frame.loc[:, ['sparse_categorical_accuracy', 'val_sparse_categorical_accuracy']].plot();\n    \nif SKIP_VALIDATION:\n    history_frame = pd.DataFrame(history.history)\n    history_frame.loc[:, ['loss']].plot()\n    history_frame.loc[:, ['sparse_categorical_accuracy']].plot();","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model-2","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    pretrained_model_2 = tf.keras.applications.DenseNet201(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    pretrained_model_2.trainable = True\n    \n    model2 = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        pretrained_model_2,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.BatchNormalization(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model2.compile(\n    optimizer=tf.keras.optimizers.Adam(lr=0.0001),\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy'],\n)\n\nmodel2.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"STEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nVALID_STEPS = NUM_VALIDATION_IMAGES // BATCH_SIZE\n\nhistory = model2.fit(train_dataset, \n                    steps_per_epoch=STEPS_PER_EPOCH, \n                    epochs=EPOCHS,\n                    validation_data=None if SKIP_VALIDATION else get_validation_dataset(),\n                    validation_steps=None if SKIP_VALIDATION else VALID_STEPS,\n                    callbacks=[lr_callback])\n                    #class_weight = weight_per_class)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if not SKIP_VALIDATION:\n    history_frame = pd.DataFrame(history.history)\n    history_frame.loc[:, ['loss', 'val_loss']].plot()\n    history_frame.loc[:, ['sparse_categorical_accuracy', 'val_sparse_categorical_accuracy']].plot();\n    \nif SKIP_VALIDATION:\n    history_frame = pd.DataFrame(history.history)\n    history_frame.loc[:, ['loss']].plot()\n    history_frame.loc[:, ['sparse_categorical_accuracy']].plot();","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predictions","metadata":{}},{"cell_type":"code","source":"def to_float32(image, label):\n    return tf.cast(image, tf.float32), label","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True)\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities_1 = model.predict(test_images_ds)\nprobabilities_2 = model2.predict(test_images_ds)\nprobabilities = (probabilities_1 + probabilities_2)/2\npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Generating submission.csv file...')\n\n# Get image ids from test set and convert to unicode\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n\n# Write the submission file\nnp.savetxt(\n    'submission.csv',\n    np.rec.fromarrays([test_ids, predictions]),\n    fmt=['%s', '%d'],\n    delimiter=',',\n    header='id,label',\n    comments='',\n)\n\n# Look at the first few predictions\n!head submission.csv","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}