{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Explanation\n\nThis project is made for our computer vision's project in Institut Sains dan Teknologi Terpadu Surabaya","execution_count":null},{"metadata":{"_uuid":"c07709ef-f0ed-4311-8ba8-f1f895ba00a5","_cell_guid":"73486ba3-8bb7-4a8f-bca9-cf9c95b859b1","trusted":true},"cell_type":"markdown","source":"> Please check Version 37 for best submission\n\n## Forked from :\n\n* [Getting started with 100+ flowers on TPU](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu) version 35\n* [Rotation Augmentation GPU/TPU - [0.96+]](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96) version 1\n* [GridMask data augmentation with tensorflow](https://www.kaggle.com/xiejialun/gridmask-data-augmentation-with-tensorflow) version 1\n\n## This notebook in nutshell\n\n* Image augmentation\n    * GridMask image augmentation (https://arxiv.org/abs/2001.04086)\n    * Rotate, shear, zoom, shift from [Rotation Augmentation GPU/TPU - [0.96+]](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96)\n    * [tf.image](https://www.tensorflow.org/api_docs/python/tf/image) functions\n* Ensemble\n    * EfficientNet B7 (https://arxiv.org/abs/1905.11946)\n    * DenseNet 201 (https://arxiv.org/abs/1608.06993)\n* LAMB optimizer (https://arxiv.org/abs/1904.00962)\n* Global Average Pooling (https://arxiv.org/abs/1312.4400)\n* Split transfer learning into warm up and fine tuning phase\n* TPU\n\n## Some things i tried\n\n* Optimizer (Reference from https://github.com/jettify/pytorch-optimizer)\n    * Adam optimizer\n    * Adaboost optimizer\n* Pretrained model\n    * Inception ResNet V2\n    * ResNet 152 V2\n* Activation (for MLP)\n    * ReLU\n    * Leaky ReLU\n* Fully connected\n    * MLP\n    * GAP + MLP\n    * Local Average Pooling\n* Image augmentation\n    * No augmentation\n    * Use all available augmentation\n    * Use some selected augmentation\n    * Change intensity (brightness range, zoom in, etc.) of augmentation\n    * Use CutOut (https://arxiv.org/abs/1708.04552)\n* Static learning rate\n* Different dynamic learning rate\n* No warm up phase\n* Freeze few layers (which assummed to extract low/medium level feature)\n* Use validation data **only** for validation\n* Use external data (which later found out have data leakage)\n\n## Some things i wanted to try\n\n* Optimizer\n    * Adabound (https://arxiv.org/abs/1902.09843)\n    * Diffgrad (https://arxiv.org/abs/1909.11015)\n    * Yogi (https://papers.nips.cc/paper/8186-adaptive-methods-for-nonconvex-optimization)\n    * RAdam optimizer (https://arxiv.org/abs/1908.03265)\n* Image augmentation\n    * CutMix (https://arxiv.org/abs/1905.04899)\n    * MixUp (https://arxiv.org/abs/1710.09412)\n    * AugMix (https://arxiv.org/abs/1912.02781)\n    * [tf.image.adjust_gamma](https://www.tensorflow.org/api_docs/python/tf/image/adjust_gamma)\n    * [tf.image.per_image_standardization](https://www.tensorflow.org/api_docs/python/tf/image/per_image_standardization)\n    * [tfa.image](https://www.tensorflow.org/addons/api_docs/python/tfa/image) functions\n* FixEfficientNet (https://arxiv.org/abs/2003.08237). Currently pretrained model only available for torch library\n* KFold","execution_count":null},{"metadata":{"_uuid":"66345b4c-794f-413c-90f3-18e88e5ae9a5","_cell_guid":"22cd88d0-361c-4a4a-bc69-10e8a20f147c","trusted":true},"cell_type":"markdown","source":"## Version Note\n\n* V23 (Version 43) : ?\n    * Replace Inception Resnet V2 with EfficientNetB7 (without image augmentation)\n    * Change epoch for each model during fine tuning phase\n    * Use all available image augmentation\n    * Change order of image augmentation function\n    * Tweak LR schedule\n    * Remove unused code\n* V22 (Version 37) : 0.96032\n    * Use ensemble\n    * For warm up phase :\n        * EfficientNet B7 unfreeze layer block7* and below\n        * DensetNet 201 unfreeze layer [conv/pool]5*\n        * Inception-ResNet V2 freeze all\n    * Add DenseNet 201 pre-trained model\n    * Add Inception ResNetV2 pre-trained model\n    * Disable `tf.image` image adjustment function\n    * Tweak LR schedule\n* V21 (Version 33) : 0.93536\n    * Change fine tuning epoch to 50\n    * Disable `tf.image` rotate 90 and random flip\n    * Enable `tf.image` image adjustment function\n* V20 (Version 32) : 0.91509\n    * Enable `tf.image` rotate 90 and random flip\n    * Tweak Early Stopping parameter\n    * Change fine tuning epoch to 30\n* V19 (Version 31) : 0.95642\n    * Remove MLP from model\n    * Make block6* and below of B7 is trainable during warm up phasae\n    * Implement GridMask from [GridMask data augmentation with tensorflow](https://www.kaggle.com/xiejialun/gridmask-data-augmentation-with-tensorflow)\n    * Tweak LR schedule\n    * Enable image augmentation (only GridMask)\n    * Change warm up epoch to 3 and fine tune epoch to 35\n* V18 (Version 29) : 0.91739\n    * Back to Global Average Pooling with MLP\n    * Change Early Stopping parameter\n* V17 (Version 27) : 0.94896\n    * Change Local Average Pooling parameter `pool_size` to `(8,8)` and `strides` to `8`\n    * Change Early Stopping parameter\n* V16 (Version 26) : 0.93765\n    * Back to Local Average Pooling\n    * Use early stopping callback\n    * Change Early Stopping parameter\n* V15 (Version 24) : **0.96230**\n    * Disable image augmentation\n* V14 (Version 23) : 0.94278\n    * Disable cutout\n    * Use validation for both warm up and fine tuning\n    * Tweak LR schedule\n* V13 (Version ~~21~~ 22) : 0.93404\n    * Tone down LR schedule for fine tuning phase\n    * Add sklearn classification report\n    * Only use training data for warm up and fine tuning phase\n    * Add random hue for image augmentation\n    * Disable augmentation function from [Rotation Augmentation GPU/TPU - [0.96+]](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96)\n    * Add callback to ensure 3 hours session limit used as much as possible\n    * Implement Cutout, modified from https://github.com/tensorflow/tpu/blob/f8b2febcb2f558c8ed45de65cf806cad2b3fcb62/models/official/efficientnet/imagenet_input.py#L130\n* V12 (Version 20) : 0.94971\n    * Fix swapped warm up and fine tune epoch count\n    * Warm up stage use training and validation image without augmentation\n    * Fine tuning stage use training image with augmentation\n    * 40 epochs for fine tuning\n    * Use custom LR schedule\n* V11 (Version 19) : 0.88383\n    * Remove all external dataset\n    * Use LAMB optimizer\n    * Use Global Average Pooling\n    * Add rot90 and flip to image augmentation\n    * Tone down range of some image augmentation function\n    * Replace 'imagenet' with 'noisy-student'\n    * Add some description\n    * Add show confusion matrix function from [Getting started with 100+ flowers on TPU]\n    * 5 epochs for warm-up & 35 epochs for fine tune\n* V11 (Preview 4) : 0.82457\n    * Use GPU w/ 192 * 192\n    * Use 20 epoch\n* ~~V10 (Version 12) : Timeout Exceeded~~\n    * Use all dataset\n    * No data augmentation\n    * Change epoch to 30\n* ~~V9 (Version 11) : **0.95480**~~\n    * Fix LR schedule\n    * Change `AveragePooling2D` parameter\n    * Add `Dropout` with *0.125* rate after `AveragePooling2D`\n    * Change Image Augmentation scheme\n        * Each augmentation have 1/3 or 1/2 chance to be executed. `np.random.randint()` don't have equal distribution\n        * Change insensity of each augmentation\n* ~~V8 (Version 10) : 0.93736~~\n    * Stop using validation data for training\n    * Use external dataset from [tf_flower_photo_tfrec](https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec), excluding *oxford_102* directory\n    * Change LR schedule\n    * Fix **critical** mistake (didn't update `STEPS_PER_EPOCH` value)\n    * Change epoch to 16\n* ~~V7 (Version 9) : 0.94523~~\n    * Add some description on markdown cells\n    * Use external dataset from [Oxford Flowers TFRecords](https://www.kaggle.com/cdeotte/oxford-flowers-tfrecords)\n    * Change LR schedule\n    * Change epoch to 70\n* V6 (Version 8) : **0.95009**\n    * Train all B7 layers\n    * Remove dropout\n    * Use adam optimizer\n    * Change LR schedule\n    * Use both train and val. data for training\n    * Fix **plt** function\n    * Change epoch to 50\n* V5 (Version 7) : 0.79954\n    * Train layer block5* and below of B7\n    * Data Augmentation only zoom out\n    * Use **adadelta** optimizer\n    * Change LR schedule\n    * Change epoch to 100\n* V4 (Version 6) : 0.90568\n    * Increase patience for Early Stopping\n    * Replace Local Average Pooling with Global Average Pooling\n    * Change optimizer to Adam\n    * Change epoch to 100\n* V3 (Version 5) : 0.90801\n    * Add bias to Dense (classification) layer\n    * Add fully-connected layers\n    * Use Leaky Relu for other Dense layers\n    * Increase patience for Early Stopping\n* V2 (Version 4) : **0.92266**\n    * Fix markdown heading\n    * Train all layers of B7\n    * Use 50 epoch\n    * More aggresive Image Augmentation\n    * Remove B7+ summary\n    * Reduce Early stopping val acc./loss by 90%\n    * Change Local Average Pooling to `AveragePooling2D(pool_size=(4,4), strides=3, padding='valid')`\n* V1 (Version 3) : **0.91153**\n    * Train layers block6* and below of B7\n    * Use 10 epoch\n    * MXU usage is quite low with high idle time on training model. Will try [Data Augmentation using GPU/TPU for Maximum Speed!](https://www.kaggle.com/yihdarshieh/make-chris-deotte-s-data-augmentation-faster)","execution_count":null},{"metadata":{"_uuid":"623ff6ab-b389-478a-bb33-0283fb5c598c","_cell_guid":"85c5754b-ba0b-468a-8be6-2d16d44a7162","trusted":true},"cell_type":"markdown","source":"## Install, load and configure library","execution_count":null},{"metadata":{"_uuid":"17e34b54-d8fa-4334-ad33-f1a6effdc269","_cell_guid":"88a00a56-7fb7-408a-b986-968547f17ff3","trusted":true},"cell_type":"code","source":"!pip install efficientnet tensorflow_addons==0.9.1 tensorflow==2.1.0","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"67cc7fe6-ae69-4f8c-a148-dec8d5aa3d8f","_cell_guid":"16760e82-ac7c-46c4-b0ef-31b275e849b6","trusted":true},"cell_type":"code","source":"import math\nimport re\nimport random\nimport os\nimport tensorflow as tf\nimport tensorflow_addons as tfa\nimport numpy as np\nimport tensorflow.keras.backend as K\nimport efficientnet.tfkeras as efn\nfrom matplotlib import pyplot as plt\nfrom datetime import datetime","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e6863204-a209-4f49-94fc-fb328eeb7e1c","_cell_guid":"74714d96-44ec-4d90-b870-29c0e87aa1ba","trusted":true},"cell_type":"code","source":"print(f'Numpy version : {np.__version__}')\nprint(f'Tensorflow version {tf.__version__}')\n\nAUTO = tf.data.experimental.AUTOTUNE\nSTART_TIME = datetime.now()\n\n# this won't make result reproducible, unless you use CPU\nSEED = 42\nos.environ['PYTHONHASHSEED']=str(SEED)\nrandom.seed(SEED)\nnp.random.seed(SEED)\ntf.random.set_seed(SEED)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"db35ac57-c2d9-407e-8cfb-ac12931831ca","_cell_guid":"0a918e64-d733-4859-8ce0-2289ec35798c","trusted":true},"cell_type":"markdown","source":"## TPU or GPU detection","execution_count":null},{"metadata":{"_uuid":"295fe847-8f32-4f03-b9fc-dec73b13f846","_cell_guid":"f5ad7339-2b74-4954-a472-5ee0b1c458ee","trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bdfe27e7-5246-4a4f-8414-8435e186e98c","_cell_guid":"36731e1d-c50c-4f71-9459-e0dde8cfc3f1","trusted":true},"cell_type":"markdown","source":"## Competition data access\n\nTPUs read data directly from Google Cloud Storage (GCS). This Kaggle utility will copy the dataset to a GCS bucket co-located with the TPU. If you have multiple datasets attached to the notebook, you can pass the name of a specific dataset to the get_gcs_path function. The name of the dataset is the name of the directory it is mounted in. Use `!ls /kaggle/input/` to list attached datasets.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls -lha /kaggle/input/","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8d7d5846-689c-4812-9f83-4fbdabfbee1d","_cell_guid":"15239699-06e5-49c2-b028-8676f53361a4","trusted":true},"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\n\nGCS_DS_PATH = KaggleDatasets().get_gcs_path() # you can list the bucket with \"!gsutil ls $GCS_DS_PATH\"\nGCS_DS_PATH","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"39299429-f9ac-4486-bc84-e9701b883854","_cell_guid":"fef69ec7-a513-45ff-adf2-893e96b0e4f9","trusted":true},"cell_type":"markdown","source":"# Configuration","execution_count":null},{"metadata":{"_uuid":"72639119-2c24-424f-8b3e-e19c8ec8d1e7","_cell_guid":"82195b29-693e-4440-8478-7971eda7ae15","trusted":true},"cell_type":"code","source":"IMAGE_SIZE = [512, 512]\n\nVALIDATE_WARMUP = False\nEPOCHS_WARMUP = 3\nDO_AUG_WARMUP = False\n\nVALIDATE = False\nEPOCHS = 30\nDO_AUG = True\n\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n\nGCS_PATH_SELECT = { # available image sizes\n    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'\n}\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') # predictions on this dataset should be submitted for the competition\n\nCLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 102","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e6e318a2-7708-406f-b33f-6ce1dcc6c2ef","_cell_guid":"6f314c57-ac5c-409d-90e5-afb416e915c7","trusted":true},"cell_type":"markdown","source":"# Datasets functions","execution_count":null},{"metadata":{"_uuid":"28268dba-da6e-4ebd-b4be-7da94aab2110","_cell_guid":"d0064349-32a9-4fed-80a2-31868d60681d","trusted":true},"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset\n\ndef get_train_val_dataset(do_aug=True):\n    dataset = load_dataset(TRAINING_FILENAMES + VALIDATION_FILENAMES, labeled=True)\n    if do_aug:\n        dataset = dataset.map(image_augmentation, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_training_dataset(do_aug=True):\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    if do_aug:\n        dataset = dataset.map(image_augmentation, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\n\nprint(f'Total training image: {NUM_TRAINING_IMAGES}')\nprint(f'Total validation image: {NUM_VALIDATION_IMAGES}')\nprint(f'Total test image: {NUM_TEST_IMAGES}')\nprint(f'Steps per epoch : {STEPS_PER_EPOCH}')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"98e001f9-c2e8-4bc0-84d8-c9d02504e3a6","_cell_guid":"b829d37c-6c9d-486b-9ed1-533d805a49c5","trusted":true},"cell_type":"markdown","source":"# Image augmentation functions\n\n| Function   | Chance | Range                             |\n| ---------- | ------ | --------------------------------- |\n| Flip       | 50%    | Only Left to right                |\n| Brightness | 50%    | 0.9 to 1.1                        |\n| Contrast   | 50%    | 0.9 to 1.1                        |\n| Saturation | 50%    | 0.9 to 1.1                        |\n| Hue        | 50%    | 0.05                              |\n| Rotate     | 50%    | 17 degrees * normal distribution  |\n| Shear      | 50%    | 5.5 degrees * normal distribution |\n| Zoom Out   | 33%    | 1.0 - (normal distribution / 8.5) |\n| Shift      | 33%    | 18 pixel * normal distribution    |\n| GridMask   | 50%    | 100 - 160 pixel black rectangle   |\n|            |        | with same pixel range for gap     |","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Rotate, shear, zoom, shift","execution_count":null},{"metadata":{"_uuid":"299e214a-a4a5-456f-88e2-d1975caf9b24","_cell_guid":"afb27760-e259-4623-92ee-2a5be4d6adb9","trusted":true},"cell_type":"code","source":"def get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n    # returns 3x3 transformmatrix which transforms indicies\n        \n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    shear = math.pi * shear / 180.\n    \n    # ROTATION MATRIX\n    c1 = tf.math.cos(rotation)\n    s1 = tf.math.sin(rotation)\n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    rotation_matrix = tf.reshape( tf.concat([c1,s1,zero, -s1,c1,zero, zero,zero,one],axis=0),[3,3] )\n        \n    # SHEAR MATRIX\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)\n    shear_matrix = tf.reshape( tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3] )    \n    \n    # ZOOM MATRIX\n    zoom_matrix = tf.reshape( tf.concat([one/height_zoom,zero,zero, zero,one/width_zoom,zero, zero,zero,one],axis=0),[3,3] )\n    \n    # SHIFT MATRIX\n    shift_matrix = tf.reshape( tf.concat([one,zero,height_shift, zero,one,width_shift, zero,zero,one],axis=0),[3,3] )\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), K.dot(zoom_matrix, shift_matrix))\n\n\ndef transform(image):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated, sheared, zoomed, and shifted\n    DIM = IMAGE_SIZE[0]\n    XDIM = DIM%2 #fix for size 331\n    \n    if np.random.randint(0, 2) == 0: # 50% chance\n        rot = 17. * tf.random.normal([1],dtype='float32')\n    else:\n        rot = tf.constant([0],dtype='float32')\n    \n    if np.random.randint(0, 2) == 0: # 50% chance\n        shr = 5.5 * tf.random.normal([1],dtype='float32') \n    else:\n        shr = tf.constant([0],dtype='float32')\n    \n    if np.random.randint(0, 3) == 0: # 33% chance\n        h_zoom = tf.random.normal([1],dtype='float32')/8.5\n        if h_zoom > 0:\n            h_zoom = 1.0 + h_zoom * -1\n        else:\n            h_zoom = 1.0 + h_zoom\n    else:\n        h_zoom = tf.constant([1],dtype='float32')\n    \n    if np.random.randint(0, 3) == 0: # 33% chance\n        w_zoom = tf.random.normal([1],dtype='float32')/8.5\n        if w_zoom > 0:\n            w_zoom = 1.0 + w_zoom * -1\n        else:\n            w_zoom = 1.0 + w_zoom\n    else:\n        w_zoom = tf.constant([1],dtype='float32')\n    \n    if np.random.randint(0, 3) == 0: # 33% chance\n        h_shift = 18. * tf.random.normal([1],dtype='float32') \n    else:\n        h_shift = tf.constant([0],dtype='float32')\n    \n    if np.random.randint(0, 3) == 0: # 33% chance\n        w_shift = 18. * tf.random.normal([1],dtype='float32') \n    else:\n        w_shift = tf.constant([0],dtype='float32')\n  \n    # GET TRANSFORMATION MATRIX\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    # LIST DESTINATION PIXEL INDICES\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(m,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES           \n    idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image,tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM,DIM,3])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## GridMask","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def transform_grid_mark(image, inv_mat, image_shape):\n    h, w, c = image_shape\n    cx, cy = w//2, h//2\n\n    new_xs = tf.repeat( tf.range(-cx, cx, 1), h)\n    new_ys = tf.tile( tf.range(-cy, cy, 1), [w])\n    new_zs = tf.ones([h*w], dtype=tf.int32)\n\n    old_coords = tf.matmul(inv_mat, tf.cast(tf.stack([new_xs, new_ys, new_zs]), tf.float32))\n    old_coords_x, old_coords_y = tf.round(old_coords[0, :] + w//2), tf.round(old_coords[1, :] + h//2)\n\n    clip_mask_x = tf.logical_or(old_coords_x<0, old_coords_x>w-1)\n    clip_mask_y = tf.logical_or(old_coords_y<0, old_coords_y>h-1)\n    clip_mask = tf.logical_or(clip_mask_x, clip_mask_y)\n\n    old_coords_x = tf.boolean_mask(old_coords_x, tf.logical_not(clip_mask))\n    old_coords_y = tf.boolean_mask(old_coords_y, tf.logical_not(clip_mask))\n    new_coords_x = tf.boolean_mask(new_xs+cx, tf.logical_not(clip_mask))\n    new_coords_y = tf.boolean_mask(new_ys+cy, tf.logical_not(clip_mask))\n\n    old_coords = tf.cast(tf.stack([old_coords_y, old_coords_x]), tf.int32)\n    new_coords = tf.cast(tf.stack([new_coords_y, new_coords_x]), tf.int64)\n    rotated_image_values = tf.gather_nd(image, tf.transpose(old_coords))\n    rotated_image_channel = list()\n    for i in range(c):\n        vals = rotated_image_values[:,i]\n        sparse_channel = tf.SparseTensor(tf.transpose(new_coords), vals, [h, w])\n        rotated_image_channel.append(tf.sparse.to_dense(sparse_channel, default_value=0, validate_indices=False))\n\n    return tf.transpose(tf.stack(rotated_image_channel), [1,2,0])\n\ndef random_rotate(image, angle, image_shape):\n    def get_rotation_mat_inv(angle):\n          #transform to radian\n        angle = math.pi * angle / 180\n\n        cos_val = tf.math.cos(angle)\n        sin_val = tf.math.sin(angle)\n        one = tf.constant([1], tf.float32)\n        zero = tf.constant([0], tf.float32)\n\n        rot_mat_inv = tf.concat([cos_val, sin_val, zero,\n                                     -sin_val, cos_val, zero,\n                                     zero, zero, one], axis=0)\n        rot_mat_inv = tf.reshape(rot_mat_inv, [3,3])\n\n        return rot_mat_inv\n    angle = float(angle) * tf.random.normal([1],dtype='float32')\n    rot_mat_inv = get_rotation_mat_inv(angle)\n    return transform_grid_mark(image, rot_mat_inv, image_shape)\n\n\ndef GridMask():\n    h, w = IMAGE_SIZE[0], IMAGE_SIZE[1]\n    image_height, image_width = (h, w)\n    d1 = 100\n    d2 = 160\n    rotate_angle = 45 # 1\n    ratio = 0.5\n\n    hh = int(np.ceil(np.sqrt(h*h+w*w)))\n    hh = hh+1 if hh%2==1 else hh\n    d = tf.random.uniform(shape=[], minval=d1, maxval=d2, dtype=tf.int32)\n    l = tf.cast(tf.cast(d,tf.float32)*ratio+0.5, tf.int32)\n\n    st_h = tf.random.uniform(shape=[], minval=0, maxval=d, dtype=tf.int32)\n    st_w = tf.random.uniform(shape=[], minval=0, maxval=d, dtype=tf.int32)\n\n    y_ranges = tf.range(-1 * d + st_h, -1 * d + st_h + l)\n    x_ranges = tf.range(-1 * d + st_w, -1 * d + st_w + l)\n\n    for i in range(0, hh//d+1):\n        s1 = i * d + st_h\n        s2 = i * d + st_w\n        y_ranges = tf.concat([y_ranges, tf.range(s1,s1+l)], axis=0)\n        x_ranges = tf.concat([x_ranges, tf.range(s2,s2+l)], axis=0)\n\n    x_clip_mask = tf.logical_or(x_ranges <0 , x_ranges > hh-1)\n    y_clip_mask = tf.logical_or(y_ranges <0 , y_ranges > hh-1)\n    clip_mask = tf.logical_or(x_clip_mask, y_clip_mask)\n\n    x_ranges = tf.boolean_mask(x_ranges, tf.logical_not(clip_mask))\n    y_ranges = tf.boolean_mask(y_ranges, tf.logical_not(clip_mask))\n\n    hh_ranges = tf.tile(tf.range(0,hh), [tf.cast(tf.reduce_sum(tf.ones_like(x_ranges)), tf.int32)])\n    x_ranges = tf.repeat(x_ranges, hh)\n    y_ranges = tf.repeat(y_ranges, hh)\n\n    y_hh_indices = tf.transpose(tf.stack([y_ranges, hh_ranges]))\n    x_hh_indices = tf.transpose(tf.stack([hh_ranges, x_ranges]))\n\n    y_mask_sparse = tf.SparseTensor(tf.cast(y_hh_indices, tf.int64),  tf.zeros_like(y_ranges), [hh, hh])\n    y_mask = tf.sparse.to_dense(y_mask_sparse, 1, False)\n\n    x_mask_sparse = tf.SparseTensor(tf.cast(x_hh_indices, tf.int64), tf.zeros_like(x_ranges), [hh, hh])\n    x_mask = tf.sparse.to_dense(x_mask_sparse, 1, False)\n\n    mask = tf.expand_dims( tf.clip_by_value(x_mask + y_mask, 0, 1), axis=-1)\n\n    mask = random_rotate(mask, rotate_angle, [hh, hh, 1])\n    mask = tf.image.crop_to_bounding_box(mask, (hh-h)//2, (hh-w)//2, image_height, image_width)\n\n    return mask\n\ndef apply_grid_mask(image):\n    mask = GridMask()\n    mask = tf.concat([mask, mask, mask], axis=-1)\n\n    return image * tf.cast(mask, 'float32')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Augmentation function & tf.image functions (flip, brightness, contrast, saturation, hue)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def image_augmentation(image, label):\n    image = tf.image.random_flip_left_right(image)\n    if np.random.randint(0, 2) == 0: # 50% chance\n        image = tf.image.random_brightness(image, 0.1)\n    if np.random.randint(0, 2) == 0: # 50% chance\n        image = tf.image.random_contrast(image, 0.9, 1.1)\n    if np.random.randint(0, 2) == 0: # 50% chance\n        image = tf.image.random_saturation(image, 0.9, 1.1)\n    if np.random.randint(0, 2) == 0: # 50% chance\n        image = tf.image.random_hue(image, 0.05)\n\n    image = transform(image)\n\n    if np.random.randint(0, 2) == 0:\n        image = apply_grid_mask(image)\n\n    return image, label","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1eff3b87-cea0-4659-a2b5-a16b291c9629","_cell_guid":"376e507c-1201-4bc1-a90a-ad3a821d7bd8","trusted":true},"cell_type":"markdown","source":"# Show augmentated image","execution_count":null},{"metadata":{"_uuid":"242e542b-89e5-4c4d-a6b5-dc0bb8421695","_cell_guid":"48e61c4c-31ad-4979-a540-92dc306c636c","trusted":true},"cell_type":"code","source":"def show_augmented_image(same_image=True):\n    row, col = 3, 5\n    if same_image:\n        all_elements = get_training_dataset(do_aug=False).unbatch()\n        one_element = tf.data.Dataset.from_tensors( next(iter(all_elements)) )\n        augmented_element = one_element.repeat().map(image_augmentation).batch(row*col)\n        for img, label in augmented_element:\n            plt.figure(figsize=(15,int(15*row/col)))\n            for j in range(row*col):\n                plt.subplot(row,col,j+1)\n                plt.axis('off')\n                plt.imshow(img[j,])\n            plt.suptitle(CLASSES[label[0]])\n            plt.show()\n            break\n    else:\n        all_elements = get_training_dataset().unbatch()\n        augmented_element = all_elements.batch(row*col)\n\n        for img, label in augmented_element:\n            plt.figure(figsize=(15,int(15*row/col)))\n            for j in range(row*col):\n                plt.subplot(row,col,j+1)\n                plt.title(CLASSES[label[j]])\n                plt.axis('off')\n                plt.imshow(img[j,])\n            plt.show()\n            break","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7bd5915e-ee0e-4993-9662-3b50c7ec7911","_cell_guid":"de535aa0-e757-49ad-b4e0-4d31b48dec69","trusted":true},"cell_type":"code","source":"# run again to see different batch of image\nshow_augmented_image()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# run again to see different image\nshow_augmented_image(same_image=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a75f451d-4205-4cce-bd9c-13f0e55f8daf","_cell_guid":"1b4398f1-19d6-454d-a67a-839179a63d67","trusted":true},"cell_type":"markdown","source":"# Functions for model training","execution_count":null},{"metadata":{"_uuid":"9d6e4962-d6b1-4a1f-9ceb-5835b2d3a28f","_cell_guid":"91c40909-1c30-4b55-a5ab-7ffc789eab6f","trusted":true},"cell_type":"code","source":"from tensorflow.keras.models import Model, Sequential\nfrom tensorflow.keras.layers import Input, Flatten, Dense, Dropout, AveragePooling2D, GlobalAveragePooling2D, SpatialDropout2D","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plt_lr(epoch_count):\n    if epoch_count > 50:\n        epoch_count = 50\n    \n    rng = [i for i in range(epoch_count)]\n\n    plt.figure()\n    y = [lrfn(x) for x in rng]\n    plt.title(f'Learning rate schedule: {y[0]} to {y[epoch_count-1]}')\n    plt.plot(rng, y)\n\ndef plt_acc(h):\n    plt.figure()\n    plt.plot(h.history[\"sparse_categorical_accuracy\"])\n    if 'val_sparse_categorical_accuracy' in h.history:\n        plt.plot(h.history[\"val_sparse_categorical_accuracy\"]) \n        plt.legend([\"training\",\"validation\"])       \n    else:\n        plt.legend([\"training\"])\n    plt.xlabel(\"epoch\")\n    plt.title(\"Sparse Categorical Accuracy\")\n    plt.show()\n\ndef plt_loss(h):\n    plt.figure()\n    plt.plot(h.history[\"loss\"])\n    if 'val_loss' in h.history:\n        plt.plot(h.history[\"val_loss\"]) \n        plt.legend([\"training\",\"validation\"])       \n    else:\n        plt.legend([\"training\"])\n    plt.legend([\"training\",\"validation\"])\n    plt.xlabel(\"epoch\")\n    plt.title(\"Loss\")\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b0943e9-1066-438e-ad1a-ce2dc41fb4d4","_cell_guid":"838b477a-e39b-4403-b5bf-e4314f77d88a","trusted":true},"cell_type":"code","source":"es_val_acc = tf.keras.callbacks.EarlyStopping(\n    monitor='val_sparse_categorical_accuracy', min_delta=0.001, patience=5, verbose=1, mode='auto',\n    baseline=None, restore_best_weights=True\n)\n\nes_val_loss = tf.keras.callbacks.EarlyStopping(\n    monitor='val_loss', min_delta=0.001, patience=5, verbose=1, mode='auto',\n    baseline=None, restore_best_weights=True\n)\n\nes_acc = tf.keras.callbacks.EarlyStopping(\n    monitor='sparse_categorical_accuracy', min_delta=0.001, patience=5, verbose=1, mode='auto',\n    baseline=None, restore_best_weights=False\n)\n\nes_loss = tf.keras.callbacks.EarlyStopping(\n    monitor='loss', min_delta=0.001, patience=5, verbose=1, mode='auto',\n    baseline=None, restore_best_weights=False\n)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"57f6b8cc-ed53-4ca1-acef-a281f81e974d","_cell_guid":"7f497bc4-6949-431a-9fbf-e92a7a346ec5","trusted":true},"cell_type":"markdown","source":"# Create model","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## EfficientNetB7 model\n\n| Layer     | Layer Type                   |\n| --------- | ---------------------------- |\n| 0         | input_1 (InputLayer)         |\n| 1         | stem_conv (Conv2D)           |\n| 2         | stem_bn (BatchNormalization) |\n| 3         | stem_activation (Activation) |\n| 4-49      | block1*                      |\n| 50 - 152  | block2*                      |\n| 153 - 255 | block3*                      |\n| 256 - 403 | block4*                      |\n| 404 - 551 | block5*                      |\n| 552 - 744 | block6*                      |\n| 745 - 802 | block7*                      |\n| 803       | top_conv (Conv2D)            |\n| 804       | top_bn (BatchNormalization)  |\n| 805       | top_activation (Activation)  |","execution_count":null},{"metadata":{"_uuid":"9962e8eb-1ccf-4879-977f-f684f1cd797e","_cell_guid":"c4c4b445-3b0a-4cf4-a272-0094c8698f01","trusted":true},"cell_type":"code","source":"with strategy.scope():\n    efn7 = efn.EfficientNetB7(weights='noisy-student', include_top=False, input_shape=(IMAGE_SIZE[0], IMAGE_SIZE[1], 3))\n    for layer in efn7.layers[:745]:\n        layer.trainable = False\n    for layer in efn7.layers[745:]:\n        layer.trainable = True\n\n    model = Sequential([\n        efn7,\n        GlobalAveragePooling2D(),\n#         AveragePooling2D(pool_size=(8,8), strides=8, padding='valid'), # valid -> drop leftover pixel, same -> add padding\n#         Flatten(),\n#         Dense(512, activation='relu'),\n        Dense(len(CLASSES), activation='softmax')\n    ], name='b7-flower')\n    \n    from tensorflow.keras.applications.densenet import DenseNet201\n    densenet201 = DenseNet201(weights='imagenet', include_top=False, input_shape=(IMAGE_SIZE[0], IMAGE_SIZE[1], 3))\n    \nmodel.compile(optimizer=tfa.optimizers.LAMB(), loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## DenseNet 201 model\n\n| Layer     | Layer Type                      |\n| --------- | ------------------------------- |\n| 0         | input_5 (InputLayer)            |\n| 1         | zero_padding2d (ZeroPadding2D)  |\n| 2 - 6     | [conv/pool]1*                   |\n| 7 - 52    | [conv/pool]2*                   |\n| 53 - 140  | [conv/pool]3*                   |\n| 141 - 480 | [conv/pool]4*                   |\n| 481 - 704 | [conv/pool]5*                   |\n| 705       | bn (BatchNormalization)         |\n| 706       | relu (Activation)               |","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    densenet201 = tf.keras.applications.densenet.DenseNet201(weights='imagenet', include_top=False, input_shape=(IMAGE_SIZE[0], IMAGE_SIZE[1], 3))\n    for layer in densenet201.layers[:481]:\n        layer.trainable = False\n    for layer in densenet201.layers[481:]:\n        layer.trainable = True\n\n    model2 = Sequential([\n        densenet201,\n        GlobalAveragePooling2D(),\n#         AveragePooling2D(pool_size=(8,8), strides=8, padding='valid'), # valid -> drop leftover pixel, same -> add padding\n#         Flatten(),\n#         Dense(512, activation='relu'),\n        Dense(len(CLASSES), activation='softmax')\n    ], name='dn201-flower')\n    \nmodel2.compile(optimizer=tfa.optimizers.LAMB(), loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\nmodel2.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eb4d331c-adb8-45d8-9505-f39549d77ae5","_cell_guid":"87f40026-6011-488d-8ece-e0cd95390238","trusted":true},"cell_type":"markdown","source":"# Warm up model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def lrfn(epoch):\n    initial_lr = 0.0001\n    current_lr = initial_lr + 0.00015 * epoch\n\n    return current_lr\n\nlr_schedule = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\nplt_lr(EPOCHS_WARMUP)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"80b794b5-89ea-4858-b456-de3d61cd7eae","_cell_guid":"47b4b0f6-3744-45d9-9ecf-72b7617a4b7a","trusted":true},"cell_type":"code","source":"if VALIDATE_WARMUP:\n    model.fit(\n        get_training_dataset(do_aug=DO_AUG_WARMUP), steps_per_epoch=STEPS_PER_EPOCH, epochs=EPOCHS_WARMUP,\n        validation_data=get_validation_dataset(), callbacks=[lr_schedule], verbose=1\n    )\nelse:\n    model.fit(\n        get_train_val_dataset(do_aug=DO_AUG_WARMUP), steps_per_epoch=STEPS_PER_EPOCH, epochs=EPOCHS_WARMUP,\n        callbacks=[lr_schedule], verbose=1\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"h = model.history\nplt_acc(h)\nplt_loss(h)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if VALIDATE_WARMUP:\n    model2.fit(\n        get_training_dataset(do_aug=DO_AUG_WARMUP), steps_per_epoch=STEPS_PER_EPOCH, epochs=EPOCHS_WARMUP,\n        validation_data=get_validation_dataset(), callbacks=[lr_schedule], verbose=1\n    )\nelse:\n    model2.fit(\n        get_train_val_dataset(do_aug=DO_AUG_WARMUP), steps_per_epoch=STEPS_PER_EPOCH, epochs=EPOCHS_WARMUP,\n        callbacks=[lr_schedule], verbose=1\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"h = model2.history\nplt_acc(h)\nplt_loss(h)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Fine tune model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"LR_START = 0.0006\nLR_MAX = 0.003 #strategy.num_replicas_in_sync\nLR_MIN = 0.0003\nLR_RAMPUP_EPOCHS = 4\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = 0.91\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\nlr_schedule = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n\nplt_lr(EPOCHS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for layer in model.layers:\n    layer.trainable = True\nmodel.compile(optimizer=tfa.optimizers.LAMB(), loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for layer in model2.layers:\n    layer.trainable = True\nmodel2.compile(optimizer=tfa.optimizers.LAMB(), loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\nmodel2.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if VALIDATE:\n    model.fit(\n        get_training_dataset(do_aug=DO_AUG), steps_per_epoch=STEPS_PER_EPOCH, epochs=35,\n        validation_data=get_validation_dataset(), callbacks=[es_acc, lr_schedule], verbose=1\n    )\nelse:\n    model.fit(\n        get_train_val_dataset(do_aug=DO_AUG), steps_per_epoch=STEPS_PER_EPOCH, epochs=35,\n        callbacks=[es_val_acc, lr_schedule], verbose=1\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"h = model.history\nplt_acc(h)\nplt_loss(h)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if VALIDATE:\n    model2.fit(\n        get_training_dataset(do_aug=DO_AUG), steps_per_epoch=STEPS_PER_EPOCH, epochs=30,\n        validation_data=get_validation_dataset(), callbacks=[es_acc, lr_schedule], verbose=1\n    )\nelse:\n    model2.fit(\n        get_train_val_dataset(do_aug=DO_AUG), steps_per_epoch=STEPS_PER_EPOCH, epochs=30,\n        callbacks=[es_val_acc, lr_schedule], verbose=1\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"h = model2.history\nplt_acc(h)\nplt_loss(h)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"POST_TRAINING_TIME_START = datetime.now()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"04d182d0-98d2-41a3-ab84-0782ca1fa5bf","_cell_guid":"51be5d27-2fcc-41c1-890c-94cf4bfd4554","trusted":true},"cell_type":"markdown","source":"# Evaluate functions\n\nNote : Evalation is useless when you include validation data for training","execution_count":null},{"metadata":{"_uuid":"0fb26b33-c066-40c3-8ff3-f8b5111b795e","_cell_guid":"60df0d9f-637b-4778-bf28-f0595ea8511d","trusted":true},"cell_type":"code","source":"from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, classification_report\n    \ndef display_confusion_matrix(cmat, score, precision, recall, acc):\n    plt.figure(figsize=(15,15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.4f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.4f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.4f} '.format(recall)\n    if recall is not None:\n        titlestring += '\\naccuracy = {:.4f} '.format(acc)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    plt.show()\n\ndef evaluate(model):\n    cmdataset = get_validation_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and labels, order matters.\n    images_ds = cmdataset.map(lambda image, label: image)\n    labels_ds = cmdataset.map(lambda image, label: label).unbatch()\n    cm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch\n    \n    if type(model) == list:\n        cm_model_pred = model[0].predict(images_ds)\n        cm_model2_pred = model[1].predict(images_ds)\n        cm_probabilities = (cm_model_pred * 0.56) + (cm_model2_pred * 0.44)\n    else:\n        cm_probabilities = model.predict(images_ds)\n    cm_predictions = np.argmax(cm_probabilities, axis=-1)\n    print(f'Correct   labels: {cm_correct_labels.shape} {cm_correct_labels}')\n    print(f'Predicated labels: {cm_predictions.shape} {cm_predictions}')\n\n    cmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))\n    score = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\n    acc = accuracy_score(cm_correct_labels, cm_predictions)\n    precision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\n    recall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\n    cmat = (cmat.T / cmat.sum(axis=1)).T # normalized\n    \n    print(classification_report(cm_correct_labels, cm_predictions, labels=range(len(CLASSES))))\n\n    print(f'F1 score: {score}')\n    print(f'Precision: {precision}')\n    print(f'Recall: {recall}')\n    print(f'Accuracy: {acc}')\n    \n    return cmat, score, precision, recall, acc","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b3775e86-2dd5-4132-ae9c-d09653408eb5","_cell_guid":"cf5ae7a3-f458-4611-9ab9-73a0d36153d8","trusted":true},"cell_type":"markdown","source":"# Evaluate Model","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Evaluate Model 1 (EfficientNetB7)","execution_count":null},{"metadata":{"_uuid":"315043e1-bb82-4f9e-b36c-2b800dd584ff","_cell_guid":"f5d24fa1-2522-42a6-be5d-7f27fb06e214","trusted":true},"cell_type":"code","source":"if VALIDATE:\n    cmat, score, precision, recall, acc = evaluate(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if VALIDATE:\n    display_confusion_matrix(cmat, score, precision, recall, acc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluate Model 2 (DensetNet 201)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if VALIDATE:\n    cmat, score, precision, recall, acc = evaluate(model2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if VALIDATE:\n    display_confusion_matrix(cmat, score, precision, recall, acc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluate Ensemble model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if VALIDATE:\n    cmat, score, precision, recall, acc = evaluate([model, model2])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if VALIDATE:\n    display_confusion_matrix(cmat, score, precision, recall, acc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visual Model Evaluation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_predict_val(m):\n    row, col = 3, 5\n    \n    dataset = get_validation_dataset()\n    dataset = dataset.unbatch().batch(row * col)\n    images, labels = next(iter(dataset))\n\n    probabilities = m.predict(images)\n    predictions = np.argmax(probabilities, axis=-1)\n    \n    plt.figure(figsize=(15,int(15*row/col)))\n    for i in range(row*col):        \n        plt.subplot(row,col,i+1)\n        \n        pred = CLASSES[predictions[i]]\n        real = CLASSES[labels[i]]\n        if pred == real:\n            plt.title(f'{pred} [OK]')\n        else:\n            plt.title(f'{pred} [NO -> {real}]', color='red')\n\n        plt.axis('off')\n        plt.imshow(images[i,])\n    \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if VALIDATE:\n    show_predict_val(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if VALIDATE:\n    show_predict_val(model2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"231b8bbe-b450-4289-add7-94959a8dbc1a","_cell_guid":"ad997638-35d4-4eb7-a578-7551d026cde2","trusted":true},"cell_type":"markdown","source":"# Submit Result","execution_count":null},{"metadata":{"_uuid":"f0290e52-599b-46cc-9bce-52578c388915","_cell_guid":"b080a816-6493-4c78-acf8-44efb66fac86","trusted":true},"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nmodel_pred = model.predict(test_images_ds)\nmodel2_pred = model2.predict(test_images_ds)\n\nprobabilities = (model_pred * 0.56) + (model2_pred * 0.44)\npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)\n\nprint('Generating submission.csv file...')\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\nnp.savetxt('submission.csv', np.rec.fromarrays([test_ids, predictions]), fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')\n!head submission.csv","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Post training time : {(datetime.now() - POST_TRAINING_TIME_START).total_seconds()} seconds')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}