{"cells":[{"metadata":{},"cell_type":"markdown","source":"In this kernel, I'll show where I failed and how to fix it with winning solutions. \n\n---\n## TL;DR\nMy original model finished public LB 0.9877 with single model, but private LB 0.9035. Obviously I failed to predict unseen labels. I naively assumed public and private were splited randomly, but of course not. My model predict very welll with seen labels, that means it can be still useful to classify seen/unseen labels. After I leaned many things from winner's solutions, I combined these with my models. Here is the result.\n\nUpdate: After I opened this kernel, Chris @cdeotte advised me how to post process correctly. Thanks, it's really amazing!\n\n| No | model                                | cv    | public | private | rank   |\n|----|--------------------------------------|-------|--------|---------|--------|\n| 1  | decode 1292 unique labels (original) | 0.999 | 0.9870 | 0.9035  | 1416th |\n| 2  | 3 heads for each r, v, c             |       | 0.9891 | 0.9228  | 639th  |\n| 3  | 3 models for each r, v, c            |       | 0.9665 | 0.9383  | 56th   |\n| 4  | seen: No.1 unseen: No.3              |       | 0.9925 | 0.9552  | 8th    |\n| 5  | No.4 with postprocess                |       | 0.9941 | 0.9704  | 2th    |\n\n---\n\n## REFERENCES\nI want to say thank you to the author of these references. All of them are great and my code was just conbined these.\n\n### TPU BASELINE\nI started from see---'s really great kernels which use TPU as a accelerator. I realized how TPU runs faster than GPU and how easy to use it with keras compared to pytorch xla. Many thanks to see---.\n1. https://www.kaggle.com/seesee/1-create-tfrecords\n2. https://www.kaggle.com/seesee/2-train\n3. https://www.kaggle.com/seesee/3-submit\n\n### TPU DATA AUGMENTATIONS\nAs see-- himself says, above kernels are plain vanilla one, which means there is a lot of room to improve it. The first thing I wanted to try was data augmentation. But with TPU it's not so easy to apply data augmentation, at least it was for me. But fortunately I found there are many useful kernels to learn how to hundle TPU in [this copmetion](https://www.kaggle.com/c/flower-classification-with-tpus/notebooks), I also want to appreciate these kernel's authors.\n1. https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n2. https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu\n3. https://www.kaggle.com/xiejialun/gridmask-data-augmentation-with-tensorflow\n\n### METRIC LEARNING\nAfter playing around with data augmentation and EfficientNet architectures, my cv score was stacked around 0.986 untils 3~4 days before the deadline. Then I finally found metric learning approach was useful to deal with relatively rare and difficult classes. I mainly refered HumpBack competition's solutions, but I borrowed keras code from the kernel mentioned bellow.\n\nreference solutions\n1. https://www.kaggle.com/c/humpback-whale-identification/discussion/82366\n2. https://www.kaggle.com/c/humpback-whale-identification/discussion/83885\n3. https://www.kaggle.com/c/humpback-whale-identification/discussion/82484\n\ncode comes from\n1. https://github.com/4uiiurz1/keras-arcface\n\n---\n\n## AFTER FINISHING COMPETITIONS\n\n### SEEN/UNSEEN CLASSIFICATION\n\nHere is the good summary of winner's solution.\nhttps://www.kaggle.com/c/bengaliai-cv19/discussion/136030\n\nAs most of winner says, the most important things to achieve good result in private lb was to deal with unseen data. I borrowed the idea to classify seen/unseen by Arcface moudle output from [3rd](https://www.kaggle.com/c/bengaliai-cv19/discussion/135982) or [8th](https://www.kaggle.com/c/bengaliai-cv19/discussion/135990).\n\n### POSTPROCESSING\n\nAnd also it seems postprocessing to optimize the competition metric macro recall may also work, but somehow I couldn't get good result. WIP.\n\n---\n\n## TRAINING DETAIL\nThese are not so critical, but I'll summarize what I did.\n- initial lr: 0.01 (reduce on plateau with patience=20)\n- batch size: 512 (TPU runs way faster if batch size is bigger)\n- num_epochs: around 100~150\n- backbone: efficientnet b3 (b4~6 also works)\n- weights: noisy-student (but almost same as imagenet)\n- augment: gridmask + cutmix\n- loss: focal loss with gamma=2.0"},{"metadata":{},"cell_type":"markdown","source":"*The next hedden cell is the change to make the code run with tf2.1.0. Fix minor bug in saving model weight. (It will be fixed in 2.2.0)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%%writefile /opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py\n\n# Copyright 2018 The TensorFlow Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n# pylint: disable=protected-access\n\"\"\"Functions for saving and loading a Keras Model from HDF5 format.\n\"\"\"\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport json\nimport os\n\nimport numpy as np\nfrom six.moves import zip  # pylint: disable=redefined-builtin\n\nfrom tensorflow.python.framework import ops\nfrom tensorflow.python.keras import backend as K\nfrom tensorflow.python.keras import optimizers\nfrom tensorflow.python.keras.saving import model_config as model_config_lib\nfrom tensorflow.python.keras.saving import saving_utils\nfrom tensorflow.python.keras.utils import conv_utils\nfrom tensorflow.python.keras.utils.io_utils import ask_to_proceed_with_overwrite\nfrom tensorflow.python.ops import variables as variables_module\nfrom tensorflow.python.platform import tf_logging as logging\nfrom tensorflow.python.util import serialization\n\n# pylint: disable=g-import-not-at-top\ntry:\n  import h5py\n  HDF5_OBJECT_HEADER_LIMIT = 64512\nexcept ImportError:\n  h5py = None\n# pylint: enable=g-import-not-at-top\n\n\ndef save_model_to_hdf5(model, filepath, overwrite=True, include_optimizer=True):\n  \"\"\"Saves a model to a HDF5 file.\n  The saved model contains:\n      - the model's configuration (topology)\n      - the model's weights\n      - the model's optimizer's state (if any)\n  Thus the saved model can be reinstantiated in\n  the exact same state, without any of the code\n  used for model definition or training.\n  Arguments:\n      model: Keras model instance to be saved.\n      filepath: One of the following:\n          - String, path where to save the model\n          - `h5py.File` object where to save the model\n      overwrite: Whether we should overwrite any existing\n          model at the target location, or instead\n          ask the user with a manual prompt.\n      include_optimizer: If True, save optimizer's state together.\n  Raises:\n      ImportError: if h5py is not available.\n  \"\"\"\n\n  if h5py is None:\n    raise ImportError('`save_model` requires h5py.')\n\n  # TODO(psv) Add warning when we save models that contain non-serializable\n  # entities like metrics added using `add_metric` and losses added using\n  # `add_loss.`\n  if len(model.weights) != len(model._undeduplicated_weights):\n    logging.warning('Found duplicated `Variable`s in Model\\'s `weights`. '\n                    'This is usually caused by `Variable`s being shared by '\n                    'Layers in the Model. These `Variable`s will be treated '\n                    'as separate `Variable`s when the Model is restored. To '\n                    'avoid this, please save with `save_format=\"tf\"`.')\n\n  if not isinstance(filepath, h5py.File):\n    # If file exists and should not be overwritten.\n    if not overwrite and os.path.isfile(filepath):\n      proceed = ask_to_proceed_with_overwrite(filepath)\n      if not proceed:\n        return\n\n    f = h5py.File(filepath, mode='w')\n    opened_new_file = True\n  else:\n    f = filepath\n    opened_new_file = False\n\n  try:\n    model_metadata = saving_utils.model_metadata(model, include_optimizer)\n    for k, v in model_metadata.items():\n      if isinstance(v, (dict, list, tuple)):\n        f.attrs[k] = json.dumps(\n            v, default=serialization.get_json_type).encode('utf8')\n      else:\n        f.attrs[k] = v\n\n    model_weights_group = f.create_group('model_weights')\n    model_layers = model.layers\n    save_weights_to_hdf5_group(model_weights_group, model_layers)\n\n    # TODO(b/128683857): Add integration tests between tf.keras and external\n    # Keras, to avoid breaking TF.js users.\n    if (include_optimizer and model.optimizer and\n        not isinstance(model.optimizer, optimizers.TFOptimizer)):\n      save_optimizer_weights_to_hdf5_group(f, model.optimizer)\n\n    f.flush()\n  finally:\n    if opened_new_file:\n      f.close()\n\n\ndef load_model_from_hdf5(filepath, custom_objects=None, compile=True):  # pylint: disable=redefined-builtin\n  \"\"\"Loads a model saved via `save_model_to_hdf5`.\n  Arguments:\n      filepath: One of the following:\n          - String, path to the saved model\n          - `h5py.File` object from which to load the model\n      custom_objects: Optional dictionary mapping names\n          (strings) to custom classes or functions to be\n          considered during deserialization.\n      compile: Boolean, whether to compile the model\n          after loading.\n  Returns:\n      A Keras model instance. If an optimizer was found\n      as part of the saved model, the model is already\n      compiled. Otherwise, the model is uncompiled and\n      a warning will be displayed. When `compile` is set\n      to False, the compilation is omitted without any\n      warning.\n  Raises:\n      ImportError: if h5py is not available.\n      ValueError: In case of an invalid savefile.\n  \"\"\"\n  if h5py is None:\n    raise ImportError('`load_model` requires h5py.')\n\n  if not custom_objects:\n    custom_objects = {}\n\n  opened_new_file = not isinstance(filepath, h5py.File)\n  if opened_new_file:\n    f = h5py.File(filepath, mode='r')\n  else:\n    f = filepath\n\n  model = None\n  try:\n    # instantiate model\n    model_config = f.attrs.get('model_config')\n    if model_config is None:\n      raise ValueError('No model found in config file.')\n    model_config = json.loads(model_config.decode('utf-8'))\n    model = model_config_lib.model_from_config(model_config,\n                                               custom_objects=custom_objects)\n\n    # set weights\n    load_weights_from_hdf5_group(f['model_weights'], model.layers)\n\n    if compile:\n      # instantiate optimizer\n      training_config = f.attrs.get('training_config')\n      if training_config is None:\n        logging.warning('No training configuration found in the save file, so '\n                        'the model was *not* compiled. Compile it manually.')\n        return model\n      training_config = json.loads(training_config.decode('utf-8'))\n\n      # Compile model.\n      model.compile(**saving_utils.compile_args_from_training_config(\n          training_config, custom_objects))\n\n      # Set optimizer weights.\n      if 'optimizer_weights' in f:\n        # Build train function (to get weight updates).\n        # Models that aren't graph networks must wait until they are called\n        # with data to _make_train_function() and so can't load optimizer\n        # weights.\n        if model._is_graph_network:  # pylint: disable=protected-access\n          if not ops.executing_eagerly_outside_functions():\n            model._make_train_function()\n          optimizer_weight_values = load_optimizer_weights_from_hdf5_group(f)\n          try:\n            model.optimizer.set_weights(optimizer_weight_values)\n          except ValueError:\n            logging.warning('Error in loading the saved optimizer '\n                            'state. As a result, your model is '\n                            'starting with a freshly initialized '\n                            'optimizer.')\n        else:\n          logging.warning('Sequential models without an `input_shape` '\n                          'passed to the first layer cannot reload their '\n                          'optimizer state. As a result, your model is'\n                          'starting with a freshly initialized optimizer.')\n\n  finally:\n    if opened_new_file:\n      f.close()\n  return model\n\n\ndef preprocess_weights_for_loading(layer,\n                                   weights,\n                                   original_keras_version=None,\n                                   original_backend=None):\n  \"\"\"Preprocess layer weights between different Keras formats.\n  Converts layers weights from Keras 1 format to Keras 2 and also weights of\n  CuDNN layers in Keras 2.\n  Arguments:\n      layer: Layer instance.\n      weights: List of weights values (Numpy arrays).\n      original_keras_version: Keras version for the weights, as a string.\n      original_backend: Keras backend the weights were trained with,\n          as a string.\n  Returns:\n      A list of weights values (Numpy arrays).\n  \"\"\"\n  def convert_nested_bidirectional(weights):\n    \"\"\"Converts layers nested in `Bidirectional` wrapper.\n    This function uses `preprocess_weights_for_loading()` for converting\n    layers.\n    Arguments:\n        weights: List of weights values (Numpy arrays).\n    Returns:\n        A list of weights values (Numpy arrays).\n    \"\"\"\n    num_weights_per_layer = len(weights) // 2\n    forward_weights = preprocess_weights_for_loading(\n        layer.forward_layer, weights[:num_weights_per_layer],\n        original_keras_version, original_backend)\n    backward_weights = preprocess_weights_for_loading(\n        layer.backward_layer, weights[num_weights_per_layer:],\n        original_keras_version, original_backend)\n    return forward_weights + backward_weights\n\n  def convert_nested_time_distributed(weights):\n    \"\"\"Converts layers nested in `TimeDistributed` wrapper.\n    This function uses `preprocess_weights_for_loading()` for converting nested\n    layers.\n    Arguments:\n        weights: List of weights values (Numpy arrays).\n    Returns:\n        A list of weights values (Numpy arrays).\n    \"\"\"\n    return preprocess_weights_for_loading(\n        layer.layer, weights, original_keras_version, original_backend)\n\n  def convert_nested_model(weights):\n    \"\"\"Converts layers nested in `Model` or `Sequential`.\n    This function uses `preprocess_weights_for_loading()` for converting nested\n    layers.\n    Arguments:\n        weights: List of weights values (Numpy arrays).\n    Returns:\n        A list of weights values (Numpy arrays).\n    \"\"\"\n    trainable_weights = weights[:len(layer.trainable_weights)]\n    non_trainable_weights = weights[len(layer.trainable_weights):]\n\n    new_trainable_weights = []\n    new_non_trainable_weights = []\n\n    for sublayer in layer.layers:\n      num_trainable_weights = len(sublayer.trainable_weights)\n      num_non_trainable_weights = len(sublayer.non_trainable_weights)\n      if sublayer.weights:\n        preprocessed = preprocess_weights_for_loading(\n            layer=sublayer,\n            weights=(trainable_weights[:num_trainable_weights] +\n                     non_trainable_weights[:num_non_trainable_weights]),\n            original_keras_version=original_keras_version,\n            original_backend=original_backend)\n        new_trainable_weights.extend(preprocessed[:num_trainable_weights])\n        new_non_trainable_weights.extend(preprocessed[num_trainable_weights:])\n\n        trainable_weights = trainable_weights[num_trainable_weights:]\n        non_trainable_weights = non_trainable_weights[\n            num_non_trainable_weights:]\n\n    return new_trainable_weights + new_non_trainable_weights\n\n  # Convert layers nested in Bidirectional/Model/Sequential.\n  # Both transformation should be ran for both Keras 1->2 conversion\n  # and for conversion of CuDNN layers.\n  if layer.__class__.__name__ == 'Bidirectional':\n    weights = convert_nested_bidirectional(weights)\n  if layer.__class__.__name__ == 'TimeDistributed':\n    weights = convert_nested_time_distributed(weights)\n  elif layer.__class__.__name__ in ['Model', 'Sequential']:\n    weights = convert_nested_model(weights)\n\n  if original_keras_version == '1':\n    if layer.__class__.__name__ == 'TimeDistributed':\n      weights = preprocess_weights_for_loading(\n          layer.layer, weights, original_keras_version, original_backend)\n\n    if layer.__class__.__name__ == 'Conv1D':\n      shape = weights[0].shape\n      # Handle Keras 1.1 format\n      if shape[:2] != (layer.kernel_size[0], 1) or shape[3] != layer.filters:\n        # Legacy shape:\n        # (filters, input_dim, filter_length, 1)\n        assert shape[0] == layer.filters and shape[2:] == (layer.kernel_size[0],\n                                                           1)\n        weights[0] = np.transpose(weights[0], (2, 3, 1, 0))\n      weights[0] = weights[0][:, 0, :, :]\n\n    if layer.__class__.__name__ == 'Conv2D':\n      if layer.data_format == 'channels_first':\n        # old: (filters, stack_size, kernel_rows, kernel_cols)\n        # new: (kernel_rows, kernel_cols, stack_size, filters)\n        weights[0] = np.transpose(weights[0], (2, 3, 1, 0))\n\n    if layer.__class__.__name__ == 'Conv2DTranspose':\n      if layer.data_format == 'channels_last':\n        # old: (kernel_rows, kernel_cols, stack_size, filters)\n        # new: (kernel_rows, kernel_cols, filters, stack_size)\n        weights[0] = np.transpose(weights[0], (0, 1, 3, 2))\n      if layer.data_format == 'channels_first':\n        # old: (filters, stack_size, kernel_rows, kernel_cols)\n        # new: (kernel_rows, kernel_cols, filters, stack_size)\n        weights[0] = np.transpose(weights[0], (2, 3, 0, 1))\n\n    if layer.__class__.__name__ == 'Conv3D':\n      if layer.data_format == 'channels_first':\n        # old: (filters, stack_size, ...)\n        # new: (..., stack_size, filters)\n        weights[0] = np.transpose(weights[0], (2, 3, 4, 1, 0))\n\n    if layer.__class__.__name__ == 'GRU':\n      if len(weights) == 9:\n        kernel = np.concatenate([weights[0], weights[3], weights[6]], axis=-1)\n        recurrent_kernel = np.concatenate(\n            [weights[1], weights[4], weights[7]], axis=-1)\n        bias = np.concatenate([weights[2], weights[5], weights[8]], axis=-1)\n        weights = [kernel, recurrent_kernel, bias]\n\n    if layer.__class__.__name__ == 'LSTM':\n      if len(weights) == 12:\n        # old: i, c, f, o\n        # new: i, f, c, o\n        kernel = np.concatenate(\n            [weights[0], weights[6], weights[3], weights[9]], axis=-1)\n        recurrent_kernel = np.concatenate(\n            [weights[1], weights[7], weights[4], weights[10]], axis=-1)\n        bias = np.concatenate(\n            [weights[2], weights[8], weights[5], weights[11]], axis=-1)\n        weights = [kernel, recurrent_kernel, bias]\n\n    if layer.__class__.__name__ == 'ConvLSTM2D':\n      if len(weights) == 12:\n        kernel = np.concatenate(\n            [weights[0], weights[6], weights[3], weights[9]], axis=-1)\n        recurrent_kernel = np.concatenate(\n            [weights[1], weights[7], weights[4], weights[10]], axis=-1)\n        bias = np.concatenate(\n            [weights[2], weights[8], weights[5], weights[11]], axis=-1)\n        if layer.data_format == 'channels_first':\n          # old: (filters, stack_size, kernel_rows, kernel_cols)\n          # new: (kernel_rows, kernel_cols, stack_size, filters)\n          kernel = np.transpose(kernel, (2, 3, 1, 0))\n          recurrent_kernel = np.transpose(recurrent_kernel, (2, 3, 1, 0))\n        weights = [kernel, recurrent_kernel, bias]\n\n  conv_layers = ['Conv1D', 'Conv2D', 'Conv3D', 'Conv2DTranspose', 'ConvLSTM2D']\n  if layer.__class__.__name__ in conv_layers:\n    if original_backend == 'theano':\n      weights[0] = conv_utils.convert_kernel(weights[0])\n      if layer.__class__.__name__ == 'ConvLSTM2D':\n        weights[1] = conv_utils.convert_kernel(weights[1])\n    if K.int_shape(layer.weights[0]) != weights[0].shape:\n      weights[0] = np.transpose(weights[0], (3, 2, 0, 1))\n      if layer.__class__.__name__ == 'ConvLSTM2D':\n        weights[1] = np.transpose(weights[1], (3, 2, 0, 1))\n\n  # convert CuDNN layers\n  return _convert_rnn_weights(layer, weights)\n\n\ndef _convert_rnn_weights(layer, weights):\n  \"\"\"Converts weights for RNN layers between native and CuDNN format.\n  Input kernels for each gate are transposed and converted between Fortran\n  and C layout, recurrent kernels are transposed. For LSTM biases are summed/\n  split in half, for GRU biases are reshaped.\n  Weights can be converted in both directions between `LSTM` and`CuDNNSLTM`\n  and between `CuDNNGRU` and `GRU(reset_after=True)`. Default `GRU` is not\n  compatible with `CuDNNGRU`.\n  For missing biases in `LSTM`/`GRU` (`use_bias=False`) no conversion is made.\n  Arguments:\n      layer: Target layer instance.\n      weights: List of source weights values (input kernels, recurrent\n          kernels, [biases]) (Numpy arrays).\n  Returns:\n      A list of converted weights values (Numpy arrays).\n  Raises:\n      ValueError: for incompatible GRU layer/weights or incompatible biases\n  \"\"\"\n\n  def transform_kernels(kernels, func, n_gates):\n    \"\"\"Transforms kernel for each gate separately using given function.\n    Arguments:\n        kernels: Stacked array of kernels for individual gates.\n        func: Function applied to kernel of each gate.\n        n_gates: Number of gates (4 for LSTM, 3 for GRU).\n    Returns:\n        Stacked array of transformed kernels.\n    \"\"\"\n    return np.hstack([func(k) for k in np.hsplit(kernels, n_gates)])\n\n  def transpose_input(from_cudnn):\n    \"\"\"Makes a function that transforms input kernels from/to CuDNN format.\n    It keeps the shape, but changes between the layout (Fortran/C). Eg.:\n    ```\n    Keras                 CuDNN\n    [[0, 1, 2],  <--->  [[0, 2, 4],\n     [3, 4, 5]]          [1, 3, 5]]\n    ```\n    It can be passed to `transform_kernels()`.\n    Arguments:\n        from_cudnn: `True` if source weights are in CuDNN format, `False`\n            if they're in plain Keras format.\n    Returns:\n        Function that converts input kernel to the other format.\n    \"\"\"\n    order = 'F' if from_cudnn else 'C'\n\n    def transform(kernel):\n      return kernel.T.reshape(kernel.shape, order=order)\n\n    return transform\n\n  target_class = layer.__class__.__name__\n\n  # convert the weights between CuDNNLSTM and LSTM\n  if target_class in ['LSTM', 'CuDNNLSTM'] and len(weights) == 3:\n    # determine if we're loading a CuDNNLSTM layer\n    # from the number of bias weights:\n    # CuDNNLSTM has (units * 8) weights; while LSTM has (units * 4)\n    # if there's no bias weight in the file, skip this conversion\n    units = weights[1].shape[0]\n    bias_shape = weights[2].shape\n    n_gates = 4\n\n    if bias_shape == (2 * units * n_gates,):\n      source = 'CuDNNLSTM'\n    elif bias_shape == (units * n_gates,):\n      source = 'LSTM'\n    else:\n      raise ValueError('Invalid bias shape: ' + str(bias_shape))\n\n    def convert_lstm_weights(weights, from_cudnn=True):\n      \"\"\"Converts the weights between CuDNNLSTM and LSTM.\n      Arguments:\n        weights: Original weights.\n        from_cudnn: Indicates whether original weights are from CuDNN layer.\n      Returns:\n        Updated weights compatible with LSTM.\n      \"\"\"\n\n      # Transpose (and reshape) input and recurrent kernels\n      kernels = transform_kernels(weights[0], transpose_input(from_cudnn),\n                                  n_gates)\n      recurrent_kernels = transform_kernels(weights[1], lambda k: k.T, n_gates)\n      if from_cudnn:\n        # merge input and recurrent biases into a single set\n        biases = np.sum(np.split(weights[2], 2, axis=0), axis=0)\n      else:\n        # Split single set of biases evenly to two sets. The way of\n        # splitting doesn't matter as long as the two sets sum is kept.\n        biases = np.tile(0.5 * weights[2], 2)\n      return [kernels, recurrent_kernels, biases]\n\n    if source != target_class:\n      weights = convert_lstm_weights(weights, from_cudnn=source == 'CuDNNLSTM')\n\n  # convert the weights between CuDNNGRU and GRU(reset_after=True)\n  if target_class in ['GRU', 'CuDNNGRU'] and len(weights) == 3:\n    # We can determine the source of the weights from the shape of the bias.\n    # If there is no bias we skip the conversion since\n    # CuDNNGRU always has biases.\n\n    units = weights[1].shape[0]\n    bias_shape = weights[2].shape\n    n_gates = 3\n\n    def convert_gru_weights(weights, from_cudnn=True):\n      \"\"\"Converts the weights between CuDNNGRU and GRU.\n      Arguments:\n        weights: Original weights.\n        from_cudnn: Indicates whether original weights are from CuDNN layer.\n      Returns:\n        Updated weights compatible with GRU.\n      \"\"\"\n\n      kernels = transform_kernels(weights[0], transpose_input(from_cudnn),\n                                  n_gates)\n      recurrent_kernels = transform_kernels(weights[1], lambda k: k.T, n_gates)\n      biases = np.array(weights[2]).reshape((2, -1) if from_cudnn else -1)\n      return [kernels, recurrent_kernels, biases]\n\n    if bias_shape == (2 * units * n_gates,):\n      source = 'CuDNNGRU'\n    elif bias_shape == (2, units * n_gates):\n      source = 'GRU(reset_after=True)'\n    elif bias_shape == (units * n_gates,):\n      source = 'GRU(reset_after=False)'\n    else:\n      raise ValueError('Invalid bias shape: ' + str(bias_shape))\n\n    if target_class == 'CuDNNGRU':\n      target = 'CuDNNGRU'\n    elif layer.reset_after:\n      target = 'GRU(reset_after=True)'\n    else:\n      target = 'GRU(reset_after=False)'\n\n    # only convert between different types\n    if source != target:\n      types = (source, target)\n      if 'GRU(reset_after=False)' in types:\n        raise ValueError('%s is not compatible with %s' % types)\n      if source == 'CuDNNGRU':\n        weights = convert_gru_weights(weights, from_cudnn=True)\n      elif source == 'GRU(reset_after=True)':\n        weights = convert_gru_weights(weights, from_cudnn=False)\n\n  return weights\n\n\ndef save_optimizer_weights_to_hdf5_group(hdf5_group, optimizer):\n  \"\"\"Saves optimizer weights of a optimizer to a HDF5 group.\n  Arguments:\n      hdf5_group: HDF5 group.\n      optimizer: optimizer instance.\n  \"\"\"\n\n  symbolic_weights = getattr(optimizer, 'weights')\n  if symbolic_weights:\n    weights_group = hdf5_group.create_group('optimizer_weights')\n    weight_names = [str(w.name).encode('utf8') for w in symbolic_weights]\n    save_attributes_to_hdf5_group(weights_group, 'weight_names', weight_names)\n    weight_values = K.batch_get_value(symbolic_weights)\n    for name, val in zip(weight_names, weight_values):\n      param_dset = weights_group.create_dataset(\n          name, val.shape, dtype=val.dtype)\n      if not val.shape:\n        # scalar\n        param_dset[()] = val\n      else:\n        param_dset[:] = val\n\n\ndef load_optimizer_weights_from_hdf5_group(hdf5_group):\n  \"\"\"Load optimizer weights from a HDF5 group.\n  Arguments:\n      hdf5_group: A pointer to a HDF5 group.\n  Returns:\n      data: List of optimizer weight names.\n  \"\"\"\n  weights_group = hdf5_group['optimizer_weights']\n  optimizer_weight_names = load_attributes_from_hdf5_group(\n      weights_group, 'weight_names')\n  return [weights_group[weight_name] for weight_name in optimizer_weight_names]\n\n\ndef save_weights_to_hdf5_group(f, layers):\n  \"\"\"Saves the weights of a list of layers to a HDF5 group.\n  Arguments:\n      f: HDF5 group.\n      layers: List of layer instances.\n  \"\"\"\n  from tensorflow.python.keras import __version__ as keras_version  # pylint: disable=g-import-not-at-top\n\n  save_attributes_to_hdf5_group(\n      f, 'layer_names', [layer.name.encode('utf8') for layer in layers])\n  f.attrs['backend'] = K.backend().encode('utf8')\n  f.attrs['keras_version'] = str(keras_version).encode('utf8')\n\n  # Sort model layers by layer name to ensure that group names are strictly\n  # growing to avoid prefix issues.\n  for layer in sorted(layers, key=lambda x: x.name):\n    g = f.create_group(layer.name)\n    weights = _legacy_weights(layer)\n    weight_values = K.batch_get_value(weights)\n    weight_names = [w.name.encode('utf8') for w in weights]\n    save_attributes_to_hdf5_group(g, 'weight_names', weight_names)\n    for name, val in zip(weight_names, weight_values):\n      param_dset = g.create_dataset(name, val.shape, dtype=val.dtype)\n      if not val.shape:\n        # scalar\n        param_dset[()] = val\n      else:\n        param_dset[:] = val\n\n\ndef load_weights_from_hdf5_group(f, layers):\n  \"\"\"Implements topological (order-based) weight loading.\n  Arguments:\n      f: A pointer to a HDF5 group.\n      layers: a list of target layers.\n  Raises:\n      ValueError: in case of mismatch between provided layers\n          and weights file.\n  \"\"\"\n  if 'keras_version' in f.attrs:\n    original_keras_version = f.attrs['keras_version'].decode('utf8')\n  else:\n    original_keras_version = '1'\n  if 'backend' in f.attrs:\n    original_backend = f.attrs['backend'].decode('utf8')\n  else:\n    original_backend = None\n\n  filtered_layers = []\n  for layer in layers:\n    weights = _legacy_weights(layer)\n    if weights:\n      filtered_layers.append(layer)\n\n  layer_names = load_attributes_from_hdf5_group(f, 'layer_names')\n  filtered_layer_names = []\n  for name in layer_names:\n    g = f[name]\n    weight_names = load_attributes_from_hdf5_group(g, 'weight_names')\n    if weight_names:\n      filtered_layer_names.append(name)\n  layer_names = filtered_layer_names\n  if len(layer_names) != len(filtered_layers):\n    raise ValueError('You are trying to load a weight file '\n                     'containing ' + str(len(layer_names)) +\n                     ' layers into a model with ' + str(len(filtered_layers)) +\n                     ' layers.')\n\n  # We batch weight value assignments in a single backend call\n  # which provides a speedup in TensorFlow.\n  weight_value_tuples = []\n  for k, name in enumerate(layer_names):\n    g = f[name]\n    weight_names = load_attributes_from_hdf5_group(g, 'weight_names')\n    weight_values = [np.asarray(g[weight_name]) for weight_name in weight_names]\n    layer = filtered_layers[k]\n    symbolic_weights = _legacy_weights(layer)\n    weight_values = preprocess_weights_for_loading(\n        layer, weight_values, original_keras_version, original_backend)\n    if len(weight_values) != len(symbolic_weights):\n      raise ValueError('Layer #' + str(k) + ' (named \"' + layer.name +\n                       '\" in the current model) was found to '\n                       'correspond to layer ' + name + ' in the save file. '\n                       'However the new layer ' + layer.name + ' expects ' +\n                       str(len(symbolic_weights)) +\n                       ' weights, but the saved weights have ' +\n                       str(len(weight_values)) + ' elements.')\n    weight_value_tuples += zip(symbolic_weights, weight_values)\n  K.batch_set_value(weight_value_tuples)\n\n\ndef load_weights_from_hdf5_group_by_name(\n    f, layers, skip_mismatch=False):\n  \"\"\"Implements name-based weight loading.\n  (instead of topological weight loading).\n  Layers that have no matching name are skipped.\n  Arguments:\n      f: A pointer to a HDF5 group.\n      layers: a list of target layers.\n      skip_mismatch: Boolean, whether to skip loading of layers\n          where there is a mismatch in the number of weights,\n          or a mismatch in the shape of the weights.\n  Raises:\n      ValueError: in case of mismatch between provided layers\n          and weights file and skip_match=False.\n  \"\"\"\n  if 'keras_version' in f.attrs:\n    original_keras_version = f.attrs['keras_version'].decode('utf8')\n  else:\n    original_keras_version = '1'\n  if 'backend' in f.attrs:\n    original_backend = f.attrs['backend'].decode('utf8')\n  else:\n    original_backend = None\n\n  # New file format.\n  layer_names = load_attributes_from_hdf5_group(f, 'layer_names')\n\n  # Reverse index of layer name to list of layers with name.\n  index = {}\n  for layer in layers:\n    if layer.name:\n      index.setdefault(layer.name, []).append(layer)\n\n  # We batch weight value assignments in a single backend call\n  # which provides a speedup in TensorFlow.\n  weight_value_tuples = []\n  for k, name in enumerate(layer_names):\n    g = f[name]\n    weight_names = load_attributes_from_hdf5_group(g, 'weight_names')\n    weight_values = [np.asarray(g[weight_name]) for weight_name in weight_names]\n\n    for layer in index.get(name, []):\n      symbolic_weights = _legacy_weights(layer)\n      weight_values = preprocess_weights_for_loading(\n          layer, weight_values, original_keras_version, original_backend)\n      if len(weight_values) != len(symbolic_weights):\n        if skip_mismatch:\n          logging.warning('Skipping loading of weights for '\n                          'layer {}'.format(layer.name) + ' due to mismatch '\n                          'in number of weights ({} vs {}).'.format(\n                              len(symbolic_weights), len(weight_values)))\n          continue\n        raise ValueError('Layer #' + str(k) + ' (named \"' + layer.name +\n                         '\") expects ' + str(len(symbolic_weights)) +\n                         ' weight(s), but the saved weights' + ' have ' +\n                         str(len(weight_values)) + ' element(s).')\n      # Set values.\n      for i in range(len(weight_values)):\n        if K.int_shape(symbolic_weights[i]) != weight_values[i].shape:\n          if skip_mismatch:\n            logging.warning('Skipping loading of weights for '\n                            'layer {}'.format(layer.name) + ' due to '\n                            'mismatch in shape ({} vs {}).'.format(\n                                symbolic_weights[i].shape,\n                                weight_values[i].shape))\n            continue\n          raise ValueError('Layer #' + str(k) +' (named \"' + layer.name +\n                           '\"), weight ' + str(symbolic_weights[i]) +\n                           ' has shape {}'.format(K.int_shape(\n                               symbolic_weights[i])) +\n                           ', but the saved weight has shape ' +\n                           str(weight_values[i].shape) + '.')\n\n        else:\n          weight_value_tuples.append((symbolic_weights[i], weight_values[i]))\n  K.batch_set_value(weight_value_tuples)\n\n\ndef save_attributes_to_hdf5_group(group, name, data):\n  \"\"\"Saves attributes (data) of the specified name into the HDF5 group.\n  This method deals with an inherent problem of HDF5 file which is not\n  able to store data larger than HDF5_OBJECT_HEADER_LIMIT bytes.\n  Arguments:\n      group: A pointer to a HDF5 group.\n      name: A name of the attributes to save.\n      data: Attributes data to store.\n  Raises:\n    RuntimeError: If any single attribute is too large to be saved.\n  \"\"\"\n  # Check that no item in `data` is larger than `HDF5_OBJECT_HEADER_LIMIT`\n  # because in that case even chunking the array would not make the saving\n  # possible.\n  bad_attributes = [x for x in data if len(x) > HDF5_OBJECT_HEADER_LIMIT]\n\n  # Expecting this to never be true.\n  if bad_attributes:\n    raise RuntimeError('The following attributes cannot be saved to HDF5 '\n                       'file because they are larger than %d bytes: %s' %\n                       (HDF5_OBJECT_HEADER_LIMIT, ', '.join(bad_attributes)))\n\n  data_npy = np.asarray(data)\n\n  num_chunks = 1\n  chunked_data = np.array_split(data_npy, num_chunks)\n\n  # This will never loop forever thanks to the test above.\n  while any(x.nbytes > HDF5_OBJECT_HEADER_LIMIT for x in chunked_data):\n    num_chunks += 1\n    chunked_data = np.array_split(data_npy, num_chunks)\n\n  if num_chunks > 1:\n    for chunk_id, chunk_data in enumerate(chunked_data):\n      group.attrs['%s%d' % (name, chunk_id)] = chunk_data\n  else:\n    group.attrs[name] = data\n\n\ndef load_attributes_from_hdf5_group(group, name):\n  \"\"\"Loads attributes of the specified name from the HDF5 group.\n  This method deals with an inherent problem\n  of HDF5 file which is not able to store\n  data larger than HDF5_OBJECT_HEADER_LIMIT bytes.\n  Arguments:\n      group: A pointer to a HDF5 group.\n      name: A name of the attributes to load.\n  Returns:\n      data: Attributes data.\n  \"\"\"\n  if name in group.attrs:\n    data = [n.decode('utf8') for n in group.attrs[name]]\n  else:\n    data = []\n    chunk_id = 0\n    while '%s%d' % (name, chunk_id) in group.attrs:\n      data.extend(\n          [n.decode('utf8') for n in group.attrs['%s%d' % (name, chunk_id)]])\n      chunk_id += 1\n  return data\n\n\ndef _legacy_weights(layer):\n  \"\"\"DO NOT USE.\n  For legacy reason, the layer.weights was in the order of\n  [self.trainable_weights + self.non_trainable_weights], and this order was\n  used for preserving the weights in h5 format. The new order of layer.weights\n  are the same as layer.get_weights() which is more intuitive for user. To\n  keep supporting the existing saved h5 file, this method should be used to\n  save/load weights. In future version, we will delete this method and\n  introduce a breaking change for h5 and stay with the new order for weights.\n  Args:\n    layer: a `tf.keras.Model` or `tf.keras.layers.Layer` instance.\n  Returns:\n    A list of variables with the order of trainable_weights, followed by\n      non_trainable_weights.\n  \"\"\"\n  weights = layer.trainable_weights + layer.non_trainable_weights\n  if any([not isinstance(w, variables_module.Variable) for w in weights]):\n    raise NotImplementedError(\n        'Save or restore weights that is not an instance of `tf.Variable` is '\n        'not supported in h5, use `save_format=\\'tf\\'` instead. Got a model '\n        'or layer {} with weights {}'.format(layer.__class__.__name__, weights))\n  return weights","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# INSTALL AND IMPORT LIBRARIES"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!pip install ../input/kaggle-efficientnet-repo/efficientnet-1.0.0-py3-none-any.whl\n\nimport os\nimport numpy as np\nimport pandas as pd\nimport argparse\nimport math\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.losses import categorical_crossentropy\nfrom tensorflow.keras.metrics import categorical_accuracy, top_k_categorical_accuracy\nfrom kaggle_datasets import KaggleDatasets\nfrom tensorflow.keras.models import load_model\nfrom tensorflow.keras.callbacks import (ModelCheckpoint, LearningRateScheduler,\n                             EarlyStopping, ReduceLROnPlateau, TensorBoard, CSVLogger)\nfrom tensorflow.keras import regularizers\nfrom tensorflow.keras import layers as L\nimport efficientnet.tfkeras as efn\nfrom tensorflow.keras.layers import Dense, Lambda, Dropout, Layer, BatchNormalization, Flatten","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# UTILS"},{"metadata":{"trusted":true},"cell_type":"code","source":"def normalize(image):\n    # https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/main.py#L325-L326\n    # https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/efficientnet_builder.py#L31-L32\n    image -= tf.constant([0.485 * 255, 0.456 * 255, 0.406 * 255])  # RGB\n    image /= tf.constant([0.229 * 255, 0.224 * 255, 0.225 * 255])  # RGB\n    return image\n\ndef one_hot(image, label):\n    label['root'] = tf.one_hot(label['root'], 168)\n    label['vowel'] = tf.one_hot(label['vowel'], 11)\n    label['consonant'] = tf.one_hot(label['consonant'], 7)\n    label['unique'] = tf.one_hot(label['unique'], 1292)\n    label['root2'] = tf.one_hot(label['root2'], 168)\n    label['vowel2'] = tf.one_hot(label['vowel2'], 11)\n    label['consonant2'] = tf.one_hot(label['consonant2'], 7)\n    label['unique2'] = tf.one_hot(label['unique2'], 1292)\n    label['root3'] = tf.one_hot(label['root3'], 168)\n    label['vowel3'] = tf.one_hot(label['vowel3'], 11)\n    label['consonant3'] = tf.one_hot(label['consonant3'], 7)\n    label['unique3'] = tf.one_hot(label['unique3'], 1292)\n    return image, label\n\ndef prepare_metric_learning(image, label, mode='train'):\n    if model == 'train':\n        return (image, label['root'], label['vowel'], label['consonant'], label['unique']), label\n    else:\n        return (image, tf.zeros_like(label['root']), tf.zeros_like(label['vowel']), tf.zeros_like(label['consonant']), tf.zeros_like(label['unique'])), label\n\ndef get_callbacks(work_dir):\n    # model check point\n    checkpoint = ModelCheckpoint(work_dir + '/best.h5', \n                                 monitor = 'val_loss', \n                                 verbose = 0, save_best_only=True, \n                                 mode = 'min',\n                                 save_weights_only = True)\n    csv_logger = CSVLogger(work_dir + '/log.csv')\n    early = EarlyStopping(monitor='val_loss', mode='min', patience=20)\n    scheduler = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=10)    \n    return [checkpoint, csv_logger, early, scheduler]\n\ndef read_tfrecords(example, input_size):\n  features = {\n      'img': tf.io.FixedLenFeature([], tf.string),\n      'image_id': tf.io.FixedLenFeature([], tf.int64),\n      'grapheme_root': tf.io.FixedLenFeature([], tf.int64),\n      'vowel_diacritic': tf.io.FixedLenFeature([], tf.int64),\n      'consonant_diacritic': tf.io.FixedLenFeature([], tf.int64),\n      'unique_tuple': tf.io.FixedLenFeature([], tf.int64),\n  }\n  example = tf.io.parse_single_example(example, features)\n  img = tf.image.decode_image(example['img'])\n  img = tf.reshape(img, input_size + (1, ))\n  img = tf.cast(img, tf.float32)\n  # grayscale -> RGB\n  img = tf.repeat(img, 3, -1)\n\n  # image_id = tf.cast(example['image_id'], tf.int32)\n  grapheme_root = tf.cast(example['grapheme_root'], tf.int32)\n  vowel_diacritic = tf.cast(example['vowel_diacritic'], tf.int32)\n  consonant_diacritic = tf.cast(example['consonant_diacritic'], tf.int32)\n  unique_tuple = tf.cast(example['unique_tuple'], tf.int32)\n  #return img, unique_tuple\n  return img, {'root': grapheme_root, 'vowel': vowel_diacritic, 'consonant': consonant_diacritic, 'unique': unique_tuple, 'root2': grapheme_root, 'vowel2': vowel_diacritic, 'consonant2': consonant_diacritic, 'unique2': unique_tuple, 'root3': grapheme_root, 'vowel3': vowel_diacritic, 'consonant3': consonant_diacritic, 'unique3': unique_tuple}\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MODEL"},{"metadata":{"trusted":true},"cell_type":"code","source":"class Generalized_mean_pooling2D(tf.keras.layers.Layer):\n    def __init__(self, p=3, epsilon=1e-6, name='', **kwargs):\n      super(Generalized_mean_pooling2D, self).__init__(name, **kwargs)\n      self.init_p = p\n      self.epsilon = epsilon\n    \n    def build(self, input_shape):\n      if isinstance(input_shape, list) or len(input_shape) != 4:\n        raise ValueError('`GeM` pooling layer only allow 1 input with 4 dimensions(b, h, w, c)')\n      self.build_shape = input_shape\n      self.p = self.add_weight(\n              name='p',\n              shape=[1,],\n              initializer=tf.keras.initializers.Constant(value=self.init_p),\n              regularizer=None,\n              trainable=True,\n              dtype=tf.float32\n              )\n      self.built=True\n\n    def call(self, inputs):\n      input_shape = inputs.get_shape()\n      if isinstance(inputs, list) or len(input_shape) != 4:\n        raise ValueError('`GeM` pooling layer only allow 1 input with 4 dimensions(b, h, w, c)')\n      return (tf.reduce_mean(tf.abs(inputs**self.p), axis=[1,2], keepdims=False) + self.epsilon)**(1.0/self.p)\n\n\nclass CosFace(Layer):\n    def __init__(self, n_classes=10, s=30.0, m=0.35, regularizer=None, **kwargs):\n        super(CosFace, self).__init__(**kwargs)\n        self.n_classes = n_classes\n        self.s = s\n        self.m = m\n        self.regularizer = regularizers.get(regularizer)\n\n    def build(self, input_shape):\n        super(CosFace, self).build(input_shape[0])\n        self.W = self.add_weight(shape=(input_shape[0][-1], self.n_classes),\n                                initializer='glorot_uniform',\n                                trainable=True,\n                                regularizer=self.regularizer)\n\n    def call(self, inputs):\n        x, y = inputs\n        c = K.shape(x)[-1]\n        \n        # normalize weights\n        W = tf.nn.l2_normalize(self.W, axis=0)\n        # dot product\n        logits = x @ W\n        # add margin\n        target_logits = logits - self.m\n        logits = logits * (1 - y) + target_logits * y\n        # feature re-scale\n        logits *= self.s\n        out = tf.nn.softmax(logits)\n\n        return out\n\n    def compute_output_shape(self, input_shape):\n        return (None, self.n_classes)\n\n\ndef get_model(input_size, backbone='efficientnet-b0', weights='imagenet', tta=False):\n    print(f'Using backbone {backbone} and weights {weights}')\n    x_input = L.Input(shape=input_size, name='imgs', dtype='float32')\n    r_label = L.Input(shape=(168,))\n    v_label = L.Input(shape=(11,))\n    c_label = L.Input(shape=(7,))\n    u_label = L.Input(shape=(1292,))\n\n    if backbone.startswith('efficientnet'):\n        model_fn = getattr(efn, f'EfficientNetB{backbone[-1]}')\n\n    x = model_fn(input_shape=input_size, weights=weights, include_top=False)(x_input)\n    x = Generalized_mean_pooling2D()(x)\n\n    # feature vector\n    weight_decay = 1e-4\n    x = BatchNormalization()(x)\n    x = Dropout(0.2)(x)\n    x = Flatten()(x)\n\n    # model architecture is inspired humpback comp's solution.\n    # I prepared cosface head and dense head for each outputs.\n    \n    # root\n    x1 = Dense(512, kernel_initializer='he_normal', kernel_regularizer=regularizers.l2(weight_decay))(x)\n    x1 = BatchNormalization()(x1)\n    x1 = tf.nn.l2_normalize(x1, axis=1)\n    root = CosFace(168, regularizer=regularizers.l2(weight_decay), name='root')([x1, r_label])\n    x1 = Dense(168, use_bias=False)(x1)\n    root2 = Lambda(lambda x: K.softmax(x), name='root2')(x1)\n\n    # vowel\n    x2 = Dense(512, kernel_initializer='he_normal', kernel_regularizer=regularizers.l2(weight_decay))(x)\n    x2 = BatchNormalization()(x2)\n    x2 = tf.nn.l2_normalize(x2, axis=1)\n    vowel = CosFace(11, regularizer=regularizers.l2(weight_decay), name='vowel')([x2, v_label])\n    x2 = Dense(11, use_bias=False)(x2)\n    vowel2 = Lambda(lambda x: K.softmax(x), name='vowel2')(x2)\n\n    # consonant\n    x3 = Dense(512, kernel_initializer='he_normal', kernel_regularizer=regularizers.l2(weight_decay))(x)\n    x3 = BatchNormalization()(x3)\n    x3 = tf.nn.l2_normalize(x3, axis=1)\n    consonant = CosFace(7, regularizer=regularizers.l2(weight_decay), name='consonant')([x3, c_label])\n    x3 = Dense(7, use_bias=False)(x3)\n    consonant2 = Lambda(lambda x: K.softmax(x), name='consonant2')(x3)\n\n    # unique\n    x4 = Dense(1024, kernel_initializer='he_normal', kernel_regularizer=regularizers.l2(weight_decay))(x)\n    x4 = BatchNormalization()(x4)\n    x4 = tf.nn.l2_normalize(x4, axis=1)\n    unique = CosFace(1292, regularizer=regularizers.l2(weight_decay), name='unique')([x4, u_label])\n    x4 = Dense(1292, use_bias=False)(x4)\n    unique2 = Lambda(lambda x: K.softmax(x), name='unique2')(x4)\n\n    # I thought it may useful to know what is other heads' prediction, so I concat all logits and predict each output again.\n    # But it turned out this head ddin't perform better than other branch. You may remove it.\n\n    # concat all logits\n    xx = tf.concat([x1, x2, x3, x4], axis=1)\n    xx = tf.keras.activations.relu(xx)\n\n    # Wow, I just realized these 4 dense layers exist when I refactored the codes....lol\n    # These are not supposed to be here, but my final model weight use them, so I don't remove it...\n    xx1 = Dense(168)(xx)\n    xx2 = Dense(11)(xx)\n    xx3 = Dense(7)(xx)\n    xx4 = Dense(1292)(xx)\n\n    xx1 = Dense(168, use_bias=False)(xx1)\n    root3 = Lambda(lambda x: K.softmax(x), name='root3')(xx1)\n\n    xx2 = Dense(11, use_bias=False)(xx2)\n    vowel3 = Lambda(lambda x: K.softmax(x), name='vowel3')(xx2)\n\n    xx3 = Dense(7, use_bias=False)(xx3)\n    consonant3 = Lambda(lambda x: K.softmax(x), name='consonant3')(xx3)\n\n    xx4 = Dense(1292, use_bias=False)(xx4)\n    unique3 = Lambda(lambda x: K.softmax(x), name='unique3')(xx4)\n\n    # model\n    model = tf.keras.Model(\n        inputs = [x_input,r_label,v_label,c_label,u_label],\n        outputs = [root, vowel, consonant, unique, root2, vowel2, consonant2, unique2, root3, vowel3, consonant3, unique3]\n    )\n\n    return model\n\n# Additionaly, I trained 3 models for unseen labels for each root, vowel and consonant prediction.\n# In this kernel, you can train only above multi-head model. To train below 3 models, you need to modify the codes a little bit.\n\ndef get_r_model(input_size, backbone='efficientnet-b0', weights='imagenet', tta=False):\n    print(f'Using backbone {backbone} and weights {weights}')\n    x_input = L.Input(shape=input_size, name='imgs', dtype='float32')\n    r_label = L.Input(shape=(168,))\n\n    if backbone.startswith('efficientnet'):\n        model_fn = getattr(efn, f'EfficientNetB{backbone[-1]}')\n\n    x = model_fn(input_shape=input_size, weights=weights, include_top=False)(x_input)\n    x = Generalized_mean_pooling2D()(x)\n\n    # feature vector\n    weight_decay = 1e-4\n    x = BatchNormalization()(x)\n    x = Dropout(0.2)(x)\n    x = Flatten()(x)\n\n    # root\n    x1 = Dense(512, kernel_initializer='he_normal', kernel_regularizer=regularizers.l2(weight_decay))(x)\n    x1 = BatchNormalization()(x1)\n    x1 = tf.nn.l2_normalize(x1, axis=1)\n    root = CosFace(168, regularizer=regularizers.l2(weight_decay), name='root')([x1, r_label])\n    x1 = Dense(168, use_bias=False)(x1)\n    root2 = Lambda(lambda x: K.softmax(x), name='root2')(x1)\n\n    # model\n    model = tf.keras.Model(\n        inputs = [x_input,r_label],\n        outputs = [root, root2]\n    )\n\n    return model\n\ndef get_v_model(input_size, backbone='efficientnet-b0', weights='imagenet', tta=False):\n    print(f'Using backbone {backbone} and weights {weights}')\n    x_input = L.Input(shape=input_size, name='imgs', dtype='float32')\n    v_label = L.Input(shape=(11,))\n\n    if backbone.startswith('efficientnet'):\n        model_fn = getattr(efn, f'EfficientNetB{backbone[-1]}')\n\n    x = model_fn(input_shape=input_size, weights=weights, include_top=False)(x_input)\n    x = Generalized_mean_pooling2D()(x)\n\n    # feature vector\n    weight_decay = 1e-4\n    x = BatchNormalization()(x)\n    x = Dropout(0.2)(x)\n    x = Flatten()(x)\n\n    # vowel\n    x2 = Dense(512, kernel_initializer='he_normal', kernel_regularizer=regularizers.l2(weight_decay))(x)\n    x2 = BatchNormalization()(x2)\n    x2 = tf.nn.l2_normalize(x2, axis=1)\n    vowel = CosFace(11, regularizer=regularizers.l2(weight_decay), name='vowel')([x2, v_label])\n    x2 = Dense(11, use_bias=False)(x2)\n    vowel2 = Lambda(lambda x: K.softmax(x), name='vowel2')(x2)\n\n    # model\n    model = tf.keras.Model(\n        inputs = [x_input,v_label],\n        outputs = [vowel, vowel2]\n    )\n\n    return model\n\ndef get_c_model(input_size, backbone='efficientnet-b0', weights='imagenet', tta=False):\n    print(f'Using backbone {backbone} and weights {weights}')\n    x_input = L.Input(shape=input_size, name='imgs', dtype='float32')\n    c_label = L.Input(shape=(7,))\n\n    if backbone.startswith('efficientnet'):\n        model_fn = getattr(efn, f'EfficientNetB{backbone[-1]}')\n\n    x = model_fn(input_shape=input_size, weights=weights, include_top=False)(x_input)\n    x = Generalized_mean_pooling2D()(x)\n\n    # feature vector\n    weight_decay = 1e-4\n    x = BatchNormalization()(x)\n    x = Dropout(0.2)(x)\n    x = Flatten()(x)\n\n    # consonant\n    x3 = Dense(512, kernel_initializer='he_normal', kernel_regularizer=regularizers.l2(weight_decay))(x)\n    x3 = BatchNormalization()(x3)\n    x3 = tf.nn.l2_normalize(x3, axis=1)\n    consonant = CosFace(7, regularizer=regularizers.l2(weight_decay), name='consonant')([x3, c_label])\n    x3 = Dense(7, use_bias=False)(x3)\n    consonant2 = Lambda(lambda x: K.softmax(x), name='consonant2')(x3)\n\n    # model\n    model = tf.keras.Model(\n        inputs = [x_input,c_label],\n        outputs = [consonant, consonant2]\n    )\n\n    return model\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# LOSS"},{"metadata":{"trusted":true},"cell_type":"code","source":"def categorical_focal_loss(num_classes, gamma=2., alpha=.25, smooth_alpha=0.05):\n    \"\"\"\n    Softmax version of focal loss.\n           m\n      FL = ∑  -alpha * (1 - p_o,c)^gamma * y_o,c * log(p_o,c)\n          c=1\n      where m = number of classes, c = class and o = observation\n    Parameters:\n      alpha -- the same as weighing factor in balanced cross entropy\n      gamma -- focusing parameter for modulating factor (1-p)\n    Default value:\n      gamma -- 2.0 as mentioned in the paper\n      alpha -- 0.25 as mentioned in the paper\n    References:\n        Official paper: https://arxiv.org/pdf/1708.02002.pdf\n        https://www.tensorflow.org/api_docs/python/tf/keras/backend/categorical_crossentropy\n    Usage:\n     model.compile(loss=[categorical_focal_loss(alpha=.25, gamma=2)], metrics=[\"accuracy\"], optimizer=adam)\n    \"\"\"\n    def categorical_focal_loss_fixed(y_true, y_pred):\n        \"\"\"\n        :param y_true: A tensor of the same shape as `y_pred`\n        :param y_pred: A tensor resulting from a softmax\n        :return: Output tensor.\n        \"\"\"\n        if smooth_alpha > 0:\n            y_true = y_true * (1 - smooth_alpha) + smooth_alpha / num_classes\n\n        # Scale predictions so that the class probas of each sample sum to 1\n        y_pred /= K.sum(y_pred, axis=-1, keepdims=True)\n\n        # Clip the prediction value to prevent NaN's and Inf's\n        epsilon = K.epsilon()\n        y_pred = K.clip(y_pred, epsilon, 1. - epsilon)\n\n        # Calculate Cross Entropy\n        cross_entropy = -y_true * K.log(y_pred)\n\n        # Calculate Focal Loss\n        loss = alpha * K.pow(1 - y_pred, gamma) * cross_entropy\n\n        # Sum the losses in mini_batch\n        return K.sum(loss, axis=1)\n\n    return categorical_focal_loss_fixed  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# AUGMENTATIONS"},{"metadata":{"trusted":true},"cell_type":"code","source":"def cutmix(images, labels, batch_size, image_size):\n    \n    DIM = image_size[0]\n    \n    # DO CUTMIX WITH PROBABILITY DEFINED ABOVE\n    # This is a tensor containing 0 or 1 -- 0: no cutmix.\n    # shape = [batch_size]\n    #do_cutmix = tf.cast(tf.random.uniform([batch_size], 0, 1) <= 1.0, tf.int32)\n    \n    # Choose random images in the batch for cutmix\n    # shape = [batch_size]\n    new_image_indices = tf.cast(tf.random.uniform([batch_size], 0, batch_size), tf.int32)\n    \n    # Choose random location in the original image to put the new images\n    # shape = [batch_size]\n    new_x = tf.cast(tf.random.uniform([batch_size], 0, DIM), tf.int32)\n    new_y = tf.cast(tf.random.uniform([batch_size], 0, DIM), tf.int32)\n    \n    # Random width for new images, shape = [batch_size]\n    b = tf.random.uniform([batch_size], 0, 1) # this is beta dist with alpha=1.0\n    new_width = tf.cast(DIM * tf.math.sqrt(1-b), tf.int32) #* do_cutmix\n    \n    # shape = [batch_size]\n    new_y0 = tf.math.maximum(0, new_y - new_width // 2)\n    new_y1 = tf.math.minimum(DIM, new_y + new_width // 2)\n    new_x0 = tf.math.maximum(0, new_x - new_width // 2)\n    new_x1 = tf.math.minimum(DIM, new_x + new_width // 2)\n    \n    # shape = [batch_size, DIM]\n    target = tf.broadcast_to(tf.range(DIM), shape=(batch_size, DIM))\n    \n    # shape = [batch_size, DIM]\n    mask_y = tf.math.logical_and(new_y0[:, tf.newaxis] <= target, target <= new_y1[:, tf.newaxis])\n    \n    # shape = [batch_size, DIM]\n    mask_x = tf.math.logical_and(new_x0[:, tf.newaxis] <= target, target <= new_x1[:, tf.newaxis])    \n    \n    # shape = [batch_size, DIM, DIM]\n    mask = tf.cast(tf.math.logical_and(mask_y[:, :, tf.newaxis], mask_x[:, tf.newaxis, :]), tf.float32)\n\n    # All components are of shape [batch_size, DIM, DIM, 3]\n    new_images =  images * tf.broadcast_to(1 - mask[:, :, :, tf.newaxis], [batch_size, DIM, DIM, 3]) + \\\n                    tf.gather(images, new_image_indices) * tf.broadcast_to(mask[:, :, :, tf.newaxis], [batch_size, DIM, DIM, 3])\n\n    a = tf.cast(new_width ** 2 / DIM ** 2, tf.float32)    \n        \n    #new_labels =  (1-a)[:, tf.newaxis] * labels + a[:, tf.newaxis] * tf.gather(labels, new_image_indices)        \n    for c in ['root', 'vowel', 'consonant', 'unique', 'root2', 'vowel2', 'consonant2', 'unique2', 'root3', 'vowel3', 'consonant3', 'unique3']:\n        y1, y2 = labels[c], tf.gather(labels[c], new_image_indices)\n        labels[c] = y1 * (1-a)[:, tf.newaxis] + y2 * a[:, tf.newaxis]\n\n    return new_images, labels\n\ndef transform(image, inv_mat, image_shape):\n\n    h, w, c = image_shape\n    cx, cy = w//2, h//2\n\n    new_xs = tf.repeat( tf.range(-cx, cx, 1), h)\n    new_ys = tf.tile( tf.range(-cy, cy, 1), [w])\n    new_zs = tf.ones([h*w], dtype=tf.int32)\n\n    old_coords = tf.matmul(inv_mat, tf.cast(tf.stack([new_xs, new_ys, new_zs]), tf.float32))\n    old_coords_x, old_coords_y = tf.round(old_coords[0, :] + w//2), tf.round(old_coords[1, :] + h//2)\n\n    clip_mask_x = tf.logical_or(old_coords_x<0, old_coords_x>w-1)\n    clip_mask_y = tf.logical_or(old_coords_y<0, old_coords_y>h-1)\n    clip_mask = tf.logical_or(clip_mask_x, clip_mask_y)\n\n    old_coords_x = tf.boolean_mask(old_coords_x, tf.logical_not(clip_mask))\n    old_coords_y = tf.boolean_mask(old_coords_y, tf.logical_not(clip_mask))\n    new_coords_x = tf.boolean_mask(new_xs+cx, tf.logical_not(clip_mask))\n    new_coords_y = tf.boolean_mask(new_ys+cy, tf.logical_not(clip_mask))\n\n    old_coords = tf.cast(tf.stack([old_coords_y, old_coords_x]), tf.int32)\n    new_coords = tf.cast(tf.stack([new_coords_y, new_coords_x]), tf.int64)\n    rotated_image_values = tf.gather_nd(image, tf.transpose(old_coords))\n    rotated_image_channel = list()\n    for i in range(c):\n        vals = rotated_image_values[:,i]\n        sparse_channel = tf.SparseTensor(tf.transpose(new_coords), vals, [h, w])\n        rotated_image_channel.append(tf.sparse.to_dense(sparse_channel, default_value=0, validate_indices=False))\n\n    return tf.transpose(tf.stack(rotated_image_channel), [1,2,0])\n\ndef random_rotate(image, angle, image_shape):\n\n    def get_rotation_mat_inv(angle):\n          #transform to radian\n        angle = math.pi * angle / 180\n\n        cos_val = tf.math.cos(angle)\n        sin_val = tf.math.sin(angle)\n        one = tf.constant([1], tf.float32)\n        zero = tf.constant([0], tf.float32)\n\n        rot_mat_inv = tf.concat([cos_val, sin_val, zero,\n                                     -sin_val, cos_val, zero,\n                                     zero, zero, one], axis=0)\n        rot_mat_inv = tf.reshape(rot_mat_inv, [3,3])\n\n        return rot_mat_inv\n    angle = float(angle) * tf.random.normal([1],dtype='float32')\n    rot_mat_inv = get_rotation_mat_inv(angle)\n    return transform(image, rot_mat_inv, image_shape)\n\n\ndef GridMask(image_height, image_width, d1, d2, rotate_angle=1, ratio=0.5):\n\n    h, w = image_height, image_width\n    hh = int(np.ceil(np.sqrt(h*h+w*w)))\n    hh = hh+1 if hh%2==1 else hh\n    d = tf.random.uniform(shape=[], minval=d1, maxval=d2, dtype=tf.int32)\n    l = tf.cast(tf.cast(d,tf.float32)*ratio+0.5, tf.int32)\n\n    st_h = tf.random.uniform(shape=[], minval=0, maxval=d, dtype=tf.int32)\n    st_w = tf.random.uniform(shape=[], minval=0, maxval=d, dtype=tf.int32)\n\n    y_ranges = tf.range(-1 * d + st_h, -1 * d + st_h + l)\n    x_ranges = tf.range(-1 * d + st_w, -1 * d + st_w + l)\n\n    for i in range(0, hh//d+1):\n        s1 = i * d + st_h\n        s2 = i * d + st_w\n        y_ranges = tf.concat([y_ranges, tf.range(s1,s1+l)], axis=0)\n        x_ranges = tf.concat([x_ranges, tf.range(s2,s2+l)], axis=0)\n\n    x_clip_mask = tf.logical_or(x_ranges <0 , x_ranges > hh-1)\n    y_clip_mask = tf.logical_or(y_ranges <0 , y_ranges > hh-1)\n    clip_mask = tf.logical_or(x_clip_mask, y_clip_mask)\n\n    x_ranges = tf.boolean_mask(x_ranges, tf.logical_not(clip_mask))\n    y_ranges = tf.boolean_mask(y_ranges, tf.logical_not(clip_mask))\n\n    hh_ranges = tf.tile(tf.range(0,hh), [tf.cast(tf.reduce_sum(tf.ones_like(x_ranges)), tf.int32)])\n    x_ranges = tf.repeat(x_ranges, hh)\n    y_ranges = tf.repeat(y_ranges, hh)\n\n    y_hh_indices = tf.transpose(tf.stack([y_ranges, hh_ranges]))\n    x_hh_indices = tf.transpose(tf.stack([hh_ranges, x_ranges]))\n\n    y_mask_sparse = tf.SparseTensor(tf.cast(y_hh_indices, tf.int64),  tf.zeros_like(y_ranges), [hh, hh])\n    y_mask = tf.sparse.to_dense(y_mask_sparse, 1, False)\n\n    x_mask_sparse = tf.SparseTensor(tf.cast(x_hh_indices, tf.int64), tf.zeros_like(x_ranges), [hh, hh])\n    x_mask = tf.sparse.to_dense(x_mask_sparse, 1, False)\n\n    mask = tf.expand_dims( tf.clip_by_value(x_mask + y_mask, 0, 1), axis=-1)\n\n    mask = random_rotate(mask, rotate_angle, [hh, hh, 1])\n    mask = tf.image.crop_to_bounding_box(mask, (hh-h)//2, (hh-w)//2, image_height, image_width)\n\n    return mask\n\ndef apply_grid_mask(image, image_shape):\n    AugParams = {\n        'd1' : 10,\n        'd2': 100,\n        'rotate' : 15,\n        'ratio' : 0.5\n    }\n    mask = GridMask(image_shape[0],\n                    image_shape[1],\n                    AugParams['d1'],\n                    AugParams['d2'],\n                    AugParams['rotate'],\n                    #AugParams['ratio'],\n                    tf.random.uniform(shape=[], minval=0.3, maxval=0.6)\n                    )\n    \n    if image_shape[-1] == 3:\n        mask = tf.concat([mask, mask, mask], axis=-1)\n\n    return image * tf.cast(mask, tf.float32)\n\ndef augmentation(image, label, input_size):\n    image = apply_grid_mask(image, (*input_size,3))\n    return image, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MAIN\n\nyou can train the models by changing epochs and turning on TPU accelerator."},{"metadata":{"trusted":true},"cell_type":"code","source":"EXP_NAME = 'efnet-b6_cosface_gridmask_cutmix'\n\nparser = argparse.ArgumentParser()\nparser.add_argument('--seed', type=int, default=123)\nparser.add_argument('--lr', type=float, default=1e-3)\nparser.add_argument('--input_size', type=str, default='224,224')\nparser.add_argument('--batch_size', type=int, default=512)\nparser.add_argument('--epochs', type=int, default=0)\nparser.add_argument('--backbone', type=str, default='efficientnet-b3')\nparser.add_argument('--weights', type=str, default='noisy-student')\nparser.add_argument('--root_dir', type=str, default='./')\nparser.add_argument('--resume_from', type=str, default=None)\nargs, _ = parser.parse_known_args([\n    '--lr' '0.0001',\n    '--batch_size', '32',\n    '--aug', 'gridmask',\n    '--mix', 'cutmix',\n])\n\nwork_dir = args.root_dir + EXP_NAME\nos.makedirs(work_dir, exist_ok=True)\n\nargs.input_size = tuple(int(x) for x in args.input_size.split(','))\nnp.random.seed(args.seed)\ntf.random.set_seed(args.seed)\n\nif args.epochs > 0:\n    try:\n      tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection\n      print('Running on TPU ', tpu.cluster_spec().as_dict()['worker'])\n    except ValueError:\n      raise BaseException('ERROR: Not connected to a TPU runtime; please see the previous cell in this notebook for instructions!')\n\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\n\n\n    with strategy.scope():\n        model = get_model(input_size=args.input_size + (3, ), backbone=args.backbone,\n        weights=args.weights)\n\n        model.compile(optimizer = Adam(lr = args.lr),\n                        loss = {'root': categorical_focal_loss(168),\n                                'vowel': categorical_focal_loss(11),\n                                'consonant': categorical_focal_loss(7),\n                                'unique': categorical_focal_loss(1292),\n                                'root2': categorical_focal_loss(168),\n                                'vowel2': categorical_focal_loss(11),\n                                'consonant2': categorical_focal_loss(7),\n                                'unique2': categorical_focal_loss(1292),\n                                'root3': categorical_focal_loss(168),\n                                'vowel3': categorical_focal_loss(11),\n                                'consonant3': categorical_focal_loss(7),\n                                'unique3': categorical_focal_loss(1292)},\n                        loss_weights = {'root': 0.25,        \n                                        'vowel': 0.25,\n                                        'consonant': 0.25,\n                                        'unique': 0.25,\n                                        'root2': 0.25,        \n                                        'vowel2': 0.25,\n                                        'consonant2': 0.25,\n                                        'unique2': 0.25,\n                                        'root3': 0.25,        \n                                        'vowel3': 0.25,\n                                        'consonant3': 0.25,\n                                        'unique3': 0.25},\n                        metrics = { 'root': ['categorical_accuracy', tf.keras.metrics.Recall()],\n                                    'vowel': ['categorical_accuracy', tf.keras.metrics.Recall()],\n                                    'consonant': ['categorical_accuracy', tf.keras.metrics.Recall()],\n                                    'unique': ['categorical_accuracy', tf.keras.metrics.Recall()],\n                                    'root2': ['categorical_accuracy', tf.keras.metrics.Recall()],\n                                    'vowel2': ['categorical_accuracy', tf.keras.metrics.Recall()],\n                                    'consonant2': ['categorical_accuracy', tf.keras.metrics.Recall()],\n                                    'unique2': ['categorical_accuracy', tf.keras.metrics.Recall()],\n                                    'root3': ['categorical_accuracy', tf.keras.metrics.Recall()],\n                                    'vowel3': ['categorical_accuracy', tf.keras.metrics.Recall()],\n                                    'consonant3': ['categorical_accuracy', tf.keras.metrics.Recall()],\n                                    'unique3': ['categorical_accuracy', tf.keras.metrics.Recall()]\n                                                }\n                        )\n\n    from kaggle_datasets import KaggleDatasets\n    ds_path = KaggleDatasets().get_gcs_path('tfrecords-224')\n    train_fns = tf.io.gfile.glob(os.path.join(ds_path, 'train*.tfrec'))\n    val_fns = tf.io.gfile.glob(os.path.join(ds_path, 'val*.tfrec'))\n\n    # get train dataset with augmentations\n    AUTO = tf.data.experimental.AUTOTUNE\n    train_ds = tf.data.TFRecordDataset(train_fns)\n    train_ds = train_ds.map(lambda e: read_tfrecords(e, args.input_size))\n    train_ds = train_ds.map(lambda a, b: augmentation(a, b, args.input_size), num_parallel_calls=AUTO)\n    train_ds = train_ds.repeat().batch(args.batch_size)\n    train_ds = train_ds.map(one_hot)\n    train_ds = train_ds.map(lambda a, b: cutmix(a, b, args.batch_size, args.input_size))\n    train_ds = train_ds.map(lambda a, b: prepare_metric_learning(a, b, 'train'))\n\n    # get valid dataset\n    val_ds = tf.data.TFRecordDataset(val_fns)\n    val_ds = val_ds.map(lambda e: read_tfrecords(e, args.input_size))\n    val_ds = val_ds.batch(args.batch_size)\n    val_ds = val_ds.map(one_hot)\n    val_ds = val_ds.map(lambda a, b: prepare_metric_learning(a, b, 'valid'))\n\n    # train\n    print(EXP_NAME)\n    if args.resume_from:\n        checkpoint_path = args.resume_from\n        print('load model from ', checkpoint_path)\n        model.load_weights(checkpoint_path)\n\n    num_train_samples = sum(int(fn.split('_')[2]) for fn in train_fns)\n    steps_per_epoch = num_train_samples // args.batch_size\n    print(f'Training on {num_train_samples} samples. Each epochs requires {steps_per_epoch} steps')\n    h = model.fit(train_ds, steps_per_epoch=steps_per_epoch, epochs=args.epochs, verbose=1,\n        validation_data=val_ds, callbacks=get_callbacks(work_dir))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# INFERENCE\n\n1. Predict with multi-head model.\n2. Classify seen/unseen by maximum value of cosface head's output.\n3. For seen data, use multi-heads model r, c, v outputs.\n4. For unseen data, use separate 3 models for each r, c, v."},{"metadata":{"trusted":true},"cell_type":"code","source":"import cv2\nimport numpy as np\nimport os\nfrom tqdm.notebook import tqdm\nimport gc\n\ndef normalize_image(img, org_width, org_height, new_width, new_height):\n  # Invert\n  img = 255 - img\n  # Normalize\n  img = (img * (255.0 / img.max())).astype(np.uint8)\n  # Reshape\n  img = img.reshape(org_height, org_width)\n  image_resized = cv2.resize(img, (new_width, new_height), interpolation=cv2.INTER_LINEAR)\n  return image_resized\n\n        \ndef predict_batch(img_batch, model, all_preds, unseen_th=0.5):\n    img_batch = np.float32(img_batch)\n    bs = img_batch.shape[0]\n    r,v,c,u = np.zeros((bs,168)), np.zeros((bs,11)), np.zeros((bs,7)), np.zeros((bs,1292))\n    # deal with single image\n    if img_batch.ndim != 4:\n        img_batch = np.expand_dims(img_batch, 0)\n\n    y_pred = [\n        model[0].predict([img_batch, r,v,c,u]),\n        model[1].predict([img_batch, r]),\n        model[2].predict([img_batch, v]),\n        model[3].predict([img_batch, c]),\n    ]\n    \n    for k in range(len(y_pred[0][0])):\n        uu_p = y_pred[0][3][k].max(-1)\n        if uu_p > unseen_th:\n            rr=y_pred[0][0][k].astype('float16')\n            vv=y_pred[0][1][k].astype('float16')\n            cc=y_pred[0][2][k].astype('float16')       \n        else:\n            rr=y_pred[1][0][k].astype('float16')\n            vv=y_pred[2][0][k].astype('float16')\n            cc=y_pred[3][0][k].astype('float16')\n        all_preds['r'].append(rr)\n        all_preds['v'].append(vv)\n        all_preds['c'].append(cc)\n\ndef postprocess(preds, num_classes, EXP = -1.2):\n    p0 = np.argmax(preds,axis=1)\n\n    s = pd.Series(p0)\n    vc = s.value_counts().sort_index()\n    df = pd.DataFrame({'a':np.arange(num_classes),'b':np.ones(num_classes)})\n    df.b = df.a.map(vc)\n    df.fillna(df.b.min(),inplace=True)\n    mat1 = np.diag(df.b.astype('float32')**EXP)\n\n    p0 = np.argmax(preds.dot(mat1), axis=1)\n    \n    return p0\n\ndef main():\n    parser = argparse.ArgumentParser()\n    parser.add_argument('--seed', type=int, default=123)\n    parser.add_argument('--input_size', type=str, default='224,224')\n    parser.add_argument('--batch_size', type=int, default=32)\n    parser.add_argument('--backbone', type=str, default='efficientnet-b3')\n    parser.add_argument('--weights', type=str, default='../input/efnet-b3-cosface-val9-3/best.h5')\n    parser.add_argument('--r_weights', type=str, default='../input/efnet-b3-rcv-models/r_best.h5')\n    parser.add_argument('--v_weights', type=str, default='../input/efnet-b3-rcv-models/v_best.h5')\n    parser.add_argument('--c_weights', type=str, default='../input/efnet-b3-rcv-models/c_best.h5')\n    parser.add_argument('--mode',type=str, default='all')\n    args, _ = parser.parse_known_args()\n\n    org_height = 137\n    org_width = 236\n    args.input_size = tuple(int(x) for x in args.input_size.split(','))\n    np.random.seed(args.seed)\n    tf.random.set_seed(args.seed)\n\n    u_model = get_model(input_size=args.input_size + (3, ), backbone=args.backbone,weights=None)\n    u_model.load_weights(args.weights)\n    r_model = get_r_model(input_size=args.input_size + (3, ), backbone=args.backbone,weights=None)\n    r_model.load_weights(args.r_weights)\n    v_model = get_v_model(input_size=args.input_size + (3, ), backbone=args.backbone,weights=None)\n    v_model.load_weights(args.v_weights)\n    c_model = get_c_model(input_size=args.input_size + (3, ), backbone=args.backbone,weights=None)\n    c_model.load_weights(args.c_weights)\n    \n    model = [u_model, r_model, v_model, c_model]\n    \n    #print(model.summary())\n    \n    all_preds = dict(r = [], v = [], c = [])\n    all_image_ids = []\n    img_batch = []\n    for i in tqdm(range(4)):\n        parquet_fn = f'../input/bengaliai-cv19/test_image_data_{i}.parquet'\n        #parquet_fn = f'../input/bengaliai-cv19/train_image_data_{i}.parquet' # to check memory error\n        all_images = pd.read_parquet(parquet_fn)\n        image_ids = all_images['image_id'].values\n        all_images = all_images.iloc[:, 1:].values\n        for k in tqdm(range(len(image_ids))):\n            all_image_ids.append(image_ids[k])\n            img = all_images[k]\n            img = normalize_image(img, org_width, org_height, args.input_size[1], args.input_size[0])\n            img_batch.append(np.dstack([img] * 3))\n            if len(img_batch) >= args.batch_size:\n                predict_batch(img_batch, model, all_preds)\n                img_batch = []\n\n        # process remaining batch\n        if len(img_batch) > 0:\n            predict_batch(img_batch, model, all_preds)\n            img_batch = []\n            \n        np.save(f'r_preds{i}', all_preds['r'])\n        np.save(f'v_preds{i}', all_preds['v'])\n        np.save(f'c_preds{i}', all_preds['c'])\n        all_preds = dict(r = [], v = [], c = [])\n        \n        del all_images\n        gc.collect()\n\n    del u_model, r_model, v_model, c_model, model\n    gc.collect()\n    \n    r_preds = np.concatenate([np.load(f'r_preds{i}.npy') for i in range(4)] ,axis=0)\n    r_preds = postprocess(r_preds, 168, -1.2)\n    #r_preds = r_preds.argmax(axis=1)\n    \n    v_preds = np.concatenate([np.load(f'v_preds{i}.npy') for i in range(4)] ,axis=0)\n    v_preds = postprocess(v_preds, 11, -1.2)\n    #v_preds = v_preds.argmax(axis=1)\n    \n    c_preds = np.concatenate([np.load(f'c_preds{i}.npy') for i in range(4)] ,axis=0)\n    c_preds = postprocess(c_preds, 7, -0.5)\n    #c_preds = c_preds.argmax(axis=1)\n    \n    # create submission\n    row_id, target = [], []\n    for iid, r, v, c in zip(all_image_ids, r_preds, v_preds, c_preds):\n        row_id.append(iid + '_grapheme_root')\n        target.append(r)\n        row_id.append(iid + '_vowel_diacritic')\n        target.append(v)\n        row_id.append(iid + '_consonant_diacritic')\n        target.append(c)\n\n    sub_fn = 'submission.csv'\n    sub = pd.DataFrame({'row_id': row_id, 'target': target})\n    sub.to_csv(sub_fn, index=False)\n    print(f'Done wrote to {sub_fn}')\n\nmain()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv('submission.csv')\nsub.head(20)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}