{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Update6 \n\nUsing same pipeline with an efficientnetb3, efficientnetb5 using arc face margin 0.15 and NUM_TO_RERANK = 1. Also, retrain the model with image size 512.\n\n\n# Update5\n\nUsing same pipeline with an efficientnetb3 using arc face margin 0.15.\n\n\n# Update4\n\nBlend efficientnetB1 and B2 and change the NUM_TO_RANK to 3. \n\nBlending did not improve the score that much, i believe it is because both efficientnets are correlated. To blend 2 models, just average the embeddings.\n\n# Update3\n\nChange validation to 2%, efficientnetB1 and train with more epochs.\n\nThe number of unique training classes is 81313 of 81313 total classes\n\nThe number of unique validation classes is 24574 of 81313 total classes\n\nTraining with 1548860 images\n\nValidating with 31610 images\n\n\n# Update2\n\nUsing retrieval with private train dataset increase public score a lot. Non landmark images and images with a few observations are hard to predict. For this case I use a arcface layer with an efficientnetB0, and the val loss accuracy was 0.730. The nice thing is that leaderboard is much better than last experiment. Well there is a los of ground for improvement. I updated the script so you can experiment with this pipeline.\n\nThe number of unique training classes is 80937 of 81313 total classes\n\nThe number of unique validation classes is 64024 of 81313 total classes\n\n\n# Update1\n\nUsed EfficientNetB5 with 100% of the data using 20% validation.\n\nThe number of unique training classes is 80937 of 81313 total classes\n\nThe number of unique validation classes is 64024 of 81313 total classes\n\nThis model got a validation accuracy of 0.86.\n\n\n# Comments\n\nSup kaggle, in this pipeline and script i want to share my results trianing my own models so that you can use this as a baseline. Here i trained a basic efficientnetB3 with the total amount of classes \"81313\". I only used 80% of the total training data and my experiments show me that if you train with more data, the validation score improves and also the public leaderboard. \n\nBecuase we are only trianing with 80% of the data, we are not actually trianing all the classes because there are some classes that have 2 samples. Here are some basic stats for the trained model.\n\nThe number of unique training classes is 74450 of 81313 total classes\n\nThe number of unique validation classes is 66213 of 81313 total classes\n\nThe validation accuracy score is 0.82 and the gap is 0.80\n\nImportant:\n\nIm still not sure why my validation is not align with the public leaderboard score, i believe it is because is has another target distribution + non landmark images, but this is just an hypothesis.\n\nSome insights that you may find usefull are the following:\n\n* Here is a discussion where I comment the dataset im using https://www.kaggle.com/c/landmark-recognition-2020/discussion/180056\n* Bigger image size gives better score but they need more resources to train, in other words it will take more time\n* Bigger efficientnets give better scores\n* Retrieval methods adjust to this problem much more that normal classification, so you want to try that approach, this is just an example to get you started building and fitting your own models\n* Model was trained in colab tpu, with a regular free account you have 12 hours to train your model, if you need more time just save the model each epoch and then reload it in another session and continue training"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# Building TF Records\n\nHere is the code to build 50 tf records, they are stratified by the target and i had to run 50 sessions to get them all."},{"metadata":{"trusted":true},"cell_type":"code","source":"# import numpy as np\n# import pandas as pd\n# import cv2\n# import tensorflow as tf\n# import pathlib\n# from tqdm import tqdm\n\n# def _bytes_feature(value):\n#     \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n#     if isinstance(value, type(tf.constant(0))):\n#         value = value.numpy() # BytesList won't unpack a string from an EagerTensor.\n#     return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n\n# def _float_feature(value):\n#     \"\"\"Returns a float_list from a float / double.\"\"\"\n#     return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))\n\n# def _int64_feature(value):\n#     \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n#     return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))\n\n# def serialize_example(feature0, feature1, feature2):\n#     feature = {\n#         'id': _bytes_feature(feature0),\n#         'image': _bytes_feature(feature1),\n#         'target': _int64_feature(feature2)\n#     }\n#     example_proto = tf.train.Example(features = tf.train.Features(feature = feature))\n#     return example_proto.SerializeToString()\n# TRAIN_IMAGE_DIR = '../input/landmark-recognition-2020/train'\n# TRAIN = '../input/landmark-image-train/train_encoded.csv'\n\n# # Read image and resize it\n# def read_image(image_path, size = (384, 384)):\n#     img = cv2.imread(image_path)\n#     img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n#     img = cv2.resize(img, size)\n#     return img\n\n\n# def get_tf_records(record = 0, size = (384, 384)):\n#     df = pd.read_csv(TRAIN)\n#     # Get image paths\n#     image_paths = [x for x in pathlib.Path(TRAIN_IMAGE_DIR).rglob('*.jpg')]\n#     # Get only one group, this is a slow process so we need to make 50 different sessions\n#     df = df[df['group'] == record]\n#     # Reset index \n#     df.reset_index(drop = True, inplace = True)\n#     # Get a list of ids\n#     ids_list = list(df['id'].unique())\n#     # Write tf records\n#     with tf.io.TFRecordWriter('train_{}.tfrec'.format(record)) as writer:\n#         for image_path in tqdm(image_paths):\n#             image_id = image_path.name.split('.')[0]\n#             if image_id in ids_list:\n#                 # Get target\n#                 target = df[df['id'] == image_id]['landmark_id_encode']\n#                 img = read_image(str(image_path), size)\n#                 img = cv2.imencode('.jpg', img, (cv2.IMWRITE_JPEG_QUALITY, 100))[1].tostring()\n#                 example = serialize_example(\n#                     str.encode(image_id), img, target.values[0]\n#                 )\n#                 writer.write(example)\n                \n# get_tf_records(record = 0, size = (384, 384))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Pipeline\n\nHere is the model pipeline, very basic pipeline"},{"metadata":{"trusted":true},"cell_type":"code","source":"# %tensorflow_version 2.x\n# from google.colab import drive\n# drive.mount('/content/drive')\n\n# !pip install -q efficientnet\n# import os\n# import re\n# import numpy as np\n# import pandas as pd\n# import random\n# import math\n# from sklearn import metrics\n# from sklearn.model_selection import train_test_split\n# import tensorflow as tf\n# import efficientnet.tfkeras as efn\n# from tensorflow.keras import backend as K\n# import tensorflow_addons as tfa\n# !pip install gcsfs\n# from tqdm.notebook import tqdm as tqdm\n\n# !pip install tensorflow~=2.2.0 tensorflow_gcs_config~=2.2.0\n# import requests\n# resp = requests.post(\"http://{}:8475/requestversion/{}\".format(os.environ[\"COLAB_TPU_ADDR\"].split(\":\")[0], tf.__version__))\n# if resp.status_code != 200:\n#   print(\"Failed to switch the TPU to TF {}\".format(version))\n\n# try:\n#     # TPU detection. No parameters necessary if TPU_NAME environment variable is\n#     # set: this is always the case on Kaggle.\n#     tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n#     print('Running on TPU ', tpu.master())\n# except ValueError:\n#     tpu = None\n\n# if tpu:\n#     tf.config.experimental_connect_to_cluster(tpu)\n#     tf.tpu.experimental.initialize_tpu_system(tpu)\n#     strategy = tf.distribute.experimental.TPUStrategy(tpu)\n# else:\n#     # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n#     strategy = tf.distribute.get_strategy()\n\n# print(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\n# # For tf.dataset\n# AUTO = tf.data.experimental.AUTOTUNE\n\n# # Data access\n# GCS_PATH = 'gs://kds-8e6633c4a6d544ae006948f95c01d818cf70ee95ed8ea3731ddbd5dc'\n# GCS_PATH_2 = 'gs://kds-6c5f45cfe497efd7115b4ccc111abe0d435e12a356d98167abf66c21'\n# DICT_PATH = 'gs://kds-80f8b28815daf39c39d710eca9c78b31e9f396674d64cad8af10e75e/train_encoded.csv'\n\n# # Configuration\n# EPOCHS = 20\n# BATCH_SIZE = 32 * strategy.num_replicas_in_sync\n# IMAGE_SIZE = [384, 384]\n# # Seed\n# SEED = 100\n# # Learning rate\n# LR = 0.0001\n# # Number of classes\n# NUMBER_OF_CLASSES = 81313\n\n# # Training filenames directory\n# FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train*.tfrec') + tf.io.gfile.glob(GCS_PATH_2 + '/train*.tfrec')\n# # Read csv file\n# df = pd.read_csv(DICT_PATH)\n# # Using 20% of the data to validate\n# TRAINING_FILENAMES, VALIDATION_FILENAMES = train_test_split(FILENAMES, test_size = 0.20, random_state = SEED)\n# training_groups = [int(re.compile(r\"_([0-9]*)\\.\").search(filename).group(1)) for filename in TRAINING_FILENAMES]\n# validation_groups = [int(re.compile(r\"_([0-9]*)\\.\").search(filename).group(1)) for filename in VALIDATION_FILENAMES]\n# n_trn_classes = df[df['group'].isin(training_groups)]['landmark_id_encode'].nunique()\n# n_val_classes = df[df['group'].isin(validation_groups)]['landmark_id_encode'].nunique()\n# print(f'The number of unique training classes is {n_trn_classes} of {NUMBER_OF_CLASSES} total classes')\n# print(f'The number of unique validation classes is {n_val_classes} of {NUMBER_OF_CLASSES} total classes')\n\n# # Seed everything\n# def seed_everything(seed):\n#     random.seed(seed)\n#     np.random.seed(seed)\n#     os.environ['PYTHONHASHSEED'] = str(seed)\n#     tf.random.set_seed(seed)\n\n# # Function to decode our images (normalize and reshape)\n# def decode_image(image_data):\n#     image = tf.image.decode_jpeg(image_data, channels = 3)\n#     # Convert image to floats in [0, 1] range\n#     image = tf.cast(image, tf.float32) / 255.0\n#     # Explicit size needed for TPU\n#     image = tf.reshape(image, [*IMAGE_SIZE, 3])\n#     return image\n\n# # This function parse our images and also get the target variable\n# def read_tfrecord(example):\n#     TFREC_FORMAT = {\n#         # tf.string means bytestring\n#         \"image\": tf.io.FixedLenFeature([], tf.string), \n#         # shape [] means single element\n#         \"target\": tf.io.FixedLenFeature([], tf.int64)\n#         }\n#     example = tf.io.parse_single_example(example, TFREC_FORMAT)\n#     image = decode_image(example['image'])\n#     target = tf.cast(example['target'], tf.int32)\n#     return image, target\n\n# # This function load our tf records and parse our data with the previous function\n# def load_dataset(filenames, ordered = False):\n#     # Read from TFRecords. For optimal performance, reading from multiple files at once and\n#     # Diregarding data order. Order does not matter since we will be shuffling the data anyway\n    \n#     ignore_order = tf.data.Options()\n#     if not ordered:\n#         # Disable order, increase speed\n#         ignore_order.experimental_deterministic = False \n        \n#     # Automatically interleaves reads from multiple files\n#     dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads = AUTO)\n#     # Use data as soon as it streams in, rather than in its original order\n#     dataset = dataset.with_options(ignore_order)\n#     # Returns a dataset of (image, label) pairs\n#     dataset = dataset.map(read_tfrecord, num_parallel_calls = AUTO) \n#     return dataset\n\n# # This function output the data so that we can use arcface\n# def arcface_format(image, target):\n#     return {'inp1': image, 'inp2': target}, target\n\n# # Training data pipeline\n# def get_training_dataset(filenames, ordered = False):\n#     dataset = load_dataset(filenames, ordered = ordered)\n#     dataset = dataset.map(arcface_format, num_parallel_calls = AUTO)\n#     # The training dataset must repeat for several epochs\n#     dataset = dataset.repeat() \n#     dataset = dataset.shuffle(2048)\n#     dataset = dataset.batch(BATCH_SIZE)\n#     # Prefetch next batch while training (autotune prefetch buffer size)\n#     dataset = dataset.prefetch(AUTO)\n#     return dataset\n\n# # Validation data pipeline\n# def get_validation_dataset(filenames, ordered = True, prediction = False):\n#     dataset = load_dataset(filenames, ordered = ordered)\n#     dataset = dataset.map(arcface_format, num_parallel_calls = AUTO)\n#     # If we are in prediction mode, use bigger batch size for faster prediction\n#     if prediction:\n#         dataset = dataset.batch(BATCH_SIZE * 4)\n#     else:\n#         dataset = dataset.batch(BATCH_SIZE)\n#     # Prefetch next batch while training (autotune prefetch buffer size)\n#     dataset = dataset.prefetch(AUTO) \n#     return dataset\n\n# # Count the number of observations with the tabular csv\n# def count_data_items(filenames):\n#     records = [int(re.compile(r\"_([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n#     df = pd.read_csv(DICT_PATH)\n#     n = df[df['group'].isin(records)].shape[0]\n#     return n\n\n# NUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\n# NUM_VALIDATION_IMAGES  = count_data_items(VALIDATION_FILENAMES)\n# print(f'Training with {NUM_TRAINING_IMAGES} images')\n# print(f'Validating with {NUM_VALIDATION_IMAGES} images')\n\n# # Function for a custom learning rate scheduler with warmup and decay\n# def get_lr_callback():\n#     lr_start   = 0.000001\n#     lr_max     = 0.0000005 * BATCH_SIZE\n#     lr_min     = 0.000001\n#     lr_ramp_ep = 5\n#     lr_sus_ep  = 0\n#     lr_decay   = 0.8\n   \n#     def lrfn(epoch):\n#         if epoch < lr_ramp_ep:\n#             lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start   \n#         elif epoch < lr_ramp_ep + lr_sus_ep:\n#             lr = lr_max    \n#         else:\n#             lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min    \n#         return lr\n\n#     lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = False)\n#     return lr_callback\n\n# # Function to calculate global average precision score\n# def gap_vector(pred, conf, true, return_x = False):\n#     '''\n#     Compute Global Average Precision (aka micro AP), the metric for the\n#     Google Landmark Recognition competition. \n#     This function takes predictions, labels and confidence scores as vectors.\n#     In both predictions and ground-truth, use None/np.nan for \"no label\".\n\n#     Args:\n#         pred: vector of integer-coded predictions\n#         conf: vector of probability or confidence scores for pred\n#         true: vector of integer-coded labels for ground truth\n#         return_x: also return the data frame used in the calculation\n\n#     Returns:\n#         GAP score\n#     '''\n#     x = pd.DataFrame({'pred': pred, 'conf': conf, 'true': true})\n#     x.sort_values('conf', ascending = False, inplace = True, na_position = 'last')\n#     x['correct'] = (x.true == x.pred).astype(int)\n#     x['prec_k'] = x.correct.cumsum() / (np.arange(len(x)) + 1)\n#     x['term'] = x.prec_k * x.correct\n#     gap = x.term.sum() / x.true.count()\n#     if return_x:\n#         return gap, x\n#     else:\n#         return gap\n\n# class ArcMarginProduct(tf.keras.layers.Layer):\n#     '''\n#     Implements large margin arc distance.\n\n#     Reference:\n#         https://arxiv.org/pdf/1801.07698.pdf\n#         https://github.com/lyakaap/Landmark2019-1st-and-3rd-Place-Solution/\n#             blob/master/src/modeling/metric_learning.py\n#     '''\n#     def __init__(self, n_classes, s=30, m=0.50, easy_margin=False,\n#                  ls_eps=0.0, **kwargs):\n\n#         super(ArcMarginProduct, self).__init__(**kwargs)\n\n#         self.n_classes = n_classes\n#         self.s = s\n#         self.m = m\n#         self.ls_eps = ls_eps\n#         self.easy_margin = easy_margin\n#         self.cos_m = tf.math.cos(m)\n#         self.sin_m = tf.math.sin(m)\n#         self.th = tf.math.cos(math.pi - m)\n#         self.mm = tf.math.sin(math.pi - m) * m\n\n#     def get_config(self):\n\n#         config = super().get_config().copy()\n#         config.update({\n#             'n_classes': self.n_classes,\n#             's': self.s,\n#             'm': self.m,\n#             'ls_eps': self.ls_eps,\n#             'easy_margin': self.easy_margin,\n#         })\n#         return config\n\n#     def build(self, input_shape):\n#         super(ArcMarginProduct, self).build(input_shape[0])\n\n#         self.W = self.add_weight(\n#             name='W',\n#             shape=(int(input_shape[0][-1]), self.n_classes),\n#             initializer='glorot_uniform',\n#             dtype='float32',\n#             trainable=True,\n#             regularizer=None)\n\n#     def call(self, inputs):\n#         X, y = inputs\n#         y = tf.cast(y, dtype=tf.int32)\n#         cosine = tf.matmul(\n#             tf.math.l2_normalize(X, axis=1),\n#             tf.math.l2_normalize(self.W, axis=0)\n#         )\n#         sine = tf.math.sqrt(1.0 - tf.math.pow(cosine, 2))\n#         phi = cosine * self.cos_m - sine * self.sin_m\n#         if self.easy_margin:\n#             phi = tf.where(cosine > 0, phi, cosine)\n#         else:\n#             phi = tf.where(cosine > self.th, phi, cosine - self.mm)\n#         one_hot = tf.cast(\n#             tf.one_hot(y, depth=self.n_classes),\n#             dtype=cosine.dtype\n#         )\n#         if self.ls_eps > 0:\n#             one_hot = (1 - self.ls_eps) * one_hot + self.ls_eps / self.n_classes\n\n#         output = (one_hot * phi) + ((1.0 - one_hot) * cosine)\n#         output *= self.s\n#         return output\n\n\n# # Function to build our model using fine tunning (efficientnet)\n# def get_model():\n\n#     with strategy.scope():\n\n#         margin = ArcMarginProduct(\n#             n_classes = NUMBER_OF_CLASSES, \n#             s = 64, \n#             m = 0.05, \n#             name='head/arc_margin', \n#             dtype='float32'\n#             )\n\n#         inp = tf.keras.layers.Input(shape = (*IMAGE_SIZE, 3), name = 'inp1')\n#         label = tf.keras.layers.Input(shape = (), name = 'inp2')\n#         x0 = efn.EfficientNetB0(weights = 'imagenet', include_top = False)(inp)\n#         x = tf.keras.layers.GlobalAveragePooling2D()(x0)\n#         x = tf.keras.layers.Dropout(0.3)(x)\n#         x = tf.keras.layers.Dense(512)(x)\n#         x = margin([x, label])\n        \n#         output = tf.keras.layers.Softmax(dtype='float32')(x)\n\n#         model = tf.keras.models.Model(inputs = [inp, label], outputs = [output])\n\n#         opt = tf.keras.optimizers.Adam(learning_rate = LR)\n\n#         model.compile(\n#             optimizer = opt,\n#             loss = [tf.keras.losses.SparseCategoricalCrossentropy()],\n#             metrics = [tf.keras.metrics.SparseCategoricalAccuracy()]\n#             ) \n        \n#         return model\n\n# # Seed everything\n# seed_everything(SEED)\n\n# # Build training and validation generators\n# train_dataset = get_training_dataset(TRAINING_FILENAMES, ordered = False)\n# val_dataset = get_validation_dataset(VALIDATION_FILENAMES, ordered = True, prediction = False)\n# STEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\n\n# model = get_model()\n# # Using a checkpoint to save best model (want the entire model, not only the weights)\n# checkpoint = tf.keras.callbacks.ModelCheckpoint(f'/content/drive/My Drive/Models/baseline_model_effb0_arcface.h5', \n#                                                  monitor = 'val_loss', \n#                                                  save_best_only = True, \n#                                                  save_weights_only = False)\n# # Using learning rate scheduler\n# cb_lr_schedule = tf.keras.callbacks.ReduceLROnPlateau(monitor = 'val_loss', \n#                                                        mode = 'min', \n#                                                        factor = 0.5, \n#                                                        patience = 1, \n#                                                        verbose = 1, \n#                                                        min_delta = 0.0001)\n\n# # Train and evaluate our model\n# history = model.fit(train_dataset,  \n#                     steps_per_epoch = STEPS_PER_EPOCH,\n#                     epochs = EPOCHS,\n#                     callbacks = [get_lr_callback(), checkpoint],\n#                     validation_data = val_dataset,\n#                     verbose = 1\n#                     )\n\n# # Restart tpu\n# tf.tpu.experimental.initialize_tpu_system(tpu)\n# # Load best model\n# model = tf.keras.models.load_model('/content/drive/My Drive/Models/baseline_model_effb0_arcface.h5')\n\n# # Reset val dataset, now in prediction mode\n# val_dataset = get_validation_dataset(VALIDATION_FILENAMES, ordered = True, prediction = True)\n# # Get ground truth target for the fold\n# val_target = val_dataset.map(lambda image, target: target).unbatch()\n# val_targets = list(next(iter(val_target.batch(NUM_VALIDATION_IMAGES))).numpy())\n\n#  # Predictions\n# val_image = val_dataset.map(lambda image, target: image['inp1'])\n# # Transform validation dataset as a numpy iterator\n# val_image = val_image.as_numpy_iterator()\n# # Initiate empty list to store predictions and confidences\n# target_predictions = []\n# target_confidences = []\n# # Iterate over validation images and predict in batches of 1024 images\n# batches = math.ceil(NUM_VALIDATION_IMAGES / (BATCH_SIZE * 4))\n# for image in tqdm(val_image, total = batches):\n#     prediction = model.predict(image)\n#     target_prediction = np.argmax(prediction, axis = -1)\n#     target_confidence = np.max(prediction, axis = -1)\n#     target_predictions.extend(list(target_prediction))\n#     target_confidences.extend(list(target_confidence))\n\n# # Calculate global average precision for the fold\n# gap = gap_vector(target_predictions, target_confidences, val_targets)\n# accuracy_score = metrics.accuracy_score(val_targets, target_predictions)\n# print(f'Our global average precision score is {gap}')\n# print(f'Our accuracy score is {accuracy_score}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference Pipeline\n\nI did some test to be make sure that my inference pipeline matches the trainig pipeline, for this I validate a tf records and the validation score was exactly the same!.\n\nAlso we add a condition to just save the sample submission if it is the public test set so we can make the submission faster"},{"metadata":{"trusted":true},"cell_type":"code","source":"import operator\nimport gc\nimport pathlib\nimport shutil\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow.keras import backend as K\nfrom scipy import spatial\nimport cv2\n!pip install ../input/landmark-baseline-model/Keras_Applications-1.0.8-py3-none-any.whl\n!pip install ../input/landmark-baseline-model/efficientnet-1.1.0-py3-none-any.whl\nimport efficientnet.tfkeras as efn\nimport math\n\nNUMBER_OF_CLASSES = 81313\nIMAGE_SIZE = [384, 384]\nLR = 0.0001\n\nclass ArcMarginProduct(tf.keras.layers.Layer):\n    '''\n    Implements large margin arc distance.\n\n    Reference:\n        https://arxiv.org/pdf/1801.07698.pdf\n        https://github.com/lyakaap/Landmark2019-1st-and-3rd-Place-Solution/\n            blob/master/src/modeling/metric_learning.py\n    '''\n    def __init__(self, n_classes, s=30, m=0.50, easy_margin=False,\n                 ls_eps=0.0, **kwargs):\n\n        super(ArcMarginProduct, self).__init__(**kwargs)\n\n        self.n_classes = n_classes\n        self.s = s\n        self.m = m\n        self.ls_eps = ls_eps\n        self.easy_margin = easy_margin\n        self.cos_m = tf.math.cos(m)\n        self.sin_m = tf.math.sin(m)\n        self.th = tf.math.cos(math.pi - m)\n        self.mm = tf.math.sin(math.pi - m) * m\n\n    def get_config(self):\n\n        config = super().get_config().copy()\n        config.update({\n            'n_classes': self.n_classes,\n            's': self.s,\n            'm': self.m,\n            'ls_eps': self.ls_eps,\n            'easy_margin': self.easy_margin,\n        })\n        return config\n\n    def build(self, input_shape):\n        super(ArcMarginProduct, self).build(input_shape[0])\n\n        self.W = self.add_weight(\n            name='W',\n            shape=(int(input_shape[0][-1]), self.n_classes),\n            initializer='glorot_uniform',\n            dtype='float32',\n            trainable=True,\n            regularizer=None)\n\n    def call(self, inputs):\n        X, y = inputs\n        y = tf.cast(y, dtype=tf.int32)\n        cosine = tf.matmul(\n            tf.math.l2_normalize(X, axis=1),\n            tf.math.l2_normalize(self.W, axis=0)\n        )\n        sine = tf.math.sqrt(1.0 - tf.math.pow(cosine, 2))\n        phi = cosine * self.cos_m - sine * self.sin_m\n        if self.easy_margin:\n            phi = tf.where(cosine > 0, phi, cosine)\n        else:\n            phi = tf.where(cosine > self.th, phi, cosine - self.mm)\n        one_hot = tf.cast(\n            tf.one_hot(y, depth=self.n_classes),\n            dtype=cosine.dtype\n        )\n        if self.ls_eps > 0:\n            one_hot = (1 - self.ls_eps) * one_hot + self.ls_eps / self.n_classes\n\n        output = (one_hot * phi) + ((1.0 - one_hot) * cosine)\n        output *= self.s\n        return output\n\n\n# Function to build our model using fine tunning (efficientnet)\ndef get_model(eff = 1):\n    \n\n    margin = ArcMarginProduct(\n        n_classes = NUMBER_OF_CLASSES, \n        s = 64, \n        m = 0.15, \n        name='head/arc_margin', \n        dtype='float32'\n        )\n\n    inp = tf.keras.layers.Input(shape = (*IMAGE_SIZE, 3), name = 'inp1')\n    label = tf.keras.layers.Input(shape = (), name = 'inp2')\n    if eff == 0:\n        x = efn.EfficientNetB0(weights = None, include_top = False)(inp)\n    elif eff == 1:\n        x = efn.EfficientNetB1(weights = None, include_top = False)(inp)\n    elif eff == 2:\n        x = efn.EfficientNetB2(weights = None, include_top = False)(inp)\n    elif eff == 3:\n        x = efn.EfficientNetB3(weights = None, include_top = False)(inp)\n    elif eff == 4:\n        x = efn.EfficientNetB4(weights = None, include_top = False)(inp)\n    elif eff == 5:\n        x = efn.EfficientNetB5(weights = None, include_top = False)(inp)\n    elif eff == 6:\n        x = efn.EfficientNetB6(weights = None, include_top = False)(inp)\n    elif eff == 7:\n        x = efn.EfficientNetB7(weights = None, include_top = False)(inp)\n        \n    x = tf.keras.layers.GlobalAveragePooling2D()(x)\n    x = tf.keras.layers.Dropout(0.3)(x)\n    x = tf.keras.layers.Dense(512)(x)\n    x = margin([x, label])\n\n    output = tf.keras.layers.Softmax(dtype='float32')(x)\n\n    model = tf.keras.models.Model(inputs = [inp, label], outputs = [output])\n\n    opt = tf.keras.optimizers.Adam(learning_rate = LR)\n\n    model.compile(\n        optimizer = opt,\n        loss = [tf.keras.losses.SparseCategoricalCrossentropy()],\n        metrics = [tf.keras.metrics.SparseCategoricalAccuracy()]\n        ) \n\n    return model\n\n\nNUM_EMBEDDING_DIMENSIONS = 512\nDATASET_DIR = '../input/landmark-image-train/train_encoded.csv'\nTEST_IMAGE_DIR = '../input/landmark-recognition-2020/test'\nTRAIN_IMAGE_DIR = '../input/landmark-recognition-2020/train'\nMODEL1 = get_model(eff = 3)\nMODEL1.load_weights('../input/landmark-baseline-model/baseline_model_effb3_arcface_0_15_512.h5')\nMODEL1 = tf.keras.models.Model(inputs = MODEL1.input[0], outputs = MODEL1.layers[-4].output)\nMODEL2 = get_model(eff = 5)\nMODEL2.load_weights('../input/landmark-baseline-model/baseline_model_effb5_arcface_0_15_512.h5')\nMODEL2 = tf.keras.models.Model(inputs = MODEL2.input[0], outputs = MODEL2.layers[-4].output)\nNUM_TO_RERANK = 1\n\n\nNUM_PUBLIC_TEST_IMAGES = 10345 # Used to detect if in session or re-run.\n\n# Read image and resize it\ndef read_image(image_path, size = (384, 384)):\n    img = cv2.imread(image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    img = cv2.resize(img, size)\n    img = cv2.imencode('.jpg', img, (cv2.IMWRITE_JPEG_QUALITY, 100))[1].tostring()\n    img = tf.image.decode_jpeg(img, channels = 3)\n    img = tf.image.resize(img, (512, 512))\n    img = tf.cast(img, tf.float32) / 255.0\n    img = tf.reshape(img, [1, 512, 512, 3])\n    return img\n\n# Function to get training and test embeddings\ndef generate_embeddings(filepaths):\n    image_paths = [x for x in pathlib.Path(filepaths).rglob('*.jpg')]\n    num_images = len(image_paths)\n    ids = num_images * [None]\n    # Generate an empty matrix where we can store the embeddings of each image\n    embeddings = np.empty((num_images, NUM_EMBEDDING_DIMENSIONS))\n    for i, image_path in enumerate(image_paths):\n        ids[i] = image_path.name.split('.')[0]\n        image_tensor = read_image(str(image_path), (384, 384))\n        prediction1 = MODEL1.predict(image_tensor)\n        prediction2 = MODEL2.predict(image_tensor)\n        prediction = np.average([prediction1, prediction2], axis = 0)\n        embeddings[i, :] = prediction\n    return ids, embeddings\n\n# This function get the most similar train images for each test image based on cosine similarity\ndef get_similarities(train_csv, test_directory, train_directory):\n    # Get target dictionary\n    df = pd.read_csv(train_csv)\n    df = df[['id', 'landmark_id']]\n    df.set_index('id', inplace = True)\n    df = df.to_dict()['landmark_id']\n    # Extract the test ids and global feature for the test images\n    test_ids, test_embeddings = generate_embeddings(test_directory)\n    # Extract the train ids and global features for the train images\n    train_ids, train_embeddings = generate_embeddings(train_directory)\n    # Initiate a list were we will store the similar training images for each test image (also score)\n    train_ids_labels_and_scores = [None] * test_embeddings.shape[0]\n    # Using (slow) for-loop, as distance matrix doesn't fit in memory\n    for test_index in range(test_embeddings.shape[0]):\n        distances = spatial.distance.cdist(\n            test_embeddings[np.newaxis, test_index, : ], train_embeddings, 'cosine')[0]\n        # Get the indices of the closest images\n        top_k = np.argpartition(distances, NUM_TO_RERANK)[:NUM_TO_RERANK]\n        # Get the nearest ids and distances using the previous indices\n        nearest = sorted([(train_ids[p], distances[p]) for p in top_k], key = lambda x: x[1])\n        # Get the labels and score results\n        train_ids_labels_and_scores[test_index] = [(df[train_id], 1.0 - cosine_distance) for \\\n                                                   train_id, cosine_distance in nearest]\n        \n    del test_embeddings\n    del train_embeddings\n    gc.collect()\n    return test_ids, train_ids_labels_and_scores\n\n# This function aggregate top simlarities and make predictions\ndef generate_predictions(test_ids, train_ids_labels_and_scores):\n    targets = []\n    scores = []\n    \n    # Iterate through each test id\n    for test_index, test_id in enumerate(test_ids):\n        aggregate_scores = {}\n        # Iterate through the similar images with their corresponing score for the given test image\n        for target, score in train_ids_labels_and_scores[test_index]:\n            if target not in aggregate_scores:\n                aggregate_scores[target] = 0\n            aggregate_scores[target] += score\n        # Get the best score\n        target, score = max(aggregate_scores.items(), key = operator.itemgetter(1))\n        targets.append(target)\n        scores.append(score)\n        \n    final = pd.DataFrame({'id': test_ids, 'target': targets, 'scores': scores})\n    final['landmarks'] = final['target'].astype(str) + ' ' + final['scores'].astype(str)\n    final[['id', 'landmarks']].to_csv('submission.csv', index = False)\n    return final\n\ndef inference_and_save_submission_csv(train_csv, test_directory, train_directory):\n    image_paths = [x for x in pathlib.Path(test_directory).rglob('*.jpg')]\n    test_len = len(image_paths)\n    if test_len == NUM_PUBLIC_TEST_IMAGES:\n        # Dummy submission\n        shutil.copyfile('../input/landmark-recognition-2020/sample_submission.csv', 'submission.csv')\n        return 'Job Done'\n    else:\n        test_ids, train_ids_labels_and_scores = get_similarities(train_csv, test_directory, train_directory)\n        final = generate_predictions(test_ids, train_ids_labels_and_scores)\n        return final\n    \nfinal = inference_and_save_submission_csv(DATASET_DIR, TEST_IMAGE_DIR, TRAIN_IMAGE_DIR)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I hope this pipeline helps you"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}