{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Baseline model evaluation","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In order to make sure that model is learning correctly, it is useful to frequently run evaluation loops and keep track whether statistic of interest is being improved. In case of the problem of Landmark Retrieval, the statistic often used is Mean Average Precision (mAP). It is exactly the statistic used for model evaluation in this competiton. It is defined, in [contest evaluation protocol](https://www.kaggle.com/c/landmark-retrieval-2020/overview/evaluation), as:","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![mAP](https://i.imgur.com/ukUp5cy.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"There are a lot of great blogposts exaplaining this metric in-depth, for example [this one](https://towardsdatascience.com/breaking-down-mean-average-precision-map-ae462f623a52), but the simple thing to know is this: hidden between all these obscure math symbols is a recipe for coming up with a single number that summarizes our model in a way that if it provides embeddings (model output) that are useful for image retrieval, the mAP score gets bigger. In particular, for each query image, embeddings for all index images are compared with the embedding for the query image (using Euclidean distance between embedding vectors), and sorted in a way, such that index images with most similar embedding vectors are placed first. The metric used in this contest takes 100 most similar images and gets bigger, if ones corresponding to the landmark presented in query image are listed first.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Since the dataset used in the competition is very large (>1.5M images), it is not practical to use them all for evaluation - inference on such an enormous dataset takes a significant amount of time. Moreover, it is hard to pick a specific small subset of landmarks to use in evaluation - the dataset is very diverse and it is a challenge to come up with representative set of images.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In order to address this problem, smaller datasets are used in literature as well. One of those datasets are Revisited Oxford and Paris datasets, presented originally in [this paper](https://arxiv.org/abs/1803.11285), as a relabelling of popular [Oxford Buildings Dataset](https://www.robots.ox.ac.uk/~vgg/data/oxbuildings/) and [The Paris Dataset](https://www.robots.ox.ac.uk/~vgg/data/parisbuildings/). The relabeling was a considerate effort, which adressed noisiness present in original datasets. That means that the dataset can be treated with high confidence as a representative one. Moreover, the mAP metrics on these datasets are often reported in majority of recent Image Retrieval literature.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In this notebook, I present how to evaluate baseline model provided by Google on Revisited Oxford & Paris datasets. The evaluation loops can be easily re-used for your custom model trained for this competition. The code is based on https://github.com/filipradenovic/revisitop","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 1. Define utility functions","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport pickle\nimport random\n\nimport numpy as np\nimport tensorflow as tf\nfrom tqdm  import tqdm\nfrom PIL import Image, ImageFile\nfrom scipy.io import savemat, loadmat\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Utility script - parsing dataset config .pkl files and mAP computation\nfrom roxfordparis_tools import configdataset, compute_map","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Adapted from: https://github.com/filipradenovic/revisitop/blob/master/python/example_process_images.py\n\ndef pil_loader(path):\n    # to avoid crashing for truncated (corrupted images)\n    ImageFile.LOAD_TRUNCATED_IMAGES = True\n    # open path as file to avoid ResourceWarning \n    # (https://github.com/python-pillow/Pillow/issues/835)\n    with open(path, 'rb') as f:\n        img = Image.open(f)\n        return img.convert('RGB')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Adapted from: https://github.com/filipradenovic/revisitop/blob/master/python/example_process_images.py\n\ndef extract_features(test_dataset, cfg, model):\n    \"\"\"\n    Generates file with serialized model outputs for each image from test_dataset.\n    \n    Arguments\n    ---------\n    test_dataset   : name of dataset of interest (roxford5k | rparis6k)\n    cfg            : unserialized dataset config, containing annotation metadata\n    model          : loaded Tensorflow baseline model object\n    \"\"\"\n    \n    print('>> Processing query images...', flush=True)\n    Q = []\n    for i in tqdm(np.arange(cfg['nq'])):\n        qim = pil_loader(cfg['qim_fname'](cfg, i)).crop(cfg['gnd'][i]['bbx'])\n        image_data = np.array(qim)\n        image_tensor = tf.convert_to_tensor(image_data)\n        Q.append(model(image_tensor)['global_descriptor'].numpy())\n    Q = np.array(Q, dtype=np.float32)\n    Q = Q.transpose()\n    \n    print('>> Processing index images...', flush=True)\n    X = []\n    for i in tqdm(np.arange(cfg['n'])):\n        im = pil_loader(cfg['im_fname'](cfg, i))\n        image_data = np.array(im)\n        image_tensor = tf.convert_to_tensor(image_data)\n        X.append(model(image_tensor)['global_descriptor'].numpy())\n    X = np.array(X, dtype=np.float32)\n    X = X.transpose()\n\n    feature_dict = {'X': X, 'Q': Q}\n    mat_save_path = \"{}_delg_baseline.mat\".format(test_dataset)\n    print('>> Saving model outputs to: {}'.format(mat_save_path))\n    savemat(mat_save_path, feature_dict)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Adapted from: https://github.com/filipradenovic/revisitop/blob/master/python/example_evaluate.py\n\ndef run_evaluation(test_dataset, cfg, features_dir):\n    ks = [1, 5, 10]\n    gnd = cfg['gnd']\n    features = loadmat(os.path.join(features_dir, '{}_delg_baseline.mat'.format(test_dataset)))\n\n    Q = features['Q']\n    X = features['X']\n    sim = np.dot(X.T, Q)\n    ranks = np.argsort(-sim, axis=0)\n\n    # search for easy\n    gnd_t = []\n    for i in range(len(gnd)):\n        g = {}\n        g['ok'] = np.concatenate([gnd[i]['easy']])\n        g['junk'] = np.concatenate([gnd[i]['junk'], gnd[i]['hard']])\n        gnd_t.append(g)\n    mapE, apsE, mprE, prsE = compute_map(ranks, gnd_t, ks)\n\n    # search for easy & hard\n    gnd_t = []\n    for i in range(len(gnd)):\n        g = {}\n        g['ok'] = np.concatenate([gnd[i]['easy'], gnd[i]['hard']])\n        g['junk'] = np.concatenate([gnd[i]['junk']])\n        gnd_t.append(g)\n    mapM, apsM, mprM, prsM = compute_map(ranks, gnd_t, ks)\n\n    # search for hard\n    gnd_t = []\n    for i in range(len(gnd)):\n        g = {}\n        g['ok'] = np.concatenate([gnd[i]['hard']])\n        g['junk'] = np.concatenate([gnd[i]['junk'], gnd[i]['easy']])\n        gnd_t.append(g)\n    mapH, apsH, mprH, prsH = compute_map(ranks, gnd_t, ks)\n\n    print('>> {}: mAP E: {}, M: {}, H: {}'.format(test_dataset, np.around(mapE*100, decimals=2), np.around(mapM*100, decimals=2), np.around(mapH*100, decimals=2)))\n    print('>> {}: mP@k{} E: {}, M: {}, H: {}'.format(test_dataset, np.array(ks), np.around(mprE*100, decimals=2), np.around(mprM*100, decimals=2), np.around(mprH*100, decimals=2)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2. Feature extraction","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"data_root = '/kaggle/input/roxfordparis'\nmodel_root = '/kaggle/input/baseline-landmark-retrieval-model/baseline_landmark_retrieval_model'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = tf.saved_model.load(model_root).signatures['serving_default']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This notebook uses cached feature extractions from `delg-baseline-roxfordparis-output` Kaggle dataset. Those features can be generated in `/kaggle/working` directory as well, by changing `should_extract_features` flag value to `True`. Be vary that feature extraction takes around ~2h with GPU accelerator on.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extract features for roxford5k & rparis6k datasets\n\nshould_extract_features = False\n\ndatasets = ['roxford5k', 'rparis6k']\nfor test_dataset in datasets:\n    if should_extract_features:\n        print('Processing dataset: {}'.format(test_dataset), flush=True)\n        cfg = configdataset(test_dataset, data_root)\n        extract_features(test_dataset, cfg, model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3. Model evaluation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Evaluate on roxford5k & rparis6k datasets\n\ndatasets = ['roxford5k', 'rparis6k']\nfor test_dataset in datasets:\n    print('Evaluating on dataset: {}'.format(test_dataset), flush=True)\n    cfg = configdataset(test_dataset, data_root)\n    run_evaluation(test_dataset, cfg, '/kaggle/input/delg-baseline-roxfordparis-output')\n    print()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The final mAP metrics are:\n- roxford5k - Easy: 90.92, Medium: 76.23, Hard: 55.54\n- rparis5k - Easy: 94.06, Medium: 87.25, Hard: 74.2\n\nIt's interesting to noticed that these metrics are higher than ones presented in [original DELG paper](https://arxiv.org/pdf/2001.05027.pdf) the baseline model is based on (check rows corresponding to DELG in columns marked as `ROxf` and `RPar`). That suggests that the baseline model is more powerful than the one presented in the paper, which means it is a considerate challenge to come up with model that improves baseline score on the Leaderboard.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![DELG figure](https://i.imgur.com/IMQZei7.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### BONUS: Testing model with concrete input","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The following code cells perform image retrieval of 5 most similar images using baseline model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_random_image(test_dataset, cfg):\n    query_image_id = random.choice(range(cfg['nq']))\n    im = pil_loader(cfg['im_fname'](cfg, query_image_id))\n    return query_image_id, im","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_inference(pil_image, model):\n    image_data = np.array(pil_image)\n    image_tensor = tf.convert_to_tensor(image_data)\n    return model(image_tensor)['global_descriptor'].numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dataset = 'roxford5k' # (roxford5k | rparis6k)\ncfg = configdataset(test_dataset, data_root)\n\nim_id, im = get_random_image(test_dataset, cfg)\nim_feats = run_inference(im, model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features_dir = '/kaggle/input/delg-baseline-roxfordparis-output'\n\nfeatures = loadmat(os.path.join(features_dir, '{}_delg_baseline.mat'.format(test_dataset)))\nX = features['X']\nsim = np.dot(X.T, im_feats)\nranks = np.argsort(-sim, axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(np.array(im))\nplt.title('Query Image')\nplt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig=plt.figure(figsize=(16, 12))\ncolumns = 5\nrows = 1\nfor i in range(1, columns*rows +1):\n    img = np.array(pil_loader(cfg['im_fname'](cfg, ranks[i-1])))\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(img)\n    plt.axis('off')\nprint('Five most similar images from query dataset:')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}