{"cells":[{"metadata":{},"cell_type":"markdown","source":"<h1><center>Google Landmark Recognition 2020</center></h1>"},{"metadata":{},"cell_type":"markdown","source":"# About the Kernel\n\n1. First version      - EDA\n2. 3rd to 5th Version - Model Building\n3. 6th Version - Some Bug fixing related to saving image paths to the training and test datasets"},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nThis is the third Landmark Recognition competition with a new set of test images.\n\nThis technology (predicting landmarks labels) directly from image pixels, will help people better understand and organize their photo collections. \n\n<div class=\"alert alert-block alert-info\">\n<b>Biggest challenge in this competition:</b> \n\nThis seems to be an extremely challenging competition because it contains a much larger number of classes (there are more than 81K classes in this challenge), and the number of training examples per class may not be very large. </div>\n\n\n<div class=\"alert alert-block alert-info\">\n<b>Another Challenge:</b> \n\nFor quite a lot of classes, there are only 2 images provided in the training set and for most of the classes training samples are less than 100 for that particular class, This means training dataset is highly imbalanced.</div>"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom tqdm import tqdm_notebook as tqdm\n\nimport glob\nimport cv2\nimport os\n\nfrom colorama import Fore, Back, Style\n\n# Setting color palette.\nplt.rcdefaults()\nplt.style.use('dark_background')\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Let's declare PATH variables"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Assigning paths to variables\nINPUT_PATH = os.path.join('..', 'input')\nDATASET_PATH = os.path.join(INPUT_PATH, 'landmark-recognition-2020')\nTRAIN_IMAGE_PATH = os.path.join(DATASET_PATH, 'train')\nTEST_IMAGE_PATH = os.path.join(DATASET_PATH, 'test')\nTRAIN_CSV_PATH = os.path.join(DATASET_PATH, 'train.csv')\nSUBMISSION_CSV_PATH = os.path.join(DATASET_PATH, 'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load CSV files 📃"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(TRAIN_CSV_PATH)\nprint(\"training dataset has {} rows and {} columns\".format(train.shape[0],train.shape[1]))\n\nsubmission = pd.read_csv(SUBMISSION_CSV_PATH)\nprint(\"submission dataset has {} rows and {} columns \\n\".format(submission.shape[0],submission.shape[1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# FOLDER STRUCTURE\n\n<div class=\"alert alert-block alert-info\">\nWe have Training Data saved in folder ranging `0 to 9` and `a to f`, We will see one example below to understand the folder structure\n</div>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# understand folder structure\nprint(Fore.YELLOW + \"If you want to access image a40d00dc4fcc3a10, you should traverse as shown below:\\n\",Style.RESET_ALL)\n\nprint(Fore.GREEN + f\"Image name: {train['id'].iloc[9]}\\n\",Style.RESET_ALL)\n\nprint(Fore.BLUE + f\"First folder to look inside: {train['id'][9][0]}\")\nprint(Fore.BLUE + f\"Second folder to look inside: {train['id'][9][1]}\")\nprint(Fore.BLUE + f\"Second folder to look inside: {train['id'][9][2]}\",Style.RESET_ALL)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build dictionary to store image paths & labels"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(Fore.BLUE + f\"{'---'*20} \\n Mapping for Training Data \\n {'---'*20}\")\ndata_label_dict = {'image': [], 'target': []}\nfor i in tqdm(range(train.shape[0])):\n    data_label_dict['image'].append(\n        TRAIN_IMAGE_PATH + '/' +\n        train['id'][i][0] + '/' + \n        train['id'][i][1]+ '/' +\n        train['id'][i][2]+ '/' +\n        train['id'][i] + \".jpg\")\n    data_label_dict['target'].append(\n        train['landmark_id'][i])\n\n#Convert to dataframe\ntrain_pathlabel = pd.DataFrame(data_label_dict)\nprint(train_pathlabel.head())\n    \nprint(Fore.BLUE + f\"{'---'*20} \\n Mapping for Test Data \\n {'---'*20}\",Style.RESET_ALL)\ndata_label_dict = {'image': []}\nfor i in tqdm(range(submission.shape[0])):\n    data_label_dict['image'].append(\n        TEST_IMAGE_PATH + '/' +\n        submission['id'][i][0] + '/' + \n        submission['id'][i][1]+ '/' +\n        submission['id'][i][2]+ '/' +\n        submission['id'][i] + \".jpg\")\n\ntest_pathlabel = pd.DataFrame(data_label_dict)\nprint(test_pathlabel.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# list of unique landmark ids\ntrain.landmark_id.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# count of unique landmark_ids\nprint(\"There are\", train.landmark_id.nunique(), \"landmarks in the training dataset\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# each class count-wise\ntrain.landmark_id.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Check File Sizes of first 10 files"},{"metadata":{"trusted":true},"cell_type":"code","source":"files = train_pathlabel.image[:10]\nprint(Fore.BLUE + \"Shape of files from training dataset\",Style.RESET_ALL)\nfor i in range(10):\n    im = cv2.imread(files[i])\n    print(im.shape)\n\n\nprint(\"------------------------------------\")    \nprint(\"------------------------------------\")    \nprint(\"------------------------------------\")    \n\nfiles = test_pathlabel.image[:10]\nprint(Fore.BLUE + \"Shape of files from test dataset\",Style.RESET_ALL)\nfor i in range(10):\n    im = cv2.imread(files[i])\n    print(im.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Class Distribution Analysis"},{"metadata":{},"cell_type":"markdown","source":"### Density plot for class distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (12, 8))\n\nsns.kdeplot(train['landmark_id'], color=\"yellow\",shade=True)\nplt.xlabel(\"LandMark IDs\")\nplt.ylabel(\"Probability Density\")\nplt.title('Class Distribution - Density plot')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Top 6 Class Categories"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize = (12,8))\n\ncount = train.landmark_id.value_counts().sort_values(ascending=False)[:6]\n\nsns.countplot(x=train.landmark_id,\n             order = train.landmark_id.value_counts().sort_values(ascending=False).iloc[:6].index)\n\nplt.xlabel(\"LandMark Id\")\nplt.ylabel(\"Frequency\")\nplt.title(\"Top 6 Classes in the Dataset\")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-block alert-info\">\n    \n* Landmark id '138982' has more than 6000 images\n* Next Top 5 clasess in this table have less than 2500 images\n</div>"},{"metadata":{},"cell_type":"markdown","source":"### Let's check out images from the top 6 classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"top6 = train.landmark_id.value_counts().sort_values(ascending=False)[:6].index\n\nimages = []\n\nfor i in range(6):\n    img=cv2.imread(train_pathlabel[train_pathlabel.target == top6[i]]['image'].values[1])   \n    images.append(img)\n\nf, ax = plt.subplots(3,2, figsize=(20,15))\nfor i, img in enumerate(images):        \n        ax[i//2, i%2].imshow(img)\n        ax[i//2, i%2].axis('off')\n       ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Top 50 Class Categories"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize = (12,8))\n\ncount = train.landmark_id.value_counts().sort_values(ascending=False)[:50]\n\nsns.countplot(x=train.landmark_id,\n             order = train.landmark_id.value_counts().sort_values(ascending=False).iloc[:50].index)\n\nplt.xticks(rotation = 90)\n\nplt.xlabel(\"LandMark Id\")\nplt.ylabel(\"Frequency\")\nplt.title(\"Top 50 Classes in the Dataset\")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-block alert-info\">\n    \n* Landmark id '138982' has more than 6000 images. \n* Next Top 5 clasess in this table has less than 2500 images\n* Rest of the classes has less than 1000 samples in the training dataset\n\n</div>"},{"metadata":{},"cell_type":"markdown","source":"### Let's check out images from the top 50 classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"top50 = train.landmark_id.value_counts().sort_values(ascending=False).index[:50]\n\nimages = []\n\nfor i in range(50):\n    img=cv2.imread(train_pathlabel[train_pathlabel.target == top50[i]]['image'].values[1])   \n    images.append(img)\n\nf, ax = plt.subplots(10,5, figsize=(20,15))\nfor i, img in enumerate(images):        \n        ax[i//5, i%5].imshow(img)\n        ax[i//5, i%5].axis('off')\n       ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Bottom 6 Class Categories"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize = (12,8))\n\ncount = train.landmark_id.value_counts()[-6:]\n\nsns.countplot(x=train.landmark_id,\n             order = train_pathlabel.target.value_counts().iloc[-6:].index)\n\nplt.xlabel(\"LandMark Id\")\nplt.ylabel(\"Frequency\")\nplt.title(\"Bottom 6 Classes in the Dataset\")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-block alert-info\">    \nJust 2 images per class for the bottom 6 classes\n</div>"},{"metadata":{},"cell_type":"markdown","source":"### Let's check out images from the bottom 6 classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"bottom6 = train.landmark_id.value_counts()[-6:].index\n\nimages = []\n\nfor i in range(6):\n    img=cv2.imread(train_pathlabel[train_pathlabel.target == bottom6[i]]['image'].values[1])   \n    images.append(img)\n\nf, ax = plt.subplots(3,2, figsize=(20,15))\nfor i, img in enumerate(images):        \n        ax[i//2, i%2].imshow(img)\n        ax[i//2, i%2].axis('off')\n       ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-block alert-info\">\n    \n<b> Observations from the whole analysis done above:</b>\n* There are 81313 unique landmark_ids\n* There is only one landmark which has more than 6000 images\n* Number of images per landmark_id ranges from 2 to 6272.\n\n</div>"},{"metadata":{},"cell_type":"markdown","source":"### We will now check few images from the top 5 classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"top5 = train.landmark_id.value_counts().sort_values(ascending=False).index[:5]\nfor i in range(5):\n    images = []      \n    for j in range(12):\n        img=cv2.imread(train_pathlabel[train_pathlabel.target == top5[i]]['image'].values[j])   \n        images.append(img)           \n    f, ax = plt.subplots(3,4,figsize=(20,15))\n    for k, img in enumerate(images):        \n        ax[k//4, k%4].imshow(img)\n        ax[k//4, k%4].axis('off')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Some more Images from training set"},{"metadata":{"trusted":true},"cell_type":"code","source":"files = train_pathlabel.image[11:23]\n\nimages = []\n\nfor i in range(11,23):    \n    img=cv2.imread(files[i])   \n    images.append(img)\nf, ax = plt.subplots(3,4, figsize=(20,15))\nfor i, img in enumerate(images):\n        ax[i//4, i%4].imshow(img)\n        ax[i//4, i%4].axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Few Images from test dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"files = test_pathlabel.image[11:23]\nimages = []\n\nfor i in range(11,23):\n    img=cv2.imread(files[i])   \n    images.append(img)\nf, ax = plt.subplots(3,4, figsize=(20,15))\nfor i, img in enumerate(images):\n        ax[i//4, i%4].imshow(img)\n        ax[i//4, i%4].axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Histogram of images from training dataset"},{"metadata":{},"cell_type":"markdown","source":"### 1. Grayscale Image\n\n* We will loaded the grayscale images here & generated its histogram \n\n* Since the images are stored in the form of a 2D ordered matrix we converted it to a 1D array using the ravel() method"},{"metadata":{"trusted":true},"cell_type":"code","source":"files = train_pathlabel.image[:4]\n\nfig = plt.figure(figsize = (20,9))\n\nfor i in range(4):\n    img=cv2.imread(files[i])   \n    plt.subplot(2,2,i+1)\n    plt.hist(img.ravel(), bins = 256,color = 'gold')\n    \nplt.suptitle(\"Histogram for Grayscale Images\",fontsize = 25)    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2. Cumulative histogram\nThe cumulative histogram is a special histogram that can be derived from the normal histogram. \nWe find the counts of each intensity value from 0–255 and then add each subsequent counts"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize = (20,9))\n\nfor i in range(4):\n    img=cv2.imread(files[i])   \n    plt.subplot(2,2,i+1)\n    plt.hist(img.ravel(), bins = 256,color = 'magenta',cumulative = True)\n\nplt.suptitle(\"Cumulative Histogram for Grayscale Images\",fontsize = 25)    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3. Grayscale Image - With bins = 8\nUsually, the range of intensity values of images is from [0–255] in 8bits representation(2⁸). \n\nBut images can be also represented using 2¹⁶, 2³² bits and so on. In such cases the intensity range is high and it is hard to represent each intensity value in a histogram.\n\nWe use binning to overcome the above problem. Here we quantize the range into several buckets. For example,\n\nIf we quantize 0-255 into 8 bins, here our bins will be: 0-31, 32-63, 64-95, 96-127, 128-159, 160-191, 192-223, 224-255"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize = (20,9))\n\nfor i in range(4):\n    img=cv2.imread(files[i])   \n    plt.subplot(2,2,i+1)\n    plt.hist(img.ravel(), bins = 8, color = \"coral\")\n\nplt.suptitle(\"Cumulative Histogram for Grayscale Images - Bin Size = 8\",fontsize = 25)    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4. Color Image\n\n* In color images, we have 3 color channels representing RGB. In Combined Color Histogram the intensity count is the sum of all three color channels."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize = (20,9))\n\nfor i in range(4):\n    img=cv2.imread(files[i])   \n    plt.subplot(2,2,i+1)\n    plt.hist(img.ravel(), bins = 256, color = 'orange', )\n    plt.hist(img[:, :, 0].ravel(), bins = 256, color = 'red', alpha = 0.5)\n    plt.hist(img[:, :, 1].ravel(), bins = 256, color = 'Green', alpha = 0.5)\n    plt.hist(img[:, :, 2].ravel(), bins = 256, color = 'Blue', alpha = 0.5)\n    plt.xlabel('Intensity Value')\n    plt.ylabel('Count')\n    plt.legend(['Total', 'Red_Channel', 'Green_Channel', 'Blue_Channel'])\n\nplt.suptitle(\"Color Histograms\",fontsize = 25)    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Building"},{"metadata":{"trusted":true},"cell_type":"code","source":"import copy\nimport csv\nimport gc\nimport operator\nimport os\nimport pathlib\nimport shutil\n\nimport numpy as np\nimport PIL\nimport pydegensac\nfrom scipy import spatial\nimport tensorflow as tf","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Dataset parameters:\nINPUT_DIR = os.path.join('..', 'input')\n\nDATASET_DIR = os.path.join(INPUT_DIR, 'landmark-recognition-2020')\nTEST_IMAGE_DIR = os.path.join(DATASET_DIR, 'test')\nTRAIN_IMAGE_DIR = os.path.join(DATASET_DIR, 'train')\nTRAIN_LABELMAP_PATH = os.path.join(DATASET_DIR, 'train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# DEBUGGING PARAMS:\nNUM_PUBLIC_TRAIN_IMAGES = 1580470 # Used to detect if in session or re-run.\nMAX_NUM_EMBEDDINGS = -1  # Set to > 1 to subsample dataset while debugging.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# Retrieval & re-ranking parameters:\nNUM_TO_RERANK = 3\nTOP_K = 3 # Number of retrieved images used to make prediction for a test image.\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# RANSAC parameters:\nMAX_INLIER_SCORE = 35\nMAX_REPROJECTION_ERROR = 7.0\nMAX_RANSAC_ITERATIONS = 8500000\nHOMOGRAPHY_CONFIDENCE = 0.99","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# DELG model:\nSAVED_MODEL_DIR = '../input/delg-saved-models/local_and_global'\nDELG_MODEL = tf.saved_model.load(SAVED_MODEL_DIR)\nDELG_IMAGE_SCALES_TENSOR = tf.convert_to_tensor([0.70710677, 1.0, 1.4142135])\nDELG_SCORE_THRESHOLD_TENSOR = tf.constant(175.)\nDELG_INPUT_TENSOR_NAMES = [\n    'input_image:0', 'input_scales:0', 'input_abs_thres:0'\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Global feature extraction:\nNUM_EMBEDDING_DIMENSIONS = 2048\nGLOBAL_FEATURE_EXTRACTION_FN = DELG_MODEL.prune(DELG_INPUT_TENSOR_NAMES,\n                                                ['global_descriptors:0'])\n\n# Local feature extraction:\nLOCAL_FEATURE_NUM_TENSOR = tf.constant(1000)\nLOCAL_FEATURE_EXTRACTION_FN = DELG_MODEL.prune(\n    DELG_INPUT_TENSOR_NAMES + ['input_max_feature_num:0'],\n    ['boxes:0', 'features:0'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def to_hex(image_id) -> str:\n  return '{0:0{1}x}'.format(image_id, 16)\n\n\ndef get_image_path(subset, image_id):\n  name = to_hex(image_id)\n  return os.path.join(DATASET_DIR, subset, name[0], name[1], name[2],\n                      '{}.jpg'.format(name))\n\n\ndef load_image_tensor(image_path):\n  return tf.convert_to_tensor(\n      np.array(PIL.Image.open(image_path).convert('RGB')))\n\n\ndef extract_global_features(image_root_dir):\n  \"\"\"Extracts embeddings for all the images in given `image_root_dir`.\"\"\"\n\n  image_paths = [x for x in pathlib.Path(image_root_dir).rglob('*.jpg')]\n\n  num_embeddings = len(image_paths)\n  if MAX_NUM_EMBEDDINGS > 0:\n    num_embeddings = min(MAX_NUM_EMBEDDINGS, num_embeddings)\n\n  ids = num_embeddings * [None]\n  embeddings = np.empty((num_embeddings, NUM_EMBEDDING_DIMENSIONS))\n\n  for i, image_path in enumerate(image_paths):\n    if i >= num_embeddings:\n      break\n\n    ids[i] = int(image_path.name.split('.')[0], 16)\n    image_tensor = load_image_tensor(image_path)\n    features = GLOBAL_FEATURE_EXTRACTION_FN(image_tensor,\n                                            DELG_IMAGE_SCALES_TENSOR,\n                                            DELG_SCORE_THRESHOLD_TENSOR)\n    embeddings[i, :] = tf.nn.l2_normalize(\n        tf.reduce_sum(features[0], axis=0, name='sum_pooling'),\n        axis=0,\n        name='final_l2_normalization').numpy()\n\n  return ids, embeddings\n\n\ndef extract_local_features(image_path):\n  \"\"\"Extracts local features for the given `image_path`.\"\"\"\n\n  image_tensor = load_image_tensor(image_path)\n\n  features = LOCAL_FEATURE_EXTRACTION_FN(image_tensor, DELG_IMAGE_SCALES_TENSOR,\n                                         DELG_SCORE_THRESHOLD_TENSOR,\n                                         LOCAL_FEATURE_NUM_TENSOR)\n\n  # Shape: (N, 2)\n  keypoints = tf.divide(\n      tf.add(\n          tf.gather(features[0], [0, 1], axis=1),\n          tf.gather(features[0], [2, 3], axis=1)), 2.0).numpy()\n\n  # Shape: (N, 128)\n  descriptors = tf.nn.l2_normalize(\n      features[1], axis=1, name='l2_normalization').numpy()\n\n  return keypoints, descriptors\n\n\ndef get_putative_matching_keypoints(test_keypoints,\n                                    test_descriptors,\n                                    train_keypoints,\n                                    train_descriptors,\n                                    max_distance=0.9):\n  \"\"\"Finds matches from `test_descriptors` to KD-tree of `train_descriptors`.\"\"\"\n\n  train_descriptor_tree = spatial.cKDTree(train_descriptors)\n  _, matches = train_descriptor_tree.query(\n      test_descriptors, distance_upper_bound=max_distance)\n\n  test_kp_count = test_keypoints.shape[0]\n  train_kp_count = train_keypoints.shape[0]\n\n  test_matching_keypoints = np.array([\n      test_keypoints[i,]\n      for i in range(test_kp_count)\n      if matches[i] != train_kp_count\n  ])\n  train_matching_keypoints = np.array([\n      train_keypoints[matches[i],]\n      for i in range(test_kp_count)\n      if matches[i] != train_kp_count\n  ])\n\n  return test_matching_keypoints, train_matching_keypoints\n\n\ndef get_num_inliers(test_keypoints, test_descriptors, train_keypoints,\n                    train_descriptors):\n  \"\"\"Returns the number of RANSAC inliers.\"\"\"\n\n  test_match_kp, train_match_kp = get_putative_matching_keypoints(\n      test_keypoints, test_descriptors, train_keypoints, train_descriptors)\n\n  if test_match_kp.shape[\n      0] <= 4:  # Min keypoints supported by `pydegensac.findHomography()`\n    return 0\n\n  try:\n    _, mask = pydegensac.findHomography(test_match_kp, train_match_kp,\n                                        MAX_REPROJECTION_ERROR,\n                                        HOMOGRAPHY_CONFIDENCE,\n                                        MAX_RANSAC_ITERATIONS)\n  except np.linalg.LinAlgError:  # When det(H)=0, can't invert matrix.\n    return 0\n\n  return int(copy.deepcopy(mask).astype(np.float32).sum())\n\n\ndef get_total_score(num_inliers, global_score):\n  local_score = min(num_inliers, MAX_INLIER_SCORE) / MAX_INLIER_SCORE\n  return local_score + global_score\n\n\ndef rescore_and_rerank_by_num_inliers(test_image_id,\n                                      train_ids_labels_and_scores):\n  \"\"\"Returns rescored and sorted training images by local feature extraction.\"\"\"\n\n  test_image_path = get_image_path('test', test_image_id)\n  test_keypoints, test_descriptors = extract_local_features(test_image_path)\n\n  for i in range(len(train_ids_labels_and_scores)):\n    train_image_id, label, global_score = train_ids_labels_and_scores[i]\n\n    train_image_path = get_image_path('train', train_image_id)\n    train_keypoints, train_descriptors = extract_local_features(\n        train_image_path)\n\n    num_inliers = get_num_inliers(test_keypoints, test_descriptors,\n                                  train_keypoints, train_descriptors)\n    total_score = get_total_score(num_inliers, global_score)\n    train_ids_labels_and_scores[i] = (train_image_id, label, total_score)\n\n  train_ids_labels_and_scores.sort(key=lambda x: x[2], reverse=True)\n\n  return train_ids_labels_and_scores\n\n\ndef load_labelmap():\n  with open(TRAIN_LABELMAP_PATH, mode='r') as csv_file:\n    csv_reader = csv.DictReader(csv_file)\n    labelmap = {row['id']: row['landmark_id'] for row in csv_reader}\n\n  return labelmap\n\n\ndef get_prediction_map(test_ids, train_ids_labels_and_scores):\n  \"\"\"Makes dict from test ids and ranked training ids, labels, scores.\"\"\"\n\n  prediction_map = dict()\n\n  for test_index, test_id in enumerate(test_ids):\n    hex_test_id = to_hex(test_id)\n\n    aggregate_scores = {}\n    for _, label, score in train_ids_labels_and_scores[test_index][:TOP_K]:\n      if label not in aggregate_scores:\n        aggregate_scores[label] = 0\n      aggregate_scores[label] += score\n\n    label, score = max(aggregate_scores.items(), key=operator.itemgetter(1))\n\n    prediction_map[hex_test_id] = {'score': score, 'class': label}\n\n  return prediction_map\n\n\ndef get_predictions(labelmap):\n  \"\"\"Gets predictions using embedding similarity and local feature reranking.\"\"\"\n\n  test_ids, test_embeddings = extract_global_features(TEST_IMAGE_DIR)\n\n  train_ids, train_embeddings = extract_global_features(TRAIN_IMAGE_DIR)\n\n  train_ids_labels_and_scores = [None] * test_embeddings.shape[0]\n\n  # Using (slow) for-loop, as distance matrix doesn't fit in memory.\n  for test_index in range(test_embeddings.shape[0]):\n    distances = spatial.distance.cdist(\n        test_embeddings[np.newaxis, test_index, :], train_embeddings,\n        'cosine')[0]\n    partition = np.argpartition(distances, NUM_TO_RERANK)[:NUM_TO_RERANK]\n\n    nearest = sorted([(train_ids[p], distances[p]) for p in partition],\n                     key=lambda x: x[1])\n\n    train_ids_labels_and_scores[test_index] = [\n        (train_id, labelmap[to_hex(train_id)], 1. - cosine_distance)\n        for train_id, cosine_distance in nearest\n    ]\n\n  del test_embeddings\n  del train_embeddings\n  del labelmap\n  gc.collect()\n\n  pre_verification_predictions = get_prediction_map(\n      test_ids, train_ids_labels_and_scores)\n\n#  return None, pre_verification_predictions\n\n  for test_index, test_id in enumerate(test_ids):\n    train_ids_labels_and_scores[test_index] = rescore_and_rerank_by_num_inliers(\n        test_id, train_ids_labels_and_scores[test_index])\n\n  post_verification_predictions = get_prediction_map(\n      test_ids, train_ids_labels_and_scores)\n\n  return pre_verification_predictions, post_verification_predictions\n\n\ndef save_submission_csv(predictions=None):\n  \"\"\"Saves optional `predictions` as submission.csv.\n\n  The csv has columns {id, landmarks}. The landmarks column is a string\n  containing the label and score for the id, separated by a ws delimeter.\n\n  If `predictions` is `None` (default), submission.csv is copied from\n  sample_submission.csv in `IMAGE_DIR`.\n\n  Args:\n    predictions: Optional dict of image ids to dicts with keys {class, score}.\n  \"\"\"\n\n  if predictions is None:\n    # Dummy submission!\n    shutil.copyfile(\n        os.path.join(DATASET_DIR, 'sample_submission.csv'), 'submission.csv')\n    return\n\n  with open('submission.csv', 'w') as submission_csv:\n    csv_writer = csv.DictWriter(submission_csv, fieldnames=['id', 'landmarks'])\n    csv_writer.writeheader()\n    for image_id, prediction in predictions.items():\n      label = prediction['class']\n      score = prediction['score']\n      csv_writer.writerow({'id': image_id, 'landmarks': f'{label} {score}'})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def main():\n  labelmap = load_labelmap()\n  num_training_images = len(labelmap.keys())\n  print(f'Found {num_training_images} training images.')\n\n  if num_training_images == NUM_PUBLIC_TRAIN_IMAGES:\n    print(\n        f'Found {NUM_PUBLIC_TRAIN_IMAGES} training images. Copying sample submission.'\n    )\n    save_submission_csv()\n    return\n\n  _, post_verification_predictions = get_predictions(labelmap)\n  save_submission_csv(post_verification_predictions)\n\nmain()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}