{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Vector Validation\n\nThis notebook continues on from the following. Please read them if you haven't already:\n\n* https://www.kaggle.com/code/prubyg/hotel-id-vector-extraction\n* https://www.kaggle.com/code/prubyg/hotel-id-vector-indexing/edit/run/92585356\n\nThis notebook attempted to validate the process of searching for similar images by KNN database search. TL;DR - it was a total flop, validation score 0.000. Let's see how we achieved this incredible result.","metadata":{}},{"cell_type":"markdown","source":"Like the extraction notebook, we will be extracting vectors from images - our validation set this time. For this, we need the timm library of models.","metadata":{}},{"cell_type":"code","source":"!pip install timm","metadata":{"execution":{"iopub.status.busy":"2022-04-11T22:12:11.788957Z","iopub.execute_input":"2022-04-11T22:12:11.789945Z","iopub.status.idle":"2022-04-11T22:12:23.429799Z","shell.execute_reply.started":"2022-04-11T22:12:11.789841Z","shell.execute_reply":"2022-04-11T22:12:23.429023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Import our dependencies...","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport random\n\nfrom annoy import AnnoyIndex\nimport pyarrow as pa\nfrom pyarrow.parquet import ParquetFile\n\nimport torch.nn as nn\nfrom torch.utils.data import DataLoader\nimport timm","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-04-11T22:12:23.432150Z","iopub.execute_input":"2022-04-11T22:12:23.432675Z","iopub.status.idle":"2022-04-11T22:12:25.815207Z","shell.execute_reply.started":"2022-04-11T22:12:23.432624Z","shell.execute_reply":"2022-04-11T22:12:25.814085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from PIL import Image as pil_image\nfrom tqdm import tqdm","metadata":{"execution":{"iopub.status.busy":"2022-04-11T22:12:25.816734Z","iopub.execute_input":"2022-04-11T22:12:25.817002Z","iopub.status.idle":"2022-04-11T22:12:25.822027Z","shell.execute_reply.started":"2022-04-11T22:12:25.816954Z","shell.execute_reply":"2022-04-11T22:12:25.820946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DEVICE = 'cpu' # We're dealing with small numbers of images, so CPU is fine.\nVECTOR_WIDTH = 320\nIMG_SIZE = 256\nVECTOR_FILE = '/kaggle/input/hotel-id-vector-extraction/vectors.parquet'\nINDEX_FILE = '/kaggle/input/hotel-id-vector-indexing/vectors.annoy'\n\nK = 10 # Number of nearest-neighbours to retrieve for each feature vector\n\nDATA_FOLDER = \"../input/hotelid-2022-train-images-256x256/\"\nIMAGE_FOLDER = DATA_FOLDER + \"images/\"","metadata":{"execution":{"iopub.status.busy":"2022-04-11T22:12:25.824930Z","iopub.execute_input":"2022-04-11T22:12:25.825495Z","iopub.status.idle":"2022-04-11T22:12:25.836185Z","shell.execute_reply.started":"2022-04-11T22:12:25.825446Z","shell.execute_reply":"2022-04-11T22:12:25.835084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We load our parquet file and annoy approximate-knn index.","metadata":{}},{"cell_type":"code","source":"vector_db = ParquetFile(VECTOR_FILE, read_dictionary=['file'])\nindex = AnnoyIndex(VECTOR_WIDTH, 'angular')\nindex.load(INDEX_FILE)\nN = index.get_n_items()","metadata":{"execution":{"iopub.status.busy":"2022-04-11T22:12:25.838010Z","iopub.execute_input":"2022-04-11T22:12:25.838415Z","iopub.status.idle":"2022-04-11T22:12:25.883659Z","shell.execute_reply.started":"2022-04-11T22:12:25.838365Z","shell.execute_reply":"2022-04-11T22:12:25.883011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This time, we completely read two columns of the Parquet file in to memory - the target and image file name. This is small enough to fit in memory easily.","metadata":{}},{"cell_type":"code","source":"meta = vector_db.read(['target', 'file'])\ntargets = meta['target']\nfiles = meta['file']\nassert(len(targets) == N)","metadata":{"execution":{"iopub.status.busy":"2022-04-11T22:12:25.885031Z","iopub.execute_input":"2022-04-11T22:12:25.885960Z","iopub.status.idle":"2022-04-11T22:12:26.128781Z","shell.execute_reply.started":"2022-04-11T22:12:25.885916Z","shell.execute_reply":"2022-04-11T22:12:26.128054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We load our validation set from the \"extraction\" notebook. This was emitted from that notebook to be sure we're using a consistent train/validation split, and not splitting differently at the end here. The validation set's images will not appear in the stored index.","metadata":{}},{"cell_type":"code","source":"val_df = pd.read_csv('/kaggle/input/hotel-id-vector-extraction/validation.csv')\n","metadata":{"execution":{"iopub.status.busy":"2022-04-11T22:12:26.129751Z","iopub.execute_input":"2022-04-11T22:12:26.129994Z","iopub.status.idle":"2022-04-11T22:12:26.151341Z","shell.execute_reply.started":"2022-04-11T22:12:26.129942Z","shell.execute_reply":"2022-04-11T22:12:26.150377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The dataset and vector extractor below are the same as in the vector extraction notebook.","metadata":{}},{"cell_type":"code","source":"class HotelTrainDataset:\n    def __init__(self, data, transform=None, data_path=\"train_images/\"):\n        self.data = data\n        self.data_path = data_path\n        self.transform = transform\n\n    def __len__(self):\n        return len(self.data)\n    \n    def __getitem__(self, idx):\n        record = self.data.iloc[idx]\n        image_path = self.data_path + record[\"image_id\"]\n        image = np.array(pil_image.open(image_path)).astype(np.uint8)\n\n        if self.transform:\n            transformed = self.transform(image=image)\n            image = transformed[\"image\"]\n        \n        hotel_id = record['hotel_id']\n        \n        return {\n            \"image\" : image,\n            \"image_id\": record[\"image_id\"],\n            \"target\" : hotel_id\n        }","metadata":{"execution":{"iopub.status.busy":"2022-04-11T22:12:26.152783Z","iopub.execute_input":"2022-04-11T22:12:26.153030Z","iopub.status.idle":"2022-04-11T22:12:26.159794Z","shell.execute_reply.started":"2022-04-11T22:12:26.152993Z","shell.execute_reply":"2022-04-11T22:12:26.159085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class VectorExtractor(nn.Module):\n    def __init__(self, backbone_name='efficientnet_b0', layer_to_extract=4):\n        super(VectorExtractor, self).__init__()\n        self.backbone_name = backbone_name\n        self.layer_to_extract = layer_to_extract\n        self.backbone = timm.create_model(self.backbone_name, pretrained=True, features_only=True)\n\n    def forward(self, x):\n        layers = self.backbone(x)\n        return layers[self.layer_to_extract]","metadata":{"execution":{"iopub.status.busy":"2022-04-11T22:12:26.160860Z","iopub.execute_input":"2022-04-11T22:12:26.161222Z","iopub.status.idle":"2022-04-11T22:12:26.176983Z","shell.execute_reply.started":"2022-04-11T22:12:26.161193Z","shell.execute_reply":"2022-04-11T22:12:26.176278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We do have a slightly more involved image augmentation here - we simulate the masks from the test dataset using the method from Michal's notebooks.","metadata":{}},{"cell_type":"code","source":"import albumentations as A\nimport albumentations.pytorch as APT\nimport cv2 \n\n# used for validation dataset - only occlusions\nval_transform = A.Compose([\n    A.CoarseDropout(p=1.0, max_holes=1, \n                    min_height=IMG_SIZE//4, max_height=IMG_SIZE//2,\n                    min_width=IMG_SIZE//4,  max_width=IMG_SIZE//2, \n                    fill_value=(255,0,0)),# simulating occlusions\n    A.ToFloat(),\n    APT.transforms.ToTensorV2(),\n])\n\n# no augmentations\nbase_transform = A.Compose([\n    A.ToFloat(),\n    APT.transforms.ToTensorV2(),\n])","metadata":{"execution":{"iopub.status.busy":"2022-04-11T22:12:26.178807Z","iopub.execute_input":"2022-04-11T22:12:26.179204Z","iopub.status.idle":"2022-04-11T22:12:28.459736Z","shell.execute_reply.started":"2022-04-11T22:12:26.179163Z","shell.execute_reply":"2022-04-11T22:12:28.459017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Construct dataset and loader...","metadata":{}},{"cell_type":"code","source":"val_dataset = HotelTrainDataset(val_df, val_transform, data_path=IMAGE_FOLDER)\nvalid_loader = DataLoader(val_dataset, num_workers=2, batch_size=1, shuffle=False)","metadata":{"execution":{"iopub.status.busy":"2022-04-11T22:12:28.460765Z","iopub.execute_input":"2022-04-11T22:12:28.461012Z","iopub.status.idle":"2022-04-11T22:12:28.466616Z","shell.execute_reply.started":"2022-04-11T22:12:28.460958Z","shell.execute_reply":"2022-04-11T22:12:28.465454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And we do the actual work. For each validation image, we run it through efficientnet_b0, extract the same layer we used for the indexed features, and find the nearest neighbours of that vector. We look up which hotel each corresponds to, and record a vote for that hotel.\n\nI considered weighting votes by vector distance here, but never got that far before abandoning this path. You'll see why.","metadata":{}},{"cell_type":"code","source":"extractor = VectorExtractor().to(DEVICE)\ncorrect = 0\nvalid_votes = 0\nn = 0\n\nbar = tqdm(valid_loader)\nfor record in bar:\n    # Per validation image...\n    votes = {}\n    \n    # Run it through our extractor\n    vectors = extractor(record['image'])\n    vectors = vectors.detach().cpu()\n    \n    # Iterate through the x and y dimensions of the result to get each feature vector\n    for x in range(vectors.shape[2]):\n        for y in range(vectors.shape[3]):\n            features = vectors[0, :, x, y].numpy()\n            # Use our annoy index to find the closest entries to our feature vector\n            knn = index.get_nns_by_vector(features, K)\n            for nn in knn:\n                # Find the corresponding hotel in our metadata\n                hotel = targets[nn].as_py()\n                # Record a vote for the hotel\n                if hotel in votes:\n                    votes[hotel] += 1\n                else:\n                    votes[hotel] = 1\n    \n    # Find the candidate with the most votes. There's probably a more Pythonic way of doing this, but I didn't find it quickly so defaulted to C-style algorithms.\n    candidate = None\n    max_votes = 0\n    for hotel, vs in votes.items(): # Corrected bug here, thank you Michal\n        if vs > max_votes:\n            candidate = hotel\n            max_votes = vs\n    \n    # Check which hotel it actually was... were we right?\n    target = int(record['target'][0])\n    if candidate == target:\n        correct += 1\n    \n    # Count the total number of votes cast for the right target. The Result Will SHOCK YOU!\n    valid_votes += votes.get(target,0)\n    n += 1\n    \n    bar.set_postfix(Correct=correct, ValidVotes=float(valid_votes)/float(n))","metadata":{"execution":{"iopub.status.busy":"2022-04-11T22:12:28.467800Z","iopub.execute_input":"2022-04-11T22:12:28.468155Z","iopub.status.idle":"2022-04-11T22:20:56.731361Z","shell.execute_reply.started":"2022-04-11T22:12:28.468125Z","shell.execute_reply":"2022-04-11T22:20:56.730138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Wow, ten out of 3116 now correct - that's 0.003. Infinity times improvement.\n\nSo what went on here? Clearly the features vector extracted is not a dense representation of \"lamp\", \"bed post\", etc. Maybe we could have luck using this approach to augment another method, but it's not going to work with this pre-trained model.","metadata":{}}]}