{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Intro\nGoal of this notebook is to explore embeddings generated from [Hotel-ID starter - similarity- training](https://www.kaggle.com/code/michaln/hotel-id-starter-similarity-training) notebook and try out different methods to identify similar images.\n\n## Embeddings\nTo compare images we can use model to generate embeddings as their representation and then calculate distance/similarity between images to search for the most similar one.\n\nWe can use pretrained model without the last classification layer and add two linear layers. Features from pretrianed CNN will be used as input for embedding layer and result of embedding layer will be used for classification layer. Model will then output embeddings and predicted class. We can use a class prediction to calculate loss and train the model further and embeddings to search for similar images. Embeddings should contain enough information to predict correct class so they should be good representations of the image. \n\n![Embedding model](https://github.com/michal-nahlik/kaggle-hotel-id-2022/raw/master/doc/img/embedding_model.png)\n\n## Data\nThis notebook uses preprocessed images that were resized and padded to 256x256 pixel.\n\nUsed dataset: [Hotel-ID 2022 train images 256x256](https://www.kaggle.com/datasets/michaln/hotelid-2022-train-images-256x256) created by [Hotel-ID - image preprocessing - 256x256](https://www.kaggle.com/code/michaln/hotel-id-image-preprocessing-256x256) notebook.","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{"id":"MyC4gTwZ3MKJ","papermill":{"duration":0.020375,"end_time":"2022-03-23T20:08:58.820405","exception":false,"start_time":"2022-03-23T20:08:58.80003","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install timm","metadata":{"papermill":{"duration":18.268618,"end_time":"2022-03-23T20:09:17.10966","exception":false,"start_time":"2022-03-23T20:08:58.841042","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:51:05.608182Z","iopub.execute_input":"2022-04-12T16:51:05.608784Z","iopub.status.idle":"2022-04-12T16:51:15.555141Z","shell.execute_reply.started":"2022-04-12T16:51:05.608690Z","shell.execute_reply":"2022-04-12T16:51:15.554365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport random\nimport os","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","id":"u0Bz2ktn2_ap","papermill":{"duration":0.032275,"end_time":"2022-03-23T20:09:17.168351","exception":false,"start_time":"2022-03-23T20:09:17.136076","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:51:15.557164Z","iopub.execute_input":"2022-04-12T16:51:15.557430Z","iopub.status.idle":"2022-04-12T16:51:15.562920Z","shell.execute_reply.started":"2022-04-12T16:51:15.557388Z","shell.execute_reply":"2022-04-12T16:51:15.562272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from PIL import Image as pil_image\nfrom tqdm import tqdm\n\nimport matplotlib\nimport matplotlib.pyplot as plt\n\nimport plotly.graph_objects as go\nimport plotly.express as px\nfrom plotly.subplots import make_subplots","metadata":{"id":"tOszKuxt3PXn","papermill":{"duration":3.425867,"end_time":"2022-03-23T20:09:20.619496","exception":false,"start_time":"2022-03-23T20:09:17.193629","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:51:15.564032Z","iopub.execute_input":"2022-04-12T16:51:15.564275Z","iopub.status.idle":"2022-04-12T16:51:19.142070Z","shell.execute_reply.started":"2022-04-12T16:51:15.564243Z","shell.execute_reply":"2022-04-12T16:51:19.141288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nfrom torch.utils.data import DataLoader\n\nimport timm\n\nfrom sklearn.metrics.pairwise import cosine_similarity","metadata":{"id":"uQE7wYFR3QxV","papermill":{"duration":2.460075,"end_time":"2022-03-23T20:09:23.106294","exception":false,"start_time":"2022-03-23T20:09:20.646219","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:51:19.146433Z","iopub.execute_input":"2022-04-12T16:51:19.148569Z","iopub.status.idle":"2022-04-12T16:51:21.460072Z","shell.execute_reply.started":"2022-04-12T16:51:19.148528Z","shell.execute_reply":"2022-04-12T16:51:21.459255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Global","metadata":{"id":"tirOg6jm3aIB","papermill":{"duration":0.025747,"end_time":"2022-03-23T20:09:23.159882","exception":false,"start_time":"2022-03-23T20:09:23.134135","status":"completed"},"tags":[]}},{"cell_type":"code","source":"IMG_SIZE = 256\nSEED = 42\nN_MATCHES = 5\n\nPROJECT_FOLDER = \"../input/hotel-id-to-combat-human-trafficking-2022-fgvc9/\"\nDATA_FOLDER = \"../input/hotelid-2022-train-images-256x256/\"\nIMAGE_FOLDER = DATA_FOLDER + \"images/\"\nOUTPUT_FOLDER = \"\"","metadata":{"id":"DV7qHDuYGoJH","papermill":{"duration":0.086034,"end_time":"2022-03-23T20:09:23.2714","exception":false,"start_time":"2022-03-23T20:09:23.185366","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-04-12T16:51:21.462794Z","iopub.execute_input":"2022-04-12T16:51:21.463141Z","iopub.status.idle":"2022-04-12T16:51:21.468701Z","shell.execute_reply.started":"2022-04-12T16:51:21.463103Z","shell.execute_reply":"2022-04-12T16:51:21.467187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(os.listdir(PROJECT_FOLDER))","metadata":{"id":"TB9CXg8U3bbQ","papermill":{"duration":0.032796,"end_time":"2022-03-23T20:09:23.329909","exception":false,"start_time":"2022-03-23T20:09:23.297113","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-04-12T16:51:21.470073Z","iopub.execute_input":"2022-04-12T16:51:21.470353Z","iopub.status.idle":"2022-04-12T16:51:21.482406Z","shell.execute_reply.started":"2022-04-12T16:51:21.470307Z","shell.execute_reply":"2022-04-12T16:51:21.481603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True","metadata":{"id":"csp2OMgo2_ar","papermill":{"duration":0.03307,"end_time":"2022-03-23T20:09:23.38914","exception":false,"start_time":"2022-03-23T20:09:23.35607","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:51:21.483908Z","iopub.execute_input":"2022-04-12T16:51:21.484310Z","iopub.status.idle":"2022-04-12T16:51:21.491570Z","shell.execute_reply.started":"2022-04-12T16:51:21.484271Z","shell.execute_reply":"2022-04-12T16:51:21.490818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset and transformations","metadata":{"id":"8V_xuoN73lON","papermill":{"duration":0.025486,"end_time":"2022-03-23T20:09:23.440614","exception":false,"start_time":"2022-03-23T20:09:23.415128","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Coarse dropout with fill_value=(255,0,0) (full red channel) is used to simulate the occlussions like the one in test dataset. \n```python\nA.CoarseDropout(p=1., max_holes=1, \n min_height=IMG_SIZE//4, max_height=IMG_SIZE//2,\n min_width=IMG_SIZE//4, max_width=IMG_SIZE//2, \n fill_value=(255,0,0))\n```","metadata":{}},{"cell_type":"code","source":"import albumentations as A\nimport albumentations.pytorch as APT\nimport cv2 \n\n# used for validation dataset - only occlusions\nval_transform = A.Compose([\n A.CoarseDropout(p=1., max_holes=1, \n min_height=IMG_SIZE//4, max_height=IMG_SIZE//2,\n min_width=IMG_SIZE//4, max_width=IMG_SIZE//2, \n fill_value=(255,0,0)),# simulating occlusions\n A.ToFloat(),\n APT.transforms.ToTensorV2(),\n])\n\n# no augmentations\nbase_transform = A.Compose([\n A.ToFloat(),\n APT.transforms.ToTensorV2(),\n])","metadata":{"id":"8ucWZHeG2_as","papermill":{"duration":0.919086,"end_time":"2022-03-23T20:09:24.385553","exception":false,"start_time":"2022-03-23T20:09:23.466467","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-04-12T16:51:21.493061Z","iopub.execute_input":"2022-04-12T16:51:21.493326Z","iopub.status.idle":"2022-04-12T16:51:22.415772Z","shell.execute_reply.started":"2022-04-12T16:51:21.493288Z","shell.execute_reply":"2022-04-12T16:51:22.415014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class HotelTrainDataset:\n def __init__(self, data, transform=None, data_path=\"train_images/\"):\n self.data = data\n self.data_path = data_path\n self.transform = transform\n\n def __len__(self):\n return len(self.data)\n \n def __getitem__(self, idx):\n record = self.data.iloc[idx]\n image_path = self.data_path + record[\"image_id\"]\n image = np.array(pil_image.open(image_path)).astype(np.uint8)\n\n if self.transform:\n transformed = self.transform(image=image)\n image = transformed[\"image\"]\n \n return {\n \"image\" : image,\n \"target\" : record['hotel_id_code'],\n }","metadata":{"id":"EiLYsfKq2_at","papermill":{"duration":0.035446,"end_time":"2022-03-23T20:09:24.447084","exception":false,"start_time":"2022-03-23T20:09:24.411638","status":"completed"},"tags":[],"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:51:22.417232Z","iopub.execute_input":"2022-04-12T16:51:22.417531Z","iopub.status.idle":"2022-04-12T16:51:22.425216Z","shell.execute_reply.started":"2022-04-12T16:51:22.417495Z","shell.execute_reply":"2022-04-12T16:51:22.424182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model\nModel uses pretrained CNN without classification layer. Features from CNN are used as input for embedding layer (linear) and embeddings are used for final classification. Model can output both embeddings and class prediction or just embeddings.\n\nInput image -> [CNN] -> features -> [Embedding layer] -> embeddings -> [Classification layer] -> class prediction","metadata":{"id":"FpR2HfK93pvS","papermill":{"duration":0.025911,"end_time":"2022-03-23T20:09:24.49901","exception":false,"start_time":"2022-03-23T20:09:24.473099","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class EmbeddingModel(nn.Module):\n def __init__(self, n_classes=100, embedding_size=64, backbone_name=\"efficientnet_b0\"):\n super(EmbeddingModel, self).__init__()\n \n self.backbone = timm.create_model(backbone_name, num_classes=n_classes, pretrained=True)\n in_features = self.backbone.get_classifier().in_features\n \n self.backbone.classifier = nn.Identity()\n self.embedding = nn.Linear(in_features, embedding_size)\n self.classifier = nn.Linear(embedding_size, n_classes)\n\n def embed_and_classify(self, x):\n x = self.forward(x)\n return x, self.classifier(x)\n\n def forward(self, x):\n x = self.backbone(x)\n x = x.view(x.size(0), -1)\n x = self.embedding(x)\n return x","metadata":{"id":"_2mse3zX3pFQ","papermill":{"duration":0.033646,"end_time":"2022-03-23T20:09:24.558453","exception":false,"start_time":"2022-03-23T20:09:24.524807","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-04-12T16:51:22.426442Z","iopub.execute_input":"2022-04-12T16:51:22.426972Z","iopub.status.idle":"2022-04-12T16:51:22.438309Z","shell.execute_reply.started":"2022-04-12T16:51:22.426935Z","shell.execute_reply":"2022-04-12T16:51:22.437552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model helper functions","metadata":{"id":"mTFCinps35ci","papermill":{"duration":0.025639,"end_time":"2022-03-23T20:09:24.609989","exception":false,"start_time":"2022-03-23T20:09:24.58435","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# method to iterate loader and generate embeddings of images\n# returns embeddings and image class\ndef generate_embeddings(loader, model, bar_desc=\"Generating embeds\"):\n targets_all = []\n outputs_all = []\n \n model.eval()\n with torch.no_grad():\n t = tqdm(loader, desc=bar_desc)\n for i, sample in enumerate(t):\n input = sample['image'].to(args.device)\n target = sample['target'].to(args.device)\n output = model(input)\n \n targets_all.extend(target.cpu().numpy())\n outputs_all.extend(output.detach().cpu().numpy())\n\n targets_all = np.array(targets_all).astype(np.float32)\n outputs_all = np.array(outputs_all).astype(np.float32)\n \n return outputs_all, targets_all","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:51:22.439946Z","iopub.execute_input":"2022-04-12T16:51:22.440425Z","iopub.status.idle":"2022-04-12T16:51:22.448695Z","shell.execute_reply.started":"2022-04-12T16:51:22.440384Z","shell.execute_reply":"2022-04-12T16:51:22.447866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_train_valid_embeddings(args, data_df):\n model_name = f\"embedding-model-{args.backbone_name}-{IMG_SIZE}x{IMG_SIZE}\"\n print(model_name)\n\n seed_everything(seed=SEED)\n\n # split data into train and validation set\n hotel_image_count = data_df.groupby(\"hotel_id\")[\"image_id\"].count()\n # hotels that have more images than samples for validation\n valid_hotels = hotel_image_count[hotel_image_count > args.val_samples]\n # data that can be split into train and val set\n valid_data = data_df[data_df[\"hotel_id\"].isin(valid_hotels.index)]\n # if hotel had less than required val_samples it will be only in the train set\n valid_df = valid_data.groupby(\"hotel_id\").sample(args.val_samples, random_state=SEED).reset_index(drop=True)\n train_df = data_df[~data_df[\"image_id\"].isin(valid_df[\"image_id\"])].reset_index(drop=True)\n \n\n valid_dataset = HotelTrainDataset(valid_df, val_transform, data_path=IMAGE_FOLDER)\n valid_loader = DataLoader(valid_dataset, num_workers=args.num_workers, batch_size=args.batch_size, shuffle=False)\n # base dataset for image similarity search\n base_dataset = HotelTrainDataset(train_df, base_transform, data_path=IMAGE_FOLDER)\n base_loader = DataLoader(base_dataset, num_workers=args.num_workers, batch_size=args.batch_size, shuffle=False)\n\n # use trained model from Hotel-ID starter - similarity- training\n checkpoint = torch.load(args.checkpoint_path)\n model = EmbeddingModel(args.n_classes, args.embedding_size ,args.backbone_name)\n model.load_state_dict(checkpoint[\"model\"])\n model = model.to(args.device)\n \n \n train_embeds, _ = generate_embeddings(base_loader, model, \"Generate embeddings for base images\")\n train_df[\"embeddings\"] = list(train_embeds)\n train_df.to_pickle(f\"{OUTPUT_FOLDER}{model_name}_train-image-embeddings.pkl\")\n \n valid_embeds, _ = generate_embeddings(valid_loader, model, \"Generate embeddings for valid images\")\n valid_df[\"embeddings\"] = list(valid_embeds)\n valid_df.to_pickle(f\"{OUTPUT_FOLDER}{model_name}_val-image-embeddings.pkl\")\n \n return train_embeds, train_df, valid_embeds, valid_df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:51:22.450022Z","iopub.execute_input":"2022-04-12T16:51:22.450452Z","iopub.status.idle":"2022-04-12T16:51:22.463461Z","shell.execute_reply.started":"2022-04-12T16:51:22.450415Z","shell.execute_reply":"2022-04-12T16:51:22.462666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare data\nWe will generate embeddings using the pretrained model for training and validation dataset from the training notebook. For the validation dataset we will use val_transform so there will be occlusions in the images. Training dataset will not use any augmentations.","metadata":{"id":"F2xgmwBW4LjC","papermill":{"duration":0.025544,"end_time":"2022-03-23T20:09:24.788734","exception":false,"start_time":"2022-03-23T20:09:24.76319","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data_df = pd.read_csv(DATA_FOLDER + \"train.csv\")\n# encode hotel ids\ndata_df[\"hotel_id_code\"] = data_df[\"hotel_id\"].astype('category').cat.codes.values.astype(np.int64)","metadata":{"id":"Sn6HrWKQ2_aw","papermill":{"duration":0.154871,"end_time":"2022-03-23T20:09:27.317597","exception":false,"start_time":"2022-03-23T20:09:27.162726","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-04-12T16:51:22.465628Z","iopub.execute_input":"2022-04-12T16:51:22.465859Z","iopub.status.idle":"2022-04-12T16:51:22.535356Z","shell.execute_reply.started":"2022-04-12T16:51:22.465835Z","shell.execute_reply":"2022-04-12T16:51:22.534721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# save hotel_id encoding for later decoding\nhotel_id_code_df = data_df.drop(columns=[\"image_id\"]).drop_duplicates().reset_index(drop=True)\nhotel_id_code_df.to_csv(OUTPUT_FOLDER + 'hotel_id_code_mapping.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-04-12T16:51:22.539142Z","iopub.execute_input":"2022-04-12T16:51:22.539329Z","iopub.status.idle":"2022-04-12T16:51:22.557856Z","shell.execute_reply.started":"2022-04-12T16:51:22.539307Z","shell.execute_reply":"2022-04-12T16:51:22.557197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class args:\n batch_size = 256\n num_workers = 2\n val_samples = 1\n embedding_size = 128\n backbone_name = \"efficientnet_b0\"\n checkpoint_path = \"../input/hotel-id-starter-similarity-training/checkpoint-embedding-model-efficientnet_b0-256x256.pt\"\n n_classes = data_df[\"hotel_id_code\"].nunique()\n device = ('cuda' if torch.cuda.is_available() else 'cpu')\n \ntrain_embeds, train_df, valid_embeds, valid_df = get_train_valid_embeddings(args, data_df)","metadata":{"execution":{"iopub.status.busy":"2022-04-12T16:51:22.558891Z","iopub.execute_input":"2022-04-12T16:51:22.559196Z","iopub.status.idle":"2022-04-12T16:54:38.668875Z","shell.execute_reply.started":"2022-04-12T16:51:22.559162Z","shell.execute_reply":"2022-04-12T16:54:38.667735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualization\nWe can project embeddings using different methods like PCA, TSNE, UMAP to new space with reduced dimensions and then plot the results. We will use only subsample of images from 20 different hotels.","metadata":{}},{"cell_type":"code","source":"import umap\n\nfrom sklearn.decomposition import PCA\nfrom sklearn.manifold import TSNE","metadata":{"execution":{"iopub.status.busy":"2022-04-12T16:54:38.670660Z","iopub.execute_input":"2022-04-12T16:54:38.671012Z","iopub.status.idle":"2022-04-12T16:54:55.632915Z","shell.execute_reply.started":"2022-04-12T16:54:38.670974Z","shell.execute_reply":"2022-04-12T16:54:55.632084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_hotel_ids = train_df[\"hotel_id\"].unique()[:20]\nsample_df = train_df[train_df[\"hotel_id\"].isin(sample_hotel_ids)].reset_index(drop=True)\nsample_embeds = np.vstack(sample_df[\"embeddings\"].values)\nsample_labels = sample_df[\"hotel_id\"].astype(str)","metadata":{"execution":{"iopub.status.busy":"2022-04-12T16:54:55.634405Z","iopub.execute_input":"2022-04-12T16:54:55.634664Z","iopub.status.idle":"2022-04-12T16:54:55.650081Z","shell.execute_reply.started":"2022-04-12T16:54:55.634630Z","shell.execute_reply":"2022-04-12T16:54:55.649224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Selected hotels:\", sample_hotel_ids)\nprint(\"Number of samples:\", len(sample_df))\nprint(\"Emeddings dimensions:\", np.shape(sample_embeds))","metadata":{"execution":{"iopub.status.busy":"2022-04-12T16:54:55.652746Z","iopub.execute_input":"2022-04-12T16:54:55.653432Z","iopub.status.idle":"2022-04-12T16:54:55.666449Z","shell.execute_reply.started":"2022-04-12T16:54:55.653372Z","shell.execute_reply":"2022-04-12T16:54:55.665417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(sample_df, x=sample_labels, histfunc='sum',barmode='group')\nfig.update_layout(title={'text' : \"Selected samples - Image count per hotel\"}, \n xaxis_title=\"Hotel ID\",\n yaxis_title=\"Image count\", \n template=\"simple_white\", height=400)","metadata":{"execution":{"iopub.status.busy":"2022-04-12T17:00:47.888678Z","iopub.execute_input":"2022-04-12T17:00:47.889315Z","iopub.status.idle":"2022-04-12T17:00:48.007110Z","shell.execute_reply.started":"2022-04-12T17:00:47.889267Z","shell.execute_reply":"2022-04-12T17:00:48.006444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## PCA - 2 components","metadata":{}},{"cell_type":"code","source":"pca = PCA(n_components=2)\npca_embeds = pca.fit_transform(sample_embeds)\n\nfig = px.scatter(x=pca_embeds[:,0], y=pca_embeds[:,1], color=sample_labels, custom_data =[sample_labels, sample_df[\"image_id\"].values])\nfig.update_traces(hovertemplate=\"Hotel ID: %{customdata[0]}
Image: %{customdata[1]}\")\nfig.update_layout(title=\"Embeddings - 2d projection using PCA\", legend=dict(title=\"Hotel ID\"),\n height=400,)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:59:54.033200Z","iopub.execute_input":"2022-04-12T16:59:54.033834Z","iopub.status.idle":"2022-04-12T16:59:54.272183Z","shell.execute_reply.started":"2022-04-12T16:59:54.033793Z","shell.execute_reply":"2022-04-12T16:59:54.271519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## TSNE - 2 components","metadata":{}},{"cell_type":"code","source":"tsne = TSNE(n_components=2, learning_rate='auto', init='random')\ntsne_embeds = tsne.fit_transform(sample_embeds)\n\nfig = px.scatter(x=tsne_embeds[:,0], y=tsne_embeds[:,1], \n color=sample_labels, custom_data =[sample_labels, sample_df[\"image_id\"].values])\nfig.update_traces(hovertemplate=\"Hotel ID: %{customdata[0]}
Image: %{customdata[1]}\")\nfig.update_layout(title=\"Embeddings - 2d projection using TSNE\", legend=dict(title=\"Hotel ID\"),\n height=400,)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T17:00:11.387224Z","iopub.execute_input":"2022-04-12T17:00:11.387507Z","iopub.status.idle":"2022-04-12T17:00:13.182245Z","shell.execute_reply.started":"2022-04-12T17:00:11.387476Z","shell.execute_reply":"2022-04-12T17:00:13.181560Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## TSNE - 3 components","metadata":{}},{"cell_type":"code","source":"tsne = TSNE(n_components=3, learning_rate='auto', init='random')\ntsne_embeds = tsne.fit_transform(sample_embeds)\n\nfig = px.scatter_3d(x=tsne_embeds[:,0], y=tsne_embeds[:,1], z=tsne_embeds[:,2], \n color=sample_labels, custom_data =[sample_labels, sample_df[\"image_id\"].values])\nfig.update_traces(hovertemplate=\"Hotel ID: %{customdata[0]}
Image: %{customdata[1]}\")\nfig.update_layout(title=\"Embeddings - 3d projection using TSNE\", legend=dict(title=\"Hotel ID\"),\n height=400,)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:54:56.224685Z","iopub.status.idle":"2022-04-12T16:54:56.225592Z","shell.execute_reply.started":"2022-04-12T16:54:56.225274Z","shell.execute_reply":"2022-04-12T16:54:56.225302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## UMAP - 2 components","metadata":{}},{"cell_type":"code","source":"reducer = umap.UMAP(random_state=SEED)\numap_embeds = reducer.fit_transform(sample_embeds)\n\nfig = px.scatter(x=umap_embeds[:,0], y=umap_embeds[:,1], \n color=sample_labels, custom_data =[sample_labels, sample_df[\"image_id\"].values])\nfig.update_traces(hovertemplate=\"Hotel ID: %{customdata[0]}
Image: %{customdata[1]}\")\nfig.update_layout(title=\"Embeddings - 2d projection using UMAP\", legend=dict(title=\"Hotel ID\"),\n height=400,)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:54:56.226739Z","iopub.status.idle":"2022-04-12T16:54:56.227512Z","shell.execute_reply.started":"2022-04-12T16:54:56.227247Z","shell.execute_reply":"2022-04-12T16:54:56.227271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## UMAP - interactive with image display \nWanted to display image on hover but couldn't find how to do it in plotly without Dash so gonna use bokeh","metadata":{}},{"cell_type":"code","source":"# methods based on: https://www.kaggle.com/code/parulpandey/visualizing-kannada-mnist-with-t-sne/notebook\n\n# Encoding all the images for inclusion in a dataframe.\nfrom io import BytesIO\nimport base64\n\ndef embeddable_image(data):\n image = pil_image.fromarray(data, 'RGB').resize((128,128), pil_image.BICUBIC)\n buffer = BytesIO()\n image.save(buffer, format='png')\n for_encoding = buffer.getvalue()\n return 'data:image/png;base64,' + base64.b64encode(for_encoding).decode()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:54:56.228626Z","iopub.status.idle":"2022-04-12T16:54:56.229438Z","shell.execute_reply.started":"2022-04-12T16:54:56.229144Z","shell.execute_reply":"2022-04-12T16:54:56.229171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# loading up bokeh and other tools to generate a suitable interactive plot.\n\nfrom bokeh.plotting import figure, show, output_notebook\nfrom bokeh.models import HoverTool, ColumnDataSource, CategoricalColorMapper\nfrom bokeh.palettes import Category20\n\noutput_notebook()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:54:56.230550Z","iopub.status.idle":"2022-04-12T16:54:56.231314Z","shell.execute_reply.started":"2022-04-12T16:54:56.231070Z","shell.execute_reply":"2022-04-12T16:54:56.231095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# prepare data\nsample_image_data = []\nfor i in range(len(sample_df)):\n image_path = IMAGE_FOLDER + sample_df.loc[i, \"image_id\"]\n image = np.array(pil_image.open(image_path)).astype(np.uint8)\n sample_image_data.extend([image])\n \nsample_df[\"image_data\"] = list(map(embeddable_image, sample_image_data))\nsample_df[\"x\"] = umap_embeds[:, 0]\nsample_df[\"y\"] = umap_embeds[:, 1]\n\nsample_df[\"hotel_id_code\"] = sample_df[\"hotel_id\"].astype('category').cat.codes.values.astype(np.int64)+1\nsample_df[\"hotel_id_code\"] = sample_df[\"hotel_id_code\"].astype(str)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:54:56.232498Z","iopub.status.idle":"2022-04-12T16:54:56.233315Z","shell.execute_reply.started":"2022-04-12T16:54:56.233052Z","shell.execute_reply":"2022-04-12T16:54:56.233077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generating the plot itself with a custom hover tooltip \ndatasource = ColumnDataSource(sample_df)\ncolor_mapping = CategoricalColorMapper(factors=sample_df[\"hotel_id_code\"].unique(), palette=Category20[20])\n\nplot_figure = figure(\n title='Embeddings - 2d projection using UMAP',\n plot_width=700,\n plot_height=400,\n tools=('pan, wheel_zoom, reset')\n)\n\nplot_figure.add_tools(HoverTool(tooltips=\"\"\"\n
\n
\n \n
\n
\n Hotel: @hotel_id\n
\n
\n Image: @image_id\n
\n
\n\"\"\"))\n\nplot_figure.circle('x', 'y',\n source=datasource,\n color={'field': 'hotel_id_code', 'transform': color_mapping},\n line_alpha=0.6,\n fill_alpha=0.6,\n radius=0.05,\n legend_field='hotel_id',\n size=4\n)\n\nshow(plot_figure)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-12T16:54:56.234487Z","iopub.status.idle":"2022-04-12T16:54:56.235251Z","shell.execute_reply.started":"2022-04-12T16:54:56.235005Z","shell.execute_reply":"2022-04-12T16:54:56.235032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Similarity search","metadata":{}},{"cell_type":"markdown","source":"## Similarity\nTo find if images are similar we can calculate distance/similarity of their emebeddings using methods like [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance) or [Cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity).\n\nAfter training model we can get embeddings for all images with known class (hotel id) and calculate their similarity to the test image we want to classify. We can rank the train images based on their similarity to embeddings of the test image and find 5 train images from different hotels that are most similar.\n","metadata":{}},{"cell_type":"markdown","source":"## Find 5 most similar images from different hotels","metadata":{}},{"cell_type":"code","source":"# find 5 most similar images from different hotels and return their hotel_id_code\ndef find_matches_cosine_similarity(query, base_embeds, base_targets, k=N_MATCHES):\n distance_df = pd.DataFrame(index=np.arange(len(base_targets)), data={\"hotel_id_code\": base_targets})\n # calculate cosine distance of query embeds to all base embeds\n distance_df[\"distance\"] = cosine_similarity([query], base_embeds)[0]\n # sort by distance and hotel_id\n distance_df = distance_df.sort_values(by=[\"distance\", \"hotel_id_code\"], ascending=False).reset_index(drop=True)\n # return first 5 different hotel_id_codes\n return distance_df[\"hotel_id_code\"].unique()[:N_MATCHES]\n\n\ndef test_similarity(base_embeds, base_df, test_embeds, test_df):\n base_targets = base_df[\"hotel_id\"]\n test_targets = test_df[\"hotel_id\"]\n \n preds = []\n \n for query_embeds in tqdm(test_embeds, desc=\"Similarity - match finding\"):\n tmp = find_matches_cosine_similarity(query_embeds, base_embeds, base_targets)\n preds.extend([tmp])\n \n preds = np.array(preds)\n test_targets_N = np.repeat([test_targets], repeats=N_MATCHES, axis=0).T\n # check if any of top 5 predictions are correct and calculate mean accuracy\n acc_top_5 = (preds == test_targets_N).any(axis=1).mean()\n # calculate prediction accuracy\n acc_top_1 = np.mean(test_targets == preds[:, 0])\n print(f\"Cosine similarity accuracy: {acc_top_1:0.4f}, MAP@5: {acc_top_5:0.4f}\")","metadata":{"execution":{"iopub.status.busy":"2022-04-12T16:54:56.236454Z","iopub.status.idle":"2022-04-12T16:54:56.237232Z","shell.execute_reply.started":"2022-04-12T16:54:56.236984Z","shell.execute_reply":"2022-04-12T16:54:56.237010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def find_matches_distance(query, base_embeds, base_targets, distance_fc, k=N_MATCHES):\n distance_df = pd.DataFrame(index=np.arange(len(base_targets)), data={\"hotel_id_code\": base_targets})\n # calculate cosine distance of query embeds to all base embeds\n distance_df[\"distance\"] = distance_fc([query], base_embeds)[0]\n # sort by distance and hotel_id\n distance_df = distance_df.sort_values(by=[\"distance\", \"hotel_id_code\"], ascending=True).reset_index(drop=True)\n # return first 5 different hotel_id_codes\n return distance_df[\"hotel_id_code\"].unique()[:N_MATCHES]\n \n \ndef test_distance(base_embeds, base_df, test_embeds, test_df, distance_fc):\n base_targets = base_df[\"hotel_id\"]\n test_targets = test_df[\"hotel_id\"]\n \n preds = []\n \n for query_embeds in tqdm(test_embeds, desc=\"Distance - match finding\"):\n tmp = find_matches_distance(query_embeds, base_embeds, base_targets, distance_fc)\n preds.extend([tmp])\n \n preds = np.array(preds)\n test_targets_N = np.repeat([test_targets], repeats=N_MATCHES, axis=0).T\n # check if any of top 5 predictions are correct and calculate mean accuracy\n acc_top_5 = (preds == test_targets_N).any(axis=1).mean()\n # calculate prediction accuracy\n acc_top_1 = np.mean(test_targets == preds[:, 0])\n print(f\"Distance accuracy: {acc_top_1:0.4f}, MAP@5: {acc_top_5:0.4f}\")","metadata":{"execution":{"iopub.status.busy":"2022-04-12T16:54:56.238432Z","iopub.status.idle":"2022-04-12T16:54:56.239207Z","shell.execute_reply.started":"2022-04-12T16:54:56.238952Z","shell.execute_reply":"2022-04-12T16:54:56.238977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_similarity(train_embeds, train_df, valid_embeds, valid_df)","metadata":{"execution":{"iopub.status.busy":"2022-04-12T16:54:56.240348Z","iopub.status.idle":"2022-04-12T16:54:56.241122Z","shell.execute_reply.started":"2022-04-12T16:54:56.240873Z","shell.execute_reply":"2022-04-12T16:54:56.240898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics.pairwise import euclidean_distances, cosine_distances\n\nprint(\"Euclidean distance\")\ntest_distance(train_embeds, train_df, valid_embeds, valid_df, euclidean_distances)\n\nprint(\"\\nCosine distance\")\ntest_distance(train_embeds, train_df, valid_embeds, valid_df, cosine_distances)","metadata":{"execution":{"iopub.status.busy":"2022-04-12T16:54:56.242285Z","iopub.status.idle":"2022-04-12T16:54:56.243131Z","shell.execute_reply.started":"2022-04-12T16:54:56.242856Z","shell.execute_reply":"2022-04-12T16:54:56.242886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## KNN Classifier","metadata":{}},{"cell_type":"code","source":"from sklearn.neighbors import KNeighborsClassifier","metadata":{"execution":{"iopub.status.busy":"2022-04-12T16:54:56.244259Z","iopub.status.idle":"2022-04-12T16:54:56.245144Z","shell.execute_reply.started":"2022-04-12T16:54:56.244855Z","shell.execute_reply":"2022-04-12T16:54:56.244885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def test_knn(x_train, y_train, x_test, y_test, n_neighbors=5, metric=\"euclidean\"):\n knn = KNeighborsClassifier(n_neighbors=n_neighbors, metric=metric)\n knn.fit(x_train, y_train)\n preds = knn.predict_proba(x_test)\n\n top_1_pred = knn.classes_[np.argmax(preds, axis=1)]\n top_5_pred = knn.classes_[np.argsort(-preds, axis=1)[:,:5]]\n\n test_targets_N = np.repeat([y_test], repeats=N_MATCHES, axis=0).T\n # check if any of top 5 predictions are correct and calculate mean accuracy\n acc_top_5 = (top_5_pred == test_targets_N).any(axis=1).mean()\n # calculate prediction accuracy\n acc_top_1 = np.mean(y_test == top_1_pred)\n print(f\"KNN ({n_neighbors}, {metric}) accuracy: {acc_top_1:0.4f}, MAP@5: {acc_top_5:0.4f}\")","metadata":{"execution":{"iopub.status.busy":"2022-04-12T16:54:56.246436Z","iopub.status.idle":"2022-04-12T16:54:56.247204Z","shell.execute_reply.started":"2022-04-12T16:54:56.246952Z","shell.execute_reply":"2022-04-12T16:54:56.246978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_knn(train_embeds, train_df[\"hotel_id\"], valid_embeds, valid_df[\"hotel_id\"], n_neighbors=1)\ntest_knn(train_embeds, train_df[\"hotel_id\"], valid_embeds, valid_df[\"hotel_id\"], n_neighbors=5)\ntest_knn(train_embeds, train_df[\"hotel_id\"], valid_embeds, valid_df[\"hotel_id\"], n_neighbors=10)\ntest_knn(train_embeds, train_df[\"hotel_id\"], valid_embeds, valid_df[\"hotel_id\"], n_neighbors=25)\ntest_knn(train_embeds, train_df[\"hotel_id\"], valid_embeds, valid_df[\"hotel_id\"], n_neighbors=50)","metadata":{"execution":{"iopub.status.busy":"2022-04-12T16:54:56.248371Z","iopub.status.idle":"2022-04-12T16:54:56.249152Z","shell.execute_reply.started":"2022-04-12T16:54:56.248903Z","shell.execute_reply":"2022-04-12T16:54:56.248928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_knn(train_embeds, train_df[\"hotel_id\"], valid_embeds, valid_df[\"hotel_id\"], n_neighbors=1, metric=\"cosine\")\ntest_knn(train_embeds, train_df[\"hotel_id\"], valid_embeds, valid_df[\"hotel_id\"], n_neighbors=5, metric=\"cosine\")\ntest_knn(train_embeds, train_df[\"hotel_id\"], valid_embeds, valid_df[\"hotel_id\"], n_neighbors=10, metric=\"cosine\")\ntest_knn(train_embeds, train_df[\"hotel_id\"], valid_embeds, valid_df[\"hotel_id\"], n_neighbors=25, metric=\"cosine\")\ntest_knn(train_embeds, train_df[\"hotel_id\"], valid_embeds, valid_df[\"hotel_id\"], n_neighbors=50, metric=\"cosine\")","metadata":{"execution":{"iopub.status.busy":"2022-04-12T16:54:56.250265Z","iopub.status.idle":"2022-04-12T16:54:56.251044Z","shell.execute_reply.started":"2022-04-12T16:54:56.250793Z","shell.execute_reply":"2022-04-12T16:54:56.250819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_knn(train_embeds, train_df[\"hotel_id\"], valid_embeds, valid_df[\"hotel_id\"], n_neighbors=5, metric=\"minkowski\")\ntest_knn(train_embeds, train_df[\"hotel_id\"], valid_embeds, valid_df[\"hotel_id\"], n_neighbors=5, metric=\"manhattan\")","metadata":{"execution":{"iopub.status.busy":"2022-04-12T16:54:56.252194Z","iopub.status.idle":"2022-04-12T16:54:56.253819Z","shell.execute_reply.started":"2022-04-12T16:54:56.253555Z","shell.execute_reply":"2022-04-12T16:54:56.253582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Output\nYou can use output of this notebook, load train and valid data with embeddings and try your own methods to classify images.\n- training data: embedding-model-efficientnet_b0-256x256_train-image-embeddings.pkl\n- validation data: embedding-model-efficientnet_b0-256x256_val-image-embeddings.pkl","metadata":{}}]}