{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Intro\n\n","metadata":{}},{"cell_type":"markdown","source":"Starter notebook for [Hotel-ID to Combat Human Trafficking 2022 - FGVC9](https://www.kaggle.com/competitions/hotel-id-to-combat-human-trafficking-2022-fgvc9) competition using model to generate embeddings as image representation and then searching for the most similar images. \n\nInference notebook: [Hotel-ID starter - similarity- inference](https://www.kaggle.com/code/ahmedhifnawy/hotel-id-starter-similarity-inference)\n\n## Embeddings\nTo compare images we can use model to generate embeddings as their representation and then calculate distance/similarity between images to search for the most similar one.\n\nWe can use pretrained model without the last classification layer and add two linear layers. Features from pretrianed CNN will be used as input for embedding layer and result of embedding layer will be used for classification layer. Model will then output embeddings and predicted class. We can use a class prediction to calculate loss and train the model further and embeddings to search for similar images. Embeddings should contain enough information to predict correct class so they should be good representations of the image. \n\n![Embedding model](https://github.com/michal-nahlik/kaggle-hotel-id-2022/raw/master/doc/img/embedding_model.png)","metadata":{}},{"cell_type":"markdown","source":"## Similarity\nTo find if images are similar we can calculate distance/similarity of their emebeddings using methods like [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance) or [Cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity).\n\nAfter training model we can get embeddings for all images with known class (hotel id) and calculate their similarity to the test image we want to classify. We can rank the train images based on their similarity to embeddings of the test image and find 5 train images from different hotels that are most similar.","metadata":{}},{"cell_type":"markdown","source":"## Data\nThis notebook uses preprocessed images that were resized and padded to 512x512 pixel.\n\nUsed dataset: [Hotel-ID 2022 train images 512x512](https://www.kaggle.com/datasets/raj2105/mlipmodels) created by [Hotel-ID - image preprocessing - 512x512](https://www.kaggle.com/code/raj2105/hotel-id-image-preprocessing-512x512) notebook.","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{"id":"MyC4gTwZ3MKJ","papermill":{"duration":0.020375,"end_time":"2022-03-23T20:08:58.820405","exception":false,"start_time":"2022-03-23T20:08:58.80003","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install timm\n!pip install pytorch_metric_learning","metadata":{"papermill":{"duration":18.268618,"end_time":"2022-03-23T20:09:17.10966","exception":false,"start_time":"2022-03-23T20:08:58.841042","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-21T21:46:20.440069Z","iopub.execute_input":"2022-05-21T21:46:20.440616Z","iopub.status.idle":"2022-05-21T21:46:42.108515Z","shell.execute_reply.started":"2022-05-21T21:46:20.440521Z","shell.execute_reply":"2022-05-21T21:46:42.107637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport random\nimport os","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","id":"u0Bz2ktn2_ap","papermill":{"duration":0.032275,"end_time":"2022-03-23T20:09:17.168351","exception":false,"start_time":"2022-03-23T20:09:17.136076","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-21T21:46:42.110821Z","iopub.execute_input":"2022-05-21T21:46:42.111087Z","iopub.status.idle":"2022-05-21T21:46:42.114999Z","shell.execute_reply.started":"2022-05-21T21:46:42.111051Z","shell.execute_reply":"2022-05-21T21:46:42.114347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from PIL import Image as pil_image\nfrom tqdm import tqdm\n\nimport matplotlib\nimport matplotlib.pyplot as plt\n\nimport plotly.graph_objects as go\nimport plotly.express as px\nfrom plotly.subplots import make_subplots","metadata":{"id":"tOszKuxt3PXn","papermill":{"duration":3.425867,"end_time":"2022-03-23T20:09:20.619496","exception":false,"start_time":"2022-03-23T20:09:17.193629","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-21T21:46:42.116741Z","iopub.execute_input":"2022-05-21T21:46:42.117137Z","iopub.status.idle":"2022-05-21T21:46:45.453926Z","shell.execute_reply.started":"2022-05-21T21:46:42.117101Z","shell.execute_reply":"2022-05-21T21:46:45.453159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nfrom torch.utils.data import DataLoader\n\nimport timm\n\nfrom sklearn.metrics.pairwise import cosine_similarity\nfrom pytorch_metric_learning import miners, losses, samplers , distances, regularizers","metadata":{"id":"uQE7wYFR3QxV","papermill":{"duration":2.460075,"end_time":"2022-03-23T20:09:23.106294","exception":false,"start_time":"2022-03-23T20:09:20.646219","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-21T21:46:45.455295Z","iopub.execute_input":"2022-05-21T21:46:45.455543Z","iopub.status.idle":"2022-05-21T21:46:47.905062Z","shell.execute_reply.started":"2022-05-21T21:46:45.45551Z","shell.execute_reply":"2022-05-21T21:46:47.904199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Global","metadata":{"id":"tirOg6jm3aIB","papermill":{"duration":0.025747,"end_time":"2022-03-23T20:09:23.159882","exception":false,"start_time":"2022-03-23T20:09:23.134135","status":"completed"},"tags":[]}},{"cell_type":"code","source":"IMG_SIZE = 512\nSEED = 42\nN_MATCHES = 5\n\nPROJECT_FOLDER = \"../input/hotel-id-to-combat-human-trafficking-2022-fgvc9/\"\nDATA_FOLDER = \"../input/mlipmodels/\"\nIMAGE_FOLDER = DATA_FOLDER + \"images/\"\nOUTPUT_FOLDER = \"\"\n\ntrain_df = pd.read_csv(os.path.join(DATA_FOLDER, 'train.csv'))","metadata":{"id":"DV7qHDuYGoJH","papermill":{"duration":0.086034,"end_time":"2022-03-23T20:09:23.2714","exception":false,"start_time":"2022-03-23T20:09:23.185366","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-21T21:46:47.908174Z","iopub.execute_input":"2022-05-21T21:46:47.908441Z","iopub.status.idle":"2022-05-21T21:46:47.968723Z","shell.execute_reply.started":"2022-05-21T21:46:47.908407Z","shell.execute_reply":"2022-05-21T21:46:47.96791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(os.listdir(PROJECT_FOLDER))","metadata":{"id":"TB9CXg8U3bbQ","papermill":{"duration":0.032796,"end_time":"2022-03-23T20:09:23.329909","exception":false,"start_time":"2022-03-23T20:09:23.297113","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-21T21:46:47.970203Z","iopub.execute_input":"2022-05-21T21:46:47.970623Z","iopub.status.idle":"2022-05-21T21:46:47.977175Z","shell.execute_reply.started":"2022-05-21T21:46:47.970586Z","shell.execute_reply":"2022-05-21T21:46:47.976164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True","metadata":{"id":"csp2OMgo2_ar","papermill":{"duration":0.03307,"end_time":"2022-03-23T20:09:23.38914","exception":false,"start_time":"2022-03-23T20:09:23.35607","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-21T21:46:47.978806Z","iopub.execute_input":"2022-05-21T21:46:47.979314Z","iopub.status.idle":"2022-05-21T21:46:47.985315Z","shell.execute_reply.started":"2022-05-21T21:46:47.979276Z","shell.execute_reply":"2022-05-21T21:46:47.984532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset and transformations","metadata":{"id":"8V_xuoN73lON","papermill":{"duration":0.025486,"end_time":"2022-03-23T20:09:23.440614","exception":false,"start_time":"2022-03-23T20:09:23.415128","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Coarse dropout with fill_value=(255,0,0) (full red channel) is used to simulate the occlussions like the one in test dataset. \n```python\nA.CoarseDropout(p=1., max_holes=1, \n                min_height=IMG_SIZE//4, max_height=IMG_SIZE//2,\n                min_width=IMG_SIZE//4,  max_width=IMG_SIZE//2, \n                fill_value=(255,0,0))\n```","metadata":{}},{"cell_type":"code","source":"import albumentations as A\nimport albumentations.pytorch as APT\nimport cv2 \n\n# used for training dataset - augmentations and occlusions\ntrain_transform = A.Compose([\n    A.HorizontalFlip(p=0.75),\n    A.VerticalFlip(p=0.25),\n    A.ShiftScaleRotate(p=0.5, border_mode=cv2.BORDER_CONSTANT),\n    A.OpticalDistortion(p=0.25),\n    A.Perspective(p=0.25),\n    A.CoarseDropout(p=0.5, min_holes=1, max_holes=6, \n                    min_height=IMG_SIZE//16, max_height=IMG_SIZE//4,\n                    min_width=IMG_SIZE//16,  max_width=IMG_SIZE//4), # normal coarse dropout\n    \n    A.CoarseDropout(p=1., max_holes=1, \n                    min_height=IMG_SIZE//4, max_height=IMG_SIZE//2,\n                    min_width=IMG_SIZE//4,  max_width=IMG_SIZE//2, \n                    fill_value=(255,0,0)),# simulating occlusions in test data\n\n    A.RandomBrightnessContrast(p=0.75),\n    A.ToFloat(),\n    APT.transforms.ToTensorV2(),\n])\n\n# used for validation dataset - only occlusions\nval_transform = A.Compose([\n    A.CoarseDropout(p=1., max_holes=1, \n                    min_height=IMG_SIZE//4, max_height=IMG_SIZE//2,\n                    min_width=IMG_SIZE//4,  max_width=IMG_SIZE//2, \n                    fill_value=(255,0,0)),# simulating occlusions\n    A.ToFloat(),\n    APT.transforms.ToTensorV2(),\n])\n\n# no augmentations\nbase_transform = A.Compose([\n    A.ToFloat(),\n    APT.transforms.ToTensorV2(),\n])","metadata":{"id":"8ucWZHeG2_as","papermill":{"duration":0.919086,"end_time":"2022-03-23T20:09:24.385553","exception":false,"start_time":"2022-03-23T20:09:23.466467","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-21T21:46:47.986622Z","iopub.execute_input":"2022-05-21T21:46:47.986994Z","iopub.status.idle":"2022-05-21T21:46:48.809554Z","shell.execute_reply.started":"2022-05-21T21:46:47.986956Z","shell.execute_reply":"2022-05-21T21:46:48.808725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class HotelTrainDataset:\n    def __init__(self, data, transform=None, data_path=\"train_images/\"):\n        self.data = data\n        self.data_path = data_path\n        self.transform = transform\n\n    def __len__(self):\n        return len(self.data)\n    \n    def __getitem__(self, idx):\n        record = self.data.iloc[idx]\n        image_path = self.data_path + record[\"image_id\"]\n        image = np.array(pil_image.open(image_path)).astype(np.uint8)\n\n        if self.transform:\n            transformed = self.transform(image=image)\n            image = transformed[\"image\"]\n        \n        return {\n            \"image\" : image,\n            \"target\" : record['hotel_id_code'],\n        }","metadata":{"id":"EiLYsfKq2_at","papermill":{"duration":0.035446,"end_time":"2022-03-23T20:09:24.447084","exception":false,"start_time":"2022-03-23T20:09:24.411638","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-21T21:46:48.810794Z","iopub.execute_input":"2022-05-21T21:46:48.812894Z","iopub.status.idle":"2022-05-21T21:46:48.821827Z","shell.execute_reply.started":"2022-05-21T21:46:48.81286Z","shell.execute_reply":"2022-05-21T21:46:48.820967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model\nModel uses pretrained CNN without classification layer. Features from CNN are used as input for embedding layer (linear) and embeddings are used for final classification. Model can output both embeddings and class prediction or just embeddings.\n\nInput image -> [CNN] -> features -> [Embedding layer] -> embeddings -> [Classification layer] -> class prediction","metadata":{"id":"FpR2HfK93pvS","papermill":{"duration":0.025911,"end_time":"2022-03-23T20:09:24.49901","exception":false,"start_time":"2022-03-23T20:09:24.473099","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class EmbeddingModel(nn.Module):\n    def __init__(self, n_classes=100, embedding_size=64, backbone_name=\"efficientnet_b0\"):\n        super(EmbeddingModel, self).__init__()\n        \n        self.backbone = timm.create_model(backbone_name, num_classes=n_classes, pretrained=True)\n        in_features = self.backbone.get_classifier().in_features\n        \n        self.backbone.classifier = nn.Identity()\n        self.embedding = nn.Linear(in_features, embedding_size)\n        self.classifier = nn.Linear(embedding_size, n_classes)\n\n    def embed_and_classify(self, x):\n        x = self.forward(x)\n        return x, self.classifier(x)\n\n    def forward(self, x):\n        x = self.backbone(x)\n        x = x.view(x.size(0), -1)\n        x = self.embedding(x)\n        return x","metadata":{"id":"_2mse3zX3pFQ","papermill":{"duration":0.033646,"end_time":"2022-03-23T20:09:24.558453","exception":false,"start_time":"2022-03-23T20:09:24.524807","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-21T21:46:48.823394Z","iopub.execute_input":"2022-05-21T21:46:48.823942Z","iopub.status.idle":"2022-05-21T21:46:48.834833Z","shell.execute_reply.started":"2022-05-21T21:46:48.823899Z","shell.execute_reply":"2022-05-21T21:46:48.833998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model helper functions","metadata":{"id":"mTFCinps35ci","papermill":{"duration":0.025639,"end_time":"2022-03-23T20:09:24.609989","exception":false,"start_time":"2022-03-23T20:09:24.58435","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# method to iterate loader and generate embeddings of images\n# returns embeddings and image class\ndef generate_embeddings(loader, model, bar_desc=\"Generating embeds\"):\n    targets_all = []\n    outputs_all = []\n    \n    model.eval()\n    with torch.no_grad():\n        t = tqdm(loader, desc=bar_desc)\n        for i, sample in enumerate(t):\n            input = sample['image'].to(args.device)\n            target = sample['target'].to(args.device)\n            output = model(input)\n            \n            targets_all.extend(target.cpu().numpy())\n            outputs_all.extend(output.detach().cpu().numpy())\n\n    targets_all = np.array(targets_all).astype(np.float32)\n    outputs_all = np.array(outputs_all).astype(np.float32)\n            \n    return outputs_all, targets_all","metadata":{"execution":{"iopub.status.busy":"2022-05-21T21:46:48.836352Z","iopub.execute_input":"2022-05-21T21:46:48.836717Z","iopub.status.idle":"2022-05-21T21:46:48.846636Z","shell.execute_reply.started":"2022-05-21T21:46:48.836676Z","shell.execute_reply":"2022-05-21T21:46:48.845523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def save_checkpoint(model, scheduler, optimizer, epoch, name, loss=None, score=None):\n    checkpoint = {\"epoch\": epoch,\n                  \"model\": model.state_dict(),\n                  \"scheduler\": scheduler.state_dict(),\n                  \"optimizer\": optimizer.state_dict(),\n                  \"loss\": loss,\n                  \"score\": score,\n                  }\n\n    torch.save(checkpoint, f\"{OUTPUT_FOLDER}checkpoint-{name}.pt\")\n\n\ndef load_checkpoint(model, scheduler, optimizer, name):\n    checkpoint = torch.load(f\"{OUTPUT_FOLDER}checkpoint-{name}.pt\")\n\n    model.load_state_dict(checkpoint[\"model\"])\n    scheduler.load_state_dict(checkpoint[\"scheduler\"])\n    return model, scheduler, optimizer, checkpoint[\"epoch\"]","metadata":{"id":"ryZ6wE0zKPiz","papermill":{"duration":0.035168,"end_time":"2022-03-23T20:09:24.671044","exception":false,"start_time":"2022-03-23T20:09:24.635876","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-21T21:46:48.848328Z","iopub.execute_input":"2022-05-21T21:46:48.84861Z","iopub.status.idle":"2022-05-21T21:46:48.85879Z","shell.execute_reply.started":"2022-05-21T21:46:48.848572Z","shell.execute_reply":"2022-05-21T21:46:48.858006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train and validation functions","metadata":{}},{"cell_type":"code","source":"def train_epoch(args, model, loader, criterion, optimizer, scheduler, epoch):\n    losses = []\n    targets_all = []\n    outputs_all = []\n    \n    model.train()\n    t = tqdm(loader)\n    \n    for i, sample in enumerate(t):\n        optimizer.zero_grad()\n        \n        images = sample['image'].to(args.device, non_blocking=True)\n        targets = sample['target'].to(args.device, non_blocking=True)\n        \n        emb, outputs = model.embed_and_classify(images)\n        loss = criterion(emb, targets)\n        \n        loss.backward()\n        optimizer.step()\n        \n        if scheduler:\n            scheduler.step()\n                \n        losses.append(loss.item())\n        targets_all.extend(targets.cpu().numpy())\n        outputs_all.extend(torch.sigmoid(outputs).detach().cpu().numpy())\n\n        score = np.mean(targets_all == np.argmax(outputs_all, axis=1))\n        desc = f\"Training epoch {epoch}/{args.epochs} - loss:{loss:0.4f}, accuracy: {score:0.4f}\"\n        t.set_description(desc)\n        \n    return np.mean(losses), score","metadata":{"id":"SntLH82s2_au","papermill":{"duration":0.040666,"end_time":"2022-03-23T20:09:24.737525","exception":false,"start_time":"2022-03-23T20:09:24.696859","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-21T21:46:48.860373Z","iopub.execute_input":"2022-05-21T21:46:48.860791Z","iopub.status.idle":"2022-05-21T21:46:48.873168Z","shell.execute_reply.started":"2022-05-21T21:46:48.860748Z","shell.execute_reply":"2022-05-21T21:46:48.872464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def test_classification(loader, model):\n    targets_all = []\n    outputs_all = []\n    \n    model.eval()\n    t = tqdm(loader, desc=\"Classification\")\n    \n    for i, sample in enumerate(t):\n        images = sample['image'].to(args.device, non_blocking=True)\n        targets = sample['target'].to(args.device, non_blocking=True)\n        \n        _, outputs = model.embed_and_classify(images)\n        \n        targets_all.extend(targets.cpu().numpy())\n        outputs_all.extend(torch.sigmoid(outputs).detach().cpu().numpy())\n        \n    \n    # repeat targets to N_MATCHES for easy calculation of MAP@5\n    y = np.repeat([targets_all], repeats=N_MATCHES, axis=0).T\n    # sort predictions and get top 5\n    preds = np.argsort(-np.array(outputs_all), axis=1)[:, :N_MATCHES]\n    # check if any of top 5 predictions are correct and calculate mean accuracy\n    acc_top_5 = (preds == y).any(axis=1).mean()\n    # calculate prediction accuracy\n    acc_top_1 = np.mean(targets_all == np.argmax(outputs_all, axis=1))\n\n    print(f\"Classification accuracy: {acc_top_1:0.4f}, MAP@5: {acc_top_5:0.4f}\")","metadata":{"execution":{"iopub.status.busy":"2022-05-21T21:46:48.878019Z","iopub.execute_input":"2022-05-21T21:46:48.878264Z","iopub.status.idle":"2022-05-21T21:46:48.890057Z","shell.execute_reply.started":"2022-05-21T21:46:48.878209Z","shell.execute_reply":"2022-05-21T21:46:48.889287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# find 5 most similar images from different hotels and return their hotel_id_code\ndef find_matches(query, base_embeds, base_targets, k=N_MATCHES):\n    distance_df = pd.DataFrame(index=np.arange(len(base_targets)), data={\"hotel_id_code\": base_targets})\n    # calculate cosine distance of query embeds to all base embeds\n    distance_df[\"distance\"] = cosine_similarity([query], base_embeds)[0]\n    # sort by distance and hotel_id\n    distance_df = distance_df.sort_values(by=[\"distance\", \"hotel_id_code\"], ascending=False).reset_index(drop=True)\n    # return first 5 different hotel_id_codes\n    return distance_df[\"hotel_id_code\"].unique()[:N_MATCHES]\n    \n\ndef test_similarity(args, base_loader, test_loader, model):\n    base_embeds, base_targets = generate_embeddings(base_loader, model, \"Generate base embeddings\")\n    test_embeds, test_targets = generate_embeddings(test_loader, model, \"Generate test embeddings\")\n    \n    preds = []\n    for query_embeds in tqdm(test_embeds, desc=\"Similarity - match finding\"):\n        tmp = find_matches(query_embeds, base_embeds, base_targets)\n        preds.extend([tmp])\n        \n    preds = np.array(preds)\n    test_targets_N = np.repeat([test_targets], repeats=N_MATCHES, axis=0).T\n    # check if any of top 5 predictions are correct and calculate mean accuracy\n    acc_top_5 = (preds == test_targets_N).any(axis=1).mean()\n    # calculate prediction accuracy\n    acc_top_1 = np.mean(test_targets == preds[:, 0])\n    print(f\"Similarity accuracy: {acc_top_1:0.4f}, MAP@5: {acc_top_5:0.4f}\")","metadata":{"execution":{"iopub.status.busy":"2022-05-21T21:46:48.892116Z","iopub.execute_input":"2022-05-21T21:46:48.892768Z","iopub.status.idle":"2022-05-21T21:46:48.904426Z","shell.execute_reply.started":"2022-05-21T21:46:48.892716Z","shell.execute_reply":"2022-05-21T21:46:48.903631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare data","metadata":{"id":"F2xgmwBW4LjC","papermill":{"duration":0.025544,"end_time":"2022-03-23T20:09:24.788734","exception":false,"start_time":"2022-03-23T20:09:24.76319","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data_df = pd.read_csv(DATA_FOLDER + \"train.csv\")\n# encode hotel ids\ndata_df[\"hotel_id_code\"] = data_df[\"hotel_id\"].astype('category').cat.codes.values.astype(np.int64)","metadata":{"id":"Sn6HrWKQ2_aw","papermill":{"duration":0.154871,"end_time":"2022-03-23T20:09:27.317597","exception":false,"start_time":"2022-03-23T20:09:27.162726","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-21T21:46:48.907069Z","iopub.execute_input":"2022-05-21T21:46:48.907809Z","iopub.status.idle":"2022-05-21T21:46:48.957426Z","shell.execute_reply.started":"2022-05-21T21:46:48.907776Z","shell.execute_reply":"2022-05-21T21:46:48.956599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# save hotel_id encoding for later decoding\nhotel_id_code_df = data_df.drop(columns=[\"image_id\"]).drop_duplicates().reset_index(drop=True)\nhotel_id_code_df.to_csv(OUTPUT_FOLDER + 'hotel_id_code_mapping.csv', index=False)\n# hotel_id_code_map = hotel_id_code_df.set_index('hotel_id_code').to_dict()[\"hotel_id\"]","metadata":{"execution":{"iopub.status.busy":"2022-05-21T21:46:48.958999Z","iopub.execute_input":"2022-05-21T21:46:48.95942Z","iopub.status.idle":"2022-05-21T21:46:48.980771Z","shell.execute_reply.started":"2022-05-21T21:46:48.959383Z","shell.execute_reply":"2022-05-21T21:46:48.979898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Quick look at data\n\nWe can see that most hotels have less than 20 images while there are few with over hundreds and one with over 1000. In case of classification weights or sampling might be useful to handle the imbalance. There is also 11 hotels with only 1 image so we should think about how to deal with them.","metadata":{}},{"cell_type":"markdown","source":"## Image count per hotel","metadata":{}},{"cell_type":"code","source":"group_df = data_df.groupby([\"hotel_id\"]).size().to_frame(\"image_count\").sort_values(\"image_count\")[::-1].reset_index()\n\n# top and low\nlow_df = group_df.iloc[-50:]\ntop_df = group_df.iloc[:50]\n\nfig = make_subplots(rows=2, cols=2, \n                    specs=[[{\"colspan\": 2}, None], [{}, {}]],\n                    horizontal_spacing=0.02, vertical_spacing=0.2, \n                    shared_yaxes=False,\n                    subplot_titles=(\"\", \"Top 50\", \"Bottom 50\"))\n\n\nfig.add_trace(go.Scatter(x=group_df[\"hotel_id\"], y=group_df[\"image_count\"], showlegend = False), 1, 1)\nfig.add_trace(go.Bar(x=top_df[\"hotel_id\"], y=top_df[\"image_count\"], showlegend = False), 2, 1)\nfig.add_trace(go.Bar(x=low_df[\"hotel_id\"], y=low_df[\"image_count\"], showlegend = False), 2, 2)\n\nfig.update_yaxes(title_text=\"Image count\", row=1, col=1)\nfig.update_yaxes(title_text=\"Image count\", row=2, col=1)\nfig.update_xaxes(type=\"category\", visible=False, row=1, col=1)\nfig.update_xaxes(title_text=\"Hotel ID\", type=\"category\", row=2, col=1)\nfig.update_xaxes(title_text=\"Hotel ID\", type=\"category\", row=2, col=2)\n\nfig.update_layout(title=\"Image count per hotel\", height=550)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-21T21:46:48.9824Z","iopub.execute_input":"2022-05-21T21:46:48.982894Z","iopub.status.idle":"2022-05-21T21:46:49.25032Z","shell.execute_reply.started":"2022-05-21T21:46:48.982853Z","shell.execute_reply":"2022-05-21T21:46:49.249593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(group_df, x=\"image_count\", nbins=100, marginal=\"box\", height=350)\nfig.update_layout(title=\"Distribution of image count per hotel\")\nfig.update_traces(hovertemplate=\"Image count: %{x} <br>Hotel count: %{y}\")\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-21T21:46:49.251628Z","iopub.execute_input":"2022-05-21T21:46:49.252003Z","iopub.status.idle":"2022-05-21T21:46:50.138154Z","shell.execute_reply.started":"2022-05-21T21:46:49.251966Z","shell.execute_reply":"2022-05-21T21:46:50.137496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Example of images","metadata":{}},{"cell_type":"code","source":"def show_images(ds, title_text, n_images=5):\n    fig, ax = plt.subplots(1,5, figsize=(22,8))\n    \n    ax[0].set_ylabel(title_text)\n    \n    for i in range(5):\n        d = ds.__getitem__(i)\n        ax[i].imshow(d[\"image\"].T)","metadata":{"execution":{"iopub.status.busy":"2022-05-21T21:46:50.139415Z","iopub.execute_input":"2022-05-21T21:46:50.139763Z","iopub.status.idle":"2022-05-21T21:46:50.14608Z","shell.execute_reply.started":"2022-05-21T21:46:50.139727Z","shell.execute_reply":"2022-05-21T21:46:50.145296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train images without and with augmentations","metadata":{}},{"cell_type":"code","source":"train_dataset = HotelTrainDataset(data_df, base_transform, data_path=IMAGE_FOLDER)\nshow_images(train_dataset, 'No augmentations')\n\ntrain_dataset = HotelTrainDataset(data_df, train_transform, data_path=IMAGE_FOLDER)\nshow_images(train_dataset, 'Train augmentations')","metadata":{"execution":{"iopub.status.busy":"2022-05-21T21:46:50.14758Z","iopub.execute_input":"2022-05-21T21:46:50.148077Z","iopub.status.idle":"2022-05-21T21:46:52.502673Z","shell.execute_reply.started":"2022-05-21T21:46:50.148038Z","shell.execute_reply":"2022-05-21T21:46:52.501026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Test image example with occlusion","metadata":{}},{"cell_type":"code","source":"test_image = np.array(pil_image.open('../input/hotel-id-to-combat-human-trafficking-2022-fgvc9/test_images/abc.jpg')).astype(np.uint8)\nplt.figure(figsize=(6,6))\nplt.imshow(test_image)","metadata":{"execution":{"iopub.status.busy":"2022-05-21T21:46:52.504111Z","iopub.execute_input":"2022-05-21T21:46:52.504775Z","iopub.status.idle":"2022-05-21T21:46:52.878728Z","shell.execute_reply.started":"2022-05-21T21:46:52.504732Z","shell.execute_reply":"2022-05-21T21:46:52.878066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train and evaluate","metadata":{"id":"EMVYKwZ64zUN","papermill":{"duration":0.035036,"end_time":"2022-03-23T20:09:27.471338","exception":false,"start_time":"2022-03-23T20:09:27.436302","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def train_and_validate(args, data_df):\n    model_name = f\"embedding-model-{args.backbone_name}-{IMG_SIZE}x{IMG_SIZE}\"\n    print(model_name)\n\n    seed_everything(seed=SEED)\n\n    # split data into train and validation set\n    hotel_image_count = data_df.groupby(\"hotel_id\")[\"image_id\"].count()\n    # hotels that have more images than samples for validation\n    valid_hotels = hotel_image_count[hotel_image_count > args.val_samples]\n    # data that can be split into train and val set\n    valid_data = data_df[data_df[\"hotel_id\"].isin(valid_hotels.index)]\n    # if hotel had less than required val_samples it will be only in the train set\n    valid_df = valid_data.groupby(\"hotel_id\").sample(args.val_samples, random_state=SEED)\n    train_df = data_df[~data_df[\"image_id\"].isin(valid_df[\"image_id\"])]\n    \n\n    train_dataset = HotelTrainDataset(train_df, train_transform, data_path=IMAGE_FOLDER)\n    train_loader  = DataLoader(train_dataset, num_workers=args.num_workers, batch_size=args.batch_size, shuffle=True, drop_last=True)\n    valid_dataset = HotelTrainDataset(valid_df, val_transform, data_path=IMAGE_FOLDER)\n    valid_loader  = DataLoader(valid_dataset, num_workers=args.num_workers, batch_size=args.batch_size, shuffle=False)\n    # base dataset for image similarity search\n    base_dataset  = HotelTrainDataset(train_df, base_transform, data_path=IMAGE_FOLDER)\n    base_loader   = DataLoader(base_dataset, num_workers=args.num_workers, batch_size=args.batch_size*4, shuffle=False)\n\n    model = EmbeddingModel(args.n_classes, args.embedding_size ,args.backbone_name)\n    model = model.to(args.device)\n    \n#     criterion = losses.CosFaceLoss(num_classes=args.n_classes, embedding_size=args.embedding_size, embedding_regularizer = regularizers.RegularFaceRegularizer()).to(args.device)\n# margin=28.6, scale=64\n    criterion = losses.ArcFaceLoss(num_classes=args.n_classes, embedding_size=args.embedding_size, margin=10, scale=64).to(args.device)\n    optimizer = torch.optim.AdamW(model.parameters(), lr=args.lr)\n\n    scheduler = torch.optim.lr_scheduler.OneCycleLR(\n                    optimizer,\n                    max_lr=args.lr,\n                    epochs=args.epochs,\n                    steps_per_epoch=len(train_loader),\n                    div_factor=5,\n                    final_div_factor=1,\n                    pct_start=0.6,\n                    anneal_strategy=\"cos\",\n                )\n    \n    start_epoch = 1\n    \n    for epoch in range(start_epoch, args.epochs+1):\n        train_loss, train_score = train_epoch(args, model, train_loader, criterion, optimizer, scheduler, epoch)\n        save_checkpoint(model, scheduler, optimizer, epoch, model_name, train_loss, train_score)\n        test_classification(valid_loader, model)\n\n    test_similarity(args, base_loader, valid_loader, model)\n    \n    # generate embeddings for all train images and save them for inference\n    base_dataset   = HotelTrainDataset(data_df, base_transform, data_path=IMAGE_FOLDER)\n    base_loader    = DataLoader(base_dataset, num_workers=args.num_workers, batch_size=args.batch_size*4, shuffle=False)\n    base_embeds, _ = generate_embeddings(base_loader, model, \"Generate embeddings for all images\")\n    data_df[\"embeddings\"] = list(base_embeds)\n    data_df.to_pickle(f\"{OUTPUT_FOLDER}{model_name}_image-embeddings.pkl\")","metadata":{"execution":{"iopub.status.busy":"2022-05-21T21:46:52.880122Z","iopub.execute_input":"2022-05-21T21:46:52.880613Z","iopub.status.idle":"2022-05-21T21:46:52.89826Z","shell.execute_reply.started":"2022-05-21T21:46:52.880556Z","shell.execute_reply":"2022-05-21T21:46:52.89736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Efficientnet-b0 training","metadata":{}},{"cell_type":"code","source":"%%time \n\nclass args:\n    epochs = 9\n    lr = 1e-3\n    batch_size = 8\n    num_workers = 2\n    val_samples = 1\n    embedding_size = 4096\n    backbone_name = \"efficientnet_b1\"\n    n_classes = data_df[\"hotel_id_code\"].nunique()\n    device = ('cuda' if torch.cuda.is_available() else 'cpu')\n\ntrain_and_validate(args, data_df)","metadata":{"id":"YONzJBtG2_a0","papermill":{"duration":1686.013804,"end_time":"2022-03-23T20:37:33.519992","exception":false,"start_time":"2022-03-23T20:09:27.506188","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-21T21:46:52.899575Z","iopub.execute_input":"2022-05-21T21:46:52.900088Z","iopub.status.idle":"2022-05-21T21:48:15.124867Z","shell.execute_reply.started":"2022-05-21T21:46:52.89994Z","shell.execute_reply":"2022-05-21T21:48:15.124086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}