{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"[<img src=\"https://github.com/KevinMusgrave/pytorch-metric-learning/raw/master/docs/imgs/Logo2.png\">]()\n\n## Introduction\n\nThis notebook makes use of the fantastic library `pytorch-metric-learning` developed and maintained by Kevin Musgrave. You can find the github at the following link:\n\n- https://github.com/KevinMusgrave/pytorch-metric-learning\n\nYou can find a ton of useful metric learning modules there, along with a super friendly API for rapid training and evaluation. I recommend reading through the example notebooks because they are very well put together (below borrows from them heavily).\n\nHere we use the library to train a basic whale detector using an efficient net backbone (https://arxiv.org/abs/1905.11946) with ArcFace loss (https://arxiv.org/abs/1801.07698). This is a very straightforward example and there are many ways to improve. Here are some suggestions:\n\n- Change the train/validation split to better resemble the public LB.\n- Change the model trunk.\n- Pre-process the images by e.g. applying bounding boxes.\n- Experiment with the training proceedure.\n\nI will continue to develop this notebook over time and hopefully improve the results.\n\nAll feedback appreciated.","metadata":{}},{"cell_type":"markdown","source":"**Change Log**\n\n- Version 14: switched to b6, reduced validation set to 0.05.\n- Version 12 (LB: 0.309): switched to b4, added gradient accumulation and mixed precision training.\n- Version 9 (LB: 0.287): switched to 384x384 dataset, added training augmentation, and switched from Adam to SGD with cosine schedule.\n- Version 8 (LB: 0.245): fixed bug where same individual predicted multiple times for single image and increased the KNN search range.\n- Version 6 (LB: 0.229): switched to cropped YOLO5 input, switched to b3 model, reduced epochs, and updated logging.\n- Version 4 (LB: 0.190): initial notebook completed.","metadata":{}},{"cell_type":"markdown","source":"## Dependencies","metadata":{}},{"cell_type":"code","source":"!pip install timm\n!pip install pytorch-metric-learning[with-hooks]","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T22:04:05.69923Z","iopub.execute_input":"2022-02-26T22:04:05.699757Z","iopub.status.idle":"2022-02-26T22:04:21.253905Z","shell.execute_reply.started":"2022-02-26T22:04:05.699669Z","shell.execute_reply":"2022-02-26T22:04:21.251735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Imports","metadata":{}},{"cell_type":"code","source":"import os\nimport glob\nimport pandas as pd\nimport numpy as np\nimport logging\nimport timm\nfrom tqdm.notebook import tqdm\n\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\n\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision.io import ImageReadMode, read_image\nfrom torchvision.transforms import Compose, Lambda, Normalize, AutoAugment, AutoAugmentPolicy\n\nimport pytorch_metric_learning\nimport pytorch_metric_learning.utils.logging_presets as LP\nfrom pytorch_metric_learning.utils import common_functions\nfrom pytorch_metric_learning import losses, miners, samplers, testers, trainers\nfrom pytorch_metric_learning.utils.accuracy_calculator import AccuracyCalculator\nfrom pytorch_metric_learning.utils.inference import InferenceModel\n\nfor handler in logging.root.handlers[:]:\n    logging.root.removeHandler(handler)\n\nlogging.getLogger().setLevel(logging.INFO)\nlogging.info(\"VERSION %s\" % pytorch_metric_learning.__version__)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:21.256255Z","iopub.execute_input":"2022-02-26T22:04:21.256547Z","iopub.status.idle":"2022-02-26T22:04:22.87355Z","shell.execute_reply.started":"2022-02-26T22:04:21.256503Z","shell.execute_reply":"2022-02-26T22:04:22.872715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\ndevice","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:22.87512Z","iopub.execute_input":"2022-02-26T22:04:22.875379Z","iopub.status.idle":"2022-02-26T22:04:22.912621Z","shell.execute_reply.started":"2022-02-26T22:04:22.875341Z","shell.execute_reply":"2022-02-26T22:04:22.91155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Parameters","metadata":{}},{"cell_type":"markdown","source":"There is no logic behind these, really. Go wild.","metadata":{}},{"cell_type":"code","source":"MODEL_NAME='tf_efficientnet_b6_ns'\nN_CLASSES=15587\nOUTPUT_SIZE = 2304\nEMBEDDING_SIZE = 512\nN_EPOCH=10\nBATCH_SIZE=8\nACCUMULATION_STEPS = int(256 / BATCH_SIZE)\nMODEL_LR = 1e-3\nPCT_START=0.3\nPATIENCE=5\nN_WORKER=2\nN_NEIGHBOURS = 750","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:22.91656Z","iopub.execute_input":"2022-02-26T22:04:22.916919Z","iopub.status.idle":"2022-02-26T22:04:22.925406Z","shell.execute_reply.started":"2022-02-26T22:04:22.916881Z","shell.execute_reply":"2022-02-26T22:04:22.92456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Directories","metadata":{}},{"cell_type":"code","source":"TRAIN_DIR = '../input/jpeg-happywhale-384x384/train_images-384-384/train_images-384-384'\nTEST_DIR = '../input/jpeg-happywhale-384x384/test_images-384-384/test_images-384-384'\nLOG_DIR = \"../logs/{}\".format(MODEL_NAME)\nMODEL_DIR = \"../models/{}\".format(MODEL_NAME)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:22.926718Z","iopub.execute_input":"2022-02-26T22:04:22.927134Z","iopub.status.idle":"2022-02-26T22:04:22.935183Z","shell.execute_reply.started":"2022-02-26T22:04:22.92709Z","shell.execute_reply":"2022-02-26T22:04:22.934158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dataset","metadata":{}},{"cell_type":"markdown","source":"Create a basic dataset for loading images. \n\nSince we're planning to use pre-trained imagenet weights we need to normalize appropriately.","metadata":{}},{"cell_type":"code","source":"\nclass HappyWhaleDataset(Dataset):\n    def __init__(\n        self,\n        df: pd.DataFrame,\n        image_dir: str,\n        return_labels=True,\n    ):\n        self.df = df\n        self.images = self.df[\"image\"]\n        self.image_dir = image_dir\n        self.image_transform = Compose(\n            [\n                AutoAugment(AutoAugmentPolicy.IMAGENET),\n                Lambda(lambda x: x / 255),\n                \n            ]\n        )\n        self.return_labels = return_labels\n\n    def __len__(self):\n        return len(self.images)\n\n    def __getitem__(self, idx):\n        \n        image_path = os.path.join(self.image_dir, self.images.iloc[idx])\n        image = read_image(path=image_path)\n        image = self.image_transform(image)\n        \n        if self.return_labels:\n            label = self.df['label'].iloc[idx]\n            return image, label\n        else:\n            return image\n","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:22.936726Z","iopub.execute_input":"2022-02-26T22:04:22.937041Z","iopub.status.idle":"2022-02-26T22:04:22.946693Z","shell.execute_reply.started":"2022-02-26T22:04:22.937004Z","shell.execute_reply":"2022-02-26T22:04:22.94593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Split","metadata":{}},{"cell_type":"markdown","source":"Load in the csv:","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('../input/happy-whale-and-dolphin/train.csv')\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:22.948336Z","iopub.execute_input":"2022-02-26T22:04:22.948621Z","iopub.status.idle":"2022-02-26T22:04:23.027497Z","shell.execute_reply.started":"2022-02-26T22:04:22.948587Z","shell.execute_reply":"2022-02-26T22:04:23.026645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Add a label for the classes:","metadata":{}},{"cell_type":"code","source":"df['label'] = df.groupby('individual_id').ngroup()\ndf['label'].describe()","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:23.028852Z","iopub.execute_input":"2022-02-26T22:04:23.029108Z","iopub.status.idle":"2022-02-26T22:04:23.070655Z","shell.execute_reply.started":"2022-02-26T22:04:23.029059Z","shell.execute_reply":"2022-02-26T22:04:23.069961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Split into training and validation:","metadata":{}},{"cell_type":"code","source":"valid_proportion = 0.05\n\nvalid_df = df.sample(frac=valid_proportion, replace=False, random_state=1).copy()\ntrain_df = df[~df['image'].isin(valid_df['image'])].copy()\n\nprint(train_df.shape)\nprint(valid_df.shape)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:23.071874Z","iopub.execute_input":"2022-02-26T22:04:23.072116Z","iopub.status.idle":"2022-02-26T22:04:23.094928Z","shell.execute_reply.started":"2022-02-26T22:04:23.072084Z","shell.execute_reply":"2022-02-26T22:04:23.094174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Reset index on both since we want to use it for KNN lookups later:","metadata":{}},{"cell_type":"code","source":"train_df.reset_index(drop=True, inplace=True)\nvalid_df.reset_index(drop=True, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:23.096253Z","iopub.execute_input":"2022-02-26T22:04:23.096502Z","iopub.status.idle":"2022-02-26T22:04:23.100912Z","shell.execute_reply.started":"2022-02-26T22:04:23.096468Z","shell.execute_reply":"2022-02-26T22:04:23.099917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create our dataset objects:","metadata":{}},{"cell_type":"code","source":"train_dataset = HappyWhaleDataset(df=train_df, image_dir=TRAIN_DIR, return_labels=True)\nlen(train_dataset)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:23.102407Z","iopub.execute_input":"2022-02-26T22:04:23.102676Z","iopub.status.idle":"2022-02-26T22:04:23.113817Z","shell.execute_reply.started":"2022-02-26T22:04:23.102643Z","shell.execute_reply":"2022-02-26T22:04:23.112996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_dataset = HappyWhaleDataset(df=valid_df, image_dir=TRAIN_DIR, return_labels=True)\nlen(valid_dataset)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:23.115683Z","iopub.execute_input":"2022-02-26T22:04:23.116106Z","iopub.status.idle":"2022-02-26T22:04:23.122757Z","shell.execute_reply.started":"2022-02-26T22:04:23.116056Z","shell.execute_reply":"2022-02-26T22:04:23.121947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset_dict = {\"train\": train_dataset, \"val\": valid_dataset}","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:23.126635Z","iopub.execute_input":"2022-02-26T22:04:23.127412Z","iopub.status.idle":"2022-02-26T22:04:23.131827Z","shell.execute_reply.started":"2022-02-26T22:04:23.127371Z","shell.execute_reply":"2022-02-26T22:04:23.131044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Setup","metadata":{}},{"cell_type":"markdown","source":"We need to specify three components to build our model:\n\n- Trunk\n- Embedder\n- Loss\n\nSetup the trunk using a pre-trained model from timm:","metadata":{}},{"cell_type":"code","source":"trunk = timm.create_model(MODEL_NAME, pretrained=True)\ntrunk.classifier = common_functions.Identity()\ntrunk = trunk.to(device)\ntrunk_optimizer = optim.SGD(trunk.parameters(), lr=MODEL_LR, momentum=0.9)\ntrunk_schedule = optim.lr_scheduler.OneCycleLR(\n    trunk_optimizer,\n    max_lr=MODEL_LR,\n    total_steps = N_EPOCH * int(len(train_dataset)/BATCH_SIZE),\n    pct_start = PCT_START\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:23.133024Z","iopub.execute_input":"2022-02-26T22:04:23.133444Z","iopub.status.idle":"2022-02-26T22:04:25.961776Z","shell.execute_reply.started":"2022-02-26T22:04:23.133408Z","shell.execute_reply":"2022-02-26T22:04:25.961034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Add our embedder. This is just a linear layer that will create the embeddings for KNN:","metadata":{}},{"cell_type":"code","source":"embedder = nn.Linear(OUTPUT_SIZE, EMBEDDING_SIZE).to(device)\nembedder_optimizer = optim.SGD(trunk.parameters(), lr=MODEL_LR, momentum=0.9)\nembedder_schedule = optim.lr_scheduler.OneCycleLR(\n    embedder_optimizer,\n    max_lr=MODEL_LR,\n    total_steps = N_EPOCH * int(len(train_dataset)/BATCH_SIZE),\n    pct_start = PCT_START\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:25.96309Z","iopub.execute_input":"2022-02-26T22:04:25.963339Z","iopub.status.idle":"2022-02-26T22:04:25.98393Z","shell.execute_reply.started":"2022-02-26T22:04:25.963304Z","shell.execute_reply":"2022-02-26T22:04:25.983328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And add the loss function:","metadata":{}},{"cell_type":"code","source":"loss_func = losses.ArcFaceLoss(num_classes=N_CLASSES, embedding_size=EMBEDDING_SIZE).to(device)\nloss_optimizer = optim.SGD(trunk.parameters(), lr=MODEL_LR, momentum=0.9)\nloss_schedule = optim.lr_scheduler.OneCycleLR(\n    loss_optimizer,\n    max_lr=MODEL_LR,\n    total_steps = N_EPOCH * int(len(train_dataset)/BATCH_SIZE),\n    pct_start = PCT_START\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:25.985256Z","iopub.execute_input":"2022-02-26T22:04:25.985503Z","iopub.status.idle":"2022-02-26T22:04:26.078702Z","shell.execute_reply.started":"2022-02-26T22:04:25.985468Z","shell.execute_reply":"2022-02-26T22:04:26.077963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Setup some hooks for validation, logging and model saving at the end of the epoch:","metadata":{}},{"cell_type":"code","source":"record_keeper, _, _ = LP.get_record_keeper(LOG_DIR)\nhooks = LP.get_hook_container(record_keeper, primary_metric='mean_average_precision')","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:26.079911Z","iopub.execute_input":"2022-02-26T22:04:26.080258Z","iopub.status.idle":"2022-02-26T22:04:26.192152Z","shell.execute_reply.started":"2022-02-26T22:04:26.08022Z","shell.execute_reply":"2022-02-26T22:04:26.191425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tester = testers.GlobalEmbeddingSpaceTester(\n    end_of_testing_hook=hooks.end_of_testing_hook,\n    accuracy_calculator=AccuracyCalculator(\n        include=['mean_average_precision'],\n        device=torch.device(\"cpu\"),\n        k=5),\n    dataloader_num_workers=N_WORKER,\n    batch_size=BATCH_SIZE\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:26.193522Z","iopub.execute_input":"2022-02-26T22:04:26.193962Z","iopub.status.idle":"2022-02-26T22:04:26.19913Z","shell.execute_reply.started":"2022-02-26T22:04:26.193924Z","shell.execute_reply":"2022-02-26T22:04:26.198105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By adding the tester as an end of epoch hook in this way, it will automatically use the embedder model to generate train and validation embeddings, then for each validation embedding find the k nearest neighbours and evaluate MAP@5. This won't take into account the `new_individual` problem, but it should give us an idea of model performance on the task regardless.","metadata":{}},{"cell_type":"code","source":"end_of_epoch_hook = hooks.end_of_epoch_hook(\n    tester, \n    dataset_dict,\n    MODEL_DIR,\n    test_interval=1, \n    patience=PATIENCE, \n    splits_to_eval = [('val', ['train'])]\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:26.200471Z","iopub.execute_input":"2022-02-26T22:04:26.200974Z","iopub.status.idle":"2022-02-26T22:04:26.208986Z","shell.execute_reply.started":"2022-02-26T22:04:26.200936Z","shell.execute_reply":"2022-02-26T22:04:26.208277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Extend the `MetricLossOnly` class to include gradient accumulation and mixed precision training: ","metadata":{}},{"cell_type":"code","source":"\nclass HappyTrainer(trainers.MetricLossOnly):\n    def __init__(self, *args, accumulation_steps=10, **kwargs):\n        super().__init__(*args, **kwargs)\n        self.accumulation_steps = accumulation_steps\n\n    def forward_and_backward(self):\n        self.zero_losses()\n        self.update_loss_weights()\n        self.calculate_loss(self.get_batch())\n        self.loss_tracker.update(self.loss_weights)\n        self.backward()\n        self.clip_gradients()\n        if ((self.iteration + 1) % self.accumulation_steps == 0) or ((self.iteration + 1) == np.ceil(len(self.dataset) / self.batch_size)):\n            self.step_optimizers()\n            self.zero_grad()\n            \n    def calculate_loss(self, curr_batch):\n        data, labels = curr_batch\n        with torch.cuda.amp.autocast():\n            embeddings = self.compute_embeddings(data)\n            indices_tuple = self.maybe_mine_embeddings(embeddings, labels)\n            self.losses[\"metric_loss\"] = self.maybe_get_metric_loss(\n                embeddings, labels, indices_tuple\n            )","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:26.210426Z","iopub.execute_input":"2022-02-26T22:04:26.210933Z","iopub.status.idle":"2022-02-26T22:04:26.221662Z","shell.execute_reply.started":"2022-02-26T22:04:26.21089Z","shell.execute_reply":"2022-02-26T22:04:26.221011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, setup our trainer object:","metadata":{}},{"cell_type":"code","source":"trainer = HappyTrainer(\n    models={\"trunk\": trunk, \"embedder\": embedder},\n    optimizers={\"trunk_optimizer\": trunk_optimizer, \"embedder_optimizer\": embedder_optimizer, \"metric_loss_optimizer\": loss_optimizer},\n    batch_size=BATCH_SIZE,\n    loss_funcs={\"metric_loss\": loss_func},\n    mining_funcs={},\n    dataset=train_dataset,\n    dataloader_num_workers=N_WORKER,\n    end_of_epoch_hook=end_of_epoch_hook,\n    lr_schedulers={\n        'trunk_scheduler_by_iteration': trunk_schedule,\n        'embedder_scheduler_by_iteration': embedder_schedule,\n        'metric_loss_scheduler_by_iteration': loss_schedule,\n    },\n    accumulation_steps=ACCUMULATION_STEPS\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:26.224591Z","iopub.execute_input":"2022-02-26T22:04:26.224808Z","iopub.status.idle":"2022-02-26T22:04:26.235039Z","shell.execute_reply.started":"2022-02-26T22:04:26.224781Z","shell.execute_reply":"2022-02-26T22:04:26.234211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Training","metadata":{}},{"cell_type":"markdown","source":"Train the model:","metadata":{}},{"cell_type":"code","source":"trainer.train(num_epochs=N_EPOCH)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T22:04:26.236217Z","iopub.execute_input":"2022-02-26T22:04:26.236617Z","iopub.status.idle":"2022-02-26T22:05:10.133418Z","shell.execute_reply.started":"2022-02-26T22:04:26.236581Z","shell.execute_reply":"2022-02-26T22:05:10.132313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inference (validation set)","metadata":{}},{"cell_type":"markdown","source":"Here we want to use the validation set to help us choose the appropriate distance threshold between our query and reference images after which we classify the former as a `new_individual`. To do so, we loop through the validation set for a number of thresholds and find that which maximises our MAP@5.","metadata":{}},{"cell_type":"markdown","source":"Load in the best weights:","metadata":{}},{"cell_type":"code","source":"logging.getLogger().setLevel(logging.WARNING)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_trunk_weights = glob.glob('../models/{}/trunk_best*.pth'.format(MODEL_NAME))[0]\ntrunk.load_state_dict(torch.load(best_trunk_weights))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_embedder_weights = glob.glob('../models/{}/embedder_best*.pth'.format(MODEL_NAME))[0]\nembedder.load_state_dict(torch.load(best_embedder_weights))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Setup the inference model object to easily generate embeddings and find nearest neighbours:","metadata":{}},{"cell_type":"code","source":"inference_model = InferenceModel(\n    trunk=trunk,\n    embedder=embedder,\n    normalize_embeddings=True,\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Train this on the training data:","metadata":{}},{"cell_type":"code","source":"inference_model.train_knn(train_dataset)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Loop through the validation data and loop through to find k nearest neighbours:","metadata":{}},{"cell_type":"code","source":"valid_dataloader = DataLoader(valid_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=N_WORKER, pin_memory=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_labels_list = []\nvalid_distance_list = []\nvalid_indices_list = []\n\nfor images, labels in tqdm(valid_dataloader):\n\n    distances, indices = inference_model.get_nearest_neighbors(images, k=N_NEIGHBOURS)\n    valid_labels_list.append(labels)\n    valid_distance_list.append(distances)\n    valid_indices_list.append(indices)\n\nvalid_labels = torch.cat(valid_labels_list, dim=0).cpu().numpy()\nvalid_distances = torch.cat(valid_distance_list, dim=0).cpu().numpy()\nvalid_indices = torch.cat(valid_indices_list, dim=0).cpu().numpy()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have the indices of the nearest neighbours in our training set, so setup the lookups to return the `individual_id`:","metadata":{}},{"cell_type":"code","source":"new_whale_idx = -1\n\ntrain_labels = train_df['individual_id'].unique()\ntrain_idx_lookup = train_df['individual_id'].copy().to_dict()\ntrain_idx_lookup[-1] = 'new_individual'\n\nvalid_class_lookup = valid_df.set_index('label')['individual_id'].copy().to_dict()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Loop through a range of thresholds and find which maximises our MAP@5:","metadata":{}},{"cell_type":"code","source":"thresholds = [np.quantile(valid_distances, q=q) for q in np.arange(0, 1.0, 0.01)]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"results = []\n\nfor threshold in tqdm(thresholds):\n\n    prediction_list = []\n    running_map=0\n\n    for i in range(len(valid_distances)):\n\n        pred_knn_idx = valid_indices[i, :].copy()  \n        insert_idx = np.where(valid_distances[i, :] > threshold) \n\n        if insert_idx[0].size != 0:  \n            pred_knn_idx = np.insert(pred_knn_idx, np.min(insert_idx[0]), new_whale_idx) \n\n        predicted_label_list = []\n        \n        for predicted_idx in pred_knn_idx:\n            predicted_label = train_idx_lookup[predicted_idx]\n            if len(predicted_label_list) == 5:\n                break\n            if (predicted_label == 'new_individual') | (predicted_label not in predicted_label_list):\n                predicted_label_list.append(predicted_label)\n\n        gt = valid_class_lookup[valid_labels[i]]\n\n        if gt not in train_labels:\n            gt = \"new_individual\"\n\n        precision_vals = []\n\n        for j in range(5):\n            if predicted_label_list[j] == gt:\n                precision_vals.append(1/(j+1))\n            else:\n                precision_vals.append(0)\n\n        running_map += np.max(precision_vals)\n\n    results.append([threshold, running_map / len(valid_distances)])\n\nresults_df = pd.DataFrame(results, columns=['threshold','map5'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"results_df = results_df.sort_values(by='map5', ascending=False).reset_index(drop=True)\nresults_df.head(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Grab the best result:","metadata":{}},{"cell_type":"code","source":"threshold = results_df.loc[0, 'threshold']\nthreshold","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inference (test set)","metadata":{}},{"cell_type":"markdown","source":"We want to make sure we use both our training and validation images for comparison. Combine the two dataframes and add a new dataset: ","metadata":{}},{"cell_type":"code","source":"combined_df = pd.concat([train_df, valid_df], axis=0).reset_index(drop=True)\ncombined_dataset = HappyWhaleDataset(df=combined_df, image_dir=TRAIN_DIR, return_labels=True)\nlen(combined_dataset)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Re-train the KNN model on this:","metadata":{}},{"cell_type":"code","source":"inference_model.train_knn(combined_dataset)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Grab the submission file:","metadata":{}},{"cell_type":"code","source":"test_df = pd.read_csv('../input/happy-whale-and-dolphin/sample_submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create our dataset and dataloader objects for the test set:","metadata":{}},{"cell_type":"code","source":"test_dataset = HappyWhaleDataset(df=test_df, image_dir=TEST_DIR, return_labels=False)\nlen(test_dataset)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_dataloader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=N_WORKER, pin_memory=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Find the k nearest neighbours in our combined dataset:","metadata":{}},{"cell_type":"code","source":"test_distance_list = []\ntest_indices_list = []\n\nfor images in tqdm(test_dataloader):\n\n    distances, indices = inference_model.get_nearest_neighbors(images, k=N_NEIGHBOURS)\n    test_distance_list.append(distances)\n    test_indices_list.append(indices)\n\ntest_distances = torch.cat(test_distance_list, dim=0).cpu().numpy()\ntest_indices = torch.cat(test_indices_list, dim=0).cpu().numpy()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Prepare the labels for lookup based on index:","metadata":{}},{"cell_type":"code","source":"combined_idx_lookup = combined_df['individual_id'].copy().to_dict()\ncombined_idx_lookup[-1] = 'new_individual'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Loop through applying the threshold we found earlier to insert `new_individual`:","metadata":{}},{"cell_type":"code","source":"results = []\n\nprediction_list = []\n\nfor i in range(len(test_distances)):\n\n    pred_knn_idx = test_indices[i, :].copy() \n    insert_idx = np.where(test_distances[i, :] > threshold)  \n\n    if insert_idx[0].size != 0:  \n        pred_knn_idx = np.insert(pred_knn_idx, np.min(insert_idx[0]), new_whale_idx)  \n\n    predicted_label_list = []\n\n    for predicted_idx in pred_knn_idx:\n        predicted_label = combined_idx_lookup[predicted_idx]\n        if len(predicted_label_list) == 5:\n            break\n        if (predicted_label == 'new_individual') | (predicted_label not in predicted_label_list):\n            predicted_label_list.append(predicted_label)\n\n    prediction_list.append(predicted_label_list)\n\nprediction_df = pd.DataFrame(prediction_list)\nprediction_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create the prediction label:","metadata":{}},{"cell_type":"code","source":"prediction_df['predictions'] = prediction_df[0].astype(str) + ' ' + prediction_df[1].astype(str) + ' ' + prediction_df[2 ].astype(str) + ' ' + prediction_df[3].astype(str) + ' ' + prediction_df[4].astype(str)\nprediction_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Attach this to the submission:","metadata":{}},{"cell_type":"code","source":"submission = pd.read_csv('../input/happy-whale-and-dolphin/sample_submission.csv')\nsubmission['predictions'] = prediction_df['predictions']\nsubmission.head(1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Save our submission:","metadata":{}},{"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}