{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n\n

- Full Training Pipeline - Step by Step Guide with Explanation -

\n\n> 🐳 **Competition Goal:** Identify and group all images that contain the same individual through time.\n\n> 🙏 **Inspiration**: A super huge thank you to [Debarshi Chanda](https://www.kaggle.com/debarshichanda) and his very [clean comprehensive notebook](https://www.kaggle.com/debarshichanda/pytorch-arcface-gem-pooling-starter) that helped me put a start to this competition.\n\n### ⬇ Libraries\n\n🐳 **What is `timm`?** - It is a library that gathers all **PyTorch Image Models**, for ease of access and convenience. The [full documentation can be found here](https://rwightman.github.io/pytorch-image-models/models/).\n* **Feature Extraction** - All of the [models in timm](https://rwightman.github.io/pytorch-image-models/feature_extraction/) have consistent mechanisms for obtaining various types of features from the model for tasks besides classification. This will help us when extracting the embeddings from the pretrained algorithms (aka the backbone).\n* **Models** - I am also leaving here a [link](https://rwightman.github.io/pytorch-image-models/results/) to all the models and *their names* that are currently available within the library.","metadata":{}},{"cell_type":"code","source":"# Helpful Installs\n!pip install timm","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2022-03-12T11:39:29.963336Z","iopub.execute_input":"2022-03-12T11:39:29.963775Z","iopub.status.idle":"2022-03-12T11:39:39.428003Z","shell.execute_reply.started":"2022-03-12T11:39:29.963743Z","shell.execute_reply":"2022-03-12T11:39:39.427148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Libraries\nimport os\nimport gc\nimport wandb\nimport time\nimport random\nimport math\nfrom scipy import spatial\nfrom tqdm import tqdm\nimport warnings\nimport cv2\nimport pandas as pd\nimport numpy as np\nfrom numpy import dot, sqrt\nimport seaborn as sns\nimport matplotlib as mpl\nimport matplotlib.patches as patches\nimport matplotlib.pyplot as plt\nfrom IPython.display import display_html\n\nfrom sklearn.model_selection import StratifiedKFold\nimport torch\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.optim import Adam, lr_scheduler\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.cuda.amp import GradScaler, autocast\n\nfrom albumentations.pytorch import transforms\nimport albumentations\nimport timm\n\nfrom cuml.neighbors import NearestNeighbors\nfrom sklearn.preprocessing import normalize\n\n# Environment check\nwarnings.filterwarnings(\"ignore\")\nos.environ[\"WANDB_SILENT\"] = \"true\"\nCONFIG = {'competition': 'happywhale', '_wandb_kernel': 'aot'}\n\n# Custom colors\nclass clr:\n S = '\\033[1m' + '\\033[96m'\n E = '\\033[0m'\n \nmy_colors = [\"#21295C\", \"#1F4E78\", \"#1C7293\", \"#73ABAF\", \"#C9E4CA\", \"#87BBA2\", \"#618E83\", \"#3B6064\"]\nprint(clr.S+\"Notebook Color Scheme:\"+clr.E)\nsns.palplot(sns.color_palette(my_colors))\nplt.show()","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2022-03-12T11:39:39.430578Z","iopub.execute_input":"2022-03-12T11:39:39.430845Z","iopub.status.idle":"2022-03-12T11:39:47.205411Z","shell.execute_reply.started":"2022-03-12T11:39:39.430809Z","shell.execute_reply":"2022-03-12T11:39:47.204334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 🐝 W&B Fork & Run\n\nIn order to run this notebook you will need to input your own **secret API key** within the `! wandb login $secret_value_0` line. \n\n🐝**How do you get your own API key?**\n\nSuper simple! Go to **https://wandb.ai/site** -> Login -> Click on your profile in the top right corner -> Settings -> Scroll down to API keys -> copy your very own key (for more info check [this amazing notebook for ML Experiment Tracking on Kaggle](https://www.kaggle.com/ayuraj/experiment-tracking-with-weights-and-biases)).\n\n
","metadata":{}},{"cell_type":"code","source":"# 🐝 Secrets\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nsecret_value_0 = user_secrets.get_secret(\"wandb\")\n\n! wandb login $secret_value_0","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:39:47.210135Z","iopub.execute_input":"2022-03-12T11:39:47.210520Z","iopub.status.idle":"2022-03-12T11:39:49.350234Z","shell.execute_reply.started":"2022-03-12T11:39:47.210477Z","shell.execute_reply":"2022-03-12T11:39:49.349286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ⬇ Helper Functions","metadata":{}},{"cell_type":"code","source":"def plot_loss_graph(train_losses, valid_losses, epoch, fold):\n '''Lineplot of the training/validation losses.'''\n \n fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 2.5))\n fig.suptitle(f\"Fold {fold} | Epoch {epoch}\", fontsize=12, y=1.05)\n axes = [ax1, ax2]\n data = [train_losses, valid_losses]\n sns.lineplot(y=train_losses, x=range(len(train_losses)),\n lw=2.3, ls=\":\", color=my_colors[1], ax=ax1)\n sns.lineplot(y=valid_losses, x=range(len(valid_losses)),\n lw=2.3, ls=\"-\", color=my_colors[6], ax=ax2)\n for ax, t, d in zip(axes, [\"Train\", \"Valid\"], data):\n ax.set_title(f\"{t} Evolution\", size=12, weight='bold')\n ax.set_xlabel(\"Iteration\", weight='bold', size=9)\n ax.set_ylabel(\"Loss\", weight='bold', size=9)\n ax.tick_params(labelsize=9)\n plt.show()\n \n \ndef show_values_on_bars(axs, h_v=\"v\", space=0.4):\n '''Plots the value at the end of the a seaborn barplot.\n axs: the ax of the plot\n h_v: weather or not the barplot is vertical/ horizontal'''\n \n def _show_on_single_plot(ax):\n if h_v == \"v\":\n for p in ax.patches:\n _x = p.get_x() + p.get_width() / 2\n _y = p.get_y() + p.get_height()\n value = int(p.get_height())\n ax.text(_x, _y, format(value, ','), ha=\"center\") \n elif h_v == \"h\":\n for p in ax.patches:\n _x = p.get_x() + p.get_width() + float(space)\n _y = p.get_y() + p.get_height()\n value = int(p.get_width())\n ax.text(_x, _y, format(value, ','), ha=\"left\")\n\n if isinstance(axs, np.ndarray):\n for idx, ax in np.ndenumerate(axs):\n _show_on_single_plot(ax)\n else:\n _show_on_single_plot(axs)\n\n\n# === 🐝 W&B ===\ndef save_dataset_artifact(run_name, artifact_name, path):\n '''Saves dataset to W&B Artifactory.\n run_name: name of the experiment\n artifact_name: under what name should the dataset be stored\n path: path to the dataset'''\n \n run = wandb.init(project='happywhale', \n name=run_name, \n config=CONFIG)\n artifact = wandb.Artifact(name=artifact_name, \n type='dataset')\n artifact.add_file(path)\n\n wandb.log_artifact(artifact)\n wandb.finish()\n print(\"Artifact has been saved successfully.\")\n \n \ndef create_wandb_plot(x_data=None, y_data=None, x_name=None, y_name=None, title=None, log=None, plot=\"line\"):\n '''Create and save lineplot/barplot in W&B Environment.\n x_data & y_data: Pandas Series containing x & y data\n x_name & y_name: strings containing axis names\n title: title of the graph\n log: string containing name of log'''\n \n data = [[label, val] for (label, val) in zip(x_data, y_data)]\n table = wandb.Table(data=data, columns = [x_name, y_name])\n \n if plot == \"line\":\n wandb.log({log : wandb.plot.line(table, x_name, y_name, title=title)})\n elif plot == \"bar\":\n wandb.log({log : wandb.plot.bar(table, x_name, y_name, title=title)})\n elif plot == \"scatter\":\n wandb.log({log : wandb.plot.scatter(table, x_name, y_name, title=title)})\n \n \ndef create_wandb_hist(x_data=None, x_name=None, title=None, log=None):\n '''Create and save histogram in W&B Environment.\n x_data: Pandas Series containing x values\n x_name: strings containing axis name\n title: title of the graph\n log: string containing name of log'''\n \n data = [[x] for x in x_data]\n table = wandb.Table(data=data, columns=[x_name])\n wandb.log({log : wandb.plot.histogram(table, x_name, title=title)})","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2022-03-12T11:39:49.352624Z","iopub.execute_input":"2022-03-12T11:39:49.352835Z","iopub.status.idle":"2022-03-12T11:39:49.376685Z","shell.execute_reply.started":"2022-03-12T11:39:49.352808Z","shell.execute_reply":"2022-03-12T11:39:49.375608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 🌱 Setting the Seed & Device","metadata":{}},{"cell_type":"code","source":"def set_seed(seed = 1234):\n '''\n 🌱src:https://www.kaggle.com/andradaolteanu/melanoma-competiton-aug-resnet-effnet-lb-0-91\n Sets the seed of the entire notebook so results are the same every time we run.\n This is for REPRODUCIBILITY.'''\n np.random.seed(seed)\n random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n # When running on the CuDNN backend, two further options must be set\n torch.backends.cudnn.deterministic = True\n # Set a fixed value for the hash seed\n os.environ['PYTHONHASHSEED'] = str(seed)\n \nset_seed()\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint(clr.S+'Device available now:'+clr.E, device)","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:39:49.378266Z","iopub.execute_input":"2022-03-12T11:39:49.378514Z","iopub.status.idle":"2022-03-12T11:39:49.390537Z","shell.execute_reply.started":"2022-03-12T11:39:49.378478Z","shell.execute_reply":"2022-03-12T11:39:49.389615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Data & Parameters\n\n🐳 **The Data:** I am using my preprocessed `.csv` file for training and test data (for more info on how I made them [check out this notebook](https://www.kaggle.com/andradaolteanu/whales-dolphins-effnet-embedding-cos-distance)). I am also using the [images dataset](https://www.kaggle.com/phalanx/whale2-cropped-dataset) that has been resized and cropped by [phalanx](https://www.kaggle.com/phalanx) using the Detic methodology ([Discussion and Explanation here](https://www.kaggle.com/c/happy-whale-and-dolphin/discussion/305503))","metadata":{}},{"cell_type":"code","source":"# --------- INITIAL PARAMETERS ---------\nTRAIN_FOLDER = \"../input/whale2-cropped-dataset/cropped_train_images/cropped_train_images/\"\nTEST_FOLDER = \"../input/whale2-cropped-dataset/cropped_test_images/cropped_test_images/\"\n\n# Set some parameters for sanity checks & experimenting\nN_SPLITS = 5\nBATCH_SIZE = 16\nMODEL_NAME = 'efficientnet_b0'\nNUM_CLASSES = 15587\nNO_NEURONS = 250\nEMBEDDING_SIZE = 128\n# -------------------------------------","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:39:49.391985Z","iopub.execute_input":"2022-03-12T11:39:49.392249Z","iopub.status.idle":"2022-03-12T11:39:49.399363Z","shell.execute_reply.started":"2022-03-12T11:39:49.392208Z","shell.execute_reply":"2022-03-12T11:39:49.398374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Import the data\ntrain = pd.read_csv(\"../input/happywhale-2022/train.csv\")\ntest = pd.read_csv(\"../input/happywhale-2022/test.csv\")\n\n# Update path to new image folders\ntrain[\"path\"] = TRAIN_FOLDER + train[\"image\"]\ntest[\"path\"] = TEST_FOLDER + test[\"image\"]\n\nprint(clr.S+\"TRAIN:\"+clr.E)\ndisplay_html(train.head())\nprint(\"\\n\", clr.S+\"TEST:\"+clr.E)\ndisplay_html(test.head())","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:39:49.400487Z","iopub.execute_input":"2022-03-12T11:39:49.401376Z","iopub.status.idle":"2022-03-12T11:39:50.265642Z","shell.execute_reply.started":"2022-03-12T11:39:49.401337Z","shell.execute_reply":"2022-03-12T11:39:50.264884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. PyTorch Dataset","metadata":{}},{"cell_type":"code","source":"class HappyWhaleDataset(Dataset):\n \n def __init__(self, csv, trainFlag):\n '''Module to create the PyTorch Dataset.\n csv: full dataframe (train or test)\n trainFlag: True if csv is a training/validation dataset, False otherwise\n return: image and class target if trainFlag, otherwise only image'''\n \n self.csv = csv\n self.trainFlag = trainFlag\n if self.trainFlag:\n self.transform = albumentations.Compose([\n albumentations.Resize(128, 128),\n albumentations.HorizontalFlip(),\n albumentations.VerticalFlip(),\n albumentations.Rotate(),\n albumentations.Normalize(),\n # B&W?\n ])\n else:\n self.transform = albumentations.Compose([\n albumentations.Normalize()\n ])\n\n \n def __len__(self):\n return self.csv.shape[0]\n\n \n def __getitem__(self, index):\n # Get data\n row = self.csv.iloc[index]\n \n # Read and transform the image\n image = cv2.imread(row.path)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n \n transformed_img = self.transform(image=image)['image'].astype(np.float32)\n image = transformed_img.transpose(2, 0, 1)\n image = torch.tensor(image) \n\n if self.trainFlag:\n # Retrieve the target group\n target = torch.tensor(row.individual_key)\n return image, target\n \n else:\n return image","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:39:50.269749Z","iopub.execute_input":"2022-03-12T11:39:50.271026Z","iopub.status.idle":"2022-03-12T11:39:50.283242Z","shell.execute_reply.started":"2022-03-12T11:39:50.270987Z","shell.execute_reply":"2022-03-12T11:39:50.282530Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Dataset Example\n\n🐳 In the below example we have:\n* 4 batches in total for 12 images, each batch having a size of 3 images => 3 images/batch\n* The output per each batch:\n * a 4D tensor of 3 images with size 3 channels x 128 width x 128 height\n * a 1D tensor of size 3 -> one target class per each image","metadata":{}},{"cell_type":"code","source":"# Example for the Dataset data\nexample_dataset = HappyWhaleDataset(train.head(12), trainFlag=True)\nexample_loader = DataLoader(example_dataset, batch_size=3)\n\nfor k, (image, target) in enumerate(example_loader):\n print(clr.S+f\"--- Batch {k} ---\"+clr.E)\n print(\"Image Shape:\", image.shape)\n print(\"Target:\", target, \"\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:39:50.287281Z","iopub.execute_input":"2022-03-12T11:39:50.288010Z","iopub.status.idle":"2022-03-12T11:39:50.613845Z","shell.execute_reply.started":"2022-03-12T11:39:50.287935Z","shell.execute_reply":"2022-03-12T11:39:50.613138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. PyTorch Helper Functions\n\n## 3.1 Generalized Mean (or GeM)\n\n🐳 There is a great article [here](https://amaarora.github.io/2020/08/30/gempool.html) from Aman Arora that explains in detail the mathematics behind GeM Pooling. A notebook comparison between GeM Pooling and Average Pooling can also be found [here](https://github.com/amaarora/amaarora.github.io/blob/master/nbs/GeM%20Pooling.ipynb).\n\nIn short, an image has 3 dimensions: `K x H x W`, where:\n* K: the number of channels\n* H: the image height\n* W: the image width\n\nLet `Xk` be the **spatial feature map activation**, then the difference between *Max Pooling*, *Average Pooling* and *GeM Pooling* is the following:\n\n
\n\n🐳 The pooling parameter `pk` can be *set* or *learned*, since this operation can be learned during back-propagation. In other words, *GeM Pooling* can also be trainable.","metadata":{}},{"cell_type":"code","source":"# src: https://amaarora.github.io/2020/08/30/gempool.html\n\nclass GeM(nn.Module):\n def __init__(self, p=3, eps=1e-6):\n super(GeM,self).__init__()\n self.p = nn.Parameter(torch.ones(1)*p)\n self.eps = eps\n\n def forward(self, x):\n return self.gem(x, p=self.p, eps=self.eps)\n \n def gem(self, x, p=3, eps=1e-6):\n # Applies 2D average-pooling operation in kH * kW regions by step size\n return F.avg_pool2d(x.clamp(min=eps).pow(p), (x.size(-2), x.size(-1))).pow(1./p)\n \n def __repr__(self):\n return self.__class__.__name__ + '(' + 'p=' + '{:.4f}'.format(self.p.data.tolist()[0]) + ', ' + 'eps=' + str(self.eps) + ')'","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:39:50.619047Z","iopub.execute_input":"2022-03-12T11:39:50.619397Z","iopub.status.idle":"2022-03-12T11:39:50.631305Z","shell.execute_reply.started":"2022-03-12T11:39:50.619361Z","shell.execute_reply":"2022-03-12T11:39:50.630493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.2 Additive Angular Margin Loss (or ArcFace)\n\n🐳 **ArcFace**, or [Additive Angular Margin Loss](https://paperswithcode.com/method/arcface#:~:text=ArcFace%2C%20or%20Additive%20Angular%20Margin,traditionally%20used%20in%20these%20tasks.), *is a loss function* used in face recognition tasks.\n\nThe `softmax` is traditionally used in these tasks. However, the softmax loss function does not *explicitly optimise* the feature embedding to enforce **higher similarity for intraclass samples** and **diversity for inter-class samples** - in other words? We want the ambeddings that are super similar to be VERY CLOSE to each-other and the embeddings that are different to be VERY FAR from each-other:\n\n
","metadata":{}},{"cell_type":"code","source":"# src: https://github.com/lyakaap/Landmark2019-1st-and-3rd-Place-Solution/blob/master/src/modeling/metric_learning.py\n\nclass ArcMarginProduct(nn.Module):\n def __init__(self, in_features, out_features, s=30.0, \n m=0.50, easy_margin=False, ls_eps=0.0):\n '''\n in_features: dimension of the input\n out_features: dimension of the last layer (in our case the classification)\n s: norm of input feature\n m: margin\n ls_eps: label smoothing'''\n \n super(ArcMarginProduct, self).__init__()\n self.in_features, self.out_features = in_features, out_features\n self.s = s\n self.m = m\n self.ls_eps = ls_eps\n self.weight = nn.Parameter(torch.FloatTensor(out_features, in_features))\n # Fills the input `Tensor` with values according to the method described in\n # `Understanding the difficulty of training deep feedforward neural networks`\n # Glorot, X. & Bengio, Y. (2010)\n # using a uniform distribution.\n nn.init.xavier_uniform_(self.weight)\n\n self.easy_margin = easy_margin\n self.cos_m, self.sin_m = math.cos(m), math.sin(m)\n self.th = math.cos(math.pi - m)\n self.mm = math.sin(math.pi - m) * m\n\n def forward(self, input, label):\n # --------------------------- cos(theta) & phi(theta) ---------------------\n cosine = F.linear(F.normalize(input), F.normalize(self.weight))\n sine = torch.sqrt(1.0 - torch.pow(cosine, 2))\n phi = cosine * self.cos_m - sine * self.sin_m\n if self.easy_margin:\n phi = torch.where(cosine > 0, phi, cosine)\n else:\n phi = torch.where(cosine > self.th, phi, cosine - self.mm)\n # --------------------------- convert label to one-hot ---------------------\n one_hot = torch.zeros(cosine.size()).to(device)\n one_hot.scatter_(1, label.view(-1, 1).long(), 1)\n if self.ls_eps > 0:\n one_hot = (1 - self.ls_eps) * one_hot + self.ls_eps / self.out_features\n # -------------torch.where(out_i = {x_i if condition_i else y_i) ------------\n output = (one_hot * phi) + ((1.0 - one_hot) * cosine)\n output *= self.s\n\n return output","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:39:50.632893Z","iopub.execute_input":"2022-03-12T11:39:50.633430Z","iopub.status.idle":"2022-03-12T11:39:50.658847Z","shell.execute_reply.started":"2022-03-12T11:39:50.633394Z","shell.execute_reply":"2022-03-12T11:39:50.657907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. The PyTorch Model","metadata":{}},{"cell_type":"code","source":"class HappyWhaleModel(nn.Module):\n def __init__(self, modelName, numClasses, noNeurons, embeddingSize):\n \n super(HappyWhaleModel, self).__init__()\n # Retrieve pretrained weights\n self.backbone = timm.create_model(modelName, pretrained=True)\n # Save the number features from the backbone\n ### different models have different numbers e.g. EffnetB3 has 1536\n backbone_features = self.backbone.classifier.in_features\n self.backbone.classifier = nn.Identity() # ?????\n self.backbone.global_pool = nn.Identity() # ?????\n self.gem = GeM()\n # Embedding layer (what we actually need)\n self.embedding = nn.Sequential(nn.Linear(backbone_features, noNeurons),\n nn.BatchNorm1d(noNeurons),\n nn.ReLU(),\n nn.Dropout(p=0.2),\n \n nn.Linear(noNeurons, embeddingSize),\n nn.BatchNorm1d(embeddingSize),\n nn.ReLU(),\n nn.Dropout(p=0.2))\n self.arcface = ArcMarginProduct(in_features=embeddingSize, \n out_features=numClasses,\n s=30.0, m=0.50, easy_margin=False, ls_eps=0.0)\n \n \n def forward(self, image, target=None, prints=False):\n '''If there is a target it means that the model is training on the dataset.\n If there is no target, that means the model is predicting on the test dataset.\n In this case we would skip the ArcFace layer and return only the image embeddings.\n '''\n \n features = self.backbone(image)\n # flatten transforms from e.g.: [3, 1536, 1, 1] to [3, 1536]\n gem_pool = self.gem(features).flatten(1)\n embedding = self.embedding(gem_pool)\n if target != None:\n out = self.arcface(embedding, target)\n \n if prints:\n print(clr.S+\"0. IN:\", \"image shape:\"+clr.E, image.shape, \"target:\", target)\n print(clr.S+\"1. Backbone Output:\"+clr.E, features.shape)\n print(clr.S+\"2. GeM Pool Output:\"+clr.E, gem_pool.shape)\n print(clr.S+\"3. Embedding Output:\"+clr.E, embedding.shape)\n if target != None:\n print(clr.S+\"4. ArcFace Output:\"+clr.E, out.shape)\n \n if target != None:\n return out, embedding\n else:\n return embedding","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:39:50.660626Z","iopub.execute_input":"2022-03-12T11:39:50.660939Z","iopub.status.idle":"2022-03-12T11:39:50.683308Z","shell.execute_reply.started":"2022-03-12T11:39:50.660898Z","shell.execute_reply":"2022-03-12T11:39:50.682543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model Example\n\n> 🐳 **Note**: Below you can find a simple schema on what the `HappyWhaleModel()` does:\n
","metadata":{}},{"cell_type":"code","source":"# Create an example model - Effnet\nmodel_example = HappyWhaleModel(MODEL_NAME, NUM_CLASSES, NO_NEURONS, EMBEDDING_SIZE).to(device)","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:39:50.689813Z","iopub.execute_input":"2022-03-12T11:39:50.691264Z","iopub.status.idle":"2022-03-12T11:39:57.621959Z","shell.execute_reply.started":"2022-03-12T11:39:50.691222Z","shell.execute_reply":"2022-03-12T11:39:57.621197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Criterion\ncriterion_example = nn.CrossEntropyLoss()\n\n# We'll use previous datasets & dataloader\nfor k, (image, target) in enumerate(example_loader):\n print(clr.S+f\"=== Batch {k} ===\"+clr.E)\n image, target = image.to(device), target.to(device)\n out, _ = model_example(image, target, prints=True)\n loss = criterion_example(out, target)\n print(clr.S+'--- LOSS ---'+clr.E, loss.item(), \"\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:39:57.623256Z","iopub.execute_input":"2022-03-12T11:39:57.623488Z","iopub.status.idle":"2022-03-12T11:40:03.511390Z","shell.execute_reply.started":"2022-03-12T11:39:57.623456Z","shell.execute_reply":"2022-03-12T11:40:03.510674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del model_example\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:40:03.512687Z","iopub.execute_input":"2022-03-12T11:40:03.513299Z","iopub.status.idle":"2022-03-12T11:40:03.727807Z","shell.execute_reply.started":"2022-03-12T11:40:03.513261Z","shell.execute_reply":"2022-03-12T11:40:03.727120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Training Classifier Model\n\nThe problem at hand could be divided into 3 steps:\n1. Train a `classification` model on the data\n2. Extract the `embedding layer` right before the classification layer\n3. Use these embeddings in order to group the individuals together\n\n## 5.1 Training Prepping\n\n## I. Data Loaders","metadata":{}},{"cell_type":"code","source":"def get_loaders(df, train_i, valid_i):\n '''\n df: the full initial dataframe\n train_i, valid_i: list of indexes for train and validation split\n VALID_PERC: percentage of how much of valid data to preserve - leave 1 for full dataset\n return: train_loader and valid_loader\n '''\n \n train_df = df.iloc[train_i, :]\n # To go quicker through validation\n valid_df = df.iloc[valid_i, :].sample(int(len(valid_i)*VALID_PERC), random_state=23)\n\n # Datasets & Dataloader\n train_dataset = HappyWhaleDataset(train_df, trainFlag=True)\n valid_dataset = HappyWhaleDataset(valid_df, trainFlag=True)\n\n train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True)\n valid_loader = DataLoader(valid_dataset, batch_size=BATCH_SIZE, shuffle=False)\n \n return train_loader, valid_loader","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:40:03.729155Z","iopub.execute_input":"2022-03-12T11:40:03.729467Z","iopub.status.idle":"2022-03-12T11:40:03.739707Z","shell.execute_reply.started":"2022-03-12T11:40:03.729427Z","shell.execute_reply":"2022-03-12T11:40:03.738929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## II. Model, Optimizer, Criterion\n\n🐳 **How to adjust learning rate?**: `torch.optim.lr_scheduler` provides [several methods](https://pytorch.org/docs/stable/optim.html) to adjust the learning rate based on the number of epochs.\n\nHere is a full list of all PyTorch schedulers: https://pytorch.org/docs/stable/optim.html","metadata":{}},{"cell_type":"code","source":"def get_model_optimizer_criterion():\n \n model = HappyWhaleModel(MODEL_NAME, NUM_CLASSES, NO_NEURONS, EMBEDDING_SIZE).to(device)\n optimizer = Adam(model.parameters(), lr=LR, weight_decay=WEIGHT_DECAY, amsgrad=False)\n scheduler = lr_scheduler.CosineAnnealingLR(optimizer, T_max=T_MAX, eta_min=MIN_LR)\n criterion = nn.CrossEntropyLoss()\n \n return model, optimizer, scheduler, criterion","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:40:03.741119Z","iopub.execute_input":"2022-03-12T11:40:03.741665Z","iopub.status.idle":"2022-03-12T11:40:03.748724Z","shell.execute_reply.started":"2022-03-12T11:40:03.741625Z","shell.execute_reply":"2022-03-12T11:40:03.748010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## III. Training Function\n\n> Below there is a complete step by step schema of what the `train_pipeline()` function does:\n
\n\n### ⬇️ Full Training Pipeline and 🐝W&B logging below","metadata":{}},{"cell_type":"code","source":"def train_pipeline(train):\n '''\n train: the full training dataframe (to be split in train data & valid data)\n '''\n\n s = time.time()\n \n # 🐝 W&B Initialize \n RUN_CONFIG = CONFIG.copy()\n params = dict(model=MODEL_NAME, epochs=EPOCHS, split=N_SPLITS, \n batch=BATCH_SIZE, lr=LR, weight_decay=WEIGHT_DECAY,\n t_max=T_MAX, min_lr=MIN_LR)\n RUN_CONFIG.update(params)\n run = wandb.init(project='happywhale', name=RUN_NAME, config=RUN_CONFIG)\n\n \n # === CV Split ===\n skf = StratifiedKFold(n_splits=N_SPLITS)\n skf_splits = skf.split(X=train, y=train[\"individual_key\"])\n\n\n for fold, (train_i, valid_i) in enumerate(skf_splits):\n\n print(\"~\"*25)\n print(\"~\"*8, clr.S+f\"FOLD {fold}\"+clr.E, \"~\"*8)\n print(\"~\"*25)\n\n # Retrieve data loaders\n train_loader, valid_loader = get_loaders(train, train_i, valid_i)\n\n # Model/ Optimizer/ Scheduler/ Criterion\n model, optimizer, scheduler, criterion = get_model_optimizer_criterion()\n # Hooks into the torch model to collect gradients and the topology\n wandb.watch(model, log_freq=100)\n\n # Run Training\n BEST_SCORE = 9999\n\n for epoch in range(EPOCHS):\n print(\"~\"*8, clr.S+f\"Epoch {epoch}\"+clr.E, \"~\"*8)\n\n # === TRAIN ===\n model.train()\n train_losses = []\n\n for images, targets in tqdm(train_loader, desc = 'TRAIN'):\n images, targets = images.to(device), targets.to(device)\n\n # Clear gradients BEFORE prediction\n optimizer.zero_grad()\n # Make predictions\n out, _ = model(images, targets)\n # Compute Loss and Optimize\n loss = criterion(out, targets) \n loss.backward()\n optimizer.step()\n\n train_losses.append(loss.cpu().detach().numpy().tolist())\n\n # Adjust Learning Rate\n scheduler.step()\n\n mean_train_loss = np.mean(train_losses)\n print(clr.S+\"Mean Train Loss:\"+clr.E, mean_train_loss)\n wandb.log({\"mean_train_loss\": np.float(mean_train_loss)}, step=epoch)\n\n\n # === EVAL ===\n model.eval()\n valid_losses, valid_preds, valid_targets = [], [], []\n with torch.no_grad():\n for images, targets in valid_loader:\n valid_targets.append(targets)\n images, targets = images.to(device), targets.to(device)\n\n out, _ = model(images, targets)\n loss = criterion(out, targets)\n\n valid_preds.append(out)\n valid_losses.append(loss.cpu().detach().numpy().tolist())\n\n mean_valid_loss = np.mean(valid_losses)\n print(clr.S+\"Mean Valid Loss:\"+clr.E, mean_valid_loss)\n wandb.log({\"mean_valid_loss\": np.float(mean_valid_loss)}, step=epoch)\n gc.collect()\n\n plot_loss_graph(train_losses, valid_losses, epoch, fold)\n create_wandb_plot(x_data=range(len(train_losses)), y_data=train_losses,\n x_name=\"Iterations\", y_name=\"Loss\", title=\"Train Loss\",\n log=\"train_loss\", plot=\"line\")\n\n # === UPDATES ===\n\n if mean_valid_loss < BEST_SCORE: \n print(\"! Saving model in fold {} | epoch {} ...\".format(fold, epoch), \"\\n\")\n torch.save(model.state_dict(), f\"EffNetB0_fold_{fold}_loss_{round(mean_valid_loss, 3)}.pt\")\n\n BEST_SCORE = mean_valid_loss\n\n # Clean memory before next fold\n del model, optimizer, scheduler, criterion, images, targets, \\\n train_losses, valid_losses, valid_preds, valid_targets\n torch.cuda.empty_cache()\n gc.collect()\n\n\n print(clr.S+f\"Time to run: {round((time.time() - s)/60, 2)} minutes\"+clr.E)\n wandb.finish()","metadata":{"_kg_hide-input":false,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2022-03-12T11:40:03.750310Z","iopub.execute_input":"2022-03-12T11:40:03.750943Z","iopub.status.idle":"2022-03-12T11:40:03.773085Z","shell.execute_reply.started":"2022-03-12T11:40:03.750905Z","shell.execute_reply":"2022-03-12T11:40:03.772367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.2 Train Experiments\n\n🐳 **A few things to keep in mind:**\n* `NUM_CLASSES`: target labels need to start from 0 and have consecutive numbers, otherwise PyTorch will get upset\n * e.g.: bad labels example: [1, 2, 5, 10]\n * e.g.: good labels example: [0, 1, 2, 3, 4, ......100]\n* `BATCH_SIZE`: if set too hight the notebook might get a memory load. This also applies to `IMAGE_SIZE` and the model used (I have tried de EffNet B3 with 512x512 size and I couldn't run the training pipeline)\n* `VALID_PERC`: this I use so the pipeline goes faster through the validation part (so the notebook commits faster); you can set it to `1` to run the full validation dataset.","metadata":{}},{"cell_type":"code","source":"# --------- GLOBAL PARAMETERS ---------\nNUM_CLASSES = 15587\nN_SPLITS = 3\nBATCH_SIZE = 32\nMODEL_NAME = 'efficientnet_b0'\nRUN_NAME = \"B0_neurons_200_embed_200_epochs_4\"\nEPOCHS = 6\nVALID_PERC = 0.1\nNO_NEURONS = 250\nEMBEDDING_SIZE = 128\n# -> Optimizer\nLR = 0.0001\nWEIGHT_DECAY = 0.000001\n# -> Scheduler\nT_MAX = 500 # Maximum number of iterations\nMIN_LR = 0.000001 # Minimum learning rate. Default: 0\n# ------------------------------------","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:40:03.774022Z","iopub.execute_input":"2022-03-12T11:40:03.777002Z","iopub.status.idle":"2022-03-12T11:40:03.784501Z","shell.execute_reply.started":"2022-03-12T11:40:03.776962Z","shell.execute_reply":"2022-03-12T11:40:03.783735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_pipeline(train)","metadata":{"execution":{"iopub.status.busy":"2022-03-01T17:01:56.118712Z","iopub.execute_input":"2022-03-01T17:01:56.119013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Model Embeddings\n\nNow that we have let the model see the images, we can take the **parameters** from the second to last layer (not the one that creates the output, but the one before that) and use it to **create the image embeddings**.\n\n🐳 **Image embeddings** are the *juice* of an image, the very core of it's *information*. This layer has learned everything there is to know about each image and each individual. Moreover, as the model becomes more accurate, the embeddings become more precise too, not only in classifying the individuals, but also being able to recognize the differences between them.\n\n## 6.1 Retrieve the Embeddings\n\nWe will use `torch.load()` to load into a model the pretrained weights & biases that we have created during the Classification task.","metadata":{}},{"cell_type":"code","source":"# pretrained_name = \"EffNetB0_fold_0_loss_14.979\"\n# pretrained_name = \"EffNetB0_fold_1_loss_14.91\"\npretrained_name = \"EffNetB0_fold_2_loss_15.325\"\n\n# Path to trained model parameters (i.e. weights and biases)\nclassif_model_path = f\"../input/happywhale-2022/{pretrained_name}.pt\"\n\n# Load the model and append learned params\nmodel = HappyWhaleModel(MODEL_NAME, NUM_CLASSES, NO_NEURONS, EMBEDDING_SIZE).to(device)\nmodel.load_state_dict(torch.load(classif_model_path))","metadata":{"execution":{"iopub.status.busy":"2022-03-10T16:23:23.596687Z","iopub.execute_input":"2022-03-10T16:23:23.59694Z","iopub.status.idle":"2022-03-10T16:23:24.519598Z","shell.execute_reply.started":"2022-03-10T16:23:23.596913Z","shell.execute_reply":"2022-03-10T16:23:24.518943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# DataLoader\ndataset = HappyWhaleDataset(train, trainFlag=True)\ndataloader = DataLoader(dataset, batch_size=64, shuffle=False)\n\n# Retrieve all embeddings for each image\nall_embeddings = []\n\nmodel.eval()\nwith torch.no_grad():\n for image, target in tqdm(dataloader): \n image, target = image.to(device), target.to(device)\n _, embedding = model(image, target)\n embedding = embedding.detach().cpu().numpy()\n all_embeddings.append(embedding)\n \n# Concatenate batches together\nimage_embeddings = np.concatenate(all_embeddings)\n\n# Save embeddings and corresponding image\nnp.save(f'{pretrained_name}.npy', image_embeddings)","metadata":{"execution":{"iopub.status.busy":"2022-03-10T16:23:27.977115Z","iopub.execute_input":"2022-03-10T16:23:27.977869Z","iopub.status.idle":"2022-03-10T16:29:06.966576Z","shell.execute_reply.started":"2022-03-10T16:23:27.977819Z","shell.execute_reply":"2022-03-10T16:29:06.965725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 🐝Save embeddings to W&B\nsave_dataset_artifact(run_name=pretrained_name, \n artifact_name=pretrained_name, \n path=\"../input/happywhale-2022/EffNetB0_fold_0_loss_14.979.npy\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 7. RAPIDS Clustering\n\nLast step is to create the clusters. The `k_neighbors` will be set to 5, as this is the maximum number to predict ([as stated in competition guidelines](https://www.kaggle.com/c/happy-whale-and-dolphin/overview/evaluation)).","metadata":{}},{"cell_type":"code","source":"# === CLUSTERING ===\n# Use the cuml function from RAPIDS suite\nknn_model = NearestNeighbors(n_neighbors=5)\n# Train the model\nknn_model.fit(image_embeddings)\n\n# Infer on the training data\n# distances - the distance between each point in the group\n# indices - the index row of each image\ndistances, indices = knn_model.kneighbors(image_embeddings)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# === PREDICTION ===\n# Create the grouped predictions based on distances & indices\npredictions = {\"images\": [], \"embeddings\": []}\n\nfor i in tqdm(range(len(image_embeddings))):\n index = np.where(distances[k, ] < 6.0)[0]\n split = indices[i, index]\n \n grouped_images = train.iloc[split][\"image\"].values\n grouped_embeddings = image_embeddings[split]\n\n predictions[\"images\"].append(grouped_images)\n predictions[\"embeddings\"].append(grouped_embeddings)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Compute Cos Distance\n\nBelow you can see a few examples of **clustered train embeddings** and the cos distance similarity between them.","metadata":{}},{"cell_type":"code","source":"def get_cosine_similarity(embeddings):\n '''Compute cos distance between n embedding vector and itself.'''\n similarity_matrix = []\n \n for embed1 in embeddings:\n similarity_row = []\n for embed2 in embeddings:\n similarity_row.append(1 - spatial.distance.cosine(embed1, embed2))\n similarity_matrix.append(similarity_row)\n \n return np.array(similarity_matrix, dtype=\"float32\")\n\n\ndef plot_heatmap(example_paths, cos_matrix, mask):\n '''Computes cos distance heatmap.'''\n \n # Plots\n fig = plt.figure(figsize=(12, 12))\n ax1 = plt.subplot2grid(shape=(6, 6), loc=(5, 1), colspan=1)\n ax2 = plt.subplot2grid(shape=(6, 6), loc=(5, 2), colspan=1)\n ax3 = plt.subplot2grid(shape=(6, 6), loc=(5, 3), colspan=1)\n ax4 = plt.subplot2grid(shape=(6, 6), loc=(5, 4), colspan=1)\n ax5 = plt.subplot2grid(shape=(6, 6), loc=(5, 5), colspan=1)\n h_axes = [ax1, ax2, ax3, ax4, ax5]\n\n ax6 = plt.subplot2grid(shape=(6, 6), loc=(0, 0), colspan=1)\n ax7 = plt.subplot2grid(shape=(6, 6), loc=(1, 0), colspan=1)\n ax8 = plt.subplot2grid(shape=(6, 6), loc=(2, 0), colspan=1)\n ax9 = plt.subplot2grid(shape=(6, 6), loc=(3, 0), colspan=1)\n ax10 = plt.subplot2grid(shape=(6, 6), loc=(4, 0), colspan=1)\n v_axes = [ax6, ax7, ax8, ax9, ax10]\n\n ax11 = plt.subplot2grid(shape=(6, 6), loc=(0, 1), colspan=5, rowspan=5)\n\n fig.suptitle('- Cosine Distance -', size = 21, color = my_colors[7], weight='bold')\n for k, ax in enumerate(h_axes):\n ax.imshow(plt.imread(example_paths[k]))\n ax.set_axis_off()\n\n for k, ax in enumerate(v_axes):\n ax.imshow(plt.imread(example_paths[k]))\n ax.set_axis_off()\n\n sns.heatmap(cos_matrix, ax=ax11, fmt=\".5\",\n cbar=False, annot=True, linewidths=0.5, mask=mask, square=True, cmap=\"winter_r\")\n\n plt.tight_layout()\n plt.show();","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Select a clustered group\ngroup = 0\n\nexample_paths = [\"../input/whale2-cropped-dataset/cropped_train_images/cropped_train_images/\"+img \\\n for img in predictions[\"images\"][group]]\nexample_embeds = predictions[\"embeddings\"][group]\n\n# Compute similarity matrix\ncos_matrix = get_cosine_similarity(example_embeds)\nmask = np.zeros_like(cos_matrix)\nmask[np.triu_indices_from(mask)] = True\n\nplot_heatmap(example_paths, cos_matrix, mask)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Select a clustered group\ngroup = 24\n\nexample_paths = [\"../input/whale2-cropped-dataset/cropped_train_images/cropped_train_images/\"+img \\\n for img in predictions[\"images\"][group]]\nexample_embeds = predictions[\"embeddings\"][group]\n\n# Compute similarity matrix\ncos_matrix = get_cosine_similarity(example_embeds)\nmask = np.zeros_like(cos_matrix)\nmask[np.triu_indices_from(mask)] = True\n\nplot_heatmap(example_paths, cos_matrix, mask)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Select a clustered group\ngroup = 100\n\nexample_paths = [\"../input/whale2-cropped-dataset/cropped_train_images/cropped_train_images/\"+img \\\n for img in predictions[\"images\"][group]]\nexample_embeds = predictions[\"embeddings\"][group]\n\n# Compute similarity matrix\ncos_matrix = get_cosine_similarity(example_embeds)\nmask = np.zeros_like(cos_matrix)\nmask[np.triu_indices_from(mask)] = True\n\nplot_heatmap(example_paths, cos_matrix, mask)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 8. Inference\n\n🐳 Last step! Let's do the inference using the goodies we have created so far!\n\nWe will first set up the inference parameters - in this notebook these might be different than the ones during training, depending on which model I am using to make the inference. Hence, the inference parameters **must** be aligned with the model's parameters.\n\n🐝 How do I remember what I used during model training? I check out the *metrics logged* into [my W&B Dashboard](https://wandb.ai/andrada/happywhale?workspace=user-andrada) for that speciffic experiment.","metadata":{}},{"cell_type":"code","source":"# --------- INFERENCE PARAMETERS ---------\nPRETRAINED_NAME1 = \"EffNetB0_fold_0_loss_14.979\"\nPRETRAINED_NAME2 = \"EffNetB0_fold_1_loss_14.91\"\nPRETRAINED_NAME3 = \"EffNetB0_fold_2_loss_15.325\"\nMODEL_NAME = 'efficientnet_b0'\nNUM_CLASSES = 15587\nNO_NEURONS = 250\nEMBEDDING_SIZE = 128\n# ----------------------------------------","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:40:03.785644Z","iopub.execute_input":"2022-03-12T11:40:03.786268Z","iopub.status.idle":"2022-03-12T11:40:03.797361Z","shell.execute_reply.started":"2022-03-12T11:40:03.786231Z","shell.execute_reply":"2022-03-12T11:40:03.796532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### I. Get Test Embeddings\n\n🐳 The first step in this process is to retrieve the embeddings. We are doing this by loading the trained model named `final_model` and extracting the embeddings from the second to last layer. I am saving these into `test_embeddings`.","metadata":{}},{"cell_type":"code","source":"def retrieve_test_embeddings(PRETRAINED_NAME, MODEL_NAME, NUM_CLASSES, NO_NEURONS, EMBEDDING_SIZE):\n '''Get test embeddings using a pretrained model.'''\n \n # Path to [best] model\n classif_model_path = f\"../input/happywhale-2022/{PRETRAINED_NAME}.pt\"\n\n # Load the model and append learned params\n final_model = HappyWhaleModel(MODEL_NAME, NUM_CLASSES, NO_NEURONS, EMBEDDING_SIZE).to(device)\n final_model.load_state_dict(torch.load(classif_model_path))\n\n # DataLoader\n dataset = HappyWhaleDataset(test, trainFlag=False)\n dataloader = DataLoader(dataset, batch_size=64, shuffle=False)\n\n # Retrieve test embeddings from the model\n all_embeddings = []\n\n final_model.eval()\n with torch.no_grad():\n for image in tqdm(dataloader): \n image = image.to(device)\n embedding = final_model(image, target=None)\n embedding = embedding.detach().cpu().numpy()\n all_embeddings.append(embedding)\n\n # Concatenate batches together\n test_embeddings = np.concatenate(all_embeddings)\n\n # Save embeddings\n np.save(f'{PRETRAINED_NAME}_test_.npy', test_embeddings)\n \n return test_embeddings","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T11:40:03.798601Z","iopub.execute_input":"2022-03-12T11:40:03.799215Z","iopub.status.idle":"2022-03-12T11:40:03.808469Z","shell.execute_reply.started":"2022-03-12T11:40:03.799176Z","shell.execute_reply":"2022-03-12T11:40:03.807693Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ===== I. EMBEDDINGS =====\n# Here I am retrieving the 3 test embeddings and averaging them together\ntest_embeddings1 = retrieve_test_embeddings(PRETRAINED_NAME1, \n MODEL_NAME, NUM_CLASSES, NO_NEURONS, EMBEDDING_SIZE)\ntest_embeddings2 = retrieve_test_embeddings(PRETRAINED_NAME2, \n MODEL_NAME, NUM_CLASSES, NO_NEURONS, EMBEDDING_SIZE)\ntest_embeddings3 = retrieve_test_embeddings(PRETRAINED_NAME3, \n MODEL_NAME, NUM_CLASSES, NO_NEURONS, EMBEDDING_SIZE)\n\ntest_embeddings = (test_embeddings1+test_embeddings2+test_embeddings3)/3","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:40:21.148743Z","iopub.execute_input":"2022-03-12T11:40:21.149291Z","iopub.status.idle":"2022-03-12T12:07:54.397466Z","shell.execute_reply.started":"2022-03-12T11:40:21.149254Z","shell.execute_reply":"2022-03-12T12:07:54.396723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### II. Creating Clusters\n\nNow we can group the embeddings into clusters.\n\n🐳 We first load the `train_embeddings` computed during training and train a final `NearestNeighbours` classifier on them. After that we can **predict** on our `test_embeddings` using this `knn_final_model` => the output is **50 nearest Distances & their Index** for *each test embedding*.\n\n#### Embedding Blending","metadata":{}},{"cell_type":"code","source":"# ===== II. CLUSTERS =====\n# Get full train embeddings\n# Here I am adding the 3 train embeddings and averaging them together as well\ntrain_embeddings1 = np.load(f\"../input/happywhale-2022/{PRETRAINED_NAME1}.npy\")\ntrain_embeddings2 = np.load(f\"../input/happywhale-2022/{PRETRAINED_NAME2}.npy\")\ntrain_embeddings3 = np.load(f\"../input/happywhale-2022/{PRETRAINED_NAME3}.npy\")\n\ntrain_embeddings = (train_embeddings1+train_embeddings2+train_embeddings3)/3\n\ntrain_individual_ids = train[\"individual_id\"].values\nprint(clr.S+\"Train Embeddings:\"+clr.E, train_embeddings.shape, \"\\n\"+\n clr.S+\"Train Individual Id:\"+clr.E, train_individual_ids.shape, \"\\n\")\n\n\n# Train a final KNN model with the train embeddings\nknn_final_model = NearestNeighbors(n_neighbors=50)\nknn_final_model.fit(train_embeddings)\n\n# Get distances & indexes for test\n# test_embeddings = normalize(test_embeddings, axis=1, norm='l2')\n\nD, I = knn_final_model.kneighbors(test_embeddings)\nprint(clr.S+\"Distances shape:\"+clr.E, D.shape, \"\\n\"+\n clr.S+\"Index shape:\"+clr.E, I.shape)\n\n# List of the test dataframe image ids (to loop through it)\ntest_images = test[\"image\"].tolist()","metadata":{"execution":{"iopub.status.busy":"2022-03-12T12:07:54.399050Z","iopub.execute_input":"2022-03-12T12:07:54.400011Z","iopub.status.idle":"2022-03-12T12:07:56.155915Z","shell.execute_reply.started":"2022-03-12T12:07:54.399970Z","shell.execute_reply":"2022-03-12T12:07:56.155161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = []\n\n# Loop through each observation within test data\nfor k, image_id in tqdm(enumerate(test_images)):\n # Get individual_id & distances for the observation\n individual_id = train_individual_ids[I[k]]\n distances = D[k]\n # Create a df subset with this info\n subset_preds = pd.DataFrame(np.stack([individual_id, distances], axis=1),\n columns=['individual_id','distances'])\n subset_preds['image_id'] = image_id\n test_df.append(subset_preds)\n \n \n# Concatenate subset dataframes into 1 dataframe\ntest_df = pd.concat(test_df).reset_index(drop=True)\n# Choose max distance for each unique pair of individual_id & image_id\ntest_df = test_df.groupby(['image_id','individual_id'])['distances'].max().reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-03-12T12:07:56.157263Z","iopub.execute_input":"2022-03-12T12:07:56.157508Z","iopub.status.idle":"2022-03-12T12:10:19.335057Z","shell.execute_reply.started":"2022-03-12T12:07:56.157474Z","shell.execute_reply":"2022-03-12T12:10:19.334295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Have a look at the predictions dataset now\ntest_df.sample(n=5, random_state=24)","metadata":{"execution":{"iopub.status.busy":"2022-03-12T12:10:19.336844Z","iopub.execute_input":"2022-03-12T12:10:19.337086Z","iopub.status.idle":"2022-03-12T12:10:19.458725Z","shell.execute_reply.started":"2022-03-12T12:10:19.337054Z","shell.execute_reply":"2022-03-12T12:10:19.457938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### III. Final Prediction\n\n🐳 The final step is to create a `predictions` dictionary that will contain the `image_id` and a list of `individual_id` predictions for each image within the test dataset. The maximum number of *guesses* to be made is 5 according to the competition guidelines, and through these guesses a value like `new_individual` can be added too, meaning that the subject within the image wasn't found within the training data.","metadata":{}},{"cell_type":"code","source":"# ===== III. PREDICTION =====\n\n# Dictionary in format: {[\"image_id\"]: 000, [\"individual_id\"]: 9999}\npredictions = {}\nthresh = 5\n\nfor k, row in tqdm(test_df.iterrows()):\n image_id = row[\"image_id\"]\n individual_id = row[\"individual_id\"]\n distance = row[\"distances\"]\n \n # If the image_id has already been added in predictions before\n if image_id in predictions:\n # If total preds for this image_id are < 5 then add, else continue\n if len(predictions[image_id]) != 5:\n predictions[image_id].append(individual_id)\n else:\n continue\n # If the distance is greater than thresh add prediction + \"new_individual\"\n elif distance > thresh:\n predictions[image_id] = [individual_id, \"new_individual\"]\n else:\n predictions[image_id] = [\"new_individual\", individual_id]\n\n\n# Fill in all lists that have less than 5 predictions as of yet\nsample_list = ['37c7aba965a5', '114207cab555', 'a6e325d8e924', '19fbb960f07d','c995c043c353']\n\nfor image_id, preds in tqdm(predictions.items()):\n if len(preds) < 5:\n remaining = [individ_id for individ_id in sample_list if individ_id not in preds]\n preds.extend(remaining)\n predictions[image_id] = preds[:5]","metadata":{"execution":{"iopub.status.busy":"2022-03-12T12:14:13.433691Z","iopub.execute_input":"2022-03-12T12:14:13.434081Z","iopub.status.idle":"2022-03-12T12:15:25.548408Z","shell.execute_reply.started":"2022-03-12T12:14:13.434036Z","shell.execute_reply":"2022-03-12T12:15:25.547586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create final submission\npredictions = pd.Series(predictions).reset_index()\npredictions.columns = ['image','predictions']\npredictions['predictions'] = predictions['predictions'].apply(lambda x: ' '.join(x))\npredictions.to_csv('submission.csv',index=False)\n\npredictions.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-12T12:15:25.549979Z","iopub.execute_input":"2022-03-12T12:15:25.550300Z","iopub.status.idle":"2022-03-12T12:15:25.709549Z","shell.execute_reply.started":"2022-03-12T12:15:25.550256Z","shell.execute_reply":"2022-03-12T12:15:25.708816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n### 🐝 W&B Dashboard\n\n> My [W&B Dashboard](https://wandb.ai/andrada/happywhale?workspace=user-andrada).\n\n
\n\n
\n\n### My Specs\n\n* 🖥 Z8 G4 Workstation\n* 💾 2 CPUs & 96GB Memory\n* 🎮 NVIDIA Quadro RTX 8000\n* 💻 Zbook Studio G7 on the go","metadata":{}}]}