{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## A Friendly ☺️ Introduction to Siamese Networks\n\nCNN classifiers are capable enough to perform many kinds of image classification tasks. Generally though, CNN require large amount of data to train properly. For each class it may be hundreds of images to be able to classify correctly.\n\nBut what if\n\n1) You have a lot of classes\n2) You don't have enough images per class\n\nFor example, face recognition is a classification task that can have thousands or more number of target classes. And you may not have hundreds of images per person, you may be forced to make do with 4-5 of them! This is a domain of machine learning that is **Few-Shot Learning**. As the name suggests, it represents techniques to learn with few 'shots' i.e. examples.\n\nSiamese Networks is one of the simpler methods of few-shot learning. It also falls in the category of 'Contrastive Learning'- this means that the network will try to separate images of distinct classes as far as possible. We'll see the details later.","metadata":{}},{"cell_type":"markdown","source":"
\n

\n What Are Siamese Networks

\n
\n\nWhile it may say \"networks\", and images you find online show two CNN models joined together- this is not true. In reality, this is just one single model, however it takes two images as input instead of one (or three depending on the loss function). So how is it different from a normal CNN?\n\nIn a basic CNN classifier, you input an image, it is forward-passed through the architecture to get a compact representation, it is flattened and fed to a dense layer, finally to a sigmoid/softmax layer where it is compared with a target and a loss is computed and backproped to update the model weights. (If you didn't understand this, you might need to read up on CNNs somewhere else 😕)\n\nNow a siamese network can be based on a cross-entropy loss or a triplet loss. Since this notebook implements the former, I'll explain the former first.\n\nIn a nutshell, a siamese network will try to \"separate\" images of different classes, while trying to bring closer the images of same class. We will train a model that can predict if given two images belong to same class or not.\n\nIn a Siamese Network, you input two random images. The label is 1 or 0, depending on whether the images belong to same class or different class. Both of them are independently forward-passed through a CNN architecture (could be a resnet backbone, or custom Conv layers) to get a single-vector representation for each of them. These two vectors are then combined using concatenation, and fed to a dense layer to predict the label (1 or 0)\n\nThe architecture isn't limited to CNNs- the backbone can be any type of neural network that can produce a compact representation. It can be a bunch of dense layers, it can be an LSTM, Vision Transformer, etc. It's a very versatile configuration, and the basic idea remains the same.\n\n![siamese-arch](https://people.kth.se/~rosun/deep-learning/figures/siamese-arch.svg)","metadata":{}},{"cell_type":"markdown","source":"
\n

\n With Triplet Loss

","metadata":{}},{"cell_type":"markdown","source":"Triplet Loss function is a loss function that's optimized for siamese networks. However I was unable to implement it here due to memory issues, so that's for another day. This approach is compute intensive, it requires three images as input (as opposed to two here). Out of the three, two images are of same class (but they are not the same image), and one image is of a different class. The loss function will try to cluster the similar images together as far away from the negative image.","metadata":{}},{"cell_type":"markdown","source":"
\n

\n What Is This Notebook About

","metadata":{}},{"cell_type":"markdown","source":"In this notebook, we're going to implement a siamese network for classifying plant species. This dataset is from the herbarium competition, which requires you to predict a plant species given it's image. There are more than 15k species, which makes this a decent candidate for few-shot learning.\n\nWhile the dataset is fairly large, and few-shot learning is not necessarily the best approach here, we're still going with it to see if it can do well. This is not a toy dataset like MNIST, so this would be a good practice.\n\nMoreover, most siamese network tutorials do not cover the inference part - admittedly simple, but still remains a missing part in many places, especially considering a siamese network doesn't directly do classification. This notebook covers the classification task.\n\nWe use Pytorch in this notebook. For experiment tracking, we are using wandb. Highly recommend experiment tracking for your deep learning tasks, especially wandb can be very useful.","metadata":{}},{"cell_type":"markdown","source":"
\n

\n Steps in the Notebook

","metadata":{}},{"cell_type":"markdown","source":"The sequence of steps is as follows:\n\n1. Importing libraries and basic overview of data provided.\n2. Defining a siamese network using Resnet18 Backbone.\n3. Defining the image dataloaders.\n4. Define loss function, optimizers.\n5. Start the training loop. Save model at the end of it.\n6. Inference- how to use it for classification task","metadata":{}},{"cell_type":"markdown","source":"## Libraries and Overview","metadata":{}},{"cell_type":"code","source":"# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\nimport pandas as pd\nfrom PIL import Image\nimport numpy as np\nimport json\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nimport seaborn as sns\nimport cv2\nimport time\nimport wandb\nimport random\nimport glob\nimport tqdm\nimport gc\n\ntrain_dir = '../input/herbarium-2022-fgvc9/train_images/'\ntest_dir = '../input/herbarium-2022-fgvc9/test_images/'\n\nwith open(\"../input/herbarium-2022-fgvc9/train_metadata.json\") as json_file:\n train_meta = json.load(json_file)\nwith open(\"../input/herbarium-2022-fgvc9/test_metadata.json\") as json_file:\n test_meta = json.load(json_file)\nnb_start_time = time.time()","metadata":{"execution":{"iopub.status.busy":"2022-12-10T18:09:58.558701Z","iopub.execute_input":"2022-12-10T18:09:58.559137Z","iopub.status.idle":"2022-12-10T18:10:12.552438Z","shell.execute_reply.started":"2022-12-10T18:09:58.559049Z","shell.execute_reply":"2022-12-10T18:10:12.551073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Experiment Tracking\n\nWhen doing deep learning experiments, experiment tracking is extremely helpful. It is extra helpful on kaggle when you have large datasets. Wandb is one of the tools to do this, and it's a fantastic product. You can log the loss, any other metrics to it, and watch it trickle on the site anytime even on your phone. You can observe and debug the model parameters, you can observe the GPU/CPU usage metrics.\n\nAnd it only takes a few lines of code! First, sign up on wandb.ai and get the API key. Then add the API key to kaggle secrets (cuz you can't hard code it in the notebook) [using these steps.](https://www.kaggle.com/general/209530) Then you can add it to any notebook, and start logging.","metadata":{"execution":{"iopub.status.busy":"2022-12-04T15:09:48.091705Z","iopub.execute_input":"2022-12-04T15:09:48.092441Z","iopub.status.idle":"2022-12-04T15:09:48.096786Z","shell.execute_reply.started":"2022-12-04T15:09:48.092402Z","shell.execute_reply":"2022-12-04T15:09:48.095808Z"}}},{"cell_type":"code","source":"nb_eval = False # this variable is for modifying my experiments, can ignore\nuse_wandb = True # use wandb for experiment tracking\nif use_wandb:\n from kaggle_secrets import UserSecretsClient\n user_secrets = UserSecretsClient()\n wandb_api = user_secrets.get_secret(\"wandb_api\")\n wandb.login(key=wandb_api)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T18:11:26.208705Z","iopub.execute_input":"2022-12-10T18:11:26.209073Z","iopub.status.idle":"2022-12-10T18:11:29.809123Z","shell.execute_reply.started":"2022-12-10T18:11:26.209038Z","shell.execute_reply":"2022-12-10T18:11:29.808177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if use_wandb:\n wandb.init(project=\"herbarium-siamese\")","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:07.663361Z","iopub.execute_input":"2022-12-10T15:16:07.663712Z","iopub.status.idle":"2022-12-10T15:16:14.495424Z","shell.execute_reply.started":"2022-12-10T15:16:07.663677Z","shell.execute_reply":"2022-12-10T15:16:14.494518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets collect the train images and see what they look like","metadata":{}},{"cell_type":"code","source":"number_of_images = len(train_meta['images'])\nprint(f\"Number of Train Images: {number_of_images}\")\nimg_ids = [train_meta['images'][i]['image_id'] for i in range(number_of_images)]\nfilenames = [train_dir + train_meta['images'][i]['file_name'] for i in range(number_of_images)]\nfilename_dict = dict(zip(img_ids, filenames))\n\nnumber_of_annotations = len(train_meta['annotations'])\nprint(f\"Number of Annotations: {number_of_annotations}\")\nimg_ids = [train_meta['annotations'][i]['image_id'] for i in range(number_of_annotations)]\nlabels = [train_meta['annotations'][i]['category_id'] for i in range(number_of_annotations)]\nlabel_dict = dict(zip(img_ids, labels))\n\nfilename_label_dict = {fl:label_dict[ID] for ID, fl in filename_dict.items()}\n\nfiles, labels = list(filename_label_dict.keys()), list(filename_label_dict.values())\nnumber_of_labels = len(np.unique(labels))\nprint(\"Number of Unique Labels:\", number_of_labels)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T18:11:38.098688Z","iopub.execute_input":"2022-12-10T18:11:38.099052Z","iopub.status.idle":"2022-12-10T18:11:39.842743Z","shell.execute_reply.started":"2022-12-10T18:11:38.099019Z","shell.execute_reply":"2022-12-10T18:11:39.841795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 15501 target classes! That's a lot.\nLet's see what the images look like:","metadata":{}},{"cell_type":"code","source":"label_dist = pd.Series(labels).value_counts()\nlabel_dist.max(), label_dist.idxmax()\nprint(f'Class with the Largest amount of samples: {label_dist.idxmax()}, with {label_dist.max()} images.')\nprint(f'Class with the Smallest amount of samples: {label_dist.idxmin()}, with {label_dist.min()} images.')","metadata":{"execution":{"iopub.status.busy":"2022-12-10T18:15:54.113518Z","iopub.execute_input":"2022-12-10T18:15:54.113868Z","iopub.status.idle":"2022-12-10T18:15:54.312127Z","shell.execute_reply.started":"2022-12-10T18:15:54.113839Z","shell.execute_reply":"2022-12-10T18:15:54.311105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Histogram of Number of Images per Class:')\nlabel_dist.plot.hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-10T18:17:28.824777Z","iopub.execute_input":"2022-12-10T18:17:28.8252Z","iopub.status.idle":"2022-12-10T18:17:29.075299Z","shell.execute_reply.started":"2022-12-10T18:17:28.825166Z","shell.execute_reply":"2022-12-10T18:17:29.074419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From above plot, we can observe that many classes have images numbering around 60-80 which is a decent size. Some classes have less than 20 images. ","metadata":{}},{"cell_type":"code","source":"im = Image.open(filenames[1])\nplt.imshow(im)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-10T18:18:47.413498Z","iopub.execute_input":"2022-12-10T18:18:47.414064Z","iopub.status.idle":"2022-12-10T18:18:47.676267Z","shell.execute_reply.started":"2022-12-10T18:18:47.414022Z","shell.execute_reply":"2022-12-10T18:18:47.675392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Opening some random images side-by-side","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(2, 2, sharex='col', sharey='row', figsize=(10,10))\nax[0][0].imshow(Image.open(random.choice(filenames)))\nax[0][1].imshow(Image.open(random.choice(filenames)))\nax[1][0].imshow(Image.open(random.choice(filenames)))\nax[1][1].imshow(Image.open(random.choice(filenames)))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:20.299567Z","iopub.execute_input":"2022-12-10T15:16:20.300058Z","iopub.status.idle":"2022-12-10T15:16:21.430035Z","shell.execute_reply.started":"2022-12-10T15:16:20.300003Z","shell.execute_reply":"2022-12-10T15:16:21.428891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, lets take a look at the test images.","metadata":{}},{"cell_type":"code","source":"test_images = [test_dir + test_meta[i]['file_name'] for i in range(len(test_meta))]\nprint(f\"Number of Test Images: {len(test_images):,}\")","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:22.459361Z","iopub.execute_input":"2022-12-10T15:16:22.459889Z","iopub.status.idle":"2022-12-10T15:16:22.557956Z","shell.execute_reply.started":"2022-12-10T15:16:22.459804Z","shell.execute_reply":"2022-12-10T15:16:22.556892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(Image.open(test_dir + test_meta[0]['file_name']))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:22.699872Z","iopub.execute_input":"2022-12-10T15:16:22.700356Z","iopub.status.idle":"2022-12-10T15:16:23.477158Z","shell.execute_reply.started":"2022-12-10T15:16:22.700312Z","shell.execute_reply":"2022-12-10T15:16:23.474379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define the Siamese Network","metadata":{}},{"cell_type":"code","source":"import random\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nimport torchvision\nfrom torch.utils.data import Dataset\nfrom torchvision import datasets\nfrom torchvision import transforms as T\nfrom torch.optim.lr_scheduler import StepLR","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:25.200694Z","iopub.execute_input":"2022-12-10T15:16:25.201229Z","iopub.status.idle":"2022-12-10T15:16:27.311821Z","shell.execute_reply.started":"2022-12-10T15:16:25.201183Z","shell.execute_reply":"2022-12-10T15:16:27.310713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The starter code was taken from the official pytorch examples for siamese network, which uses MNIST data, and was modified from there.\n\nThere are two classes we are creating.\n\n1. First is the model class that derives from nn.Module. This includes the model architecture, forward pass, etc. \n2. Second is the dataset class that we will use for loading, preprocessing the image data.\n\nWe'll first see the code and then the explanation.","metadata":{}},{"cell_type":"code","source":"class SiameseNetwork(nn.Module):\n \"\"\"\n Siamese network for image similarity estimation.\n The network is composed of two identical networks, one for each input.\n The output of each network is concatenated and passed to a linear layer. \n The output of the linear layer passed through a sigmoid function.\n `\"FaceNet\" `_ is a variant of the Siamese network.\n This implementation varies from FaceNet as we use the `ResNet-18` model from\n `\"Deep Residual Learning for Image Recognition\" `_ as our feature extractor.\n \"\"\"\n def __init__(self):\n super(SiameseNetwork, self).__init__()\n # get resnet model\n self.resnet = torchvision.models.resnet18(pretrained=False)\n\n self.fc_in_features = self.resnet.fc.in_features\n \n # remove the last layer of resnet18 (linear layer which is before avgpool layer)\n self.resnet = torch.nn.Sequential(*(list(self.resnet.children())[:-1]))\n for name, p in self.resnet.named_parameters():\n if not name.startswith(\"7.0\"):\n p.requires_grad = False\n# for param in self.resnet.parameters():\n# param.requires_grad = False\n\n # add linear layers to compare between the features of the two images\n self.fc = nn.Sequential(\n nn.Linear(self.fc_in_features * 2, 256),\n nn.ReLU(inplace=True),\n nn.Linear(256, 1),\n )\n\n self.sigmoid = nn.Sigmoid()\n\n self.fc.apply(self.init_weights)\n def init_weights(self, m):\n if isinstance(m, nn.Linear):\n torch.nn.init.xavier_uniform(m.weight)\n m.bias.data.fill_(0.01)\n\n def forward_once(self, x):\n output = self.resnet(x)\n output = output.view(output.size()[0], -1)\n return output\n\n def forward(self, input1, input2):\n # get two images' features\n output1 = self.forward_once(input1)\n output2 = self.forward_once(input2)\n\n # concatenate both images' features\n output = torch.cat((output1, output2), 1)\n\n # pass the concatenation to the linear layers\n output = self.fc(output)\n\n # pass the out of the linear layers to sigmoid layer\n output = self.sigmoid(output)\n \n return output","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:23:02.94504Z","iopub.execute_input":"2022-12-10T15:23:02.945407Z","iopub.status.idle":"2022-12-10T15:23:02.959483Z","shell.execute_reply.started":"2022-12-10T15:23:02.945374Z","shell.execute_reply":"2022-12-10T15:23:02.95844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class APP_MATCHER(Dataset):\n def __init__(self, train_files, labels, transform=None):\n super(APP_MATCHER, self).__init__()\n self.train_files = train_files\n self.labels = labels\n self.unique_labels = np.unique(labels)\n self.transform = transform\n self.group_examples()\n\n def group_examples(self):\n \"\"\"\n To ease the accessibility of data based on the class, we will use `group_examples` to group \n examples based on class. \n \n Every key in `grouped_examples` corresponds to a class. For every key in \n `grouped_examples`, every value will conform to all of the indices for the \n dataset examples that correspond to that key.\n \"\"\"\n np_arr = np.array(self.labels)\n \n # group examples based on class\n self.grouped_examples = {}\n for i in np.unique(self.labels):\n self.grouped_examples[i] = np.where((np_arr==i))[0]\n \n def __len__(self):\n return len(self.train_files)\n \n def __getitem__(self, index):\n \"\"\"\n For every example, we will select two images. There are two cases, \n positive and negative examples. For positive examples, we will have two \n images from the same class. For negative examples, we will have two images \n from different classes.\n Given an index, if the index is even, we will pick the second image from the same class, \n but it won't be the same image we chose for the first class. This is used to ensure the positive\n example isn't trivial as the network would easily distinguish the similarity between same images. However,\n if the network were given two different images from the same class, the network will need to learn \n the similarity between two different images representing the same class. If the index is odd, we will \n pick the second image from a different class than the first image.\n \"\"\"\n\n # pick some random class for the first image\n random.shuffle(self.unique_labels)\n selected_class = self.unique_labels[0] #random.randint(0, number_of_labels-1)\n \n # pick the index to get the first image\n index_1 = random.choice(self.grouped_examples[selected_class]) \n\n # get the first image\n image_1 = self.train_files[index_1]#.clone().float()\n\n index_2 = index_1\n if index % 2 == 0:\n if index_2 == index_1:\n index_2 = random.choice(self.grouped_examples[selected_class])\n image_2 = self.train_files[index_2]\n target = 1\n else:\n other_selected_class = self.unique_labels[1]\n index_2 = random.choice(self.grouped_examples[other_selected_class])\n image_2 = self.train_files[index_2]\n target = 0\n image_1, image_2 = Image.open(image_1), Image.open(image_2)\n if self.transform is not None:\n image_1, image_2 = self.transform(image_1), self.transform(image_2)\n \n target = torch.tensor(target, dtype=torch.float)\n return image_1, image_2, target","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:27.700776Z","iopub.execute_input":"2022-12-10T15:16:27.701211Z","iopub.status.idle":"2022-12-10T15:16:27.721467Z","shell.execute_reply.started":"2022-12-10T15:16:27.70117Z","shell.execute_reply":"2022-12-10T15:16:27.720427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## What's Happening Here?\n\n### Model Class\n\nIn the model class, in the `__init__` function we have initialized a resnet18 to act as our backbone. We have removed it's last layer (which is a fully connected layer). Basically, when an image is passed to this resnet backbone, it will return a single vector.\n\nWe also initialized some new fully connected/dense layers of our own. The input to these layers is of length *double* the output of resnet. Because the input to these layers will come from *two* images. Two vectors will be concatenated and passed to this dense network (with one hidden layer of size 256) to get a final single score. Let's look at the diagram again:\n\n![siam](https://people.kth.se/~rosun/deep-learning/figures/siamese-arch.svg)\n\nThe class has a method `forward_once`, which will pass an image through resnet to get a single vector that contains it's representation. The class has a `forward` method, where two images are given as input. We get the vector representation of both images independently. We concatenate both of these vectors, which is passed to a dense network to get a final score. \n\nThis score is compared with the label. Recall, that the label is 0 or 1 depending if the two images belong to same class or not. With cross entropy loss, and an optimizer, we are updating the model during training.\n\n### Dataset Class\n\nWhat's happening in the dataset class? In pytorch we use a dataset-derived class to create a dataloader to feed to a model. We modify the `__getitem__` method to generate our input to the model. This is regular pytorch workflow.\n\nIn this case, we have an extra method `grouped_examples`. In this method, we are creating a dictionary that maps from label to a list of indices of images which belong to that label. i.e. if the images 1, 4, 78 in the list of train files belong to class 1, and 2, 33, 89 belong to class 2, the dictionary looks like:\n\n```\n{1: [1, 4, 78],\n 2: [2, 33, 89]}\n```\nand likewise.\n\nIn `__getitem__`, we take a random class. And retrieve a random index from the list of indices belonging to that class. \n\nThen, if given index is even we retrieve another random index belonging to this same class. We ensure that we don't take the same index again. Label is 1 in this case.\n\nIf given index is odd, we take a random class, that's not the previous class. We retrieve a random index from this class. Label is 0 in this case.\n\nFrom the two indices, retrieve the train file paths, and load the images, and perform transforms, and return! There are three items to return- two transformed images (tensors), and a label (integer).","metadata":{}},{"cell_type":"markdown","source":"## A Bug Story\n\nEncountered an interesting bug while using the sample code from pytorch.\nTo choose a random class, the code was iterating through classes using `range(0, number_of_classes)`. This threw a bug, but only sometimes.\n\nThen I checked that `len(np.unique(labels))` is not equal to `np.max(labels)`, because some labels in the middle are missing! Hence, using the actual list of labels, instead of creating a new list. ","metadata":{}},{"cell_type":"code","source":"# we are using GPU ofcourse\ndevice = 'cuda'","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:30.729446Z","iopub.execute_input":"2022-12-10T15:16:30.730294Z","iopub.status.idle":"2022-12-10T15:16:30.7385Z","shell.execute_reply.started":"2022-12-10T15:16:30.73025Z","shell.execute_reply":"2022-12-10T15:16:30.737583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# what transforms to use?\n# resizing to 224x224 cause that's resnet size\n# converting to tensor is mandatory step\n# added normalize in there\n# no additional augmentation, already the data is too huge\ntransforms = T.Compose([\n T.Resize((224, 224)),\n T.ToTensor(),\n T.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n])","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:32.239573Z","iopub.execute_input":"2022-12-10T15:16:32.240171Z","iopub.status.idle":"2022-12-10T15:16:32.247968Z","shell.execute_reply.started":"2022-12-10T15:16:32.240126Z","shell.execute_reply":"2022-12-10T15:16:32.246928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Due to memory issues, I'm taking only half of the entire training data. Maybe there are ways to mitigate this, but will try them later and rerun the notebook with the full dataset. But till then, we are keeping only a part of the data.\n\n`files` currently only contains the filepaths, not the loaded images. We don't load the images until absolutely required.","metadata":{}},{"cell_type":"code","source":"_, files, _, labels = train_test_split(files, labels, test_size=0.5, random_state=0, stratify=labels)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:38.37027Z","iopub.execute_input":"2022-12-10T15:16:38.370819Z","iopub.status.idle":"2022-12-10T15:16:40.116899Z","shell.execute_reply.started":"2022-12-10T15:16:38.370777Z","shell.execute_reply":"2022-12-10T15:16:40.115901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_files, val_files, train_labels, val_labels = train_test_split(files, labels, test_size=0.5, random_state=0, stratify=labels)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:40.122436Z","iopub.execute_input":"2022-12-10T15:16:40.125089Z","iopub.status.idle":"2022-12-10T15:16:40.983348Z","shell.execute_reply.started":"2022-12-10T15:16:40.125048Z","shell.execute_reply":"2022-12-10T15:16:40.98235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(train_files), len(val_files)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:40.988007Z","iopub.execute_input":"2022-12-10T15:16:40.990634Z","iopub.status.idle":"2022-12-10T15:16:41.002257Z","shell.execute_reply.started":"2022-12-10T15:16:40.990594Z","shell.execute_reply":"2022-12-10T15:16:41.000954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# defining the dataset\ntrain_dataset = APP_MATCHER(train_files=train_files, labels=train_labels, transform=transforms)\nval_dataset = APP_MATCHER(train_files=val_files, labels=val_labels, transform=transforms)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:42.339468Z","iopub.execute_input":"2022-12-10T15:16:42.340101Z","iopub.status.idle":"2022-12-10T15:16:49.23485Z","shell.execute_reply.started":"2022-12-10T15:16:42.340058Z","shell.execute_reply":"2022-12-10T15:16:49.233849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# testing out the dataset instance\nx0,y0, c = train_dataset[0]\nprint(f'Shape of Image: {x0.shape}, The target tensor: {c}')","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:49.239843Z","iopub.execute_input":"2022-12-10T15:16:49.242442Z","iopub.status.idle":"2022-12-10T15:16:49.351311Z","shell.execute_reply.started":"2022-12-10T15:16:49.242402Z","shell.execute_reply":"2022-12-10T15:16:49.350392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Initializing the Dataloader\n\nWe can keep iterating through the dataset instances like above. But that is not efficient, because batching is a great way to reduce compute time. It allows us efficient usage of the GPU capacity. For batching, we use torch's Dataloader which converts a `Dataset` to a `Dataloader`. We can use it to create batches of our training data and iterate over the batches.","metadata":{}},{"cell_type":"code","source":"# configuring some hyperparameters\nBATCH_SIZE = 64*4*3\nLEARNING_RATE = 0.0001","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:56.139127Z","iopub.execute_input":"2022-12-10T15:16:56.139487Z","iopub.status.idle":"2022-12-10T15:16:56.150427Z","shell.execute_reply.started":"2022-12-10T15:16:56.139449Z","shell.execute_reply":"2022-12-10T15:16:56.146411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=2)\nval_loader = torch.utils.data.DataLoader(val_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=2)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:49.372927Z","iopub.execute_input":"2022-12-10T15:16:49.375817Z","iopub.status.idle":"2022-12-10T15:16:49.503211Z","shell.execute_reply.started":"2022-12-10T15:16:49.37578Z","shell.execute_reply":"2022-12-10T15:16:49.502152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Wandb Config\n\nIn wandb, we can log loss and other metrics. We can also specify experiment-level config parameters, like what's the learning rate we are using in this experiment, the optimizer, literally any value about the current run. In the dashboard we can filter and see our results by those parameters.","metadata":{}},{"cell_type":"code","source":"LEARNING_RATE\nif use_wandb:\n wandb.config = {\n \"learning_rate\": LEARNING_RATE,\n \"epochs\": 2,\n \"batch_size\": BATCH_SIZE\n }","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:16:58.33898Z","iopub.execute_input":"2022-12-10T15:16:58.339339Z","iopub.status.idle":"2022-12-10T15:16:58.345891Z","shell.execute_reply.started":"2022-12-10T15:16:58.33931Z","shell.execute_reply":"2022-12-10T15:16:58.344439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = SiameseNetwork().to(device)\noptimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:23:14.523733Z","iopub.execute_input":"2022-12-10T15:23:14.524655Z","iopub.status.idle":"2022-12-10T15:23:14.716544Z","shell.execute_reply.started":"2022-12-10T15:23:14.524617Z","shell.execute_reply":"2022-12-10T15:23:14.71554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# i tried many times with triplet loss, but couldn't get it to work.\n# some other days, perhaps!\n# triplet_loss = nn.TripletMarginLoss(margin=1.0, p=2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# lets check out our loader\nx, y, c = next(iter(train_loader))","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:23:16.733644Z","iopub.execute_input":"2022-12-10T15:23:16.734341Z","iopub.status.idle":"2022-12-10T15:23:45.055545Z","shell.execute_reply.started":"2022-12-10T15:23:16.734305Z","shell.execute_reply":"2022-12-10T15:23:45.053936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# lets check our model using one single forward pass\nmodel.forward_once(x.to(device)).shape","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:23:45.058502Z","iopub.execute_input":"2022-12-10T15:23:45.059682Z","iopub.status.idle":"2022-12-10T15:23:50.149846Z","shell.execute_reply.started":"2022-12-10T15:23:45.059632Z","shell.execute_reply":"2022-12-10T15:23:50.148878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train loop contained in one function.\ndef train(model, device, train_loader, optimizer, epoch):\n model.train()\n # using BCELoss since our 'label' is binary - 0 or 1\n # this task is to determine if two images are similar or not\n criterion = nn.BCELoss()\n for batch_idx, (img1, img2, targets) in enumerate(train_loader):\n # recall that the dataset class returns three values\n img1, img2, targets = img1.to(device), img2.to(device), targets.to(device)\n optimizer.zero_grad()\n # pass both images to model, get the final score\n # outputs is of size batch_size x 1\n outputs = model(img1, img2).squeeze()\n loss = criterion(outputs, targets)\n loss.backward()\n optimizer.step()\n # log the metrics for this loop to wandb\n # now you can go to the site, and watch the loss realtime\n # instead of plotting using matplotlib after the run.\n if use_wandb: wandb.log({\"loss\": loss.item(), 'batch_idx': batch_idx})\n \n # print some stuff locally too\n if batch_idx % 100 == 0:\n print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n epoch, batch_idx * len(img1), len(train_loader.dataset),\n 100. * batch_idx / len(train_loader), loss.item()))\n time_elapsed = time.time() - nb_start_time\n if time_elapsed > 8*60*60: break\n del img1,img2\n ","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:23:59.523902Z","iopub.execute_input":"2022-12-10T15:23:59.524266Z","iopub.status.idle":"2022-12-10T15:23:59.534257Z","shell.execute_reply.started":"2022-12-10T15:23:59.524235Z","shell.execute_reply":"2022-12-10T15:23:59.533052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# function for test\ndef test(model, device, test_loader):\n model.eval()\n test_loss = 0\n correct = 0\n\n criterion = nn.BCELoss()\n with torch.no_grad():\n for (images_1, images_2, targets) in test_loader:\n images_1, images_2, targets = images_1.to(device), images_2.to(device), targets.to(device)\n outputs = model(images_1, images_2).squeeze()\n test_loss += criterion(outputs, targets).sum().item() # sum up batch loss\n pred = torch.where(outputs > 0.5, 1, 0) # get the index of the max log-probability\n correct_i = pred.eq(targets.view_as(pred)).sum().item()\n correct += correct_i\n\n test_loss /= len(test_loader.dataset)\n\n print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n test_loss, correct, len(test_loader.dataset),\n 100. * correct / len(test_loader.dataset)))","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:24:01.203558Z","iopub.execute_input":"2022-12-10T15:24:01.203933Z","iopub.status.idle":"2022-12-10T15:24:01.214123Z","shell.execute_reply.started":"2022-12-10T15:24:01.203899Z","shell.execute_reply":"2022-12-10T15:24:01.213139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if use_wandb: wandb.watch(model)","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:24:02.6536Z","iopub.execute_input":"2022-12-10T15:24:02.653958Z","iopub.status.idle":"2022-12-10T15:24:02.661997Z","shell.execute_reply.started":"2022-12-10T15:24:02.653922Z","shell.execute_reply":"2022-12-10T15:24:02.660885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n model.load_state_dict(torch.load(\"../input/herbarium-siamese/model.pkl\"))\n print('Pre-existing model successfully loaded!')\nexcept:\n print(\"No existing model found.\")\n pass\n\nif not nb_eval:\n scheduler = StepLR(optimizer, step_size=1, gamma=0.7)\n for epoch in range(0, 2):\n train(model, device, train_loader, optimizer, epoch)\n torch.cuda.empty_cache()\n test(model, device, val_loader)\n scheduler.step()","metadata":{"execution":{"iopub.status.busy":"2022-12-10T15:24:05.444449Z","iopub.execute_input":"2022-12-10T15:24:05.445082Z"}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if not nb_eval:\n torch.save(model.state_dict(), \"/kaggle/working/model.pkl\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# model.load_state_dict(torch.load(\"../input/herbarium-siamese/model.pkl\"))\n# torch.save(model.state_dict(), \"/kaggle/working/model.pkl\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train_loader, val_loader\ndel train_dataset, val_dataset","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\ntorch.cuda.empty_cache()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inference\n\nThe model we trained above doesn't actually help our task - which is to classify a given image into 15,501 classes. All the above model did, was to learn if two images are of same class.\n\nHow do we leverage the above model for doing our task?\n\nWe use the representations generated by this model.\n\nThe resnet backbone has been trained on the image differentiation. Now we do a forward pass through the CNN layers, and get an embedding of each image. Embedding/Vector Representation is just a single vector containing numbers that has contained in itself all the info about that image.\n\nNow, we have the embeddings for all train images. Similarly, do it for test images. We train a simple classifier like KNNs on the train images, and use it to predict on test images.","metadata":{}},{"cell_type":"markdown","source":"We can make do with the previous dataset/loaders but we are defining new simpler ones.","metadata":{}},{"cell_type":"code","source":"class simple_loader_train(Dataset):\n def __init__(self, train_files, labels, transform=None):\n super(simple_loader_train, self).__init__()\n self.train_files = train_files\n self.labels = labels\n self.transform = transform\n \n def __len__(self):\n return len(self.train_files)\n \n def __getitem__(self, index):\n image = Image.open(self.train_files[index])\n label = self.labels[index]\n if self.transform is not None:\n image = self.transform(image)\n return image, label\n \nclass simple_loader_test(Dataset):\n def __init__(self, test_files, transform=None):\n super(simple_loader_test, self).__init__()\n self.test_files = test_files\n self.transform = transform\n \n def __len__(self):\n return len(self.test_files)\n \n def __getitem__(self, index):\n image = Image.open(self.test_files[index])\n if self.transform is not None:\n image = self.transform(image)\n return image","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"simple_train_ds = simple_loader_train(train_files, train_labels, transform=transforms)\nsimple_train_loader = torch.utils.data.DataLoader(simple_train_ds, batch_size=BATCH_SIZE, shuffle=False, num_workers=2)\nsimple_test_ds = simple_loader_test(test_images, transform=transforms)\nsimple_test_loader = torch.utils.data.DataLoader(simple_test_ds, batch_size=BATCH_SIZE, shuffle=False, num_workers=2)","metadata":{"execution":{"iopub.status.busy":"2022-12-11T12:28:41.761828Z","iopub.execute_input":"2022-12-11T12:28:41.762328Z","iopub.status.idle":"2022-12-11T12:28:41.842297Z","shell.execute_reply.started":"2022-12-11T12:28:41.762248Z","shell.execute_reply":"2022-12-11T12:28:41.840941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"next(iter(simple_test_loader)).shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Get Embeddings For All Images\n\nFor each train image, do a forward pass through the model, and save the embeddings to disk (and the labels, one of the 15,501). Same for each test. No labels here.\n\nThen, load the embeddings+labels, train a classifier, and predict on test.\n\nNote that we are saving to disk, because doing it in memory caused issues. Otherwise you can simultaneously do forward pass, store embeddings, and train.","metadata":{}},{"cell_type":"code","source":"model.eval()\nfor i, (a, c) in tqdm.tqdm(enumerate(simple_train_loader)):\n a = a.to(device)\n a_embeddings = model.forward_once(a).to('cpu').detach().numpy()\n labels = c.to('cpu').numpy()\n torch.cuda.empty_cache()\n np.savez_compressed(f'/kaggle/working/train_{i}', a_embeddings=a_embeddings, labels=labels)\n del a, a_embeddings, labels\n \nfor i, a in tqdm.tqdm(enumerate(simple_test_loader)):\n a = a.to(device)\n a_embeddings = model.forward_once(a).to('cpu').detach().numpy()\n torch.cuda.empty_cache()\n np.savez_compressed(f'/kaggle/working/test_{i}', a_embeddings=a_embeddings, )\n del a, a_embeddings\n\ndel simple_train_ds, simple_train_loader, simple_test_ds, simple_test_loader","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"embeddings = []\nlabels = []\nfor i in glob.glob('/kaggle/working/train_*.npz'):\n loaded = np.load(i)\n embedding = loaded['a_embeddings']\n label = loaded['labels']\n embeddings.append(embedding)\n labels.append(label)\n\ntest_embeddings = []\nfor i in glob.glob('/kaggle/working/test_*.npz'):\n loaded = np.load(i)\n embedding = loaded['a_embeddings']\n test_embeddings.append(embedding)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"embeddings = np.concatenate(embeddings)\nlabels = np.concatenate(labels)\ntest_embeddings = np.concatenate(test_embeddings)\nprint(\"Train embeddings:\", embeddings.shape,\n \"Labels:\", labels.shape, \n \"Test embeddings:\", test_embeddings.shape)\n\nX_train, X_val, y_train, y_val = train_test_split(embeddings, labels, \n test_size=0.2, \n random_state=0, \n stratify=labels)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.neighbors import KNeighborsClassifier\nneigh = KNeighborsClassifier(n_neighbors=3)\nneigh.fit(X_train, y_train)\nprint(f'Training Score: {neigh.score(X_train, y_train)}')\nprint(f'Validation Score: {neigh.score(X_val, y_val)}')\npredictions = neigh.predict(test_embeddings)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission = pd.read_csv('../input/herbarium-2022-fgvc9/sample_submission.csv')\nsample_submission.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_ids = [i['image_id'] for i in test_meta]\nsubm = pd.DataFrame()\nsubm['Id'] = image_ids\nsubm['Predicted'] = predictions\nsubm.to_csv('sample_submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Future Improvements\n\n1. Can try out the entire dataset instead of subset\n2. Run for more epochs than just one\n3. Try hyperparameter tuning.\n4. Use triplet loss","metadata":{}}]}