{"cells":[{"metadata":{},"cell_type":"markdown","source":"The New York Botanical Garden (NYBG) herbarium contains more than 7.8 million plant and fungal specimens. Herbaria are a massive repository of plant diversity data. These collections not only represent a vast amount of plant diversity, but since herbarium collections include specimens dating back hundreds of years, they provide snapshots of plant diversity through time. The integrity of the plant is maintained in herbaria as a pressed, dried specimen; a specimen collected nearly two hundred years ago by Darwin looks much the same as one collected a month ago by an NYBG botanist. All specimens not only maintain their morphological features but also include collection dates and locations, and the name of the person who collected the specimen. This information, multiplied by millions of plant collections, provides the framework for understanding plant diversity on a massive scale and learning how it has changed over time."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np                                    # Array, Linear Algebra\nfrom torch.utils.data.dataset import random_split     # spliting inTrain Val\nimport pandas as pd                                   # handling CSV\nimport os                                             # For File handling\nimport random                                         # Choosing from images dataset\nimport time                                           # timing Epochs  \nfrom tqdm.notebook import tqdm                        # Testing\nfrom os.path import join                              # File Handling\nfrom torchvision import transforms                    # Data Aug\nimport torch                                          # Framework\nfrom PIL import Image                                 # Loading Image\nfrom torch.utils.data import Dataset, DataLoader      # Dataset\nimport torch.nn.functional as F                       # Function\nimport json                                           # Loading Metadat\nfrom PIL import  ImageOps                             # Data Aug \nfrom PIL.Image import open as openIm                  # Image Handling\nimport matplotlib.pyplot  as plt                      # Ploting Image\nimport cv2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Configuration"},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN       = \"../input/herbarium-2020-fgvc7/nybg2020/train/\"\nTEST        = \"../input/herbarium-2020-fgvc7/nybg2020/test/\"\nMETA        = \"metadata.json\"\nBATCH_SIZE  = 7\nNUM_WORKERS = 2\nBATCH_EVAL  = 1\nSHUFFLE     = True\nEPOCHS      = 3\nRESIZE      = (800, 600)\nCLASSES     = 32094\nLENGTH      = 2*CLASSES","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## DATA INSIGTH"},{"metadata":{},"cell_type":"markdown","source":"The dataset is in COCO Format.\n\n\nCOCO is a large image dataset designed for object detection, segmentation, person keypoints detection, stuff segmentation, and caption generation. This package provides Matlab, Python, and Lua APIs that assists in loading, parsing, and visualizing the annotations in COCO. Please visit http://cocodataset.org/ for more information on COCO, including for the data, paper, and tutorials. The exact format of the annotations is also described on the COCO website. The Matlab and Python APIs are complete, the Lua API provides only basic functionality."},{"metadata":{},"cell_type":"markdown","source":"**Here is how the json file Looks Like**"},{"metadata":{},"cell_type":"markdown","source":"### TRAIN FILE"},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(join(TRAIN,META),\"r\", encoding = \"ISO-8859-1\") as file:\n    metadata = json.load(file)\nprint(\"Metadata has {} sections. These section has all the Information regarding Images in dataset like class, id, size etc. \".format(len(list(metadata.keys()))))\nprint(\"Let us see al the sections in metadata:- \", [print(\" - \",i) for i in list(metadata.keys())])\n\nprint(\"Number of Images in our Training set is:- \", len(metadata[\"images\"]))\nprint(\"\\n Let us see how every section of Dataset Looks like:-\\n\")\nfor i in list(metadata.keys()):\n    print(\" - sample and number of elements in {} :- \".format(i),len(list(metadata[i])))\n    print(\"\\t\",list(metadata[i])[0], end = \"\\n\\n\")\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### TEST FILE"},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(join(TEST,META),\"r\", encoding = \"ISO-8859-1\") as file:\n    metadata_test = json.load(file)\nprint(\"Metadata has {} sections. These section has all the Information regarding Images in dataset like class, id, size etc. \".format(len(list(metadata_test.keys()))))\nprint(\"Let us see al the sections in metadata:- \", [print(\" - \",i) for i in list(metadata_test.keys())])\n\nprint(\"Number of Images in our Training set is:- \", len(metadata_test[\"images\"]))\nprint(\"\\n Let us see how every section of Dataset Looks like:-\\n\")\nfor i in list(metadata_test.keys()):\n    print(\" - sample and number of elements in {} :- \".format(i),len(list(metadata_test[i])))\n    print(\"\\t\",list(metadata_test[i])[0], end = \"\\n\\n\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are 1030747 Images in Train set.\n\nThere are 32094 Classes in The dataset."},{"metadata":{},"cell_type":"markdown","source":"Now let us see the Image Sample."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_img = pd.DataFrame(metadata['images'])\ntrain_ann = pd.DataFrame(metadata['annotations'])\ntrain_df = pd.merge(train_ann, train_img, left_on='image_id', right_on='id', how='left').drop('image_id', axis=1).sort_values(by=['category_id'])\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"im = Image.open(\"../input/herbarium-2020-fgvc7/nybg2020/train/images/156/72/354106.jpg\")\nprint(\"Category Id is 15672 and Image Id is 354106 is shown below\")\nim","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**The Distribution**"},{"metadata":{"trusted":true},"cell_type":"code","source":"size_of_img = (28, 28)\nfig=plt.figure(figsize=(72,72))\nfor i in range(60):\n    ax=fig.add_subplot(12,12,i+1)\n    img = cv2.imread(TRAIN + metadata[\"images\"][i][\"file_name\"])\n    img = cv2.resize(img,size_of_img)\n    ax.imshow(img)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## DATALOADER"},{"metadata":{},"cell_type":"markdown","source":"A lot of effort in solving any machine learning problem goes in to preparing the data. PyTorch provides many tools to make data loading easy and hopefully, to make your code more readable. In this tutorial, we will see how to load and preprocess/augment data from a non trivial dataset.\n\n\nDataset class\ntorch.utils.data.Dataset is an abstract class representing a dataset. Your custom dataset should inherit Dataset and override the following methods:\n\n__len__ so that len(dataset) returns the size of the dataset.\n__getitem__ to support the indexing such that dataset[i] can be used to get ith sample\nLet’s create a dataset class for our face landmarks dataset. We will read the csv in __init__ but leave the reading of images to __getitem__. This is memory efficient because all the images are not stored in the memory at once but read as required."},{"metadata":{"trusted":true},"cell_type":"code","source":"def random_img(cat,df, tmp):\n    \n    cat_df = train_df[train_df[\"category_id\"] == cat]\n    try: \n        return TRAIN + cat_df.iloc[tmp, 3]\n    except:\n        TRAIN + cat_df.iloc[0, 3]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fill(df, cat, sim):\n    pair = [random_img(cat,df, 0)]\n    ncat = cat if sim else random.randint(0,CLASSES)\n    pair.append(random_img(ncat,df,1))\n    return pair","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"class HerbariumDataset(Dataset): \n\n    def __init__(self, Folder,metadata, trans, length, phase):\n        \n        self.transform  =  trans\n        self.length     =  length if length != None else len(metadata[\"images\"])\n        self.root       =  Folder\n        self.metadata   =  metadata\n        self.phase      =  phase\n        if self.phase == \"TRAIN\":\n            for i in self.metadata[\"categories\"]:\n                self.cat_id =  {i[\"family\"]:i[\"id\"]}\n                self.id_cat =  {i[\"id\"]:i[\"family\"]}\n            self.id_cat     =  {a[\"id\"] : a[\"category_id\"] for a in self.metadata[\"annotations\"]}\n            self.classes    =  list(self.cat_id.keys())\n        self.paths      =  {im[\"id\"] : join(self.root , im[\"file_name\"]) for im in self.metadata[\"images\"]}\n        \n    \n    def _rand_another(self, idx):\n        pool = np.arange(self.length)\n        return np.random.choice(pool)\n                                \n    def image(self, x):\n        \n        try:\n            im  = openIm(x[0])\n            im1 = openIm(x[1])\n            \n            return im, im1\n        \n        except:\n            print(\"The Image id {} not found \".format(x))\n            return None , None\n    \n                     \n\n    def __len__(self):\n        \n        return self.length\n        \n        \n\n    def __getitem__(self, idx):\n        idx = idx.tolist() if torch.is_tensor(idx) else idx\n        if self.phase == \"TRAIN\":\n            while True:\n                sim = idx % 2\n                cat = idx // 2\n                pair = fill(train_df, cat, sim)\n                im, im2 = self.image(pair)\n                if im is None or im2 is None:\n                    print(\"\\nfrom \", idx,end =  \"\")\n                    idx = self._rand_another(idx)\n                    print(\" Moving to Image id :-\",idx)\n                    continue\n                if self.transform:\n                    im  = self.transform(im)\n                    im1 = self.transform(im2)\n                    return im, im1, sim\n                return im, im1, sim\n\n        elif self.phase == \"TEST\":\n            im, im2 = self.image(pair)\n            if self.transform:\n                im = self.transform(im)\n                im2 = self.transform(im2)\n\n            return im, im2  \n        else:\n            im   = Image.open(TRAIN + train_df[train_df[\"category_id\"]  == idx].iloc[0,3])\n            im   = trans(im)\n            return im\n            ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## DATA AUGMENTATION"},{"metadata":{},"cell_type":"markdown","source":"***Data augmentation*** is the technique of increasing the size of data used for training a model. For reliable predictions, the deep learning models often require a lot of training data, which is not always available. Therefore, the existing data is augmented in order to make a better generalized model.\n\nAlthough data augmentation can be applied in various domains, it's commonly used in computer vision. Some of the most common data augmentation techniques used for images are:\n\n1) Position augmentation\n* Scaling\n* Cropping\n* Flipping\n* Padding\n* Rotation\n* Translation\n\n2) Affine transformation\n* Color augmentation\n* Brightness\n* Contrast\n* Saturation\n* Hue"},{"metadata":{"trusted":true},"cell_type":"code","source":"class invert:\n    def invert(self, img):\n        r\"\"\"Invert the input PIL Image.\n        Args:\n            img (PIL Image): Image to be inverted.\n        Returns:\n            PIL Image: Inverted image.\n        \"\"\"\n\n        return ImageOps.invert(img)\n    def __call__(self, img):\n            \n        return self.invert(img)\n\n    def __repr__(self):\n        \n        return self.__class__.__name__ + '()'\ntrans = transforms.Compose(\n    [invert(),\n     transforms.CenterCrop(400),\n     transforms.ColorJitter(brightness=0.3, contrast=0.3, saturation=0.1, hue=0.1),\n     transforms.RandomHorizontalFlip(p=0.50),\n     transforms.RandomRotation(25), \n     transforms.Resize(RESIZE, interpolation=2),\n     transforms.ToTensor(),\n     #transforms.Normalize([123.675, 116.28, 103.53],[58.395, 57.12, 57.375]),\n    ])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def torch_loader( dataset, batch_size, num_workers, shuffle):\n    return DataLoader(dataset = dataset, batch_size = batch_size, num_workers = num_workers, shuffle = shuffle)\n\ntrain_dataset = HerbariumDataset(TRAIN, metadata, trans, LENGTH, \"TRAIN\")\ntest_dataset  = HerbariumDataset(TEST, metadata_test, trans, None, \"TEST\")\ntrain_loader  = torch_loader( train_dataset, BATCH_SIZE, NUM_WORKERS, SHUFFLE)\ntest_loader   = torch_loader( test_dataset, 3*(BATCH_SIZE), 2, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**This is How our Image is Going to look before feeding it to model**"},{"metadata":{"trusted":true},"cell_type":"code","source":"im = train_dataset.__getitem__(127)[0]\ntrans = transforms.ToPILImage()\nim = trans(im)\nprint(np.array(im).shape)\nim","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## MODEL"},{"metadata":{},"cell_type":"markdown","source":"> In case of **standard classification**, the input image is fed into a series of layers, and finally at the output we generate a probability distribution over all the classes (typically using a Softmax). For example, if we are trying to classify an image as cat or dog or horse or elephant, then for every input image, we generate 4 probabilities, indicating the probability of the image belonging to each of the 4 classes. Two important points must be noticed here. First, during the training process, we require a large number of images for each of the class (cats, dogs, horses and elephants). Second, if the network is trained only on the above 4 classes of images, then we cannot expect to test it on any other class, example “zebra”. If we want our model to classify the images of zebra as well, then we need to first get a lot of zebra images and then we must re-train the model again. There are applications wherein we neither have enough data for each class and the total number classes is huge as well as dynamically changing. Thus, the cost of data collection and periodical re-training is too high.\nOn the other hand, in a **one shot classification**, we require only **one training example** for each class. Yes you got that right, just one. Hence the name One Shot."},{"metadata":{},"cell_type":"markdown","source":"### Few shot learning\nOne of the main requisites of highly accurate deep learning models is large amount of data. The set of hyperparameters a Deep Model need to be tuned are very large, and the amount of data needed to get the right set of value for these hyperparameters is also large.\n\nBut what if we need an automated system, which can successfully classify images to various classes given the data for each image class is quite less.\n\nFew shot learning is such a problem. We can Few shot learning as a problem to classify data into K classes where each class has only few examples. The paper written by Gregory et. al, suggest ideas for building a Neural Network Architecture to solve this problem.\n\n![](https://camo.githubusercontent.com/1d29ae8092dea858f45e4519b7454782df3c9328/68747470733a2f2f656e637279707465642d74626e302e677374617469632e636f6d2f696d616765733f713d74626e3a414e643947635154684d757375386232754b386b4777724673672d63755a58614e38576337486b666779694d2d3859416643664e5f3275694a51)"},{"metadata":{},"cell_type":"markdown","source":"The above image has been chosen from the Coursera course on Deep Learning by DeepLearning.ai\nMachine learning has been successfully used to achieve state-ofthe-art performance in a variety of applications such as web search, spam detection, caption generation, and speech and image recognition. However, these algorithms often break down when forced to make predictions about data for which little supervised information is available. We desire to generalize to these unfamiliar categories without necessitating extensive retraining which may be either expensive or impossible due to limited data or in an online prediction setting, such as web retrieval.\n\nOne particularly interesting task is classification under the restriction that we may only observe a single example of each possible class before making a prediction about a test instance. This is called one-shot learning and it is the primary focus of our model presented in this work"},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\ndevice","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn as nn\nfrom torch.nn import functional as F\nimport torch.optim as optim\nimport torchvision \nmodel = torchvision.models.resnet50(pretrained=True).to(device)\n    \nfor param in model.parameters():\n    param.requires_grad = True   \n    \nmodel.fc = nn.Sequential(*list(model.fc.children())[:-1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class ContrastiveLoss(nn.Module):\n\n    def __init__(self, margin=2.0):\n        super(ContrastiveLoss, self).__init__()\n        self.margin = margin\n\n    def forward(self, output1, output2, label):\n        euclidean_distance = F.pairwise_distance(output1, output2, keepdim = True)\n        loss_contrastive = torch.mean((1-label) * torch.pow(euclidean_distance, 2) +\n                                      (label) * torch.pow(torch.clamp(self.margin - euclidean_distance, min=0.0), 2))\n\n\n        return loss_contrastive","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_model(model, num_epochs=1):\n    \n    criterion  = ContrastiveLoss()\n    optimizer  = optim.Adam(model.parameters(), lr=0.01)\n    for epoch in range(num_epochs):\n        phase = \"train\"\n        print('Epoch {}/{}'.format(epoch+1, num_epochs), end = \"\\n\"+\"_\" * 75 + \"\\n\")\n        model.train()\n\n        running_loss     = 0.0\n        starttime = time.time()\n        for i, (inputs1, inputs2, labels) in enumerate(train_loader):\n            try:\n                inputs1 = inputs1.to(device)\n                inputs2 = inputs2.to(device)            \n                labels = labels.to(device)\n                outputs1 = model(inputs1)\n                outputs2 = model(inputs2)\n                loss = criterion(outputs1, outputs2, labels)\n                optimizer.zero_grad()\n                loss.backward()\n                optimizer.step()\n\n\n                running_loss += loss.item() \n                if i % 100  == 0 and i != 0:\n                    print(\"running_loss:- {:.4f} \".format(running_loss))\n            except:\n                print(\"ERROR IN BATCH\", i)\n                continue\n\n                \n        epoch_loss = running_loss / len(train_loader)\n        print('{} epoch_loss: {:.4f}, time:-{:.4f}  '.format(phase,epoch_loss,starttime - time.time()))\n        \n        if epoch != 0:  # Due to Memory Constrain\n            torch.save(model, \"/kaggle/working/epoch1_{}.pth\".format(epoch+1))\n            \n        \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_trained = train_model(model, num_epochs=EPOCHS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.cuda.empty_cache()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}