{"cells":[{"metadata":{},"cell_type":"markdown","source":"The Herbarium 2021: Half-Earth Challenge is to identify vascular plant specimens provided by the New York Botanical Garden (NY), Bishop Museum (BPBM), Naturalis Biodiversity Center (NL), Queensland Herbarium (BRI), and Auckland War Memorial Museum (AK).\n\nThe Herbarium 2021: Half-Earth Challenge dataset includes more than 2.5M images representing nearly 65,000 species from the Americas and Oceania that have been aligned to a standardized plant list (LCVP v1.0.2)."},{"metadata":{},"cell_type":"markdown","source":"## Disclaimer\n\nThis kernel is heavily inspired from [@yasufuminakama](https://www.kaggle.com/yasufuminakama)'s kernel from last year's competition [Herbarium 2020 PyTorch Resnet18 [train]](https://www.kaggle.com/yasufuminakama/herbarium-2020-pytorch-resnet18-train)"},{"metadata":{},"cell_type":"markdown","source":"# Table of Content\n\n1. [Packages 📦 and Basic Setup](#packages)\n2. [🏷 Label Encoder](#label-encoder)\n3. [💿 Dataset, DataLoader and K-Fold](#data)\n4. [The Model 👷‍♀️](#Model)\n5. [Training 💪🏻](#Training)"},{"metadata":{},"cell_type":"markdown","source":"<a id = \"packages\"></a>\n# Packages 📦 and Basic Setup\n\nIn the following **hidden** code cell, we: \n\n* Import the required libraries (Main ones being [`torch`](https://pytorch.org/), [`torchvision`](https://pytorch.org/vision/0.8/index.html) and [`sklearn`](https://scikit-learn.org/stable/index.html))\n* Print the device configuration\n* Set Random Seed 🌱 to ensure reproducibility\n* Create a [Logger 📃](https://docs.python.org/3/library/logging.html) for Event Logging"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"%%capture\n!pip install --upgrade wandb","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Import Statements\nimport os # To set Random Seed for Reproducibility\nimport cv2 # For Image 🌌 Processing\nimport json # For Reading in the JSON file\nimport torch # The Main Machine Learning Framework\nimport random # To set Random Seed for Reproducibility\nimport logging # For Event Logging\nimport sklearn # For LabelEncoder and Metrics\nimport torchvision # For creating a pretrained model\nimport numpy as np # For Numerical Processing\nimport pandas as pd # For creating DataFrames \nimport albumentations # For Image Augmentations\nfrom tqdm import tqdm # For Creating ProgressBar\nfrom sklearn import preprocessing # For the 🏷 Label Encoder\nfrom albumentations.pytorch import ToTensorV2 # For Converting to torch.Tensor\nfrom sklearn.model_selection import StratifiedKFold # For Cross Validation\n\n\n# Device Configuration\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\ndevice\n\n# Setting RandomSeed🌱 for Reproducibility \ndef seed_torch(seed:int =42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n\nseed_torch()\n\n# Creating a logger 📃\ndef init_logger(log_file:str ='training.log'):\n    \n    # Specify the format \n    formatter = logging.Formatter('%(levelname)s:%(name)s:%(message)s')\n    \n    # Create a StreamHandler Instance\n    stream_handler = logging.StreamHandler()\n    stream_handler.setLevel(logging.DEBUG)\n    stream_handler.setFormatter(formatter)\n    \n    # Create a FileHandler Instance\n    file_handler = logging.FileHandler(log_file)\n    file_handler.setFormatter(formatter)\n    \n    # Create a logging.Logger Instance\n    logger = logging.getLogger('Herbarium')\n    logger.setLevel(logging.DEBUG)\n    logger.addHandler(stream_handler)\n    logger.addHandler(file_handler)\n    \n    return logger\n\nLOGGER = init_logger()\nLOGGER.info(\"Logger Initialized\")\n\n# Logging Into Weights and Biases\nimport wandb\nfrom kaggle_secrets import UserSecretsClient\n\nuser_secrets = UserSecretsClient()\napi_key = user_secrets.get_secret(\"WANDB_API_KEY\")\nwandb.login(key=api_key);\nLOGGER.info(\"Logged into Weights and Biases\")\nwandb.init(project='Herbarium 2021', entity='sauravmaheshkar')\nLOGGER.info(\"New run Created\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Basic Parameters for the Model\nN_CLASSES = 64500\nHEIGHT = 128\nWIDTH = 128\nbatch_size = 512\nn_epochs = 1\nlr = 4e-4","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nwith open('../input/herbarium-2021-fgvc8/train/metadata.json', \"r\", encoding=\"ISO-8859-1\") as file:\n    train = json.load(file)\n\ntrain_img = pd.DataFrame(train['images'])\ntrain_ann = pd.DataFrame(train['annotations']).drop(columns='image_id')\ntrain_df = train_img.merge(train_ann, on='id')\nLOGGER.info(\"Train DataFrame Created: ✅\")\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see the DataFrame contains the following columns:\n\n* `file_name: str`: Contains the Relative Path to the Images\n* `height: int`: The Height of the Image\n* `width: int`: The Width of the Image\n* `id: int`: The Image ID\n* `license: int`: Either 0, 1 or 2\n* `category_id: int`: Our **Target Variable** determining the species corresponding to the Image"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['category_id'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.hist(column=\"category_id\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nwith open('../input/herbarium-2021-fgvc8/test/metadata.json', \"r\", encoding=\"ISO-8859-1\") as file:\n    test = json.load(file)\n\ntest_df = pd.DataFrame(test['images'])\nLOGGER.info(\"Test DataFrame Created: ✅\")\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"label-encoder\"></a>\n# 🏷 Label Encoder\n\nWe use the [LabelEncoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html) from sklearn.preprocessing in order to encode **target labels** with value between `0` and `n_classes-1`."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create a Instance of LabelEncoder\nencoder = preprocessing.LabelEncoder()\nLOGGER.info(\"LabelEncoder Instance created ✅\")\n\n# Fits the label encoder instance\nLOGGER.info(\"Fitting the LabelEncoder Instance\")\nencoder.fit(train_df['category_id'])\n\n# To Transform labels to normalized encoding\nLOGGER.info(\"Converting Labels to Normalized Encoding\")\ntrain_df['category_id'] = encoder.transform(train_df['category_id'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"Data\"></a>\n# 💿 Dataset, DataLoader and K-Fold"},{"metadata":{},"cell_type":"markdown","source":"The following code cell aims to convert the Herbarium dataset into a torch `torch..utils.data.Dataset` object.\n\nAll Dataset objects in pytorch represent a map from keys to data samples. We create a subclass which overwrites the __getitem__() and __len__() method (for it to work well with the `torch.utils.data.DataLoader`).\n\nIn the __getitem__() method, we use df[<column_name>].values[] to get the `file_name `and then use `cv2` to read the image. If the transform bool is set to True, we apply the transforms. \n\nEach element of our dataset returns:\n\n* Image\n* Encoded Label"},{"metadata":{"trusted":true},"cell_type":"code","source":"class TrainDataset(torch.utils.data.Dataset):\n    \"\"\"\n    Custom Dataset Class\n    \"\"\"\n    def __init__(self, df, labels, transform=None):\n        self.df = df\n        self.labels = labels\n        self.transform = transform\n        \n    def __len__(self) -> int:\n        return len(self.df)\n\n    def __getitem__(self, idx):\n        file_name = self.df['file_name'].values[idx]\n        file_path = f'../input/herbarium-2021-fgvc8/train/{file_name}'\n        image = cv2.imread(file_path)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        label = self.labels.values[idx]\n        \n        if self.transform:\n            augmented = self.transform(image=image)\n            image = augmented['image']\n        \n        return image, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Image Augmentation:🌆 -> 🌇\n\nApplying Standard Image Augmentation Techniques, such as `Resize`, `Normalize` and conversion to `torch.Tensor`"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_transforms(*, data: str):\n    \n    # Only allows transforms for \"train\" and \"valid\"\n    assert data in ('train', 'valid')\n    \n    # Data Transformations for Training set\n    if data == 'train':\n        return albumentations.Compose([\n            albumentations.Resize(HEIGHT, WIDTH),\n            albumentations.Normalize(\n                mean=[0.485, 0.456, 0.406],\n                std=[0.229, 0.224, 0.225],\n            ),\n            ToTensorV2(),\n        ])\n    \n    # Data Transformations for Validation set\n    elif data == 'valid':\n        return albumentations.Compose([\n            albumentations.Resize(HEIGHT, WIDTH),\n            albumentations.Normalize(\n                mean=[0.485, 0.456, 0.406],\n                std=[0.229, 0.224, 0.225],\n            ),\n            ToTensorV2(),\n        ])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## StratifiedKFold\n\nStratified kfold cross validation is an extension of regular kfold cross validation but specifically for **classification problems** where rather than the splits being completely random, the ratio between the target classes is the same in each fold as it is in the full dataset. We choose to use StratifiedKFold because we have some class imbalance in our Dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"# For Debugging\nDEBUG = False\n\nif DEBUG:\n    folds = train_df.sample(n=10000, random_state=0).reset_index(drop=True).copy()\nelse:\n    folds = train_df.copy()\n\n# Create a seperate train_labels variable\ntrain_labels = folds['category_id'].values\n\n# Create StratifiedKFold Instance\nLOGGER.info(\"Creating StratifiedKFold Instance\")\nkf = StratifiedKFold(n_splits=2)\n\n# Creating Splits\nLOGGER.info(\"Creating Splits\")\nfor fold, (train_index, val_index) in enumerate(kf.split(folds.values, train_labels)):\n    folds.loc[val_index, 'fold'] = int(fold)\n    \nfolds['fold'] = folds['fold'].astype(int)\n\n# Exporting to CSV\nLOGGER.info(\"Exporting Folds to csv\")\nfolds.to_csv('folds.csv', index=None)\n\n#save it as model artifact on W&B\nartifact =  wandb.Artifact(name=\"folds\", type=\"dataset\")\nartifact.add_file(\"folds.csv\")\nLOGGER.info(\"Logging folds.csv to W&B Artifacts\")\nwandb.log_artifact(artifact)\n\n# Creating Train and Valid sets\nFOLD = 0\ntrn_idx = folds[folds['fold'] != FOLD].index\nval_idx = folds[folds['fold'] == FOLD].index","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It takes a lot of time to generate folds, that's we use Weights and Biases Artifacts to store our `folds.csv`. You can store different versions of your datasets and models in the cloud as Artifacts. Think of an Artifact as of a folder of data to which we can add individual files, and then upload to the cloud as a part of our W&B project, which also supports automatic versioning of datasets and models. Artifacts also track the training pipelines as DAGs. Here's an example of a  artifacts graph.\n\n![](https://i.imgur.com/QQULnpP.gif)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create Training Dataset\ntrain_dataset = TrainDataset(folds.loc[trn_idx].reset_index(drop=True), \n                             folds.loc[trn_idx]['category_id'], \n                             transform=get_transforms(data='train'))\nLOGGER.info(\"Training Dataset Object Created ✅\")\n\n# Create Validation Dataset\nvalid_dataset = TrainDataset(folds.loc[val_idx].reset_index(drop=True), \n                             folds.loc[val_idx]['category_id'], \n                             transform=get_transforms(data='valid'))\nLOGGER.info(\"Validation Dataset Object Created ✅\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"LOGGER.info(\"Creating Training DataLoader\")\ntrain_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True, num_workers=4)\nLOGGER.info(\"Training DataLoader created\")\n\nLOGGER.info(\"Creating Validation DataLoader\")\nvalid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=batch_size, shuffle=False, num_workers=4)\nLOGGER.info(\"Validation DataLoader created\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id = \"Model\"></a>\n# The Model 👷‍♀️\n\n---\n\n## Transfer Learning\n\nThe main aim of transfer learning (TL) is to implement a model quickly i.e. instead of creating a DNN (dense neural network) from scratch, the model will transfer the features it has learned from the different dataset that has performed the same task. This transaction is also known as **knowledge transfer**.\n\n## Resnet18\n\nA residual network, or ResNet for short, is a DNN that helps to build deeper neural networks by utilizing skip connections or shortcuts to jump over some layers. This helps solve the problem of **vanishing gradients**.\n\nThere are different versions of ResNet, including ResNet-18, ResNet-34, ResNet-50, and so on. The numbers denote layers, although the architecture is the same.\n\n![](https://i.imgur.com/XwcnU5x.png)\n\nIn the end, we just add a Adaptive Pooling Layer and a Fully Connected Layer with output dimensions equal to the number of classes."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%capture\n# Creating a instance of a Resnet18 pretrained Model\nmodel = torchvision.models.resnet18(pretrained=True)\n\n# Add a Adaptive Average Pooling Layer\nmodel.avgpool = torch.nn.AdaptiveAvgPool2d(1)\n\n# Add a Fully connected Layer with N_CLASSES as the output dimension\nmodel.fc = torch.nn.Linear(model.fc.in_features, N_CLASSES)\n\nLOGGER.info(\"Model Created ✅\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"Training\"></a>\n\n# Training 💪🏻\n\nlooping over our data iterator, and feed the inputs to the network and optimize."},{"metadata":{"trusted":true},"cell_type":"code","source":"# moves the model to the device\nmodel.to(device)\n    \n# Creating Optimizers and Schedulers\noptimizer = torch.optim.Adam(model.parameters(), lr=lr, amsgrad=False)\nscheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.75, patience=5, verbose=True, eps=1e-6)\n    \n# Loss Function\ncriterion = torch.nn.CrossEntropyLoss()\nbest_score = 0.\nbest_loss = np.inf\n\nLOGGER.info(\"Starting Training\")\nwandb.watch(model)\n    \nfor epoch in range(n_epochs):\n        \n    # Sets the model in training mode\n    model.train()\n    avg_loss = 0.\n\n    # Sets gradients of all model parameters to zero\n    optimizer.zero_grad()\n\n    for i, (images, labels) in tqdm(enumerate(train_loader)):\n        \n        # Move Image and Labels to device\n        images = images.to(device)\n        labels = labels.to(device)\n            \n        # Forward Pass\n        y_preds = model(images)\n        \n        # Calculate Loss\n        loss = criterion(y_preds, labels)\n        \n        # Log our Training Loss to W&B\n        wandb.log({\"loss\": loss})\n            \n        # Backpropagation\n        loss.backward()\n        optimizer.step()\n        optimizer.zero_grad()\n\n        # Update Average Loss\n        avg_loss += loss.item() / len(train_loader)\n            \n    # Set Model in Eval mode\n    model.eval()\n    avg_val_loss = 0.\n    preds = np.zeros((len(valid_dataset)))\n\n    for i, (images, labels) in enumerate(valid_loader):\n            \n        # Move Image and Labels to device\n        images = images.to(device)\n        labels = labels.to(device)\n            \n        # To perform inference without Gradient Calculation\n        with torch.no_grad():\n            y_preds = model(images)\n            \n        preds[i * batch_size: (i+1) * batch_size] = y_preds.argmax(1).to('cpu').numpy()\n\n        # Calculate Loss\n        loss = criterion(y_preds, labels)\n        \n        # Log our Validation Loss to W&B\n        wandb.log({\"val_loss\": loss})\n        \n        # Update Average Validation Loss\n        avg_val_loss += loss.item() / len(valid_loader)\n        \n    # Updates Learning Rate wrt to the Average Validation Loss\n    scheduler.step(avg_val_loss)\n            \n    # Calculate F1 Score\n    score = sklearn.metrics.f1_score(folds.loc[val_idx]['category_id'].values, preds, average='macro')\n    \n    # Log our F1 Score to W&B\n    wandb.log({\"f1_score\": score})\n\n    # Prints Stats\n    LOGGER.debug(f'  Epoch {epoch+1} - avg_train_loss: {avg_loss:.4f}  avg_val_loss: {avg_val_loss:.4f}  F1: {score:.6f}')\n\n    # Update Best Score\n    if score>best_score:\n        best_score = score\n        LOGGER.debug(f'  Epoch {epoch+1} - Save Best Score: {best_score:.6f} Model')\n        torch.save(model.state_dict(), f'fold{FOLD}_best_score.pth')\n\n    if avg_val_loss<best_loss:\n        best_loss = avg_val_loss\n        LOGGER.debug(f'  Epoch {epoch+1} - Save Best Loss: {best_loss:.4f} Model')\n        torch.save(model.state_dict(), f'fold{FOLD}_best_loss.pth')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Resources\n\n- Logging Tutorial Series by Corey Schafer:\n    - [Video 🎥 1: Python Tutorial: Logging Basics - Logging to Files, Setting Levels, and Formatting\n](https://youtu.be/-ARI4Cz-awo)\n    - [Video 🎥 2: Python Tutorial: Logging Advanced - Loggers, Handlers, and Formatters\n](https://youtu.be/jxmzY9soFXg)\n\n- [Stratified K-Fold: What It Is & How to Use It](https://towardsdatascience.com/stratified-k-fold-what-it-is-how-to-use-it-cf3d107d3ea2)\n\n- [Weights and Biases: Getting Started](https://wandb.ai/site/getting-started)\n\n- [Compete More Effectively on Kaggle using Weights and Biases](https://youtu.be/Y7rjzfpWSo8)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}