{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Herbarium 2021: Half-Earth Challenge - FGVC8 - Exploratory Data Analysis\n\nQuick Exploratory Data Analysis for [Herbarium 2021: Half-Earth Challenge - FGVC8](https://www.kaggle.com/c/herbarium-2021-fgvc8) challenge    \n\nThe Herbarium 2021: Half-Earth Challenge is to identify vascular plant specimens provided by the [New York Botanical Garden (NY)](https://www.nybg.org/), [Bishop Museum (BPBM)](https://www.bishopmuseum.org/), [Naturalis Biodiversity Center (NL)](https://www.naturalis.nl/en), [Queensland Herbarium (BRI)](https://www.qld.gov.au/environment/plants-animals/plants/herbarium), and [Auckland War Memorial Museum (AK)](https://www.aucklandmuseum.com/)."},{"metadata":{},"cell_type":"markdown","source":"![](https://storage.googleapis.com/kaggle-competitions/kaggle/25558/logos/header.png)"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"top\"></a>\n\n<div class=\"list-group\" id=\"list-tab\" role=\"tablist\">\n<h3 class=\"list-group-item list-group-item-action active\" data-toggle=\"list\" style='color:white; background:brown; border:0' role=\"tab\" aria-controls=\"home\"><center>Quick Navigation</center></h3>\n\n* [Overview](#1)\n* [Data Visualization](#2)\n    \n    \n* [Competition Metric](#100)\n* [Sample Submission](#101)\n    \n    \n* [Modeling](#200)"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"1\"></a>\n<h2 style='background:brown; border:0; color:white'><center>Overview<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"The training and test set contain images of herbarium specimens from nearly 65,000 species of vascular plants. Each image contains exactly one specimen. The text labels on the specimen images have been blurred to remove category information in the image."},{"metadata":{},"cell_type":"markdown","source":"The data has been approximately split 80%/20% for training/test. Each category has at least 1 instance in both the training and test datasets. Note that the test set distribution is slightly different from the training set distribution. The training set contains species with hundreds of examples, but the test set has the number of examples per species capped at a maximum of 10."},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport json\nimport collections\n\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Read the metadata file"},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH_BASE = \"../input/herbarium-2021-fgvc8/\"\nPATH_TRAIN = os.path.join(PATH_BASE, \"train/\")\nPATH_TRAIN_META = os.path.join(PATH_TRAIN, \"metadata.json\")\n\n\nwith open(PATH_TRAIN_META) as json_file:\n    metadata = json.load(json_file)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### First level elements"},{"metadata":{"trusted":true},"cell_type":"code","source":"metadata.keys()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Check the number of images and their annotations"},{"metadata":{"trusted":true},"cell_type":"code","source":"len(metadata[\"annotations\"]), len(metadata[\"images\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Check first samples from each key"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(metadata[\"annotations\"][0])\nprint(metadata[\"images\"][0])\nprint(metadata[\"categories\"][0])\nprint(metadata[\"licenses\"][0])\nprint(metadata[\"institutions\"][0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Calculate the total number of classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"len(set([annotation[\"category_id\"] for annotation in metadata[\"annotations\"]]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Create DataFrame with main information"},{"metadata":{"trusted":true},"cell_type":"code","source":"ids = []\ncategories = []\npaths = []\n\nfor annotation, image in zip(metadata[\"annotations\"], metadata[\"images\"]):\n    assert annotation[\"image_id\"] == image[\"id\"]\n    ids.append(image[\"id\"])\n    categories.append(annotation[\"category_id\"])\n    paths.append(image[\"file_name\"])\n        \ndf_meta = pd.DataFrame({\"id\": ids, \"category\": categories, \"path\": paths})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_meta","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Classes distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_meta[\"category\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Find name and family of the classes by their ids"},{"metadata":{"trusted":true},"cell_type":"code","source":"d_categories = {category[\"id\"]: category[\"name\"] for category in metadata[\"categories\"]}\nd_families = {category[\"id\"]: category[\"family\"] for category in metadata[\"categories\"]}\nd_orders = {category[\"id\"]: category[\"order\"] for category in metadata[\"categories\"]}\n\ndf_meta[\"category_name\"] = df_meta[\"category\"].map(d_categories)\ndf_meta[\"family_name\"] = df_meta[\"category\"].map(d_families)\ndf_meta[\"order_name\"] = df_meta[\"category\"].map(d_orders)\ndf_meta","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2\"></a>\n<h2 style='background:brown; border:0; color:white'><center>Data Visualization<center><h2>"},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_train_batch(paths, categories, families, orders):\n    plt.figure(figsize=(16, 16))\n    \n    for ind, info in enumerate(zip(paths, categories, families, orders)):\n        path, category, family, order = info\n        \n        plt.subplot(2, 3, ind + 1)\n        \n        image = cv2.imread(os.path.join(PATH_TRAIN, path))\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n        plt.imshow(image)\n        \n        plt.title(\n            f\"FAMILY: {family} ORDER: {order}\\n{category}\", \n            fontsize=10,\n        )\n        plt.axis(\"off\")\n    \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_by_id(df, _id=None):\n    tmp = df.sample(6)\n    if _id is not None:\n        tmp = df[df[\"category\"] == _id].sample(6)\n\n    visualize_train_batch(\n        tmp[\"path\"].tolist(), \n        tmp[\"category_name\"].tolist(),\n        tmp[\"family_name\"].tolist(),\n        tmp[\"order_name\"].tolist(),\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_by_id(df_meta, 22344)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_by_id(df_meta, 42811)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_by_id(df_meta, 1719)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_by_id(df_meta, 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Random samples"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_by_id(df_meta)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"100\"></a>\n<h2 style='background:brown; border:0; color:white'><center>Competition Metric<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"Submissions are evaluated using the [macro F1 score](#https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html)."},{"metadata":{},"cell_type":"markdown","source":"$$F_1 = 2\\frac{precision \\cdot recall}{precision+recall}$$"},{"metadata":{},"cell_type":"markdown","source":"where:"},{"metadata":{},"cell_type":"markdown","source":"$$precision = \\frac{TP}{TP+FP}$$"},{"metadata":{},"cell_type":"markdown","source":"$$recall = \\frac{TP}{TP+FN}$$"},{"metadata":{},"cell_type":"markdown","source":"In \"macro\" F1 a separate F1 score is calculated for each species value and then averaged."},{"metadata":{},"cell_type":"markdown","source":"<a id=\"101\"></a>\n<h2 style='background:brown; border:0; color:white'><center>Sample Submission<center><h2>"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submission = pd.read_csv(\n    \"../input/herbarium-2021-fgvc8/sample_submission.csv\",\n    index_col=0,\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### One of the most frequently class from train data"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submission[\"Predicted\"] = 25229","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submission.to_csv(\"submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.read_csv(\"submission.csv\", index_col=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"200\"></a>\n<h2 style='background:brown; border:0; color:white'><center>Modeling<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"### The idea: Create for each category abstract vector from some model (MobileNetV2) and find nearest vector for each train sample"},{"metadata":{"trusted":true},"cell_type":"code","source":"FULL_PIPELINE = False","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Import libraries"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"import os\nimport random\n\nimport numpy as np\nfrom numpy import save, load\nimport pandas as pd\nimport cv2\nimport albumentations as A\nfrom albumentations import pytorch as ATorch\nimport torch\nfrom torch.utils import data as torch_data\nfrom torch import nn as torch_nn\nfrom torch.nn import functional as torch_functional\nimport torchvision\nfrom tqdm import tqdm\nfrom sklearn.metrics.pairwise import euclidean_distances","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define the model\n\nYou can use any of the pretrained models, for example:\n- [PYTORCH HUB FOR RESEARCHERS](https://pytorch.org/hub/research-models)\n- [TORCHVISION.MODELS](https://pytorch.org/vision/stable/models.html)"},{"metadata":{"trusted":true},"cell_type":"code","source":"class MobileNetV2(torch.nn.Module):\n    def __init__(self):\n        super().__init__()\n        tmp_net = torch.hub.load(\n            \"pytorch/vision:v0.6.0\", \"mobilenet_v2\", pretrained=True\n        )\n        self.net = torch_nn.Sequential(*(list(tmp_net.children())[:-1]))\n\n    def forward(self, x):\n        return self.net(x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define your dataset class for getting image samples"},{"metadata":{"trusted":true},"cell_type":"code","source":"class DataRetriever(torch_data.Dataset):\n    def __init__(\n        self, \n        paths, \n        categories=None,\n        transforms=None,\n        base_path=PATH_TRAIN\n    ):\n        self.paths = paths\n        self.categories = categories\n        self.transforms = transforms\n        self.base_path = base_path\n          \n    def __len__(self):\n        return len(self.paths)\n    \n    def __getitem__(self, index):\n        img = cv2.imread(os.path.join(self.base_path, self.paths[index]))\n        img = cv2.resize(img, (224, 224))\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        \n        if self.transforms:\n            img = self.transforms(image=img)[\"image\"]\n        \n        if self.categories is None:\n            return img\n        \n        y = self.categories[index] \n        return img, y\n    \n    \ndef get_transforms():\n    return A.Compose(\n        [\n            A.Normalize(\n                mean=[0.485, 0.456, 0.406], \n                std=[0.229, 0.224, 0.225], \n                p=1.0\n            ),\n            ATorch.transforms.ToTensorV2(p=1.0),\n        ], \n        p=1.0\n    )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Let's take for each category (target) all images from the train set and after processing average their vectors\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = df_meta[[\"category\", \"path\"]].sort_values(by=\"category\")\n\ndf_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tmp_path = df_train[\"path\"].tolist()\ntmp_category = df_train[\"category\"].tolist()\n# If FULL_PIPELINE is False we use small subset of data\nif not FULL_PIPELINE:\n    tmp_path = tmp_path[:256 * 8]\n    tmp_category = tmp_category[:256 * 8]\n\ntrain_data_retriever = DataRetriever(\n    tmp_path,\n    tmp_category,\n    transforms=get_transforms(),\n)\n\ntrain_loader = torch_data.DataLoader(\n    train_data_retriever,\n    batch_size=256,\n    shuffle=False,\n    num_workers=8,\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Initialize the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nmodel = MobileNetV2()\nmodel.to(device)\nmodel.eval();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Save output vectors from the model and average by category"},{"metadata":{"trusted":true},"cell_type":"code","source":"category_counts = collections.Counter(df_train[\"category\"].tolist())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_vectors = np.zeros((len(category_counts), 1280))\n\nwith torch.no_grad():\n    for batch in tqdm(train_loader):\n        X, y = batch\n        vectors = model(X.to(device)).mean(axis=(2, 3))\n        \n        _y = y.numpy().tolist()\n        for ind in range(len(_y)):\n            final_vectors[_y[ind]] += vectors[ind].cpu().numpy().copy() / category_counts[_y[ind]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Save and load category vectors (you can pretrain them)"},{"metadata":{"trusted":true},"cell_type":"code","source":"save(\"average_vectors.npy\", final_vectors)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_vectors = load(\"average_vectors.npy\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Get test paths"},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH_TEST = os.path.join(PATH_BASE, \"test/\")\nPATH_TEST_META = os.path.join(PATH_TEST, \"metadata.json\")\n\n\nwith open(PATH_TEST_META) as json_file:\n    metadata = json.load(json_file)\n\n    \nid2path = {\n    img[\"id\"]: img[\"file_name\"] for img in metadata[\"images\"]\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submission = pd.read_csv(\n    \"../input/herbarium-2021-fgvc8/sample_submission.csv\",\n    index_col=0,\n)\n\ndf_submission[\"Id\"] = df_submission.index\ndf_submission[\"Path\"] = df_submission[\"Id\"].map(lambda x: id2path[x])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Create test data loader"},{"metadata":{"trusted":true},"cell_type":"code","source":"tmp_path = df_submission[\"Path\"].tolist()\n# If FULL_PIPELINE is False we use small subset of data\nif not FULL_PIPELINE:\n    tmp_path = tmp_path[:256 * 2]\n\ntest_data_retriever = DataRetriever(\n    tmp_path,\n    transforms=get_transforms(),\n    base_path=PATH_TEST,\n)\n\ntest_loader = torch_data.DataLoader(\n    test_data_retriever,\n    batch_size=256,\n    shuffle=False,\n    num_workers=8,\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Get test output vectors and find the nearest train vector (by euclidean distance) and take its category"},{"metadata":{"trusted":true},"cell_type":"code","source":"res = []\n\nwith torch.no_grad():\n    for ind, X in enumerate(tqdm(test_loader)):\n        vectors = model(X.to(device)).mean(axis=(2, 3))\n        tmp = euclidean_distances(vectors.cpu().numpy(), final_vectors)\n        res.extend(list(tmp.argmin(axis=1)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Save results to submission file"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submission.iloc[:len(res), 0] = res\n\ndf_submission[[\"Predicted\"]].to_csv(\"submission.csv\")\n\npd.read_csv(\"submission.csv\", index_col=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### I prepared processed by the algorithm described above submission file for all data, you can use it for fast submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH_PREPARED_SUBMISSION = \"../input/herbarium-2021-submissions/submission-mobilenetv2-mean.csv\"\n\nprepared_subnission = pd.read_csv(PATH_PREPARED_SUBMISSION, index_col=0)\nprepared_subnission.to_csv(\"prepared_submission.csv\")\n\npd.read_csv(\"prepared_submission.csv\", index_col=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Work In Progress..."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}