{"cells":[{"metadata":{},"cell_type":"markdown","source":"![image](https://github.com/Lexie88rus/iWildCam/raw/master/assets/cover_image.jpg)"},{"metadata":{},"cell_type":"markdown","source":"*Photo by [Sarah Bliss](https://unsplash.com/@squeakypeachphotos?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/s/photos/wild-animals?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)*"},{"metadata":{},"cell_type":"markdown","source":"# iWildCam EDA"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport os\nimport json\nfrom datetime import datetime\n\nfrom PIL import Image\n\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Introduction\nCamera Traps (or Wild Cams) enable the automatic collection of large quantities of image data. Biologists all over the world use camera traps to monitor biodiversity and population density of animal species. However, as we try to expand the scope of the species classification models we are faced with an interesting problem: how do we train models that perform well on new (unseen during training) camera trap locations? Can we leverage data from other modalities, such as citizen science data and remote sensing data?\n\nThe challenge of [this competition](https://www.kaggle.com/c/iwildcam-2020-fgvc7/overview) is to classify species in the cameras correctly given the test and training data from different cameras spread across the globe. The set of species seen in each camera overlap, but are not identical. \n\nIn this notebook I'd like to give some overview of the data."},{"metadata":{},"cell_type":"markdown","source":"## Load Data"},{"metadata":{},"cell_type":"markdown","source":"Get the list of files in the dataset:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"! ls ../input/iwildcam-2020-fgvc7","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load the train annotations data:"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# setup the directories\nDATA_DIR = '../input/iwildcam-2020-fgvc7/'\nTRAIN_DIR = DATA_DIR + 'train/'\nTEST_DIR = DATA_DIR + 'test/'\n\n# load the megadetector results\nmegadetector_results = json.load(open(DATA_DIR + 'iwildcam2020_megadetector_results.json'))\n#megadetector_results['images'][:2]\n\n# load train images annotations\ntrain_info = json.load(open(DATA_DIR + 'iwildcam2020_train_annotations.json'))\n# split json into several pandas dataframes\ntrain_annotations = pd.DataFrame(train_info['annotations'])\ntrain_images = pd.DataFrame(train_info['images'])\ntrain_categories = pd.DataFrame(train_info['categories'])\n\n# load test images info\ntest_info = json.load(open(DATA_DIR + 'iwildcam2020_test_information.json'))\n# split json into several pandas dataframes\ntest_images = pd.DataFrame(test_info['images'])\ntest_categories = pd.DataFrame(test_info['categories'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Look at the dataframes containing the train set annotations:"},{"metadata":{},"cell_type":"markdown","source":"1. The list of images and corresponding category ids:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_annotations.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2. Train images sequence information (locations of cameras, sequences of images from same locations, image timestamps):"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_images.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"3. The description of categories:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_categories.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Set Statistics"},{"metadata":{},"cell_type":"markdown","source":"`1` Get the total number of images in the train and test sets:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Number of images in the train set is {}'.format(train_annotations.image_id.nunique()))\nprint('Number of images in the test set is {}'.format(test_images.file_name.nunique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visualize train and test sets:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.pie([train_annotations.image_id.nunique(), test_images.file_name.nunique()], labels=['Train', 'Test'], autopct='%1.1f%%', \n           startangle=90, colors=['#fa4252', '#91bd3a'])\nplt.axis('equal')\nplt.title('Number of images in train and test sets', fontsize=14)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`2` Let's check the number of animals found per image:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# get the number of counts per image\n# sort values in descending order\ncounts_per_image = train_annotations.groupby(by=['image_id']).sum().reset_index()[['image_id', 'count']].sort_values(by=['count'], ascending=False)\n# output top-5\ncounts_per_image.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# output tail-5\ncounts_per_image.tail(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As wee see, some of the images have 80 counts of animals on them!\nSome of the images have negative counts, which probably corresponds to no animals there.\n\nThis is a distribution of the count of animals found on the images:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.hist(counts_per_image['count'].values, bins=14, color='#91bd3a')\nplt.title('The distribution of the count of animals per image', fontsize=14)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that:\n* most of the images are empty;\n* having more than 20 animals on one image is very rare."},{"metadata":{},"cell_type":"markdown","source":"`3` Let's look at the number of different categories of animals per image:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# get the number of counts per image\n# sort values in descending order\ncategory_counts_per_image = train_annotations.groupby(by=['image_id', 'category_id']).sum().reset_index()[['image_id', 'category_id', 'count']]\\\n.sort_values(by=['count'], ascending=False)\n# merge with category names\ncategory_counts_per_image = category_counts_per_image.merge(train_categories[['id', 'name']].rename(columns={'id':'category_id'}), on=['category_id'])\n# output top-5\ncategory_counts_per_image.head(5)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.hist(category_counts_per_image['count'].values, bins=14, color='#91bd3a')\nplt.title('The distribution of the animal category counts per image', fontsize=14)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This distribution is quite similar to the number of the animals per image."},{"metadata":{},"cell_type":"markdown","source":"`4` Now let's look at the number of images for each category:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# sort the categories in descending order\nnum_categories = train_categories.sort_values(by=['count'], ascending=False)\n# list the top-5\nnum_categories.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_categories.tail(5)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axs = plt.subplots(1,2, figsize=(20,7))\nwidth = 0.8\nn_categories = 20\n# plot the top-n\naxs[0].bar(x=range(n_categories), height=num_categories['count'].values[:n_categories], width=width, color='#91bd3a')\naxs[0].set_xticks(np.array(range(n_categories)))\naxs[0].set_xticklabels(num_categories['name'].values[:n_categories], rotation=90)\naxs[0].set_title('TOP-{}'.format(n_categories))\n# plot the tail-n\naxs[1].bar(x=range(n_categories), height=num_categories[num_categories['count'] > 0]['count'].values[-n_categories:], width=width, color='#fa4252')\naxs[1].set_xticks(np.array(range(n_categories)))\naxs[1].set_xticklabels(num_categories[num_categories['count'] > 0]['name'].values[-n_categories:], rotation=90)\naxs[1].set_title('LAST-{}'.format(n_categories))\n\nplt.suptitle('The most and the least frequent categories', fontsize=16)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some quick conclusions:\n\n* Most of the images are empty;\n* A lot of images captured humans;\n* Some of the categories were not captured at all;\n* The least frequent categories were captured by cameras just once!"},{"metadata":{},"cell_type":"markdown","source":"`5` Let's explore the locations related to the train set images:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('The number of unique locations is {}'.format(train_images.location.nunique()))\nprint('The average number of images per location is {}'.format(train_images.groupby(by=['location']).id.count().mean()))\nprint('The minimum number of images per location is {}'.format(train_images.groupby(by=['location']).id.count().min()))\nprint('The maximum number of images per location is {}'.format(train_images.groupby(by=['location']).id.count().max()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The distribution of the number of the images per location:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,5))\nplt.hist(train_images.groupby(by=['location']).id.count(), bins=40, color='#91bd3a')\nplt.title('The distribution of the number of the images per location', fontsize=14)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Most of the locations have less than 500 images, but for some of the locations the total number of images exceeds 8000.\n* For some of the locations there is only 1 image captured! Probably, this is why we want to have models, which extend to different locations."},{"metadata":{},"cell_type":"markdown","source":"`6` Analyze the timelines for the captured images:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# convert datetimes to just dates\ndef to_date(datetime_str):\n    \"\"\"Convert datetime string to date.\"\"\"\n    # datetime string example: 2013-08-08 11:45:00.000\n    dt = datetime_str.split(' ')[0]\n    return dt\n    \ntrain_images['date'] = train_images.apply(lambda row: to_date(row.datetime), axis=1)\n# group by date\nimg_per_date = train_images.groupby(by=['date']).id.count()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('The average number of images per day is {}'.format(img_per_date.mean()))\nprint('The maximum number of images per day is {}'.format(img_per_date.max()))\nprint('The minimum number of images per day is {}'.format(img_per_date.min()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# visualize the number of images for each date\n# normalize dates\nmax_date = datetime.strptime(img_per_date.index[-1], '%Y-%m-%d')\nmin_date = datetime.strptime(img_per_date.index[0], '%Y-%m-%d')\n\ndef get_date_norm(date_str):\n    \"\"\"Helper function to get the noemalized date.\"\"\"\n    date = datetime.strptime(date_str, '%Y-%m-%d')\n    return (date - min_date) * 1.0 / (max_date - min_date)\n    \ncopy = img_per_date.reset_index().copy()\ncopy['date_norm'] = copy.apply(lambda row: get_date_norm(row.date), axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# plot the scatter plot\nplt.figure(figsize=(20, 7))\nplt.plot(copy.date_norm, copy.id, color='#91bd3a', linestyle='--')\nplt.scatter(copy.date_norm, copy.id, s=50, color='#91bd3a')\nplt.xticks([copy.date_norm[i] for i in range(0,len(copy),10)], [copy.date[i] for i in range(0,len(copy),10)], rotation=90)\n\nplt.title('Number of images per day', fontsize=14)\nplt.xlabel('date')\nplt.ylabel('number of images')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Most of the data is collected in January - September 2013;\n* More than 4700 images are captured on single day;\n* Just 1 image is capture on some days."},{"metadata":{},"cell_type":"markdown","source":"`7` Analyze the number of frames in sequences:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# group by sequence id\nframes_per_sequence = train_images.groupby(by=['seq_id']).frame_num.max()\n\nprint('The average number of frames is {}'.format(frames_per_sequence.mean()))\nprint('The minimum number of frames is {}'.format(frames_per_sequence.min()))\nprint('The maximum number of frames is {}'.format(frames_per_sequence.max()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The distribution of the number of frames in sequences:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.hist(frames_per_sequence.values, bins=40, color='#91bd3a')\nplt.title('The distribution of the number of frames')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The number of frames in sequences is mostly small. Large sequences are exceptions."},{"metadata":{},"cell_type":"markdown","source":"`8` Explore the dimensions of the images:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('The minimum width of the images is {}'.format(train_images.width.min()))\nprint('The maximum width of the images is {}'.format(train_images.width.max()))\nprint('The minimum height of the images is {}'.format(train_images.height.min()))\nprint('The maximum height of the images is {}'.format(train_images.height.max()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The `-1` are the very strange values for the width and height."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# plot histograms to show the distribution of width and height values\nfig, axs = plt.subplots(1,2, figsize=(15,7))\naxs[0].hist(train_images.width.values, bins=20, color = '#91bd3a')\naxs[0].set_title('Width distribution')\naxs[0].set_xlim(1000, 3000)\n\naxs[1].hist(train_images.width.values, bins=20, color = '#91bd3a')\naxs[1].set_title('Height distribution')\naxs[1].set_xlim(1000, 3000)\n\nplt.suptitle('Image Dimensions')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The dimensions of the images vary greatly."},{"metadata":{},"cell_type":"markdown","source":"## Train Set Images Exploration"},{"metadata":{},"cell_type":"markdown","source":"`1` Visualize random images from the train set and their categories (just one first category):"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def get_first_category(img_id):\n    \"\"\"Find first the image category by id.\"\"\"\n    # get category id\n    category_id = train_annotations[train_annotations.image_id == img_id].category_id.values[0]\n    # get category name\n    category_name = train_categories[train_categories.id == category_id].name.values[0]\n    return category_id, category_name\n\ndef visualize_image_grid(rows, cols):\n    \"\"\"Visualize random grid of images with the first category.\"\"\"\n    filenames = train_images.file_name.unique()\n    \n    np.random.seed(42)\n    img_idx = np.random.randint(len(filenames), size=rows * cols)\n    \n    fig, axs = plt.subplots(rows, cols, figsize=(15,7))\n    \n    for r in range(rows):\n        for c in range(cols):\n            # get the image and image id\n            filename = filenames[img_idx[rows*r + c]]\n            img_id = filename.split('.')[0]\n            # get the category\n            category_id, category = get_first_category(img_id)\n            \n            img = Image.open(TRAIN_DIR + filename)\n            \n            axs[r,c].imshow(img)\n            axs[r,c].axis('off')\n            axs[r,c].set_title('{}:{}'.format(category_id, category))\n            \n    plt.suptitle('Train images', fontsize=16)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_image_grid(3, 3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that there are images of different sizes with different lighting conditions. For some of the images it is hard even for a human to find an animal. "},{"metadata":{},"cell_type":"markdown","source":"`2` Visualize images for a specific category"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def visualize_cetagory(category_id, rows=3, cols=3, seed=42):\n    \"\"\"Function to visualize images of a specific category.\"\"\"\n    # filter by the category_id\n    copy = train_annotations[train_annotations.category_id == category_id]\n    # get the category name\n    category_name = train_categories[train_categories.id == category_id].name.values[0]\n    \n    # get random indices\n    np.random.seed(seed)\n    img_idx = np.random.randint(len(copy), size=rows * cols)\n    \n    # plot images\n    fig, axs = plt.subplots(rows, cols, figsize=(15,7))\n    \n    for r in range(rows):\n        for c in range(cols):\n            # get the image and image id\n            filename = copy.iloc[img_idx[rows*r + c]].image_id + '.jpg'\n            img_id = filename.split('.')[0]\n            \n            img = Image.open(TRAIN_DIR + filename)\n            \n            axs[r,c].imshow(img)\n            axs[r,c].axis('off')\n            axs[r,c].set_title('{}:{}'.format(category_id, category_name))\n            \n    plt.suptitle('Train images for {}:{}'.format(category_id, category_name), fontsize=16)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_cetagory(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# central american agouty\n# https://en.wikipedia.org/wiki/Central_American_agouti\nvisualize_cetagory(3) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# giraffe\n# https://en.wikipedia.org/wiki/Giraffe\nvisualize_cetagory(112) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems that we can mostly see his legs 🦒."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Ocellated turkey\n# https://en.wikipedia.org/wiki/Ocellated_turkey\nvisualize_cetagory(372) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`3` Visualize images for top categories:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def visualize_top_categories(n_categories=5, n_cols=5, seed=42):\n    \"\"\"Function to plot a grid of n_cols images for each of the top n_categories from the train set.\"\"\"\n    np.random.seed(seed)\n    \n    # get ids for the top n_categories\n    # excluding empty and human\n    top_categories = num_categories['id'].values[2:n_categories+2]\n    \n    # setup the image grid\n    fig, axs = plt.subplots(n_categories, n_cols, figsize=(18,10))\n    \n    for row in range(0, n_categories):\n        # get the category \n        category_id = top_categories[row]\n        \n        # get the category name\n        category_name = train_categories[train_categories.id == category_id].name.values[0]\n        \n        # filter the images by category\n        copy = train_annotations[train_annotations.category_id == category_id]\n        \n        # get random indices\n        img_idx = np.random.randint(len(copy), size=n_cols)\n        \n        for col in range(0, n_cols):\n            # get the image and image id\n            filename = copy.iloc[img_idx[col]].image_id + '.jpg'\n            img_id = filename.split('.')[0]\n            \n            img = Image.open(TRAIN_DIR + filename)\n            \n            axs[row,col].imshow(img)\n            axs[row,col].axis('off')\n            axs[row,col].set_title('{}:{}'.format(category_id, category_name))\n            \n    plt.suptitle('Train images for top-{} categories'.format(n_categories), fontsize=16)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_top_categories(n_categories=5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## References"},{"metadata":{},"cell_type":"markdown","source":"Some papers and github repos you may find helpful for the competition:\n1. [Efficient Method for Categorize Animals in the Wild](https://arxiv.org/pdf/1907.13037v1.pdf):  We transfer the state-of-the-art ImagaNet pretrained models to the problem. To improve the generalization and robustness of the model, we utilize efficient image augmentation and regularization strategies, like cutout, mixup and label-smoothing. Finally, we use ensemble learning to increase the performance of the model. [Github repo](https://github.com/Walleclipse/iWildCam_2019_FGVC6)\n2. [Animal Classification System: A Block Based Approach](https://arxiv.org/pdf/1609.01829v1.pdf): In this work, we propose a method for the classification of animal in images. Initially, a graph cut based method is used to perform segmentation in order to eliminate the background from the given image. The segmented animal images are partitioned in to number of blocks and then the color texture moments are extracted from different blocks. Probabilistic neural network and K-nearest neighbors are considered here for classification. To corroborate the efficacy of the proposed method, an experiment was conducted on our own data set of 25 classes of animals, which consisted of 4000 sample images. The experiment was conducted by picking images randomly from the database to study the effect of classification accuracy, and the results show that the K-nearest neighbors classifier achieves good performance."},{"metadata":{},"cell_type":"markdown","source":"## Conclusion\nIn this notebook I observed:\n1. The statistics for the images from the train set:\n    * the number of images;\n    * the number of categories per image;\n    * the number of images per category;\n    * the number of images per day.\n2. The examples of the images from the train set."},{"metadata":{},"cell_type":"markdown","source":"__The notebook is being updated. Your comments are very welcome!__"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}