{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Exploratory Analysis\n\n\nThis notebook is the starting point of a data science project about image recognition. I am going to perform an exploratory analysis (**EDA**) on **Google Landmark Recognition 2020** dataset. My main objective is giving as detailed explanations as possible about the characteristics of the data. So that the conclusions extracted from this notebook can help others and also me for further researches and the implementation of instance-level recognition system."},{"metadata":{},"cell_type":"markdown","source":"# Index of contents\n1. **Importing libraries**\n2. **Loading the dataset**\n3. **Data Visualization**\n4. **Image Visualization**\n5. **Conclusions**\n6. **References**"},{"metadata":{},"cell_type":"markdown","source":"# Importing libraries\nThe modules I am using are the most basic and common ones for this kind of work. Just trying to make things simple. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nimport cv2 \nimport seaborn as sns\nimport matplotlib.pyplot as plt \nimport matplotlib.image as mpimg\n\n#Seed for making reproducible experiments\nseed = 256","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading data\nThe `csv` file of the training set would be enough for now."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_data = pd.read_csv(\"../input/landmark-recognition-2020/train.csv\")\ntrain_data.sample(5, random_state=seed) #Checking if it is loaded correctly","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The file contains two columns, the **id** of the image that comes from the folders names as it is explained on the competition basis and the **landmark_id** which is the unique label of each image."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Training data size\", train_data.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are over one million and half images what is a very huge amount of data to deal with. "},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of unique labels\", train_data[\"landmark_id\"].nunique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are nearly one hundred thousand different types of landmarks what is very far from \"classical\" multi-class problems in which the range of classes is not so extensive. "},{"metadata":{},"cell_type":"markdown","source":"# Data visualizations\nOnce the data is loaded, let's see the data with different graphic visualizations to understand it and also think about how can be processed later on."},{"metadata":{},"cell_type":"markdown","source":"## Number of images per labels"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Calculating the n_images per landmark\ntemp =train_data.groupby([\"landmark_id\"]).size().reset_index(name='n_images')\ntemp = temp.sort_values(by='n_images', ascending=False)\ntemp.sample(5,random_state=seed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,6))\nscatter = sns.scatterplot(x='landmark_id', y='n_images', data=temp, s=20)\nscatter.set(yscale=\"log\")\nplt.ylabel(\"Number of images\")\nplt.xlabel(\"Landmark_id\")\nplt.title(\"Number of images for each landmark_id\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems that the most part of the classes has under one hundred images. The tangle of points is concentrated on the middle range, between 50 and 100 images. Below this range, apparently there are some kind of stairs steps where the minoritary classes are represented. Also above the commented range there are some distinguished points which corresponds to the most frequent landmarks of the database."},{"metadata":{},"cell_type":"markdown","source":"## Most frequent landmarks (Top 10)\n\nBar plot of the most common landmarks of the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"temp.head(10).style.background_gradient(subset='n_images') ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (12,5))\nplt.title(\"Top 10 most frequent landmarkrs\")\nsns.barplot(x='landmark_id', y='n_images', data=temp.head(10), palette=\"mako\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is one landmark that clearly stand out from the rest with a very notable difference on the number of instances. Also it can be said that most of the top-10 landmark has an average of one thousand ocurrences each one.  "},{"metadata":{},"cell_type":"markdown","source":"## Least frequent landmarks (Bottom 10)\nBar plot of the least common landmarks of the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Calculating the bottom 10 landmarks\ntemp.tail(10).style.background_gradient(subset='n_images') ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (12,5))\nplt.title(\"Bottom 10 least frequent landmarkrs\")\nsns.barplot(x='landmark_id', y='n_images', data=temp.tail(10), palette='mako')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is easy to think about the enormous difference on instances in respect of the previous ones. This fact could lead us to decision as data augmentation or cleaning the least frequent ones from the data set."},{"metadata":{},"cell_type":"markdown","source":"## Landmark distribution"},{"metadata":{},"cell_type":"markdown","source":"With the previous plots it is clear that there is a high imbalance on the distribution of labels. For that reason, let's see by parts how they are distributed."},{"metadata":{"trusted":true},"cell_type":"code","source":"temp.describe(percentiles=[0.25,0.5,0.75,0.8,0.9,0.99])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The `99%` of the labels have under 156 images what is far away from the most frequent ones as we have seen before."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (10,6))\nplt.title(\"Distribution of the Top 50 landmarks\")\nplt.ylabel('Density')\nsns.distplot(temp['n_images'][:50], color='tomato')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, it is more evident than the most frequent labels oscilates among five hundred and one thousand images with the clear outlier landmark commented previously."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (10,6))\nplt.title(\"Distribution of rest of landmarks\")\nplt.ylabel('Density')\nsns.distplot(temp['n_images'][51:], color='blue')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The rest of the landmarks has less than one hundred images each one."},{"metadata":{},"cell_type":"markdown","source":"## Number of landmarks within an number image range"},{"metadata":{},"cell_type":"markdown","source":"Once seen the imbalance distribution, we can group landmarks based on their number of images. With that we can infer how to make partitions of the database."},{"metadata":{"trusted":true},"cell_type":"code","source":"range_labels = ['2-10', '10-50', '50-100', '100-250', '250-500', '>500']\nrange_counts = [temp[temp['n_images'] <= 10]['landmark_id'].count(), temp[(temp['n_images'] > 10) & (temp['n_images'] <= 50)]['landmark_id'].count(),\n                temp[(temp['n_images'] > 50) & (temp['n_images'] <= 100)]['landmark_id'].count(),temp[(temp['n_images'] > 100) & (temp['n_images'] <= 250)]['landmark_id'].count(),\n                temp[(temp['n_images'] > 250) & (temp['n_images'] <= 500)]['landmark_id'].count(),temp[temp['n_images'] > 500]['landmark_id'].count() ]\n\nplt.figure(figsize=(12,5))\nplt.title(\"Number of landmarks within an image range\")\nplt.ylabel(\"Image range count\")\nplt.xlabel(\"Number of landmarks\")\nsns.barplot(range_counts, range_labels, palette='mako')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"On the bar plot above we can see that almost all landmarks has under 50 images what is seen at first sight at the beggining. Just few points had above five hundred images."},{"metadata":{"trusted":true},"cell_type":"code","source":"total = temp['n_images'].sum()\nrange_percents = [temp[temp['n_images'] <= 10]['n_images'].sum()/total, temp[(temp['n_images'] > 10) & (temp['n_images'] <= 50)]['n_images'].sum()/total,\n                temp[(temp['n_images'] > 50) & (temp['n_images'] <= 100)]['n_images'].sum()/total,temp[(temp['n_images'] > 100) & (temp['n_images'] <= 250)]['n_images'].sum()/total,\n                temp[(temp['n_images'] > 250) & (temp['n_images'] <= 500)]['n_images'].sum()/total,temp[temp['n_images'] > 500]['n_images'].sum()/total ]\n\nplt.figure(figsize=(12,5))\nplt.title(\"Percent of landmarks photos by range\")\nplt.ylabel(\"Image range count\")\nplt.xlabel(\"Percent\")\nsns.barplot(range_percents, range_labels, palette='rocket')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This bar plot is just a confirmation of the percentiles described before. At first sight, it is noticeable that `90%` of the data has less than one hundred photos. Moreover, the range from **10-50 photos per landmark** is the predominant in the database being nearly the half of it. Also, we must be careful because there is a high percent of landmark images on the ones that have just a couple of photos per class. "},{"metadata":{},"cell_type":"markdown","source":"# Image visualization\n\nNow it is turn to see some random images from the dataset. So let's start by loading the corresponding paths."},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_DIR= \"../input/landmark-recognition-2020/train\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Some Landmark photos"},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_images(images):\n    f = plt.figure(figsize=(20,20))\n    \n    for i, id in enumerate(images):\n        image_path = os.path.join(TRAIN_DIR, f'{id[0][0]}/{id[0][1]}/{id[0][2]}/{id[0]}.jpg')\n        image = mpimg.imread(image_path)\n        f.add_subplot(1,5,i+1)\n        plt.imshow(image)\n        plt.title(f'Landmark_id: {id[1]} \\nID: {id[0]}')\n    \n    plt.show()\n\nlandmark_id = temp.head(1)['landmark_id'].values[0]\ntop = train_data[train_data['landmark_id'] == landmark_id].sample(5,random_state=seed).values  #the seed affects to the plotted images!!\ndisplay_images(top)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is very interesting the fact that the images of the most common landmark are so diferrent one from other. In the sample, there are no common elements in none of the photos, maybe just the dessert, but that is all and it is barely distinguihed in two of them. This fact shows us the intra-class imbalance as other challenging characteristic of this dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"landmark_id = temp.tail(1)['landmark_id'].values[0]\nbottom = train_data[train_data['landmark_id'] == landmark_id].sample(2,random_state=seed).values\ndisplay_images(bottom)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is one of the least common landmarks along the dataset. In contrast of the previous one, here it is more likely to identify by first sight that there are photos of the same beach. However, it is not so easy because the persepctive, light and place which were taken are very different."},{"metadata":{"trusted":true},"cell_type":"code","source":"landmark_id = temp.sample(1,random_state=seed)['landmark_id'].values[0]\nrandom = train_data[train_data['landmark_id'] == landmark_id].sample(5,random_state=seed).values\ndisplay_images(random)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, the display of one random landmark. In this case a church, identifying the common points is more clear, but again the perspectives are several as well as the light conditions. Also, one of the photos is some kind of painting that is supposed to belong to the inside part of the church. Meanwhile there is another photo that is taken from the sky and it is very difficult to recognize where is the church on it. These are the aspects with which the models will struggle on their learning. Another thing that I have forgotten to mention before is the variance on the image sizes. So, let's see some statistics about the photos characteristics."},{"metadata":{},"cell_type":"markdown","source":"## Photo characteristics"},{"metadata":{},"cell_type":"markdown","source":"As there are one million and half photos, it is not very efficient to analyze all of them, so I am going to use just a significant bunch of them."},{"metadata":{},"cell_type":"markdown","source":"## Image sizes"},{"metadata":{"trusted":true},"cell_type":"code","source":"def image_data(images):\n    widths = []\n    heights = []\n    pixels = []\n    for i, id in enumerate(images):\n        image_path = os.path.join(TRAIN_DIR, f'{id[0]}/{id[1]}/{id[2]}/{id}.jpg')\n        img = cv2.imread(image_path)\n        h, w = img.shape[0], img.shape[1]  #Don't need the color channel now\n        p = img.size\n        widths.append(w)\n        heights.append(h)\n        pixels.append(p)\n        \n    data = {\"height\": heights, \"width\": widths,\"pixels\": pixels }\n    return pd.DataFrame(data)\n\n\nbunch_photos = train_data.sample(10000, random_state=seed)\n\nphoto_data = image_data(bunch_photos[\"id\"].values)\nphoto_data.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(7,7))\nsns.scatterplot(x='width', y='height', data=photo_data, color='orange')\nplt.title(\"Sizes of 10000 random images\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are a big variaty of image sizes, but it seems that the predominant height and width are 800 pixels. This is something important to consider for the pre-processing of the image data."},{"metadata":{},"cell_type":"markdown","source":"## Image aspect ratios"},{"metadata":{"trusted":true},"cell_type":"code","source":"photo_data['a_ratio'] = photo_data['width'] / photo_data['height']\n\nplt.figure(figsize=(10,6))\nsns.distplot(photo_data['a_ratio'], color='deepskyblue')\nplt.title(\"Aspect ratios of 10000 random images\")\nplt.ylabel(\"Density\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Linked to the width and height of the photos we have the aspect ratio. In this sample the most common one is between one and two what means portrait photos. However, there is an important percentege which are vertical due to their ratio is under one (width smaller than height) "},{"metadata":{},"cell_type":"markdown","source":"## Total pixel per image"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,6))\nsns.distplot(photo_data['pixels'], color='plum')\nplt.title(\"Total pixels distribution of 10000 random images\")\nplt.ylabel(\"Density\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The samples of photos oscilates between one and one and half megapixels, so they are not so heavy sizes as the ones we are commonly doing on our daily life."},{"metadata":{},"cell_type":"markdown","source":"# Conclusions\nIn conclusion there are some facts that we must pay special attention in the creation of a model:\n* Huge unbalanced distribution of labels\n* High variance inside classes\n* Very different image sizes\n* Perspective, light conditions, common points and other image features"},{"metadata":{},"cell_type":"markdown","source":"# References\n* [Visualizing Landmarks (+more EDA)](https://www.kaggle.com/jeffreybraun/visualizing-landmarks-more-eda)\n* [[EDA] : Google Landmark Recognition 2020](https://www.kaggle.com/namanj27/eda-google-landmark-recognition-2020)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}