{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Exploratory Data Analysis"},{"metadata":{},"cell_type":"markdown","source":"![Google Landmark Dataset](https://1.bp.blogspot.com/-EQNRBJuBXJo/XMthch8bdWI/AAAAAAAAEG0/oHVw1fxfiXoMfsSn_rNQPR-fyRqM_N4CwCEwYBhgL/s1600/image1.png)\n\n\nThe most Important step of any machine learning project is understanding the underlying patterns of our dataset. In this particular case of landmark dataset our data analysis is mainly focused on the class distribution. as stated by the Google's Landmark Dataset [Paper](https://arxiv.org/pdf/2004.01804.pdf) the class distrubution is **\"extremely long-tailed\"**. In order to see the other unique features of the landmark dataset we will take a look at some of the images in the train and test datasets."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport glob\nimport PIL.Image as Image\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_folder_names(mode, dataid= str):\n    return f'../input/landmark-recognition-2020/{mode}/{dataid[0]}/{dataid[1]}/{dataid[2]}/{dataid}.jpg'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_csv = pd.read_csv('../input/landmark-recognition-2020/train.csv')\nprint('Number of landmark images in train set:', len(train_csv))\ntrain_csv.head()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_csv= pd.read_csv('../input/landmark-recognition-2020/sample_submission.csv')\nsubmission_csv.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Tidy Up "},{"metadata":{},"cell_type":"markdown","source":"Clean data results in happy models. The first step in any ML project. Well at least in the case of raw data. In the lion's share of kaggle datasets this part can be overlooked due to the already cleaned and well-organized datasets."},{"metadata":{},"cell_type":"markdown","source":"1. Check for missing values"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Any missing values?\\n', train_csv.isnull().values.any()) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2. Check for Duplicate rows"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Any Duplicates?', train_csv.duplicated().values.any())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Explore landmarks"},{"metadata":{},"cell_type":"markdown","source":"This dataset has a large amount of unique landmarks which we have to predicts. However, the problem does not lie in the total number of labels. The real challenge is the high variance in the label volume.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"landmark_count = train_csv.landmark_id.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize = (20, 7))\nsns.distplot(landmark_count, hist = False);\nplt.title('Class Distribution', size = 20);\nplt.xlabel('number of images', size = 15);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"limits = [None, (0,200), (0,100)]\nfig = plt.figure(figsize = (20, 7))\nfor i, lim in enumerate(limits):\n    plt.subplot(len(limits),1,i+1)\n    sns.boxplot(landmark_count)\n    plt.xlim(lim)\n    plt.title(lim)\nplt.tight_layout()\n\nprint((landmark_count>200).sum())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"While one class contains **6000** images, ***75%*** contain less then **25**. Only **483** of the classes contain more then **200** images. "},{"metadata":{},"cell_type":"markdown","source":"# Sightseeing"},{"metadata":{},"cell_type":"markdown","source":"Now let's take a look at some of the landmarks. First we will look at the classes with the greatest image count, and then we will go over some of the data on the other side.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"landmark_count_id = list(landmark_count.index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_images(data, landmarkid, num):\n    sub = data.id[data.landmark_id == landmarkid] \n    fig = plt.figure(figsize= (10, 10))\n    fig.suptitle(f\"landmark ID  {landmarkid}\")\n    for i in range(num):\n        if num > 3:\n            plt.subplot(num ** (1/2), num ** (1/2), i+1)\n        else:\n            plt.subplot(1, num, i+1)\n        img = Image.open(get_folder_names('train' ,list(sub)[i]))\n        plt.imshow(img)\n        plt.axis('off')\n    \n\nget_images(train_csv, landmark_count_id[0], 9)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_images(train_csv, landmark_count_id[1], 9)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_images(train_csv, landmark_count_id[8], 9)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The pictures suggest that the dataset suffers from **Intra-Class Variation**. In other words, there is a high variation in the pictures of each class. For instance, in the 126637 landmark pictures which apparently represents a coast, the images show different parts of the landmark and from different angles. This means that our predictive model should be highly robust. "},{"metadata":{"trusted":true},"cell_type":"code","source":"get_images(train_csv, landmark_count_id[-1], 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_images(train_csv, landmark_count_id[-2], 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_images(train_csv, landmark_count_id[-3], 2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualize Testset Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"a = glob.glob('../input/landmark-recognition-2020/test/*/*/*/*.jpg')\nnum = 10\nfig = plt.figure(figsize = (20, 10))\nfor i in range(num):\n    plt.subplot(1, num, i+1)\n    plt.axis('off')\n    randint = np.random.randint(0, len(a))\n    img = Image.open(a[randint])\n    plt.imshow(img)\n    \n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, the last feature of Google's landmark dataset lies in its testset. Most of the pictures in the testset do not belong to any landmark. Therefore, we need to leave a white space for some of the testset pictures."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}