{"cells":[{"metadata":{},"cell_type":"markdown","source":"# GLR for beginners: fetch the data, basic EDA\n\nI am making this notebook for anyone who's starting right now with this competition and realized that it's much more complicated than the example scenarios of minicourses here in Kaggle, coursera and so on. This is in part to try to make things clear for myself, but I hope it can also help others.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I will be handling the imports as they are needed, so that if you have problems with just a subset of what I'm doing you can get to that cell and know which imports you need. Here at the top I will just import some things that are used widely throughout the notebook.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot  as plt # data visualization\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First thing, let's set some paths.","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import os\n\n# Dataset parameters:\nINPUT_DIR = os.path.join('..', 'input')\n\nDATASET_DIR = os.path.join(INPUT_DIR, 'landmark-recognition-2020')\nTEST_IMAGE_DIR = os.path.join(DATASET_DIR, 'test')\nTRAIN_IMAGE_DIR = os.path.join(DATASET_DIR, 'train')\nTRAIN_LABELMAP_PATH = os.path.join(DATASET_DIR, 'train.csv')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's fetch the data:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(f'{DATASET_DIR}/train.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And let's have a first look at it:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Shape of train_data :\", train.shape)\nprint(\"Number of unique landmarks :\", train[\"landmark_id\"].nunique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have 1580470 entries, corresponding to 81313 unique landmarks.\n\nLet's look at the first few rows of what we imported:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If we visualize the first few rowns of train we can see that the dataset is composed by the id of the figure followed by the id of the landmark. We need to use this figure ID to load the figures.\n\n# Image visualization\n\nLet's look at how the indexing of images works.\n\nOne example of an image id is:\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"idx = train.id[1]\nidx","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If we look at the folder structure (on the right-hand side column) we can quickly notice that each image is nested three times based on the first three digits of the id. \n\nSo, in our case, we want to open:\n\n> /input/landmark-recognition-2020/train/9/2/b/92b6290d571448f6.jpg\n\nGiven this structure, it's useful to make a quick helper function:\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_image_full_path(idx):\n    return os.path.join(TRAIN_IMAGE_DIR,  f'{idx[0]}/{idx[1]}/{idx[2]}/{idx}.jpg')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from PIL import Image, ImageDraw\n\nimage = Image.open(get_image_full_path(idx))\nplt.imshow(image) \nimage.close()       \nplt.axis(\"off\")\n\nplt.show() \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also look at all the images with a certain Landmark Id:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"example = train[train[\"landmark_id\"]==1]\nfor idx in example[\"id\"]:\n    image = Image.open(get_image_full_path(idx))\n    plt.imshow(image) \n    image.close()       \n    plt.axis(\"off\")\n    plt.show() \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image sizes\n\nSince we will need to feed our classifier images that are all of the same size, it is reasonable to check of which sizes the images of the training set currently are. To do so, we take ispiration from [EDA + Data Augmentation for Beginners](https://www.kaggle.com/azaemon/eda-data-augmentation-for-beginners) and use the package basic_image_eda, applying our exploratory analysis only to one of the subfolders to keep the computation time short enough.\n\n***Important note: In the end, the next two code blocks were taken from [EDA + Data Augmentation for Beginners](https://www.kaggle.com/azaemon/eda-data-augmentation-for-beginners) as they were (this might change in the future). I am still including them for completeness, since I think looking at this aspect of the data is important... but if your upvote my notebook, please consider upvoting also the one I took this bit of code from.***","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install basic_image_eda\nfrom basic_image_eda import BasicImageEDA","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_dir = \"../input/landmark-recognition-2020/train/0\"\nextensions = ['png', 'jpg', 'jpeg']\nthreads = 0\ndimension_plot = True\nchannel_hist = True\nnonzero = False\nhw_division_factor = 1.0\n\nBasicImageEDA.explore(data_dir, extensions, threads, dimension_plot, channel_hist, nonzero, hw_division_factor)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's notice that\n\n> min height                               |  49\n> max height                               |  800\n> \n> min width                                |  120\n> max width                                |  800\n\nthe function reccomends:\n\n> recommended input size(by mean)          |  [608 736] \n\nin the height/width scatterplot we can notice that there are some popular dimensions for either height and width (the orizontal and vertical \"lines\") and some popular form factors (the tilted lines)\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Landmark distribution\n\nNot all landmarks are created equal, and in this dataset we have a huge variety of how much they are represented: some have are in thousands of images, other in as few as just two. Let's first look at the distribution, we might want to exclude those that appear very rarely from out classification.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns \n\nad = sns.distplot(train['landmark_id'].value_counts()[-75000:])\nad.set(xlabel='Landmark Counts', ylabel='Probability Density', title='Distribution of less common landmarks')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}