{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Landmark Retrieval\n![The Taj Mahal, one of the most recognizable landmarks in the world](https://images.unsplash.com/photo-1564507592333-c60657eea523?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=1502&q=80)\nA picture is worth a thousand words - or so the saying goes. Surely we've all been there, reminiscing about a past trip by scrolling through our photos, stopping every few pics to think to yourself *why in the world did I take **this** photo?*. Perhaps at the time you could readily identify each historic landmark you snapped a pic of, but now that information has been lost to the sands of time and you're left scratching your head at your old photos.\n\nGoogle's Landmark Recognition Competition is looking to harness the power of Computer Vision to better identify these landmarks in photos. There are certainly challenges that must be overcome to build a superior model, such as: coping with CPU/GPU limits, dealing with a lopsided dataset, and constructing the appropriate model architecture. To help in the brainstorming process for how to conquer these challenges, let's do a deep dive into the data itself to see if there are any insights to be gained.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 1. Data Import + Inspection","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport plotly as py\nimport plotly_express as px\nimport plotly.graph_objects as go\nfrom matplotlib import pyplot as plt\nimport folium\nfrom folium import plugins\nfrom plotly.offline import init_notebook_mode, iplot\nimport os\ninit_notebook_mode()\n\nprint(\"First 5 entries in train.csv: \")\ndf_train = pd.read_csv('/kaggle/input/landmark-recognition-2020/train.csv')\nprint(df_train.head())\nprint('\\n')\n\nprint(\"Number of photos in train: \")\nprint(df_train.shape[0])\nprint('\\n')\n\nprint(\"Number of unique landmarks: \")\nprint(df_train.landmark_id.nunique())\nprint('\\n')\n\nprint(\"Number of photos in test: \")\ntest_count = sum(len(files) for _, _, files in os.walk(r'/kaggle/input/landmark-recognition-2020/test'))\nprint(test_count)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So the test set is roughly 0.65% the size of the training set. Let's take a look at how the landmarks are distributed in train.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_landmark = pd.DataFrame(df_train.landmark_id.value_counts()).reset_index()\ndf_landmark.rename(columns = {'index':'landmark_id', 'landmark_id':'num_photos'}, inplace=True)\ndf_landmark['landmark_id'] = df_landmark['landmark_id'].astype(str)\n\nfig = px.bar(df_landmark[0:10], x = 'landmark_id', y = 'num_photos', hover_name = 'landmark_id', color = 'num_photos', title = 'Top 10 most frequent landmarks')\nfig.update_layout(xaxis_type='category')\nfig.show()\n\n\nfig = px.bar(df_landmark[0:100], x = 'landmark_id', y = 'num_photos', hover_name = 'landmark_id', color = 'num_photos', title = 'Top 100 most frequent landmarks')\nfig.update_layout(xaxis_type='category')\nfig.show()\n\nfig = px.violin(df_landmark, x = 'num_photos', title='Violin plot for landmark frequency')\nfig.show()\n\nfig = px.histogram(df_landmark, x = 'num_photos', title='Histogram for landmark frequency (truncated)')\nfig.update_layout(yaxis_title='number of landmarks', xaxis=dict(range=[0,100]))\nfig.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that we've got a visual sense for how frequently the landmarks appear in the training set, lets look at some statistics.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_landmark.describe(percentiles = [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.95,0.99, 0.999])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"99% of all landmakrs appear in 156 or fewer training photos. Additionally, the top 1% of landmarks by photo frequency account for roughly 14% of all of the training photos. Let's look at the actual images associated with the most popular landmarks in the training set.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Landmark Photo Visualizatio","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from PIL import Image \n\ndef get_file_path(photo_id, ttype):\n    file_path = '/kaggle/input/landmark-recognition-2020/' + ttype\n    file_path = file_path + '/' + photo_id[0] + '/' + photo_id[1] + '/' + photo_id[2] + '/' + photo_id + '.jpg'\n    return file_path\n\n\ndef get_landmarks(landmark_ids, df_train, ttype, ninstances):\n    df = pd.DataFrame([], columns=['id', 'landmark_id'])\n    if ttype not in ['train', 'test']:\n        print('Please enter a valid ttype: train or test')\n        return df\n    photos_url = \"/kaggle/input/landmark-recognition-2020/\" + ttype\n    for lmark in landmark_ids:\n        df_lmark = df_train[df_train.landmark_id == lmark]\n        if len(df_lmark) < ninstances:\n            print('Too many instances for landmark_id ' + str(lmark) + ' (there are only ' + str(len(df_lmark)) + ')')\n            return df\n        else:\n            df_lmark = df_lmark[0:ninstances]\n            df = pd.concat([df, df_lmark])\n    df['file_path'] = df['id'].apply(lambda x: get_file_path(x, ttype))\n    return df.reset_index()\n    \n    \ndef show_landmark(landmark_file, name):\n    img_array = np.array(Image.open(landmark_file))\n    plt.title(name)\n    plt.imshow(img_array)\n    plt.show()\n    \n\ndef show_grid(landmark_id, rows, cols, df_train, ttype):\n    landmark_ids = [landmark_id]\n    instances = len(df_train[df_train.landmark_id == landmark_id])\n    df_show_landmarks = get_landmarks(landmark_ids, df_train, ttype, rows*cols)\n    fig, axs = plt.subplots(rows, cols, figsize = (rows*4,cols*4))\n    count = 0\n    for i in range(0,rows):\n        for j in range(0,cols):\n            landmark_file = df_show_landmarks.loc[count, 'file_path']\n            img_array = np.array(Image.open(landmark_file))\n            axs[i,j].imshow(img_array)\n            count += 1\n    plt.suptitle('Instances of landmark_id: '+ str(landmark_id) + ', which has ' + str(instances) + ' instances in the data.')\n    plt.show()\n    ","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"\nlandmark_ids = list(df_landmark[0:10]['landmark_id'].astype(int))\nfor landmark_id in landmark_ids:\n    show_grid(landmark_id, 5, 5, df_train, 'train')\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Right away, we see there are some interesting features of the training data:\n\n1) The most common landmark looks to be images with a specific label style at the bottom, since the images themselves are not similar.\n\n2) Some landmarks refer to a very specific place or object, but some landmarks appear to be more general (ie Niagara Falls vs. Waterfall in general)\n\nMaybe the landmarks that are over-represented are more susceptible to the aforementioned issues. Let's check out some landmarks that have less associated photos.\n","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"\nlandmark_ids = list(df_landmark[15452:15462]['landmark_id'].astype(int))\nfor landmark_id in landmark_ids:\n    show_grid(landmark_id, 5, 5, df_train, 'train')\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The specificity of the landmark has definitely increased. Finally, let's check out the landmarks that are among the least represented.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"\nlandmark_ids = list(df_landmark[64016:64026]['landmark_id'].astype(int))\nfor landmark_id in landmark_ids:\n    show_grid(landmark_id, 2, 2, df_train, 'train')\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The specificity has increased again.\n\nNext question: what are the dimensions of the images? I haven't found an efficient way to do it for all of the images, so for now we'll just be looking at a subset (say, 100,000 images).","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from tqdm._tqdm_notebook import tqdm_notebook\nimport os\nimport struct\nimport io\nimport warnings\nwarnings.filterwarnings('ignore')\n\ndef get_image_size(file_path):\n    \"\"\"\n    :Name:        get_image_size\n    :Purpose:     extract image dimensions given a file path\n    :Author:      Paulo Scardine (based on code from Emmanuel VAÏSSE)\n    :Created:     26/09/2013\n    :Copyright:   (c) Paulo Scardine 2013\n    :Licence:     MIT\n    \n    \"\"\"\n    size = os.path.getsize(file_path)\n    with io.open(file_path, \"rb\") as input:\n        height = -1\n        width = -1\n        data = input.read(26)\n        input.seek(0)\n        input.read(2)\n        b = input.read(1)\n        while (b and ord(b) != 0xDA):\n            while (ord(b) != 0xFF):\n                b = input.read(1)\n            while (ord(b) == 0xFF):\n                b = input.read(1)\n            if (ord(b) >= 0xC0 and ord(b) <= 0xC3):\n                input.read(3)\n                h, w = struct.unpack(\">HH\", input.read(4))\n                break\n            else:\n                input.read(int(struct.unpack(\">H\", input.read(2))[0]) - 2)\n            b = input.read(1)\n        width = int(w)\n        height = int(h)\n        return [width, height]\n    \n\ndef get_width(dim):\n    return dim[0]\n\n\ndef get_height(dim):\n    return dim[1]\n\nNUM_SUBSET = 100000\n\ndf_train_subset = df_train[0:NUM_SUBSET]\ndf_train_subset['file_path'] = df_train_subset['id'].apply(lambda x: get_file_path(x, 'train'))\ndf_train_subset['dimension'] = df_train_subset['file_path'].apply(lambda x: get_image_size(x))\ndf_train_subset['width'] = df_train_subset['dimension'].apply(lambda x: get_width(x))\ndf_train_subset['height'] = df_train_subset['dimension'].apply(lambda x: get_height(x))\ndf_train_subset['total_pixels'] = df_train_subset['width'] * df_train_subset['height']\ndf_train_subset['ratio'] = df_train_subset['width'] / df_train_subset['height']\ndf_train_subset['ratio'] = df_train_subset['width'] / df_train_subset['height']","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = px.scatter(df_train_subset, x = 'width', y = 'height', hover_name='dimension', color='total_pixels', title='Image dimension (first 100,000 images)')\nfig.show()\n\nfig = px.histogram(df_train_subset, x = 'total_pixels', title='Total Pixels per Image (first 100,000 images)')\nfig.show()\n\nfig = px.scatter(df_train_subset, x = 'width', y = 'height', hover_name='dimension', color='ratio', range_color=(0.5,1.5), title='Image dimension (first 100,000 images)')\nfig.show()\n\nfig = px.histogram(df_train_subset, x = 'ratio', title='Image Ratio (first 100,000 images)')\nfig.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}