{"cells":[{"metadata":{},"cell_type":"markdown","source":"# <center> Google Landmark Recognition</center>\n# ![image](https://images.unsplash.com/photo-1526548583898-58820894ac9b?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=500&q=300)\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Hi Everyone this is my first kernel, which gives a brief idea of using PCA and t-SNE to visualize the features extracted from images.\n\n- **Please Upvote if you find this kernel interesting**\n- Suggestions and feedbacks are most welcome","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### About the Competition","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The Third Landmark Recognition competition! This year, we have worked to set this up as a code competition and collected a new set of test images.\n\nHave you ever gone through your vacation photos and asked yourself: What was the name of that place I visited in 5 years ago? or Who created this monument I saw in France? Landmark recognition can help! This technology can predict landmark labels directly from image pixels, to help people better understand and organize their photo collections. This competition challenges Kagglers to build models that recognize the correct landmark (if any) in a dataset of challenging test images.\n\n**EVALUATION**\n\nSubmissions are evaluated using Global Average Precision (GAP) at k, where k=1. This metric is also known as micro Average Precision (μAP) It works as follows:\n\nFor each test image, you will predict one landmark label and a corresponding confidence score. The evaluation treats each prediction as an individual data point in a long list of predictions (sorted in descending order by confidence scores), and computes the Average Precision based on this list.\n\nIf a submission has N predictions (label/confidence pairs) sorted in descending order by their confidence scores, then the Global Average Precision is computed as:\n\n**$$GAP=\\frac{1}{M}\\sum_{i=1}^{N}P(i)rel(i)$$**\n\nwhere:\n\n- N is the total number of predictions returned by the system, across all queries\n- M is the total number of queries with at least one landmark from the training set visible in it (note that some queries may not depict landmarks)\n- P(i) is the precision at rank i\n- rel(i) denotes the relevance of prediciton i: it’s 1 if the i-th prediction is correct, and 0 otherwise\n\n**TIMELINE**\n- September 22, 2020 - Entry deadline. You must accept the competition rules before this date in order to compete.\n\n- September 22, 2020 - Team Merger deadline. This is the last day participants may join or merge teams.\n\n- September 29, 2020 - Final submission deadline.\n\n- October 6, 2020 - Paper Submission deadline (winners only).**\n\n* Winners will be required to provide a detailed description of their method in order to claim their prize (minimum of 2 pages double-column).\n\nAll deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os\nimport cv2\nimport time\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport matplotlib.patheffects as PathEffects\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.decomposition import PCA\nfrom sklearn.manifold import TSNE\n\nfrom tqdm import tqdm_notebook\n\nimport torch,torchvision\nfrom albumentations import *\nfrom albumentations.pytorch import ToTensor\n\nfrom torchvision import models\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nsns.set()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/landmark-recognition-2020/train.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Trian Data**\n- Total Number of Training Data: **1580470**\n- Total Number of Classes in Trian data: **81313**","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df['landmark_id'].value_counts().values.max(),train_df['landmark_id'].value_counts().values.min()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,5))\nsns.distplot(train_df['landmark_id'].value_counts().values,bins=10)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**From the above graph it is clear that train data is unequally distributed across all the classes**\n- Max Number of Images avalible for a class - 6272 (class name: 138982)\n- Min Number of Images available for a class - 2 (class name: 197219)","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"value_counts_df = pd.DataFrame(train_df['landmark_id'].value_counts())\nvalue_counts_df.index.name='Class_name'\nvalue_counts_sorted = value_counts_df.sort_values('Class_name')\nvalue_counts_sorted.reset_index('Class_name',inplace=True)\nplt.figure(figsize=(20,5))\nplt.plot(value_counts_sorted.Class_name.values,value_counts_sorted['landmark_id'].values,linestyle='--',\n         marker='*',color = 'red')\nplt.xlabel('Landmark_id',fontsize=12)\nplt.ylabel('Number of Images',fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = pd.DataFrame(value_counts_df['landmark_id'].value_counts())\nplt.figure(figsize=(20,5))\nplt.subplot(121)\nplt.plot(df['landmark_id'].values,df.index,linestyle='-',\n         marker='.',color = '#52B9FB',label='Complete Data')\nplt.plot(df[df['landmark_id']>1]['landmark_id'].values,df[df['landmark_id']>1].index,linestyle='-',\n         marker='.',color = '#EE0D73',label='class count>1')\nplt.plot(df[df['landmark_id']>25]['landmark_id'].values,df[df['landmark_id']>25].index,linestyle='-',\n         marker='.',color = '#397D09',label='class count>25')\nplt.plot(df[df['landmark_id']>50]['landmark_id'].values,df[df['landmark_id']>50].index,linestyle='-',\n         marker='.',color = '#C58A17',label='class count>50')\nplt.plot(df[df['landmark_id']>100]['landmark_id'].values,df[df['landmark_id']>100].index,linestyle='-',\n         marker='.',color = '#F2625E',label='class count>100')\nplt.ylabel('Images Count',fontsize=12)\nplt.xlabel('Number of Landmark_id(classes)',fontsize=12)\nplt.legend()\n\nplt.subplot(122)\nx_label = ['2-5','5-10','10-50','50-100','100-250','250-500','>500']\ny_label = [df[(df.index <= 5)]['landmark_id'].sum(),df[(df.index > 5) & (df.index<=10)]['landmark_id'].sum(),\n          df[(df.index > 10) & (df.index<=50)]['landmark_id'].sum(),df[(df.index > 50) & (df.index<=100)]['landmark_id'].sum(),\n          df[(df.index > 100) & (df.index<=250)]['landmark_id'].sum(),df[(df.index > 250) & (df.index<=500)]['landmark_id'].sum(),\n          df[(df.index > 500)]['landmark_id'].sum()]\n\nplt.barh(x_label,y_label)\nfor i, v in enumerate(y_label):\n    plt.text(v + 3, i-0.1, str(v),fontsize=8)\n    \nplt.ylabel('Images Count',fontsize=12)\nplt.xlabel('Number of Landmark_id(classes)',fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- **92%** of the existing classes has less than 50 image count","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Images","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"- Since the Dataset is huge images are stored in three subset of folders where the first 3 digits in \"train_df.id\" represents the folders. So we can access the data as shown below","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_path = '../input/landmark-recognition-2020/train/'\n# 5 images from 5 different class\nnp.random.seed(27)\nclasses = np.random.choice(train_df['landmark_id'].unique(),5,replace=False)\nfor cls in classes:\n    image_names = np.random.choice(train_df[train_df['landmark_id'] == cls]['id'],5,replace=False)\n    \n    c=1\n    plt.figure(figsize=(25,5))\n    for image_name in image_names:\n        image_label = cls\n        image = cv2.imread(os.path.join(train_path,image_name[0],image_name[1],image_name[2],image_name+'.jpg'))\n        image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB)\n        plt.subplot(1,5,c)\n        plt.title('Image_name: '+image_name + '\\n\\nLabel: '+str(image_label))\n        plt.axis('off')\n        plt.imshow(image)\n        c+=1\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"**Observation from the above grids of images**\n- **Label 100601**,the statue is constant in all images\n- **Label - 55401**, has different views, I wonder how a model is going to train to detect the structure when a boat is placed infront.\n- **Label 172124**, the features of the image inside the building is completly different from outside.(Images 1,2 and 4 are similar whereas 3 and 5 are completely different)  ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Classes With Images >500 ( 51 Classes)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class Load_Dataset(torch.utils.data.Dataset):\n    def __init__(self, df):\n        self.image_paths = df['id']\n        self.labels = df['landmark_id']\n        self.default_transform = Compose([\n            Normalize((0.485, 0.456, 0.406),\n                                 (0.229, 0.224, 0.225),always_apply=True),\n            Resize(224,224),\n            ToTensor()\n        ])\n        \n    def __len__(self):\n        return self.image_paths.shape[0]\n    \n    def __getitem__(self,i):\n        image_name = self.image_paths[i]\n        img_path = os.path.join('../input/landmark-recognition-2020/train/',\n                                                     image_name[0],image_name[1],image_name[2],image_name+'.jpg')\n        image = cv2.imread(img_path)\n        image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB)\n        image = self.default_transform(image=image)['image']\n        mean = torch.mean(image)\n        std = torch.std(image)\n        label = torch.tensor(self.labels[i])\n        return image,label,mean,std","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# class_500 = value_counts_df[value_counts_df['landmark_id']>500].index.values\n# class_500_df = train_df[train_df['landmark_id'].isin(class_500)]\n# class_500_df.reset_index(inplace=True,drop=True)\n# class_500_dataset = Load_Dataset(class_500_df)\n# class_500_dataset_loader = torch.utils.data.DataLoader(class_500_dataset,batch_size=64)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Extracting Feature Descriptors Using Resnet-50","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"# model = models.resnet50(pretrained=True)\n# model = torch.nn.Sequential(*(list(model.children())[:-1]))\n# device='cuda'\n# model.to(device)\n\n# output_descriptor = np.zeros((1,2048))\n# output_label = np.zeros((1))\n# output_means = np.zeros((1))\n# output_stds = np.zeros((1))\n\n# with torch.no_grad():\n#     for _,(images,labels,means,stds) in tqdm_notebook(enumerate(class_500_dataset_loader)):\n#         images,labels,means,stds = images.to(device),labels.to(device),means.to(device),stds.to(device)\n#         model.eval()\n#         predictions = model(images)\n#         output_descriptor = np.concatenate((output_descriptor,predictions.cpu().numpy().squeeze()),0)\n#         output_label = np.concatenate((output_label,labels.cpu().numpy()))\n#         output_means = np.concatenate((output_means,means.cpu().numpy()))\n#         output_stds = np.concatenate((output_stds,stds.cpu().numpy()))\n\n        \n# output_descriptor = output_descriptor[1:]\n# output_label = output_label[1:]\n# output_means = output_means[1:]\n# output_stds = output_stds[1:]\n\n# output_500 = pd.concat([pd.DataFrame(output_descriptor),pd.DataFrame(output_label),pd.DataFrame(output_means),pd.DataFrame(output_stds)],1)\n# col_names = []\n# for idx,col in enumerate(output_500.columns):\n#     if idx < 2048:\n#         col_names.append('f_'+str(col))\n#     elif idx == 2048:\n#         col_names.append('label')\n#     elif idx == 2049:\n#         col_names.append('mean')\n#     else:\n#         col_names.append('std')     \n# output_500.columns = col_names\n# output_500.to_csv('resnet_50_features.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"output_500 = pd.read_csv('../input/gld-resnet-50-features/resnet_50_features.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(30,10))\nplt.suptitle('Mean and Standard Deviation of Images', fontsize=14,fontweight='bold')\nplt.subplot(211)\nsns.boxplot(x=output_500.label.astype(int),y=output_500['mean'].astype(np.float64))\nplt.xticks(rotation=90)\nplt.xlabel('Class')\n\nplt.subplot(212)\nsns.boxplot(x=output_500.label.astype(int),y=output_500['std'].astype(np.float64))\nplt.xticks(rotation=90)\nplt.xlabel('Class')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"output_500['label'] = output_500['label'].astype(int)\noutput_500['mapped_label'] = output_500['label'].astype('category')\noutput_500['mapped_label'] = output_500['mapped_label'].cat.codes\noutput_500.head(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Validation Split(80-20)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train,valid = train_test_split(output_500,stratify=output_500['mapped_label'],\n                              test_size=0.20)\nx_train = train.iloc[:,:-4]\nx_valid = valid.iloc[:,:-4]\ny_train = train['mapped_label']\ny_valid= valid['mapped_label']\nx_train.shape,x_valid.shape","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_scatter(x, colors,true_label,train=True):\n    # choose a color palette with seaborn.\n    num_classes = len(np.unique(colors))\n    palette = np.array(sns.color_palette(\"Set1\", num_classes))\n\n    # create a scatter plot.\n    f = plt.figure(figsize=(8, 8))\n    ax = plt.subplot(aspect='equal')\n    sc = ax.scatter(x[:,0], x[:,1], lw=0, s=40, c=palette[colors.astype(np.int)])\n    plt.xlim(-25, 25)\n    plt.ylim(-25, 25)\n    ax.axis('off')\n    ax.axis('tight')\n\n    # add the labels for each digit corresponding to the label\n    txts = []\n\n    for i in range(num_classes):\n\n        # Position of each label at median of data points.\n\n        xtext, ytext = np.median(x[colors == i, :], axis=0)\n        txt = ax.text(xtext, ytext, str(i), fontsize=14)\n#         txt = ax.text(xtext, ytext, str(true_label[i]), fontsize=14)\n#         txt.set_path_effects([\n#             PathEffects.Stroke(linewidth=1, foreground=\"w\"),\n#             PathEffects.Normal()])\n#         txts.append(txt)\n    if train:\n        plt.title(\"Trian Data\")\n    else:\n        plt.title(\"Valid Data\")\n    plt.show()\n\n#     return f, ax, sc, txt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Principal Component Analysis(PCA)--> Linear Dimensionality Reduction","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"st_time = time.time()\npca = PCA(n_components=5)\npca_result_train = pca.fit_transform(x_train)\npca_result_test = pca.fit_transform(x_valid)\n\nprint('PCA done; Time take {} seconds'.format(time.time()-st_time))\nprint('Variance: {}'.format(pca.explained_variance_ratio_))\nprint('Sum of variance in data by first top five components: {:.2f}%'.format(100*(pca.explained_variance_ratio_.sum())))\n\n##PCA df\npca_tr = pd.DataFrame(columns=['pca1','pca2'])\npca_te = pd.DataFrame(columns=['pca1','pca2'])\n\npca_tr['pca1'] = pca_result_train[:,0]\npca_tr['pca2'] = pca_result_train[:,1]\n\npca_te['pca1'] = pca_result_test[:,0]\npca_te['pca2'] = pca_result_test[:,1]\n\nplot_scatter(pca_tr[['pca1','pca2']].values,y_train,train['label'].values)\nplot_scatter(pca_te[['pca1','pca2']].values,y_valid,valid['label'].values,False)\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('True Label for 31st class as shown in above plot is {}'.format(str(train[train['mapped_label'] == 31]['label'].values[0])))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can PCA was not able to cluster for all the classes, this may be due to the extracted features using resnet-50 (trained on Imagenet dataset). PCA might be able to give better clustering if we could try extracting features using model trained on similar dataset.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## t- Distributed Stochastic Neighbout (t-SNE)-->Non-linear Dimensionality Reduction Technique","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"T-sne took almost **2 Hours 14 Minutes to run** in CPU mode so i have saved the features as numpy as array and saved in \"GLDR RESNET 50 FEATURES\" dataset and made public  \n* Here is the link to access GLDR RESNET 50 FEATURES Dataset - https://www.kaggle.com/ramjib/gld-resnet-50-features\n* DATASET includes only features for the 51 classes for which number of images where more than 500. (Will try to extract for all the images and update the dataset asap)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# st_time = time.time()\n# t_sne = TSNE(random_state=2020)\n# t_sne_tr = t_sne.fit_transform(x_train)\n# t_sne_va = t_sne.fit_transform(x_valid)\n\n# np.save('t_sne_tr.npy',t_sne_tr)\n# np.save('t_sne_va.npy',t_sne_va)\n\n# print('TNSE done; Time take {} seconds'.format(time.time()-st_time))\n\n\n# plot_scatter(t_sne_tr, y_train,train['label'].values)\n# plot_scatter(t_sne_va, y_valid,valid['label'].values,False)\n\nimage = cv2.cvtColor(cv2.imread('../input/gld-resnet-50-features/tsne_train.png'), cv2.COLOR_BGR2RGB)\nplt.figure(figsize=(8, 8))\nplt.imshow(image)\nplt.axis('off')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"t-SNE was able to cluster the classes more clearly than PCA as we can observe","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Using PCA features for t-sne","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"st_time = time.time()\npca = PCA(n_components=100)\npca_result_train = pca.fit_transform(x_train)\npca_result_test = pca.fit_transform(x_valid)\n\nprint('PCA done; Time take {} seconds'.format(time.time()-st_time))\n# print('Variance: {}'.format(pca.explained_variance_ratio_))\nprint('Sum of variance in data by first top five components: {:.2f}%'.format(100*(pca.explained_variance_ratio_.sum())))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"st_time = time.time()\nt_sne = TSNE(random_state=2020)\nt_sne_tr = t_sne.fit_transform(pca_result_train)\nt_sne_va = t_sne.fit_transform(pca_result_test)\nprint('TNSE done; Time take {} seconds'.format(time.time()-st_time))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_scatter(t_sne_tr, y_train,train['label'].values)\nplot_scatter(t_sne_va, y_valid,valid['label'].values,False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"t_sne_tr.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Will try to do more visualizations in coming days. Thank You!!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## References:\n- https://www.datacamp.com/community/tutorials/introduction-t-sne","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}