{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There has been a lot of confusion on how exactly we are supposed to submit our model. As the ***Data*** section of the competition states:\n> Your model must be named submission.zip and be compatible with TensorFlow 2.2. The submission.zip should contain all files and directories created by the tf.saved_model_save function using Tensorflow's SavedModel format.\n\nNow question is what exactly in the [SavedModel](https://www.tensorflow.org/tutorials/keras/save_and_load#savedmodel_format) format do we need to submit.\n\nAlso, majority of us don't want to use tensorflow to train our models. And we don't know how to preprocess. So we'll tackle two things mainly.\n\n1. Use our own keras model in submission.\n2. How to preprocess.\n\nLet's get started.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's reverse engineer the model that organisers gave us as baseline. We'll use saved_model_cli to visualize it's structure. You may want to check out this [discussion thread](https://www.kaggle.com/c/landmark-retrieval-2020/discussion/163589).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!saved_model_cli show --dir \"../input/baseline-landmark-retrieval-model/baseline_landmark_retrieval_model\" --all","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Important things to notice are:\n\n    inputs['input_image'] tensor_info:\n    dtype: DT_UINT8\n    shape: (-1, -1, 3)\n        \n    outputs['global_descriptor'] tensor_info:\n    dtype: DT_FLOAT\n    shape: (2048)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Armed with this information, let's create our own model.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom PIL import Image\nimport matplotlib.pyplot as plt\n\nimport os\nimport cv2\nimport glob\n\nimport tensorflow as tf\nimport keras\nfrom keras.models import load_model, save_model\nfrom keras.layers import Input, GlobalAveragePooling2D, GlobalMaxPooling2D\nimport keras.backend as K\nfrom keras.models import Model, load_model\nfrom keras.applications import VGG16\nfrom keras.applications.vgg16 import preprocess_input","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# import the data\ntrain_df = pd.read_csv('../input/landmark-retrieval-2020/train.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[train_df.id == '000171b259e48280']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check the jpg images for the same landmark id\ntrain_df[train_df.landmark_id == 5724]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# first get the number of images with the same idnumber\nnumber_image = train_df[train_df.landmark_id == 107382].count().values[1]\nnumber_image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for number in np.arange(0,number_image):\n    print (number)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"the rule of naming the picture depends on the id, which start with the number or letter.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# check the folder structure of train folder\narr = os.listdir('../input/landmark-retrieval-2020/train/0/0/0/')\nprint(arr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[train_df.landmark_id == 107382].id.values[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# split the name of id and find the path to the image\nfolder_path_number = []\n\nfor i in np.arange(0,3):\n    folder_path_number.append(train_df[train_df.landmark_id == 107382].id.values[0][i])\n\nfolder_path_number","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# path of image\nnameofimage = train_df[train_df.landmark_id == 107382].id.values[0]\nimage_path = (f'../input/landmark-retrieval-2020/train/{folder_path_number[0]}/{folder_path_number[1]}/{folder_path_number[2]}/{nameofimage}.jpg')\nimage_path","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# show the image\nimage = cv2.imread(image_path)\nplt.figure(figsize = (12,9))\nplt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))\nplt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# the shape of image with 3 bit RGB info\nimage.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"number_image = train_df[train_df.landmark_id == 5724].count().values[1]\nlength = 8*number_image\nfig, axes = plt.subplots(1, number_image, figsize=(length, 6))\n\nfor number in np.arange(0,number_image): \n    folder_path_number = []\n    for i in np.arange(0,3):\n        folder_path_number.append(train_df[train_df.landmark_id == 5724].id.values[number][i])\n    \n    nameofimage = train_df[train_df.landmark_id == 5724].id.values[number]\n    #print (nameofimage)\n    image_path = (f'../input/landmark-retrieval-2020/train/{folder_path_number[0]}/{folder_path_number[1]}/{folder_path_number[2]}/{nameofimage}.jpg')\n    #print (image_path)\n    image = cv2.imread(image_path)\n    axes[number].imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))\n    axes[number].axis('off')\n    axes[number].set_title(f'id of jpg image: {nameofimage}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# define the function to get the path of image\ndef get_path(landmark_id):\n    # first get the number of images with the same idnumber\n    number_image = train_df[train_df.landmark_id == landmark_id].count().values[1]\n    \n    image_path = []\n    for number in np.arange(0,number_image):\n        folder_path_number = []    \n        for i in np.arange(0,3):\n            folder_path_number.append(train_df[train_df.landmark_id == landmark_id].id.values[number][i])\n          \n        nameofimage = train_df[train_df.landmark_id == landmark_id].id.values[0]\n        image_path.append((f'../input/landmark-retrieval-2020/train/{folder_path_number[0]}/{folder_path_number[1]}/{folder_path_number[2]}/{nameofimage}.jpg')) \n        \n    return image_path","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print the image with the same landmark id\ndef print_image(landmark_id):\n    # first get the number of images with the same idnumber\n    number_image = train_df[train_df.landmark_id == landmark_id].count().values[1]\n    \n    # build a figure subplot within a row\n    length = 8*number_image\n    fig, axes = plt.subplots(1, number_image, figsize=(length, 6))    \n    \n    for number in np.arange(0,number_image): \n        folder_path_number = []\n        for i in np.arange(0,3):\n            folder_path_number.append(train_df[train_df.landmark_id == landmark_id].id.values[number][i])\n    \n        nameofimage = train_df[train_df.landmark_id == landmark_id].id.values[number]\n        #print (nameofimage)\n        image_path = (f'../input/landmark-retrieval-2020/train/{folder_path_number[0]}/{folder_path_number[1]}/{folder_path_number[2]}/{nameofimage}.jpg')\n        #print (image_path)\n        image = cv2.imread(image_path)\n        axes[number].imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))\n        axes[number].axis('off')\n        axes[number].set_title(f'id of jpg image: {nameofimage}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_path(107382)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print_image(107382)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# the number of the unique landmark in the dataset\nprint ('The number of the unique landmark id in the dataset is', train_df.landmark_id.nunique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are varying shapes of images as you can see below, meaning we'll need to resize images inside the model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"files = glob.glob(\"../input/landmark-retrieval-2020/train/a/b/c/*.jpg\")\nfor i in range(10):\n    im = cv2.imread(files[i])\n    print(im.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's load our model. In this case the vanilla VGG16 pretrained model of Keras for demonstration purposes. Since this is not trained on any retrieval dataset, the score will most probably be zero.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"vgg = VGG16(input_shape=(224,224,3), weights=None, include_top=False)\nvgg.load_weights(\"../input/keras-pretrained-models/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5\")\n\ninput_image = Input((224,224,3))\nx = vgg(input_image)\noutput = GlobalMaxPooling2D()(x)\n\nmodel = Model(inputs=[input_image], outputs=[output])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now the main part! The *input_image* will be in it's own variable shape and hence we need to resize it within the model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\n\nclass MyModel(tf.keras.Model):\n    def __init__(self):\n        super(MyModel, self).__init__()\n        self.model = model\n    \n    @tf.function(input_signature=[\n      tf.TensorSpec(shape=[None, None, 3], dtype=tf.uint8, name='input_image')\n    ])\n    def call(self, input_image):\n        output_tensors = {}\n        \n        # resizing\n        im = tf.image.resize(input_image, (224,224))\n        \n        # preprocessing\n        im = preprocess_input(im)\n        \n        extracted_features = self.model(tf.convert_to_tensor([im], dtype=tf.uint8))[0]\n        output_tensors['global_descriptor'] = tf.identity(extracted_features, name='global_descriptor')\n        return output_tensors","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Now we create and save our model instance.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"m = MyModel() #creating our model instance\n\nserved_function = m.call\ntf.saved_model.save(\n      m, export_dir=\"./my_model\", signatures={'serving_default': served_function})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ./my_model/variables","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from zipfile import ZipFile\n\nwith ZipFile('submission.zip','w') as zip:           \n    zip.write('./my_model/saved_model.pb', arcname='saved_model.pb') \n    zip.write('./my_model/variables/variables.data-00000-of-00001', arcname='variables/variables.data-00000-of-00001')\n    #zip.write('./my_model/variables/variables.data-00001-of-00002', arcname='variables/variables.data-00001-of-00002') \n    zip.write('./my_model/variables/variables.index', arcname='variables/variables.index') \n    #zip.write('./my_model/assets', arcname='assets') ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Last but not the least, let's visualize our model to see if the structure is as per the requirements.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!saved_model_cli show --dir ./my_model/ --all","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Please upvote and let me know if this helps!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}