{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### This notebook consists of two parts, the 1st demonstrates how can we create startified tfrecords and the other explains how to set up training pipeline with those tfrecords!! \n\nIn Development (For 2021 Data)","metadata":{}},{"cell_type":"markdown","source":"# 1. Creating Stratified TFRecords For Google Landmark Recognition 2021\nThis notebook discusses how you can create Stratified TFRecords to train tensorflow models for Google Landmark Recognition 2021. Some parts of the code is taken from this excellent notebook by Chris Doette: https://www.kaggle.com/cdeotte/how-to-create-tfrecords/output. The folds are created by stratifying on landmark_id. Further groups are created on the basis of density of landmark id to which image belongs. This grouping makes it easy to train models on landmark_id whose sample size is greater than certain threshold. You are encouraged to try your own groups based on your training strategy. \n\nFor each (fold,group), there is a single TFRecord. \nFor example **train-0-1.tfrec** corresponds to Fold 0 & Group 1\n\nFurther the notebook discusses how you can read these TFRecords & prepare dataset which can directly be passed to keras.fit method.\n\n**Suggestion:** If you train only single model, choose fold 0 or 1 as validation set to ensure all landmark ids have at least one sample in validation set\n\nLast but not the least, I have prepared all the 60 tfrecords and the link for Kaggle Dataset is mentioned at the end.","metadata":{}},{"cell_type":"markdown","source":"## Load Libraries","metadata":{}},{"cell_type":"code","source":"import os, json, random, cv2\nimport numpy as np, pandas as pd\nimport matplotlib.pyplot as plt\nimport tensorflow as tf, re, math\nfrom tqdm import tqdm","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:08.91328Z","iopub.execute_input":"2021-08-24T16:23:08.913706Z","iopub.status.idle":"2021-08-24T16:23:15.111391Z","shell.execute_reply.started":"2021-08-24T16:23:08.913621Z","shell.execute_reply":"2021-08-24T16:23:15.110233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Login to Kaggle API\n\nKaggle API allows you to perform basic dataset operations (creation, versioning, etc). We are going to create tfrecords in /tmp/ directory which can store upto 100 GBs of data and then create a single kaggle dataset using kaggle api. For more details regarding kaggle api, please refer https://www.kaggle.com/docs/api or https://github.com/Kaggle/kaggle-api","metadata":{}},{"cell_type":"code","source":"%%time\n\n### Create Kaggle Dataset if not exists \n\nDATASET_NAME = 'landmark-recognition-2021-tfrecords-subset'\n\n!rm -r /tmp/{DATASET_NAME}\n\nos.makedirs(f'/tmp/{DATASET_NAME}', exist_ok=True)\n\nwith open('../input/kaggle-api-creds/kaggle.json') as f:\n    kaggle_creds = json.load(f)\n    \nos.environ['KAGGLE_USERNAME'] = kaggle_creds['username']\nos.environ['KAGGLE_KEY'] = kaggle_creds['key']\n\n!kaggle datasets init -p /tmp/{DATASET_NAME}\n\nwith open(f'/tmp/{DATASET_NAME}/dataset-metadata.json') as f:\n    dataset_meta = json.load(f)\ndataset_meta['id'] = f'ks2019/{DATASET_NAME}'\ndataset_meta['title'] = DATASET_NAME\nwith open(f'/tmp/{DATASET_NAME}/dataset-metadata.json', \"w\") as outfile:\n    json.dump(dataset_meta, outfile)\nprint(dataset_meta)\n\n!cp /tmp/{DATASET_NAME}/dataset-metadata.json /tmp/{DATASET_NAME}/meta.json\n!ls /tmp/{DATASET_NAME}\n\n!kaggle datasets create -u -p /tmp/{DATASET_NAME} ","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:15.116476Z","iopub.execute_input":"2021-08-24T16:23:15.116902Z","iopub.status.idle":"2021-08-24T16:23:22.824779Z","shell.execute_reply.started":"2021-08-24T16:23:15.11686Z","shell.execute_reply":"2021-08-24T16:23:22.823657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Configurations","metadata":{}},{"cell_type":"code","source":"RESIZE = True\nN_LABELS = 81313\nN_LABELS_SUBSET = 100\nIMAGE_SIZE = 224\nN_GROUPS = 12\nN_FOLDS = 5\nN_TFRs = N_GROUPS*N_FOLDS\nSUBSET = True  # Keep SUBSET=True while debugging (Faster Execution)\nBATCH_SIZE = 32\nFOLDS = [0]\nGROUPS = [11]\nassert max(FOLDS)<N_FOLDS, \"ELEMENTS OF FOLDS can't be greater than N_FOLDS\"\nassert max(GROUPS)<N_GROUPS, \"ELEMENTS OF FOLDS can't be greater than N_FOLDS\"","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:22.826734Z","iopub.execute_input":"2021-08-24T16:23:22.827056Z","iopub.status.idle":"2021-08-24T16:23:22.833641Z","shell.execute_reply.started":"2021-08-24T16:23:22.827017Z","shell.execute_reply":"2021-08-24T16:23:22.832344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preparing Folds and Groups\n### You may change heuristics for Groups as per your requirements","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv('../input/landmark-recognition-2021/train.csv')\nif SUBSET:\n    landmarks = random.sample(list(train_df.landmark_id.unique()),100)\n    train_df = train_df[train_df.landmark_id.isin(landmarks)].reset_index(drop=True)\n    N_LABELS = N_LABELS_SUBSET\ntrain_df['original_landmark_id'] = train_df.landmark_id\nprint(train_df.shape)\ntrain_df['order'] = np.arange(train_df.shape[0])\ntrain_df['order'] = train_df.groupby('landmark_id').order.rank()-1\nlandmark_counts = train_df.landmark_id.value_counts()\ntrain_df['landmark_counts'] = landmark_counts.loc[train_df.landmark_id.values].values\ntrain_df['fold'] = (train_df['order']%N_FOLDS).astype(int)\nall_groups = [(1/N_GROUPS)*x for x in range(N_GROUPS)]\nfor i,partition_val in enumerate(train_df.landmark_counts.quantile(all_groups).values):\n                     train_df.loc[train_df.landmark_counts>=partition_val,'group'] = i \n        \nlandmark_map = train_df.sort_values(by='landmark_counts').landmark_id.drop_duplicates().reset_index(drop=True)\nlandmark_dict = {landmark_map.loc[x]:N_LABELS-x-1 for x in range(N_LABELS)}\ntrain_df['landmark_id'] = train_df.original_landmark_id.apply(lambda x: landmark_dict[x])\ntrain_df = train_df.sample(frac=1).reset_index(drop=True)\ntrain_df.to_csv(f'/tmp/{DATASET_NAME}/train_meta_data.csv',index=False)\ntrain_df.sample(5)","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:22.835504Z","iopub.execute_input":"2021-08-24T16:23:22.835805Z","iopub.status.idle":"2021-08-24T16:23:24.782942Z","shell.execute_reply.started":"2021-08-24T16:23:22.835776Z","shell.execute_reply":"2021-08-24T16:23:24.781959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Checking Null values\ntrain_df.isna().sum().sum()","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:24.784218Z","iopub.execute_input":"2021-08-24T16:23:24.78451Z","iopub.status.idle":"2021-08-24T16:23:24.792469Z","shell.execute_reply.started":"2021-08-24T16:23:24.78448Z","shell.execute_reply":"2021-08-24T16:23:24.791408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Group Partitions","metadata":{}},{"cell_type":"code","source":"train_df.groupby('group').landmark_counts.agg(['min','max'])","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:24.793869Z","iopub.execute_input":"2021-08-24T16:23:24.794194Z","iopub.status.idle":"2021-08-24T16:23:24.823691Z","shell.execute_reply.started":"2021-08-24T16:23:24.794163Z","shell.execute_reply":"2021-08-24T16:23:24.822641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Some Statistics","metadata":{}},{"cell_type":"code","source":"#Landmark Counts\ntrain_df.landmark_id.value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:24.825129Z","iopub.execute_input":"2021-08-24T16:23:24.825809Z","iopub.status.idle":"2021-08-24T16:23:24.836313Z","shell.execute_reply.started":"2021-08-24T16:23:24.825765Z","shell.execute_reply":"2021-08-24T16:23:24.835301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#No of images GroupBy landmark counts\ntrain_df.landmark_counts.value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:24.838788Z","iopub.execute_input":"2021-08-24T16:23:24.839072Z","iopub.status.idle":"2021-08-24T16:23:24.852144Z","shell.execute_reply.started":"2021-08-24T16:23:24.839043Z","shell.execute_reply":"2021-08-24T16:23:24.850909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#No of Images in Each Folds\ntrain_df.fold.value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:24.853989Z","iopub.execute_input":"2021-08-24T16:23:24.854302Z","iopub.status.idle":"2021-08-24T16:23:24.863411Z","shell.execute_reply.started":"2021-08-24T16:23:24.854272Z","shell.execute_reply":"2021-08-24T16:23:24.862407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#No of Images in Each Group\ntrain_df.group.value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:24.864881Z","iopub.execute_input":"2021-08-24T16:23:24.865284Z","iopub.status.idle":"2021-08-24T16:23:24.878628Z","shell.execute_reply.started":"2021-08-24T16:23:24.865252Z","shell.execute_reply":"2021-08-24T16:23:24.877642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# No of Landmark in each Fold\ntrain_df.groupby('fold').landmark_id.nunique()","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:24.87981Z","iopub.execute_input":"2021-08-24T16:23:24.880082Z","iopub.status.idle":"2021-08-24T16:23:24.889193Z","shell.execute_reply.started":"2021-08-24T16:23:24.880056Z","shell.execute_reply":"2021-08-24T16:23:24.888077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## No of images in each (fold,group)\n### Each row corresponds to single tf-records ","metadata":{}},{"cell_type":"code","source":"pd.pivot_table(train_df.groupby(['fold','group']).id.count().reset_index(),index='fold',columns='group')","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2021-08-24T16:23:24.891062Z","iopub.execute_input":"2021-08-24T16:23:24.891423Z","iopub.status.idle":"2021-08-24T16:23:24.930434Z","shell.execute_reply.started":"2021-08-24T16:23:24.891391Z","shell.execute_reply":"2021-08-24T16:23:24.929734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Creating TF-Records","metadata":{}},{"cell_type":"code","source":"def _bytes_feature(value):\n  \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n  if isinstance(value, type(tf.constant(0))):\n    value = value.numpy() # BytesList won't unpack a string from an EagerTensor.\n  return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n\ndef _float_feature(value):\n  \"\"\"Returns a float_list from a float / double.\"\"\"\n  return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))\n\ndef _int64_feature(value):\n  \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n  return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))\n\ndef serialize_example(image,image_name,label):\n    feature = {\n        'image': _bytes_feature(image),\n        'image_name': _bytes_feature(image_name),\n        'target': _int64_feature(label),\n      }\n    example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n    return example_proto.SerializeToString()","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:24.93162Z","iopub.execute_input":"2021-08-24T16:23:24.932066Z","iopub.status.idle":"2021-08-24T16:23:24.940196Z","shell.execute_reply.started":"2021-08-24T16:23:24.932035Z","shell.execute_reply":"2021-08-24T16:23:24.939562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def create_tf_records(fold  = 0, group = 0):\n    df = train_df[(train_df.fold==fold) & (train_df.group==group)]\n    tfr_filename = f'/tmp/{DATASET_NAME}/landmark-2021-train-{fold}-{group}-{df.shape[0]}.tfrec'\n    with tf.io.TFRecordWriter(tfr_filename) as writer:\n        for i,row in df.iterrows():\n            image_id = row.id\n            target = row.landmark_id\n            image_path = \"../input/landmark-recognition-2020/train/{}/{}/{}/{}.jpg\".format(image_id[0],image_id[1],image_id[2],image_id) \n            image_encoded = tf.io.read_file(image_path)\n            image_name = str.encode(image_id)\n            example = serialize_example(image_encoded,image_name,target)\n            writer.write(example)","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:24.941712Z","iopub.execute_input":"2021-08-24T16:23:24.942277Z","iopub.status.idle":"2021-08-24T16:23:24.955497Z","shell.execute_reply.started":"2021-08-24T16:23:24.942245Z","shell.execute_reply":"2021-08-24T16:23:24.954505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import joblib\nfor fold in range(N_FOLDS):\n    _ = joblib.Parallel(n_jobs=8)(\n        joblib.delayed(create_tf_records)(fold,group) for group in tqdm(range(N_GROUPS))\n    )","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:24.956916Z","iopub.execute_input":"2021-08-24T16:23:24.957223Z","iopub.status.idle":"2021-08-24T16:23:40.428196Z","shell.execute_reply.started":"2021-08-24T16:23:24.957196Z","shell.execute_reply":"2021-08-24T16:23:40.427165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Upload Dataset","metadata":{}},{"cell_type":"code","source":"from datetime import datetime\nversion_name = datetime.now().strftime(\"%Y%m%d-%H%M%S\")\nprint(version_name)","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:40.42953Z","iopub.execute_input":"2021-08-24T16:23:40.429825Z","iopub.status.idle":"2021-08-24T16:23:40.434918Z","shell.execute_reply.started":"2021-08-24T16:23:40.429791Z","shell.execute_reply":"2021-08-24T16:23:40.434055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls /tmp/{DATASET_NAME}","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:40.436071Z","iopub.execute_input":"2021-08-24T16:23:40.436563Z","iopub.status.idle":"2021-08-24T16:23:41.187324Z","shell.execute_reply.started":"2021-08-24T16:23:40.436522Z","shell.execute_reply":"2021-08-24T16:23:41.186203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!kaggle datasets version -m {version_name} -p /tmp/{DATASET_NAME} -r zip -q","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:23:41.188855Z","iopub.execute_input":"2021-08-24T16:23:41.189199Z","iopub.status.idle":"2021-08-24T16:25:32.13376Z","shell.execute_reply.started":"2021-08-24T16:23:41.189144Z","shell.execute_reply":"2021-08-24T16:25:32.13251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Reading TFRecords & preparing Tensorflow Dataset for model training","metadata":{}},{"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.image.resize(image,IMAGE_SIZE_)\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        'target': tf.io.FixedLenFeature([], tf.int64),\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = example['target']\n    return image, label # returns a dataset of (image, label) pairs\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset\n\ndef get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:25:32.135479Z","iopub.execute_input":"2021-08-24T16:25:32.135794Z","iopub.status.idle":"2021-08-24T16:25:32.149207Z","shell.execute_reply.started":"2021-08-24T16:25:32.135757Z","shell.execute_reply":"2021-08-24T16:25:32.147818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"IMAGE_SIZE_ = [IMAGE_SIZE,IMAGE_SIZE]\nAUTO = tf.data.experimental.AUTOTUNE\nTRAINING_FILENAMES = tf.io.gfile.glob(f'/tmp/{DATASET_NAME}/landmark-2021-train*.tfrec')\nprint(len(TRAINING_FILENAMES))\ndataset = load_dataset(TRAINING_FILENAMES, labeled=True)\ndataset = dataset.repeat()\ndataset = dataset.shuffle(2048)\ndataset = dataset.batch(BATCH_SIZE)\ndataset = dataset.prefetch(AUTO) #This dataset can directly be passed to keras.fit method\ncount_data_items(TRAINING_FILENAMES)","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:25:32.151005Z","iopub.execute_input":"2021-08-24T16:25:32.151436Z","iopub.status.idle":"2021-08-24T16:25:32.399387Z","shell.execute_reply.started":"2021-08-24T16:25:32.151392Z","shell.execute_reply":"2021-08-24T16:25:32.398356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for x,y in dataset:\n    print(x.shape,y.shape)\n    break","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:25:32.400886Z","iopub.execute_input":"2021-08-24T16:25:32.40132Z","iopub.status.idle":"2021-08-24T16:25:44.395534Z","shell.execute_reply.started":"2021-08-24T16:25:32.401276Z","shell.execute_reply":"2021-08-24T16:25:44.394368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Verifying Dataset is written in correct format","metadata":{}},{"cell_type":"code","source":"# numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)\nCLASSES = [0,1]\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    #if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n    #    numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef display_single_sample(image, label, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    title = str(label)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch):\n    \"\"\"\n    Display single batch Of images \n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        correct = True\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_single_sample(image, label, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:25:44.396971Z","iopub.execute_input":"2021-08-24T16:25:44.397311Z","iopub.status.idle":"2021-08-24T16:25:44.412745Z","shell.execute_reply.started":"2021-08-24T16:25:44.397276Z","shell.execute_reply":"2021-08-24T16:25:44.411599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Displaying single batch of TFRecord\ntrain_batch = iter(dataset)\ndisplay_batch_of_images(next(train_batch))","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:25:44.418078Z","iopub.execute_input":"2021-08-24T16:25:44.418582Z","iopub.status.idle":"2021-08-24T16:25:58.567718Z","shell.execute_reply.started":"2021-08-24T16:25:44.418533Z","shell.execute_reply":"2021-08-24T16:25:58.566575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Great, Now you have to run this notebook for different folds & prepare all the tf records!!The good part is that I have already done this & you can find all the tf records & metadata here:\n1. Fold 0: https://www.kaggle.com/ks2019/landmark-recognition-2021-tfrecords-fold0\n2. Fold 1: https://www.kaggle.com/ks2019/landmark-recognition-2021-tfrecords-fold1\n3. Fold 2: https://www.kaggle.com/ks2019/landmark-recognition-2021-tfrecords-fold2\n4. Fold 3: https://www.kaggle.com/ks2019/landmark-recognition-2021-tfrecords-fold3\n5. Fold 4: https://www.kaggle.com/ks2019/landmark-recognition-2021-tfrecords-fold4\n6. Metadata: https://www.kaggle.com/ks2019/landmark-recognition-2021-tfrecords-fold1?select=train_meta_data.csv\n\nPlease note for training models, you don't need to add any dataset, you can directly use the gcs paths. The GCS path for a dataset can be found by using the following commands: \n\n```from kaggle_datasets import KaggleDatasets\nKaggleDatasets().get_gcs_path(f'landmark-recognition-2021-tfrecords-fold{fold}')```\n","metadata":{}},{"cell_type":"markdown","source":"# 2. Setting up Training Pipeline\nThis section discusses how to train efficient net model using TFRecords prepared above!! Most of the part of this section is taken from this excellent chris notebook: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords !! \n\nThis only demonstrates training pipeline for a subset of training data!! You may want to play with \"GROUPS\" argument to add more data!! \n### The arcface implementation is taken from:\n1. https://www.kaggle.com/akensert/glrec-resnet50-arcface-tf2-2\n2. https://www.kaggle.com/ragnar123/shopee-efficientnetb3-arcmarginproduct","metadata":{}},{"cell_type":"code","source":"!pip install -q efficientnet\n!pip install tensorflow_addons\nimport re\nimport os\nimport numpy as np\nimport pandas as pd\nimport random\nimport math\nimport tensorflow as tf\nimport efficientnet.tfkeras as efn\nfrom sklearn import metrics\nfrom sklearn.model_selection import KFold, train_test_split\nfrom tensorflow.keras import backend as K\nimport tensorflow_addons as tfa\nfrom tqdm.notebook import tqdm\nfrom kaggle_datasets import KaggleDatasets\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:25:58.573748Z","iopub.execute_input":"2021-08-24T16:25:58.574168Z","iopub.status.idle":"2021-08-24T16:26:15.906394Z","shell.execute_reply.started":"2021-08-24T16:25:58.574104Z","shell.execute_reply":"2021-08-24T16:26:15.905257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:26:15.908213Z","iopub.execute_input":"2021-08-24T16:26:15.908661Z","iopub.status.idle":"2021-08-24T16:26:21.325793Z","shell.execute_reply.started":"2021-08-24T16:26:15.908612Z","shell.execute_reply":"2021-08-24T16:26:21.324978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# For tf.dataset\nAUTO = tf.data.experimental.AUTOTUNE\n\n### Note that: These GCS paths are updated every week. To get updated GCS path, follow the instructions mentioned above.\n\n# Data access\nGCS_PATHS = {\n    0: 'gs://kds-c430f4dc931cb05decf924854e81afc21a367188a0feb7718872ebf1',\n    1: 'gs://kds-47fd736d084594eec1e33a8bba3c79c0539e21279a3207233c12dcb6',\n    2: 'gs://kds-de7798fc5a5e4670e6421cd846bf3dd1dcc9c6340acef9fc5a85f247',\n    3: 'gs://kds-89b8e7f8f9fe4836d0092e6b753505a82af130e9c270d91c1ee092c7',\n    4: 'gs://kds-697a86388d216aba3e3119019447f5053586208639b827f5bb0fb53a'\n}\n\n# Configuration\nEPOCHS = 4\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nIMAGE_SIZE = [256, 256]\n# Seed\nSEED = 42\nFOLD_TO_RUN = [0,1,2,3,4]\n# Learning rate\nLR = 0.001\n# Verbosity\nVERBOSE = 2\n# Number of classes\nN_CLASSES = 81313\n# Number of folds\nFOLDS = 5\n# EfficientNet\nEFF_NET = 0\n# Freeze Batch Norm\nFREEZE_BATCH_NORM = False\nSNAPSHOT_THRESHOLD = 0\n\n# Training filenames directory\nTRAINING_FILENAMES = []\nfor fold in GCS_PATHS:\n    TRAINING_FILENAMES += tf.io.gfile.glob(GCS_PATHS[fold] + '/*.tfrec')\n    ","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:26:21.326938Z","iopub.execute_input":"2021-08-24T16:26:21.327267Z","iopub.status.idle":"2021-08-24T16:26:21.747252Z","shell.execute_reply.started":"2021-08-24T16:26:21.327234Z","shell.execute_reply":"2021-08-24T16:26:21.746359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(TRAINING_FILENAMES)","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:26:21.748564Z","iopub.execute_input":"2021-08-24T16:26:21.74921Z","iopub.status.idle":"2021-08-24T16:26:21.75669Z","shell.execute_reply.started":"2021-08-24T16:26:21.749146Z","shell.execute_reply":"2021-08-24T16:26:21.755524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to get our f1 score\ndef f1_score(y_true, y_pred):\n    y_true = y_true.apply(lambda x: set(x.split()))\n    y_pred = y_pred.apply(lambda x: set(x.split()))\n    intersection = np.array([len(x[0] & x[1]) for x in zip(y_true, y_pred)])\n    len_y_pred = y_pred.apply(lambda x: len(x)).values\n    len_y_true = y_true.apply(lambda x: len(x)).values\n    f1 = 2 * intersection / (len_y_pred + len_y_true)\n    return f1\n\n# Function to seed everything\ndef seed_everything(seed):\n    random.seed(seed)\n    np.random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    tf.random.set_seed(seed)\n    \ndef arcface_format(posting_id, image, label_group, matches):\n    return posting_id, {'inp1': image, 'inp2': label_group}, label_group, matches\n\n# Data augmentation function\ndef data_augment(posting_id, image, label_group, matches):\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_hue(image, 0.01)\n    image = tf.image.random_saturation(image, 0.70, 1.30)\n    image = tf.image.random_contrast(image, 0.80, 1.20)\n    image = tf.image.random_brightness(image, 0.10)\n    return posting_id, image, label_group, matches\n\n# Function to decode our images\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels = 3)\n    image = tf.image.resize(image, IMAGE_SIZE)\n    image = tf.cast(image, tf.float32) / 255.0\n    return image\n\n# This function parse our images and also get the target variable\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.int64),\n#         \"matches\": tf.io.FixedLenFeature([], tf.string)\n    }\n\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    posting_id = example['image_name']\n    image = decode_image(example['image'])\n#     label_group = tf.one_hot(tf.cast(example['label_group'], tf.int32), depth = N_CLASSES)\n    label_group = tf.cast(example['target'], tf.int32)\n#     matches = example['matches']\n    matches = 1\n    return posting_id, image, label_group, matches\n\n# This function loads TF Records and parse them into tensors\ndef load_dataset(filenames, ordered = False, cache=False):\n    \n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False \n        \n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads = AUTO)\n    if cache:\n        dataset = dataset.cache()\n    dataset = dataset.with_options(ignore_order)\n    dataset = dataset.map(read_labeled_tfrecord, num_parallel_calls = AUTO) \n    return dataset\n\n# This function is to get our training tensors\ndef get_training_dataset(filenames, ordered = False):\n    dataset = load_dataset(filenames, ordered = ordered)\n    dataset = dataset.map(data_augment, num_parallel_calls = AUTO)\n    dataset = dataset.map(arcface_format, num_parallel_calls = AUTO)\n    dataset = dataset.map(lambda posting_id, image, label_group, matches: (image, label_group))\n    dataset = dataset.repeat()\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\n# This function is to get our validation tensors\ndef get_validation_dataset(filenames, ordered = True):\n    dataset = load_dataset(filenames, ordered = ordered)\n    dataset = dataset.map(arcface_format, num_parallel_calls = AUTO)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) \n    return dataset\n\n# Function to count how many photos we have in\ndef count_data_items(filenames):\n    # The number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nprint(f'Dataset: {NUM_TRAINING_IMAGES} training images')","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:26:21.75852Z","iopub.execute_input":"2021-08-24T16:26:21.759125Z","iopub.status.idle":"2021-08-24T16:26:21.785434Z","shell.execute_reply.started":"2021-08-24T16:26:21.759081Z","shell.execute_reply":"2021-08-24T16:26:21.784254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_lr_callback(plot=False):\n    lr_start   = 0.000001\n    lr_max     = 0.000005 * BATCH_SIZE  \n    lr_min     = 0.000001\n    lr_ramp_ep = 5\n    lr_sus_ep  = 0\n    lr_decay   = 0.8\n   \n    def lrfn(epoch):\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n            \n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n            \n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n            \n        return lr\n        \n    if plot:\n        epochs = list(range(EPOCHS))\n        learning_rates = [lrfn(x) for x in epochs]\n        plt.scatter(epochs,learning_rates)\n        plt.show()\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback\n\nget_lr_callback(plot=True)","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:26:21.786909Z","iopub.execute_input":"2021-08-24T16:26:21.787488Z","iopub.status.idle":"2021-08-24T16:26:21.979344Z","shell.execute_reply.started":"2021-08-24T16:26:21.787444Z","shell.execute_reply":"2021-08-24T16:26:21.978223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Arcmarginproduct class keras layer\nclass ArcMarginProduct(tf.keras.layers.Layer):\n    '''\n    Implements large margin arc distance.\n\n    Reference:\n        https://arxiv.org/pdf/1801.07698.pdf\n        https://github.com/lyakaap/Landmark2019-1st-and-3rd-Place-Solution/\n            blob/master/src/modeling/metric_learning.py\n    '''\n    def __init__(self, n_classes, s=30, m=0.50, easy_margin=False,\n                 ls_eps=0.0, **kwargs):\n\n        super(ArcMarginProduct, self).__init__(**kwargs)\n\n        self.n_classes = n_classes\n        self.s = s\n        self.m = m\n        self.ls_eps = ls_eps\n        self.easy_margin = easy_margin\n        self.cos_m = tf.math.cos(m)\n        self.sin_m = tf.math.sin(m)\n        self.th = tf.math.cos(math.pi - m)\n        self.mm = tf.math.sin(math.pi - m) * m\n\n    def get_config(self):\n\n        config = super().get_config().copy()\n        config.update({\n            'n_classes': self.n_classes,\n            's': self.s,\n            'm': self.m,\n            'ls_eps': self.ls_eps,\n            'easy_margin': self.easy_margin,\n        })\n        return config\n\n    def build(self, input_shape):\n        super(ArcMarginProduct, self).build(input_shape[0])\n\n        self.W = self.add_weight(\n            name='W',\n            shape=(int(input_shape[0][-1]), self.n_classes),\n            initializer='glorot_uniform',\n            dtype='float32',\n            trainable=True,\n            regularizer=None)\n\n    def call(self, inputs):\n        X, y = inputs\n        y = tf.cast(y, dtype=tf.int32)\n        cosine = tf.matmul(\n            tf.math.l2_normalize(X, axis=1),\n            tf.math.l2_normalize(self.W, axis=0)\n        )\n        sine = tf.math.sqrt(1.0 - tf.math.pow(cosine, 2))\n        phi = cosine * self.cos_m - sine * self.sin_m\n        if self.easy_margin:\n            phi = tf.where(cosine > 0, phi, cosine)\n        else:\n            phi = tf.where(cosine > self.th, phi, cosine - self.mm)\n        one_hot = tf.cast(\n            tf.one_hot(y, depth=self.n_classes),\n            dtype=cosine.dtype\n        )\n        if self.ls_eps > 0:\n            one_hot = (1 - self.ls_eps) * one_hot + self.ls_eps / self.n_classes\n\n        output = (one_hot * phi) + ((1.0 - one_hot) * cosine)\n        output *= self.s\n        return output\n\nEFNS = [efn.EfficientNetB0, efn.EfficientNetB1, efn.EfficientNetB2, efn.EfficientNetB3, \n        efn.EfficientNetB4, efn.EfficientNetB5, efn.EfficientNetB6, efn.EfficientNetB7]\n\ndef freeze_BN(model):\n    # Unfreeze layers while leaving BatchNorm layers frozen\n    for layer in model.layers:\n        if not isinstance(layer, tf.keras.layers.BatchNormalization):\n            layer.trainable = True\n        else:\n            layer.trainable = False\n\n# Function to create our EfficientNetB3 model\ndef get_model():\n\n    with strategy.scope():\n\n        margin = ArcMarginProduct(\n            n_classes = N_CLASSES, \n            s = 30, \n            m = 0.5, \n            name='head/arc_margin', \n            dtype='float32'\n            )\n\n        inp = tf.keras.layers.Input(shape = (*IMAGE_SIZE, 3), name = 'inp1')\n        label = tf.keras.layers.Input(shape = (), name = 'inp2')\n        x = EFNS[EFF_NET](weights = 'imagenet', include_top = False)(inp)\n        x = tf.keras.layers.GlobalAveragePooling2D()(x)\n        x = margin([x, label])\n        \n        output = tf.keras.layers.Softmax(dtype='float32')(x)\n\n        model = tf.keras.models.Model(inputs = [inp, label], outputs = [output])\n\n        opt = tf.keras.optimizers.Adam(learning_rate = LR)\n        if FREEZE_BATCH_NORM:\n            freeze_BN(model)\n\n        model.compile(\n            optimizer = opt,\n            loss = [tf.keras.losses.SparseCategoricalCrossentropy()],\n            metrics = [tf.keras.metrics.SparseCategoricalAccuracy()]\n            ) \n        \n        return model\nget_model().summary()","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:26:21.980739Z","iopub.execute_input":"2021-08-24T16:26:21.981033Z","iopub.status.idle":"2021-08-24T16:26:37.001341Z","shell.execute_reply.started":"2021-08-24T16:26:21.981004Z","shell.execute_reply":"2021-08-24T16:26:37.000247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"row = 10; col = 2;\nrow = min(row,BATCH_SIZE//col)\nN_TRAIN = count_data_items(TRAINING_FILENAMES)\nprint(N_TRAIN)\nds = get_training_dataset(TRAINING_FILENAMES, ordered = False)\n\nfor (sample,label) in ds:\n    img = sample['inp1']\n    plt.figure(figsize=(25,int(25*row/col)))\n    for j in range(row*col):\n        plt.subplot(row,col,j+1)\n        plt.title(label[j].numpy())\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break\nprint(img.shape)","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:26:37.002906Z","iopub.execute_input":"2021-08-24T16:26:37.003346Z","iopub.status.idle":"2021-08-24T16:26:50.489119Z","shell.execute_reply.started":"2021-08-24T16:26:37.003309Z","shell.execute_reply":"2021-08-24T16:26:50.488331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample.keys()","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:26:50.490147Z","iopub.execute_input":"2021-08-24T16:26:50.490574Z","iopub.status.idle":"2021-08-24T16:26:50.496482Z","shell.execute_reply.started":"2021-08-24T16:26:50.490542Z","shell.execute_reply":"2021-08-24T16:26:50.495292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def is_interactive():\n    return 'runtime'    in get_ipython().config.IPKernelApp.connection_file\nIS_INTERACTIVE = is_interactive()\nprint(IS_INTERACTIVE)","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:26:50.497886Z","iopub.execute_input":"2021-08-24T16:26:50.498355Z","iopub.status.idle":"2021-08-24T16:26:50.509902Z","shell.execute_reply.started":"2021-08-24T16:26:50.498307Z","shell.execute_reply":"2021-08-24T16:26:50.508794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Snapshot(tf.keras.callbacks.Callback):\n    \n    def __init__(self,snapshot_min_epoch,fold):\n        super(Snapshot, self).__init__()\n        self.snapshot_min_epoch = snapshot_min_epoch\n        self.fold = fold\n        \n        \n    def on_epoch_end(self, epoch, logs=None):\n        # logs is a dictionary\n#         print(f\"epoch: {epoch}, train_acc: {logs['acc']}, valid_acc: {logs['val_acc']}\")\n        if epoch >=self.snapshot_min_epoch: # your custom condition         \n            self.model.save_weights(f\"EF{EFF_NET}_fold{self.fold}_epoch{epoch}.h5\")","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:26:50.511623Z","iopub.execute_input":"2021-08-24T16:26:50.511992Z","iopub.status.idle":"2021-08-24T16:26:50.526308Z","shell.execute_reply.started":"2021-08-24T16:26:50.51195Z","shell.execute_reply":"2021-08-24T16:26:50.524285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"VERBOSE = 1\nseed_everything(SEED)\ntrain = np.array(TRAINING_FILENAMES)\nSTEPS_PER_EPOCH = count_data_items(train) // BATCH_SIZE\ntrain_dataset = get_training_dataset(train, ordered = False)\nmodel = get_model()\nsnap = Snapshot(snapshot_min_epoch=SNAPSHOT_THRESHOLD,fold=0)\nhistory = model.fit(train_dataset,\n                        steps_per_epoch = STEPS_PER_EPOCH,\n                        epochs = EPOCHS,\n                        callbacks = [snap,get_lr_callback()], \n                        verbose = VERBOSE)","metadata":{"execution":{"iopub.status.busy":"2021-08-24T16:26:50.528258Z","iopub.execute_input":"2021-08-24T16:26:50.528596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}