{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **TF Similarity**","metadata":{}},{"cell_type":"markdown","source":"# Notebook goal","metadata":{}},{"cell_type":"markdown","source":"<p>Today we are releasing the first version of TensorFlow Similarity, a python package designed to make it easy and fast to train similarity models using TensorFlow. This notebook demonstrates how to use TensorFlow Similarity to train a SimilarityModel()</p>","metadata":{}},{"cell_type":"markdown","source":"# Introduction","metadata":{}},{"cell_type":"markdown","source":"<p>TensorFlow Similarity provides all the necessary components to make similarity training evaluation and querying intuitive and easy. In particular, as illustrated below, TensorFlow Similarity introduces the SimilarityModel(), a new Keras model that natively supports embedding indexing and querying. This allows you to perform end-to-end training and evaluation quickly and efficiently.</p>","metadata":{}},{"cell_type":"markdown","source":"# Methodology","metadata":{}},{"cell_type":"markdown","source":"<p><ul><li>This notebook demonstrates how can we use Tensorflow Similarity to classify whales & dolphins.</li><li>This notebook is inspired by the blog published by tensorflow, you can read the blog through <a href=\"https://blog.tensorflow.org/2021/09/introducing-tensorflow-similarity.html\">here</a></li><li>This notebook uses codes from Tensorflow Similarity tutorial on <a href=\"https://github.com/tensorflow/similarity/blob/master/examples/supervised_hello_world.ipynb\">Github</a> and also the notebook <a href=\"https://www.kaggle.com/nicapotato/keras-efficientnet#Preparation-for-modeling\">Keras EfficientNet</a> from Humpback Whale Identification Challenge</li></ul></p>","metadata":{}},{"cell_type":"markdown","source":"# Loading Libraries & Packages","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport datetime\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score\nimport tensorflow as tf\nfrom tensorflow.keras import models, layers\nfrom tensorflow.keras.preprocessing import image\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau\nfrom tensorflow.keras.optimizers import Adam\n\nimport gc\nimport os\nfrom tabulate import tabulate\n\n# INFO messages are not printed.\n# This must be run before loading other modules.\nos.environ[\"TF_CPP_MIN_LOG_LEVEL\"] = \"1\"\n\n# ignoring warnings\nimport warnings\nwarnings.simplefilter(\"ignore\")\n\nimport os, cv2, json\nfrom PIL import Image\nfrom random import randint","metadata":{"execution":{"iopub.status.busy":"2022-02-05T13:58:41.982010Z","iopub.execute_input":"2022-02-05T13:58:41.982818Z","iopub.status.idle":"2022-02-05T13:58:48.091501Z","shell.execute_reply.started":"2022-02-05T13:58:41.982730Z","shell.execute_reply":"2022-02-05T13:58:48.090712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# install TF similarity if needed\ntry:\n    import tensorflow_similarity as tfsim  # main package\nexcept ModuleNotFoundError:\n    !pip install tensorflow_similarity\n    import tensorflow_similarity as tfsim","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-05T13:58:48.094883Z","iopub.execute_input":"2022-02-05T13:58:48.095500Z","iopub.status.idle":"2022-02-05T13:59:12.587593Z","shell.execute_reply.started":"2022-02-05T13:58:48.095467Z","shell.execute_reply":"2022-02-05T13:59:12.586854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tfsim.utils.tf_cap_memory()","metadata":{"execution":{"iopub.status.busy":"2022-02-05T13:59:12.589230Z","iopub.execute_input":"2022-02-05T13:59:12.589476Z","iopub.status.idle":"2022-02-05T13:59:12.734981Z","shell.execute_reply.started":"2022-02-05T13:59:12.589436Z","shell.execute_reply":"2022-02-05T13:59:12.733401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Clear out any old model state.\ngc.collect()\ntf.keras.backend.clear_session()","metadata":{"execution":{"iopub.status.busy":"2022-02-05T13:59:12.740668Z","iopub.execute_input":"2022-02-05T13:59:12.741385Z","iopub.status.idle":"2022-02-05T13:59:13.082131Z","shell.execute_reply.started":"2022-02-05T13:59:12.741347Z","shell.execute_reply":"2022-02-05T13:59:13.081350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"TensorFlow:\", tf.__version__)\nprint(\"TensorFlow Similarity\", tfsim.__version__)","metadata":{"execution":{"iopub.status.busy":"2022-02-05T13:59:13.083846Z","iopub.execute_input":"2022-02-05T13:59:13.084409Z","iopub.status.idle":"2022-02-05T13:59:13.091002Z","shell.execute_reply.started":"2022-02-05T13:59:13.084368Z","shell.execute_reply":"2022-02-05T13:59:13.089992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset","metadata":{}},{"cell_type":"code","source":"# File Parameters\nWORK_DIR = \"../input/happy-whale-and-dolphin\"\nlabel_col = \"individual_id\"\nimg_col = \"image\"\ntrain_folder = \"train_images\"\ntest_folder = \"test_images\"\n\nos.listdir(WORK_DIR)","metadata":{"execution":{"iopub.status.busy":"2022-02-05T13:59:13.092917Z","iopub.execute_input":"2022-02-05T13:59:13.093588Z","iopub.status.idle":"2022-02-05T13:59:13.112065Z","shell.execute_reply.started":"2022-02-05T13:59:13.093538Z","shell.execute_reply":"2022-02-05T13:59:13.111352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Train images: %d' %len(os.listdir(\n    os.path.join(WORK_DIR, train_folder))))","metadata":{"execution":{"iopub.status.busy":"2022-02-05T13:59:13.115859Z","iopub.execute_input":"2022-02-05T13:59:13.116387Z","iopub.status.idle":"2022-02-05T13:59:13.852205Z","shell.execute_reply.started":"2022-02-05T13:59:13.116351Z","shell.execute_reply":"2022-02-05T13:59:13.851351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels = pd.read_csv(os.path.join(WORK_DIR, \"train.csv\"))\nlabel_names = train_labels[label_col].value_counts().index\nlabel_map = {name:i for (i,name) in enumerate(label_names)}\ninv_label_map = {v: k for k, v in label_map.items()}\n\ntrain_labels['label_name'] = train_labels[label_col].copy()\ntrain_labels[label_col] = train_labels[label_col].map(label_map)\ndisplay(train_labels.head())","metadata":{"execution":{"iopub.status.busy":"2022-02-05T13:59:13.855261Z","iopub.execute_input":"2022-02-05T13:59:13.855555Z","iopub.status.idle":"2022-02-05T13:59:14.010447Z","shell.execute_reply.started":"2022-02-05T13:59:13.855518Z","shell.execute_reply":"2022-02-05T13:59:14.009520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-05T13:59:14.012184Z","iopub.execute_input":"2022-02-05T13:59:14.012484Z","iopub.status.idle":"2022-02-05T13:59:14.025391Z","shell.execute_reply.started":"2022-02-05T13:59:14.012442Z","shell.execute_reply":"2022-02-05T13:59:14.024311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Notebook Configuration","metadata":{}},{"cell_type":"code","source":"# Main parameters\nBATCH_SIZE = 32\nSTEPS_PER_EPOCH = len(train_labels)*0.8 / BATCH_SIZE\nVALIDATION_STEPS = len(train_labels)*0.2 / BATCH_SIZE\nEPOCHS = 4\nTARGET_SIZE = 512","metadata":{"execution":{"iopub.status.busy":"2022-02-05T13:59:14.030353Z","iopub.execute_input":"2022-02-05T13:59:14.030675Z","iopub.status.idle":"2022-02-05T13:59:14.036496Z","shell.execute_reply.started":"2022-02-05T13:59:14.030631Z","shell.execute_reply":"2022-02-05T13:59:14.035341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Generation","metadata":{}},{"cell_type":"code","source":"# Traning data\ntrain_datagen = ImageDataGenerator(validation_split = 0.2,\n                                     preprocessing_function = None,\n                                     rotation_range = 45,\n                                     zoom_range = 0.2,\n                                     horizontal_flip = True,\n                                     vertical_flip = True,\n                                     fill_mode = 'nearest',\n                                     shear_range = 0.1,\n                                     height_shift_range = 0.1,\n                                     width_shift_range = 0.1)\n\n\ntrain_generator = train_datagen.flow_from_dataframe(train_labels,\n                         directory = os.path.join(WORK_DIR, train_folder),\n                         subset = \"training\",\n                         x_col = img_col,\n                         y_col = label_col,\n                         color_mode='grayscale',\n                         target_size = (TARGET_SIZE, TARGET_SIZE),\n                         batch_size = 8,\n                         class_mode = \"raw\")\n\n# Validation Data\nvalidation_datagen = ImageDataGenerator(validation_split = 0.2)\n\n\nvalidation_generator = validation_datagen.flow_from_dataframe(train_labels,\n                         directory = os.path.join(WORK_DIR, train_folder),\n                         subset = \"validation\",\n                         color_mode='grayscale',\n                         x_col = img_col,\n                         y_col = label_col,\n                         target_size = (TARGET_SIZE, TARGET_SIZE),\n                         batch_size = 8,\n                         class_mode = \"raw\")","metadata":{"execution":{"iopub.status.busy":"2022-02-05T13:59:14.038442Z","iopub.execute_input":"2022-02-05T13:59:14.039071Z","iopub.status.idle":"2022-02-05T14:00:20.006877Z","shell.execute_reply.started":"2022-02-05T13:59:14.038919Z","shell.execute_reply":"2022-02-05T14:00:20.006162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model definition","metadata":{}},{"cell_type":"markdown","source":"<p><b>SimilarityModel()</b> models extend <b>tensorflow.keras.model</b>. Model with additional features and functionality that allow you to index and search for similar looking examples.\n\nAs visible in the model definition below, similarity models output a 64 dimensional float embedding using the <b>MetricEmbedding()</b> layers. This layer is a Dense layer with L2 normalization. Thanks to the loss, the model learns to minimize the distance between similar examples and maximize the distance between dissimilar examples. As a result, the distance between examples in the embedding space is meaningful; the smaller the distance the more similar the examples are.\n\nBeing able to use a distance as a meaningful proxy for how similar two examples are, is what enables the fast ANN (aproximate nearest neighbor) search. Using a sub-linear ANN search instead of a standard quadratic NN search is what allows deep similarity search to scale to millions of items. </p>","metadata":{}},{"cell_type":"code","source":"def get_model():\n    inputs = tf.keras.layers.Input(shape=(TARGET_SIZE, TARGET_SIZE, 1))\n    x = tf.keras.layers.experimental.preprocessing.Rescaling(1 / 255)(inputs)\n    x = tf.keras.layers.Conv2D(32, 3, activation=\"relu\")(x)\n    x = tf.keras.layers.Conv2D(32, 3, activation=\"relu\")(x)\n    x = tf.keras.layers.MaxPool2D()(x)\n    x = tf.keras.layers.Conv2D(64, 3, activation=\"relu\")(x)\n    x = tf.keras.layers.Conv2D(64, 3, activation=\"relu\")(x)\n    x = tf.keras.layers.Flatten()(x)\n    # smaller embeddings will have faster lookup times while a larger embedding will improve the accuracy up to a point.\n    outputs = tfsim.layers.MetricEmbedding(64)(x)\n    return tfsim.models.SimilarityModel(inputs, outputs)\n\n\nmodel = get_model()\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2022-02-05T14:00:20.008019Z","iopub.execute_input":"2022-02-05T14:00:20.008849Z","iopub.status.idle":"2022-02-05T14:00:22.087514Z","shell.execute_reply.started":"2022-02-05T14:00:20.008809Z","shell.execute_reply":"2022-02-05T14:00:22.086778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loss definition","metadata":{}},{"cell_type":"markdown","source":"<p>Overall what makes Metric losses different from tradional losses is that:\n\n**They expect different inputs**. Instead of having the prediction equal the true values, they expect embeddings as **y_preds** and the id (as an int32) of the class as **y_true**.\n\n**They require a distance**. You need to specify which distance function to use to compute the distance between embeddings. cosine is usually a great starting point and the default.\n\nIn this example we are using the **MultiSimilarityLoss()**. This loss takes a weighted combination of all valid positive and negative pairs, making it one of the best loss that you can use for similarity training.</p>","metadata":{}},{"cell_type":"code","source":"distance = \"cosine\"  # @param [\"cosine\", \"L2\", \"L1\"]{allow-input: false}\nloss = tfsim.losses.MultiSimilarityLoss(distance=distance)","metadata":{"execution":{"iopub.status.busy":"2022-02-05T14:00:22.088642Z","iopub.execute_input":"2022-02-05T14:00:22.088890Z","iopub.status.idle":"2022-02-05T14:00:22.095809Z","shell.execute_reply.started":"2022-02-05T14:00:22.088852Z","shell.execute_reply":"2022-02-05T14:00:22.095061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Compilation","metadata":{}},{"cell_type":"markdown","source":"<p>Tensorflow similarity use an extended <b>compile()</b> method that allows you to optionally specify distance_metrics (metrics that are computed over the distance between the embeddings), and the distance to use for the indexer.\n\nBy default the **compile()** method tries to infer what type of distance you are using by looking at the first loss specified. If you use multiple losses, and the distance loss is not the first one, then you need to specify the distance function used as distance= parameter in the compile function.</p>","metadata":{}},{"cell_type":"code","source":"LR = 0.001  # @param {type:\"number\"}\nmodel.compile(optimizer=tf.keras.optimizers.Adam(LR), loss=loss)","metadata":{"execution":{"iopub.status.busy":"2022-02-05T14:00:22.098813Z","iopub.execute_input":"2022-02-05T14:00:22.099072Z","iopub.status.idle":"2022-02-05T14:00:22.123375Z","shell.execute_reply.started":"2022-02-05T14:00:22.099042Z","shell.execute_reply":"2022-02-05T14:00:22.122684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{}},{"cell_type":"markdown","source":"<p>Similarity models are trained like normal models.</p>","metadata":{}},{"cell_type":"code","source":"EPOCHS = 10  # @param {type:\"integer\"}\nhistory = model.fit(train_generator,\n    steps_per_epoch = STEPS_PER_EPOCH,\n    epochs = EPOCHS,\n    validation_data = validation_generator,\n    validation_steps = VALIDATION_STEPS)","metadata":{"execution":{"iopub.status.busy":"2022-02-05T14:00:22.124728Z","iopub.execute_input":"2022-02-05T14:00:22.125015Z","iopub.status.idle":"2022-02-05T17:51:51.650033Z","shell.execute_reply.started":"2022-02-05T14:00:22.124975Z","shell.execute_reply":"2022-02-05T17:51:51.647672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plotting","metadata":{}},{"cell_type":"code","source":"plt.plot(history.history[\"loss\"])\nplt.plot(history.history[\"val_loss\"])\nplt.legend([\"loss\", \"val_loss\"])\nplt.title(f\"Loss: {loss.name} - LR: {LR}\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-05T17:51:51.654209Z","iopub.execute_input":"2022-02-05T17:51:51.656896Z","iopub.status.idle":"2022-02-05T17:51:51.896437Z","shell.execute_reply.started":"2022-02-05T17:51:51.656859Z","shell.execute_reply":"2022-02-05T17:51:51.895767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction","metadata":{}},{"cell_type":"code","source":"ss = pd.read_csv(os.path.join(WORK_DIR, \"sample_submission.csv\"))\nss","metadata":{"execution":{"iopub.status.busy":"2022-02-05T17:51:51.899784Z","iopub.execute_input":"2022-02-05T17:51:51.900114Z","iopub.status.idle":"2022-02-05T17:51:51.982185Z","shell.execute_reply.started":"2022-02-05T17:51:51.900086Z","shell.execute_reply":"2022-02-05T17:51:51.981493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds = []\ntop_n = 5\nfor image_id in ss[img_col]:\n    image = Image.open(os.path.join(WORK_DIR, test_folder, image_id)).convert('L')\n    image = image.resize((TARGET_SIZE, TARGET_SIZE))\n    image = np.expand_dims(image, axis = 0)\n    arr = model.predict(image)[0].argsort()[-top_n:][::-1]\n    p = \" \".join(np.vectorize(inv_label_map.get)(arr))\n    preds.append(p)\nss[label_col] = preds\nss","metadata":{"execution":{"iopub.status.busy":"2022-02-05T17:51:51.983240Z","iopub.execute_input":"2022-02-05T17:51:51.983643Z","iopub.status.idle":"2022-02-05T19:09:44.432622Z","shell.execute_reply.started":"2022-02-05T17:51:51.983604Z","shell.execute_reply":"2022-02-05T19:09:44.431857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ss.drop(['predictions'], axis = 1)","metadata":{"execution":{"iopub.status.busy":"2022-02-05T19:09:44.433993Z","iopub.execute_input":"2022-02-05T19:09:44.434247Z","iopub.status.idle":"2022-02-05T19:09:44.457483Z","shell.execute_reply.started":"2022-02-05T19:09:44.434213Z","shell.execute_reply":"2022-02-05T19:09:44.456785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ss.rename(columns = {'individual_id':'predictions'}, inplace = True)","metadata":{"execution":{"iopub.status.busy":"2022-02-05T19:09:44.458706Z","iopub.execute_input":"2022-02-05T19:09:44.458972Z","iopub.status.idle":"2022-02-05T19:09:44.465615Z","shell.execute_reply.started":"2022-02-05T19:09:44.458922Z","shell.execute_reply":"2022-02-05T19:09:44.464978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv(os.path.join(WORK_DIR, \"sample_submission.csv\"))","metadata":{"execution":{"iopub.status.busy":"2022-02-05T19:20:39.945911Z","iopub.execute_input":"2022-02-05T19:20:39.946540Z","iopub.status.idle":"2022-02-05T19:20:40.005268Z","shell.execute_reply.started":"2022-02-05T19:20:39.946500Z","shell.execute_reply":"2022-02-05T19:20:40.004578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub[\"predictions\"]=ss.iloc[:,2]","metadata":{"execution":{"iopub.status.busy":"2022-02-05T19:23:01.281148Z","iopub.execute_input":"2022-02-05T19:23:01.281686Z","iopub.status.idle":"2022-02-05T19:23:01.290191Z","shell.execute_reply.started":"2022-02-05T19:23:01.281648Z","shell.execute_reply":"2022-02-05T19:23:01.289502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub","metadata":{"execution":{"iopub.status.busy":"2022-02-05T19:23:28.042458Z","iopub.execute_input":"2022-02-05T19:23:28.042732Z","iopub.status.idle":"2022-02-05T19:23:28.055013Z","shell.execute_reply.started":"2022-02-05T19:23:28.042699Z","shell.execute_reply":"2022-02-05T19:23:28.054334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub.to_csv('submission_whale_and_dolphin.csv', index = False)\nprint(ss.shape)","metadata":{"execution":{"iopub.status.busy":"2022-02-05T19:24:53.136236Z","iopub.execute_input":"2022-02-05T19:24:53.136737Z","iopub.status.idle":"2022-02-05T19:24:53.263388Z","shell.execute_reply.started":"2022-02-05T19:24:53.136696Z","shell.execute_reply":"2022-02-05T19:24:53.262616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reference","metadata":{}},{"cell_type":"markdown","source":"As mentioned at the beginning of the notebook, this notebook is inspired from the example on<a href= https://blog.tensorflow.org/2021/09/introducing-tensorflow-similarity.html> TF Similarity</a>, do check it out.","metadata":{}},{"cell_type":"markdown","source":"<h2> If you found it interesting & helpful, then <b>Upvote</b> the notebook!!</h2>","metadata":{}}]}