{"cells":[{"metadata":{},"cell_type":"markdown","source":"## A brief outline\n\nIn this notebook, we show how to search for duplicated images using image embeddings and the DBSCAN clustering algorithm. The embeddings are extracted with the help of a pre-trained EfficientNet model. We find 5 pairs of images in the Cassava train set which are potential duplicates. Visual inspection of the images in each pair reveals that all of them are true duplicates of one another, so the precission of our method on the training dataset is 100% (the recall is unknown). We find that one pair of duplicates contains images which are labeled differently which is a clear demonstration of the presence of noissy labels in the competition dataset.\n\nThis method can also be applied to search for the overlap between the current competition dataset and the [2019 data](https://www.kaggle.com/c/cassava-disease/data) (we are leaving this as an excercise for an interested reader). This is very important to keep all duplicates under control when doing cross-validation because we do not want to validate our model on the same data that the model was shown during the training phase. Identifying duplicated images will help us to enforce the uniqueness of the images in the training and the validation sets.\n\nThe discussion topic is here: [Searching for duplicated images with DBSCAN](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/210723)."},{"metadata":{"id":"YmXVe_bBuglC"},"cell_type":"markdown","source":"## Loading libraries"},{"metadata":{"id":"8T0OVkF6uglD","executionInfo":{"status":"ok","timestamp":1610407441570,"user_tz":360,"elapsed":21684,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"79d0fa40-12aa-4287-ff0b-6386c5b37899","trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"id":"wkB3rarVuglH","executionInfo":{"status":"ok","timestamp":1610407444328,"user_tz":360,"elapsed":24433,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"9e451c52-8718-49c9-dd05-350da5ba032a","trusted":true},"cell_type":"code","source":"import re\nimport gc  \nimport os\nimport math\nimport json\nimport random\nimport warnings\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\nimport tensorflow as tf\nfrom pathlib import Path\nfrom sklearn.cluster import DBSCAN\nimport efficientnet.tfkeras as efn\nfrom collections import defaultdict\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\n\nprint(\"Tensorflow version \" + tf.__version__)\nAUTO = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{"id":"iwZD-He0uglL"},"cell_type":"markdown","source":"## Loading data"},{"metadata":{"id":"9Yb5aY0cuglL","executionInfo":{"status":"ok","timestamp":1610407444329,"user_tz":360,"elapsed":24424,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"def seed_everything(seed):\n    random.seed(seed)\n    np.random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    os.environ['TF_DETERMINISTIC_OPS'] = '1'\n    tf.random.set_seed(seed)","execution_count":null,"outputs":[]},{"metadata":{"id":"mCiMLPNDuglP","executionInfo":{"status":"ok","timestamp":1610407445681,"user_tz":360,"elapsed":25770,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"09d8a313-ab19-4db0-f73a-f44faf762b8b","trusted":true},"cell_type":"code","source":"MODEL='EffNB4'\n\nHEIGHT=512\nWIDTH=512\nIMAGE_SIZE = [HEIGHT, WIDTH] # At this size, a GPU will run out of memory. Use the TPU.\n                             # For GPU training, please select 224 x 224 px image size.    \nDESCRIPTION='embeddings'\nNAME=f'{MODEL}_{HEIGHT}x{WIDTH}_{DESCRIPTION}'\nSEED=311\nBATCH_SIZE_FACTOR = 32\n\nmodel_selector={'EffNB0': efn.EfficientNetB0,\n                'EffNB1': efn.EfficientNetB1,\n                'EffNB2': efn.EfficientNetB2,\n                'EffNB3': efn.EfficientNetB3,\n                'EffNB4': efn.EfficientNetB4,\n                'EffNB5': efn.EfficientNetB5,\n                'EffNB6': efn.EfficientNetB6,\n                'EffNB7': efn.EfficientNetB7,\n               }\n\nPATH=Path('/kaggle/input/cassava-leaf-disease-classification/')\ntrain=pd.read_csv(PATH/'train.csv')\n              \nseed_everything(SEED)\nwarnings.filterwarnings('ignore')\nprint(f\"Model name: {NAME}.\")","execution_count":null,"outputs":[]},{"metadata":{"id":"v971oHGKNpLH","executionInfo":{"status":"ok","timestamp":1610407445682,"user_tz":360,"elapsed":25762,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"17d11085-899b-448d-f102-57f05c1613fd","trusted":true},"cell_type":"code","source":"print(f\"The shape of the training set is {train.shape}.\")\nprint(f\"The columns in `train`:\\n {list(train.columns)}.\\n\")","execution_count":null,"outputs":[]},{"metadata":{"id":"mJvUZmrtNxwH","executionInfo":{"status":"ok","timestamp":1610407445683,"user_tz":360,"elapsed":25753,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"3fda4e56-607a-495c-d74e-de0562db9f09","trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"71YNwselMB2Q","executionInfo":{"status":"ok","timestamp":1610407445684,"user_tz":360,"elapsed":25743,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"35eea61c-c647-45be-848f-f72fffaf68ad","trusted":true},"cell_type":"code","source":"labels=np.sort(train['label'].unique())\nlabels","execution_count":null,"outputs":[]},{"metadata":{"id":"sSiaLFqkMB2Q"},"cell_type":"markdown","source":"Now, let's download the mapping between the label numbers and the disease names."},{"metadata":{"id":"IU81aPdQMB2Q","executionInfo":{"status":"ok","timestamp":1610407445937,"user_tz":360,"elapsed":25985,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"a1d6c6df-695d-4b6a-8673-0b828906d12c","trusted":true},"cell_type":"code","source":"with open(os.path.join(PATH, \"label_num_to_disease_map.json\")) as file:\n    label_mapping = json.loads(file.read())\n    \nlabel_mapping","execution_count":null,"outputs":[]},{"metadata":{"id":"nfBzkQgrMB2R"},"cell_type":"markdown","source":"Let's remove the abbreviations at the end of each decesase name and turn the keys of the dictionary into integers."},{"metadata":{"id":"Elvz4DpBMB2R","executionInfo":{"status":"ok","timestamp":1610407445938,"user_tz":360,"elapsed":25977,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"label_mapping={0: 'Cassava Bacterial Blight',\n               1: 'Cassava Brown Streak Disease',\n               2: 'Cassava Green Mottle',\n               3: 'Cassava Mosaic Disease',\n               4: 'Healthy',\n              }","execution_count":null,"outputs":[]},{"metadata":{"id":"YoH1DdvCMB2R"},"cell_type":"markdown","source":"Save the abbreviated class names as follows:"},{"metadata":{"id":"9VpETkUDMB2S","executionInfo":{"status":"ok","timestamp":1610407445938,"user_tz":360,"elapsed":25970,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"SHORT_CLASSES=['CBB', 'CBSD', 'CGM', 'CMD', 'Healthy']","execution_count":null,"outputs":[]},{"metadata":{"id":"ztNaH9raMB2S","executionInfo":{"status":"ok","timestamp":1610407446043,"user_tz":360,"elapsed":26068,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"94f65929-8308-4cd3-eee9-78e74a179dee","trusted":true},"cell_type":"code","source":"train['desease'] = train['label'].map(label_mapping)\ntrain['desease'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nCLASSES = [label_mapping[i] for i in range(len(label_mapping))]\nCLASSES","execution_count":null,"outputs":[]},{"metadata":{"id":"zwrju15xuglg"},"cell_type":"markdown","source":"## TPU or GPU detection"},{"metadata":{"id":"L1cjq7Jkuglh","executionInfo":{"status":"ok","timestamp":1610407461715,"user_tz":360,"elapsed":41731,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"8137850e-1d2e-4283-9e39-012deedc40e1","trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment \n    # variable is set. On Kaggle this is always the case.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n    \nBATCH_SIZE = BATCH_SIZE_FACTOR * strategy.num_replicas_in_sync\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\nprint(\"BATCH SIZE: \", BATCH_SIZE)","execution_count":null,"outputs":[]},{"metadata":{"id":"4JHxDRriuglk"},"cell_type":"markdown","source":"## Data access"},{"metadata":{"id":"Q-C8b_Tnuglo","executionInfo":{"status":"ok","timestamp":1610407461825,"user_tz":360,"elapsed":41823,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"582ace4d-e242-4e7c-d099-84643239bfcb","trusted":true},"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path('cassava-leaf-disease-classification')\nALL_TFRECS=np.array(tf.io.gfile.glob(GCS_PATH + '/train_tfrecords/*.tfrec'))\n\nprint(GCS_PATH)","execution_count":null,"outputs":[]},{"metadata":{"id":"hYHvsS7yMB2U","executionInfo":{"status":"ok","timestamp":1610407461826,"user_tz":360,"elapsed":41814,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"46a8e88e-ded2-4ecd-e943-98e7bb1a403f","trusted":true},"cell_type":"code","source":"ALL_TFRECS","execution_count":null,"outputs":[]},{"metadata":{"id":"XfuTfS_7ugmI"},"cell_type":"markdown","source":"## Configuration"},{"metadata":{"id":"AU7BuL2Pugmo","executionInfo":{"status":"ok","timestamp":1610407461827,"user_tz":360,"elapsed":41796,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"def count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, \n    # i.e. test10-687.tfrec = 687 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    \n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"id":"jkLeA7eNPadb","executionInfo":{"status":"ok","timestamp":1610407461827,"user_tz":360,"elapsed":41789,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"4a24e939-384f-46c3-9013-1d96fd0786b8","trusted":true},"cell_type":"code","source":"%%time\n\nprint(f\"The total number of images is {count_data_items(ALL_TFRECS)}.\")","execution_count":null,"outputs":[]},{"metadata":{"id":"j4Kua-WHugmt"},"cell_type":"markdown","source":"## Datasets utility functions\n\nBelow are the standard functions that we will be using to read and process the data from the `.tfrec` files. "},{"metadata":{"id":"em6PEXrtMB2V","executionInfo":{"status":"ok","timestamp":1610407566625,"user_tz":360,"elapsed":296,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"def decode_image(image_data):\n    \"\"\"\n        1. Decode a JPEG-encoded image to a uint8 tensor.\n        2. Cast the tensor to float and normalizes (range between 0 and 1).\n        3. Reshape the image to the expected shape.\n    \"\"\"\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    # we normalize our inputs by subtracting ImageNet mean of 0.449 \n    # and dividing by ImageNet standard deviation of 0.226. \n    # We have to do it because we won't be fine-tuning our model.\n    image = ((tf.cast(image, tf.float32) / 255.0) - 0.449) / 0.226\n    image = tf.reshape(image, [HEIGHT, WIDTH, 3])\n    return image","execution_count":null,"outputs":[]},{"metadata":{"id":"89iEeMruMB2W","executionInfo":{"status":"ok","timestamp":1610407566738,"user_tz":360,"elapsed":398,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"def read_tfrecord(example, labeled=True):\n    \"\"\"\n        1. Parse data based on the 'TFREC_FORMAT' map.\n        2. Decode image.\n        3. If 'labeled' returns (image, label) if not (image, name).\n    \"\"\"\n    if labeled:\n        TFREC_FORMAT = {\n            'image': tf.io.FixedLenFeature([], tf.string), \n            'target': tf.io.FixedLenFeature([], tf.int64), \n        }\n    else:\n        TFREC_FORMAT = {\n            'image': tf.io.FixedLenFeature([], tf.string), \n            'image_name': tf.io.FixedLenFeature([], tf.string), \n        }\n    example = tf.io.parse_single_example(example, TFREC_FORMAT)\n    image = decode_image(example['image'])\n    if labeled:\n        label_or_name = tf.cast(example['target'], tf.int32)\n    else:\n        label_or_name =  example['image_name']\n    return image, label_or_name","execution_count":null,"outputs":[]},{"metadata":{"id":"STtYQMrhMB2W","executionInfo":{"status":"ok","timestamp":1610407566738,"user_tz":360,"elapsed":391,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n    \"\"\"\n        Create a Tensorflow dataset from TFRecords.\n    \"\"\"\n    ignore_order = tf.data.Options()\n    if not ordered:\n        # disable order, increase speed. Makes sense to do\n        # if we are going to shuffle the data anyway\n        ignore_order.experimental_deterministic = False\n    \n    # automatically interleaves reads from multiple files\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO)\n    # if ordered=False uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.with_options(ignore_order)\n    # returns a dataset of (image, label) pairs if labeled=True \n    # or (image, id) pairs if labeled=False\n    dataset = dataset.map(lambda x: read_tfrecord(x, labeled=labeled), num_parallel_calls=AUTO)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"id":"nIpWozy9MB2W"},"cell_type":"markdown","source":"Let's take a quick look at one example of data:"},{"metadata":{"id":"u5QMwJq7ugm5","executionInfo":{"status":"ok","timestamp":1610407578101,"user_tz":360,"elapsed":11740,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"2cd4400a-cd1d-4216-82ae-fab58bcf12bb","trusted":true},"cell_type":"code","source":"%%time\n\ndataset = load_dataset(ALL_TFRECS)\n\nprint(\"Example of the training data:\")\nfor image, label in dataset.take(1):\n    print(\"The image batch size:\", image.numpy().shape)\n    print(\"Label:\", label.numpy())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We won't be training our model, so our `get_dataset` function is very simple."},{"metadata":{"id":"YznJIwtCwN9H","executionInfo":{"status":"ok","timestamp":1610407578103,"user_tz":360,"elapsed":11733,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"def get_dataset(FILENAMES):\n    \"\"\"\n        Return a Tensorflow dataset ready for training or inference.\n    \"\"\"     \n    dataset = load_dataset(FILENAMES, labeled=False, ordered=True)\n    dataset = dataset.cache()\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO)\n    \n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"id":"bXM39S6zugnG"},"cell_type":"markdown","source":"## Visualization utilities\n\nAnd here are the standard visualization utilities that we will use to visualize the dataset."},{"metadata":{"id":"E656VW7cugnH","executionInfo":{"status":"ok","timestamp":1610407578104,"user_tz":360,"elapsed":11727,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"# numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)","execution_count":null,"outputs":[]},{"metadata":{"id":"uq3v8VzlugnJ","executionInfo":{"status":"ok","timestamp":1610407578105,"user_tz":360,"elapsed":11722,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"def batch_to_numpy_images_and_labels(databatch):\n    images, labels = databatch\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n\n    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        # If no labels, only image IDs, return None for labels (this is the case for test data)\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n\n    return numpy_images, numpy_labels","execution_count":null,"outputs":[]},{"metadata":{"id":"o8Ry9wnTugnN","executionInfo":{"status":"ok","timestamp":1610407578106,"user_tz":360,"elapsed":11716,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"def title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return SHORT_CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(SHORT_CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" \n                                if not correct else '', \n                                SHORT_CLASSES[correct_label] if not correct else ''), correct","execution_count":null,"outputs":[]},{"metadata":{"id":"sX6JyMKIugnP","executionInfo":{"status":"ok","timestamp":1610407578106,"user_tz":360,"elapsed":11709,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"def display_one_image(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), \n                  color='red' if red else 'black', fontdict={'verticalalignment':'center'}, \n                  pad=int(titlesize/1.5)\n                 )\n    return (subplot[0], subplot[1], subplot[2]+1)","execution_count":null,"outputs":[]},{"metadata":{"id":"gLnRL6T8ugnS","executionInfo":{"status":"ok","timestamp":1610407578107,"user_tz":360,"elapsed":11703,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"def display_batch_of_images(databatch, show_class_names=True, predictions=None):\n    \"\"\" This will work with:\n        display_batch_of_images(images)\n        display_batch_of_images(images, predictions)\n        display_batch_of_images((images, labels))\n        display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if not any(l is None for l in labels):\n        labels = np.argmax(labels, axis=-1)\n        \n    # auto-squaring: this will drop data that does  \n    # not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        if show_class_names:\n            title = '' if label is None else SHORT_CLASSES[label]\n        else:\n            title = ''\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        # magic formula tested to work from 1x1 to 10x10 images\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3\n        subplot = display_one_image(image, title, subplot, \n                                     not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"6oAIv35XYTtI"},"cell_type":"markdown","source":"## Dataset visualizations"},{"metadata":{"id":"SdZfGpvtMB2c","executionInfo":{"status":"ok","timestamp":1610407578173,"user_tz":360,"elapsed":11763,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"# Peek at training data\n\ndataset = get_dataset(ALL_TFRECS)\ndataset = dataset.unbatch().batch(20)\nimg_batch = iter(dataset)","execution_count":null,"outputs":[]},{"metadata":{"id":"5al8HL3xMB2c","executionInfo":{"status":"ok","timestamp":1610407582292,"user_tz":360,"elapsed":15875,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"22eeb68b-4b66-46cc-d303-e200f4be6d6c","trusted":true},"cell_type":"code","source":"# run this cell again for next set of images\ndisplay_batch_of_images(next(img_batch))","execution_count":null,"outputs":[]},{"metadata":{"id":"fcVmp7CsMB2c","executionInfo":{"status":"ok","timestamp":1610407582293,"user_tz":360,"elapsed":15869,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"1028e29e-17ff-4229-92d5-e834f47a3b5b","trusted":true},"cell_type":"code","source":"del dataset, img_batch\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"id":"Dah04KImZA4t"},"cell_type":"markdown","source":"## Extract image features\n\nTo extract image embeddings, we follow the procedure suggested by Chris Deotte in [one of his Melanoma competition public kernels](https://www.kaggle.com/cdeotte/rapids-cuml-knn-find-duplicates). The idea is very simple -- we add a `GlobalAveragePooling2D` layer to a pre-trained EfficientNet model and then send every image through the resulting neural net making predictions. The output of the `GlobalAveragePooling2D` layer is a very long vector wiht thousands of components. The exact number of these components depends on the type of the EfficientNet model. This vector will be used to represent the corresponding image. In other words, we will be using it as an embedding for the image. "},{"metadata":{"id":"1xYz4W8ZRgif","executionInfo":{"status":"ok","timestamp":1610407582294,"user_tz":360,"elapsed":15863,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"# EXTRACT LAST LAYER OF EFFICIENT NET WITH GLOBAL AVERAGE POOLING\n\ndef build_model():\n    with strategy.scope():\n        pretrained_model = model_selector[MODEL](input_shape=(*IMAGE_SIZE, 3),\n                                                      weights='imagenet',\n                                                      include_top=False\n                                                      )\n        inp = tf.keras.layers.Input(shape=(*IMAGE_SIZE, 3))\n        x = pretrained_model(inp)\n        out = tf.keras.layers.GlobalAveragePooling2D()(x)\n        model = tf.keras.Model(inputs=inp, outputs=out)\n        return model","execution_count":null,"outputs":[]},{"metadata":{"id":"oAfM07yLgYF_","executionInfo":{"status":"ok","timestamp":1610407609472,"user_tz":360,"elapsed":43034,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"fc602fe6-6123-46f5-a54a-489ee4a1c66d","trusted":true},"cell_type":"code","source":"%%time\n\nmodel = build_model()","execution_count":null,"outputs":[]},{"metadata":{"id":"upLFgqU6dMSW","executionInfo":{"status":"ok","timestamp":1610407609473,"user_tz":360,"elapsed":43029,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"def retrieve_image(image, image_id_or_label):\n    return image","execution_count":null,"outputs":[]},{"metadata":{"id":"fNcwi1MYebya","executionInfo":{"status":"ok","timestamp":1610407609473,"user_tz":360,"elapsed":43019,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"trusted":true},"cell_type":"code","source":"def retrieve_id_or_label(image, image_id_or_label):\n    return image_id_or_label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Computing the embeddings and the corresponding image ID's."},{"metadata":{"id":"b0qL-NAAaM7Q","executionInfo":{"status":"ok","timestamp":1610407666918,"user_tz":360,"elapsed":100458,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"5de5225d-6883-4a3f-8b1e-dedfbeed706a","trusted":false},"cell_type":"code","source":"%%time\n\nds = get_dataset(ALL_TFRECS)\n\nds_imgs = ds.map(retrieve_image, num_parallel_calls=AUTO) \nds_ids = ds.map(retrieve_id_or_label, num_parallel_calls=AUTO).unbatch()\n\nembs = model.predict(ds_imgs,verbose=1)\n\nimage_ids=next(iter(ds_ids.batch(count_data_items(ALL_TFRECS)))).numpy().astype('U')","execution_count":null,"outputs":[]},{"metadata":{"id":"zfNEoTTQxwuI","executionInfo":{"status":"ok","timestamp":1610407733644,"user_tz":360,"elapsed":179,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"defb9550-f743-492d-95f4-084d9d86903f","trusted":false},"cell_type":"code","source":"print(f\"The shapes of the embeddings and image ID's are {embs.shape} and {image_ids.shape}, respectively.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Saving the embeddings and the image ID's."},{"metadata":{"id":"f-XSTuTDgTog","executionInfo":{"status":"ok","timestamp":1610407668333,"user_tz":360,"elapsed":101860,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"ed3f6bdd-4bad-4eb0-9c4d-876da77166fb","trusted":false},"cell_type":"code","source":"%%time\n\nnp.save(f'embeddings_train_{MODEL}_{HEIGHT}x{WIDTH}', embs)\nnp.save(f'image_ids_train_{MODEL}_{HEIGHT}x{WIDTH}', image_ids)","execution_count":null,"outputs":[]},{"metadata":{"id":"oXl6X4aXy-4p"},"cell_type":"markdown","source":"## Finding duplicates with DBSCAN\n\nTo search for duplicated images, we apply the method suggested by Alex Shonenkov in his Melanoma compentition [public kernenl](https://www.kaggle.com/shonenkov/dbscan-clustering-check-marking). The idea is to tune the parameters of the `DBSCAN` clustering algorithm to split the dataset into clusters of similar images. Then we will visually inspect the resulting clusters to see whether or not they contain true duplicates.\n\nIf you want to learn more about DBSCAN and how it works please refer to the following scikit-learn page: [DBSCAN](https://scikit-learn.org/stable/modules/clustering.html#dbscan)."},{"metadata":{"id":"jjSlej2uzobQ","executionInfo":{"status":"ok","timestamp":1610408933147,"user_tz":360,"elapsed":504685,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"285d516d-6fd5-4cb3-87d2-0bd8d2da9767","trusted":false},"cell_type":"code","source":"%%time\n\nclusters = defaultdict(list)\nfor image_name, cluster_id in zip(image_ids, DBSCAN(eps=3.0, min_samples=1, n_jobs=4).fit_predict(embs)):\n    clusters[cluster_id].append(image_name)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Find clusters with more than one element."},{"metadata":{"id":"uGdAdTWM2zDU","executionInfo":{"status":"ok","timestamp":1610409237852,"user_tz":360,"elapsed":216,"user":{"displayName":"Alexey Pronin","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GgFO04ue-bNrj83tCRld7Qz_P5Chn-2fMGkWtxLGQ=s64","userId":"07372911786440619972"}},"outputId":"b330fd76-0dda-41d8-f46f-d08ca8136619","trusted":false},"cell_type":"code","source":"potential_duplicates = np.array([c for c in clusters.values() if len(c)>1])\npotential_duplicates","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We found 5 pairs of potential duplicates. Let's plot them for visual inspection."},{"metadata":{"id":"WS32Tmdb9c-B","trusted":true},"cell_type":"code","source":"def plot_duplicates(dups, h_factor=7, v_factor=6, font_size=14):\n    n_dups_max=max([len(d) for d in dups])\n    n_rows=len(dups)\n\n    fig, ax = plt.subplots(n_rows, n_dups_max, figsize=(n_dups_max*h_factor, n_rows*v_factor))\n\n    for j, image_ids in enumerate(dups):\n        for i, image_id in enumerate(image_ids):\n            \n            PATH_IMG = PATH/'train_images'\n            label = str(train.loc[train['image_id'] == image_id, 'label'].values[0])\n                \n            image = Image.open(PATH_IMG/image_id)\n\n            ax[j][i].imshow(image)\n            ax[j][i].set_title(\"image_id: \" + image_id + \";  label: \" + label, \n                               fontsize=font_size)\n            ax[j][i].axis('off')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_duplicates(potential_duplicates)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Brief discussion of the result\n\nVisual inspection reveals that all of these pairs contain duplicated images. The method yields zero number of false positives, so it's *precision* $ = TP/(TP + FP) = 1$, where $TP$ and $FP$ stand for the true and false positives, respectively. (Of course, this is the precision on the competition training set -- I cannot guarantee that the method's performance will be the same on some arbitrary data.)\n\nAlso, note that the `['1562043567.jpg', '3551135685.jpg']` pair contains duplicated images carrying different labels. This is a great examples of the noissy labels that we have to deal with in this competition."},{"metadata":{},"cell_type":"markdown","source":"## Saving the duplicate filenames"},{"metadata":{"trusted":true},"cell_type":"code","source":"np.save('duplicates', potential_duplicates)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thank you for reading! Please kindly upvote this notebook if you find it helpful!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}