{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n
\n \"my_img\"/\n Brief summary🐋\n
\n \n We use fingerprints and facial recognition to identify people, but can we use similar approaches with animals? In this competition, you’ll develop a model to match individual whales and dolphins by unique—but often subtle—characteristics of their natural markings. You'll pay particular attention to dorsal fins and lateral body views in image sets from a multi-species dataset built by 28 research institutions. The best submissions will suggest photo-ID solutions that are fast and accurate.\n \n
\n
\n Competition goal:
\n Identify whales and dolphins by unique characteristics\n

\n Evaluation:
\n Submissions are evaluated according to the Mean Average Precision @ 5 (MAP@5):\n
$MAP@5 = \\frac{1}{U}\\displaystyle \\sum_{u=1}^{U}\\sum_{k=1}^{min(n, 5)}P(k)\\times rel(k)$
\n
\n where U is the number of images, P(k) is the precision at cutoff k, n is the number of predictions per image, and rel(k) is an indicator function equaling 1 if the item at rank k is a relevant (correct) label, zero otherwise.\n
\n \n Let's understand this better below👇 (Huge thanks to this notebook for understanding this evaluation metric)\n \n
","metadata":{}},{"cell_type":"code","source":"import numpy as np \n\ndef map_per_image(label, predictions):\n \"\"\"Computes the precision score of one image.\n Parameters\n ----------\n label : string\n The true label of the image\n predictions : list\n A list of predicted elements (order does matter, 5 predictions allowed per image)\n Returns\n -------\n score : double\n \"\"\" \n try:\n return 1 / (predictions[:5].index(label) + 1)\n except ValueError:\n return 0.0\ndef map_per_set(labels, predictions):\n \"\"\"Computes the average over multiple images.\n Parameters\n ----------\n labels : list\n A list of the true labels. (Only one true label per images allowed!)\n predictions : list of list\n A list of predicted elements (order does matter, 5 predictions allowed per image)\n Returns\n -------\n score : double\n \"\"\"\n return np.mean([map_per_image(l, p) for l,p in zip(labels, predictions)])\n\nassert map_per_image('A',['A', 'B', 'C', 'D', 'E']) == 1.0\nassert map_per_image('A', ['A', 'A', 'A', 'A', 'A']) == 1.0\nassert map_per_image('A', ['A', 'B', 'A', 'C', 'A']) == 1.0","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:12:26.052506Z","iopub.execute_input":"2022-03-08T11:12:26.053002Z","iopub.status.idle":"2022-03-08T11:12:26.080667Z","shell.execute_reply.started":"2022-03-08T11:12:26.052910Z","shell.execute_reply":"2022-03-08T11:12:26.080040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n
\n 1. Loading and understanding Data\n
\n
\n \n
\n
","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import os \nimport cv2\nimport numpy as np\nimport glob\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow import keras\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\ndef set_seed(SEED):\n np.random.seed(SEED)\n os.environ[\"PYTHONHASHSEED\"] = str(SEED)\n tf.random.set_seed(SEED)\n tf.keras.backend.clear_session()\n print(f'SEED {SEED} SET!')\n\n\nclass Config:\n seed = 42\n batch_size = 4\n epochs = 5\n img_size = 448\n learning_rate = 0.001\n n_fold:int = 5\n gpus:list = tf.config.list_physical_devices('GPU')\n\nset_seed(Config.seed)\n\nroot = '../input/happy-whale-and-dolphin'\ntrain_images = f'{root}/train_images'\ntest_images = f'{root}/test_images'\ntabular_path = f'{root}/train.csv'","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:15:55.983906Z","iopub.execute_input":"2022-03-08T11:15:55.984359Z","iopub.status.idle":"2022-03-08T11:15:55.997469Z","shell.execute_reply.started":"2022-03-08T11:15:55.984318Z","shell.execute_reply":"2022-03-08T11:15:55.996646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tabular_data = pd.read_csv(tabular_path)\ntabular_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:15:57.133626Z","iopub.execute_input":"2022-03-08T11:15:57.134353Z","iopub.status.idle":"2022-03-08T11:15:57.198686Z","shell.execute_reply.started":"2022-03-08T11:15:57.134303Z","shell.execute_reply":"2022-03-08T11:15:57.197889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"__From the sample above of the first 5 rows of the data__🐬:\n> * We have image name and extension, species to which the dolphin or whale belongs to and the individual ids of the whales and dolphins.\n> *The individual ids will be used as the labels for the similarity search to identify whales and dolphins based on their unique yet subtle markings.\n
\nLet's explore more😄\n
","metadata":{}},{"cell_type":"code","source":"from IPython.display import display, Markdown, HTML\ncoolors = [\n (\"#A2D2FF\", \"#EBF5FF\"),\n (\"#FFAFCC\", \"#FFD6E5\"),\n (\"#55C123\", \"#B2EC98\"),\n (\"#00b4d8\", \"#caf0f8\"),\n (\"#C92CAC\", \"#E99BDB\"),\n (\"#FFB35C\", \"#FFE2C2\"),\n (\"#D4A373\",\"#F1E0D0\")\n]\n\ndisplay(HTML(\" \\\n
\\\n We're back from the Orient Express and the Nile, and this is what we found \\\n 🕵️ \\\n
\"))\n\nno_rows = tabular_data.shape[0]\ndisplay(Markdown(\"* There are __{}__ rows in the dataset\".format(no_rows)))\ndisplay(Markdown(\"In the dataset, there are {} species of whales and dolphins\"\n .format(tabular_data.species.nunique())))\ndisplay(Markdown(\"#### 🐋The classes are:🐬\"))\n\nall_classes = \"
\"\nfor i in tabular_data.species.unique():\n border, bg = coolors[np.random.randint(len(coolors))]\n whale_or_dolphin_class = i#(' '.join(i.split('_'))).capitalize()\n whale_or_dolphin_class = f\"🐋{whale_or_dolphin_class}\" if 'whale' in whale_or_dolphin_class.lower() else whale_or_dolphin_class\n whale_or_dolphin_class = f\"🐬{whale_or_dolphin_class}\" if 'dolphin' in whale_or_dolphin_class.lower() else whale_or_dolphin_class\n all_classes += \"
\\\n {}
\".format(bg, border, whale_or_dolphin_class)\nall_classes += \"
\"\ndisplay(HTML(all_classes))\ndisplay(Markdown(\"🖋️The awesome colors can be found on [Coolors](https://coolors.co/palettes/trending)\"))","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:15:59.493262Z","iopub.execute_input":"2022-03-08T11:15:59.493715Z","iopub.status.idle":"2022-03-08T11:15:59.524312Z","shell.execute_reply.started":"2022-03-08T11:15:59.493677Z","shell.execute_reply":"2022-03-08T11:15:59.523643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":">* There is a typo on the species bottlenose_dolpin which needs to be corrected to bottlenose_dolphin\n>* There is a typo on the species kiler_whale which needs to be corrected to killer_whale\n>* This reduces the total species to 28\n>* Also looking at the unknown species, we find out that:\n>\n>\n>>📓 Belugas are toothed whales, and are not part of the oceanic dolphin family. They are classified under the Monodontidae family, which only consists of two species: belugas and narwhals.\n>\n>\n>>📓 Globis seems to have been confused and is supposed to be globus, a latin for whales named \"pilot whales\" because their birth pods were believed to be \"piloted\" by a leader. Let us go further into looking at the images of the species to confirm this.","metadata":{"execution":{"iopub.status.busy":"2022-03-04T20:29:40.528024Z","iopub.execute_input":"2022-03-04T20:29:40.528315Z","iopub.status.idle":"2022-03-04T20:29:40.553059Z","shell.execute_reply.started":"2022-03-04T20:29:40.528284Z","shell.execute_reply":"2022-03-04T20:29:40.551726Z"}}},{"cell_type":"code","source":"tabular_data.species.replace(\n [\"bottlenose_dolpin\", \"kiler_whale\", \"beluga\"],\n [\"bottlenose_dolphin\",\"killer_whale\",\"beluga_whale\"], inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:03.193696Z","iopub.execute_input":"2022-03-08T11:16:03.194234Z","iopub.status.idle":"2022-03-08T11:16:03.211714Z","shell.execute_reply.started":"2022-03-08T11:16:03.194195Z","shell.execute_reply":"2022-03-08T11:16:03.211054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Let's add another column for file paths to be able to reference file paths for the ~globis~ __*or rather globus*__ species ","metadata":{}},{"cell_type":"code","source":"tabular_data['path'] = tabular_data.image.apply(lambda x: f'{train_images}/{x}')\nglobis = tabular_data.loc[tabular_data.species == 'globis']\nprint(\"There are {} rows for the globis species\".format(globis.shape[0]))","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:05.123068Z","iopub.execute_input":"2022-03-08T11:16:05.123899Z","iopub.status.idle":"2022-03-08T11:16:05.160241Z","shell.execute_reply.started":"2022-03-08T11:16:05.123847Z","shell.execute_reply":"2022-03-08T11:16:05.159497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\nNow onto the images\n
","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:40:43.727538Z","iopub.execute_input":"2022-03-04T21:40:43.728129Z","iopub.status.idle":"2022-03-04T21:40:43.73863Z","shell.execute_reply.started":"2022-03-04T21:40:43.728077Z","shell.execute_reply":"2022-03-04T21:40:43.737705Z"}}},{"cell_type":"code","source":"def read_img(image):\n image = cv2.imread(image)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n return cv2.resize(image, (Config.img_size, Config.img_size))\n \nvertical_stack = []\nfor i in range(5):\n horizontal_stack = []\n for j in range(5):\n img = read_img(globis.iloc[i * 5 + j].path)\n horizontal_stack.append(img)\n horizontal_stack = np.asarray(horizontal_stack)\n horizontal_stack = np.hstack(horizontal_stack)\n vertical_stack.append(horizontal_stack)\nvertical_stack = np.asarray(vertical_stack)\nvertical_stack = np.vstack(vertical_stack)\n\nplt.figure(figsize=(16, 16))\nplt.imshow(vertical_stack)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:07.232716Z","iopub.execute_input":"2022-03-08T11:16:07.233287Z","iopub.status.idle":"2022-03-08T11:16:09.430259Z","shell.execute_reply.started":"2022-03-08T11:16:07.233248Z","shell.execute_reply":"2022-03-08T11:16:09.429007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Cross referencing with sample images from pilot whales\n","metadata":{}},{"cell_type":"code","source":"pilot_whale = tabular_data.loc[tabular_data.species == 'pilot_whale']\n\nhorizontal_stack = []\nfor j in range(5):\n img = read_img(pilot_whale.iloc[j].path)\n horizontal_stack.append(img)\nhorizontal_stack = np.asarray(horizontal_stack)\nhorizontal_stack = np.hstack(horizontal_stack)\nplt.figure(figsize=(15, 40))\nplt.imshow(horizontal_stack)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:12.883505Z","iopub.execute_input":"2022-03-08T11:16:12.883785Z","iopub.status.idle":"2022-03-08T11:16:13.702400Z","shell.execute_reply.started":"2022-03-08T11:16:12.883755Z","shell.execute_reply":"2022-03-08T11:16:13.701720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📌 And also, after looking through other amazing notebooks such as [Andrada Olteanu's](https://www.kaggle.com/andradaolteanu/whales-dolphins-effnet-embedding-cos-distance), it has been decided to rename ~globis~ globus(now confirmed to be a typo👍) and pilot_whale to short_finned_pilot_whale\n> Yeap! The detective was right!","metadata":{}},{"cell_type":"code","source":"tabular_data.species.replace(\n [\"globis\", \"pilot_whale\"],\n [\"short_finned_pilot_whale\",\"short_finned_pilot_whale\"], inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:17.891140Z","iopub.execute_input":"2022-03-08T11:16:17.891398Z","iopub.status.idle":"2022-03-08T11:16:17.918604Z","shell.execute_reply.started":"2022-03-08T11:16:17.891364Z","shell.execute_reply":"2022-03-08T11:16:17.917618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Looking now at the percentage of the classes of whales and dolphins\n\n# First, a new column is created indicating whether the sample is a whale or a dolphin\ntabular_data['class'] = tabular_data.species.apply(lambda x: 'whale' if 'whale' in x else 'dolphin')\ntotal_counts = tabular_data.shape[0] # Get all the samples of the data\nwhale_counts = tabular_data['class'].value_counts()['whale'] # Get all whale counts in the data\ndolphin_counts = tabular_data['class'].value_counts()['dolphin'] # Get all dolphin counts in the data\ndisplay(Markdown(\"The species are now {} with whales being \\\n __{} ({:.2f}%)__ and dolphins being __{} ({:.2f}%)__\" \n .format(tabular_data.species.nunique(), \n whale_counts, whale_counts/total_counts * 100, \n dolphin_counts, dolphin_counts/total_counts * 100)))","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:20.181239Z","iopub.execute_input":"2022-03-08T11:16:20.181502Z","iopub.status.idle":"2022-03-08T11:16:20.220579Z","shell.execute_reply.started":"2022-03-08T11:16:20.181472Z","shell.execute_reply":"2022-03-08T11:16:20.219816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"g = sns.catplot(x=\"species\", kind=\"count\", data=tabular_data, height=9)\ng.set_xticklabels(rotation=90)\nspecies_and_values = [[i[0], i[1]/total_counts*100] for i in tabular_data.species.value_counts().items()]\nfor p in g.ax.patches:\n height = p.get_height()\n x = p.get_x()\n g.ax.text(x - 0.25, height + 3, '{:.2f} %'.format(height/total_counts*100), size=10)\nplt.title(\"Count of species of Dolphins and Whales as a percentage\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:22.340743Z","iopub.execute_input":"2022-03-08T11:16:22.341467Z","iopub.status.idle":"2022-03-08T11:16:22.958472Z","shell.execute_reply.started":"2022-03-08T11:16:22.341426Z","shell.execute_reply":"2022-03-08T11:16:22.957798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n
\n 2. Label encoding and Data Loading pipeline\n
\n
\n \n
\n
","metadata":{}},{"cell_type":"code","source":"# Sklearn Imports\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import StratifiedKFold","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:26.221255Z","iopub.execute_input":"2022-03-08T11:16:26.221784Z","iopub.status.idle":"2022-03-08T11:16:26.363726Z","shell.execute_reply.started":"2022-03-08T11:16:26.221747Z","shell.execute_reply":"2022-03-08T11:16:26.363034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"encoder = LabelEncoder()\n\n# Let us map the individual ids to integers using integer encoding\ntabular_data['encoding'] = encoder.fit_transform(tabular_data.individual_id)","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:28.561787Z","iopub.execute_input":"2022-03-08T11:16:28.562608Z","iopub.status.idle":"2022-03-08T11:16:28.610351Z","shell.execute_reply.started":"2022-03-08T11:16:28.562559Z","shell.execute_reply":"2022-03-08T11:16:28.609562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tabular_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:28.993026Z","iopub.execute_input":"2022-03-08T11:16:28.993588Z","iopub.status.idle":"2022-03-08T11:16:29.007320Z","shell.execute_reply.started":"2022-03-08T11:16:28.993549Z","shell.execute_reply":"2022-03-08T11:16:29.006506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n📝K-fold cross validation\n
\nWhen we are using K-fold cross validation, it is often beneficial to have folds containing roughly the same percentage of observations from each of the different target classes (called stratified k-fold). For example, if our target vector contained gender and 80% of the observations were male, then each fold would contain 80% male and 20% female observation\n
\n
","metadata":{"execution":{"iopub.status.busy":"2022-03-05T18:42:09.343799Z","iopub.execute_input":"2022-03-05T18:42:09.344153Z","iopub.status.idle":"2022-03-05T18:42:09.348701Z","shell.execute_reply.started":"2022-03-05T18:42:09.344117Z","shell.execute_reply":"2022-03-05T18:42:09.347866Z"}}},{"cell_type":"code","source":"stratified_k_fold = StratifiedKFold(\n n_splits=Config.n_fold\n )\nstratified_k_fold_splits = stratified_k_fold.split(\n X = tabular_data.drop(columns = 'encoding'),\n y = tabular_data['encoding']\n)\n\nfor fold, (train_index, valid_index) in enumerate(\n stratified_k_fold_splits\n ):\n tabular_data.loc[valid_index, \"kfold\"] = np.int32(fold)\n \ntabular_data[\"kfold\"] = tabular_data[\"kfold\"].astype(int)\n\ntabular_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:31.481523Z","iopub.execute_input":"2022-03-08T11:16:31.482220Z","iopub.status.idle":"2022-03-08T11:16:32.039543Z","shell.execute_reply.started":"2022-03-08T11:16:31.482181Z","shell.execute_reply":"2022-03-08T11:16:32.038625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import albumentations as A\nfrom albumentations import transforms\nAUTOTUNE = tf.data.AUTOTUNE\n\nNormalize = transforms.Normalize\nCoarseDropout = transforms.CoarseDropout\nFlip = transforms.Flip\n\ndef read_img(path, label):\n img = tf.io.decode_image(\n tf.io.read_file(path), \n channels=3, \n dtype=tf.dtypes.float32,\n expand_animations=False\n )\n img = tf.image.resize(img, (Config.img_size, Config.img_size))\n return img, label\n\ndef get_transform():\n mean = (0.485, 0.456, 0.406)\n std = (0.229, 0.224, 0.225)\n return A.Compose([\n Normalize(mean=mean, std=std, always_apply=True, p=1.0),\n CoarseDropout(max_holes=30, max_height=10, max_width=10, fill_value=64),\n Flip(),\n ])\n\ndef augment_image(image,label, transforms):\n image = image.numpy()\n return transforms(image=image)[\"image\"], label","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:33.720617Z","iopub.execute_input":"2022-03-08T11:16:33.721019Z","iopub.status.idle":"2022-03-08T11:16:34.595795Z","shell.execute_reply.started":"2022-03-08T11:16:33.720984Z","shell.execute_reply":"2022-03-08T11:16:34.594985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Read the footer below \n# def load_dataset(data, fold):\n# dataset = tabular_data.loc[\n# tabular_data.kfold==fold, ['path', 'encoding', 'kfold']\n# ].values\n# path = dataset[:, 0]\n# labels = dataset[:, 1].astype(np.float32)\n# dataset = tf.data.Dataset.from_tensor_slices((path, labels))\n# transform = get_transform()\n# dataset = dataset.map(\n# lambda x, y: read_img(x, y),\n# num_parallel_calls = AUTOTUNE).map(\n# lambda x, y: tf.py_function(\n# func = augment_image(x, y, transform),\n# inp=[x, y],\n# Tout=[tf.float32, tf.float32]\n# ),\n# num_parallel_calls = AUTOTUNE\n# ).batch(Config.batch_size)\n# return dataset","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:38.991480Z","iopub.execute_input":"2022-03-08T11:16:38.992015Z","iopub.status.idle":"2022-03-08T11:16:38.997482Z","shell.execute_reply.started":"2022-03-08T11:16:38.991975Z","shell.execute_reply":"2022-03-08T11:16:38.995355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n \"my_img\"/\n
\n 🧐\n I don't know why, but for some reason, strings and TensorFlow dataset map don't see eye to eye. Could someone investigate this code more. 'Cause only with the invitation of Holmes and Watson could we find the real culprit!\n
\n \"my_img\"/\n
✈️Hence, I went on to use generators!
\n
","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ndef load_dataset(dataset): \n transforms = get_transform()\n def generator():\n for path, label in dataset:\n image, _ = read_img(path, label)\n image = image.numpy()\n image = transforms(image=image)[\"image\"]\n yield image, label\n data = tf.data.Dataset.from_generator(\n generator,\n output_signature=(tf.TensorSpec(shape=(Config.img_size,Config.img_size,3),dtype=tf.float32),\n tf.TensorSpec(shape=(),dtype=tf.float32))\n )\n return data.batch(Config.batch_size)\n\ndef train_and_validation_data(data, fold=0, test_size=0.33):\n data = data.loc[data.kfold==fold, ['path', 'encoding']].values\n train, valid = train_test_split(data, test_size=test_size, random_state=Config.seed)\n train = load_dataset(train)\n valid = load_dataset(valid)\n return train, valid","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:46.571356Z","iopub.execute_input":"2022-03-08T11:16:46.572017Z","iopub.status.idle":"2022-03-08T11:16:46.583782Z","shell.execute_reply.started":"2022-03-08T11:16:46.571975Z","shell.execute_reply":"2022-03-08T11:16:46.582883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 📌 A bit of the visualization","metadata":{}},{"cell_type":"code","source":"mean = (0.485, 0.456, 0.406)\nstd = (0.229, 0.224, 0.225)\n \ndata, _ = train_and_validation_data(tabular_data, 1)\n\nplt.figure(figsize=(10,10))\nfor images, labels in data.take(1):\n for i, (image, label) in enumerate(zip(images, labels)):\n image, label = image.numpy(), label.numpy()\n plt.subplot(4, 4, i + 1)\n image = ((np.asarray(image) * np.array(std) * 255) +(np.array(mean) * 255))\n plt.imshow(image)\n plt.title(f\"{label}\")\n plt.axis(\"off\")\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:16:50.560947Z","iopub.execute_input":"2022-03-08T11:16:50.561490Z","iopub.status.idle":"2022-03-08T11:16:54.428472Z","shell.execute_reply.started":"2022-03-08T11:16:50.561453Z","shell.execute_reply":"2022-03-08T11:16:54.427712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n
\n 3. Model implementation and Embeddings\n
\n
\n \n
\n
","metadata":{"execution":{"iopub.status.busy":"2022-03-07T13:02:29.144533Z","iopub.execute_input":"2022-03-07T13:02:29.14484Z","iopub.status.idle":"2022-03-07T13:02:29.150594Z","shell.execute_reply.started":"2022-03-07T13:02:29.144799Z","shell.execute_reply":"2022-03-07T13:02:29.149415Z"}}},{"cell_type":"markdown","source":"### 📌GeM (Generalized-Mean) pooling\nGeneralized Mean Pooling (GeM) computes the generalized mean of each channel in a tensor. Formally:\n$$e = \\left[(\\frac{1}{|\\Omega|}\\sum_{u\\in\\Omega}x_{cu}^p)\\right]_{c=1,...,C}$$\nwhere $p > 0$ is a parameter. Setting this exponent as $p > 0$ increases the contrast of the pooled feature map and focuses on the salient features of the image. GeM is a generalization of the average pooling commonly used in classification networks ($p = 1$) and of spatial max-pooling layer ($p = \\infty$).\n
","metadata":{}},{"cell_type":"code","source":"class GeMPoolingLayer(tf.keras.layers.Layer):\n def __init__(self, p=3., train_p=False):\n super(GeMPoolingLayer, self).__init__()\n if train_p:\n self.p = tf.Variable(p, dtype=tf.float32)\n else:\n self.p = p\n self.eps = 1e-6\n \n #def build(self, x_shape):\n #self.avg_pool_2d = tf.keras.layers.AveragePooling2D((x_shape[0], x_shape[1]))\n\n def call(self, inputs: tf.Tensor, **kwargs):\n inputs = tf.clip_by_value(inputs, clip_value_min=self.eps, clip_value_max=tf.reduce_max(inputs))\n inputs = tf.pow(inputs, self.p)\n inputs = tf.reduce_mean(inputs, axis=[1, 2], keepdims=False)\n #inputs = self.avg_pool_2d(inputs)\n inputs = tf.pow(inputs, 1./self.p)\n return inputs","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:17:01.365260Z","iopub.execute_input":"2022-03-08T11:17:01.366040Z","iopub.status.idle":"2022-03-08T11:17:01.373528Z","shell.execute_reply.started":"2022-03-08T11:17:01.365991Z","shell.execute_reply":"2022-03-08T11:17:01.372676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 📌ArcFace (Additive Angular Margin) Loss Function\n> ArcFace is a machine learning model that takes two face images as input and outputs the distance between them to see how likely they are to be the same person. It can be used for face recognition and face search. __and in this case, similarity search in subtle features of whales and dolphins__\n\n\n> ArcFace uses a similarity learning mechanism that allows distance metric learning to be solved in the classification task by introducing Angular Margin Loss to replace Softmax Loss.\nThe distance between faces is calculated using cosine distance, which is a method used by search engines and can be calculated by the inner product of two normalized vectors. If the two vectors are the same, $\\theta$ will be 0 and $cos(\\theta) = 1$. If they are orthogonal, $\\theta$ will be $\\pi/2$ and $cos\\theta$=0. Therefore, it can be used as a similarity measure.\n","metadata":{}},{"cell_type":"code","source":"import math \nclass ArcMarginProduct(tf.keras.layers.Layer):\n def __init__(self, n_classes, s=30, m=0.50, easy_margin=False,\n ls_eps=0.0, **kwargs):\n super(ArcMarginProduct, self).__init__(**kwargs)\n self.n_classes = n_classes\n self.s = s\n self.m = m\n self.ls_eps = ls_eps\n self.easy_margin = easy_margin\n self.cos_m = tf.math.cos(m)\n self.sin_m = tf.math.sin(m)\n self.th = tf.math.cos(math.pi - m)\n self.mm = tf.math.sin(math.pi - m) * m\n def get_config(self):\n config = super().get_config().copy()\n config.update({\n 'n_classes': self.n_classes,\n 's': self.s,\n 'm': self.m,\n 'ls_eps': self.ls_eps,\n 'easy_margin': self.easy_margin,\n })\n return config\n def build(self, input_shape):\n super(ArcMarginProduct, self).build(input_shape[0])\n self.W = self.add_weight(\n name='W',\n shape=(int(input_shape[0][-1]), self.n_classes),\n initializer='glorot_uniform',\n dtype='float32',\n trainable=True,\n regularizer=None)\n def call(self, inputs):\n X, y = inputs\n y = tf.cast(y, dtype=tf.int32)\n cosine = tf.matmul(\n tf.math.l2_normalize(X, axis=1),\n tf.math.l2_normalize(self.W, axis=0)\n )\n sine = tf.math.sqrt(1.0 - tf.math.pow(cosine, 2))\n phi = cosine * self.cos_m - sine * self.sin_m\n if self.easy_margin:\n phi = tf.where(cosine > 0, phi, cosine)\n else:\n phi = tf.where(cosine > self.th, phi, cosine - self.mm)\n one_hot = tf.cast(\n tf.one_hot(y, depth=self.n_classes),\n dtype=cosine.dtype\n )\n if self.ls_eps > 0:\n one_hot = (1 - self.ls_eps) * one_hot + self.ls_eps / self.n_classes\n output = (one_hot * phi) + ((1.0 - one_hot) * cosine)\n output *= self.s\n return output","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:17:03.805715Z","iopub.execute_input":"2022-03-08T11:17:03.806285Z","iopub.status.idle":"2022-03-08T11:17:03.820691Z","shell.execute_reply.started":"2022-03-08T11:17:03.806245Z","shell.execute_reply":"2022-03-08T11:17:03.820037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 📌EfficientNetB7 model implementation","metadata":{}},{"cell_type":"code","source":"EfficientNetB7 = tf.keras.applications.EfficientNetB7\n\nbaseModel = EfficientNetB7(\n include_top = False\n)\n\nfor layer in baseModel.layers:\n if not isinstance(layer, tf.keras.layers.BatchNormalization):\n layer.trainable = True\n else:\n layer.trainable = False","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:17:06.002389Z","iopub.execute_input":"2022-03-08T11:17:06.003056Z","iopub.status.idle":"2022-03-08T11:17:12.748388Z","shell.execute_reply.started":"2022-03-08T11:17:06.003018Z","shell.execute_reply":"2022-03-08T11:17:12.747656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"margin = ArcMarginProduct(\n n_classes = tabular_data.encoding.nunique(), \n s = 30, \n m = 0.3, \n name=f'head/arcface', \n dtype='float32'\n )","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:17:12.749817Z","iopub.execute_input":"2022-03-08T11:17:12.750065Z","iopub.status.idle":"2022-03-08T11:17:12.763429Z","shell.execute_reply.started":"2022-03-08T11:17:12.750031Z","shell.execute_reply":"2022-03-08T11:17:12.762709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"input_1 = tf.keras.layers.Input(shape = (Config.img_size, Config.img_size, 3))\ninput_2 = tf.keras.layers.Input(shape = ())\nx = baseModel(input_1)\nx = GeMPoolingLayer()(x)\nembed = tf.keras.layers.Dropout(0.2)(x)\nembed = tf.keras.layers.Dense(512)(embed)\nx = margin([embed, input_2])\noutput = tf.keras.layers.Softmax(dtype='float32')(x)\n\nmodel = tf.keras.models.Model(inputs = [input_1, input_2], outputs = [output])","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:17:12.764756Z","iopub.execute_input":"2022-03-08T11:17:12.765028Z","iopub.status.idle":"2022-03-08T11:17:14.810078Z","shell.execute_reply.started":"2022-03-08T11:17:12.764994Z","shell.execute_reply":"2022-03-08T11:17:14.809323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dummy = tf.random.normal((1, Config.img_size, Config.img_size, 3))\nmodel([dummy, 1])","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:17:16.553704Z","iopub.execute_input":"2022-03-08T11:17:16.554501Z","iopub.status.idle":"2022-03-08T11:17:22.235396Z","shell.execute_reply.started":"2022-03-08T11:17:16.554462Z","shell.execute_reply":"2022-03-08T11:17:22.234674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.keras.utils.plot_model(model)","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:17:23.953299Z","iopub.execute_input":"2022-03-08T11:17:23.953589Z","iopub.status.idle":"2022-03-08T11:17:24.971950Z","shell.execute_reply.started":"2022-03-08T11:17:23.953558Z","shell.execute_reply":"2022-03-08T11:17:24.971113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 📌Model training","metadata":{}},{"cell_type":"code","source":"from tqdm.notebook import tqdm\ntrain_dataset, val_dataset = train_and_validation_data(tabular_data, 0)\n\noptimizer = tf.keras.optimizers.Adam(learning_rate = Config.learning_rate)\ncheckpoint = tf.train.Checkpoint(optimizer=optimizer, model=model)\nloss = tf.keras.losses.SparseCategoricalCrossentropy()\nsparse_categorical_accuracy = tf.keras.metrics.SparseCategoricalAccuracy()\nsparse_top_k_accuracy = tf.keras.metrics.SparseTopKCategoricalAccuracy(k=5)\n\ndef compute_loss(labels, predictions):\n per_example_loss = loss(labels, predictions)\n return tf.nn.compute_average_loss([per_example_loss], global_batch_size=Config.batch_size)\n\ndef train_step(inputs):\n images, labels = inputs\n with tf.GradientTape() as tape:\n predictions = model([images,labels], training=True)\n loss = compute_loss(labels, predictions)\n\n gradients = tape.gradient(loss, model.trainable_variables)\n optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n\n sparse_categorical_accuracy.update_state(labels, predictions)\n sparse_top_k_accuracy.update_state(labels, predictions)\n return loss \n\nfor epoch in range(Config.epochs):\n # TRAIN LOOP\n for batch in tqdm(train_dataset, total=957):\n total_loss = 0.0\n num_batches = 0\n total_loss += train_step(batch)\n num_batches += 1\n train_loss = total_loss / num_batches\n if epoch % 2 == 0:\n checkpoint.save(\"efficientnetb7.h5\")\n template = (\"Epoch {}, Loss: {}, Sparse Accuracy: {}, \"\n \" Sparse Top K Accuracy: {}\")\n print (template.format(epoch+1, train_loss,\n sparse_categorical_accuracy.result()*100, \n sparse_top_k_accuracy.result()*100))","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:19:04.102846Z","iopub.execute_input":"2022-03-08T11:19:04.103442Z","iopub.status.idle":"2022-03-08T11:19:14.917358Z","shell.execute_reply.started":"2022-03-08T11:19:04.103402Z","shell.execute_reply":"2022-03-08T11:19:14.916122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!nvidia-smi","metadata":{"execution":{"iopub.status.busy":"2022-03-08T11:19:35.643691Z","iopub.execute_input":"2022-03-08T11:19:35.644367Z","iopub.status.idle":"2022-03-08T11:19:36.383311Z","shell.execute_reply.started":"2022-03-08T11:19:35.644324Z","shell.execute_reply":"2022-03-08T11:19:36.382474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}