{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# [Happywhale - Whale and Dolphin Identification](https://www.kaggle.com/c/happy-whale-and-dolphin)\n> Identify whales and dolphins by unique characteristic\n\n<img src=\"https://storage.googleapis.com/kaggle-competitions/kaggle/22962/logos/header.png?t=2021-03-17-22-44-09\">","metadata":{}},{"cell_type":"markdown","source":"# Methodology 📌\n* In this notebook we'll demonstrante how to check data distribution using **T-SNE** and **UMAP**.\n* We'll utilize `timm` library for pretrained models to generate **image-embeddings**.\n    * For this notebook we'll be using `tf-efficientnet-b0` model.\n    * Image size is kept `224x224`\n    * Final feature vector has dimension of `1280` (output of the **pooling layer** of `tf-efficientnet-b0`)\n* We'll use **cuML** library from **RAPIDS** for **T-SNE** and **UMAP**.\n* We'll also use **WandB** for visulization. Now, **WandB** allows you to plot **image embedding** directly. :)\n* We'll also check sample images from different locations of **vector-space**.\n* We'll check different data distribution\n    * Train Vs Test\n    * Whales Vs Dolphins\n    * Whales Speices\n    * Dolphins Species\n* There are individuals in the test data that are not observed in the training data, which should be predicted as `new_individual`.\n* **Question:** Will we be able to identify those `new_individual` from **image embeddings**?\n\n> So, follow me, and dare to face the unknown, and ponder the question: What if? ;)\n\n<img src=\"https://everythingmarvel.net/wp-content/uploads/what-if-quotes.jpg\" width=1000>","metadata":{}},{"cell_type":"markdown","source":"# Notebooks 📒\nHere are some of my notebooks for this competition, **please upvote if you find them useful**\n* [Happywhale: BoundingBox [YOLOv5] 🐋🐬](https://www.kaggle.com/awsaf49/happywhale-boundingbox-yolov5)\n* [Happywhale: Cropped Dataset [YOLOv5] ✂️](https://www.kaggle.com/awsaf49/happywhale-cropped-dataset-yolov5)\n* [Happywhale: Data Distribution 🐋🐬](https://www.kaggle.com/awsaf49/happywhale-data-distribution)","metadata":{}},{"cell_type":"markdown","source":"# Install Libraries 🛠","metadata":{}},{"cell_type":"code","source":"!pip install -qU timm  wandb imagesize","metadata":{"execution":{"iopub.status.busy":"2022-02-06T05:19:58.690501Z","iopub.execute_input":"2022-02-06T05:19:58.690863Z","iopub.status.idle":"2022-02-06T05:20:15.255631Z","shell.execute_reply.started":"2022-02-06T05:19:58.69076Z","shell.execute_reply":"2022-02-06T05:20:15.25459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Import Libraries 📚","metadata":{}},{"cell_type":"code","source":"import os\nfrom glob import glob\nfrom tqdm.notebook import tqdm\nimport numpy as np\nimport math\nimport random\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport plotly.express as px\nimport cv2\nimport imagesize\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import DataLoader, Dataset\nimport timm\ntry:\n    from cuml import TSNE, UMAP # if gpu is ON\nexcept:\n    from sklearn.manifold import TSNE # for cpu\nimport wandb\nimport IPython.display as ipd","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-02-06T05:20:15.259525Z","iopub.execute_input":"2022-02-06T05:20:15.259852Z","iopub.status.idle":"2022-02-06T05:20:23.677659Z","shell.execute_reply.started":"2022-02-06T05:20:15.259816Z","shell.execute_reply":"2022-02-06T05:20:23.676604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Notebook Config ⚙️","metadata":{}},{"cell_type":"code","source":"class CFG:\n    seed          = 42\n    base_path     = '../input/happy-whale-and-dolphin'\n    embed_path    = '../input/happywhale-embedding-dataset' # `None` for creating embeddings otherwise load\n    ckpt_path     = '../input/arcface-gem-dataset/Loss15.2453_epoch3.bin' # checkpoint for finetuned model by debarshichanda\n    num_samples   = None #  None for all samples\n    device        = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n    competition   = 'happywhale'\n    _wandb_kernel = 'awsaf49'","metadata":{"execution":{"iopub.status.busy":"2022-02-06T05:20:23.681449Z","iopub.execute_input":"2022-02-06T05:20:23.681721Z","iopub.status.idle":"2022-02-06T05:20:23.687821Z","shell.execute_reply.started":"2022-02-06T05:20:23.681689Z","shell.execute_reply":"2022-02-06T05:20:23.686813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reproducibility ♻","metadata":{}},{"cell_type":"code","source":"def seed_torch(seed_value):\n    random.seed(seed_value) # Python\n    np.random.seed(seed_value) # cpu vars\n    torch.manual_seed(seed_value) # cpu  vars    \n    if torch.cuda.is_available(): \n        torch.cuda.manual_seed(seed_value)\n        torch.cuda.manual_seed_all(seed_value) # gpu vars\n    if torch.backends.cudnn.is_available:\n        torch.backends.cudnn.deterministic = True\n        torch.backends.cudnn.benchmark = False\n    print('# SEEDING DONE')\nseed_torch(CFG.seed)","metadata":{"execution":{"iopub.status.busy":"2022-02-06T05:20:23.689802Z","iopub.execute_input":"2022-02-06T05:20:23.69052Z","iopub.status.idle":"2022-02-06T05:20:23.706957Z","shell.execute_reply.started":"2022-02-06T05:20:23.690461Z","shell.execute_reply":"2022-02-06T05:20:23.705621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# WandB ⭐\n<div align=center> <img src=\"https://camo.githubusercontent.com/dd842f7b0be57140e68b2ab9cb007992acd131c48284eaf6b1aca758bfea358b/68747470733a2f2f692e696d6775722e636f6d2f52557469567a482e706e67\" width=500></div>\n\nWeights & Biases (W&B) is MLOps platform for tracking our experiemnts. We can use it to Build better models faster with experiment tracking, dataset versioning, and model management. Some of the cool features of W&B:\n\n* Track, compare, and visualize ML experiments\n* Get live metrics, terminal logs, and system stats streamed to the centralized dashboard.\n* Explain how your model works, show graphs of how model versions improved, discuss bugs, and demonstrate progress towards milestones.","metadata":{}},{"cell_type":"code","source":"import wandb\n\ntry:\n    from kaggle_secrets import UserSecretsClient\n    user_secrets = UserSecretsClient()\n    api_key = user_secrets.get_secret(\"WANDB\")\n    wandb.login(key=api_key)\n    anonymous = None\nexcept:\n    anonymous = \"must\"\n    wandb.login(anonymous=anonymous)\n    print('To use your W&B account,\\nGo to Add-ons -> Secrets and provide your W&B access token. Use the Label name as WANDB. \\nGet your W&B access token from here: https://wandb.ai/authorize')","metadata":{"execution":{"iopub.status.busy":"2022-02-06T05:20:23.710636Z","iopub.execute_input":"2022-02-06T05:20:23.711021Z","iopub.status.idle":"2022-02-06T05:20:25.62152Z","shell.execute_reply.started":"2022-02-06T05:20:23.710976Z","shell.execute_reply":"2022-02-06T05:20:25.620657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Meta Data 📈\n* `train_images/` - a folder containing the training images\n* `train.csv` - provides the species and the individual_id for each of the training images\n* `test_images/` - a folder containing the test images; for each image, your task is to predict the individual_id; no species information is given for the test data; there are individuals in the test data that are not observed in the training data, which should be predicted as new_individual.\n* `sample_submission.csv` - a sample submission file in the correct format\n\n> Note: We don't have access to `species` column for **test** data. So, we can't direcly use `species` for **train**.","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(f'{CFG.base_path}/train.csv')\ndf['image_path'] = CFG.base_path+'/train_images/'+df['image']\ndf['split'] = 'Train'\n\ntest_df = pd.read_csv(f'{CFG.base_path}/sample_submission.csv')\ntest_df['image_path'] = CFG.base_path+'/test_images/'+test_df['image']\ntest_df['split'] = 'Test'\n\nprint('Train Images: {:,} | Test Images: {:,}'.format(len(df), len(test_df)))","metadata":{"execution":{"iopub.status.busy":"2022-02-06T05:20:25.623994Z","iopub.execute_input":"2022-02-06T05:20:25.624305Z","iopub.status.idle":"2022-02-06T05:20:25.810485Z","shell.execute_reply.started":"2022-02-06T05:20:25.624255Z","shell.execute_reply":"2022-02-06T05:20:25.808747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Clip Data\nSize of the dataset is huge. We can control the size using `CFG.num_samples`","metadata":{}},{"cell_type":"code","source":"if CFG.num_samples:\n    df = df.iloc[:CFG.num_samples]\n    test_df = test_df.iloc[:CFG.num_samples]","metadata":{"execution":{"iopub.status.busy":"2022-02-06T05:30:10.433007Z","iopub.execute_input":"2022-02-06T05:30:10.433337Z","iopub.status.idle":"2022-02-06T05:30:10.439079Z","shell.execute_reply.started":"2022-02-06T05:30:10.433305Z","shell.execute_reply":"2022-02-06T05:30:10.437805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Fix Meta Data\nFolowing cells,\n* Converts `beluga`, `globis` to `whales` for 2class label.\n* Fixes Duplicate Labels.","metadata":{}},{"cell_type":"code","source":"# convert beluga, globis to whales\ndf.loc[df.species.str.contains('beluga'), 'species'] = 'beluga_whale'\ndf.loc[df.species.str.contains('globis'), 'species'] = 'short_finned_pilot_whale'\ndf.loc[df.species.str.contains('pilot_whale'), 'species'] = 'short_finned_pilot_whale'\ndf['class'] = df.species.map(lambda x: 'whale' if 'whale' in x else 'dolphin')\n\n# fix duplicate labels\n# https://www.kaggle.com/c/happy-whale-and-dolphin/discussion/304633\ndf['species'] = df['species'].str.replace('bottlenose_dolpin','bottlenose_dolphin')\ndf['species'] = df['species'].str.replace('kiler_whale','killer_whale')","metadata":{"execution":{"iopub.status.busy":"2022-02-06T05:31:49.099371Z","iopub.execute_input":"2022-02-06T05:31:49.099693Z","iopub.status.idle":"2022-02-06T05:31:49.300921Z","shell.execute_reply.started":"2022-02-06T05:31:49.099663Z","shell.execute_reply":"2022-02-06T05:31:49.300008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Find Image Size","metadata":{}},{"cell_type":"code","source":"def get_imgsize(row):\n    row['width'], row['height'] = imagesize.get(row['image_path'])\n    return row","metadata":{"execution":{"iopub.status.busy":"2022-02-06T05:31:52.422875Z","iopub.execute_input":"2022-02-06T05:31:52.423355Z","iopub.status.idle":"2022-02-06T05:31:52.435059Z","shell.execute_reply.started":"2022-02-06T05:31:52.423275Z","shell.execute_reply":"2022-02-06T05:31:52.433957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train\ntqdm.pandas(desc='Train ')\ndf = df.progress_apply(get_imgsize, axis=1)\ndf.to_csv('train.csv', index=False)\n\n# Test\ntqdm.pandas(desc='Test ')\ntest_df = test_df.progress_apply(get_imgsize, axis=1)\ntest_df.to_csv('test.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2022-02-06T05:31:52.639171Z","iopub.execute_input":"2022-02-06T05:31:52.639896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Check Metadata","metadata":{}},{"cell_type":"code","source":"print('Train:')\ndisplay(df.head(2))\n\nprint('Test:')\ndisplay(test_df.head(2))","metadata":{"execution":{"iopub.status.busy":"2022-02-05T03:56:47.748445Z","iopub.execute_input":"2022-02-05T03:56:47.748761Z","iopub.status.idle":"2022-02-05T03:56:47.781263Z","shell.execute_reply.started":"2022-02-05T03:56:47.748711Z","shell.execute_reply":"2022-02-05T03:56:47.780129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA 🌈","metadata":{}},{"cell_type":"markdown","source":"## Different Species\nIt seems we also have a **Class Imablance** across different **species**. We may want to split our data **stratifying species**.","metadata":{}},{"cell_type":"code","source":"data = df.species.value_counts().reset_index()\nfig = px.bar(data, x='index', y='species', color='species',title='Species', text_auto=True)\nfig.update_traces(textfont_size=12, textangle=0, textposition=\"outside\", cliponaxis=False)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-05T03:56:42.944804Z","iopub.status.idle":"2022-02-05T03:56:42.945791Z","shell.execute_reply.started":"2022-02-05T03:56:42.945436Z","shell.execute_reply":"2022-02-05T03:56:42.945472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dolphin Vs Whale\n* We have more samples of Whales comparing Dolphins. This might be a important factor while **splitting** the data.","metadata":{}},{"cell_type":"code","source":"data = df['class'].value_counts().reset_index()\nfig = px.bar(data, x='index', y='class', color='class', title='Whale Vs Dolphin', text_auto=True)\nfig.update_traces(textfont_size=12, textangle=0, textposition=\"outside\", cliponaxis=False)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-05T03:56:42.94743Z","iopub.status.idle":"2022-02-05T03:56:42.94825Z","shell.execute_reply.started":"2022-02-05T03:56:42.947934Z","shell.execute_reply":"2022-02-05T03:56:42.947967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ImageSize Vs Class\n* It is visible that Distribution of ImageSize is similar for both Whale and Dolphin except some cases in height.","metadata":{}},{"cell_type":"code","source":"fig = px.histogram(df,\n                   x=\"width\", \n                   color=\"class\",\n                   barmode='group',\n                   log_y=True,\n                   title='Width Vs Class')\ndisplay(fig.show())\n\nfig = px.histogram(df,\n                   x=\"height\", \n                   color=\"class\",\n                   barmode='group',\n                   log_y=True,\n                   title='Height Vs Class')\ndisplay(fig.show())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-05T03:56:42.950053Z","iopub.status.idle":"2022-02-05T03:56:42.950933Z","shell.execute_reply.started":"2022-02-05T03:56:42.950618Z","shell.execute_reply":"2022-02-05T03:56:42.950651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ImageSize Vs Split(Train/Test)\n* It can be notices that distribution of width for **train** and **test** data, looks quite similar. So, we can resize without any tension.\n* For height we have some unique shapes.","metadata":{}},{"cell_type":"code","source":"fig = px.histogram(pd.concat([df, test_df]),\n                   x=\"width\", \n                   color=\"split\",\n                   barmode='group',\n                   log_y=True,\n                   title='Width Vs Split');\ndisplay(fig.show())\n\nfig = px.histogram(pd.concat([df, test_df]),\n                   x=\"height\", \n                   color=\"split\",\n                   barmode='group',\n                   log_y=True,\n                   title='Height Vs Split');\ndisplay(fig.show())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-05T03:56:42.952689Z","iopub.status.idle":"2022-02-05T03:56:42.953639Z","shell.execute_reply.started":"2022-02-05T03:56:42.953216Z","shell.execute_reply":"2022-02-05T03:56:42.953248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Pipeline 🍚\nTo create **image embedding** we will,\n* Read the image.\n* Resize it accordingly.","metadata":{}},{"cell_type":"code","source":"def load_image(path):\n    img = cv2.imread(path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    return img\n\nclass ImageDataset(Dataset):\n    def __init__(self,\n                 path,\n                 target=None,\n                 input_shape=(128, 256),\n                 transform=None,\n                 channel_first=True,\n                ):\n        super(ImageDataset, self).__init__()\n        self.path = path\n        self.target = target\n        self.input_shape = input_shape\n        self.transform = transform\n        self.channel_first = channel_first\n    def __len__(self):\n        return len(self.path)\n    \n    def __getitem__(self, idx):\n        img = load_image(self.path[idx])\n        img = cv2.resize(img, dsize=self.input_shape)\n        if self.transform is not None:\n            img = self.transform(image=img)[\"image\"]\n        if self.channel_first:\n            img = img.transpose((2, 0, 1))\n        if self.target is not None:\n            target = self.target[idx]\n            return img, target\n        else:\n            return img\n\ndef get_dataset(path, target=None, batch_size=32, input_shape=(224, 224)):\n    dataset = ImageDataset(path=path,\n                           target=target,\n                           input_shape=input_shape,\n                          )\n\n    dataloader = DataLoader(\n        dataset,\n        batch_size=batch_size,\n        num_workers=2,\n        shuffle=False,\n        pin_memory=True,\n    )\n    return dataloader","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-02-05T03:56:59.911839Z","iopub.execute_input":"2022-02-05T03:56:59.912403Z","iopub.status.idle":"2022-02-05T03:56:59.955395Z","shell.execute_reply.started":"2022-02-05T03:56:59.912285Z","shell.execute_reply":"2022-02-05T03:56:59.954072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualization 📉","metadata":{}},{"cell_type":"markdown","source":"## Utility","metadata":{}},{"cell_type":"code","source":"def plot_batch(batch, row=2, col=2, channel_first=True):\n    if isinstance(batch, tuple) or isinstance(batch, list):\n        imgs, tars = batch\n    else:\n        imgs, tars = batch, None\n    plt.figure(figsize=(col*3, row*3))\n    for i in range(row*col):\n        plt.subplot(row, col, i+1)\n        img = imgs[i].numpy()\n        if channel_first:\n            img = img.transpose((1, 2, 0))\n        plt.imshow(img)\n        if tars is not None:\n            plt.title(tars[i])\n        plt.axis('off')\n    plt.tight_layout()\n    plt.show()\n    \ndef gen_colors(n=10):\n    cmap   = plt.get_cmap('rainbow')\n    colors = [cmap(i) for i in np.linspace(0, 1, n + 2)]\n    colors = [(c[2] * 255, c[1] * 255, c[0] * 255) for c in colors]\n    return colors","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-05T03:57:01.983397Z","iopub.execute_input":"2022-02-05T03:57:01.983711Z","iopub.status.idle":"2022-02-05T03:57:01.994373Z","shell.execute_reply.started":"2022-02-05T03:57:01.983662Z","shell.execute_reply":"2022-02-05T03:57:01.993387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dataloader\nWe need to create dataloader to read images efficiently.","metadata":{}},{"cell_type":"code","source":"train_loader = get_dataset(path=df.image_path.tolist(),\n                       target=df.species.tolist(),\n                       input_shape=(224,224),\n                      )\ntest_loader = get_dataset(path=test_df.image_path.tolist(),\n                       target=None,\n                       input_shape=(224,224),\n                      )","metadata":{"execution":{"iopub.status.busy":"2022-02-05T03:57:03.595754Z","iopub.execute_input":"2022-02-05T03:57:03.596156Z","iopub.status.idle":"2022-02-05T03:57:03.606547Z","shell.execute_reply.started":"2022-02-05T03:57:03.596111Z","shell.execute_reply":"2022-02-05T03:57:03.60525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train\nLet's have a look at some images from **Train** Data","metadata":{}},{"cell_type":"code","source":"batch = iter(train_loader).next()\nplot_batch(batch, row=2, col=5)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-05T03:57:04.384669Z","iopub.execute_input":"2022-02-05T03:57:04.385306Z","iopub.status.idle":"2022-02-05T03:57:21.374866Z","shell.execute_reply.started":"2022-02-05T03:57:04.385228Z","shell.execute_reply":"2022-02-05T03:57:21.373363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Test\nLet's have a look at some images from **Test** Data","metadata":{}},{"cell_type":"code","source":"batch = iter(test_loader).next()\nplot_batch(batch, row=2, col=5)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-05T03:57:21.37732Z","iopub.execute_input":"2022-02-05T03:57:21.377657Z","iopub.status.idle":"2022-02-05T03:57:30.226068Z","shell.execute_reply.started":"2022-02-05T03:57:21.377615Z","shell.execute_reply":"2022-02-05T03:57:30.223425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model 📦","metadata":{}},{"cell_type":"code","source":"class ImageModel(nn.Module):\n    def __init__(self, backbone_name, pretrained=True):\n        super(ImageModel, self).__init__()\n        self.backbone = timm.create_model(backbone_name,\n                                          pretrained=pretrained)\n        self.backbone.reset_classifier(0) # to get pooled features\n        \n    def forward(self, x):            \n        x = self.backbone(x)\n        return x\n\nclass GeM(nn.Module):\n    def __init__(self, p=3, eps=1e-6):\n        super(GeM, self).__init__()\n        self.p = nn.Parameter(torch.ones(1)*p)\n        self.eps = eps\n\n    def forward(self, x):\n        return self.gem(x, p=self.p, eps=self.eps)\n        \n    def gem(self, x, p=3, eps=1e-6):\n        return F.avg_pool2d(x.clamp(min=eps).pow(p), (x.size(-2), x.size(-1))).pow(1./p)\n        \n    def __repr__(self):\n        return self.__class__.__name__ + \\\n                '(' + 'p=' + '{:.4f}'.format(self.p.data.tolist()[0]) + \\\n                ', ' + 'eps=' + str(self.eps) + ')'\n\nclass FTModel(nn.Module):\n    \"\"\"FineTune (on happywhale dataset) Model\"\"\"\n    def __init__(self, model_name, pretrained=True):\n        super(FTModel, self).__init__()\n        self.model = timm.create_model(model_name, pretrained=pretrained)\n        in_features = self.model.classifier.in_features\n        self.model.classifier = nn.Identity()\n        self.model.global_pool = nn.Identity()\n        self.pooling = GeM()\n        self.fc = nn.Identity()\n\n    def forward(self, images):\n        features = self.model(images)\n        pooled_features = self.pooling(features).flatten(1)\n        return pooled_features\n    \ndef load_model(ckpt_path):\n    model = FTModel(model_name='tf_efficientnet_b0', pretrained=False)\n    model.load_state_dict(torch.load(ckpt_path), strict=False)\n    model.fc = nn.Identity()\n    return model\n    ","metadata":{"execution":{"iopub.status.busy":"2022-02-05T03:59:36.53227Z","iopub.execute_input":"2022-02-05T03:59:36.532641Z","iopub.status.idle":"2022-02-05T03:59:36.549709Z","shell.execute_reply.started":"2022-02-05T03:59:36.532593Z","shell.execute_reply":"2022-02-05T03:59:36.548481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model1 = ImageModel('tf_efficientnet_b0')\nmodel2 = load_model(CFG.ckpt_path)","metadata":{"execution":{"iopub.status.busy":"2022-02-05T03:59:36.908257Z","iopub.execute_input":"2022-02-05T03:59:36.909219Z","iopub.status.idle":"2022-02-05T03:59:39.912746Z","shell.execute_reply.started":"2022-02-05T03:59:36.909182Z","shell.execute_reply":"2022-02-05T03:59:39.911709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Embeddings 🔰\n* What is Embeddings?: \n    * An embedding is a **low-dimensional space** into which one can translate high-dimensional vectors. Ideally, an embedding captures some of the semantics of the input by placing semantically similar inputs close together in the embedding space. An embedding can be learned and reused across models.\n    * Embeddings from different models belongs to different **vector-space**. Hence, we shouldn't directly compare them.\n    \n<div align=center> <img src=\"https://www.pinecone.io/images/vector_embeddings.jpg\" width=600></div>\n    \n* How to Generate Embeddings?\n    * We can train a simple **classification** model and remove its **classification head** and **TADA**, we have embedding model. For a image output of this model is **embedding** of that image.\n    * In other words, we have to take output from the **pooling layers**","metadata":{}},{"cell_type":"markdown","source":"## Utility","metadata":{}},{"cell_type":"code","source":"@torch.no_grad()\ndef predict(model, dataloader):\n    model.eval() # turn off layers such as BatchNorm or Dropout\n    model.to(CFG.device) # cpu -> gpu\n    embeds = []\n    pbar = tqdm(dataloader, total=len(dataloader))\n    for img in pbar:\n        img = img.type(torch.float32) # uint8 -> float32\n        img = img.to(CFG.device) # cpu -> gpu\n        embed = model(img) # this is where magic happens ;)\n        gpu_mem = torch.cuda.memory_reserved() / 1E9 if torch.cuda.is_available() else 0\n        pbar.set_postfix(gpu_mem=f'{gpu_mem:0.2f} GB')\n        embeds.append(embed.cpu().detach().numpy())\n    return np.concatenate(embeds)","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-02-05T03:59:44.151676Z","iopub.execute_input":"2022-02-05T03:59:44.152015Z","iopub.status.idle":"2022-02-05T03:59:44.171447Z","shell.execute_reply.started":"2022-02-05T03:59:44.151982Z","shell.execute_reply":"2022-02-05T03:59:44.167953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dataloader","metadata":{}},{"cell_type":"code","source":"train_loader = get_dataset(\n    path=df.image_path.tolist(),\n    target=None,\n    input_shape=(224,224),\n    batch_size=128*4,\n)\ntest_loader = get_dataset(\n    path=test_df.image_path.tolist(),\n    target=None,\n    input_shape=(224,224),\n    batch_size=128*4,\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-05T03:59:45.906094Z","iopub.execute_input":"2022-02-05T03:59:45.906442Z","iopub.status.idle":"2022-02-05T03:59:45.918729Z","shell.execute_reply.started":"2022-02-05T03:59:45.906408Z","shell.execute_reply":"2022-02-05T03:59:45.917688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Generate Embeddings","metadata":{}},{"cell_type":"code","source":"if CFG.embed_path:\n    print('# Load Train Embeddings:')\n    train_embeds = np.load(f'{CFG.embed_path}/train_embeds.npy')\n    print('# Load Test Embeddings:')\n    test_embeds = np.load(f'{CFG.embed_path}/test_embeds.npy')\n    print('# Load Train Embeddings (Finetune):')\n    train_embeds2 = np.load(f'{CFG.embed_path}/train_embeds2.npy')\n    print('# Test Embeddings  (Finetune):')\n    test_embeds2 = np.load(f'{CFG.embed_path}/test_embeds2.npy')\nelse:\n    print('# Train Embeddings:')\n    train_embeds = predict(model1, train_loader)\n    print('# Test Embeddings:')\n    test_embeds = predict(model1, test_loader)\n    print('# Train Embeddings (Finetune):')\n    train_embeds2 = predict(model2, train_loader)\n    print('# Test Embeddings (Finetune):')\n    test_embeds2 = predict(model2, test_loader)\n    \n# Save Embeddings\nnp.save('train_embeds.npy', train_embeds) \nnp.save('test_embeds.npy', test_embeds)\nnp.save('train_embeds2.npy', train_embeds2)\nnp.save('test_embed2.npy', test_embeds2)","metadata":{"execution":{"iopub.status.busy":"2022-02-05T03:59:47.188988Z","iopub.execute_input":"2022-02-05T03:59:47.18929Z","iopub.status.idle":"2022-02-05T03:59:55.871406Z","shell.execute_reply.started":"2022-02-05T03:59:47.189257Z","shell.execute_reply":"2022-02-05T03:59:55.870285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# T-SNE 💫\n* What is T-SNE?\n    * t-Distributed Stochastic Neighbor Embedding (T-SNE) is an unsupervised, non-linear technique primarily used for data exploration and visualizing high-dimensional data. In other words, t-SNE proves the intuition of how the data is arranged in a high-dimensional space.\n    \n<div align=center> <img src=\"https://images.squarespace-cdn.com/content/v1/5bc760564d546e3f72b825c1/1556571430321-VFB052K3PFO78F5V2I2H/174Yb_MHCII_19_04_29_14_22_11_106.png\" width=400></div>\n\n* How to use T-SNE?\n    * To take advantage of **GPU** we'll be using **cuml** library from **NVIDIA RAPIDS**. It can boost up the speed up to even **100x**. ;)","metadata":{}},{"cell_type":"code","source":"tsne = TSNE()\n\n# Concatenate both train and test\nembeds = np.concatenate([train_embeds,test_embeds])\nembeds2 = np.concatenate([train_embeds2,test_embeds2])\n\n# Fit TSNE on the embeddings and then transfer data\ntsne_embed = tsne.fit_transform(embeds)\ntsne_embed2 = tsne.fit_transform(embeds2)\n\n# Train\ndf['x'] = tsne_embed[:len(train_embeds),0]\ndf['y'] = tsne_embed[:len(train_embeds),1]\n\ndf['x2'] = tsne_embed2[:len(train_embeds2),0]\ndf['y2'] = tsne_embed2[:len(train_embeds2),1]\n\n# Test\ntest_df['x'] = tsne_embed[len(train_embeds):,0]\ntest_df['y'] = tsne_embed[len(train_embeds):,1]\n\ntest_df['x2'] = tsne_embed2[len(train_embeds2):,0]\ntest_df['y2'] = tsne_embed2[len(train_embeds2):,1]","metadata":{"execution":{"iopub.status.busy":"2022-02-05T03:59:57.807063Z","iopub.execute_input":"2022-02-05T03:59:57.807393Z","iopub.status.idle":"2022-02-05T04:00:11.841489Z","shell.execute_reply.started":"2022-02-05T03:59:57.807361Z","shell.execute_reply":"2022-02-05T04:00:11.840533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# UMAP 🌠","metadata":{}},{"cell_type":"code","source":"umap = UMAP()\n\n# Fit TSNE on the embeddings and then transfer data\numap_embed = umap.fit_transform(embeds)\numap_embed2 = umap.fit_transform(embeds2)\n\n# Train\ndf['x3'] = umap_embed[:len(train_embeds),0]\ndf['y3'] = umap_embed[:len(train_embeds),1]\n\ndf['x4'] = umap_embed2[:len(train_embeds2),0]\ndf['y4'] = umap_embed2[:len(train_embeds2),1]\n\n# Test\ntest_df['x3'] = umap_embed[len(train_embeds):,0]\ntest_df['y3'] = umap_embed[len(train_embeds):,1]\n\ntest_df['x4'] = umap_embed2[len(train_embeds2):,0]\ntest_df['y4'] = umap_embed2[len(train_embeds2):,1]","metadata":{"execution":{"iopub.status.busy":"2022-02-05T04:00:18.831634Z","iopub.execute_input":"2022-02-05T04:00:18.832009Z","iopub.status.idle":"2022-02-05T04:00:24.671659Z","shell.execute_reply.started":"2022-02-05T04:00:18.831964Z","shell.execute_reply":"2022-02-05T04:00:24.67061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# WandB Logging ⭐\n* We'll also log our result in **WandB** so that we can try out other plotting methods such as **PCA**, **UMAP**>\n* We can plot embeddings using **WandB** in following two ways,\n    1. We can directly save the **image embeddings** in wandb then plot using `2D Projection: Plot`.\n    2. We can compute **t-sne** like features first and then save their result in wandb. Then plot result using `2D Projection: Plot`. In this notebook, We'll be using this 2nd way but you are encouraged to try both.\n* We'll also save our **embeddings** to **WandB** so that we can re-use them later.","metadata":{}},{"cell_type":"code","source":"# convert config from class to dict\nconfig = {k:v for k,v in dict(vars(CFG)).items() if '__' not in k}\n\n# initialize wandb project\nwandb.init(project='happywhale-public', config=config)\n\n# process data for wandb\nwdf1 = pd.concat([df, test_df]).drop(columns=['image_path','predictions']) # train + test\nwdf2 = df.copy() # only train as some columns of test don't have any value e.g: species\n\n# log the data\nwandb.log({\"All\":wdf1, \n           \"Train\":wdf2}) # log both result\n\n# save embeddings to wandb for later use\nwandb.save('test_embeds.npy'); # save train embeddings\nwandb.save('train_embeds.npy'); # save test embeddings\nwandb.save('test_embeds2.npy'); # save train embeddings\nwandb.save('train_embeds2.npy'); # save test embeddings\n\n# show wandb dashboard\ndisplay(ipd.IFrame(wandb.run.url, width=1080, height=720)) # show wandb dashboard\n\n# finish logging\nwandb.finish()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-05T03:56:42.986823Z","iopub.status.idle":"2022-02-05T03:56:42.987785Z","shell.execute_reply.started":"2022-02-05T03:56:42.987392Z","shell.execute_reply":"2022-02-05T03:56:42.987424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After logging **WandB** output directory will look like this,\n\n<img src=\"https://i.ibb.co/56S299q/w-b03.png\" alt=\"w-b03\" border=\"0\">\n\nAnd **Embedding Plot** will look something like this,\n\n<img src=\"https://i.ibb.co/W0gYY0P/w-b01.png\" alt=\"w-b01\" border=\"0\">\n<img src=\"https://i.ibb.co/G5SnMMv/w-b02.png\" alt=\"w-b02\" border=\"0\">\n<img src=\"https://i.ibb.co/1MxnZqX/w-b04.png\" alt=\"w-b04\" border=\"0\">\n\n### [Complete Dashboard can be accessed here 🡪](https://wandb.ai/awsaf49/happywhale-public)","metadata":{}},{"cell_type":"markdown","source":"# Visualize **T-SNE** & UMAP 🌠☀️\nHuuuf! We have come so far. Let's visualize the **image embeddings** using **T-SNE**.","metadata":{}},{"cell_type":"code","source":"x_min = df.x.min()\nx_max = df.x.max()\ny_min = df.y.min()\ny_max = df.y.max()\n\ndef plot(df, ROW, COL):\n    plt.figure(figsize=(15,16*ROW/COL))\n    for k in range(ROW):\n        for j in range(COL):\n            plt.subplot(ROW,COL,k*COL+j+1)\n            row = df.iloc[k*COL+j]\n            img = cv2.imread(row.image_path)\n            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n            plt.axis('off')\n            id_ = row['image']\n            try:\n                species = row['species']\n                class_ = row['class']\n            except:\n                species = None\n                class_ = None\n            plt.title('id:{}\\nclass:{}\\nspecies:{}'.format(id_, class_, species))\n            plt.imshow(img)  \n    plt.tight_layout()\n    plt.show()\n    \ndef plot_tsne(df1, df2, labels=['Train', 'Test'], colors=['orange','blue']):\n    plt.figure(figsize=(10,10))\n    plt.scatter(df1.x, df1.y,color=colors[0],s=10,label=labels[0])\n    plt.scatter(df2.x, df2.y,color=colors[1],s=10,label=labels[1], alpha=0.4)\n    plt.plot([xa_mx,xa_mx],[ya_mx,yb_mx],color='black')\n    plt.plot([xa_mx,xb_mx],[ya_mx,ya_mx],color='black')\n    plt.plot([xb_mx,xb_mx],[ya_mx,yb_mx],color='black')\n    plt.plot([xa_mx,xb_mx],[yb_mx,yb_mx],color='black')\n    plt.legend()\n    plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-05T04:00:55.352912Z","iopub.execute_input":"2022-02-05T04:00:55.353222Z","iopub.status.idle":"2022-02-05T04:00:55.373638Z","shell.execute_reply.started":"2022-02-05T04:00:55.353175Z","shell.execute_reply":"2022-02-05T04:00:55.372566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train Vs Test 1️⃣\n* From the **T-SNE** it seems that `new_individual` can't be indentified from **image embedding**. So, we can say,\n    * The `new_individual` class looks very similar to `old_indiviual`.\n    * Or there aren't many samples for `new_individual` class.\n* We need to build model keeping that in mind that `new_individual` has similar distribution as `old_individual`.\n* As the **Train** and **Test** data distribution is similar, there are less chance of **shakeup**. Which is a sigh of relief for us. So, we can worry less about the mysterious **test** data and concentrate on the **train** data.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(15,15))\n\nplt.subplot(2, 2, 1)\nplt.scatter(df.x,df.y,color='orange',s=10,label='Train')\nplt.scatter(test_df.x,test_df.y,color='blue',s=10,label='Test', alpha=0.5)\nplt.title('T-SNE')\nplt.legend(prop={'size': 12})\n\nplt.subplot(2, 2, 2)\nplt.scatter(df.x2,df.y2,color='orange',s=10,label='Train')\nplt.scatter(test_df.x2,test_df.y2,color='blue',s=10,label='Test', alpha=0.5)\nplt.title('T-SNE (Finetune)')\nplt.legend(prop={'size': 12})\n\nplt.subplot(2, 2, 3)\nplt.scatter(df.x3,df.y3,color='red',s=10,label='Train')\nplt.scatter(test_df.x3,test_df.y3,color='green',s=10,label='Test', alpha=0.5)\nplt.title('UMAP')\nplt.legend(prop={'size': 12})\n\nplt.subplot(2, 2, 4)\nplt.scatter(df.x4,df.y4,color='red',s=10,label='Train')\nplt.scatter(test_df.x4,test_df.y4,color='green',s=10,label='Test', alpha=0.5)\nplt.title('UMAP (Finetune)')\nplt.legend(prop={'size': 12})\n\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-05T04:04:35.631858Z","iopub.execute_input":"2022-02-05T04:04:35.632225Z","iopub.status.idle":"2022-02-05T04:04:39.337542Z","shell.execute_reply.started":"2022-02-05T04:04:35.632185Z","shell.execute_reply":"2022-02-05T04:04:39.336451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Random **Train/Test** 2️⃣\nLet's plot some train and test imaes. \n* We'll be choosing random location from vector space for images.\n* For same location we'll be plotting images from both **train** & **test** data.\n* The similarity between images depend on the window of our search. If we search using large window we'll have less similar images and if we use tiny window we'll have very similar images **(black rectangle)**. But choosing tiny window for search could result in **empty** images as not at all location we have images.\n* Search window can be controlled using `X_DIV` and `Y_DIV`. The bigger the value the smaller the dimension of the window.","metadata":{}},{"cell_type":"code","source":"ROW = 2\nCOL = 5\n\nX_DIV = 20; Y_DIV = 20;\nx_step = (x_max - x_min)/X_DIV\ny_step = (y_max - y_min)/Y_DIV\n\nfor it in range(5):\n    i = 0; i2=0; trial=0;\n    while i<ROW*COL:\n        trial+=1\n        if trial>50:\n            break\n        k = np.random.randint(0,X_DIV)\n        j = np.random.randint(0,Y_DIV)\n        xa_mx = k*x_step + x_min\n        xb_mx = (k+1)*x_step + x_min\n        ya_mx = j*y_step + y_min\n        yb_mx = (j+1)*y_step + y_min\n        df1 = df.loc[(df.x>xa_mx)&(df.x<xb_mx)&(df.y>ya_mx)&(df.y<yb_mx)]\n        df2 = test_df.loc[(test_df.x>xa_mx)&(test_df.x<xb_mx)&(test_df.y>ya_mx)&(test_df.y<yb_mx)]\n        i  = len(df1)\n        i2 = len(df2)\n    print(f'### RANDOM: {it}')\n    \n    print('>>TSNE:')\n    plot_tsne(df, test_df)\n\n    print('>>Train:')\n    if i>=ROW*COL:\n        plot(df1, ROW, COL)\n    else:\n        print('Not Found')\n\n    print('>>Test')\n    if i2>=ROW*COL:\n        plot(df2, ROW, COL)\n    else:\n        print('Not Found')\n    print('\\n\\n')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-04T15:09:05.699754Z","iopub.execute_input":"2022-02-04T15:09:05.700495Z","iopub.status.idle":"2022-02-04T15:09:08.253734Z","shell.execute_reply.started":"2022-02-04T15:09:05.700431Z","shell.execute_reply":"2022-02-04T15:09:08.25261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Whale Vs Dolphin 3️⃣\nLet's plot the **image embeddings** of Whales and Dolphins using **T-SNE**.\n* From **Train/Test** section if you're wondering that what we were expecting to see, check out following images.\n* It is clearly visible that there are some Whales which look very different from Dolphins. We were actually expecting somewhat similar pattern in **Train/Test** Plot.","metadata":{}},{"cell_type":"code","source":"w_df = df[df['class']=='whale']\nd_df = df[df['class']=='dolphin']\n\nplt.figure(figsize=(15,15))\n\nplt.subplot(2, 2, 1)\nplt.scatter(w_df.x,w_df.y,color='orange',s=10,label='Whale')\nplt.scatter(d_df.x,d_df.y,color='blue',s=10,label='Dolphin', alpha=0.4)\nplt.title('T-SNE')\nplt.legend(prop={'size': 12})\n\nplt.subplot(2, 2, 2)\nplt.scatter(w_df.x2,w_df.y2,color='orange',s=10,label='Whale')\nplt.scatter(d_df.x2,d_df.y2,color='blue',s=10,label='Dolphin', alpha=0.4)\nplt.title('T-SNE (Finetune)')\nplt.legend(prop={'size': 12})\n\nplt.subplot(2, 2, 3)\nplt.scatter(w_df.x3,w_df.y3,color='red',s=10,label='Whale')\nplt.scatter(d_df.x3,d_df.y3,color='green',s=10,label='Dolphin', alpha=0.4)\nplt.title('UMAP')\nplt.legend(prop={'size': 12})\n\nplt.subplot(2, 2, 4)\nplt.scatter(w_df.x4,w_df.y4,color='red',s=10,label='Whale')\nplt.scatter(d_df.x4,d_df.y4,color='green',s=10,label='Dolphin', alpha=0.4)\nplt.title('UMAP (Finetune)')\nplt.legend(prop={'size': 12})\n\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-05T04:10:05.394889Z","iopub.execute_input":"2022-02-05T04:10:05.395181Z","iopub.status.idle":"2022-02-05T04:10:08.399801Z","shell.execute_reply.started":"2022-02-05T04:10:05.395148Z","shell.execute_reply":"2022-02-05T04:10:08.398824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Random **Whale/Dolphin** 4️⃣\nLet's plot some images from Whales and Dolphins. It seems that they lookalike.","metadata":{}},{"cell_type":"code","source":"ROW = 2\nCOL = 5\n\nX_DIV = 15; Y_DIV = 15;\nx_step = (x_max - x_min)/X_DIV\ny_step = (y_max - y_min)/Y_DIV\n\nw_df = df[df['class']=='whale']\nd_df = df[df['class']=='dolphin']\n\nfor it in range(5):\n    i = 0; i2=0; trial=0;\n    while i<ROW*COL:\n        trial+=1\n        if trial>50:\n            break\n        k = np.random.randint(0,X_DIV)\n        j = np.random.randint(0,Y_DIV)\n        xa_mx = k*x_step + x_min\n        xb_mx = (k+1)*x_step + x_min\n        ya_mx = j*y_step + y_min\n        yb_mx = (j+1)*y_step + y_min\n        df1 = w_df.loc[(w_df.x>xa_mx)&(w_df.x<xb_mx)&(w_df.y>ya_mx)&(w_df.y<yb_mx)]\n        df2 = d_df.loc[(d_df.x>xa_mx)&(d_df.x<xb_mx)&(d_df.y>ya_mx)&(d_df.y<yb_mx)]\n        i  = len(df1)\n        i2 = len(df2)\n    print(f'### RANDOM: {it}')\n    \n    print('>>TSNE:')\n    plot_tsne(w_df, d_df, labels=['Whale', 'Dolphin'], colors=['red', 'green'])\n\n    print('>>Whale:')\n    if i>=ROW*COL:\n        plot(df1, ROW, COL)\n    else:\n        print('Not Found')\n\n    print('>>Dolphin:')\n    if i2>=ROW*COL:\n        plot(df2, ROW, COL)\n    else:\n        print('Not Found')\n    print('\\n\\n')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-02T20:28:11.650263Z","iopub.status.idle":"2022-02-02T20:28:11.650988Z","shell.execute_reply.started":"2022-02-02T20:28:11.650733Z","shell.execute_reply":"2022-02-02T20:28:11.650759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Whale Species 5️⃣\nLet's look at the species of Whales in **T-SNE**.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(20,10))\nn_species = w_df.species.nunique()\ncolors = gen_colors(n=n_species)\n\nplt.subplot(1, 2, 1)\nfor i, species in enumerate(w_df.species.unique()):\n    s_df = w_df.query(\"species==@species\")\n    color = '#%02x%02x%02x'%tuple(int(c) for c in colors[i])\n    plt.scatter(s_df.x,s_df.y,s=10,color=color, label=species)\nplt.title('T-SNE')\nplt.legend(prop={'size': 10})\n\nplt.subplot(1, 2, 2)\nfor i, species in enumerate(w_df.species.unique()):\n    s_df = w_df.query(\"species==@species\")\n    color = '#%02x%02x%02x'%tuple(int(c) for c in colors[i])\n    plt.scatter(s_df.x2,s_df.y2,s=10,color=color, label=species)\nplt.title('T-SNE (Finetune)')\nplt.legend(prop={'size': 10})\n\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-04T15:37:03.535625Z","iopub.execute_input":"2022-02-04T15:37:03.535985Z","iopub.status.idle":"2022-02-04T15:37:05.197279Z","shell.execute_reply.started":"2022-02-04T15:37:03.53594Z","shell.execute_reply":"2022-02-04T15:37:05.196361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dolphin Species 6️⃣\nLet's look at the species of Dolphin in **T-SNE**.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(20,10))\nn_species = d_df.species.nunique()\ncolors = gen_colors(n=n_species)\n\nplt.subplot(1, 2, 1)\nfor i, species in enumerate(d_df.species.unique()):\n    s_df = d_df.query(\"species==@species\")\n    color = '#%02x%02x%02x'%tuple(int(c) for c in colors[i])\n    plt.scatter(s_df.x,s_df.y,s=10,color=color, label=species)\nplt.title('T-SNE')\nplt.legend(prop={'size': 10})\n\nplt.subplot(1, 2, 2)\nfor i, species in enumerate(d_df.species.unique()):\n    s_df = d_df.query(\"species==@species\")\n    color = '#%02x%02x%02x'%tuple(int(c) for c in colors[i])\n    plt.scatter(s_df.x2,s_df.y2,s=10,color=color, label=species)\nplt.title('T-SNE (Finetune)')\nplt.legend(prop={'size': 10})\n\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-04T15:40:39.139702Z","iopub.execute_input":"2022-02-04T15:40:39.140105Z","iopub.status.idle":"2022-02-04T15:40:40.577246Z","shell.execute_reply.started":"2022-02-04T15:40:39.14007Z","shell.execute_reply":"2022-02-04T15:40:40.572036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion 🏁\n* It can be noticed that **train** and **test** distirubtion are very close to earch other.\n* Even though `new_individual` never appears in **train** data, they have similar distribution as **train** data.\n* It is also possible that there aren't many samples of `new_individual` in the **test** data. Hence, their distribution wasn't much visible in **T-SNE** plot.\n* Some Whales are very distinct comparing Dolphins whereas most of the Dolphins resemblance Whales.\n* The `killer`, `southern` and `pilot` Whales dominates over other species.\n* The `bottlenose` and `dusky` Dolphins dominates over other speceis.","metadata":{}},{"cell_type":"markdown","source":"# Reference 💡\n* [RAPIDS cuML kNN - Find Duplicates](https://www.kaggle.com/cdeotte/rapids-cuml-knn-find-duplicates) by Chris Deotte\n* [[Pytorch] ArcFace + GeM Pooling Starter](https://www.kaggle.com/debarshichanda/pytorch-arcface-gem-pooling-starter) by Debarsh Chanda","metadata":{}},{"cell_type":"markdown","source":"# Remove Files ✂️","metadata":{}},{"cell_type":"code","source":"!rm -rf ./wandb","metadata":{"execution":{"iopub.status.busy":"2022-02-02T20:28:11.656198Z","iopub.status.idle":"2022-02-02T20:28:11.656598Z","shell.execute_reply.started":"2022-02-02T20:28:11.656378Z","shell.execute_reply":"2022-02-02T20:28:11.656401Z"},"trusted":true},"execution_count":null,"outputs":[]}]}