{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Visualize Embedding using W&B Embedding Projector\n\nW&B has recently launched it's own embedding projector using which we can easily visualize the embedding space in 2D using TSNE, UMAP, or PCA techniques. \n\nYou can refer the [documentation](https://docs.wandb.ai/ref/app/features/panels/weave/embedding-projector) for more information. ","metadata":{}},{"cell_type":"markdown","source":"# Imports and Setup","metadata":{}},{"cell_type":"code","source":"import os\nimport gc\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nfrom functools import partial\nfrom argparse import Namespace\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-04T19:56:00.517935Z","iopub.execute_input":"2022-02-04T19:56:00.518372Z","iopub.status.idle":"2022-02-04T19:56:06.837901Z","shell.execute_reply.started":"2022-02-04T19:56:00.518278Z","shell.execute_reply":"2022-02-04T19:56:06.836795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import wandb\nwandb.login()","metadata":{"execution":{"iopub.status.busy":"2022-02-03T20:29:11.356597Z","iopub.execute_input":"2022-02-03T20:29:11.356808Z","iopub.status.idle":"2022-02-03T20:29:32.440672Z","shell.execute_reply.started":"2022-02-03T20:29:11.356783Z","shell.execute_reply":"2022-02-03T20:29:32.439944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"IMG_DIR = '../input/jpeg-happywhale-128x128/train_images-128-128/train_images-128-128/'\ndf = pd.read_csv('../input/happy-whale-and-dolphin/train.csv')\n\ndef source_path(row):\n    return f'{IMG_DIR}/{row.image}'\n\ndf['img_path'] = df.apply(lambda row: source_path(row), axis=1)\n\n# Fix wrong label issue\ndf = df.replace({'bottlenose_dolpin': 'bottlenose_dolphin',\n                 'kiler_whale': 'killer_whale'})\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-03T20:29:35.087350Z","iopub.execute_input":"2022-02-03T20:29:35.088117Z","iopub.status.idle":"2022-02-03T20:29:36.086941Z","shell.execute_reply.started":"2022-02-03T20:29:35.088079Z","shell.execute_reply":"2022-02-03T20:29:36.086209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels_list = list(df.species.unique())\nlabel2ids = {label: idx for idx, label in enumerate(labels_list)}\nid2labels = {val:key for key, val in label2ids.items()}\n\n# Add a column of label ids\ndf['target'] = df['species'].replace(label2ids)","metadata":{"execution":{"iopub.status.busy":"2022-02-03T20:29:38.141615Z","iopub.execute_input":"2022-02-03T20:29:38.142315Z","iopub.status.idle":"2022-02-03T20:29:38.229289Z","shell.execute_reply.started":"2022-02-03T20:29:38.142276Z","shell.execute_reply":"2022-02-03T20:29:38.228584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Config","metadata":{}},{"cell_type":"code","source":"args = Namespace(\n    labels = label2ids,\n    num_labels = len(label2ids),\n    image_height = 128,\n    image_width = 128,\n    resize=False,\n    batch_size = 128,\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-03T20:40:18.917763Z","iopub.execute_input":"2022-02-03T20:40:18.918035Z","iopub.status.idle":"2022-02-03T20:40:18.923730Z","shell.execute_reply.started":"2022-02-03T20:40:18.918007Z","shell.execute_reply":"2022-02-03T20:40:18.922392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataloader","metadata":{}},{"cell_type":"code","source":"AUTOTUNE = tf.data.AUTOTUNE\n\n@tf.function\ndef decode_image(img, resize=True):\n    # convert the compressed string to a 3D uint8 tensor\n    img = tf.image.decode_jpeg(img, channels=3)\n    # Normalize image\n    img = tf.image.convert_image_dtype(img, dtype=tf.float32)\n    # resize the image to the desired size\n    if resize:\n        img = tf.image.resize(img, [args.image_height, args.image_width], \n                              method='bicubic', preserve_aspect_ratio=False)\n        img = tf.clip_by_value(img, 0.0, 1.0)\n\n    return img\n\n@tf.function\ndef parse_data(df_dict, resize=True):\n    # Parse Image\n    image = tf.io.read_file(df_dict['img_path'])\n    image = decode_image(image, resize)\n    # Parse Target\n    label = df_dict['target']\n    label = tf.one_hot(indices=label, depth=args.num_labels)\n    \n    return image, label","metadata":{"execution":{"iopub.status.busy":"2022-02-03T20:36:59.815273Z","iopub.execute_input":"2022-02-03T20:36:59.815541Z","iopub.status.idle":"2022-02-03T20:36:59.825947Z","shell.execute_reply.started":"2022-02-03T20:36:59.815512Z","shell.execute_reply":"2022-02-03T20:36:59.825244Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_dataloader(df):\n    dataloader = tf.data.Dataset.from_tensor_slices(dict(df))\n\n    # Training Dataset\n    dataloader = (\n        dataloader\n        .map(partial(parse_data, resize=args.resize), num_parallel_calls=AUTOTUNE)\n        .batch(args.batch_size)\n        .prefetch(AUTOTUNE)\n    )\n    \n    return dataloader","metadata":{"execution":{"iopub.status.busy":"2022-02-03T22:00:46.944112Z","iopub.execute_input":"2022-02-03T22:00:46.944386Z","iopub.status.idle":"2022-02-03T22:00:46.950046Z","shell.execute_reply.started":"2022-02-03T22:00:46.944347Z","shell.execute_reply":"2022-02-03T22:00:46.949174Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_label_name(one_hot_label):\n    label = np.argmax(one_hot_label, axis=0)\n    return id2labels[label]\n\ndef show_batch(image_batch, label_batch):\n  plt.figure(figsize=(20,20))\n  for n in range(25):\n      ax = plt.subplot(5,5,n+1)\n      plt.imshow(image_batch[n])\n      plt.title(get_label_name(label_batch[n].numpy()))\n      plt.axis('off')\n\n# Sanity Check\ndataloader = get_dataloader(df)\n\nsample_imgs, sample_labels = next(iter(dataloader))\n    \nshow_batch(sample_imgs, sample_labels)","metadata":{"execution":{"iopub.status.busy":"2022-02-03T22:01:10.995080Z","iopub.execute_input":"2022-02-03T22:01:10.995357Z","iopub.status.idle":"2022-02-03T22:01:12.978152Z","shell.execute_reply.started":"2022-02-03T22:01:10.995327Z","shell.execute_reply":"2022-02-03T22:01:12.977496Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Log the Images as W&B Table [Optional]\n\nNote that running the cell below might take some time as you are uploading images onto W&B. Later on we will be using reference to the uploaded images, keep reading. :D\n\nIf you are okay with images of resolution 128x128 for analysis you can ignore to run the cell below. Instead go to section 1.1 to download the table I have logged. \n\nHere's the link to the logged table: https://wandb.ai/ayut/happywhale/artifacts/run_table/run-34086otd-data_table/d529d0d65d18ff406099/files/data_table.table.json","metadata":{}},{"cell_type":"code","source":"LOG_FULL_TABLE = False","metadata":{"execution":{"iopub.status.busy":"2022-02-03T20:56:15.581751Z","iopub.status.idle":"2022-02-03T20:56:15.582300Z","shell.execute_reply.started":"2022-02-03T20:56:15.582070Z","shell.execute_reply":"2022-02-03T20:56:15.582096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if LOG_FULL_TABLE:\n    # Initialize a W&B run\n    run = wandb.init(project='happywhale')\n\n    # Initialize an empty W&B Table\n    data_table = wandb.Table(columns=['individual_id', 'image'])\n\n    # for unique_id, tmp_df in tqdm(df.groupby('individual_id')):\n    for i in tqdm(range(len(df))):\n        row = df.loc[i]\n        # Add data to the table row-wise\n        data_table.add_data(row.individual_id,\n                            wandb.Image(f'{IMG_DIR}/{row.image}'))\n\n    # Log the table\n    wandb.log({'data_table': data_table})\n\n    # Finish the run\n    wandb.finish()","metadata":{"execution":{"iopub.status.busy":"2022-02-03T17:42:20.008843Z","iopub.execute_input":"2022-02-03T17:42:20.009533Z","iopub.status.idle":"2022-02-03T18:17:30.421887Z","shell.execute_reply.started":"2022-02-03T17:42:20.009494Z","shell.execute_reply":"2022-02-03T18:17:30.420917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1.1 Grab the Table and Get Image Index [Optional]\n\n","metadata":{}},{"cell_type":"code","source":"# Initialize a W&B run\nrun = wandb.init(project='happywhale')\n\n# Use the already logged dataset\ndata_artifact = run.use_artifact('ayut/happywhale/run-34086otd-data_table:latest')\n\n# Get the data_table to access the data\ndata_table = data_artifact.get(\"data_table\")\n\nwandb.finish()","metadata":{"execution":{"iopub.status.busy":"2022-02-03T21:03:17.840882Z","iopub.execute_input":"2022-02-03T21:03:17.841544Z","iopub.status.idle":"2022-02-03T21:03:23.000167Z","shell.execute_reply.started":"2022-02-03T21:03:17.841415Z","shell.execute_reply":"2022-02-03T21:03:22.999386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1.2 Sample the data\n\nWe will not be logging the embedding for the entire dataset but a stratified sample.","metadata":{}},{"cell_type":"code","source":"_, sampled_df = train_test_split(df, test_size=0.33, random_state=42, stratify=df.species.values)\nsampled_ids = sampled_df.index\n\nprint(sampled_df.shape)","metadata":{"execution":{"iopub.status.busy":"2022-02-03T21:07:09.514975Z","iopub.execute_input":"2022-02-03T21:07:09.515257Z","iopub.status.idle":"2022-02-03T21:07:09.594990Z","shell.execute_reply.started":"2022-02-03T21:07:09.515215Z","shell.execute_reply":"2022-02-03T21:07:09.594271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sampledloader = get_dataloader(sampled_df)","metadata":{"execution":{"iopub.status.busy":"2022-02-03T22:01:29.422154Z","iopub.execute_input":"2022-02-03T22:01:29.422421Z","iopub.status.idle":"2022-02-03T22:01:29.450400Z","shell.execute_reply.started":"2022-02-03T22:01:29.422391Z","shell.execute_reply":"2022-02-03T22:01:29.449684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Pretrained Weights\n\nIt would be interesting to look at the 2D projection of embedding from pretrained weights. The model was trained on ImageNet dataset which has natural images. Since the images in this competition have some domain similarity it would be interesting to look at the embedding projection.","metadata":{}},{"cell_type":"code","source":"EMBEDDING_DIM = 1280","metadata":{"execution":{"iopub.status.busy":"2022-02-03T21:54:39.060438Z","iopub.execute_input":"2022-02-03T21:54:39.060723Z","iopub.status.idle":"2022-02-03T21:54:39.068519Z","shell.execute_reply.started":"2022-02-03T21:54:39.060690Z","shell.execute_reply":"2022-02-03T21:54:39.067633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_pretrained_model():\n    base_model = tf.keras.applications.EfficientNetB0(\n        input_shape=(args.image_height, args.image_width, 3),\n        include_top=False, \n        weights='imagenet'\n    )\n    \n    base_model.trainabe = False\n    feature = tf.keras.layers.GlobalAveragePooling2D()(base_model.output)\n\n    return tf.keras.models.Model(base_model.input, feature)\n\ntf.keras.backend.clear_session()\npretrained_model = get_pretrained_model()\n# pretrained_model.summary()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-03T21:48:31.557618Z","iopub.execute_input":"2022-02-03T21:48:31.558120Z","iopub.status.idle":"2022-02-03T21:48:33.005141Z","shell.execute_reply.started":"2022-02-03T21:48:31.558081Z","shell.execute_reply":"2022-02-03T21:48:33.004418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Initialize a W&B run\nrun = wandb.init(project='happywhale')\n\n# Initialize an empty W&B Table\nembedding_cols = [f'e_{i}' for i in range(EMBEDDING_DIM)]\npretrained_embedding_table = wandb.Table(columns=['image', 'truth']+embedding_cols)\n\n# Get embedding\nprint('Getting embedding...')\nembedding = pretrained_model.predict(sampledloader)\n\n# Get the row number associated to the logged table\nfor i, table_row_id in tqdm(enumerate(sampled_ids)):\n    # Add data to the table row-wise\n    pretrained_embedding_table.add_data(\n        wandb.Image(data_table.data[table_row_id][1]), # Notice here!!!\n        sampled_df.loc[table_row_id].species,\n        *embedding[i]\n    )\n\n# Log the table\nwandb.log({'embedding_table': pretrained_embedding_table})\n\n# Finish the run\nwandb.finish()","metadata":{"execution":{"iopub.status.busy":"2022-02-03T22:12:35.565586Z","iopub.execute_input":"2022-02-03T22:12:35.566437Z","iopub.status.idle":"2022-02-03T22:26:50.849175Z","shell.execute_reply.started":"2022-02-03T22:12:35.566392Z","shell.execute_reply":"2022-02-03T22:26:50.848399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. FineTuned Weights (Model)\n\nNext up I have trained EfficientNetB0 on this competition's dataset and the finetuned weights are attached as Kaggle dataset. It was trained for 30 epochs with 5 fold stratified split of the dataset. No fancy augmentation or training techniques was used. `ReduceLRonPlateua` was used as a learning rate scheduler while training. \n\nYou can find the W&B run page for the training metrics here: https://wandb.ai/ayut/happywhale/groups/effnetb0/workspace\n\n![img](https://i.imgur.com/RCqv3Zz.png)\n\n","metadata":{}},{"cell_type":"code","source":"!tar -xvf ../input/happywhale-supervised/model.tar\n!ls","metadata":{"execution":{"iopub.status.busy":"2022-02-03T21:22:07.775333Z","iopub.execute_input":"2022-02-03T21:22:07.776119Z","iopub.status.idle":"2022-02-03T21:22:12.077081Z","shell.execute_reply.started":"2022-02-03T21:22:07.776075Z","shell.execute_reply":"2022-02-03T21:22:12.076251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_finetuned_model(model_path):\n    finetuned_model = tf.keras.models.load_model(model_path)\n    feature_extractor = tf.keras.models.Model(\n                            finetuned_model.input,\n                            finetuned_model.get_layer('global_average_pooling2d').output\n                        )\n    return feature_extractor","metadata":{"execution":{"iopub.status.busy":"2022-02-03T21:33:14.415609Z","iopub.execute_input":"2022-02-03T21:33:14.416592Z","iopub.status.idle":"2022-02-03T21:33:14.421381Z","shell.execute_reply.started":"2022-02-03T21:33:14.416541Z","shell.execute_reply":"2022-02-03T21:33:14.420677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.keras.backend.clear_session()\nfinetuned_model = get_finetuned_model('./model_0.h5')\nfinetuned_model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-02-03T21:33:16.483040Z","iopub.execute_input":"2022-02-03T21:33:16.483591Z","iopub.status.idle":"2022-02-03T21:33:19.678234Z","shell.execute_reply.started":"2022-02-03T21:33:16.483551Z","shell.execute_reply":"2022-02-03T21:33:19.677509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Initialize a W&B run\nrun = wandb.init(project='happywhale')\n\n# Initialize an empty W&B Table\nembedding_cols = [f'e_{i}' for i in range(EMBEDDING_DIM)]\nfinetuned_embedding_table = wandb.Table(columns=['image', 'truth']+embedding_cols)\n\n# Get embedding\nprint('Getting embedding...')\nembeddings = []\nfor i in tqdm(range(5)):\n    finetuned_model = get_finetuned_model(f'./model_{i}.h5')\n    embedding = finetuned_model.predict(sampledloader)\n    embeddings.append(embedding)\n    \nembeddings = np.mean(embeddings, axis=0)\n\n# Get the row number associated to the logged table\nfor i, table_row_id in tqdm(enumerate(sampled_ids)):\n    # Add data to the table row-wise\n    finetuned_embedding_table.add_data(\n        wandb.Image(data_table.data[table_row_id][1]), # Notice here!!!\n        sampled_df.loc[table_row_id].species,\n        *embeddings[i]\n    )\n\n# Log the table\nwandb.log({'embedding_table': finetuned_embedding_table})\n\n# Finish the run\nwandb.finish()","metadata":{"execution":{"iopub.status.busy":"2022-02-03T22:35:03.737422Z","iopub.execute_input":"2022-02-03T22:35:03.738178Z","iopub.status.idle":"2022-02-03T22:49:59.753177Z","shell.execute_reply.started":"2022-02-03T22:35:03.738141Z","shell.execute_reply":"2022-02-03T22:49:59.752436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Embeddings\n\nNow that the embeddings are logged onto W&B, we can easily visualize the 2D projection of the embeddings using PCA, TSNE, and UMAP with just few clicks. \n\nHere's the [documentation](https://docs.wandb.ai/ref/app/features/panels/weave/embedding-projector) page. ","metadata":{}},{"cell_type":"markdown","source":"# 4.0 How to Easily Create Embeddings\n\nOnce you have logged the embedding as shown above, open the W&B run page and follow the following steps:\n* Click on \"Merge Tables: 2D Projection: Plot\"\n* It will automatically create the embedding (either one of PCA, t-SNE, or UMAP).\n* Click on the \"gear\" icon to open the controls. \n* Select the dimentionality reduction algorightm. W&B will automatically compute the 2D projection. \n* You can also play with the hyperparameters of the algorithms. \n* IF you hover your mouse over the dots, you can see the logged images as well, which is a very handy way to visually validate the projection.\n\nCheck out the video below to follow along. \n\n![img](https://imgur.com/VmhtPkd.mp4)\n","metadata":{}},{"cell_type":"markdown","source":"# 4.1 PCA\n\nPretrained:\n\n![img](https://i.imgur.com/81ixxTW.png)\n\nFinetuned:\n\n![img](https://i.imgur.com/OIbadx9.png)","metadata":{}},{"cell_type":"markdown","source":"# 4.2 TSNE\n\nPretrained:\n\n![img](https://i.imgur.com/nyKynT0.png) \n\nFinetuned: \n\n![img](https://i.imgur.com/j21gN2m.png)","metadata":{}},{"cell_type":"markdown","source":"# 4.3 UMAP\n\nPretrained:\n\n![img](https://i.imgur.com/959vjPK.png)\n\nFinetuned:\n\n![img](https://i.imgur.com/hJV0DtV.png)\n\nBy changing the `Neighbors` parameter to 5. The UMAP shows better clusters.\n\n![img](https://i.imgur.com/Hl301zy.png)","metadata":{}}]}