{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"![](https://api.time.com/wp-content/uploads/2015/04/humpback-whale.jpg)\n**A Breaching HumpBack (image credits: time)**\n","metadata":{}},{"cell_type":"markdown","source":"**Whales are Intelligent and Emotional beings. Here is a recommended watch on humpback,where the humpback saved the life of a biologist from the threat of a lurking tiger shark**","metadata":{}},{"cell_type":"code","source":"from IPython.display import YouTubeVideo\n\nYouTubeVideo(id='OXNCCdcBhcY',width=1000,height=600, allow_autoplay=False)\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-07T06:01:39.377847Z","iopub.execute_input":"2022-02-07T06:01:39.378640Z","iopub.status.idle":"2022-02-07T06:01:39.499070Z","shell.execute_reply.started":"2022-02-07T06:01:39.378593Z","shell.execute_reply":"2022-02-07T06:01:39.498168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# About \n\n**We use fingerprints and facial recognition to identify people, but can we use similar approaches with animals? In fact, researchers manually track marine life by the shape and markings on their tails, dorsal fins, heads and other body parts. Identification by natural markings via photographs—known as photo-ID—is a powerful tool for marine mammal science. It allows individual animals to be tracked over time and enables assessments of population status and trends.**\n![](https://storage.googleapis.com/kaggle-media/competitions/Happywhale/AU%20Kaggle%20Competition%20Description%20Image-03.jpg)\n\n**Algorithms developed in this competition will be implemented in Happywhale, a research collaboration and citizen science web platform. Its mission is to increase global understanding and caring for marine environments through high quality conservation science and education.**\n\n\n**In this competition, you’ll develop a model to match individual whales and dolphins by unique—but often subtle—characteristics of their natural markings. You'll pay particular attention to dorsal fins and lateral body views in image sets from a multi-species dataset built by 28 research institutions. The best submissions will suggest photo-ID solutions that are fast and accurate.**","metadata":{}},{"cell_type":"markdown","source":"> **To Summarize, the aim of the competitions is to develop algorithm to identify individual whales and dolphins , based on thier dorsal fin and lateral body images.**","metadata":{}},{"cell_type":"markdown","source":"# References and Resources \n* Fantastic Notebook by @ruchi798 : https://www.kaggle.com/ruchi798/and-identification-eda-augmentation\n* Fantastic notebook by  @awsaf49 : https://www.kaggle.com/awsaf49/happywhale-data-distribution\n","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"import os \nimport gc\nimport matplotlib.pyplot as plt \nimport seaborn as sns\nimport numpy as np \nimport pandas as pd \nfrom glob import glob\nfrom tqdm import tqdm\n#colored print\nfrom termcolor import colored\n\n#deep learning \nimport tensorflow as tf \n\n\n\n\ntry:  # if gpu is ON\n    from cuml import TSNE, UMAP   # cuml is gpu accelerated library \nexcept:\n    from sklearn.manifold import TSNE # for cpu\n    from umap import UMAP","metadata":{"execution":{"iopub.status.busy":"2022-02-07T07:36:27.448652Z","iopub.execute_input":"2022-02-07T07:36:27.449146Z","iopub.status.idle":"2022-02-07T07:36:42.972144Z","shell.execute_reply.started":"2022-02-07T07:36:27.449098Z","shell.execute_reply":"2022-02-07T07:36:42.971405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Setting a Configuration Object**\n    \n\nDoing this as it will be easy to change, track or retrieve the parameters(like say image height and width) easily.\n","metadata":{}},{"cell_type":"code","source":"class config:\n    def __init__(self,\n                 seed=7,\n                 img_size=(264,264),\n                 batch=32):\n        self.seed = seed\n        self.img_size= img_size  #image size \n        self.batch_size = batch\n    \n    def set_seed(self):\n        '''set seed for reproduciblity'''\n\n        tf.random.set_seed(self.seed)\n        os.environ['PYTHONHASHSEED'] = str(self.seed)\n        np.random.seed(self.seed)\n        print(f'Setting Random Seed  to {self.seed}')\n\ncfg = config()\ncfg.set_seed()","metadata":{"execution":{"iopub.status.busy":"2022-02-07T05:37:34.833591Z","iopub.execute_input":"2022-02-07T05:37:34.833857Z","iopub.status.idle":"2022-02-07T05:37:34.842776Z","shell.execute_reply.started":"2022-02-07T05:37:34.833828Z","shell.execute_reply":"2022-02-07T05:37:34.842041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Helper Functions","metadata":{}},{"cell_type":"code","source":"def load_image(path,\n               img_size=None,\n               expand_dims=False,\n               rescale=False):\n    '''load image at given path'''\n    img= tf.io.read_file(path)             # load image\n    img = tf.image.decode_image(img, expand_animations = False)   \n    img = tf.cast(img,float)\n    \n    if rescale:\n        img = img /255.0                           # convert img pixels in range [0,1]\n    if img_size:\n        img = tf.image.resize(img,\n                              size=img_size)  # resize image \n    if expand_dims:\n        img = tf.expand_dims(img,axis=0)\n        \n    \n    # handle single channel images \n    if img.shape[2] == 1 or len(img.shape)==2: # if image has single band (b and w), stack that band together                 \n#         tf.print(img.shape)\n        if img.shape[2]:\n            img = tf.squeeze(img,axis=-1) # squeeze (n,n,1) to (n,n)\n        \n        #stack the single band to create n,n,3 image\n        img = tf.stack([img,img,img],axis=-1)\n#         tf.print(img.shape)\n        \n        \n    return img\n\n\ndef plot_image_grid(image_list,\n                    label_list,\n                    sample_images=False,\n                    num_images=12,\n                    pre_title='class',\n                    num_img_per_row=3,\n                    cmap=None,\n                    img_h_w=3):\n    '''viz images from a list of images and labels\n    INPUTS:\n    image_list: a list of images to be plotted,\n    label_list: a list of correspomding image labels'''\n    \n\n\n    #number of img rows\n    n_row= num_images//num_img_per_row\n\n    plt.subplots(n_row,num_img_per_row,figsize=(img_h_w*num_img_per_row,(img_h_w-2) *n_row))\n\n    if sample_images:\n    #select_random images \n        sampled_ids = random.choices(np.arange(0,len(image_list)),k=num_images)\n\n        for i,idx in enumerate(sampled_ids):\n\n            img = image_list[idx]\n            label = label_list[i]\n            plt.subplot(n_row,num_img_per_row,i+1)\n            plt.title(f'{pre_title} - {label}')\n            plt.axis('off')\n            plt.imshow(img,cmap=cmap)\n    else:\n        for i,img in enumerate(image_list):\n\n            label = label_list[i]\n            plt.subplot(n_row,num_img_per_row,i+1)\n            plt.title(f'{pre_title} - {label}')\n            plt.axis('off')\n            plt.imshow(img,cmap=cmap)\n\n            # break the loop \n            if i==num_images-1 :\n                  break \n\n    #show\n    plt.tight_layout()\n    plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-07T05:37:34.844552Z","iopub.execute_input":"2022-02-07T05:37:34.844804Z","iopub.status.idle":"2022-02-07T05:37:34.862177Z","shell.execute_reply.started":"2022-02-07T05:37:34.844777Z","shell.execute_reply":"2022-02-07T05:37:34.861226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading Data","metadata":{}},{"cell_type":"markdown","source":"**Train Csv Description**\n\n    image : Image Path in train directory.\n    \n    species: Species name of the whale or the dolphin in the Image.\n    \n    individual_id: Unique Id associated with the individual(something like a Tag).","metadata":{}},{"cell_type":"code","source":"train_dir = '../input/happy-whale-and-dolphin/train_images/' #train directory\ntest_dir = '../input/happy-whale-and-dolphin/test_images/'   #test directory\n\n#add data dirs to config for ease of access\ncfg.train = train_dir\ncfg.test  = test_dir\n\n\ntrain = pd.read_csv('../input/happy-whale-and-dolphin/train.csv') #train csv\ntrain.drop_duplicates(inplace=True)\n\n\n#adding path to train \ntrain['path'] = train_dir + train['image']\n\n#there are some duplicates in species \n\n# fixing duplicate labels\ntrain['species'] = train['species'].str.replace('bottlenose_dolpin','bottlenose_dolphin')\ntrain['species'] = train['species'].str.replace('kiler_whale','killer_whale')\n\n#beluga and globis are whales, so replacing them as that\ntrain[\"species\"].replace({\"beluga\": \"beluga_whale\",\n                         \"globis\": \"globis_whale\"},\n                          inplace=True)\n\n#there is a typo in bottlenose_dolpin,correcting that \ntrain[\"species\"].replace({'bottlenose_dolpin':'bottlenose_dolphin'},\n                        inplace=True)\n\n\n\nsample_sub = pd.read_csv('../input/happy-whale-and-dolphin/sample_submission.csv') #sample submission \n\n\n\nprint_color = 'green'\n\nprint(colored(f'Number of Images in train directory {train.image.nunique()}',print_color))\nprint(colored(f'Number of Images in test directory {len(os.listdir(test_dir))}',print_color))\nprint(colored(f'Number of Unique Individuals in train directory {train.individual_id.nunique()}',print_color))\nprint(colored(f'Number of Unique Species in train directory {train.species.nunique()}',print_color))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-07T05:37:34.864403Z","iopub.execute_input":"2022-02-07T05:37:34.865128Z","iopub.status.idle":"2022-02-07T05:37:35.617736Z","shell.execute_reply.started":"2022-02-07T05:37:34.865085Z","shell.execute_reply":"2022-02-07T05:37:35.616730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Checking the Distribution of Dolphin and Whales Species ","metadata":{}},{"cell_type":"markdown","source":"**Whales**","metadata":{}},{"cell_type":"code","source":"\n\nwhales = [x for x in train.species.unique() if 'whale' in x]\n\n\nprint(f'Number of Unique Whale Species in dataset {len(whales)}')\n\nwhls = train[train['species'].isin(whales)]\ntrain.loc[train['species'].isin(whales),'type'] = 'whales'\n\nprint(colored(f'Number of Whales Images in train directory {whls.image.nunique()}',print_color))\nprint(colored(f'Number of Unique Whales in train directory {whls.individual_id.nunique()}',print_color))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-07T06:12:47.121210Z","iopub.execute_input":"2022-02-07T06:12:47.121484Z","iopub.status.idle":"2022-02-07T06:12:47.157878Z","shell.execute_reply.started":"2022-02-07T06:12:47.121454Z","shell.execute_reply":"2022-02-07T06:12:47.156989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20,10))\nplt.yticks(fontsize=16)\nsns.countplot(y=\"species\",\n              data=whls,\n              order=whls.iloc[0:][\"species\"].value_counts().index,\n              palette=\"GnBu_r\",\n              linewidth=3)\nplt.title(\"Whale Species Distribution\",font=\"Serif\", size=20,color='k')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-07T06:12:47.301355Z","iopub.execute_input":"2022-02-07T06:12:47.301618Z","iopub.status.idle":"2022-02-07T06:12:47.958853Z","shell.execute_reply.started":"2022-02-07T06:12:47.301591Z","shell.execute_reply":"2022-02-07T06:12:47.957949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Dolphins**","metadata":{}},{"cell_type":"code","source":"dolphin = [x for x in train.species.unique() if 'dolphin' in x]\n\nprint(f'Number of Unique Dolphin Species in dataset {len(dolphin)}')\n\ndlps = train[train['species'].isin(dolphin)]\ntrain.loc[train['species'].isin(dolphin),'type'] = 'dolphins'\n\nprint(colored(f'Number of dolphin Images in train directory {dlps.image.nunique()}',print_color))\nprint(colored(f'Number of Unique Dolphins in train directory {dlps.individual_id.nunique()}',print_color))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-07T06:12:47.960731Z","iopub.execute_input":"2022-02-07T06:12:47.960998Z","iopub.status.idle":"2022-02-07T06:12:47.992514Z","shell.execute_reply.started":"2022-02-07T06:12:47.960969Z","shell.execute_reply":"2022-02-07T06:12:47.991833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20,10))\nplt.yticks(fontsize=16)\nsns.countplot(y=\"species\",\n              data=dlps,\n              order=dlps.iloc[0:][\"species\"].value_counts().index,\n              palette=\"GnBu_r\",\n              linewidth=3)\nplt.title(\"Dolphin Species Distribution\",font=\"Serif\", size=20,color='Green')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-07T06:12:47.993593Z","iopub.execute_input":"2022-02-07T06:12:47.993807Z","iopub.status.idle":"2022-02-07T06:12:48.236498Z","shell.execute_reply.started":"2022-02-07T06:12:47.993781Z","shell.execute_reply":"2022-02-07T06:12:48.235517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualizing Whale Species","metadata":{}},{"cell_type":"code","source":"sample_ids = []\nsample_sp  =[]\n\nfor species in whales:\n    sample = whls[whls['species']==species].sample(3)\n    sample_ids.extend(sample.path) # sample paths \n    sample_sp.extend(sample.species) # sample species \n    \n\nwhale_imgs = [load_image(path,rescale =True) for path in sample_ids] # load sample images to view \n\nplot_image_grid(whale_imgs[:12],\n                sample_sp[:12],\n                sample_images=False,\n                num_images=12,\n                pre_title='',\n                num_img_per_row=3,\n                cmap=None,\n                img_h_w=5)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-07T06:12:48.331014Z","iopub.execute_input":"2022-02-07T06:12:48.331284Z","iopub.status.idle":"2022-02-07T06:12:58.920546Z","shell.execute_reply.started":"2022-02-07T06:12:48.331255Z","shell.execute_reply":"2022-02-07T06:12:58.919578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nplot_image_grid(whale_imgs[12:24],\n                sample_sp[12:24],\n                sample_images=False,\n                num_images=12,\n                pre_title='',\n                num_img_per_row=3,\n                cmap=None,\n                img_h_w=5)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-07T06:12:58.922486Z","iopub.execute_input":"2022-02-07T06:12:58.922801Z","iopub.status.idle":"2022-02-07T06:13:22.048059Z","shell.execute_reply.started":"2022-02-07T06:12:58.922763Z","shell.execute_reply":"2022-02-07T06:13:22.046893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nplot_image_grid(whale_imgs[24:36],\n                sample_sp[24:36],\n                sample_images=False,\n                num_images=12,\n                pre_title='',\n                num_img_per_row=3,\n                cmap=None,\n                img_h_w=5)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-07T06:13:22.049770Z","iopub.execute_input":"2022-02-07T06:13:22.050143Z","iopub.status.idle":"2022-02-07T06:13:34.270794Z","shell.execute_reply.started":"2022-02-07T06:13:22.050094Z","shell.execute_reply":"2022-02-07T06:13:34.269701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nplot_image_grid(whale_imgs[36:54],\n                sample_sp[36:54],\n                sample_images=False,\n                num_images=18,\n                pre_title='',\n                num_img_per_row=3,\n                cmap=None,\n                img_h_w=5)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-07T06:13:34.273436Z","iopub.execute_input":"2022-02-07T06:13:34.273773Z","iopub.status.idle":"2022-02-07T06:13:46.943689Z","shell.execute_reply.started":"2022-02-07T06:13:34.273719Z","shell.execute_reply":"2022-02-07T06:13:46.942635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#  Visualizing Dolphin Species","metadata":{}},{"cell_type":"code","source":"sample_ids = []\nsample_sp  =[]\n\nfor species in dolphin:\n    sample = dlps[dlps['species']==species].sample(3)\n    sample_ids.extend(sample.path) # sample paths \n    sample_sp.extend(sample.species) # sample species \n    \n\ndolp_imgs = [load_image(path,rescale =True) for path in sample_ids]\n\nplot_image_grid(dolp_imgs[:15],\n                sample_sp[:15],\n                sample_images=False,\n                num_images=15,\n                pre_title='',\n                num_img_per_row=3,\n                cmap=None,\n                img_h_w=5)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-07T06:13:46.945006Z","iopub.execute_input":"2022-02-07T06:13:46.945257Z","iopub.status.idle":"2022-02-07T06:13:59.650929Z","shell.execute_reply.started":"2022-02-07T06:13:46.945229Z","shell.execute_reply":"2022-02-07T06:13:59.649172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nplot_image_grid(dolp_imgs[15:],\n                sample_sp[15:],\n                sample_images=False,\n                num_images=15,\n                pre_title='',\n                num_img_per_row=3,\n                cmap=None,\n                img_h_w=5)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-07T06:13:59.652406Z","iopub.execute_input":"2022-02-07T06:13:59.652659Z","iopub.status.idle":"2022-02-07T06:14:09.174724Z","shell.execute_reply.started":"2022-02-07T06:13:59.652630Z","shell.execute_reply":"2022-02-07T06:14:09.174096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#delete imgs loaded in RAM\ndel whale_imgs,dolp_imgs,sample_ids,sample_sp,whls,dlps; gc.collect()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-07T06:14:09.175915Z","iopub.execute_input":"2022-02-07T06:14:09.176308Z","iopub.status.idle":"2022-02-07T06:14:09.588349Z","shell.execute_reply.started":"2022-02-07T06:14:09.176269Z","shell.execute_reply":"2022-02-07T06:14:09.587779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Getting Image Embeddings\n\n    What are these Embeddings? Well we just predict on the Images to a Pretrained Convolutional Nueral Network (without the final softmax layer), which gives us the representations that the model has extracted from the Images.","metadata":{}},{"cell_type":"code","source":"#loading EfficientNet B0 pretrained on ImageNet\nefnet = tf.keras.applications.EfficientNetB0(include_top=False, # exclude the final prediction layer \n                                            weights='imagenet')# trained on imagenet \n                                            ","metadata":{"execution":{"iopub.status.busy":"2022-02-07T05:38:53.904808Z","iopub.execute_input":"2022-02-07T05:38:53.905366Z","iopub.status.idle":"2022-02-07T05:38:57.530679Z","shell.execute_reply.started":"2022-02-07T05:38:53.905327Z","shell.execute_reply":"2022-02-07T05:38:57.529966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_model(backbone):\n    '''Generate Embeddings from pretrained models\n    \n    Inputs:\n    backbone : pretrained models'''\n    inp = tf.keras.layers.Input(shape=(*cfg.img_size,3))\n    x = backbone(inp)\n    output = tf.keras.layers.GlobalAveragePooling2D()(x)\n    \n    return tf.keras.Model(inputs=inp,outputs=output)\n\nmodel = build_model(backbone=efnet)","metadata":{"execution":{"iopub.status.busy":"2022-02-07T05:38:57.531958Z","iopub.execute_input":"2022-02-07T05:38:57.532386Z","iopub.status.idle":"2022-02-07T05:38:58.186380Z","shell.execute_reply.started":"2022-02-07T05:38:57.532357Z","shell.execute_reply":"2022-02-07T05:38:58.185765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading Data for Generating Embeddings","metadata":{}},{"cell_type":"markdown","source":"**Taking 5000 images from training and test set to check thier distribution**","metadata":{}},{"cell_type":"code","source":"#loading train data \nnum_samples = 5000\n\n\n#store the paths to train files in these lists\ntr_files = glob(train_dir + '*.jpg')\ntrain_files = tf.data.Dataset.list_files(file_pattern=tr_files, \n                                      shuffle=True,\n                                      seed= cfg.seed).take(num_samples)\n\n#store the paths to test files in these lists\nts_files = glob(test_dir + '*.jpg') \ntest_files = tf.data.Dataset.list_files(ts_files,\n                                      shuffle=True,\n                                      seed= cfg.seed).take(num_samples)\n","metadata":{"execution":{"iopub.status.busy":"2022-02-07T05:38:58.187579Z","iopub.execute_input":"2022-02-07T05:38:58.188051Z","iopub.status.idle":"2022-02-07T05:40:35.375220Z","shell.execute_reply.started":"2022-02-07T05:38:58.188018Z","shell.execute_reply":"2022-02-07T05:40:35.374295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**AUTOTUNE FUNCTION**","metadata":{}},{"cell_type":"code","source":"# this function will autotune number of parallel calls.\nAUTOTUNE = tf.data.experimental.AUTOTUNE","metadata":{"execution":{"iopub.status.busy":"2022-02-07T05:40:35.376490Z","iopub.execute_input":"2022-02-07T05:40:35.376730Z","iopub.status.idle":"2022-02-07T05:40:35.383281Z","shell.execute_reply.started":"2022-02-07T05:40:35.376700Z","shell.execute_reply":"2022-02-07T05:40:35.382454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**LOADING IMAGES**","metadata":{}},{"cell_type":"code","source":"train_ds = train_files.map(load_image,num_parallel_calls=AUTOTUNE)\ntest_ds = test_files.map(load_image,num_parallel_calls=AUTOTUNE)","metadata":{"execution":{"iopub.status.busy":"2022-02-07T05:40:35.386412Z","iopub.execute_input":"2022-02-07T05:40:35.386637Z","iopub.status.idle":"2022-02-07T05:40:35.618311Z","shell.execute_reply.started":"2022-02-07T05:40:35.386610Z","shell.execute_reply":"2022-02-07T05:40:35.617686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**PREPROCESS IMAGES**","metadata":{}},{"cell_type":"code","source":"def preprocess_img(img,\n                  img_size=cfg.img_size,\n                  expand_dims =False):\n    '''resize images to common shape and rescale by / 255'''\n    \n    img = tf.image.resize(images=img,size=img_size)\n    img = img / 255.0    \n    \n    if expand_dims:\n        img = tf.expand_dims(img,\n                             axis=-1)\n    \n    return img","metadata":{"execution":{"iopub.status.busy":"2022-02-07T05:40:35.619300Z","iopub.execute_input":"2022-02-07T05:40:35.619636Z","iopub.status.idle":"2022-02-07T05:40:35.625511Z","shell.execute_reply.started":"2022-02-07T05:40:35.619608Z","shell.execute_reply":"2022-02-07T05:40:35.624715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#apply preprocessing function to dataset\n\ntrain_ds = train_ds.map(preprocess_img,num_parallel_calls=AUTOTUNE)\ntest_ds  = train_ds.map(preprocess_img,num_parallel_calls=AUTOTUNE)","metadata":{"execution":{"iopub.status.busy":"2022-02-07T05:40:35.626664Z","iopub.execute_input":"2022-02-07T05:40:35.626905Z","iopub.status.idle":"2022-02-07T05:40:35.718319Z","shell.execute_reply.started":"2022-02-07T05:40:35.626878Z","shell.execute_reply":"2022-02-07T05:40:35.717397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def optimize_pipeline(tf_dataset,\n                      batch_size = cfg.batch_size,\n                      Autotune_fn = AUTOTUNE,\n                      cache= False,\n                      batch = True):\n    if cache:\n        tf_dataset = tf_dataset.cache()                        # store data in RAM  \n        \n    tf_dataset =  tf_dataset.shuffle(buffer_size=100)         # shuffle \n    \n    if batch:\n        tf_dataset = tf_dataset.padded_batch(batch_size)              #split the data in batches\n    \n    # prefetch(load the data with cpu,while gpu is training) the data in memory \n    tf_dataset = tf_dataset.prefetch(buffer_size=Autotune_fn)    \n    \n    return tf_dataset","metadata":{"execution":{"iopub.status.busy":"2022-02-07T05:40:35.719596Z","iopub.execute_input":"2022-02-07T05:40:35.719844Z","iopub.status.idle":"2022-02-07T05:40:35.725945Z","shell.execute_reply.started":"2022-02-07T05:40:35.719814Z","shell.execute_reply":"2022-02-07T05:40:35.725032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# optimize for performance \ntrain_ds=optimize_pipeline(train_ds)\ntest_ds=optimize_pipeline(test_ds)","metadata":{"execution":{"iopub.status.busy":"2022-02-07T05:40:35.727128Z","iopub.execute_input":"2022-02-07T05:40:35.727452Z","iopub.status.idle":"2022-02-07T05:40:35.752195Z","shell.execute_reply.started":"2022-02-07T05:40:35.727424Z","shell.execute_reply":"2022-02-07T05:40:35.751269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Now Our data is ready to be passed into the model**","metadata":{}},{"cell_type":"markdown","source":"# Generate Embeddings","metadata":{}},{"cell_type":"code","source":"def predict(dataset,model = model):\n    \n    embeddings = []\n    \n    for batch in tqdm(train_ds.as_numpy_iterator()):   #iterate over the batches \n        \n        embeds = model(batch)   # pass (predict) with pretrained model \n        \n        embeddings.extend(embeds)        # append batch predictions to the list \n        \n    return np.array(embeddings)    ","metadata":{"execution":{"iopub.status.busy":"2022-02-07T05:40:35.753481Z","iopub.execute_input":"2022-02-07T05:40:35.753902Z","iopub.status.idle":"2022-02-07T05:40:35.759195Z","shell.execute_reply.started":"2022-02-07T05:40:35.753871Z","shell.execute_reply":"2022-02-07T05:40:35.758383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get embeddings\ntrain_embeddings = predict(train_ds)","metadata":{"execution":{"iopub.status.busy":"2022-02-07T05:40:35.760797Z","iopub.execute_input":"2022-02-07T05:40:35.761342Z","iopub.status.idle":"2022-02-07T05:45:57.821735Z","shell.execute_reply.started":"2022-02-07T05:40:35.761296Z","shell.execute_reply":"2022-02-07T05:45:57.820830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_embeddings = predict(test_ds)","metadata":{"execution":{"iopub.status.busy":"2022-02-07T05:45:57.825354Z","iopub.execute_input":"2022-02-07T05:45:57.826087Z","iopub.status.idle":"2022-02-07T05:53:20.142707Z","shell.execute_reply.started":"2022-02-07T05:45:57.826046Z","shell.execute_reply":"2022-02-07T05:53:20.141820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'train embeddings are in range{train_embeddings.min(),train_embeddings.max()}')\nprint(f'test embeddings are in range{test_embeddings.min(),test_embeddings.max()}')","metadata":{"execution":{"iopub.status.busy":"2022-02-07T06:24:26.432711Z","iopub.execute_input":"2022-02-07T06:24:26.433244Z","iopub.status.idle":"2022-02-07T06:24:26.448078Z","shell.execute_reply.started":"2022-02-07T06:24:26.433208Z","shell.execute_reply":"2022-02-07T06:24:26.447148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Checking the distribution of the data using UMAP and TSNE ","metadata":{}},{"cell_type":"markdown","source":"**TSNE**\n\n* **T-SNE is a Non-Linera dimensionality reduction technique that preserves the local clusters , while reducing the dimensionality.**\n* **It is useful in visualizing higher dimensional data,by projecting it into lower dimensions that can be visualized.**","metadata":{}},{"cell_type":"code","source":"%%time\n#instantiate tsne object with following params\ntsne = TSNE(n_components=3,perplexity=20,n_iter=2500,random_state=cfg.seed)\n\n#project the (5000,1280) array to (5000,3)\ntrain_tsne = tsne.fit_transform(train_embeddings)","metadata":{"execution":{"iopub.status.busy":"2022-02-07T07:23:23.243853Z","iopub.execute_input":"2022-02-07T07:23:23.244187Z","iopub.status.idle":"2022-02-07T07:30:26.015792Z","shell.execute_reply.started":"2022-02-07T07:23:23.244155Z","shell.execute_reply":"2022-02-07T07:30:26.014819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#project the (5000,1280) array to (5000,3)\ntest_tsne = tsne.fit_transform(test_embeddings)","metadata":{"execution":{"iopub.status.busy":"2022-02-07T07:30:26.018002Z","iopub.execute_input":"2022-02-07T07:30:26.018451Z","iopub.status.idle":"2022-02-07T07:36:26.772795Z","shell.execute_reply.started":"2022-02-07T07:30:26.018403Z","shell.execute_reply":"2022-02-07T07:36:26.771807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_tsne = pd.DataFrame(train_tsne,columns=['dim1','dim2','dim3'])\ndf_tsne['set'] = 'train'\n\ntest_tsne = pd.DataFrame(test_tsne,columns=['dim1','dim2','dim3'])\ntest_tsne['set'] = 'test'\n\n\n#append train and test df together \ndf_tsne = df_tsne.append(other=test_tsne)\n\n#set the train color to green\ndf_tsne.loc[df_tsne['set']=='train','color'] = 'green'\n\n#set test color to blue\ndf_tsne.loc[df_tsne['set']=='test','color'] = 'blue'\n\ndf_tsne.shape","metadata":{"execution":{"iopub.status.busy":"2022-02-07T07:36:26.774301Z","iopub.execute_input":"2022-02-07T07:36:26.777796Z","iopub.status.idle":"2022-02-07T07:36:26.790126Z","shell.execute_reply.started":"2022-02-07T07:36:26.777736Z","shell.execute_reply":"2022-02-07T07:36:26.789277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.style.use('seaborn-white')\nfrom mpl_toolkits import mplot3d \n\nfig = plt.figure(figsize =(15,15)) \nax = plt.axes(projection ='3d') \n\nax.scatter(xs=df_tsne['dim2'].values,\n           ys=df_tsne['dim1'].values,\n           zs=df_tsne['dim3'].values,\n           c=df_tsne['color'],\n           s= 3)\n\n\nax.set_xlabel('dim 2')\nax.set_ylabel('dim 1')\nax.set_zlabel('dim 3')\n\nplt.legend(df_tsne['set'])\nplt.title('TSNE reduced embeddings')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-07T07:51:46.521912Z","iopub.execute_input":"2022-02-07T07:51:46.522212Z","iopub.status.idle":"2022-02-07T07:51:47.209938Z","shell.execute_reply.started":"2022-02-07T07:51:46.522181Z","shell.execute_reply":"2022-02-07T07:51:47.208807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**UMAP**\n\n* It is an similar dimensionality reduction algorithm.","metadata":{}},{"cell_type":"code","source":"%%time\numap =UMAP(n_neighbors=20,\n          n_components=3,\n          min_dist=0.3,\n          metric='euclidean')\n\n#project the (5000,1280) array to (5000,3)\ntrain_umap = umap.fit_transform(train_embeddings)","metadata":{"execution":{"iopub.status.busy":"2022-02-07T07:54:44.167169Z","iopub.execute_input":"2022-02-07T07:54:44.167518Z","iopub.status.idle":"2022-02-07T07:54:57.829417Z","shell.execute_reply.started":"2022-02-07T07:54:44.167481Z","shell.execute_reply":"2022-02-07T07:54:57.828397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#project the (5000,1280) array to (5000,3)\ntest_umap = umap.fit_transform(test_embeddings)","metadata":{"execution":{"iopub.status.busy":"2022-02-07T07:54:57.831676Z","iopub.execute_input":"2022-02-07T07:54:57.832039Z","iopub.status.idle":"2022-02-07T07:55:11.576518Z","shell.execute_reply.started":"2022-02-07T07:54:57.831992Z","shell.execute_reply":"2022-02-07T07:55:11.575576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_umap = pd.DataFrame(train_umap,columns=['dim1','dim2','dim3'])\ndf_umap['set'] = 'train'\n\ntest_umap = pd.DataFrame(test_umap,columns=['dim1','dim2','dim3'])\ntest_umap['set'] = 'test'\n\n#append train and test df together \ndf_umap = df_umap.append(other=test_umap)\n\n#set the train color to green\ndf_umap.loc[df_umap['set']=='train','color'] = 'blue'\n\n#set test color to blue\ndf_umap.loc[df_umap['set']=='test','color'] = 'red'\n\ndf_umap.shape","metadata":{"execution":{"iopub.status.busy":"2022-02-07T07:55:11.578461Z","iopub.execute_input":"2022-02-07T07:55:11.578814Z","iopub.status.idle":"2022-02-07T07:55:11.600259Z","shell.execute_reply.started":"2022-02-07T07:55:11.578768Z","shell.execute_reply":"2022-02-07T07:55:11.599307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nfig = plt.figure(figsize =(15,15)) \nax = plt.axes(projection ='3d') \n\nax.scatter(xs=df_umap['dim1'].values,\n           ys=df_umap['dim2'].values,\n           zs=df_umap['dim3'].values,\n           c=df_umap['color'],\n           s=3)\n\n\nax.set_xlabel('dim 1')\nax.set_ylabel('dim 2')\nax.set_zlabel('dim 3')\n\nplt.legend(df_umap['set'])\nplt.title('UMAP reduced embeddings')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-07T07:58:50.831706Z","iopub.execute_input":"2022-02-07T07:58:50.832196Z","iopub.status.idle":"2022-02-07T07:58:51.533379Z","shell.execute_reply.started":"2022-02-07T07:58:50.832144Z","shell.execute_reply":"2022-02-07T07:58:51.532329Z"},"trusted":true},"execution_count":null,"outputs":[]}]}