{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"![](https://drive.google.com/uc?id=1vBNDDTuZG6ptoOw2iEEpXKOx3rQCSDiC)\n\n[Image Source](https://www.facebook.com/happywhales)\n\n# **Goal**\n \nThe goal of this competition is to develop a model to match individual whales and dolphins by unique—but often subtle—characteristics of their natural markings like dorsal fins and lateral body views .\n\n# **Data**\n\nThe data in this competition contains images of over 15,000 unique individual marine mammals from 30 different species collected from 28 different research organizations. Individuals have been manually identified and given an `individual_id` by marine researches, and your task is to correctly identify these individuals in images. \n\n`train_images/` - a folder containing the training images\n\n `train.csv` - provides the species and the individual_id for each of the training images\n \n`test_images/` - a folder containing the test images; for each image, your task is to predict the individual_id; no species information is given for the test data; there are individuals in the test data that are not observed in the training data, which should be predicted as new_individual.\n\n`sample_submission.csv` - a sample submission file in the correct format\n\n# **Metric**\n\nThe evaluation metric in the competition's description is Mean Average Precision @ 5 (MAP@5):\n$$MAP@5 = {1 \\over U} \\sum_{u=1}^{U} \\sum_{k=1}^{min(n,5)}P(k) × rel(k)$$\n\nwhere `U` is the number of images, `P(k)` is the precision at cutoff `k`, rel(k) is an indicator function equaling 1 if the item at rank k is a relevant (correct) label, zero otherwise and `n` is the number of predictions per image.\n\n\n\n\n\n> I will be integrating W&B for visualizations and logging artifacts!\n> \n> [Happy Whale - Involutional Neural Networks](https://wandb.ai/usharengaraju/InvolutionalNN)\n\n\n> \n> - To get the API key, create an account in the [website](https://wandb.ai/site) .\n> - Use secrets to use API Keys more securely ","metadata":{}},{"cell_type":"code","source":"!pip install tensorflow_addons","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport re\nimport os\nimport numpy as np\nimport pandas as pd\nimport random\nimport math\nimport pickle\nimport json\n\nimport wandb\n\nfrom sklearn import metrics\nfrom sklearn.model_selection import KFold, train_test_split\n\nfrom tqdm.auto import tqdm\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom datetime import datetime\n\nfrom kaggle_datasets import KaggleDatasets\n\nfrom functools import partial\n\n\nimport tensorflow as tf\nfrom tensorflow import keras\nimport tensorflow_hub as tfhub\nfrom tensorflow.keras import backend as K\nimport tensorflow_addons as tfa\n\nimport cv2\n\n# Set seed for reproducibility.\ntf.random.set_seed(42)\n\n#ignore warnings\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n%matplotlib inline\n\nROOT_PATH = '../input/happy-whale-and-dolphin'\nIMGS_DIR = f'{ROOT_PATH}/train_images'","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n from kaggle_secrets import UserSecretsClient\n user_secrets = UserSecretsClient()\n secret_value_0 = user_secrets.get_secret(\"api_key\")\n wandb.login(key=secret_value_0)\n anony=None\nexcept:\n anony = \"must\"\n print('If you want to use your W&B account, go to Add-ons -> Secrets and provide your W&B access token. Use the Label name as wandb_api. \\nGet your W&B access token from here: https://wandb.ai/authorize')\n \nCONFIG = dict(competition = 'InvolutionalNN',_wandb_kernel = 'tensorgirl')","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Data PreProcessing**","metadata":{}},{"cell_type":"markdown","source":"Code for preprocessing and Visualization is inspired from the following Sources [1](https://www.kaggle.com/ayuraj/happywhale-eda-ids-to-images-using-w-b-tables) [2](https://www.kaggle.com/abhranta/starter-eda-aug) [3](https://www.kaggle.com/newra008/happywhale-explore)\n[4](https://www.kaggle.com/rajankumar/visuals-of-happywhale)\n\n","metadata":{}},{"cell_type":"code","source":"# code copied from https://www.kaggle.com/newra008/happywhale-explore\ntrain_df = pd.read_csv(\"../input/happy-whale-and-dolphin/train.csv\")\ntrain_df.loc[train_df.species.str.contains('beluga'), 'species'] = 'beluga_whale'\ntrain_df.loc[train_df.species.str.contains('globis'), 'species'] = 'globis_whale'\n\ntrain_df['species'] = train_df['species'].str.replace('bottlenose_dolpin','bottlenose_dolphin')\ntrain_df['species'] = train_df['species'].str.replace('kiler_whale','killer_whale')\n\ntrain_df['class'] = train_df.species.map(lambda x: 'dolphin' if 'dolphin' in x else 'whale')","metadata":{"execution":{"iopub.status.busy":"2022-02-21T01:51:35.597438Z","iopub.execute_input":"2022-02-21T01:51:35.597978Z","iopub.status.idle":"2022-02-21T01:51:35.883836Z","shell.execute_reply.started":"2022-02-21T01:51:35.597947Z","shell.execute_reply":"2022-02-21T01:51:35.883076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Species Distribution - Whale vs Dolphin**","metadata":{}},{"cell_type":"code","source":"# Let's look at the train image by species class\nfig, ax = plt.subplots(figsize=(15, 7))\nclass_count = train_df.groupby(['class']).size().reset_index(name='counts')\nsns.barplot(x = class_count['counts'],y = class_count['class'],data = class_count,palette = \"Blues\") \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-21T01:51:38.3566Z","iopub.execute_input":"2022-02-21T01:51:38.357468Z","iopub.status.idle":"2022-02-21T01:51:38.59137Z","shell.execute_reply.started":"2022-02-21T01:51:38.357413Z","shell.execute_reply":"2022-02-21T01:51:38.590492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Most Frequently Occuring Individual IDs**","metadata":{}},{"cell_type":"code","source":"# code copied from https://www.kaggle.com/newra008/happywhale-explore\nfig, ax = plt.subplots(figsize=(15, 7))\nmost_frequent_ids = train_df.individual_id.value_counts().head(10)\nmost_frequent_ids = pd.DataFrame({'individual_id':most_frequent_ids.index, 'frequency':most_frequent_ids.values})\nsns.barplot(y = most_frequent_ids['individual_id'],x = most_frequent_ids['frequency'],data = most_frequent_ids,palette = \"Blues\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-21T01:52:08.909583Z","iopub.execute_input":"2022-02-21T01:52:08.909915Z","iopub.status.idle":"2022-02-21T01:52:09.210504Z","shell.execute_reply.started":"2022-02-21T01:52:08.909883Z","shell.execute_reply":"2022-02-21T01:52:09.209644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Dolphin Subspecies Distribution**","metadata":{}},{"cell_type":"code","source":"# code copied from https://www.kaggle.com/rajankumar/visuals-of-happywhale\nfig, ax = plt.subplots(figsize=(15, 7))\ndolphin_count = train_df[train_df['class'] == 'dolphin'].groupby('species').image.count()\ndolphin_count = pd.DataFrame(dolphin_count.sort_values(ascending = False))\nsns.barplot(data = dolphin_count, y = dolphin_count.index, x = 'image', palette = \"Blues\")\nplt.title('Dolphin')","metadata":{"execution":{"iopub.status.busy":"2022-02-21T01:52:26.414534Z","iopub.execute_input":"2022-02-21T01:52:26.415091Z","iopub.status.idle":"2022-02-21T01:52:26.744981Z","shell.execute_reply.started":"2022-02-21T01:52:26.415049Z","shell.execute_reply":"2022-02-21T01:52:26.744161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Whale Subspecies Distribution**","metadata":{}},{"cell_type":"code","source":"# code copied from https://www.kaggle.com/rajankumar/visuals-of-happywhale\nfig, ax = plt.subplots(figsize=(15, 7))\nwhale_count = train_df[train_df['class'] == 'whale'].groupby('species').image.count()\nwhale_count = pd.DataFrame(whale_count.sort_values(ascending = False))\nsns.barplot(data = whale_count, y = whale_count.index, x = 'image', palette = \"Blues\")\nplt.title('Whale')","metadata":{"execution":{"iopub.status.busy":"2022-02-21T01:52:38.49389Z","iopub.execute_input":"2022-02-21T01:52:38.494172Z","iopub.status.idle":"2022-02-21T01:52:38.938636Z","shell.execute_reply.started":"2022-02-21T01:52:38.494145Z","shell.execute_reply":"2022-02-21T01:52:38.937731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Species distribution by Individual IDs**","metadata":{}},{"cell_type":"code","source":"# code copied from https://www.kaggle.com/abhranta/starter-eda-aug\nid_perspecies = train_df.groupby([\"species\"])[\"individual_id\"].nunique()\nid_perspecies = pd.DataFrame({'Species': id_perspecies.index,'Unique ID Count': id_perspecies.values})\nid_perspecies = id_perspecies.sort_values(['Unique ID Count'], ascending=False)\nfig, ax = plt.subplots(figsize=(15, 7))\nplt.title('Species distribution by Individual IDs')\nsns.barplot(y = 'Species', x=\"Unique ID Count\", data=id_perspecies ,palette = \"Blues\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-21T01:52:50.391811Z","iopub.execute_input":"2022-02-21T01:52:50.392372Z","iopub.status.idle":"2022-02-21T01:52:50.864491Z","shell.execute_reply.started":"2022-02-21T01:52:50.392337Z","shell.execute_reply":"2022-02-21T01:52:50.863686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Visualizing Whales**","metadata":{}},{"cell_type":"code","source":"sample_whale = train_df[train_df[\"class\"] == 'whale'].sample(9)\nfig, ax = plt.subplots(figsize=(15, 7)) \nfor ind, (image_id, label) in enumerate(zip(sample_whale.image, sample_whale.species)):\n plt.subplot(3, 3, ind + 1)\n image = cv2.imread(os.path.join(\"../input/happy-whale-and-dolphin/train_images\", image_id))\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n plt.imshow(image)\n plt.axis(\"off\")\n \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-21T01:53:05.878127Z","iopub.execute_input":"2022-02-21T01:53:05.878418Z","iopub.status.idle":"2022-02-21T01:53:10.539271Z","shell.execute_reply.started":"2022-02-21T01:53:05.878389Z","shell.execute_reply":"2022-02-21T01:53:10.538232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Visualizing Dolphins**","metadata":{}},{"cell_type":"code","source":"sample_dolphin = train_df[train_df[\"class\"] == 'dolphin'].sample(9)\nfig, ax = plt.subplots(figsize=(15, 7))\nfor ind, (image_id, label) in enumerate(zip(sample_dolphin.image, sample_dolphin.species)):\n plt.subplot(3, 3, ind + 1)\n image = cv2.imread(os.path.join(\"../input/happy-whale-and-dolphin/train_images\", image_id))\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n plt.imshow(image)\n plt.axis(\"off\")\n \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-21T01:53:18.742288Z","iopub.execute_input":"2022-02-21T01:53:18.742906Z","iopub.status.idle":"2022-02-21T01:53:23.231471Z","shell.execute_reply.started":"2022-02-21T01:53:18.742859Z","shell.execute_reply":"2022-02-21T01:53:23.230689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# **Map individual_id to Images using W&B Tables**\n\n[Source](https://docs.wandb.ai/guides/data-vis/tables)\n\nW&B Tables are used to log and visualize data and model predictions. Interactively explore your data:\n\n📌 Compare changes precisely across models, epochs, or individual examples\n\n📌 Understand higher-level patterns in your data\n\n📌 Capture and communicate your insights with visual samples\n","metadata":{}},{"cell_type":"code","source":"# code copied from https://www.kaggle.com/ayuraj/happywhale-eda-ids-to-images-using-w-b-tables\n# Initialize a W&B run\nrun = wandb.init(project='InvolutionalNN')\n\ntrain_df1 = train_df.sample(1000)\n\n# Initialize an empty W&B Table\ndata_table = wandb.Table(columns=['individual_id', 'image_1', 'image_2', 'image_3', 'image_4', 'image_5'])\n\nfor unique_id, tmp_df in tqdm(train_df1.groupby('individual_id')):\n if len(tmp_df) > 5:\n # Sample 5 images randomly\n sample_imgs = random.sample(list(tmp_df.image.values), 5)\n # Add data to the table row-wise\n data_table.add_data(unique_id,\n wandb.Image(f'{IMGS_DIR}/{sample_imgs[0]}'),\n wandb.Image(f'{IMGS_DIR}/{sample_imgs[1]}'),\n wandb.Image(f'{IMGS_DIR}/{sample_imgs[2]}'),\n wandb.Image(f'{IMGS_DIR}/{sample_imgs[3]}'),\n wandb.Image(f'{IMGS_DIR}/{sample_imgs[4]}'))\n \n# Log the table\nwandb.log({'mapping_table': data_table})\n\n# Finish the run\nwandb.finish()","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **TPU Intialization and Distribution Strategy **\n\n\n![](https://drive.google.com/uc?id=1q6AUi9XZRRWBjov49PSl3thB9idGsUKV)\n\nTensor Processing Units (TPUs) are Google's custom-developed application-specific integrated circuits (ASICs) used to accelerate machine learning workloads.It's easy to run replicated models on Cloud TPU using High-level Tensorflow APIs .\n\n### **Performance**\n\nTPU can achieve a high computational throughput on massive multiplications and additions for neural networks , at blazingly fast speeds with much less power consumption and smaller footprint.\n\n📌 **TPU Initialization:** TPUs are usually on Cloud TPU workers and hence have to be connected to remote clusters and then initialized .\n\n📌 **Distribution strategies :** A distribution strategy is an abstraction that can be used to drive models on CPU, GPUs or TPUs.","metadata":{}},{"cell_type":"code","source":"\ntry:\n # TPU detection. No parameters necessary if TPU_NAME environment variable is\n # set: this is always the case on Kaggle.\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Running on TPU ', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.TPUStrategy(tpu)\nelse:\n # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n strategy = tf.distribute.get_strategy()\n\nAUTO = tf.data.experimental.AUTOTUNE\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\nNote: TPUs read data directly from Google Cloud Storage (GCS) This Kaggle utility will copy the dataset to a GCS bucket co-located with the TPU. If you have multiple datasets attached to the notebook, you can pass the name of a specific dataset to the get_gcs_path function. The name of the dataset is the name of the directory it is mounted in. Use !ls /kaggle/input/ to list attached datasets.\n
","metadata":{}},{"cell_type":"markdown","source":"# **TF Records**\n\n📌TF-Record is Tensorflow’s own binary storage format.\n\n📌Any byte-string that can be decoded in TensorFlow could be stored in a TFRecord file. Examples include: Lines of text, JSON ,encoded image data, or serialized tf.Tensors .\n\n📌A TFRecord file contains a sequence of records and can only be read sequentially.Each record contains a byte-string, for the data-payload, plus the data-length, and CRC32C (32-bit CRC using the Castagnoli polynomial) hashes for integrity checking.","metadata":{}},{"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path('happywhale-tfrecords-v1') \ntrain_files = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/happywhale-2022-train*.tfrec')))\ntest_files = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/happywhale-2022-test*.tfrec')))\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\nBATCH_SIZE = 32\nIMAGE_SIZE = [256, 256]","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Train-Validation-Test Split**\n\n![](https://drive.google.com/uc?id=1qomd1gdCbw7CA-vdlqtBWAYqsU6-aCtN)\n\nImage Source : mc.ai","metadata":{}},{"cell_type":"code","source":"TRAINING_FILENAMES, VALID_FILENAMES = train_test_split(\n tf.io.gfile.glob(GCS_PATH + '/happywhale-2022-train*.tfrec'),\n test_size=0.1, random_state=42\n)\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/happywhale-2022-test*.tfrec')","metadata":{"_kg_hide-output":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **TF-Data**\n\n![](https://drive.google.com/uc?id=10M07nrMGrKKVP0KGwAqsm7oNK1mXexsD)\n\n📌The tf.data API enables you to build complex input pipelines from simple, reusable pieces. The tf.data API also makes it possible to handle large amounts of data, read from different data formats, and perform complex transformations.\n\n📌The tf.data API introduces a tf.data.Dataset abstraction that represents a sequence of elements, in which each element consists of one or more components. For example, in an image pipeline, an element might be a single training example, with a pair of tensor components representing the image and its label.\n\n🎯 **Input Pipeline :**\n\n tf.data.TFRecordDataset() is used to create an input pipeline for data stored in TFRecord format .\n \n🎯 **Transformations :**\n\nThe Dataset object can be transformed into a new Dataset by chaining method calls on the tf.data.Dataset object . Some of the transformations which can be applied are Dataset.map() , Dataset.batch() , Dataset.shuffle() , Dataset.prefetch() .\n\nThe Dataset object is a Python iterable which it possible to consume its elements using a for loop .\n","metadata":{}},{"cell_type":"code","source":"def decode_to_tensor(image):\n image = tf.image.decode_jpeg(image, channels=3)\n image = tf.cast(image, tf.float32) / 255.0\n image = tf.reshape(image, [*IMAGE_SIZE, 3])\n return image\n\ndef read_tfrecord(example, labeled):\n tfrecord_format = {\n \"image\": tf.io.FixedLenFeature([], tf.string),\n \"target\": tf.io.FixedLenFeature([], tf.int64)\n } if labeled else {\n \"image\": tf.io.FixedLenFeature([], tf.string),\n \"image_name\": tf.io.FixedLenFeature([], tf.string)\n }\n example = tf.io.parse_single_example(example, tfrecord_format)\n image = decode_to_tensor(example['image'])\n if labeled:\n label = tf.cast(example['target'], tf.int32)\n return image, label\n idnum = example['image_name']\n return image, idnum\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n ignore_order = tf.data.Options()\n if not ordered:\n ignore_order.experimental_deterministic = False # disable order, increase speed\n dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTOTUNE) # automatically interleaves reads from multiple files\n dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n dataset = dataset.map(partial(read_tfrecord, labeled=labeled), num_parallel_calls=AUTOTUNE)\n # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n return dataset\n\n","metadata":{"execution":{"iopub.status.busy":"2022-02-19T21:13:17.327717Z","iopub.execute_input":"2022-02-19T21:13:17.328539Z","iopub.status.idle":"2022-02-19T21:13:17.340383Z","shell.execute_reply.started":"2022-02-19T21:13:17.328477Z","shell.execute_reply":"2022-02-19T21:13:17.339531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# **Data Augmentation**\n\n🎯 Data augmentation is a technique through which one can increase the size of the data for the training of the model without adding the new data. Techniques like padding, cropping, rotating, and flipping are the most common methods that are used over the images to increase the data size. \n\n
\nNote: Both Keras Preprocessing Layers and tf.image can be used for data augmentation. For finer control , tf.image is preferred.\n
\n","metadata":{}},{"cell_type":"code","source":"def augmentation_pipeline(image, label):\n image = tf.image.random_flip_left_right(image)\n return image, label\n\ndef get_training_dataset():\n dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n dataset = dataset.map(augmentation_pipeline, num_parallel_calls=AUTOTUNE)\n dataset = dataset.repeat()\n dataset = dataset.shuffle(2048)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTOTUNE)\n return dataset\n\n\ndef get_validation_dataset(ordered=False):\n dataset = load_dataset(VALID_FILENAMES, labeled=True, ordered=ordered)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.cache()\n dataset = dataset.prefetch(AUTOTUNE)\n return dataset\n\ndef get_test_dataset(ordered=False):\n dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTOTUNE)\n return dataset\n\ndef count_data_items(filenames):\n n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n return np.sum(n)\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALID_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nprint(\n 'Dataset: {} training images, {} validation images, {} unlabeled test images'.format(\n NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES\n )\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-19T21:13:20.766349Z","iopub.execute_input":"2022-02-19T21:13:20.766662Z","iopub.status.idle":"2022-02-19T21:13:20.779959Z","shell.execute_reply.started":"2022-02-19T21:13:20.766629Z","shell.execute_reply":"2022-02-19T21:13:20.779335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Cyclic Learning Rate\n\n
\nCyclic learning rates is a learning rate scheduling technique for (1) faster training of a network and (2) a finer understanding of the optimal learning rate. Cyclic learning rates have an effect on the model training process known somewhat fancifully as\"superconvergence\" \n
\n\nSource : @residentmario Notebook\n\n![](https://drive.google.com/uc?id=1bnsglJByd1lc1h9hGrz2WZa1LzVBgtWW)\n\nImage Source : https://arxiv.org/abs/1803.09820","metadata":{}},{"cell_type":"code","source":"def build_lrfn(lr_start=0.00001, lr_max=0.000075, lr_min=0.000001, lr_rampup_epochs=20, lr_sustain_epochs=0, lr_exp_decay=.8):\n lr_max = lr_max * strategy.num_replicas_in_sync\n def lrfn(epoch):\n if epoch < lr_rampup_epochs:\n lr = (lr_max - lr_start) / lr_rampup_epochs * epoch + lr_start\n elif epoch < lr_rampup_epochs + lr_sustain_epochs:\n lr = lr_max\n else:\n lr = (lr_max - lr_min) * lr_exp_decay ** (epoch - lr_rampup_epochs - lr_sustain_epochs) + lr_min\n return lr\n return lrfn","metadata":{"execution":{"iopub.status.busy":"2022-02-19T21:13:24.889308Z","iopub.execute_input":"2022-02-19T21:13:24.889767Z","iopub.status.idle":"2022-02-19T21:13:24.896412Z","shell.execute_reply.started":"2022-02-19T21:13:24.889728Z","shell.execute_reply":"2022-02-19T21:13:24.895832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Involutional Neural Networks**\n\n\n[Source](https://arxiv.org/pdf/2103.06255.pdf)\n\nInvolutions has symmetrically inverse inherent characteristics like spatial-specific and channel-agnostic . Some of the advantages of Involution over Convolution are \n\n📌 Ivolution could summarize the context in a wider spatial arrangement, thus overcome the difficulty of modeling long-range interactions well\n\n📌 Ivolution could adaptively allocate\n\n\n![](https://drive.google.com/uc?id=17dqDpHWpLAXnWfV40EDNoWOTbl1bGUZD)","metadata":{}},{"cell_type":"code","source":"class Involution(keras.layers.Layer):\n def __init__(\n self, channel, group_number, kernel_size, stride, reduction_ratio, name\n ):\n super().__init__(name=name)\n\n # Initialize the parameters.\n self.channel = channel\n self.group_number = group_number\n self.kernel_size = kernel_size\n self.stride = stride\n self.reduction_ratio = reduction_ratio\n\n def build(self, input_shape):\n # Get the shape of the input.\n (_, height, width, num_channels) = input_shape\n\n # Scale the height and width with respect to the strides.\n height = height // self.stride\n width = width // self.stride\n\n # Define a layer that average pools the input tensor\n # if stride is more than 1.\n self.stride_layer = (\n keras.layers.AveragePooling2D(\n pool_size=self.stride, strides=self.stride, padding=\"same\"\n )\n if self.stride > 1\n else tf.identity\n )\n # Define the kernel generation layer.\n self.kernel_gen = keras.Sequential(\n [\n keras.layers.Conv2D(\n filters=self.channel // self.reduction_ratio, kernel_size=1\n ),\n keras.layers.BatchNormalization(),\n keras.layers.ReLU(),\n keras.layers.Conv2D(\n filters=self.kernel_size * self.kernel_size * self.group_number,\n kernel_size=1,\n ),\n ]\n )\n # Define reshape layers\n self.kernel_reshape = keras.layers.Reshape(\n target_shape=(\n height,\n width,\n self.kernel_size * self.kernel_size,\n 1,\n self.group_number,\n )\n )\n self.input_patches_reshape = keras.layers.Reshape(\n target_shape=(\n height,\n width,\n self.kernel_size * self.kernel_size,\n num_channels // self.group_number,\n self.group_number,\n )\n )\n self.output_reshape = keras.layers.Reshape(\n target_shape=(height, width, num_channels)\n )\n\n def call(self, x):\n # Generate the kernel with respect to the input tensor.\n # B, H, W, K*K*G\n kernel_input = self.stride_layer(x)\n kernel = self.kernel_gen(kernel_input)\n\n # reshape the kerenl\n # B, H, W, K*K, 1, G\n kernel = self.kernel_reshape(kernel)\n\n # Extract input patches.\n # B, H, W, K*K*C\n input_patches = tf.image.extract_patches(\n images=x,\n sizes=[1, self.kernel_size, self.kernel_size, 1],\n strides=[1, self.stride, self.stride, 1],\n rates=[1, 1, 1, 1],\n padding=\"SAME\",\n )\n\n # Reshape the input patches to align with later operations.\n # B, H, W, K*K, C//G, G\n input_patches = self.input_patches_reshape(input_patches)\n\n # Compute the multiply-add operation of kernels and patches.\n # B, H, W, K*K, C//G, G\n output = tf.multiply(kernel, input_patches)\n # B, H, W, C//G, G\n output = tf.reduce_sum(output, axis=3)\n\n # Reshape the output kernel.\n # B, H, W, C\n output = self.output_reshape(output)\n\n # Return the output tensor and the kernel.\n return output, kernel","metadata":{"execution":{"iopub.status.busy":"2022-02-19T21:13:27.426372Z","iopub.execute_input":"2022-02-19T21:13:27.426829Z","iopub.status.idle":"2022-02-19T21:13:27.44375Z","shell.execute_reply.started":"2022-02-19T21:13:27.426799Z","shell.execute_reply":"2022-02-19T21:13:27.442802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n# Build the involution model.\n\n train_dataset = get_training_dataset()\n valid_dataset = get_validation_dataset()\n \n print(\"building the involution model...\")\n\n inputs = keras.Input(shape=(512, 512, 3))\n x, _ = Involution(\n channel=3, group_number=1, kernel_size=3, stride=1, reduction_ratio=2, name=\"inv_1\"\n )(inputs)\n x = keras.layers.ReLU()(x)\n x = keras.layers.MaxPooling2D((2, 2))(x)\n x, _ = Involution(\n channel=3, group_number=1, kernel_size=3, stride=1, reduction_ratio=2, name=\"inv_2\"\n )(x)\n x = keras.layers.ReLU()(x)\n x = keras.layers.MaxPooling2D((2, 2))(x)\n x, _ = Involution(\n channel=3, group_number=1, kernel_size=3, stride=1, reduction_ratio=2, name=\"inv_3\"\n )(x)\n x = keras.layers.ReLU()(x)\n x = keras.layers.Flatten()(x)\n x = keras.layers.Dense(64, activation=\"relu\")(x)\n outputs = keras.layers.Dense(10)(x)\n\n inv_model = keras.Model(inputs=[inputs], outputs=[outputs], name=\"inv_model\")\n \n inv_model.compile(\n optimizer = tf.keras.optimizers.Adam(),\n loss = [tf.keras.losses.SparseCategoricalCrossentropy()],\n metrics = [tf.keras.metrics.SparseCategoricalAccuracy(),tf.keras.metrics.SparseTopKCategoricalAccuracy(k=5)]\n )\n \ninv_model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-02-19T21:13:31.81481Z","iopub.execute_input":"2022-02-19T21:13:31.815232Z","iopub.status.idle":"2022-02-19T21:13:32.859807Z","shell.execute_reply.started":"2022-02-19T21:13:31.815203Z","shell.execute_reply":"2022-02-19T21:13:32.85881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Paper Summary **\n\nInvolution is defined as the inverse operation of convolution . While Convolution is spatial agnostic and channel specific , Involution is spatial specific and channel agnostic .The above implies Involution kernels are unique in spatial space whereas they are shared across channels .\n\nInvolution marries Convolution and self attention with a twist . Involutional Neural Networks retains only the query and positional embedding and discards the Key Vector .\n\n### **RedNet :**\n\n\nRedNet architecture is constructed from ResNet by replacing the 3 × 3 convolutions with 3 × 3 Involutions . 1 × 1 convolution responsible for channel projection and fusion is retained . \n\n### **Performance in Image Classification :**\n\nRedNet significantly outperforms other state of the art self attention architectures in image classification tasks.\n\n![](https://drive.google.com/uc?id=1uM2dI0GDIF4SLU_-rJnCSbpzxbkCGKCh)","metadata":{}},{"cell_type":"markdown","source":"### **Acknowledgements**\n\nGoogle supported this work by providing Google Cloud credit\n\nAritra Roy Gosthipaty work on Involutional Neural Network inspired me to write this notebook\n\n### **Attributions**\n\nhttps://www.kaggle.com/ayuraj/happywhale-eda-ids-to-images-using-w-b-tables\n\nhttps://docs.wandb.ai/guides/data-vis/tables\n\nhttps://www.kaggle.com/abhranta/starter-eda-aug\n\nhttps://www.kaggle.com/newra008/happywhale-explore\n\nhttps://www.kaggle.com/rajankumar/visuals-of-happywhale\n\nhttps://www.kaggle.com/ks2019/happywhale-tfrecords\n\nhttps://www.kaggle.com/ks2019/happywhale-arcface-baseline-tpu\n\nhttps://arxiv.org/pdf/2103.06255.pdf\n\n[Involutional Neural Networks](https://github.com/keras-team/keras-io/blob/master/examples/vision/involution.py)\n\n### Work in progress 🚧","metadata":{}}]}