{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"","metadata":{"id":"RcgrrsmsJXwT","papermill":{"duration":0.026719,"end_time":"2022-06-19T15:11:23.114845","exception":false,"start_time":"2022-06-19T15:11:23.088126","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n\nReferences :\n\nhttps://www.kaggle.com/code/susnato/birdclef-2022-eda-resnet50-training-tf\n\nhttps://www.kaggle.com/code/spsayakpaul/mae-keras\n","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nfrom tqdm import tqdm\nimport numpy as np\nfrom sklearn.model_selection import StratifiedKFold\nimport cv2\nimport matplotlib.pyplot as plt\nfrom math import ceil\nimport tensorflow as tf\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport wandb\n\nimport re\nimport gc\nimport glob\nimport random\nimport seaborn as sns\nimport plotly.graph_objects as go\n%matplotlib inline\n\nimport IPython.display as ipd\n\n# Map libraries\nimport descartes\nimport geopandas as gpd\nfrom shapely.geometry import Point, Polygon\n\n\n# Audio specific imports\nimport librosa as lb\nimport librosa.display","metadata":{"id":"OnPwxirZuSI3","papermill":{"duration":7.300885,"end_time":"2022-06-19T15:11:30.493655","exception":false,"start_time":"2022-06-19T15:11:23.19277","status":"completed"},"tags":[],"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-06-24T01:52:23.698198Z","iopub.execute_input":"2022-06-24T01:52:23.698546Z","iopub.status.idle":"2022-06-24T01:52:34.084577Z","shell.execute_reply.started":"2022-06-24T01:52:23.698433Z","shell.execute_reply":"2022-06-24T01:52:34.083361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![](\"https://camo.githubusercontent.com/dd842f7b0be57140e68b2ab9cb007992acd131c48284eaf6b1aca758bfea358b/68747470733a2f2f692e696d6775722e636f6d2f52557469567a482e706e67\")
\n\n> I will be integrating W&B for visualizations and logging artifacts!\n> \n> [Birdclef 2022 project on W&B Dashboard](https://wandb.ai/usharengaraju/BirdClef2022)\n> \n> - To get the API key, create an account in the [website](https://wandb.ai/site) .\n> - Use secrets to use API Keys more securely","metadata":{}},{"cell_type":"code","source":"from kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nwandb_key = user_secrets.get_secret(\"api_key\")\nwandb.login(key = wandb_key)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-06-24T01:52:34.087014Z","iopub.execute_input":"2022-06-24T01:52:34.087369Z","iopub.status.idle":"2022-06-24T01:52:34.968226Z","shell.execute_reply.started":"2022-06-24T01:52:34.087322Z","shell.execute_reply":"2022-06-24T01:52:34.967070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"METADATA_FILE_PATH = '../input/birdclef-2022/train_metadata.csv'\nmetadata_df = pd.read_csv(METADATA_FILE_PATH)\nmetadata_df.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-06-24T01:52:34.970206Z","iopub.execute_input":"2022-06-24T01:52:34.970592Z","iopub.status.idle":"2022-06-24T01:52:35.127592Z","shell.execute_reply.started":"2022-06-24T01:52:34.970547Z","shell.execute_reply":"2022-06-24T01:52:35.126693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Top 20 Primary Labels**","metadata":{}},{"cell_type":"code","source":"species = metadata_df['primary_label'].value_counts().head(20)\nfig, ax = plt.subplots(figsize=(15, 7))\nsns.barplot(x=species.values,y=species.index,data = species,palette = \"copper\") \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-24T01:52:35.129492Z","iopub.execute_input":"2022-06-24T01:52:35.129825Z","iopub.status.idle":"2022-06-24T01:52:35.514288Z","shell.execute_reply.started":"2022-06-24T01:52:35.129795Z","shell.execute_reply":"2022-06-24T01:52:35.513590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nax = sns.countplot(x = metadata_df['rating'], order = metadata_df['rating'].value_counts().index , palette = \"copper\")\n\nplt.title(\"Sound quality rating\", fontsize=16)\nplt.ylabel(\"Count\", fontsize=14)\nplt.yticks(fontsize=13)\nplt.xticks(rotation=45, fontsize=13)\nplt.xlabel(\"\");","metadata":{"execution":{"iopub.status.busy":"2022-06-24T01:52:35.515352Z","iopub.execute_input":"2022-06-24T01:52:35.515745Z","iopub.status.idle":"2022-06-24T01:52:35.820337Z","shell.execute_reply.started":"2022-06-24T01:52:35.515712Z","shell.execute_reply":"2022-06-24T01:52:35.819424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_audio = '../input/birdclef-2022/train_audio/'\nprint(f'Number of unique bird species: {len(os.listdir(train_audio))}')","metadata":{"execution":{"iopub.status.busy":"2022-06-24T01:52:35.821611Z","iopub.execute_input":"2022-06-24T01:52:35.822272Z","iopub.status.idle":"2022-06-24T01:52:35.848065Z","shell.execute_reply.started":"2022-06-24T01:52:35.822235Z","shell.execute_reply":"2022-06-24T01:52:35.847274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Visualize number of recordings per Label using W&B**\n\nW&B Tables are used to log and visualize data and model predictions. \n\nšInteractively explore your data\n\nšCompare changes precisely across models, epochs, or individual examples\n\nšUnderstand higher-level patterns in your data\n\nšCapture and communicate your insights with visual samples","metadata":{}},{"cell_type":"code","source":"recordings_per_label = {'species_id': [], \n 'num_audio': []}\n\nfor label in os.listdir(train_audio):\n num_recordings = len(os.listdir(train_audio+label))\n recordings_per_label['species_id'].append(label)\n recordings_per_label['num_audio'].append(num_recordings)\n \nrecordings_per_label = pd.DataFrame.from_dict(recordings_per_label)\n\nrun = wandb.init(project='BirdClef2022', group='EDA')\ndata = [[label, val] for (val, label) in sorted(zip(recordings_per_label.num_audio.values, recordings_per_label.species_id.values))[::-1]]\ntable = wandb.Table(data=data, columns = [\"species_id\", \"num_audio\"])\nwandb.log({\"recordings_per_label\" : wandb.plot.bar(table, \"species_id\", \"num_audio\",\n title=\"Number of recordings per label\")})\n\n\n\nwandb.finish()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-06-24T01:52:35.849219Z","iopub.execute_input":"2022-06-24T01:52:35.849888Z","iopub.status.idle":"2022-06-24T01:52:39.902928Z","shell.execute_reply.started":"2022-06-24T01:52:35.849847Z","shell.execute_reply":"2022-06-24T01:52:39.900329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"def configure_device():\n try:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect() # connect to tpu cluster\n strategy = tf.distribute.TPUStrategy(tpu) # get strategy for tpu\n print('Num of TPUs: ', strategy.num_replicas_in_sync)\n device='TPU'\n except: # otherwise detect GPUs\n tpu = None\n gpus = tf.config.list_logical_devices('GPU') # get logical gpus\n ngpu = len(gpus)\n if ngpu: # if number of GPUs are 0 then CPU\n strategy = tf.distribute.MirroredStrategy(gpus) # single-GPU or multi-GPU\n print(\"> Running on GPU\", end=' | ')\n print(\"Num of GPUs: \", ngpu)\n device='GPU'\n else:\n print(\"> Running on CPU\")\n strategy = tf.distribute.get_strategy() # connect to single gpu or cpu\n device='CPU'\n return strategy, device, tpu","metadata":{"id":"ASDwpoVPj1JC","papermill":{"duration":0.035845,"end_time":"2022-06-19T15:11:30.555538","exception":false,"start_time":"2022-06-19T15:11:30.519693","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.904453Z","iopub.status.idle":"2022-06-24T01:52:39.904840Z","shell.execute_reply.started":"2022-06-24T01:52:39.904656Z","shell.execute_reply":"2022-06-24T01:52:39.904675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"strategy, device, tpu = configure_device()\nAUTO = tf.data.experimental.AUTOTUNE\n","metadata":{"id":"rJd40S9ElIFZ","outputId":"eb72d2bd-5429-4396-9635-fcf86b597087","papermill":{"duration":6.173833,"end_time":"2022-06-19T15:11:36.757112","exception":false,"start_time":"2022-06-19T15:11:30.583279","status":"completed"},"tags":[],"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-06-24T01:52:39.905959Z","iopub.status.idle":"2022-06-24T01:52:39.906287Z","shell.execute_reply.started":"2022-06-24T01:52:39.906123Z","shell.execute_reply":"2022-06-24T01:52:39.906140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import re\ndef count_data_items(filenames):\n n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n return np.sum(n)\n\nGCS_PATH = \"gs://kds-cd6394f23429c6b928662c3e4c9479f1b4e4371b159e5633d5a79b6d\"\nALL_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/*.tfrec')\nprint('NUM TFRECORD FILES: {:,}'.format(len(ALL_FILENAMES)))\n","metadata":{"id":"lBtNTWAliCwv","outputId":"b7936e23-8f2d-49f6-d3a9-a56568c7035f","papermill":{"duration":0.297757,"end_time":"2022-06-19T15:11:37.087435","exception":false,"start_time":"2022-06-19T15:11:36.789678","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.907359Z","iopub.status.idle":"2022-06-24T01:52:39.907730Z","shell.execute_reply.started":"2022-06-24T01:52:39.907544Z","shell.execute_reply":"2022-06-24T01:52:39.907562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# tf.config.run_functions_eagerly(True)\ndef parse_tfr_element(element):\n #use the same structure as above; it's kinda an outline of the structure we now want to create\n data = {\n 'filename': tf.io.FixedLenFeature([], tf.string),\n 'time':tf.io.FixedLenFeature([], tf.int64),\n 'audio' : tf.io.FixedLenFeature([], tf.string),\n 'label':tf.io.FixedLenFeature([], tf.int64),\n }\n\n \n content = tf.io.parse_single_example(element, data)\n \n filename = content['filename']\n\n time = content['time']\n label = content['label']\n audio = content['audio']\n\n return (audio, label)\n\ndataset = tf.data.TFRecordDataset(ALL_FILENAMES)\n#pass every single feature through our mapping function\ndataset=dataset.shuffle(75000)\ndataset = dataset.map(parse_tfr_element)\n# dataset = dataset.batch(10)","metadata":{"id":"TuAeQTATRGy_","papermill":{"duration":0.152002,"end_time":"2022-06-19T15:11:37.401271","exception":false,"start_time":"2022-06-19T15:11:37.249269","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.908897Z","iopub.status.idle":"2022-06-24T01:52:39.909208Z","shell.execute_reply.started":"2022-06-24T01:52:39.909049Z","shell.execute_reply":"2022-06-24T01:52:39.909065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tqdm import tqdm\ntrain_x=[]\ntrain_y=[]\n\nfor sample in tqdm(dataset.take(200)):\n x = np.fromstring(sample[0].numpy(), dtype='uint8')\n image = cv2.imdecode(x, cv2.IMREAD_UNCHANGED)\n image = cv2.resize(image,(128,48))\n train_x.append(image)\n train_y.append(sample[1].numpy())","metadata":{"id":"UpHNaXaGTdOv","outputId":"0b79e69a-74f0-44a9-94e1-c7184b71a983","papermill":{"duration":49.76547,"end_time":"2022-06-19T15:12:27.194241","exception":false,"start_time":"2022-06-19T15:11:37.428771","status":"completed"},"tags":[],"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-06-24T01:52:39.910089Z","iopub.status.idle":"2022-06-24T01:52:39.910393Z","shell.execute_reply.started":"2022-06-24T01:52:39.910234Z","shell.execute_reply":"2022-06-24T01:52:39.910249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_X = np.asarray(train_x)\ntrain_Y = np.asarray(train_y)\ndel train_y","metadata":{"id":"ow3pOXa3xP68","papermill":{"duration":0.038836,"end_time":"2022-06-19T15:12:27.264599","exception":false,"start_time":"2022-06-19T15:12:27.225763","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.912409Z","iopub.status.idle":"2022-06-24T01:52:39.912786Z","shell.execute_reply.started":"2022-06-24T01:52:39.912604Z","shell.execute_reply":"2022-06-24T01:52:39.912626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_Y = train_Y.ravel()","metadata":{"id":"_qE2nsUMxWA-","papermill":{"duration":0.038094,"end_time":"2022-06-19T15:12:27.333944","exception":false,"start_time":"2022-06-19T15:12:27.29585","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.914308Z","iopub.status.idle":"2022-06-24T01:52:39.914783Z","shell.execute_reply.started":"2022-06-24T01:52:39.914579Z","shell.execute_reply":"2022-06-24T01:52:39.914601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.unique(train_Y).reshape(-1, 1).shape","metadata":{"id":"-0yGKO2K8NzD","outputId":"0a5d02cb-9e45-4b88-8e55-3cdb1aa75a3a","papermill":{"duration":0.043904,"end_time":"2022-06-19T15:12:27.409781","exception":false,"start_time":"2022-06-19T15:12:27.365877","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.915978Z","iopub.status.idle":"2022-06-24T01:52:39.916657Z","shell.execute_reply.started":"2022-06-24T01:52:39.916429Z","shell.execute_reply":"2022-06-24T01:52:39.916452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"jg = [[float(i)] for i in range(151)]\njg=np.array(jg)\njg.shape","metadata":{"id":"RustmjSN_H4m","outputId":"a47811e6-216f-4b13-9a1b-d2328042effd","papermill":{"duration":0.041106,"end_time":"2022-06-19T15:12:27.482536","exception":false,"start_time":"2022-06-19T15:12:27.44143","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.917800Z","iopub.status.idle":"2022-06-24T01:52:39.918130Z","shell.execute_reply.started":"2022-06-24T01:52:39.917959Z","shell.execute_reply":"2022-06-24T01:52:39.917979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.preprocessing import OneHotEncoder\nohe = OneHotEncoder()\nohe.fit(jg)\nY_vals=[]\nfor y in tqdm(train_Y):\n res = ohe.transform(np.array([y]).reshape(-1, 1)).todense()\n y_arr = np.array(res).reshape(-1, 151)\n Y_vals.extend(y_arr)\nY_vals=np.array(Y_vals)\nprint(Y_vals.shape)\ndel train_Y\n\ntrain_x_placeholder=[]\ntrain_y = []\nfor i in range(len(train_x)):\n try:\n assert train_x[i].shape == (48, 128, 3)\n train_x_placeholder.append(train_x[i])\n train_y.append(Y_vals[i])\n except:\n pass\ndel Y_vals","metadata":{"id":"4BfrtV1c7R_6","outputId":"c5b04866-2c6d-4ad7-d4da-40ff6b311bf6","papermill":{"duration":0.118521,"end_time":"2022-06-19T15:12:27.633619","exception":false,"start_time":"2022-06-19T15:12:27.515098","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.919325Z","iopub.status.idle":"2022-06-24T01:52:39.919689Z","shell.execute_reply.started":"2022-06-24T01:52:39.919513Z","shell.execute_reply":"2022-06-24T01:52:39.919532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_x_placeholder = tf.convert_to_tensor(train_x_placeholder)\ntrain_y = tf.convert_to_tensor(train_y)","metadata":{"id":"g6XvtB0K7SAK","papermill":{"duration":2.984266,"end_time":"2022-06-19T15:12:30.65101","exception":false,"start_time":"2022-06-19T15:12:27.666744","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.920816Z","iopub.status.idle":"2022-06-24T01:52:39.921171Z","shell.execute_reply.started":"2022-06-24T01:52:39.920984Z","shell.execute_reply":"2022-06-24T01:52:39.921007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The tutorial aims to explain the concepts and terminologies of the research paper \" Masked Autoencoders Are Scalable Vision Learners \" .\n\n\n","metadata":{"id":"xGdFB6lSJXwW","papermill":{"duration":0.032793,"end_time":"2022-06-19T15:12:30.719149","exception":false,"start_time":"2022-06-19T15:12:30.686356","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## **Context**\n\n`MAE (Masked autoencoders)` are self-supervised models used to reconstruct the image with missing pixels. It has two core designs consisting of encoder-decoder and masking(hiding) of pixels. Firstly for training purposes, up to 75% of the tokens are masked and removed and the remaining patch is encoded. Since the output of an autoencoder has the same number of tokens as input, the masked tokens are inserted again and with the lightweight decoder, the input image is reconstructed. After pre-training, the decoder is discarded and the encoder is applied to complete images(no missing pixels) for recognition tasks. The above model allows us to train the data faster and improve accuracy. This model works well on a variety of image cases and outperforms supervised training and also scales effectively.\n\nThe transformer architecture has been successfully applied to Natural Language Processing(NLP) using autoregressive language modelling and masked encoding, wherein a portion of the data is removed and models are trained to predict the missing data. However, computer vision has been predominantly associated with Convolutional Neural Network(CNN). The paper explores the usage of masked autoencoders in computer vision\n\nPreviously, autoencoding wasnāt used in computer vision due to the following reasons-\n\nš It was difficult to integrate masked tokens or positional embedding into CNN. However with Vision Transformers(ViT), this problem was solved. ViT slightly outperforms CNN for a large dataset(more than 100 million images). In ViT, we split the image into fixed size patches, vectorize them, add positional embedding and feed the vectors into a transformer encoder to train the model. \n\nš Languages are information-dense and predicting missing words is a sophisticated task which needs sophisticated language understanding. While missing patches of images can be recreated with little high-level understanding. To overcome this,large proportion of patches of image are masked(removed) , reducing redundancy and requiring a higher level of understanding.\n\nš Decoder plays different roles in reconstructing images and text. As text has a high level of semantic information(information that refers to facts, concepts and ideas which we have accumulated over the course of our lives) while images have a low level of semantic information. Thus the decoder design plays an important role for reconstructing images.\n\nThe paper presents a simple, effective and scalable form of MAE (Masked Autoenocoder) for visual representation learning. In MAE, random patches from the input space are masked and these random patches are reconstructed in pixel space. It has an asymmetric encoder-decoder design. The encoder operates on tokens which remain after removal of masked tokens and a lightweight decoder reconstructs the image from the latent representation and masked tokens. With a high masking ratio (75%), high accuracy can be achieved, reducing the overall training time by more than 3x and also reducing memory consumption. MAE pre-training helps data-hungry modela like ViT-Huge to improve their performance. The paper also evaluates transfer learning on a variety of downstream tasks such as object detection, instance segmentation, and semantic segmentation. In these tasks, the proposed pre-trained model achieves better results than supervised pre-trained models.\n","metadata":{"id":"B_J8mjneJXwX","papermill":{"duration":0.032513,"end_time":"2022-06-19T15:12:30.784688","exception":false,"start_time":"2022-06-19T15:12:30.752175","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"\n## **Masked language modelling**\n\n`Masked language modelling` is successful for pre-training in NLP. In methods such as BERT and GPT(methods used for pre-training in NLP), the sequences of words from the input were removed and the model is trained to predict the missing sequence. These models scale efficiently and work for a variety of downstream tasks .\n\n## **Autoencoding**\n\n`Autoencoding` type of neural network that is trained to copy its input to its output. It has an encoder that converts input vector into code vector(latent representation) using recognition weights and a decoder that regenerates the input from the code vector using generative weights. Denoising autoencoders(DAE) are used to corrupt an input signal and predict the original signal. DAE is used to extract a representation from the encoder that is robust to the introduction of noise. DAE can be constructed in many ways such as masking pixels or removing colour of the input. MAE is a kind of denoising autoencoder but different from the classical DAE in many ways. \n\n## **Masked image encoding**\n\n`Masked image encoding` is used to recreate images that have been corrupted by masking. Context encoders are Convolutional Neural networks that generate patches of missing pixels on the basis of its surrounding pixels. The success of unsupervised learning using transformers in NLP has prompted a similar method to be applied to images. `iGPT` operates on sequences of pixels using a sequence transformer to predict unknown pixels autoregressive. `ViT(vision transformer)` masks patches of images and using transformers predicts the image type. Vision Transformer (ViT), using self supervision, attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. Recently, `BEiT (Bidirectional Encoder representation from Image Transformers)` is used to pretrain image transformers. We break the original image into tokens(patches) and randomly mask a few tokens and try to recover the original tokens by fine-tuning the model .\n\n## **Self Supervised Learning**\n\n`Self Supervised Learning` has been used significantly in language and is now being used in computer vision. Contrastive learning is used to train a CNN to classify similar and dissimilar images. Contrastive methods strongly depend on data augmentation. Autoencoding is based on a different concept, and it exhibits different behaviours as we will present. ","metadata":{"id":"PeHTE-VqJXwY","papermill":{"duration":0.033322,"end_time":"2022-06-19T15:12:30.851706","exception":false,"start_time":"2022-06-19T15:12:30.818384","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## **Masked Autoencoders are Scalable Vision Learners**\n\nMasked Autoencoder (MAE) follows a simple autoencoding approach, wherein an encoder maps an input into a latent representation and decoder reconstructs the original image from the latent representation and masked tokens. The paper proposes an unsymmetric design as it allows the encoder to operate on input after removing the masked tokens from it.\n\n## **Masking**\n\nThe image is divided into regular non-overlapping patches and a subset of the patches are randomly sampled , masking(removing) the other patches. This is also called ārandom samplingā, i.e., sampling random patches without replacement, along with uniform distribution. Random Sampling with a high masking ratio eliminates redundancy and reduces the chance of being solved by extrapolation from the visible patches. A uniform distribution prevents centre bias(i.e. More masked patches near centre) and we get a highly sparse input, which helps in designing an efficient encoder. \n\n## **MAE Encoder**\n\nViT(Vision Transformer) is applied only on the patches that are not masked. The encoder embeds the visible patches by linear projection with added positional embedding and then processes it using transformer blocks. Masked tokens are vectors that indicate the presence of a missing patch to be predicted. Since masked patches are removed and no mask tokens are used, large encoders can be trained with a fraction of computation and memory. MAE Encoder is used during pre training and testing as well.\n\n\n\n## **MAE Decoder**\n\nThe inputs to the decoder are encoded visible patches and mask tokens. The total count of tokens is same as in input images. All the tokens have positional embedding in them to know their location in the image. The decoder has transformer blocks in them. MAE decoders are used only during pre-training and hence can be designed independently of the encoder. Due to asymmetric design, a full set of tokens can be processed by lightweight decoders, which significantly reduce pre-training time.\n\n\n\n## **Reconstruction Target**\n \nMAE reconstructs the input image by predicting the pixel values for each masked patch. Each element in the decoderās output is a vector of pixel values representing a patch. The last layer of the decoder is a linear projection whose number of output channels equals the number of pixel values in a patch. The decoderās output is reshaped to form a reconstructed image. The loss function compares the mean squared error between reconstructed and original images on the masked patches.We also study the results after normalising the pixel values of each masked patch as using normalised pixel values improves the accuracy of the experiments. \n\n## **Implementation**\n\nIn the MAE pre-training,a token for every input is generated. The token for each input patch is generated by adding positional embedding to the linear projection of the input. The list of tokens is randomly shuffled and the last portion of the list is removed depending on the masking ratio. The remaining tokens are encoded and then a list of masked tokens is appended to make the total number of tokens equal to the number of input tokens. The full list is unshuffled to align tokens in their original position. The full list is decoded and the original image is reconstructed. This process has negligible overhead as shuffling and unshuffling operations are fast and no sparse operations(operations performed on matrices consisting of row and column numbers of non-zero numbers) are needed.\n\n\n\n","metadata":{"id":"xldwdC4tJXwY","papermill":{"duration":0.032664,"end_time":"2022-06-19T15:12:30.91752","exception":false,"start_time":"2022-06-19T15:12:30.884856","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install -q noisereduce ","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"id":"LxTl3mN5JXwZ","papermill":{"duration":10.420241,"end_time":"2022-06-19T15:12:41.372615","exception":false,"start_time":"2022-06-19T15:12:30.952374","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.922586Z","iopub.status.idle":"2022-06-24T01:52:39.923178Z","shell.execute_reply.started":"2022-06-24T01:52:39.922971Z","shell.execute_reply":"2022-06-24T01:52:39.922994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install tensorflow_addons ","metadata":{"id":"ZR41eUuIl0al","outputId":"090f4578-1636-42f9-e388-aa37093edc0f","papermill":{"duration":7.936036,"end_time":"2022-06-19T15:12:49.342328","exception":false,"start_time":"2022-06-19T15:12:41.406292","status":"completed"},"tags":[],"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-24T01:52:39.924269Z","iopub.status.idle":"2022-06-24T01:52:39.924634Z","shell.execute_reply.started":"2022-06-24T01:52:39.924427Z","shell.execute_reply":"2022-06-24T01:52:39.924448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport json\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\npd.set_option('max_rows', 250)\npd.set_option('max_columns', 100)\n\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import models\nimport tensorflow as tf\nimport tensorflow_addons as tfa\nfrom tensorflow import keras\n\nimport noisereduce as nr\nfrom math import ceil\n\nimport random\n\n# Setting seeds for reproducibility.\nseed = 42\ntf.random.set_seed(seed)\nnp.random.seed(seed)\nrandom.seed(seed)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"id":"8PXj70QaJXwa","papermill":{"duration":1.955358,"end_time":"2022-06-19T15:12:51.332816","exception":false,"start_time":"2022-06-19T15:12:49.377458","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.926146Z","iopub.status.idle":"2022-06-24T01:52:39.926505Z","shell.execute_reply.started":"2022-06-24T01:52:39.926306Z","shell.execute_reply":"2022-06-24T01:52:39.926328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#seed = 42\nos.environ['PYTHONHASHSEED'] = str(seed)\nnp.random.seed(seed)\n\nDURATION = 15\nSPEC_SHAPE = (48, 128)\nSAMPLE_RATE = 32000\nTEST_DURATION = 5\nSPEC_SHAPE = (48, 128)\nFMIN = 500\nFMAX = 12500\n\n# DATA\nBUFFER_SIZE = 1024\nBATCH_SIZE = 256\nAUTO = tf.data.AUTOTUNE\nINPUT_SHAPE = (48, 128, 3)\nNUM_CLASSES = 151\n\n# OPTIMIZER\nLEARNING_RATE = 5e-3\nWEIGHT_DECAY = 1e-4\n\n# TRAINING\nEPOCHS = 5\n\n# AUGMENTATION\nIMAGE_SIZE = 48 # We'll resize input images to this size.\nIMAGE_SIZE1 = 128 # We'll resize input images to this size.\nPATCH_SIZE = 6 # Size of the patches to be extract from the input images.\nNUM_PATCHES = 168#(IMAGE_SIZE // PATCH_SIZE) ** 2\n\n# ENCODER and DECODER\nLAYER_NORM_EPS = 1e-6\nENC_PROJECTION_DIM = 128\nENC_NUM_HEADS = 4\nENC_LAYERS = 3\nENC_TRANSFORMER_UNITS = [\n ENC_PROJECTION_DIM * 2,\n ENC_PROJECTION_DIM,\n] # Size of the transformer layers.","metadata":{"id":"lh29TujyJXwb","papermill":{"duration":0.045798,"end_time":"2022-06-19T15:12:51.415425","exception":false,"start_time":"2022-06-19T15:12:51.369627","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.928714Z","iopub.status.idle":"2022-06-24T01:52:39.929229Z","shell.execute_reply.started":"2022-06-24T01:52:39.928948Z","shell.execute_reply":"2022-06-24T01:52:39.928974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Masking ratio**\n\nA high masking ratio is optimal for MAE. Ratio of 75% is good for both linear probing and fine tuning. In other models of computer vision, masking ratio is less (between 20%-50%). While in language, the masking ratio is even lesser, around 15%.\n\n## **Mask Tokens**\n\n\nThe masked tokens are dropped and applied again after encoding. If masked tokens are used during encoding , the accuracy drops by 14% in linear probing and 1% in fine-tuning. This is because, in pre-training, a large proportion of tokens are masked and the encoder pre-trains on these tokens which are not part of the uncorrupted image, reducing accuracy. The masked tokens are removed as then encoder pre-trains only on the patches that exist in un-corrupted images.\nRemoving masked token reduces the computational resources .\n\n## **Mask Sampling Strategy**\n\nIn mask sampling strategy, a large block of pixels is removed. At a masking ratio of 50%, fine-tuning and linear probing do not degrade much. But with a higher masking ratio of 75%, the accuracy decreases considerably. Also the reconstruction observed is much blurrier due to higher training loss.Grid-wise sampling has lower training loss, however, the representation quality is low. Simple random sampling works best for MAE, with a high masking ratio providing high speed and good accuracy.\n\n","metadata":{"id":"v3EOTYUxJXwe","papermill":{"duration":0.033521,"end_time":"2022-06-19T15:12:51.483229","exception":false,"start_time":"2022-06-19T15:12:51.449708","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class Patches(layers.Layer):\n def __init__(self, patch_size=PATCH_SIZE):\n super(Patches, self).__init__()\n self.patch_size = patch_size\n\n def call(self, images):\n batch_size = tf.shape(images)[0]\n patches = tf.image.extract_patches(\n images=images,\n sizes=[1, self.patch_size, self.patch_size, 1],\n strides=[1, self.patch_size, self.patch_size, 1],\n rates=[1, 1, 1, 1],\n padding=\"VALID\",\n )\n patch_dims = patches.shape[-1]\n patches = tf.reshape(patches, [batch_size, -1, patch_dims])\n return patches","metadata":{"id":"qAXPHasTJXwe","papermill":{"duration":0.044234,"end_time":"2022-06-19T15:12:51.561336","exception":false,"start_time":"2022-06-19T15:12:51.517102","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.930438Z","iopub.status.idle":"2022-06-24T01:52:39.930934Z","shell.execute_reply.started":"2022-06-24T01:52:39.930678Z","shell.execute_reply":"2022-06-24T01:52:39.930703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class PatchEncoder(layers.Layer):\n def __init__(self, num_patches=NUM_PATCHES, projection_dim=ENC_PROJECTION_DIM):\n super(PatchEncoder, self).__init__()\n self.num_patches = num_patches\n self.projection = layers.Dense(units=projection_dim)\n self.position_embedding = layers.Embedding(\n input_dim=num_patches, output_dim=projection_dim\n )\n\n def call(self, patch):\n positions = tf.range(start=0, limit=self.num_patches, delta=1)\n encoded = self.projection(patch) + self.position_embedding(positions)\n return encoded","metadata":{"id":"3HUW96iDJXwe","papermill":{"duration":0.04412,"end_time":"2022-06-19T15:12:51.63948","exception":false,"start_time":"2022-06-19T15:12:51.59536","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.932034Z","iopub.status.idle":"2022-06-24T01:52:39.932542Z","shell.execute_reply.started":"2022-06-24T01:52:39.932266Z","shell.execute_reply":"2022-06-24T01:52:39.932290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def mlp(x, dropout_rate, hidden_units):\n for units in hidden_units:\n x = layers.Dense(units, activation=tf.nn.gelu)(x)\n x = layers.Dropout(dropout_rate)(x)\n return x\n","metadata":{"id":"f4N-fQIAJXwf","papermill":{"duration":0.044485,"end_time":"2022-06-19T15:12:51.720823","exception":false,"start_time":"2022-06-19T15:12:51.676338","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.934595Z","iopub.status.idle":"2022-06-24T01:52:39.935070Z","shell.execute_reply.started":"2022-06-24T01:52:39.934818Z","shell.execute_reply":"2022-06-24T01:52:39.934842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def create_vit_classifier():\n inputs = layers.Input(shape=(IMAGE_SIZE, IMAGE_SIZE1, 3))\n # Create patches.\n patches = Patches()(inputs)\n # Encode patches.\n encoded_patches = PatchEncoder()(patches)\n\n # Create multiple layers of the Transformer block.\n for _ in range(ENC_LAYERS):\n # Layer normalization 1.\n x1 = layers.LayerNormalization(epsilon=LAYER_NORM_EPS)(encoded_patches)\n # Create a multi-head attention layer.\n attention_output = layers.MultiHeadAttention(\n num_heads=ENC_NUM_HEADS, key_dim=ENC_PROJECTION_DIM, dropout=0.1\n )(x1, x1)\n # Skip connection 1.\n x2 = layers.Add()([attention_output, encoded_patches])\n # Layer normalization 2.\n x3 = layers.LayerNormalization(epsilon=LAYER_NORM_EPS)(x2)\n # MLP.\n x3 = mlp(x3, hidden_units=ENC_TRANSFORMER_UNITS, dropout_rate=0.1)\n # Skip connection 2.\n encoded_patches = layers.Add()([x3, x2])\n \n\n # Create a [batch_size, projection_dim] tensor.\n representation = layers.LayerNormalization(epsilon=LAYER_NORM_EPS)(encoded_patches)\n representation = layers.GlobalAveragePooling1D()(representation)\n \n # Classify outputs.\n outputs = layers.Dense(NUM_CLASSES, activation=\"softmax\")(representation)\n \n # Create the Keras model.\n model = keras.Model(inputs=inputs, outputs=outputs)\n return model","metadata":{"id":"V7nRpIG9JXwf","papermill":{"duration":0.048263,"end_time":"2022-06-19T15:12:51.803857","exception":false,"start_time":"2022-06-19T15:12:51.755594","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.937341Z","iopub.status.idle":"2022-06-24T01:52:39.937852Z","shell.execute_reply.started":"2022-06-24T01:52:39.937587Z","shell.execute_reply":"2022-06-24T01:52:39.937613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **WandbCallback**\n\n","metadata":{}},{"cell_type":"code","source":"run = wandb.init(project='BirdClef2022', group='EDA')\nwandb_callback = wandb.keras.WandbCallback(log_weights=True)\n","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-06-24T01:52:39.939029Z","iopub.status.idle":"2022-06-24T01:52:39.939371Z","shell.execute_reply.started":"2022-06-24T01:52:39.939207Z","shell.execute_reply":"2022-06-24T01:52:39.939224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n vit_model = create_vit_classifier()\n vit_model.compile(\n optimizer='adam',\n loss=\"categorical_crossentropy\",#sparse_categorical_crossentropy\n metrics=[\"accuracy\"]\n )\n \n vit_model.fit(train_x_placeholder,train_y, batch_size= 1,epochs=EPOCHS,callbacks=wandb_callback,verbose=1)\n ","metadata":{"id":"sqiBLt3AJXwf","outputId":"352669e7-4068-4300-e60b-b814f9167d2e","papermill":{"duration":28.816086,"end_time":"2022-06-19T15:13:20.66117","exception":false,"start_time":"2022-06-19T15:12:51.845084","status":"completed"},"tags":[],"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-06-24T01:52:39.940624Z","iopub.status.idle":"2022-06-24T01:52:39.940937Z","shell.execute_reply.started":"2022-06-24T01:52:39.940776Z","shell.execute_reply":"2022-06-24T01:52:39.940792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vit_model.summary()","metadata":{"id":"LkkD0vCNJXwg","papermill":{"duration":0.149282,"end_time":"2022-06-19T15:13:20.9244","exception":false,"start_time":"2022-06-19T15:13:20.775118","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.941961Z","iopub.status.idle":"2022-06-24T01:52:39.942315Z","shell.execute_reply.started":"2022-06-24T01:52:39.942111Z","shell.execute_reply":"2022-06-24T01:52:39.942165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vit_model.save_weights('BirdClef.h5', overwrite=True)","metadata":{"id":"Vogtp31JZoA-","papermill":{"duration":0.314067,"end_time":"2022-06-19T15:13:21.353399","exception":false,"start_time":"2022-06-19T15:13:21.039332","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-06-24T01:52:39.943404Z","iopub.status.idle":"2022-06-24T01:52:39.943751Z","shell.execute_reply.started":"2022-06-24T01:52:39.943587Z","shell.execute_reply":"2022-06-24T01:52:39.943604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The core of deep learning consists of simple algorithms that scale up well. While self supervised learning methods are used in NLP due to exponential scaling models, computer vision still primarily has supervised models. In this paper, authors observe that using autoencoder - a simple self-supervised method similar to techniques in NLP- provides scalable benefits. Self-supervised learning in vision is on the same path as in NLP. Images and languages are different types of signals and these differences must be addressed carefully. Images do not have a semantic decomposition like languages and instead of attempting to remove objects like we do in language, random patches that do not most likely form semantic segment are removed. Thus, the MAE model reconstructs pixels, which are not semantic entities. This behaviour occurs by way of a rich hidden representation inside the MAE.The method predicts content based on statistics learned from the training dataset and will reflect biases in those data, including the ones with a negative societal impact or inexistent content. \n","metadata":{"id":"CaghtRcXJXwg","papermill":{"duration":0.112869,"end_time":"2022-06-19T15:13:21.590872","exception":false,"start_time":"2022-06-19T15:13:21.478003","status":"completed"},"tags":[]}}]}