{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"execution":{"iopub.status.idle":"2021-10-24T08:29:47.628711Z","shell.execute_reply.started":"2021-10-24T08:29:33.728585Z","shell.execute_reply":"2021-10-24T08:29:47.628080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Vision Transformer\n\nThis notebook contains different implementations and variations of the classical Vision Transformer which has been created by Alexey Dosovitskiy et al. In the field of vision based classification, ViT has been instrumental as it uses the self-attention mechanism which is the building blcok of a generic Transformer.The original paper has been [linked](https://arxiv.org/pdf/2010.11929.pdf). Most of the implementations have been abstracted from the [Google Research Repo](https://github.com/google-research/vision_transformer). \n\n","metadata":{}},{"cell_type":"markdown","source":"### Understanding the Vision Transformer\n\nViT has 3 important aspects which involves splitting the image pixels into regular sized patches, applying a linear transformation on them and then adding positional embeddings to the patches to retain spatial embeddings as a trainable input to the neural network. The neural network consists of a standard [Transformer](https://arxiv.org/abs/1706.03762) which uses multihead splitted self attention mechanism for preserving better image characteristics.\n\n\n#### Patch Embedding\n\nThe standard Transformer in case of NLP receives input as a 1D sequence of token embeddings. To handle 2D images, we reshape the image x∈R^{H×W×C} into a sequence of flattened 2D patches.Where, (H, W) is the resolution of the original image and (P, P) is the resolution of each image patch. N = HW/P² is then the effective sequence length for the Transformer. The image is split into fixed-size patches, in the image below, patch size is taken as 16×16. So the dimensions of the image will be 48×48.\n\n\n#### Linear Transformation/Projection\n\nThe patches are then rolled out in a linear manner and passed into an Embedding layer to create Patched Embeddings. The Patched embedding matrix is created by multiplying the trainable embedding weight with the patches. \n\n#### Positional Embedding\n\nPosition embeddings are added to the patched embeddings to retain positional information. We explore different 2D-aware variants of position embeddings without any significant gains over standard 1D position embeddings. The joint embedding serves as input to the Transformer encoder.Each unrolled patch (before Linear Projection) has a sequence of numbers associated with it, in this paper the authors chose it to 1,2,3,4…. no of patches. These numbers are nothing but learnable vectors. Each vector is parameterized and stacked row-wise to form a learnable positional embedding table.Similar to [BERT](https://arxiv.org/abs/1810.04805) which has [cls] tokens, the ViT is prepended with a learnable embedding to the sequence of embedded patches, whose state at the output of the Transformer encoder (zₗ⁰) serves as the image representation y. Both during pre-training and fine-tuning, the classification head is attached to zₗ⁰.\n\n\n\n\n\n ","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nimport tensorflow as tf\nimport tensorflow_datasets as tfds\nimport tensorflow_addons as tfa\nfrom tensorflow.keras.layers import (Dense,Dropout,LayerNormalization)\nfrom tensorflow.keras.layers.experimental.preprocessing import Rescaling\n","metadata":{"execution":{"iopub.status.busy":"2021-10-23T16:38:42.942601Z","iopub.execute_input":"2021-10-23T16:38:42.943199Z","iopub.status.idle":"2021-10-23T16:38:47.843247Z","shell.execute_reply.started":"2021-10-23T16:38:42.943163Z","shell.execute_reply":"2021-10-23T16:38:47.842467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Implementing the base architecture\n\nThe following section provides a basic architecture of the ViT. It consists of a Transformer Block which has a series of [Multihead Attention layers](https://paperswithcode.com/method/graph-self-attention) followed by Layer Normalization and FFNN. \n\n\n\nThe Multihead attention class implements the classical self attention mechanism involving softmax(Q.Kt/temperature)* values computation. More on attention mechanism can be found in [Jay's blogs](https://jalammar.github.io/illustrated-transformer/)\n\n\n\nThe Vision Transformer class contains the mergeing of split attention cells in the Transformer block with the patches created in the previous step. It adds a positional embedding trainable matrix to the input and provides a learnable embedding head for the cls token. It then wraps a series of Transformer blocks and passes the output through a series of Dense FFNN based on the number of labels/classes.","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nimport tensorflow as tf\nimport tensorflow_datasets as tfds\nimport tensorflow_addons as tfa\nfrom tensorflow.keras.layers import (Dense,Dropout,LayerNormalization)\nfrom tensorflow.keras.layers.experimental.preprocessing import Rescaling\n\nclass TransformerBlock(tf.keras.layers.Layer):\n def __init__(self, embed_dim, num_heads, feedforward_dim, dropout=0.1):\n super(TransformerBlock, self).__init__()\n self.multiheadselfattention = MultiHeadAttention(embed_dim, num_heads)\n self.ffn = tf.keras.Sequential(\n [Dense(feedforward_dim, activation=\"relu\"), Dense(embed_dim),]\n )\n self.layernorm1 = LayerNormalization(epsilon=1e-6)\n self.layernorm2 = LayerNormalization(epsilon=1e-6)\n self.dropout1 = Dropout(dropout)\n self.dropout2 = Dropout(dropout)\n \n def call(self, inputs, training):\n out1 = self.layernorm1(inputs) \n attention_output = self.multiheadselfattention(out1)\n attention_output = self.dropout1(attention_output, training=training) \n out2 = self.layernorm1(inputs + attention_output)\n ffn_output = self.ffn(out2)\n ffn_output = self.dropout2(ffn_output, training=training)\n return self.layernorm2(out2 + ffn_output)\n\n\n\nclass MultiHeadAttention(tf.keras.layers.Layer):\n def __init__(self,embed_dim,num_heads):\n super(MultiHeadAttention, self).__init__()\n self.num_heads=num_heads\n self.embed_dim=embed_dim\n assert self.embed_dim%self.num_heads==0\n self.projection_dim=self.embed_dim//self.num_heads\n self.query_dense=Dense(self.embed_dim)\n self.key_dense=Dense(self.embed_dim)\n self.value_dense=Dense(self.embed_dim)\n self.combine_heads=Dense(self.embed_dim)\n \n def self_attention(self,query,key,value):\n q_kt=tf.matmul(a=query,b=key,transpose_b=True)\n key_dims=tf.cast(self.embed_dim**(-0.5),tf.float32)\n normalized_score=q_kt/key_dims\n softmax_wts=tf.nn.softmax(normalized_score,axis=-1)\n output=tf.matmul(softmax_wts,value)\n return output,softmax_wts\n \n def separate_heads(self,x,batch_size):\n x = tf.reshape(x, (batch_size, -1, self.num_heads, self.projection_dim))\n return tf.transpose(x, perm=[0, 2, 1, 3])\n \n def call(self,inputs):\n batch_size = tf.shape(inputs)[0]\n query = self.query_dense(inputs)\n key = self.key_dense(inputs)\n value = self.value_dense(inputs)\n query = self.separate_heads(query, batch_size)\n key = self.separate_heads(key, batch_size)\n value = self.separate_heads(value, batch_size)\n attention, weights = self.self_attention(query, key, value)\n attention = tf.transpose(attention, perm=[0, 2, 1, 3])\n concat_attention = tf.reshape(\n attention, (batch_size, -1, self.embed_dim)\n )\n output = self.combine_heads(concat_attention)\n return output\n\nclass VisionTransformer(tf.keras.Model):\n def __init__(self,image_size,patch_size,num_layers,num_classes,d_model,num_heads,mlp_dim,channels=3,dropout=0.1,**kwargs):\n super(VisionTransformer, self).__init__()\n self.patch_size = patch_size\n num_patches = (image_size // patch_size) ** 2\n self.patch_dim = channels * patch_size ** 2\n self.num_layers=num_layers\n self.d_model = d_model\n self.rescale = Rescaling(1./255)\n self.pos_emb = self.add_weight( \"positional_emb\", shape=(1, num_patches + 1, d_model))\n self.cls_emb = self.add_weight(\"cls_embedding\", shape=(1, 1, d_model))\n self.patch_proj = Dense(d_model)\n self.enc_layers = [\n TransformerBlock(d_model, num_heads, mlp_dim, dropout)\n for _ in range(num_layers)\n ]\n self.mlp_head = tf.keras.Sequential(\n [\n Dense(mlp_dim, activation=tfa.activations.gelu),\n Dropout(dropout),\n Dense(num_classes),\n ]\n )\n \n \n \n def extract_patches(self, images):\n batch_size = tf.shape(images)[0]\n patches = tf.image.extract_patches(\n images=images,\n sizes=[1, self.patch_size, self.patch_size, 1],\n strides=[1, self.patch_size, self.patch_size, 1],\n rates=[1, 1, 1, 1],\n padding=\"VALID\",\n )\n patches = tf.reshape(patches, [batch_size, -1, self.patch_dim])\n return patches\n \n def call(self, x, training):\n batch_size = tf.shape(x)[0]\n x = self.rescale(x)\n patches = self.extract_patches(x)\n x = self.patch_proj(patches)\n\n cls_emb = tf.broadcast_to(self.cls_emb, [batch_size, 1, self.d_model])\n x = tf.concat([cls_emb, x], axis=1)\n x = x + self.pos_emb\n\n for layer in self.enc_layers:\n x = layer(x, training)\n\n # First (cls token) is used for classification\n x = self.mlp_head(x[:, 0])\n return x\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\nIMAGE_SIZE=32\nPATCH_SIZE=7 \nNUM_LAYERS=8\nNUM_HEADS=16\nMLP_DIM=128\nlr=1e-3\nWEIGHT_DECAY=1e-4\nBATCH_SIZE=64\nepochs=1\n\nmodel = VisionTransformer(image_size=IMAGE_SIZE,patch_size=PATCH_SIZE,num_layers=NUM_LAYERS,num_classes=10,d_model=64,num_heads=NUM_HEADS,mlp_dim=MLP_DIM,channels=3,dropout=0.1)\n \nds = tfds.load(\"cifar10\", as_supervised=True)\nds_train = (\n ds[\"train\"]\n .cache()\n .shuffle(1024)\n .batch(BATCH_SIZE)\n .prefetch(AUTOTUNE)\n)\nds_test = (\n ds[\"test\"]\n .cache()\n .batch(BATCH_SIZE)\n .prefetch(AUTOTUNE)\n)\nmodel.compile( loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),optimizer=tfa.optimizers.AdamW(learning_rate=lr, weight_decay=WEIGHT_DECAY),\n metrics=[\"accuracy\"],\n)\nearly_stop = tf.keras.callbacks.EarlyStopping(patience=10),\n# mcp = tf.keras.callbacks.ModelCheckpoint(filepath='../weights/best.h5', \n# save_best_only=True, \n# monitor='val_loss', \n# mode='min')\n# reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', \n# factor=0.1, \n# patience=3, \n# verbose=0, \n# mode='auto',\n#min_delta=0.0001, cooldown=0, min_lr=0)\nmodel.fit(\n ds_train,\n validation_data=ds_test,\n epochs=epochs,\n callbacks=[early_stop],\n)\nprint(model.summary())\n","metadata":{"execution":{"iopub.status.busy":"2021-10-23T16:38:47.844811Z","iopub.execute_input":"2021-10-23T16:38:47.846003Z","iopub.status.idle":"2021-10-23T16:40:32.578095Z","shell.execute_reply.started":"2021-10-23T16:38:47.845966Z","shell.execute_reply":"2021-10-23T16:40:32.577320Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2021-10-23T16:40:32.579673Z","iopub.execute_input":"2021-10-23T16:40:32.580025Z","iopub.status.idle":"2021-10-23T16:40:32.599578Z","shell.execute_reply.started":"2021-10-23T16:40:32.579987Z","shell.execute_reply":"2021-10-23T16:40:32.598852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ViT on the Casava dataset\n\nThis is an implementation of a compact ViT on the casava dataset. The initial part remains augmenting the dataset by rotating and flipping .In this case keras image generator is used to generate the images after preprocessing and splitting them into train and test sets. Some useful links for preprocessing is provided [here](https://www.tensorflow.org/tutorials/load_data/images)","metadata":{}},{"cell_type":"code","source":"import glob\nimport pandas as pd\nimport numpy as np\nimage_size = 224\nbatch_size = 16\nn_classes = 5\n\ntrain_path = '/kaggle/input/cassava-leaf-disease-classification/train_images'\ntest_path = '/kaggle/input/cassava-leaf-disease-classification/test_images'\n\ndf_train = pd.read_csv('/kaggle/input/cassava-leaf-disease-classification/train.csv', dtype = 'str')\n\ntest_images = glob.glob(test_path + '/*.jpg')\ndf_test = pd.DataFrame(test_images, columns = ['image_path'])\n\nclasses = {0 : \"Cassava Bacterial Blight (CBB)\",\n 1 : \"Cassava Brown Streak Disease (CBSD)\",\n 2 : \"Cassava Green Mottle (CGM)\",\n 3 : \"Cassava Mosaic Disease (CMD)\",\n 4 : \"Healthy\"}","metadata":{"execution":{"iopub.status.busy":"2021-10-23T16:40:32.600780Z","iopub.execute_input":"2021-10-23T16:40:32.601112Z","iopub.status.idle":"2021-10-23T16:40:32.634011Z","shell.execute_reply.started":"2021-10-23T16:40:32.601078Z","shell.execute_reply":"2021-10-23T16:40:32.633337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def data_augment(image):\n p_spatial = tf.random.uniform([], 0, 1.0, dtype = tf.float32)\n p_rotate = tf.random.uniform([], 0, 1.0, dtype = tf.float32)\n image = tf.image.random_flip_left_right(image)\n image = tf.image.random_flip_up_down(image)\n \n if p_spatial > .75:\n image = tf.image.transpose(image)\n \n # Rotates\n if p_rotate > .75:\n image = tf.image.rot90(image, k = 3) # rotate 270º\n elif p_rotate > .5:\n image = tf.image.rot90(image, k = 2) # rotate 180º\n elif p_rotate > .25:\n image = tf.image.rot90(image, k = 1) # rotate 90º\n return image\ndatagen = tf.keras.preprocessing.image.ImageDataGenerator(samplewise_center = True,samplewise_std_normalization = True,validation_split = 0.2,preprocessing_function = data_augment)\n\ntrain_gen = datagen.flow_from_dataframe(dataframe = df_train,\n directory = train_path,\n x_col = 'image_id',\n y_col = 'label',\n subset = 'training',\n batch_size = batch_size,\n seed = 1,\n color_mode = 'rgb',\n shuffle = True,\n class_mode = 'categorical',\n target_size = (image_size, image_size))\n\nvalid_gen = datagen.flow_from_dataframe(dataframe = df_train,\n directory = train_path,\n x_col = 'image_id',\n y_col = 'label',\n subset = 'validation',\n batch_size = batch_size,\n seed = 1,\n color_mode = 'rgb',\n shuffle = False,\n class_mode = 'categorical',\n target_size = (image_size, image_size))\n\ntest_gen = datagen.flow_from_dataframe(dataframe = df_test,\n x_col = 'image_path',\n y_col = None,\n batch_size = batch_size,\n seed = 1,\n color_mode = 'rgb',\n shuffle = False,\n class_mode = None,\n target_size = (image_size, image_size))\n","metadata":{"execution":{"iopub.status.busy":"2021-10-23T16:40:32.635870Z","iopub.execute_input":"2021-10-23T16:40:32.636145Z","iopub.status.idle":"2021-10-23T16:40:47.871117Z","shell.execute_reply.started":"2021-10-23T16:40:32.636111Z","shell.execute_reply":"2021-10-23T16:40:47.870339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Evaluating with the Base ViT \n\nThe following code snippet is for training the base ViT implementation on hte Casava dataset.","metadata":{}},{"cell_type":"code","source":"STEP_SIZE_TRAIN = train_gen.n // train_gen.batch_size\nSTEP_SIZE_VALID = valid_gen.n // valid_gen.batch_size\nnum_epochs=2\n\nmodel = VisionTransformer(image_size=image_size,patch_size=7,num_layers=NUM_LAYERS,num_classes=n_classes,d_model=64,num_heads=NUM_HEADS,mlp_dim=MLP_DIM,channels=3,dropout=0.1)\nmodel.compile(optimizer = 'adam', \n loss = tf.keras.losses.CategoricalCrossentropy(label_smoothing = 0.1), \n metrics = ['accuracy'])\nearlystopping = tf.keras.callbacks.EarlyStopping(monitor = 'val_accuracy',\n min_delta = 1e-4,\n patience = 5,\n mode = 'max',\n restore_best_weights = True,\n verbose = 1)\n\ncheckpointer = tf.keras.callbacks.ModelCheckpoint(filepath = './model.hdf5',\n monitor = 'val_accuracy', \n verbose = 1, \n save_best_only = True,\n save_weights_only = True,\n mode = 'max')\n\ncallbacks = [earlystopping, checkpointer]\n\nmodel.fit(x = train_gen,\n steps_per_epoch = STEP_SIZE_TRAIN,\n validation_data = valid_gen,\n validation_steps = STEP_SIZE_VALID,\n epochs = num_epochs,\n callbacks = callbacks)","metadata":{"execution":{"iopub.status.busy":"2021-10-23T17:16:48.834308Z","iopub.execute_input":"2021-10-23T17:16:48.834678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Compact Alternative ViT\n\nThis is another implementation of ViT which is similar to the previous one but has minor modifications in terms of input splitting and patch embedding generation.Also the trainable matrix for MLP can be 2D as it allows better expressivity of the spatial features of the encoded patches. The transformer block can be wrapped around as many blocks but a restriction has to be kept on the memory usage of such large blocks. \nThe compact architecture divides the images into patch embeddings in semantic space and then uses a relative positional embedding for correct spatial representation. \n","metadata":{}},{"cell_type":"code","source":"#Compact transformer\n\n\nclass Patch(tf.keras.layers.Layer):\n def __init__(self, patch_size):\n super(Patch, self).__init__()\n self.patch_size = patch_size\n\n def call(self, images):\n batch_size = tf.shape(images)[0]\n patches = tf.image.extract_patches(\n images = images,\n sizes = [1, self.patch_size, self.patch_size, 1],\n strides = [1, self.patch_size, self.patch_size, 1],\n rates = [1, 1, 1, 1],\n padding = 'VALID',\n )\n patch_dims = patches.shape[-1]\n patches = tf.reshape(patches, [batch_size, -1, patch_dims])\n return patches\n \n\nclass Patch_Embeddings(tf.keras.layers.Layer):\n def __init__(self,num_patches,projection_dim):\n super(Patch_Embeddings,self).__init__()\n \n self.num_patches=num_patches\n self.projection=tf.keras.layers.Dense(projection_dim)\n self.position_embedding=tf.keras.layers.Embedding(input_dim=self.num_patches,output_dim=projection_dim)\n \n def call(self,patch):\n positions=tf.range(start=0,limit=self.num_patches,delta=1)\n x=self.projection(patch)+self.position_embedding(positions)\n return x\n\n \ndef mlp(x, hidden_units, dropout_rate):\n for units in hidden_units:\n x = tf.keras.layers.Dense(units, activation = tf.nn.gelu)(x)\n x = tf.keras.layers.Dropout(dropout_rate)(x)\n return x\n\ndef compactVIT(image_size,patch_size,num_patches,projection_dim,transformer_layers,num_heads,transformer_units,mlp_head_units):\n inputs=tf.keras.layers.Input(shape=(image_size,image_size,3))\n patch=Patch(patch_size)(inputs)\n encoded_patches=Patch_Embeddings(num_patches,projection_dim)(patch)\n for _ in range(transformer_layers):\n x1 = tf.keras.layers.LayerNormalization(epsilon = 1e-6)(encoded_patches)\n attention_output =tf.keras.layers.MultiHeadAttention(num_heads = num_heads, key_dim = projection_dim, dropout = 0.1)(x1, x1)\n x2 = tf.keras.layers.Add()([attention_output, encoded_patches])\n x3 = tf.keras.layers.LayerNormalization(epsilon = 1e-6)(x2)\n x3 = mlp(x3, hidden_units = transformer_units, dropout_rate = 0.1)\n encoded_patches = tf.keras.layers.Add()([x3, x2])\n\n representation = tf.keras.layers.LayerNormalization(epsilon = 1e-6)(encoded_patches)\n representation = tf.keras.layers.Flatten()(representation)\n representation = tf.keras.layers.Dropout(0.5)(representation)\n \n features = mlp(representation, hidden_units = mlp_head_units, dropout_rate = 0.5)\n logits = tf.keras.layers.Dense(n_classes)(features)\n model = tf.keras.Model(inputs = inputs, outputs = logits)\n print(model.summary())\n return model\n\npatch_size = 7 \ntransformer_layers=3\nnum_patches = (image_size // patch_size) ** 2\nprojection_dim = 64\nnum_heads=8\nmlp_head_units = [128, 28]\ntransformer_units = [projection_dim * 2,projection_dim,]\nmodel=compactVIT(image_size,patch_size,num_patches,projection_dim,transformer_layers,num_heads,transformer_units,mlp_head_units)\nmodel.compile(optimizer = 'adam', \n loss = tf.keras.losses.CategoricalCrossentropy(label_smoothing = 0.1), \n metrics = ['accuracy'])\n\nmodel.fit(x = train_gen,\n steps_per_epoch = STEP_SIZE_TRAIN,\n validation_data = valid_gen,\n validation_steps = STEP_SIZE_VALID,\n epochs = num_epochs,\n callbacks = callbacks)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Hybrid ViT \n\n\n\nHybrid ViT is a version of visual transformers where a preliminary convolution kernel is added before passing the patched embeddings to the MHSA of the transformer model. This allows better visual reception of the pixels. The initial embedding space can be compounded with any variation of Convolution networks such as Resnet,Exception,Inception ,Efficient Net and so on. Some of the most popular architectures for these are provided in [Tf blog here](https://www.tensorflow.org/api_docs/python/tf/keras/applications) . A \"hybrid\" model is ideally a model that uses convolutional layers in the earlier layers followed by self-attention. Hybrid ViT, BotNet, Visformer are all examples of this approach. A pictorial depiction of the same is provided here: \n\n\n\n\n### Visformer and Bottleneck Transformer\n\nHere we see that a convolution kernel based embedding is passed as input to the MHSA Transformer net. The variety of the architectures which can be added before the Transformer kernel makes it highly efficient.\nSome resources:\n\n- [Visformer](https://arxiv.org/abs/2104.12533)\n- [Bottleneck Transformer](https://arxiv.org/abs/2101.11605)\n\nA rational direction is then to consider hybrid architectures that mix convolutional layers where they are most convenient with transformer self-attention layers where their impact is maximal.In this hybrid vein, there’s the Visformer along with the bottleneck transformer and standalone self-attention. The following implements a minimalistic ResNet bottleneck Visformer :\n\n\n\n","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nimport tensorflow as tf\nimport tensorflow_datasets as tfds\nimport tensorflow_addons as tfa\nfrom tensorflow.keras.layers import (Dense,Dropout,LayerNormalization)\nfrom tensorflow.keras.layers.experimental.preprocessing import Rescaling\n\nclass TransformerBlock(tf.keras.layers.Layer):\n def __init__(self, embed_dim, num_heads, feedforward_dim, dropout=0.1):\n super(TransformerBlock, self).__init__()\n self.multiheadselfattention = MultiHeadAttention(embed_dim, num_heads)\n self.ffn = tf.keras.Sequential(\n [Dense(feedforward_dim, activation=\"relu\"), Dense(embed_dim),]\n )\n self.layernorm1 = LayerNormalization(epsilon=1e-6)\n self.layernorm2 = LayerNormalization(epsilon=1e-6)\n self.dropout1 = Dropout(dropout)\n self.dropout2 = Dropout(dropout)\n \n def call(self, inputs, training):\n out1 = self.layernorm1(inputs) \n attention_output = self.multiheadselfattention(out1)\n attention_output = self.dropout1(attention_output, training=training) \n out2 = self.layernorm1(inputs + attention_output)\n ffn_output = self.ffn(out2)\n ffn_output = self.dropout2(ffn_output, training=training)\n return self.layernorm2(out2 + ffn_output)\n\n\n\nclass MultiHeadAttention(tf.keras.layers.Layer):\n def __init__(self,embed_dim,num_heads):\n super(MultiHeadAttention, self).__init__()\n self.num_heads=num_heads\n self.embed_dim=embed_dim\n assert self.embed_dim%self.num_heads==0\n self.projection_dim=self.embed_dim//self.num_heads\n self.query_dense=Dense(self.embed_dim)\n self.key_dense=Dense(self.embed_dim)\n self.value_dense=Dense(self.embed_dim)\n self.combine_heads=Dense(self.embed_dim)\n \n def self_attention(self,query,key,value):\n q_kt=tf.matmul(a=query,b=key,transpose_b=True)\n key_dims=tf.cast(self.embed_dim**(-0.5),tf.float32)\n normalized_score=q_kt/key_dims\n softmax_wts=tf.nn.softmax(normalized_score,axis=-1)\n output=tf.matmul(softmax_wts,value)\n return output,softmax_wts\n \n def separate_heads(self,x,batch_size):\n x = tf.reshape(x, (batch_size, -1, self.num_heads, self.projection_dim))\n return tf.transpose(x, perm=[0, 2, 1, 3])\n \n def call(self,inputs):\n batch_size = tf.shape(inputs)[0]\n query = self.query_dense(inputs)\n key = self.key_dense(inputs)\n value = self.value_dense(inputs)\n query = self.separate_heads(query, batch_size)\n key = self.separate_heads(key, batch_size)\n value = self.separate_heads(value, batch_size)\n attention, weights = self.self_attention(query, key, value)\n attention = tf.transpose(attention, perm=[0, 2, 1, 3])\n concat_attention = tf.reshape(\n attention, (batch_size, -1, self.embed_dim)\n )\n output = self.combine_heads(concat_attention)\n return output\n\nclass VisionTransformer(tf.keras.Model):\n def __init__(self,image_size,patch_size,num_layers,num_classes,d_model,num_heads,mlp_dim,channels=3,dropout=0.1,**kwargs):\n super(VisionTransformer, self).__init__()\n self.patch_size = patch_size\n num_patches = (image_size // patch_size) ** 2\n self.patch_dim = channels * patch_size ** 2\n self.num_layers=num_layers\n self.d_model = d_model\n self.rescale = Rescaling(1./255)\n self.pos_emb = self.add_weight( \"positional_emb\", shape=(1, num_patches + 1, d_model))\n self.cls_emb = self.add_weight(\"cls_embedding\", shape=(1, 1, d_model))\n self.patch_proj = Dense(d_model)\n self.enc_layers = [\n TransformerBlock(d_model, num_heads, mlp_dim, dropout)\n for _ in range(num_layers)\n ]\n self.mlp_head = tf.keras.Sequential(\n [\n Dense(mlp_dim, activation=tfa.activations.gelu),\n Dropout(dropout),\n Dense(num_classes),\n ]\n )\n \n \n \n def extract_patches(self, images):\n batch_size = tf.shape(images)[0]\n patches = tf.image.extract_patches(\n images=images,\n sizes=[1, self.patch_size, self.patch_size, 1],\n strides=[1, self.patch_size, self.patch_size, 1],\n rates=[1, 1, 1, 1],\n padding=\"VALID\",\n )\n patches = tf.reshape(patches, [batch_size, -1, self.patch_dim])\n return patches\n \n def call(self, x, training):\n print(\"shapes\",x.shape)\n batch_size = tf.shape(x)[0]\n x = self.rescale(x)\n patches = self.extract_patches(x)\n x = self.patch_proj(patches)\n\n cls_emb = tf.broadcast_to(self.cls_emb, [batch_size, 1, self.d_model])\n x = tf.concat([cls_emb, x], axis=1)\n x = x + self.pos_emb\n\n for layer in self.enc_layers:\n x = layer(x, training)\n\n # First (cls token) is used for classification\n x = self.mlp_head(x[:, 0])\n return x\n\nclass HybridFormer(tf.keras.Model):\n def __init__(self,image_size,patch_size,num_layers,num_classes,d_model,num_heads,mlp_dim,channels=3,dropout=0.1,**kwargs):\n super(HybridFormer,self).__init__()\n self.pretrained_model=tf.keras.applications.resnet.ResNet101(input_shape=(image_size,image_size,3),include_top=False,weights='imagenet')\n self.pretrained_model.trainable=False\n self.vision_transformer=VisionTransformer(image_size=IMAGE_SIZE,patch_size=PATCH_SIZE,num_layers=NUM_LAYERS,num_classes=10,d_model=64,num_heads=NUM_HEADS,mlp_dim=MLP_DIM,channels=3,dropout=0.1)\n \n def call(self,x,training):\n output_p=self.pretrained_model(x)\n output_trans=self.vision_transformer(output_p,training)\n return output_trans\n \n \n \n########### Fitting on CIFAR ##########################\nprint(\"Fitting on CIFAR Dataset\")\nAUTOTUNE = tf.data.experimental.AUTOTUNE\nIMAGE_SIZE=32\nPATCH_SIZE=7 \nNUM_LAYERS=8\nNUM_HEADS=16\nMLP_DIM=128\nlr=1e-3\nWEIGHT_DECAY=1e-4\nBATCH_SIZE=64\nepochs=1\n\nmodel = HybridFormer(image_size=IMAGE_SIZE,patch_size=PATCH_SIZE,num_layers=NUM_LAYERS,num_classes=10,d_model=64,num_heads=NUM_HEADS,mlp_dim=MLP_DIM,channels=3,dropout=0.1)\n \nds = tfds.load(\"cifar10\", as_supervised=True)\nds_train = (\n ds[\"train\"]\n .cache()\n .shuffle(1024)\n .batch(BATCH_SIZE)\n .prefetch(AUTOTUNE)\n)\nds_test = (\n ds[\"test\"]\n .cache()\n .batch(BATCH_SIZE)\n .prefetch(AUTOTUNE)\n)\nmodel.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),optimizer=tfa.optimizers.AdamW(learning_rate=lr, weight_decay=WEIGHT_DECAY),\n metrics=[\"accuracy\"],\n)\nearly_stop = tf.keras.callbacks.EarlyStopping(patience=10),\n# mcp = tf.keras.callbacks.ModelCheckpoint(filepath='../weights/best.h5', \n# save_best_only=True, \n# monitor='val_loss', \n# mode='min')\n# reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', \n# factor=0.1, \n# patience=3, \n# verbose=0, \n# mode='auto',\n#min_delta=0.0001, cooldown=0, min_lr=0)\nmodel.fit(\n ds_train,\n validation_data=ds_test,\n epochs=epochs,\n callbacks=[early_stop],\n)\nprint(model.summary())\n\n\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Compact Hybrid transformer\n\nSTEP_SIZE_TRAIN = train_gen.n // train_gen.batch_size\nSTEP_SIZE_VALID = valid_gen.n // valid_gen.batch_size\nnum_epochs=2\n\nclass Patch(tf.keras.layers.Layer):\n def __init__(self, patch_size):\n super(Patch, self).__init__()\n self.patch_size = patch_size\n\n def call(self, images):\n batch_size = tf.shape(images)[0]\n patches = tf.image.extract_patches(\n images = images,\n sizes = [1, self.patch_size, self.patch_size, 1],\n strides = [1, self.patch_size, self.patch_size, 1],\n rates = [1, 1, 1, 1],\n padding = 'VALID',\n )\n patch_dims = patches.shape[-1]\n patches = tf.reshape(patches, [batch_size, -1, patch_dims])\n return patches\n \n\nclass Patch_Embeddings(tf.keras.layers.Layer):\n def __init__(self,num_patches,projection_dim):\n super(Patch_Embeddings,self).__init__()\n \n self.num_patches=num_patches\n self.projection=tf.keras.layers.Dense(projection_dim)\n self.position_embedding=tf.keras.layers.Embedding(input_dim=self.num_patches,output_dim=projection_dim)\n \n def call(self,patch):\n positions=tf.range(start=0,limit=self.num_patches,delta=1)\n x=self.projection(patch)+self.position_embedding(positions)\n return x\n\n \ndef mlp(x, hidden_units, dropout_rate):\n for units in hidden_units:\n x = tf.keras.layers.Dense(units, activation = tf.nn.gelu)(x)\n x = tf.keras.layers.Dropout(dropout_rate)(x)\n return x\n\ndef compacthybrid_VIT(image_size,patch_size,num_patches,projection_dim,transformer_layers,num_heads,transformer_units,mlp_head_units):\n inputs=tf.keras.layers.Input(shape=(image_size,image_size,3))\n embeddings=tf.keras.applications.resnet.ResNet101(input_shape=(image_size,image_size,3),include_top=False,weights='imagenet')(inputs)\n patch=Patch(patch_size)(embeddings)\n encoded_patches=Patch_Embeddings(num_patches,projection_dim)(patch)\n for _ in range(transformer_layers):\n x1 = tf.keras.layers.LayerNormalization(epsilon = 1e-6)(encoded_patches)\n attention_output =tf.keras.layers.MultiHeadAttention(num_heads = num_heads, key_dim = projection_dim, dropout = 0.1)(x1, x1)\n x2 = tf.keras.layers.Add()([attention_output, encoded_patches])\n x3 = tf.keras.layers.LayerNormalization(epsilon = 1e-6)(x2)\n x3 = mlp(x3, hidden_units = transformer_units, dropout_rate = 0.1)\n encoded_patches = tf.keras.layers.Add()([x3, x2])\n\n representation = tf.keras.layers.LayerNormalization(epsilon = 1e-6)(encoded_patches)\n representation = tf.keras.layers.Flatten()(representation)\n representation = tf.keras.layers.Dropout(0.5)(representation)\n \n features = mlp(representation, hidden_units = mlp_head_units, dropout_rate = 0.5)\n logits = tf.keras.layers.Dense(n_classes)(features)\n model = tf.keras.Model(inputs = inputs, outputs = logits)\n print(model.summary())\n return model\n\npatch_size = 7 \ntransformer_layers=3\nnum_patches = (image_size // patch_size) ** 2\nprojection_dim = 64\nnum_heads=8\nmlp_head_units = [128, 28]\ntransformer_units = [projection_dim * 2,projection_dim,]\nmodel=compacthybrid_VIT(image_size,patch_size,num_patches,projection_dim,transformer_layers,num_heads,transformer_units,mlp_head_units)\nmodel.compile(optimizer = 'adam', \n loss = tf.keras.losses.CategoricalCrossentropy(label_smoothing = 0.1), \n metrics = ['accuracy'])\nearlystopping = tf.keras.callbacks.EarlyStopping(monitor = 'val_accuracy',\n min_delta = 1e-4,\n patience = 5,\n mode = 'max',\n restore_best_weights = True,\n verbose = 1)\n\ncheckpointer = tf.keras.callbacks.ModelCheckpoint(filepath = './model.hdf5',\n monitor = 'val_accuracy', \n verbose = 1, \n save_best_only = True,\n save_weights_only = True,\n mode = 'max')\n\ncallbacks = [earlystopping, checkpointer]\nmodel.fit(x = train_gen,\n steps_per_epoch = STEP_SIZE_TRAIN,\n validation_data = valid_gen,\n validation_steps = STEP_SIZE_VALID,\n epochs = num_epochs,\n callbacks = callbacks)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import wandb\nfrom wandb.keras import WandbCallback\nfrom kaggle_secrets import UserSecretsClient\n\nuser_secrets = UserSecretsClient()\nwandb_api = user_secrets.get_secret(\"wandb_api\")\n\nwandb.login(key=wandb_api)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class TransformerBlock(tf.keras.layers.Layer):\n def __init__(self, embed_dim, num_heads, feedforward_dim, dropout=0.1):\n super(TransformerBlock, self).__init__()\n self.multiheadselfattention = MultiHeadAttention(embed_dim, num_heads)\n self.ffn = tf.keras.Sequential(\n [Dense(feedforward_dim, activation=\"relu\"), Dense(embed_dim),]\n )\n self.layernorm1 = LayerNormalization(epsilon=1e-6)\n self.layernorm2 = LayerNormalization(epsilon=1e-6)\n self.dropout1 = Dropout(dropout)\n self.dropout2 = Dropout(dropout)\n \n def call(self, inputs, training):\n out1 = self.layernorm1(inputs) \n attention_output = self.multiheadselfattention(out1)\n attention_output = self.dropout1(attention_output, training=training) \n out2 = self.layernorm1(inputs + attention_output)\n ffn_output = self.ffn(out2)\n ffn_output = self.dropout2(ffn_output, training=training)\n return self.layernorm2(out2 + ffn_output)\n\n\n\nclass MultiHeadAttention(tf.keras.layers.Layer):\n def __init__(self,embed_dim,num_heads):\n super(MultiHeadAttention, self).__init__()\n self.num_heads=num_heads\n self.embed_dim=embed_dim\n assert self.embed_dim%self.num_heads==0\n self.projection_dim=self.embed_dim//self.num_heads\n self.query_dense=Dense(self.embed_dim)\n self.key_dense=Dense(self.embed_dim)\n self.value_dense=Dense(self.embed_dim)\n self.combine_heads=Dense(self.embed_dim)\n self.proj=self.add_weight(\"test\",shape=(256,self.projection_dim))\n \n def self_attention(self,query,key,value):\n q_kt=tf.matmul(a=query,b=key,transpose_b=True)\n key_dims=tf.cast(self.embed_dim**(-0.5),tf.float32)\n normalized_score=q_kt/key_dims\n softmax_wts=tf.nn.softmax(normalized_score,axis=-1)\n output=tf.matmul(softmax_wts,value)\n return output,softmax_wts\n \n def separate_heads(self,x,batch_size):\n x = tf.reshape(x, (batch_size, -1, self.num_heads, self.projection_dim))\n return tf.transpose(x, perm=[0, 2, 1, 3])\n \n def call(self,inputs):\n batch_size = tf.shape(inputs)[0]\n query = self.query_dense(inputs)\n key = self.key_dense(inputs)\n value = self.value_dense(inputs)\n query = self.separate_heads(query, batch_size)\n key = self.separate_heads(key, batch_size)\n value = self.separate_heads(value, batch_size)\n attention, weights = self.self_attention(query, key, value)\n attention = tf.transpose(attention, perm=[0, 2, 1, 3])\n concat_attention = tf.reshape(\n attention, (batch_size, -1, self.embed_dim)\n )\n output = self.combine_heads(concat_attention)\n return output\n\nclass VisionTransformer(tf.keras.Model):\n def __init__(self,image_size,patch_size,num_layers,num_classes,d_model,num_heads,mlp_dim,channels=3,dropout=0.1,**kwargs):\n super(VisionTransformer, self).__init__()\n self.patch_size = patch_size\n num_patches = (image_size // patch_size) ** 2\n self.patch_dim = channels * patch_size ** 2\n self.num_layers=num_layers\n self.d_model = d_model\n self.rescale = Rescaling(1./255)\n self.pos_emb = self.add_weight( \"positional_emb\", shape=(1, num_patches + 1, d_model))\n self.cls_emb = self.add_weight(\"cls_embedding\", shape=(1, 1, d_model))\n self.patch_proj = Dense(d_model)\n self.enc_layers = [\n TransformerBlock(d_model, num_heads, mlp_dim, dropout)\n for _ in range(num_layers)\n ]\n self.mlp_head = tf.keras.Sequential(\n [\n Dense(mlp_dim, activation=tfa.activations.gelu),\n Dropout(dropout),\n Dense(num_classes),\n ]\n )\n \n \n \n def extract_patches(self, images):\n batch_size = tf.shape(images)[0]\n patches = tf.image.extract_patches(\n images=images,\n sizes=[1, self.patch_size, self.patch_size, 1],\n strides=[1, self.patch_size, self.patch_size, 1],\n rates=[1, 1, 1, 1],\n padding=\"VALID\",\n )\n patches = tf.reshape(patches, [batch_size, -1, self.patch_dim])\n return patches\n \n def call(self, x, training):\n print(\"shapes\",x.shape)\n batch_size = tf.shape(x)[0]\n x = self.rescale(x)\n patches = self.extract_patches(x)\n x = self.patch_proj(patches)\n\n cls_emb = tf.broadcast_to(self.cls_emb, [batch_size, 1, self.d_model])\n x = tf.concat([cls_emb, x], axis=1)\n x = x + self.pos_emb\n\n for layer in self.enc_layers:\n x = layer(x, training)\n\n # First (cls token) is used for classification\n x = self.mlp_head(x[:, 0])\n return x\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## YOLO\n\nYolo is an algorithm that uses convolutional neural networks for object detection.\nSo what's great about object detection? In comparison to recognition algorithms, a detection algorithm does not only predict class labels, but detects locations of objects as well.\n\n\n\n\n\n### How does YOLOv3 work? (Overview)\n\n[YOLO](https://arxiv.org/pdf/1804.02767.pdf) is a Convolutional Neural Network (CNN) for performing object detection in real-time. CNNs are classifier-based systems that can process input images as structured arrays of data and identify patterns between them (view image below). YOLO has the advantage of being much faster than other networks and still maintains accuracy.\n\nIt allows the model to look at the whole image at test time, so its predictions are informed by the global context in the image. YOLO and other convolutional neural network algorithms “score” regions based on their similarities to predefined classes.\n\nHigh-scoring regions are noted as positive detections of whatever class they most closely identify with. For example, in a live feed of traffic, YOLO can be used to detect different kinds of vehicles depending on which regions of the video score highly in comparison to predefined classes of vehicles.\n\n\n\n\nThe YOLOv3 algorithm first separates an image into a grid. Each grid cell predicts some number of boundary boxes (sometimes referred to as anchor boxes) around objects that score highly with the aforementioned predefined classes.\n\nEach boundary box has a respective confidence score of how accurate it assumes that prediction should be and detects only one object per bounding box. The boundary boxes are generated by clustering the dimensions of the ground truth boxes from the original dataset to find the most common shapes and sizes.\n\nOther comparable algorithms that can carry out the same objective are R-CNN (Region-based Convolutional Neural Networks made in 2015) and Fast R-CNN (R-CNN improvement developed in 2017), and [Mask R-CNN](https://arxiv.org/abs/1703.06870).\n\n- [Kaggle kernel for MRCNN](https://www.kaggle.com/drt2290078/mask-rcnn-sample-starter-code)\n- [Resource](https://paperswithcode.com/paper/mask-r-cnn)\n\nHowever, unlike systems like R-CNN and Fast R-CNN, YOLO is trained to do classification and bounding box regression at the same time.\n\n\n### Model Weights\n\nWeights and cfg (or configuration) files can be downloaded from the website of the original creator of [YOLOv3](https://pjreddie.com/darknet/yolo). You can also (more easily) use YOLO’s COCO pretrained weights by initializing the model with model = YOLOv3().\n\nUsing COCO’s pre-trained weights means that you can only use YOLO for object detection with any of the 80 pretrained classes that come with the COCO dataset. This is a good option for beginners because it requires the least amount of new code and customization.\n\nThe following 80 classes are available using COCO’s pretrained weights:\n\n\n``'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', 'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis','snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush'\n``\n\n### Model Definition\n\nBatch norm and fixed padding: It's useful to define batch_norm function since the model uses batch norms with shared parameters heavily. Also, same as ResNet, Yolo uses convolution with fixed padding, which means that padding is defined only by the size of the kernel.\n\n","metadata":{}},{"cell_type":"code","source":"import tensorflow.compat.v1 as tf\nimport numpy as np\nfrom PIL import Image, ImageDraw, ImageFont\nfrom IPython.display import display\nfrom seaborn import color_palette\nimport cv2\ntf.compat.v1.disable_eager_execution()\n#hyperparams\n_BATCH_NORM_DECAY = 0.9\n_BATCH_NORM_EPSILON = 1e-05\n_LEAKY_RELU = 0.1\n_ANCHORS = [(10, 13), (16, 30), (33, 23),\n (30, 61), (62, 45), (59, 119),\n (116, 90), (156, 198), (373, 326)]\n_MODEL_SIZE = (416, 416)\n\ndef batch_norm(inputs, training, data_format):\n \"\"\"Performs a batch normalization using a standard set of parameters.\"\"\"\n return tf.layers.batch_normalization(\n inputs=inputs, axis=1 if data_format == 'channels_first' else 3,\n momentum=_BATCH_NORM_DECAY, epsilon=_BATCH_NORM_EPSILON,\n scale=True, training=training)\n\n\ndef fixed_padding(inputs, kernel_size, data_format):\n \"\"\"ResNet implementation of fixed padding.\n\n Pads the input along the spatial dimensions independently of input size.\n\n Args:\n inputs: Tensor input to be padded.\n kernel_size: The kernel to be used in the conv2d or max_pool2d.\n data_format: The input format.\n Returns:\n A tensor with the same format as the input.\n \"\"\"\n pad_total = kernel_size - 1\n pad_beg = pad_total // 2\n pad_end = pad_total - pad_beg\n\n if data_format == 'channels_first':\n padded_inputs = tf.pad(inputs, [[0, 0], [0, 0],\n [pad_beg, pad_end],\n [pad_beg, pad_end]])\n else:\n padded_inputs = tf.pad(inputs, [[0, 0], [pad_beg, pad_end],\n [pad_beg, pad_end], [0, 0]])\n return padded_inputs\n\n\ndef conv2d_fixed_padding(inputs, filters, kernel_size, data_format, strides=1):\n \"\"\"Strided 2-D convolution with explicit padding.\"\"\"\n if strides > 1:\n inputs = fixed_padding(inputs, kernel_size, data_format)\n\n return tf.layers.conv2d(\n inputs=inputs, filters=filters, kernel_size=kernel_size,\n strides=strides, padding=('SAME' if strides == 1 else 'VALID'),\n use_bias=False, data_format=data_format)\n","metadata":{"execution":{"iopub.status.busy":"2021-10-24T07:21:48.075369Z","iopub.execute_input":"2021-10-24T07:21:48.075617Z","iopub.status.idle":"2021-10-24T07:21:53.122388Z","shell.execute_reply.started":"2021-10-24T07:21:48.075591Z","shell.execute_reply":"2021-10-24T07:21:53.121594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Feature extraction: Darknet-53\n\nFor feature extraction Yolo uses Darknet-53 neural net pretrained on ImageNet. Same as ResNet, Darknet-53 has shortcut (residual) connections, which help information from earlier layers flow further. We omit the last 3 layers (Avgpool, Connected and Softmax) since we only need the features.","metadata":{}},{"cell_type":"code","source":"def darknet53_residual_block(inputs, filters, training, data_format,\n strides=1):\n shortcut = inputs\n\n inputs = conv2d_fixed_padding(\n inputs, filters=filters, kernel_size=1, strides=strides,\n data_format=data_format)\n inputs = batch_norm(inputs, training=training, data_format=data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n\n inputs = conv2d_fixed_padding(\n inputs, filters=2 * filters, kernel_size=3, strides=strides,\n data_format=data_format)\n inputs = batch_norm(inputs, training=training, data_format=data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n\n inputs += shortcut\n\n return inputs\n\n#Darknet with Res 53\ndef darknet53(inputs, training, data_format):\n \"\"\"Creates Darknet53 model for feature extraction.\"\"\"\n inputs = conv2d_fixed_padding(inputs, filters=32, kernel_size=3,\n data_format=data_format)\n inputs = batch_norm(inputs, training=training, data_format=data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n inputs = conv2d_fixed_padding(inputs, filters=64, kernel_size=3,\n strides=2, data_format=data_format)\n inputs = batch_norm(inputs, training=training, data_format=data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n\n inputs = darknet53_residual_block(inputs, filters=32, training=training,\n data_format=data_format)\n\n inputs = conv2d_fixed_padding(inputs, filters=128, kernel_size=3,\n strides=2, data_format=data_format)\n inputs = batch_norm(inputs, training=training, data_format=data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n\n for _ in range(2):\n inputs = darknet53_residual_block(inputs, filters=64,\n training=training,\n data_format=data_format)\n\n inputs = conv2d_fixed_padding(inputs, filters=256, kernel_size=3,\n strides=2, data_format=data_format)\n inputs = batch_norm(inputs, training=training, data_format=data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n\n for _ in range(8):\n inputs = darknet53_residual_block(inputs, filters=128,\n training=training,\n data_format=data_format)\n\n route1 = inputs\n\n inputs = conv2d_fixed_padding(inputs, filters=512, kernel_size=3,\n strides=2, data_format=data_format)\n inputs = batch_norm(inputs, training=training, data_format=data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n\n for _ in range(8):\n inputs = darknet53_residual_block(inputs, filters=256,\n training=training,\n data_format=data_format)\n\n route2 = inputs\n\n inputs = conv2d_fixed_padding(inputs, filters=1024, kernel_size=3,\n strides=2, data_format=data_format)\n inputs = batch_norm(inputs, training=training, data_format=data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n\n for _ in range(4):\n inputs = darknet53_residual_block(inputs, filters=512,\n training=training,\n data_format=data_format)\n\n return route1, route2, inputs","metadata":{"execution":{"iopub.status.busy":"2021-10-24T07:21:53.124100Z","iopub.execute_input":"2021-10-24T07:21:53.124382Z","iopub.status.idle":"2021-10-24T07:21:53.143728Z","shell.execute_reply.started":"2021-10-24T07:21:53.124349Z","shell.execute_reply":"2021-10-24T07:21:53.142814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Convolution layers\n\nYolo has a large number of convolutional layers. It's useful to group them in blocks.","metadata":{}},{"cell_type":"code","source":"#Yolo convolution blocks\ndef yolo_convolution_block(inputs, filters, training, data_format):\n \"\"\"Creates convolution operations layer used after Darknet.\"\"\"\n inputs = conv2d_fixed_padding(inputs, filters=filters, kernel_size=1,\n data_format=data_format)\n inputs = batch_norm(inputs, training=training, data_format=data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n\n inputs = conv2d_fixed_padding(inputs, filters=2 * filters, kernel_size=3,\n data_format=data_format)\n inputs = batch_norm(inputs, training=training, data_format=data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n\n inputs = conv2d_fixed_padding(inputs, filters=filters, kernel_size=1,\n data_format=data_format)\n inputs = batch_norm(inputs, training=training, data_format=data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n\n inputs = conv2d_fixed_padding(inputs, filters=2 * filters, kernel_size=3,\n data_format=data_format)\n inputs = batch_norm(inputs, training=training, data_format=data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n\n inputs = conv2d_fixed_padding(inputs, filters=filters, kernel_size=1,\n data_format=data_format)\n inputs = batch_norm(inputs, training=training, data_format=data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n\n route = inputs\n\n inputs = conv2d_fixed_padding(inputs, filters=2 * filters, kernel_size=3,\n data_format=data_format)\n inputs = batch_norm(inputs, training=training, data_format=data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n\n return route, inputs\n","metadata":{"execution":{"iopub.status.busy":"2021-10-24T07:21:53.315366Z","iopub.execute_input":"2021-10-24T07:21:53.315600Z","iopub.status.idle":"2021-10-24T07:21:53.327118Z","shell.execute_reply.started":"2021-10-24T07:21:53.315576Z","shell.execute_reply":"2021-10-24T07:21:53.326391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Detection layers\n\nYolo has 3 detection layers, that detect on 3 different scales using respective anchors. For each cell in the feature map the detection layer predicts n_anchors * (5 + n_classes) values using 1x1 convolution. For each scale we have n_anchors = 3. 5 + n_classes means that respectively to each of 3 anchors we are going to predict 4 coordinates of the box, its confidence score (the probability of containing an object) and class probabilities.\n\n\n### Upsample layer\n\nIn order to concatenate with shortcut outputs from Darknet-53 before applying detection on a different scale, we are going to upsample the feature map using nearest neighbor interpolation.\n\n### Non-max suppression\n\nThe model is going to produce a lot of boxes, so we need a way to discard the boxes with low confidence scores. Also, to avoid having multiple boxes for one object, we will discard the boxes with high overlap as well using non-max suppresion for each class.","metadata":{}},{"cell_type":"code","source":"#detection layer\ndef yolo_layer(inputs, n_classes, anchors, img_size, data_format):\n \"\"\"Creates Yolo final detection layer.\n \"\"\"\n n_anchors = len(anchors)\n\n inputs = tf.layers.conv2d(inputs, filters=n_anchors * (5 + n_classes),\n kernel_size=1, strides=1, use_bias=True,\n data_format=data_format)\n\n shape = inputs.get_shape().as_list()\n grid_shape = shape[2:4] if data_format == 'channels_first' else shape[1:3]\n if data_format == 'channels_first':\n inputs = tf.transpose(inputs, [0, 2, 3, 1])\n inputs = tf.reshape(inputs, [-1, n_anchors * grid_shape[0] * grid_shape[1],\n 5 + n_classes])\n\n strides = (img_size[0] // grid_shape[0], img_size[1] // grid_shape[1])\n\n box_centers, box_shapes, confidence, classes = \\\n tf.split(inputs, [2, 2, 1, n_classes], axis=-1)\n\n x = tf.range(grid_shape[0], dtype=tf.float32)\n y = tf.range(grid_shape[1], dtype=tf.float32)\n x_offset, y_offset = tf.meshgrid(x, y)\n x_offset = tf.reshape(x_offset, (-1, 1))\n y_offset = tf.reshape(y_offset, (-1, 1))\n x_y_offset = tf.concat([x_offset, y_offset], axis=-1)\n x_y_offset = tf.tile(x_y_offset, [1, n_anchors])\n x_y_offset = tf.reshape(x_y_offset, [1, -1, 2])\n box_centers = tf.nn.sigmoid(box_centers)\n box_centers = (box_centers + x_y_offset) * strides\n\n anchors = tf.tile(anchors, [grid_shape[0] * grid_shape[1], 1])\n box_shapes = tf.exp(box_shapes) * tf.to_float(anchors)\n\n confidence = tf.nn.sigmoid(confidence)\n\n classes = tf.nn.sigmoid(classes)\n\n inputs = tf.concat([box_centers, box_shapes,\n confidence, classes], axis=-1)\n\n return inputs\n#upsample\ndef upsample(inputs, out_shape, data_format):\n \"\"\"Upsamples to `out_shape` using nearest neighbor interpolation.\"\"\"\n if data_format == 'channels_first':\n inputs = tf.transpose(inputs, [0, 2, 3, 1])\n new_height = out_shape[3]\n new_width = out_shape[2]\n else:\n new_height = out_shape[2]\n new_width = out_shape[1]\n\n inputs = tf.image.resize_nearest_neighbor(inputs, (new_height, new_width))\n\n if data_format == 'channels_first':\n inputs = tf.transpose(inputs, [0, 3, 1, 2])\n\n return inputs\n\n# bounding box & non maximal suppression\n\ndef build_boxes(inputs):\n \"\"\"Computes top left and bottom right points of the boxes.\"\"\"\n center_x, center_y, width, height, confidence, classes = \\\n tf.split(inputs, [1, 1, 1, 1, 1, -1], axis=-1)\n\n top_left_x = center_x - width / 2\n top_left_y = center_y - height / 2\n bottom_right_x = center_x + width / 2\n bottom_right_y = center_y + height / 2\n\n boxes = tf.concat([top_left_x, top_left_y,\n bottom_right_x, bottom_right_y,\n confidence, classes], axis=-1)\n\n return boxes\n\n\ndef non_max_suppression(inputs, n_classes, max_output_size, iou_threshold,\n confidence_threshold):\n \"\"\"Performs non-max suppression separately for each class.\n \"\"\"\n batch = tf.unstack(inputs)\n boxes_dicts = []\n for boxes in batch:\n boxes = tf.boolean_mask(boxes, boxes[:, 4] > confidence_threshold)\n classes = tf.argmax(boxes[:, 5:], axis=-1)\n classes = tf.expand_dims(tf.to_float(classes), axis=-1)\n boxes = tf.concat([boxes[:, :5], classes], axis=-1)\n\n boxes_dict = dict()\n for cls in range(n_classes):\n mask = tf.equal(boxes[:, 5], cls)\n mask_shape = mask.get_shape()\n if mask_shape.ndims != 0:\n class_boxes = tf.boolean_mask(boxes, mask)\n boxes_coords, boxes_conf_scores, _ = tf.split(class_boxes,\n [4, 1, -1],\n axis=-1)\n boxes_conf_scores = tf.reshape(boxes_conf_scores, [-1])\n indices = tf.image.non_max_suppression(boxes_coords,\n boxes_conf_scores,\n max_output_size,\n iou_threshold)\n class_boxes = tf.gather(class_boxes, indices)\n boxes_dict[cls] = class_boxes[:, :5]\n\n boxes_dicts.append(boxes_dict)\n\n return boxes_dicts","metadata":{"execution":{"iopub.status.busy":"2021-10-24T07:21:55.092116Z","iopub.execute_input":"2021-10-24T07:21:55.092370Z","iopub.status.idle":"2021-10-24T07:21:55.116185Z","shell.execute_reply.started":"2021-10-24T07:21:55.092343Z","shell.execute_reply":"2021-10-24T07:21:55.115364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"class Yolo_v3:\n \"\"\"Yolo v3 model class.\"\"\"\n\n def __init__(self, n_classes, model_size, max_output_size, iou_threshold,\n confidence_threshold, data_format=None):\n \"\"\"Creates the model.\n\n Args:\n n_classes: Number of class labels.\n model_size: The input size of the model.\n max_output_size: Max number of boxes to be selected for each class.\n iou_threshold: Threshold for the IOU.\n confidence_threshold: Threshold for the confidence score.\n data_format: The input format.\n\n Returns:\n None.\n \"\"\"\n if not data_format:\n if tf.test.is_built_with_cuda():\n data_format = 'channels_first'\n else:\n data_format = 'channels_last'\n\n self.n_classes = n_classes\n self.model_size = model_size\n self.max_output_size = max_output_size\n self.iou_threshold = iou_threshold\n self.confidence_threshold = confidence_threshold\n self.data_format = data_format\n\n def __call__(self, inputs, training):\n \"\"\"Add operations to detect boxes for a batch of input images.\n\n Args:\n inputs: A Tensor representing a batch of input images.\n training: A boolean, whether to use in training or inference mode.\n\n Returns:\n A list containing class-to-boxes dictionaries\n for each sample in the batch.\n \"\"\"\n with tf.variable_scope('yolo_v3_model'):\n if self.data_format == 'channels_first':\n inputs = tf.transpose(inputs, [0, 3, 1, 2])\n\n inputs = inputs / 255\n\n route1, route2, inputs = darknet53(inputs, training=training,\n data_format=self.data_format)\n\n route, inputs = yolo_convolution_block(\n inputs, filters=512, training=training,\n data_format=self.data_format)\n detect1 = yolo_layer(inputs, n_classes=self.n_classes,\n anchors=_ANCHORS[6:9],\n img_size=self.model_size,\n data_format=self.data_format)\n\n inputs = conv2d_fixed_padding(route, filters=256, kernel_size=1,\n data_format=self.data_format)\n inputs = batch_norm(inputs, training=training,\n data_format=self.data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n upsample_size = route2.get_shape().as_list()\n inputs = upsample(inputs, out_shape=upsample_size,\n data_format=self.data_format)\n axis = 1 if self.data_format == 'channels_first' else 3\n inputs = tf.concat([inputs, route2], axis=axis)\n route, inputs = yolo_convolution_block(\n inputs, filters=256, training=training,\n data_format=self.data_format)\n detect2 = yolo_layer(inputs, n_classes=self.n_classes,\n anchors=_ANCHORS[3:6],\n img_size=self.model_size,\n data_format=self.data_format)\n\n inputs = conv2d_fixed_padding(route, filters=128, kernel_size=1,\n data_format=self.data_format)\n inputs = batch_norm(inputs, training=training,\n data_format=self.data_format)\n inputs = tf.nn.leaky_relu(inputs, alpha=_LEAKY_RELU)\n upsample_size = route1.get_shape().as_list()\n inputs = upsample(inputs, out_shape=upsample_size,\n data_format=self.data_format)\n inputs = tf.concat([inputs, route1], axis=axis)\n route, inputs = yolo_convolution_block(\n inputs, filters=128, training=training,\n data_format=self.data_format)\n detect3 = yolo_layer(inputs, n_classes=self.n_classes,\n anchors=_ANCHORS[0:3],\n img_size=self.model_size,\n data_format=self.data_format)\n\n inputs = tf.concat([detect1, detect2, detect3], axis=1)\n\n inputs = build_boxes(inputs)\n\n boxes_dicts = non_max_suppression(\n inputs, n_classes=self.n_classes,\n max_output_size=self.max_output_size,\n iou_threshold=self.iou_threshold,\n confidence_threshold=self.confidence_threshold)\n\n return boxes_dicts\n\n","metadata":{"execution":{"iopub.status.busy":"2021-10-24T07:21:57.914769Z","iopub.execute_input":"2021-10-24T07:21:57.915179Z","iopub.status.idle":"2021-10-24T07:21:57.933624Z","shell.execute_reply.started":"2021-10-24T07:21:57.915146Z","shell.execute_reply":"2021-10-24T07:21:57.932899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Utility functions\n\nHere are some utility functions that will help us load images as NumPy arrays, load class names from the official file and draw the predicted boxes.\n\n\n### Converting weights to Tensorflow format\n\nNow it's time to load the official weights. We are going to iterate through the file and gradually create tf.assign operations.","metadata":{}},{"cell_type":"code","source":"def load_images(img_names, model_size):\n \"\"\"Loads images in a 4D array.\n\n Args:\n img_names: A list of images names.\n model_size: The input size of the model.\n data_format: A format for the array returned\n ('channels_first' or 'channels_last').\n\n Returns:\n A 4D NumPy array.\n \"\"\"\n imgs = []\n\n for img_name in img_names:\n img = Image.open(img_name)\n img = img.resize(size=model_size)\n img = np.array(img, dtype=np.float32)\n img = np.expand_dims(img, axis=0)\n imgs.append(img)\n\n imgs = np.concatenate(imgs)\n\n return imgs\n\n\ndef load_class_names(file_name):\n \"\"\"Returns a list of class names read from `file_name`.\"\"\"\n with open(file_name, 'r') as f:\n class_names = f.read().splitlines()\n return class_names\n\n\ndef draw_boxes(img_names, boxes_dicts, class_names, model_size):\n \"\"\"Draws detected boxes.\"\"\"\n colors = ((np.array(color_palette(\"hls\", 80)) * 255)).astype(np.uint8)\n for num, img_name, boxes_dict in zip(range(len(img_names)), img_names,\n boxes_dicts):\n img = Image.open(img_name)\n draw = ImageDraw.Draw(img)\n font = ImageFont.truetype(font='../input/data-for-yolo-v3-kernel/futur.ttf',\n size=(img.size[0] + img.size[1]) // 100)\n resize_factor = \\\n (img.size[0] / model_size[0], img.size[1] / model_size[1])\n for cls in range(len(class_names)):\n boxes = boxes_dict[cls]\n if np.size(boxes) != 0:\n color = colors[cls]\n for box in boxes:\n xy, confidence = box[:4], box[4]\n xy = [xy[i] * resize_factor[i % 2] for i in range(4)]\n x0, y0 = xy[0], xy[1]\n thickness = (img.size[0] + img.size[1]) // 200\n for t in np.linspace(0, 1, thickness):\n xy[0], xy[1] = xy[0] + t, xy[1] + t\n xy[2], xy[3] = xy[2] - t, xy[3] - t\n draw.rectangle(xy, outline=tuple(color))\n text = '{} {:.1f}%'.format(class_names[cls],\n confidence * 100)\n text_size = draw.textsize(text, font=font)\n draw.rectangle(\n [x0, y0 - text_size[1], x0 + text_size[0], y0],\n fill=tuple(color))\n draw.text((x0, y0 - text_size[1]), text, fill='black',\n font=font)\n\n display(img)\n \n \n \ndef load_weights(variables, file_name):\n \"\"\"Reshapes and loads official pretrained Yolo weights.\n\n Args:\n variables: A list of tf.Variable to be assigned.\n file_name: A name of a file containing weights.\n\n Returns:\n A list of assign operations.\n \"\"\"\n with open(file_name, \"rb\") as f:\n # Skip first 5 values containing irrelevant info\n np.fromfile(f, dtype=np.int32, count=5)\n weights = np.fromfile(f, dtype=np.float32)\n\n assign_ops = []\n ptr = 0\n\n # Load weights for Darknet part.\n # Each convolution layer has batch normalization.\n for i in range(52):\n conv_var = variables[5 * i]\n gamma, beta, mean, variance = variables[5 * i + 1:5 * i + 5]\n batch_norm_vars = [beta, gamma, mean, variance]\n\n for var in batch_norm_vars:\n shape = var.shape.as_list()\n num_params = np.prod(shape)\n var_weights = weights[ptr:ptr + num_params].reshape(shape)\n ptr += num_params\n assign_ops.append(tf.assign(var, var_weights))\n\n shape = conv_var.shape.as_list()\n num_params = np.prod(shape)\n var_weights = weights[ptr:ptr + num_params].reshape(\n (shape[3], shape[2], shape[0], shape[1]))\n var_weights = np.transpose(var_weights, (2, 3, 1, 0))\n ptr += num_params\n assign_ops.append(tf.assign(conv_var, var_weights))\n\n # Loading weights for Yolo part.\n # 7th, 15th and 23rd convolution layer has biases and no batch norm.\n ranges = [range(0, 6), range(6, 13), range(13, 20)]\n unnormalized = [6, 13, 20]\n for j in range(3):\n for i in ranges[j]:\n current = 52 * 5 + 5 * i + j * 2\n conv_var = variables[current]\n gamma, beta, mean, variance = \\\n variables[current + 1:current + 5]\n batch_norm_vars = [beta, gamma, mean, variance]\n\n for var in batch_norm_vars:\n shape = var.shape.as_list()\n num_params = np.prod(shape)\n var_weights = weights[ptr:ptr + num_params].reshape(shape)\n ptr += num_params\n assign_ops.append(tf.assign(var, var_weights))\n\n shape = conv_var.shape.as_list()\n num_params = np.prod(shape)\n var_weights = weights[ptr:ptr + num_params].reshape(\n (shape[3], shape[2], shape[0], shape[1]))\n var_weights = np.transpose(var_weights, (2, 3, 1, 0))\n ptr += num_params\n assign_ops.append(tf.assign(conv_var, var_weights))\n\n bias = variables[52 * 5 + unnormalized[j] * 5 + j * 2 + 1]\n shape = bias.shape.as_list()\n num_params = np.prod(shape)\n var_weights = weights[ptr:ptr + num_params].reshape(shape)\n ptr += num_params\n assign_ops.append(tf.assign(bias, var_weights))\n\n conv_var = variables[52 * 5 + unnormalized[j] * 5 + j * 2]\n shape = conv_var.shape.as_list()\n num_params = np.prod(shape)\n var_weights = weights[ptr:ptr + num_params].reshape(\n (shape[3], shape[2], shape[0], shape[1]))\n var_weights = np.transpose(var_weights, (2, 3, 1, 0))\n ptr += num_params\n assign_ops.append(tf.assign(conv_var, var_weights))\n\n return assign_ops","metadata":{"execution":{"iopub.status.busy":"2021-10-24T07:22:00.600481Z","iopub.execute_input":"2021-10-24T07:22:00.600770Z","iopub.status.idle":"2021-10-24T07:22:00.633961Z","shell.execute_reply.started":"2021-10-24T07:22:00.600741Z","shell.execute_reply":"2021-10-24T07:22:00.633078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_names = ['../input/data-for-yolo-v3-kernel/dog.jpg', '../input/data-for-yolo-v3-kernel/office.jpg']\nfor img in img_names: display(Image.open(img))\n\nbatch_size = len(img_names)\nbatch = load_images(img_names, model_size=_MODEL_SIZE)\nclass_names = load_class_names('../input/data-for-yolo-v3-kernel/coco.names')\nn_classes = len(class_names)\nmax_output_size = 10\niou_threshold = 0.5\nconfidence_threshold = 0.5\n\nmodel = Yolo_v3(n_classes=n_classes, model_size=_MODEL_SIZE,\n max_output_size=max_output_size,\n iou_threshold=iou_threshold,\n confidence_threshold=confidence_threshold)\n\ninputs = tf.placeholder(tf.float32, [batch_size, 416, 416, 3])\n\ndetections = model(inputs, training=False)\n\nmodel_vars = tf.global_variables(scope='yolo_v3_model')\nassign_ops = load_weights(model_vars, '../input/data-for-yolo-v3-kernel/yolov3.weights')\n\nwith tf.Session() as sess:\n sess.run(assign_ops)\n detection_result = sess.run(detections, feed_dict={inputs: batch})\n \ndraw_boxes(img_names, detection_result, class_names, _MODEL_SIZE)","metadata":{"execution":{"iopub.status.busy":"2021-10-24T07:22:03.749434Z","iopub.execute_input":"2021-10-24T07:22:03.749956Z","iopub.status.idle":"2021-10-24T07:22:37.588573Z","shell.execute_reply.started":"2021-10-24T07:22:03.749921Z","shell.execute_reply":"2021-10-24T07:22:37.585385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Making a Prediction\n\nThe convolutional layers included in the YOLOv3 architecture produce a detection prediction after passing the features learned onto a classifier or regressor. These features include the class label, coordinates of the bounding boxes, sizes of the bounding boxes, and more.\n\nSince the prediction with YOLO uses 1 x 1 convolutions (hence the name, “you only look once”), the size of the prediction map is exactly the size of the feature map before it.\n\nIn YOLOv3 and its other versions, the way this prediction map is interpreted is that each cell predicts a fixed number of bounding boxes. Then, whichever cell contains the center of the ground truth box of an object of interest is designated as the cell that will be finally responsible for predicting the object. There is a ton of mathematics behind the inner workings of the prediction architecture.\n\n### Anchor Boxes\n\nAlthough anchor boxes, or bounding boxes, were discussed a little bit at the beginning of this article, there is a bit more detail about implementing them and using them with YOLOv3. Object detectors using YOLOv3 usually predict log-space transforms, which are offsets to predefined “default” bounding boxes. Those specific bounding boxes are called anchors. The transforms are later applied to the anchor boxes to receive a prediction.YOLOv3 in particular has three anchors. This results in the prediction of three bounding boxes per cell (the cell is also called a neuron in more technical terms).\n\n### Non-Maximum Suppression\n\nObjects can sometimes be detected multiple times when more than one bounding box detects the object as a positive class detection. Non-maximum suppression helps avoid this situation and only passes detections if they haven’t already been detected. Using the NMS threshold value and confidence threshold value, NMS is implemented to prevent double detections. It is an imperative part of using YOLOv3 effectively. Here, we briefly described a few of the features that make the predictions possible, such as anchor boxes and non-maximum suppression (NMS) values. This is, however, not a complete representation of all the features that go into creating a successful prediction with YOLOv3. For full descriptions of YOLOv3’s mathematical background, I suggest reading the official YOLOv3 paper linked at the end of this article.\n\n### Interpreting Results\n\nInterpreting the results of a YOLO model prediction is just as nuanced as the actual implementation of the model. Multiple factors go into a successful interpretation and accuracy rating, such as the box confidence score and class confidence score used when creating a YOLOv3 computer vision model.\n\nThere are many other ways and features used when interpreting results, but these are just a few. Other YOLOv3 prediction features include the classification loss, loss function, objectness score, and more.\n\n### Class Confidence and Box Confidence Scores\n\nEach bounding box has an x, y, w, h, and box confidence score value. The confidence score is the value of how probable a class is contained by that box, as well as how accurate that bounding box is.\n\nThe bounding box width and height (w and h) is first set to the width and height of the image given. Then, x and y are offsets of the cell in question and all 4 bounding box values are between 0 and 1. Then, each cell has 20 conditional class probabilities implemented by the YOLOv3 algorithm.\n\nThe class confidence score for each final boundary box used as a positive prediction is equal to the box confidence score multiplied by the conditional class probability. The conditional class probability in this context is the probability that the detected object is part of a certain class (the class being the object of interest’s identification). YOLOv3’s prediction, therefore, has 3 values of h, w, and depth.\n\nThere is some math that then takes place involving the spatial dimensions of the images and the tensors used in order to produce boundary box predictions, but that is complicated. [Yolo](https://arxiv.org/pdf/1506.02640.pdf)\n\nFor the final step, the boundary boxes with high confidence scores (more than 0.25) are kept as final predictions.\n\n\n\n[Resource](https://www.atlantis-press.com/journals/ijcis/125943382/view)","metadata":{}},{"cell_type":"code","source":"class Transformer(tf.keras.Model):\n \"\"\"Transformer model with Keras.\n Implemented as described in: https://arxiv.org/pdf/1706.03762.pdf\n The Transformer model consists of an encoder and decoder. The input is an int\n sequence (or a batch of sequences). The encoder produces a continuous\n representation, and the decoder uses the encoder output to generate\n probabilities for the output sequence.\n \"\"\"\n\n def __init__(self, params, name=None):\n \"\"\"Initialize layers to build Transformer model.\n Args:\n params: hyperparameter object defining layer sizes, dropout values, etc.\n name: name of the model.\n \"\"\"\n super(Transformer, self).__init__(name=name)\n self.params = params\n self.embedding_softmax_layer = embedding_layer.EmbeddingSharedWeights(\n params[\"vocab_size\"], params[\"hidden_size\"])\n self.encoder_stack = EncoderStack(params)\n self.decoder_stack = DecoderStack(params)\n self.position_embedding = position_embedding.RelativePositionEmbedding(\n hidden_size=self.params[\"hidden_size\"])\n\n def get_config(self):\n return {\n \"params\": self.params,\n }\n\n def call(self, inputs, training):\n \"\"\"Calculate target logits or inferred target sequences.\n Args:\n inputs: input tensor list of size 1 or 2.\n First item, inputs: int tensor with shape [batch_size, input_length].\n Second item (optional), targets: None or int tensor with shape\n [batch_size, target_length].\n training: boolean, whether in training mode or not.\n Returns:\n If targets is defined, then return logits for each word in the target\n sequence. float tensor with shape [batch_size, target_length, vocab_size]\n If target is none, then generate output sequence one token at a time.\n returns a dictionary {\n outputs: int tensor with shape [batch_size, decoded_length]\n scores: float tensor with shape [batch_size]}\n Even when float16 is used, the output tensor(s) are always float32.\n Raises:\n NotImplementedError: If try to use padded decode method on CPU/GPUs.\n \"\"\"\n inputs = inputs if isinstance(inputs, list) else [inputs]\n if len(inputs) == 2:\n inputs, targets = inputs[0], inputs[1]\n else:\n # Decoding path.\n inputs, targets = inputs[0], None\n if self.params[\"padded_decode\"]:\n if not self.params[\"num_replicas\"]:\n raise NotImplementedError(\n \"Padded decoding on CPU/GPUs is not supported.\")\n decode_batch_size = int(self.params[\"decode_batch_size\"] /\n self.params[\"num_replicas\"])\n inputs.set_shape([decode_batch_size, self.params[\"decode_max_length\"]])\n\n # Variance scaling is used here because it seems to work in many problems.\n # Other reasonable initializers may also work just as well.\n with tf.name_scope(\"Transformer\"):\n # Calculate attention bias for encoder self-attention and decoder\n # multi-headed attention layers.\n attention_bias = model_utils.get_padding_bias(inputs)\n\n # Run the inputs through the encoder layer to map the symbol\n # representations to continuous representations.\n encoder_outputs = self.encode(inputs, attention_bias, training)\n # Generate output sequence if targets is None, or return logits if target\n # sequence is known.\n if targets is None:\n return self.predict(encoder_outputs, attention_bias, training)\n else:\n logits = self.decode(targets, encoder_outputs, attention_bias, training)\n return logits\n\n def encode(self, inputs, attention_bias, training):\n \"\"\"Generate continuous representation for inputs.\n Args:\n inputs: int tensor with shape [batch_size, input_length].\n attention_bias: float tensor with shape [batch_size, 1, 1, input_length].\n training: boolean, whether in training mode or not.\n Returns:\n float tensor with shape [batch_size, input_length, hidden_size]\n \"\"\"\n with tf.name_scope(\"encode\"):\n # Prepare inputs to the layer stack by adding positional encodings and\n # applying dropout.\n embedded_inputs = self.embedding_softmax_layer(inputs)\n embedded_inputs = tf.cast(embedded_inputs, self.params[\"dtype\"])\n inputs_padding = model_utils.get_padding(inputs)\n attention_bias = tf.cast(attention_bias, self.params[\"dtype\"])\n\n with tf.name_scope(\"add_pos_encoding\"):\n pos_encoding = self.position_embedding(inputs=embedded_inputs)\n pos_encoding = tf.cast(pos_encoding, self.params[\"dtype\"])\n encoder_inputs = embedded_inputs + pos_encoding\n\n if training:\n encoder_inputs = tf.nn.dropout(\n encoder_inputs, rate=self.params[\"layer_postprocess_dropout\"])\n\n return self.encoder_stack(\n encoder_inputs, attention_bias, inputs_padding, training=training)\n\n def decode(self, targets, encoder_outputs, attention_bias, training):\n \"\"\"Generate logits for each value in the target sequence.\n Args:\n targets: target values for the output sequence. int tensor with shape\n [batch_size, target_length]\n encoder_outputs: continuous representation of input sequence. float tensor\n with shape [batch_size, input_length, hidden_size]\n attention_bias: float tensor with shape [batch_size, 1, 1, input_length]\n training: boolean, whether in training mode or not.\n Returns:\n float32 tensor with shape [batch_size, target_length, vocab_size]\n \"\"\"\n with tf.name_scope(\"decode\"):\n # Prepare inputs to decoder layers by shifting targets, adding positional\n # encoding and applying dropout.\n decoder_inputs = self.embedding_softmax_layer(targets)\n decoder_inputs = tf.cast(decoder_inputs, self.params[\"dtype\"])\n attention_bias = tf.cast(attention_bias, self.params[\"dtype\"])\n with tf.name_scope(\"shift_targets\"):\n # Shift targets to the right, and remove the last element\n decoder_inputs = tf.pad(decoder_inputs,\n [[0, 0], [1, 0], [0, 0]])[:, :-1, :]\n with tf.name_scope(\"add_pos_encoding\"):\n length = tf.shape(decoder_inputs)[1]\n pos_encoding = self.position_embedding(decoder_inputs)\n pos_encoding = tf.cast(pos_encoding, self.params[\"dtype\"])\n decoder_inputs += pos_encoding\n if training:\n decoder_inputs = tf.nn.dropout(\n decoder_inputs, rate=self.params[\"layer_postprocess_dropout\"])\n\n # Run values\n decoder_self_attention_bias = model_utils.get_decoder_self_attention_bias(\n length, dtype=self.params[\"dtype\"])\n outputs = self.decoder_stack(\n decoder_inputs,\n encoder_outputs,\n decoder_self_attention_bias,\n attention_bias,\n training=training)\n logits = self.embedding_softmax_layer(outputs, mode=\"linear\")\n logits = tf.cast(logits, tf.float32)\n return logits\n\n def _get_symbols_to_logits_fn(self, max_decode_length, training):\n \"\"\"Returns a decoding function that calculates logits of the next tokens.\"\"\"\n timing_signal = self.position_embedding(\n inputs=None, length=max_decode_length + 1)\n timing_signal = tf.cast(timing_signal, self.params[\"dtype\"])\n decoder_self_attention_bias = model_utils.get_decoder_self_attention_bias(\n max_decode_length, dtype=self.params[\"dtype\"])\n\n def symbols_to_logits_fn(ids, i, cache):\n \"\"\"Generate logits for next potential IDs.\n Args:\n ids: Current decoded sequences. int tensor with shape [batch_size *\n beam_size, i + 1].\n i: Loop index.\n cache: dictionary of values storing the encoder output, encoder-decoder\n attention bias, and previous decoder attention values.\n Returns:\n Tuple of\n (logits with shape [batch_size * beam_size, vocab_size],\n updated cache values)\n \"\"\"\n # Set decoder input to the last generated IDs\n decoder_input = ids[:, -1:]\n\n # Preprocess decoder input by getting embeddings and adding timing signal.\n decoder_input = self.embedding_softmax_layer(decoder_input)\n decoder_input += timing_signal[i]\n if self.params[\"padded_decode\"]:\n bias_shape = decoder_self_attention_bias.shape.as_list()\n self_attention_bias = tf.slice(\n decoder_self_attention_bias, [0, 0, i, 0],\n [bias_shape[0], bias_shape[1], 1, bias_shape[3]])\n else:\n self_attention_bias = decoder_self_attention_bias[:, :, i:i + 1, :i + 1]\n\n decoder_outputs = self.decoder_stack(\n decoder_input,\n cache.get(\"encoder_outputs\"),\n self_attention_bias,\n cache.get(\"encoder_decoder_attention_bias\"),\n training=training,\n cache=cache,\n decode_loop_step=i if self.params[\"padded_decode\"] else None)\n logits = self.embedding_softmax_layer(decoder_outputs, mode=\"linear\")\n logits = tf.squeeze(logits, axis=[1])\n return logits, cache\n\n return symbols_to_logits_fn\n\n def predict(self, encoder_outputs, encoder_decoder_attention_bias, training):\n \"\"\"Return predicted sequence.\"\"\"\n encoder_outputs = tf.cast(encoder_outputs, self.params[\"dtype\"])\n if self.params[\"padded_decode\"]:\n batch_size = encoder_outputs.shape.as_list()[0]\n input_length = encoder_outputs.shape.as_list()[1]\n else:\n batch_size = tf.shape(encoder_outputs)[0]\n input_length = tf.shape(encoder_outputs)[1]\n max_decode_length = input_length + self.params[\"extra_decode_length\"]\n encoder_decoder_attention_bias = tf.cast(encoder_decoder_attention_bias,\n self.params[\"dtype\"])\n\n symbols_to_logits_fn = self._get_symbols_to_logits_fn(\n max_decode_length, training)\n\n # Create initial set of IDs that will be passed into symbols_to_logits_fn.\n initial_ids = tf.zeros([batch_size], dtype=tf.int32)\n\n # Create cache storing decoder attention values for each layer.\n # pylint: disable=g-complex-comprehension\n init_decode_length = (\n max_decode_length if self.params[\"padded_decode\"] else 0)\n num_heads = self.params[\"num_heads\"]\n dim_per_head = self.params[\"hidden_size\"] // num_heads\n cache = {\n \"layer_%d\" % layer: {\n \"k\":\n tf.zeros(\n [batch_size, init_decode_length, num_heads, dim_per_head],\n dtype=self.params[\"dtype\"]),\n \"v\":\n tf.zeros(\n [batch_size, init_decode_length, num_heads, dim_per_head],\n dtype=self.params[\"dtype\"])\n } for layer in range(self.params[\"num_hidden_layers\"])\n }\n # pylint: enable=g-complex-comprehension\n\n # Add encoder output and attention bias to the cache.\n cache[\"encoder_outputs\"] = encoder_outputs\n cache[\"encoder_decoder_attention_bias\"] = encoder_decoder_attention_bias\n\n # Use beam search to find the top beam_size sequences and scores.\n decoded_ids, scores = beam_search.sequence_beam_search(\n symbols_to_logits_fn=symbols_to_logits_fn,\n initial_ids=initial_ids,\n initial_cache=cache,\n vocab_size=self.params[\"vocab_size\"],\n beam_size=self.params[\"beam_size\"],\n alpha=self.params[\"alpha\"],\n max_decode_length=max_decode_length,\n eos_id=EOS_ID,\n padded_decode=self.params[\"padded_decode\"],\n dtype=self.params[\"dtype\"])\n\n # Get the top sequence for each batch element\n top_decoded_ids = decoded_ids[:, 0, 1:]\n top_scores = scores[:, 0]\n\n return {\"outputs\": top_decoded_ids, \"scores\": top_scores}\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## DEtr\n\n\n\n### Introduction\n\nDETR treats an object detection problem as a direct set prediction problem with the help of an encoder-decoder architecture based on transformers. By set, I mean the set of bounding boxes. Transformers are the new breed of deep learning models that have performed outstandingly in the NLP domain. This is the first time when someone used transformers for object detection.\nThe authors of this paper have evaluated DETR on one of the most popular object detection datasets, COCO, against a very competitive Faster R-CNN baseline.\nIn the results, the DETR achieved comparable performances. More precisely, DETR demonstrates significantly better performance on large objects. However, it didn’t perform that well on small objects.\n\n### The DETR Model\n\nMost modern object detection methods make predictions relative to some initial guesses. Two-stage detectors(R-CNN family) predict boxes w.r.t. proposals, whereas single-stage methods(YOLO) make predictions w.r.t. anchors or a grid of possible object centers. Recent work demonstrate that the final performance of these systems heavily depends on the exact way these initial guesses are set. In our model(DETR) we are able to remove this hand-crafted process and streamline the detection process by directly predicting the set of detections with absolute box prediction w.r.t. the input image rather than an anchor.\nTwo things are essential for direct set predictions in detection:\n\n- A set prediction loss that forces unique matching between predicted and ground truth boxes\n- An architecture that predicts (in a single pass) a set of objects and models their relation\n\nThe researcher at Facebook AI used bipartite matching between predicted and ground truth objects which ensures one-to-one mapping between predicted and ground truth objects/bounding boxes.\n\nDETR infers a fixed-size set of N predictions, in a single pass through the decoder, where N is set to be significantly larger than the typical number of objects in an image. This N user has to decide according to their need. Suppose in an image maximum 5 object are there so we can define (N=7,8,..). let’s say N=7, so DETR infers a set of 7 prediction. Out of this 7 prediction 5 prediction will for object and 2 prediction are for ∅(no object) means they will assign to background. Each prediction is a kind of tuple containing class and bounding box (c,b).\n\n### The DETR Architecture\n\nThe overall DETR architecture is easy to understand. It contains three main components:\n- CNN Backbone\n- Encoder-Decoder transformer\n- Simple feed-forward network\n\n\n\n\n\nDETR uses a conventional CNN backbone to learn a 2D representation of an input image. The model flattens it and supplements it with a positional encoding before passing it into a transformer encoder. A transformer decoder then takes as input a small fixed number(N) of learned positional embeddings, which we call object queries, and additionally attends to the encoder output. We pass each output embedding of the decoder to a shared feed forward network (FFN) that predicts either a detection (class and bounding box) or a ∅(no object) class.\nThe decoder follows the standard architecture of the transformer, transforming N embeddings of size d using multi-headed self- and encoder-decoder attention mechanisms. The difference with the original transformer is that our model decodes the N objects in parallel at each decoder layer while original transformer use an autoregressive model that predicts the output sequence one element at a time.\nThe N object queries are transformed into an output embedding by the decoder. They are then independently decoded into box coordinates and class labels by a feed forward network, resulting N final predictions. Using self- and encoder-decoder attention over these embeddings, the model globally reasons about all objects together using pair-wise relations between them, while being able to use the whole image as context.\nThe final prediction is computed by a 3-layer perceptron with ReLU activation function and hidden dimension d, and a linear projection layer. The FFN predicts the normalized center coordinates, height and width of the box w.r.t. the input image, and the linear layer predicts the class label using a softmax function. Since we predict a fixed-size set of N bounding boxes, where N is usually larger than the actual number of objects of interest in an image, an additional special class label ∅ is used to represent that no object is detected within a slot. This class plays a similar role to the “background” class in the standard object detection approaches.\n\n\n\n\nIn above image we can see that our DETR infer 6 prediction but out of 6 prediction only 2 prediction are assigned to object rest of the prediction are belonging to ∅(no object) class.\n\n- [Paper](https://arxiv.org/abs/2005.12872)\n- [Resource](https://github.com/facebookresearch/detr)\n\n","metadata":{}},{"cell_type":"code","source":"\n#sample tf implementation ... has errors...will correct\nimport tensorflow as tf\n\n# tf.enable_eager_execution()\n# tf.compat.v1.enable_eager_execution()\nfrom tensorflow.keras import models\nfrom tensorflow.keras.activations import softmax\nfrom tensorflow.keras.utils import get_custom_objects\nfrom tensorflow.keras.layers import (\n Activation,\n Add,\n AveragePooling2D,\n BatchNormalization,\n Conv2D,\n Dense,\n Dropout,\n GlobalAveragePooling2D,\n Input,\n MaxPool2D,\n UpSampling2D,\n ZeroPadding2D,\n MaxPooling2D,\n Conv1D,\n)\nfrom tensorflow.keras.models import Transformer\n\n\ndef get_flops(model):\n run_meta = tf.compat.v1.RunMetadata()\n opts = tf.compat.v1.profiler.ProfileOptionBuilder.float_operation()\n\n # We use the Keras session graph in the call to the profiler.\n flops = tf.compat.v1.profiler.profile(\n graph=tf.compat.v1.keras.backend.get_session().graph, run_meta=run_meta, cmd=\"op\", options=opts\n )\n\n return flops.total_float_ops # Prints the \"flops\" of the model.\n\n\nclass Mish(Activation):\n \"\"\"\n based on https://github.com/digantamisra98/Mish/blob/master/Mish/TFKeras/mish.py\n Mish Activation Function.\n \"\"\"\n\n def __init__(self, activation, **kwargs):\n super(Mish, self).__init__(activation, **kwargs)\n self.__name__ = \"Mish\"\n\n\ndef mish(inputs):\n # with tf.device(\"CPU:0\"):\n result = inputs * tf.math.tanh(tf.math.softplus(inputs))\n return result\n\n\nclass GroupedConv2D(object):\n \"\"\"Groupped convolution.\n https://github.com/tensorflow/tpu/blob/master/models/official/mnasnet/mixnet/custom_py\n Currently tf.keras and tf.layers don't support group convolution, so here we\n use split/concat to implement this op. It reuses kernel_size for group\n definition, where len(kernel_size) is number of groups. Notably, it allows\n different group has different kernel size.\n \"\"\"\n\n def __init__(self, filters, kernel_size, use_keras=True, **kwargs):\n \"\"\"Initialize the layer.\n Args:\n filters: Integer, the dimensionality of the output space.\n kernel_size: An integer or a list. If it is a single integer, then it is\n same as the original Conv2D. If it is a list, then we split the channels\n and perform different kernel for each group.\n use_keras: An boolean value, whether to use keras layer.\n **kwargs: other parameters passed to the original conv2d layer.\n \"\"\"\n self._groups = len(kernel_size)\n self._channel_axis = -1\n\n self._convs = []\n splits = self._split_channels(filters, self._groups)\n for i in range(self._groups):\n self._convs.append(self._get_conv2d(splits[i], kernel_size[i], use_keras, **kwargs))\n\n def _get_conv2d(self, filters, kernel_size, use_keras, **kwargs):\n \"\"\"A helper function to create Conv2D layer.\"\"\"\n if use_keras:\n return Conv2D(filters=filters, kernel_size=kernel_size, **kwargs)\n else:\n return Conv2D(filters=filters, kernel_size=kernel_size, **kwargs)\n\n def _split_channels(self, total_filters, num_groups):\n split = [total_filters // num_groups for _ in range(num_groups)]\n split[0] += total_filters - sum(split)\n return split\n\n def __call__(self, inputs):\n if len(self._convs) == 1:\n return self._convs[0](inputs)\n\n if tf.__version__ < \"2.0.0\":\n filters = inputs.shape[self._channel_axis].value\n else:\n filters = inputs.shape[self._channel_axis]\n splits = self._split_channels(filters, len(self._convs))\n x_splits = tf.split(inputs, splits, self._channel_axis)\n x_outputs = [c(x) for x, c in zip(x_splits, self._convs)]\n x = tf.concat(x_outputs, self._channel_axis)\n return x\n\n\nclass DETR:\n def __init__(self, verbose=False, input_shape=(224, 224, 3), active=\"relu\", n_classes=81,\n dropout_rate=0.2, fc_activation=None, blocks_set=[3, 4, 6, 3], radix=2, groups=1,\n bottleneck_width=64, deep_stem=True, stem_width=32, block_expansion=4, avg_down=True,\n avd=True, avd_first=False, preact=False, using_basic_block=False,using_cb=False,\n using_transformer=True,hidden_dim=512,nheads=8,num_encoder_layers=6,\n num_decoder_layers=6,n_query_pos=100):\n self.channel_axis = -1 # not for change\n self.verbose = verbose\n self.active = active # default relu\n self.input_shape = input_shape\n self.n_classes = n_classes\n self.dropout_rate = dropout_rate\n self.fc_activation = fc_activation\n\n #resnest set\n self.blocks_set = blocks_set\n self.radix = radix\n self.cardinality = groups\n self.bottleneck_width = bottleneck_width\n\n self.deep_stem = deep_stem\n self.stem_width = stem_width\n self.block_expansion = block_expansion\n self.avg_down = avg_down\n self.avd = avd\n self.avd_first = avd_first\n\n # self.cardinality = 1\n self.dilation = 1\n self.preact = preact\n self.using_basic_block = using_basic_block\n self.using_cb = using_cb\n\n #DETR set\n # self.training = training\n self.using_transformer = using_transformer\n self.hidden_dim = hidden_dim\n self.nheads = nheads\n self.num_encoder_layers = num_encoder_layers\n self.num_decoder_layers = num_decoder_layers\n self.n_query_pos = n_query_pos\n\n def _make_stem(self, input_tensor, stem_width=64, deep_stem=False):\n x = input_tensor\n if deep_stem:\n x = Conv2D(stem_width, kernel_size=3, strides=2, padding=\"same\", kernel_initializer=\"he_normal\",\n use_bias=False, data_format=\"channels_last\")(x)\n\n x = BatchNormalization(axis=self.channel_axis, epsilon=1.001e-5)(x)\n x = Activation(self.active)(x)\n\n x = Conv2D(stem_width, kernel_size=3, strides=1, padding=\"same\",\n kernel_initializer=\"he_normal\", use_bias=False, data_format=\"channels_last\")(x)\n\n x = BatchNormalization(axis=self.channel_axis, epsilon=1.001e-5)(x)\n x = Activation(self.active)(x)\n\n x = Conv2D(stem_width * 2, kernel_size=3, strides=1, padding=\"same\", kernel_initializer=\"he_normal\",\n use_bias=False, data_format=\"channels_last\")(x)\n\n # x = BatchNormalization(axis=self.channel_axis,epsilon=1.001e-5)(x)\n # x = Activation(self.active)(x)\n else:\n x = Conv2D(stem_width, kernel_size=7, strides=2, padding=\"same\", kernel_initializer=\"he_normal\",\n use_bias=False, data_format=\"channels_last\")(x)\n # x = BatchNormalization(axis=self.channel_axis,epsilon=1.001e-5)(x)\n # x = Activation(self.active)(x)\n return x\n\n def _rsoftmax(self, input_tensor, filters, radix, groups):\n x = input_tensor\n batch = x.shape[0]\n if radix > 1:\n x = tf.reshape(x, [-1, groups, radix, filters // groups])\n x = tf.transpose(x, [0, 2, 1, 3])\n x = tf.keras.activations.softmax(x, axis=1)\n x = tf.reshape(x, [-1, 1, 1, radix * filters])\n else:\n x = Activation(\"sigmoid\")(x)\n return x\n\n def _SplAtConv2d(self, input_tensor, filters=64, kernel_size=3, stride=1, dilation=1, groups=1, radix=0):\n x = input_tensor\n in_channels = input_tensor.shape[-1]\n\n x = GroupedConv2D(filters=filters * radix, kernel_size=[kernel_size for i in range(groups * radix)],\n use_keras=True, padding=\"same\", kernel_initializer=\"he_normal\", use_bias=False,\n data_format=\"channels_last\", dilation_rate=dilation)(x)\n\n x = BatchNormalization(axis=self.channel_axis, epsilon=1.001e-5)(x)\n x = Activation(self.active)(x)\n\n batch, rchannel = x.shape[0], x.shape[-1]\n if radix > 1:\n splited = tf.split(x, radix, axis=-1)\n gap = sum(splited)\n else:\n gap = x\n\n # print('sum',gap.shape)\n gap = GlobalAveragePooling2D(data_format=\"channels_last\")(gap)\n gap = tf.reshape(gap, [-1, 1, 1, filters])\n # print('adaptive_avg_pool2d',gap.shape)\n\n reduction_factor = 4\n inter_channels = max(in_channels * radix // reduction_factor, 32)\n\n x = Conv2D(inter_channels, kernel_size=1)(gap)\n\n x = BatchNormalization(axis=self.channel_axis, epsilon=1.001e-5)(x)\n x = Activation(self.active)(x)\n x = Conv2D(filters * radix, kernel_size=1)(x)\n\n atten = self._rsoftmax(x, filters, radix, groups)\n\n if radix > 1:\n logits = tf.split(atten, radix, axis=-1)\n out = sum([a * b for a, b in zip(splited, logits)])\n else:\n out = atten * x\n return out\n\n def _make_block(\n self, input_tensor, first_block=True, filters=64, stride=2, radix=1, avd=False, avd_first=False, is_first=False\n ):\n x = input_tensor\n inplanes = input_tensor.shape[-1]\n if stride != 1 or inplanes != filters * self.block_expansion:\n short_cut = input_tensor\n if self.avg_down:\n if self.dilation == 1:\n short_cut = AveragePooling2D(pool_size=stride, strides=stride, padding=\"same\", data_format=\"channels_last\")(\n short_cut\n )\n else:\n short_cut = AveragePooling2D(pool_size=1, strides=1, padding=\"same\", data_format=\"channels_last\")(short_cut)\n short_cut = Conv2D(filters * self.block_expansion, kernel_size=1, strides=1, padding=\"same\",\n kernel_initializer=\"he_normal\", use_bias=False, data_format=\"channels_last\")(short_cut)\n else:\n short_cut = Conv2D(filters * self.block_expansion, kernel_size=1, strides=stride, padding=\"same\",\n kernel_initializer=\"he_normal\", use_bias=False, data_format=\"channels_last\")(short_cut)\n\n short_cut = BatchNormalization(axis=self.channel_axis, epsilon=1.001e-5)(short_cut)\n else:\n short_cut = input_tensor\n\n group_width = int(filters * (self.bottleneck_width / 64.0)) * self.cardinality\n x = Conv2D(group_width, kernel_size=1, strides=1, padding=\"same\", kernel_initializer=\"he_normal\", use_bias=False,\n data_format=\"channels_last\")(x)\n x = BatchNormalization(axis=self.channel_axis, epsilon=1.001e-5)(x)\n x = Activation(self.active)(x)\n\n avd = avd and (stride > 1 or is_first)\n avd_first = avd_first\n\n if avd:\n avd_layer = AveragePooling2D(pool_size=3, strides=stride, padding=\"same\", data_format=\"channels_last\")\n stride = 1\n\n if avd and avd_first:\n x = avd_layer(x)\n\n if radix >= 1:\n x = self._SplAtConv2d(x, filters=group_width, kernel_size=3, stride=stride, dilation=self.dilation,\n groups=self.cardinality, radix=radix)\n else:\n x = Conv2D(group_width, kernel_size=3, strides=stride, padding=\"same\", kernel_initializer=\"he_normal\",\n dilation_rate=self.dilation, use_bias=False, data_format=\"channels_last\")(x)\n x = BatchNormalization(axis=self.channel_axis, epsilon=1.001e-5)(x)\n x = Activation(self.active)(x)\n\n if avd and not avd_first:\n x = avd_layer(x)\n # print('can')\n x = Conv2D(filters * self.block_expansion, kernel_size=1, strides=1, padding=\"same\", kernel_initializer=\"he_normal\",\n dilation_rate=self.dilation, use_bias=False, data_format=\"channels_last\")(x)\n x = BatchNormalization(axis=self.channel_axis, epsilon=1.001e-5)(x)\n\n m2 = Add()([x, short_cut])\n m2 = Activation(self.active)(m2)\n return m2\n\n def _make_block_basic(\n self, input_tensor, first_block=True, filters=64, stride=2, radix=1, avd=False, avd_first=False, is_first=False\n ):\n \"\"\"Conv2d_BN_Relu->Bn_Relu_Conv2d\n \"\"\"\n x = input_tensor\n x = BatchNormalization(axis=self.channel_axis, epsilon=1.001e-5)(x)\n x = Activation(self.active)(x)\n\n short_cut = x\n inplanes = input_tensor.shape[-1]\n if stride != 1 or inplanes != filters * self.block_expansion:\n if self.avg_down:\n if self.dilation == 1:\n short_cut = AveragePooling2D(pool_size=stride, strides=stride, padding=\"same\", data_format=\"channels_last\")(\n short_cut\n )\n else:\n short_cut = AveragePooling2D(pool_size=1, strides=1, padding=\"same\", data_format=\"channels_last\")(short_cut)\n short_cut = Conv2D(filters, kernel_size=1, strides=1, padding=\"same\", kernel_initializer=\"he_normal\",\n use_bias=False, data_format=\"channels_last\")(short_cut)\n else:\n short_cut = Conv2D(filters, kernel_size=1, strides=stride, padding=\"same\", kernel_initializer=\"he_normal\",\n use_bias=False, data_format=\"channels_last\")(short_cut)\n\n group_width = int(filters * (self.bottleneck_width / 64.0)) * self.cardinality\n avd = avd and (stride > 1 or is_first)\n avd_first = avd_first\n\n if avd:\n avd_layer = AveragePooling2D(pool_size=3, strides=stride, padding=\"same\", data_format=\"channels_last\")\n stride = 1\n\n if avd and avd_first:\n x = avd_layer(x)\n\n if radix >= 1:\n x = self._SplAtConv2d(x, filters=group_width, kernel_size=3, stride=stride, dilation=self.dilation,\n groups=self.cardinality, radix=radix)\n else:\n x = Conv2D(filters, kernel_size=3, strides=stride, padding=\"same\", kernel_initializer=\"he_normal\",\n dilation_rate=self.dilation, use_bias=False, data_format=\"channels_last\")(x)\n\n if avd and not avd_first:\n x = avd_layer(x)\n # print('can')\n\n x = BatchNormalization(axis=self.channel_axis, epsilon=1.001e-5)(x)\n x = Activation(self.active)(x)\n x = Conv2D(filters, kernel_size=3, strides=1, padding=\"same\", kernel_initializer=\"he_normal\",\n dilation_rate=self.dilation, use_bias=False, data_format=\"channels_last\")(x)\n m2 = Add()([x, short_cut])\n return m2\n\n def _make_layer(self, input_tensor, blocks=4, filters=64, stride=2, is_first=True):\n x = input_tensor\n if self.using_basic_block is True:\n x = self._make_block_basic(x, first_block=True, filters=filters, stride=stride, radix=self.radix,\n avd=self.avd, avd_first=self.avd_first, is_first=is_first)\n # print('0',x.shape)\n\n for i in range(1, blocks):\n x = self._make_block_basic(\n x, first_block=False, filters=filters, stride=1, radix=self.radix, avd=self.avd, avd_first=self.avd_first\n )\n # print(i,x.shape)\n\n elif self.using_basic_block is False:\n x = self._make_block(x, first_block=True, filters=filters, stride=stride, radix=self.radix, avd=self.avd,\n avd_first=self.avd_first, is_first=is_first)\n # print('0',x.shape)\n\n for i in range(1, blocks):\n x = self._make_block(\n x, first_block=False, filters=filters, stride=1, radix=self.radix, avd=self.avd, avd_first=self.avd_first\n )\n # print(i,x.shape)\n return x\n\n def _make_Composite_layer(self,input_tensor,filters=256,kernel_size=1,stride=1,upsample=True):\n x = input_tensor\n x = Conv2D(filters, kernel_size, strides=stride, use_bias=False)(x)\n x = BatchNormalization(axis=self.channel_axis, epsilon=1.001e-5)(x)\n if upsample:\n x = UpSampling2D(size=2)(x)\n return x\n\n def get_trainable_parameter(self,shape=(100,128)):\n w_init = tf.random_normal_initializer()\n parameter = tf.Variable(\n initial_value=w_init(shape=shape,\n dtype='float32'),\n trainable=True)\n return parameter\n\n def __make_transformer_top(self,x,verbose=False):\n h = Conv2D(self.hidden_dim,kernel_size=1,strides=1,\n padding='same',kernel_initializer='he_normal',\n use_bias=True,data_format='channels_last')(x)\n if verbose: print('h',h.shape)\n\n if tf.__version__ < \"2.0.0\":\n H,W = h.shape[1].value,h.shape[2].value\n else:\n H,W = h.shape[1],h.shape[2]\n if verbose: print('H,W',H,W)\n \n query_pos = self.get_trainable_parameter(shape=(self.n_query_pos, self.hidden_dim))\n row_embed = self.get_trainable_parameter(shape=(100, self.hidden_dim // 2))\n col_embed = self.get_trainable_parameter(shape=(100, self.hidden_dim // 2))\n\n cat1_col = tf.expand_dims(col_embed[:W], 0)\n cat1_col = tf.repeat(cat1_col, H, axis=0)\n if verbose: print('col_embed',cat1_col.shape)\n\n cat2_row = tf.expand_dims(row_embed[:H], 1)\n cat2_row = tf.repeat(cat2_row, W, axis=1)\n if verbose: print('row_embed',cat2_row.shape)\n\n pos = tf.concat([cat1_col,cat2_row],axis=-1)\n if tf.__version__ < \"2.0.0\":\n pos = tf.expand_dims(tf.reshape(pos,[pos.shape[0].value*pos.shape[1].value,-1]),0)\n else:\n pos = tf.expand_dims(tf.reshape(pos,[pos.shape[0]*pos.shape[1],-1]),0)\n\n h = tf.reshape(h,[-1, h.shape[1]*h.shape[2],h.shape[3]])\n temp_input = pos+h\n\n h_tag = tf.transpose(h,perm=[0, 2, 1])\n if verbose: print('h_tag transpose1',h_tag.shape)\n h_tag = Conv1D(query_pos.shape[0],kernel_size=1,strides=1,\n padding='same',kernel_initializer='he_normal',\n use_bias=True,data_format='channels_last')(h_tag)\n if verbose: print('h_tag conv',h_tag.shape)\n h_tag = tf.transpose(h_tag,perm=[0, 2, 1])\n if verbose: print('h_tag transpose2',h_tag.shape)\n\n query_pos = tf.expand_dims(query_pos,0)\n if verbose: print('query_pos',query_pos.shape)\n query_pos+=h_tag\n query_pos-=h_tag\n\n self.transformer = Transformer(\n d_model=self.hidden_dim, nhead=self.nheads, num_encoder_layers=self.num_encoder_layers,\n num_decoder_layers=self.num_decoder_layers)\n atten_out, attention_weights = self.transformer(temp_input, query_pos)\n return atten_out\n\n def build(self):\n get_custom_objects().update({'mish': Mish(mish)})\n\n input_sig = Input(shape=self.input_shape)\n x = self._make_stem(input_sig, stem_width=self.stem_width, deep_stem=self.deep_stem)\n\n if self.preact is False:\n x = BatchNormalization(axis=self.channel_axis, epsilon=1.001e-5)(x)\n x = Activation(self.active)(x)\n if self.verbose:\n print(\"stem_out\", x.shape)\n\n x = MaxPool2D(pool_size=3, strides=2, padding=\"same\", data_format=\"channels_last\")(x)\n if self.verbose:\n print(\"MaxPool2D out\", x.shape)\n\n if self.preact is True:\n x = BatchNormalization(axis=self.channel_axis, epsilon=1.001e-5)(x)\n x = Activation(self.active)(x)\n \n if self.using_cb:\n second_x = x\n second_x = self._make_layer(x, blocks=self.blocks_set[0], filters=64, stride=1, is_first=False)\n second_x_tmp = self._make_Composite_layer(second_x,filters=x.shape[-1],upsample=False)\n if self.verbose: print('layer 0 db_com',second_x_tmp.shape)\n x = Add()([second_x_tmp, x])\n x = self._make_layer(x, blocks=self.blocks_set[0], filters=64, stride=1, is_first=False)\n if self.verbose:\n print(\"-\" * 5, \"layer 0 out\", x.shape, \"-\" * 5)\n\n b1_b3_filters = [64,128,256,512]\n for i in range(3):\n idx = i+1\n if self.using_cb:\n second_x = self._make_layer(x, blocks=self.blocks_set[idx], filters=b1_b3_filters[idx], stride=2)\n second_x_tmp = self._make_Composite_layer(second_x,filters=x.shape[-1])\n if self.verbose: print('layer {} db_com out {}'.format(idx,second_x_tmp.shape))\n x = Add()([second_x_tmp, x])\n x = self._make_layer(x, blocks=self.blocks_set[idx], filters=b1_b3_filters[idx], stride=2)\n if self.verbose: print('----- layer {} out {} -----'.format(idx,x.shape))\n\n if self.using_transformer:\n x = self.__make_transformer_top(x,verbose=self.verbose)\n if self.verbose: print('transformer out',x.shape)\n else:\n x = GlobalAveragePooling2D(name='avg_pool')(x)\n if self.verbose: print('GlobalAveragePooling2D',x.shape)\n\n if self.dropout_rate > 0:\n x = Dropout(self.dropout_rate, noise_shape=None)(x)\n\n fc_out = Dense(self.n_classes, kernel_initializer=\"he_normal\", use_bias=False, name=\"fc_NObias\")(x)\n if self.verbose:\n print(\"fc_out:\", fc_out.shape)\n\n if self.fc_activation:\n fc_out = Activation(self.fc_activation)(fc_out)\n\n model = models.Model(inputs=input_sig, outputs=fc_out)\n\n if self.verbose:\n print(\"Resnest50_DETR builded with input {}, output{}\".format(input_sig.shape, fc_out.shape))\n if self.verbose:\n print(\"-------------------------------------------\")\n if self.verbose:\n print(\"\")\n\n return model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Kaggle Torch notebook\n\nThis notebook has been taken without edits from [Notebook](https://www.kaggle.com/tanulsingh077/end-to-end-object-detection-with-transformers-detr/notebook) . Since this is in pytorch and provides a good idea of the workflow of Detr.\nThis will be replaced by a customized implementation in Tf/Torch later.","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np \nimport pandas as pd \nfrom datetime import datetime\nimport time\nimport random\nfrom tqdm.autonotebook import tqdm\n\n\n#Torch\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset,DataLoader\nfrom torch.utils.data.sampler import SequentialSampler, RandomSampler\n\n#sklearn\nfrom sklearn.model_selection import StratifiedKFold\n\n#CV\nimport cv2\n\n################# DETR FUCNTIONS FOR LOSS######################## \nimport sys\nsys.path.append('./detr/')\n\nfrom detr.models.matcher import HungarianMatcher\nfrom detr.models.detr import SetCriterion\n#################################################################\n\n#Albumenatations\nimport albumentations as A\nimport matplotlib.pyplot as plt\nfrom albumentations.pytorch.transforms import ToTensorV2\n\n#Glob\nfrom glob import glob\n\n\nclass AverageMeter(object):\n \"\"\"Computes and stores the average and current value\"\"\"\n def __init__(self):\n self.reset()\n\n def reset(self):\n self.val = 0\n self.avg = 0\n self.sum = 0\n self.count = 0\n\n def update(self, val, n=1):\n self.val = val\n self.sum += val * n\n self.count += n\n self.avg = self.sum / self.count\n \n \nn_folds = 5\nseed = 42\nnum_classes = 2\nnum_queries = 100\nnull_class_coef = 0.5\nBATCH_SIZE = 8\nLR = 2e-5\nEPOCHS = 2\ndef seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = True\nseed_everything(seed)\n\nmarking = pd.read_csv('../input/global-wheat-detection/train.csv')\n\nbboxs = np.stack(marking['bbox'].apply(lambda x: np.fromstring(x[1:-1], sep=',')))\nfor i, column in enumerate(['x', 'y', 'w', 'h']):\n marking[column] = bboxs[:,i]\nmarking.drop(columns=['bbox'], inplace=True)\n\nskf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=seed)\n\ndf_folds = marking[['image_id']].copy()\ndf_folds.loc[:, 'bbox_count'] = 1\ndf_folds = df_folds.groupby('image_id').count()\ndf_folds.loc[:, 'source'] = marking[['image_id', 'source']].groupby('image_id').min()['source']\ndf_folds.loc[:, 'stratify_group'] = np.char.add(\n df_folds['source'].values.astype(str),\n df_folds['bbox_count'].apply(lambda x: f'_{x // 15}').values.astype(str)\n)\ndf_folds.loc[:, 'fold'] = 0\n\nfor fold_number, (train_index, val_index) in enumerate(skf.split(X=df_folds.index, y=df_folds['stratify_group'])):\n df_folds.loc[df_folds.iloc[val_index].index, 'fold'] = fold_number\n \n \ndef get_train_transforms():\n return A.Compose([A.OneOf([A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit= 0.2, val_shift_limit=0.2, p=0.9),\n \n A.RandomBrightnessContrast(brightness_limit=0.2, contrast_limit=0.2, p=0.9)],p=0.9),\n \n A.ToGray(p=0.01),\n \n A.HorizontalFlip(p=0.5),\n \n A.VerticalFlip(p=0.5),\n \n A.Resize(height=512, width=512, p=1),\n \n A.Cutout(num_holes=8, max_h_size=64, max_w_size=64, fill_value=0, p=0.5),\n \n ToTensorV2(p=1.0)],\n \n p=1.0,\n \n bbox_params=A.BboxParams(format='coco',min_area=0, min_visibility=0,label_fields=['labels'])\n )\n\ndef get_valid_transforms():\n return A.Compose([A.Resize(height=512, width=512, p=1.0),\n ToTensorV2(p=1.0)], \n p=1.0, \n bbox_params=A.BboxParams(format='coco',min_area=0, min_visibility=0,label_fields=['labels'])\n )\n\n\nDIR_TRAIN = '../input/global-wheat-detection/train'\n\nclass WheatDataset(Dataset):\n def __init__(self,image_ids,dataframe,transforms=None):\n self.image_ids = image_ids\n self.df = dataframe\n self.transforms = transforms\n \n \n def __len__(self) -> int:\n return self.image_ids.shape[0]\n \n def __getitem__(self,index):\n image_id = self.image_ids[index]\n records = self.df[self.df['image_id'] == image_id]\n \n image = cv2.imread(f'{DIR_TRAIN}/{image_id}.jpg', cv2.IMREAD_COLOR)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)\n image /= 255.0\n \n # DETR takes in data in coco format \n boxes = records[['x', 'y', 'w', 'h']].values\n \n #Area of bb\n area = boxes[:,2]*boxes[:,3]\n area = torch.as_tensor(area, dtype=torch.float32)\n \n # AS pointed out by PRVI It works better if the main class is labelled as zero\n labels = np.zeros(len(boxes), dtype=np.int32)\n\n \n if self.transforms:\n sample = {\n 'image': image,\n 'bboxes': boxes,\n 'labels': labels\n }\n sample = self.transforms(**sample)\n image = sample['image']\n boxes = sample['bboxes']\n labels = sample['labels']\n \n \n #Normalizing BBOXES\n \n _,h,w = image.shape\n boxes = A.augmentations.bbox_utils.normalize_bboxes(sample['bboxes'],rows=h,cols=w)\n target = {}\n target['boxes'] = torch.as_tensor(boxes,dtype=torch.float32)\n target['labels'] = torch.as_tensor(labels,dtype=torch.long)\n target['image_id'] = torch.tensor([index])\n target['area'] = area\n \n return image, target, image_id","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class DETRModel(nn.Module):\n def __init__(self,num_classes,num_queries):\n super(DETRModel,self).__init__()\n self.num_classes = num_classes\n self.num_queries = num_queries\n \n self.model = torch.hub.load('facebookresearch/detr', 'detr_resnet50', pretrained=True)\n self.in_features = self.model.class_embed.in_features\n \n self.model.class_embed = nn.Linear(in_features=self.in_features,out_features=self.num_classes)\n self.model.num_queries = self.num_queries\n \n def forward(self,images):\n return self.model(images)\n \n'''\ncode taken from github repo detr , 'code present in engine.py'\n'''\n\nmatcher = HungarianMatcher()\n\nweight_dict = weight_dict = {'loss_ce': 1, 'loss_bbox': 1 , 'loss_giou': 1}\n\nlosses = ['labels', 'boxes', 'cardinality']\n\n\ndef train_fn(data_loader,model,criterion,optimizer,device,scheduler,epoch):\n model.train()\n criterion.train()\n \n summary_loss = AverageMeter()\n \n tk0 = tqdm(data_loader, total=len(data_loader))\n \n for step, (images, targets, image_ids) in enumerate(tk0):\n \n images = list(image.to(device) for image in images)\n targets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n \n\n output = model(images)\n \n loss_dict = criterion(output, targets)\n weight_dict = criterion.weight_dict\n \n losses = sum(loss_dict[k] * weight_dict[k] for k in loss_dict.keys() if k in weight_dict)\n \n optimizer.zero_grad()\n\n losses.backward()\n optimizer.step()\n if scheduler is not None:\n scheduler.step()\n \n summary_loss.update(losses.item(),BATCH_SIZE)\n tk0.set_postfix(loss=summary_loss.avg)\n \n return summary_loss\n\n\ndef collate_fn(batch):\n return tuple(zip(*batch))\n\n\ndef run(fold):\n \n df_train = df_folds[df_folds['fold'] != fold]\n df_valid = df_folds[df_folds['fold'] == fold]\n \n train_dataset = WheatDataset(\n image_ids=df_train.index.values,\n dataframe=marking,\n transforms=get_train_transforms()\n )\n\n valid_dataset = WheatDataset(\n image_ids=df_valid.index.values,\n dataframe=marking,\n transforms=get_valid_transforms()\n )\n \n train_data_loader = DataLoader(\n train_dataset,\n batch_size=BATCH_SIZE,\n shuffle=False,\n num_workers=4,\n collate_fn=collate_fn\n )\n\n valid_data_loader = DataLoader(\n valid_dataset,\n batch_size=BATCH_SIZE,\n shuffle=False,\n num_workers=4,\n collate_fn=collate_fn\n )\n \n device = torch.device('cuda')\n model = DETRModel(num_classes=num_classes,num_queries=num_queries)\n model = model.to(device)\n criterion = SetCriterion(num_classes-1, matcher, weight_dict, eos_coef = null_class_coef, losses=losses)\n criterion = criterion.to(device)\n \n\n optimizer = torch.optim.AdamW(model.parameters(), lr=LR)\n \n best_loss = 10**5\n for epoch in range(EPOCHS):\n train_loss = train_fn(train_data_loader, model,criterion, optimizer,device,scheduler=None,epoch=epoch)\n valid_loss = eval_fn(valid_data_loader, model,criterion, device)\n \n print('|EPOCH {}| TRAIN_LOSS {}| VALID_LOSS {}|'.format(epoch+1,train_loss.avg,valid_loss.avg))\n \n if valid_loss.avg < best_loss:\n best_loss = valid_loss.avg\n print('Best model found for Fold {} in Epoch {}........Saving Model'.format(fold,epoch+1))\n torch.save(model.state_dict(), f'detr_best_{fold}.pth')\nrun(fold=0)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def view_sample(df_valid,model,device):\n '''\n Code taken from Peter's Kernel \n https://www.kaggle.com/pestipeti/pytorch-starter-fasterrcnn-train\n '''\n valid_dataset = WheatDataset(image_ids=df_valid.index.values,\n dataframe=marking,\n transforms=get_valid_transforms()\n )\n \n valid_data_loader = DataLoader(\n valid_dataset,\n batch_size=BATCH_SIZE,\n shuffle=False,\n num_workers=4,\n collate_fn=collate_fn)\n \n images, targets, image_ids = next(iter(valid_data_loader))\n _,h,w = images[0].shape # for de normalizing images\n \n images = list(img.to(device) for img in images)\n targets = [{k: v.to(device) for k, v in t.items()} for t in targets]\n \n boxes = targets[0]['boxes'].cpu().numpy()\n boxes = [np.array(box).astype(np.int32) for box in A.augmentations.bbox_utils.denormalize_bboxes(boxes,h,w)]\n sample = images[0].permute(1,2,0).cpu().numpy()\n \n model.eval()\n model.to(device)\n cpu_device = torch.device(\"cpu\")\n \n with torch.no_grad():\n outputs = model(images)\n \n outputs = [{k: v.to(cpu_device) for k, v in outputs.items()}]\n \n fig, ax = plt.subplots(1, 1, figsize=(16, 8))\n\n for box in boxes:\n cv2.rectangle(sample,\n (box[0], box[1]),\n (box[2]+box[0], box[3]+box[1]),\n (220, 0, 0), 1)\n \n\n oboxes = outputs[0]['pred_boxes'][0].detach().cpu().numpy()\n oboxes = [np.array(box).astype(np.int32) for box in A.augmentations.bbox_utils.denormalize_bboxes(oboxes,h,w)]\n prob = outputs[0]['pred_logits'][0].softmax(1).detach().cpu().numpy()[:,0]\n \n for box,p in zip(oboxes,prob):\n \n if p >0.5:\n color = (0,0,220) #if p>0.5 else (0,0,0)\n cv2.rectangle(sample,\n (box[0], box[1]),\n (box[2]+box[0], box[3]+box[1]),\n color, 1)\n \n ax.set_axis_off()\n ax.imshow(sample)\nmodel = DETRModel(num_classes=num_classes,num_queries=num_queries)\nmodel.load_state_dict(torch.load(\"./detr_best_0.pth\"))\nview_sample(df_folds[df_folds['fold'] == 0],model=model,device=torch.device('cuda'))\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"WIP","metadata":{}}]}