{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# TensorFlow baseline Implementation","metadata":{}},{"cell_type":"markdown","source":"In this notebook I have given a basline approach to this problem using Tensorflow. But that's not all.\nI have written the approach whilst following **Object Oriented Approach** and a few of **design patterns** as well. I have made sure not to use any procedural code.\n\nI am sure almost everyone is aware of the benfits of writing code using Object oriented paradigm instead of procedural paradigm. I highly recommend everyone to go through this notebook as I am sure, it would be *great learning experience* for everyone (I say that because it was surely for me witing this notebook), more so for beginners in Data Science and object oriented design patterns.","metadata":{}},{"cell_type":"markdown","source":"**Please remember to upvote the notebook if you like the content. It is always a great motivator to write such notebook in the future ;) !!**","metadata":{}},{"cell_type":"markdown","source":"I have taken a lot of help from Notebook [**here**](https://www.kaggle.com/ammarali32/molecular-translation-simple-training-starter/notebook).","metadata":{}},{"cell_type":"markdown","source":"I would very much appreciate if anyone has any **suggestions** on how I can **improve** this a bit further.\n\nThanks in Advance!!","metadata":{}},{"cell_type":"markdown","source":"**Note**: The class names used here in the code are very intuitive and easy to understand, just go though once or twice. If you still don't understand, do let me know in the comments.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport cv2\n\nimport tensorflow as tf\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.utils import shuffle\nimport collections\nimport random\nimport re\nimport numpy as np\nimport os\nimport time\nimport json\nfrom glob import glob\nfrom PIL import Image\nimport pickle","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class CFG:\n    \n    EPOCHS = 8\n    top_k = 5000\n    BATCH_SIZE = 64\n    BUFFER_SIZE = 1000\n    embedding_dim = 256\n    units = 512\n    vocab_size = top_k + 1\n    num_steps = 75\n    features_shape = 2048\n    attention_features_shape = 64\n    TRAIN_LABELS_PATH = \"../input/bms-molecular-translation/train_labels.csv\"\n    Data_path = '../input/bms-molecular-translation/train/0/0/0/'\n    tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=top_k,\n                                                  oov_token=\"<unk>\",\n                                                  filters='!\"#$%&*+.-;?@[]^`{}~ ')\n    checkpoint_path = \"./checkpoints/train\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Attention(tf.keras.Model):\n    \n    def __init__(self, units):\n    \n        super(Attention, self).__init__()\n        self.W1 = tf.keras.layers.Dense(units)\n        self.W2 = tf.keras.layers.Dense(units)\n        self.V = tf.keras.layers.Dense(1)\n\n    def call(self, features, hidden):\n        \n        hidden_with_time_axis = tf.expand_dims(hidden, 1)\n        attention_hidden_layer = (tf.nn.tanh(self.W1(features) +\n                                             self.W2(hidden_with_time_axis)))\n        score = self.V(attention_hidden_layer)\n        attention_weights = tf.nn.softmax(score, axis=1)\n        context_vector = attention_weights * features\n        context_vector = tf.reduce_sum(context_vector, axis=1)\n        return context_vector, attention_weights","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class CNN_Encoder(tf.keras.Model):\n    \n    def __init__(self, embedding_dim):\n        \n        super(CNN_Encoder, self).__init__()\n        self.fc = tf.keras.layers.Dense(embedding_dim)\n    \n    def call(self, x):\n        \n        x = self.fc(x)\n        x = tf.nn.relu(x)\n        return x","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class RNN_Decoder(tf.keras.Model):\n  \n    def __init__(self, embedding_dim, units, vocab_size):\n        \n        super(RNN_Decoder, self).__init__()\n        self.units = units\n\n        self.embedding = tf.keras.layers.Embedding(vocab_size,embedding_dim)\n        self.gru = tf.keras.layers.GRU(self.units,\n                                       return_sequences=True,\n                                       return_state=True,\n                                       recurrent_initializer='glorot_uniform')\n        self.fc1 = tf.keras.layers.Dense(self.units)\n        self.fc2 = tf.keras.layers.Dense(vocab_size)\n\n        self.attention = Attention(self.units)\n\n    def call(self, x, features, hidden):\n    \n        context_vector, attention_weights = self.attention(features, hidden)\n        x = self.embedding(x)\n        x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)\n        output, state = self.gru(x)\n        x = self.fc1(output)\n        x = tf.reshape(x, (-1, x.shape[2]))\n        x = self.fc2(x)\n        return x, state, attention_weights\n\n    def reset_state(self, batch_size):\n    \n        return tf.zeros((batch_size, self.units))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class getDataframe:\n    \n    __instance = None\n    \n    @staticmethod \n    def getInstance():\n        \"\"\" Static access method. \"\"\"\n        if getDataframe.__instance == None:\n            getDataframe(df_train_labels)\n        return getDataframe.__instance\n    \n    def __init__(self):\n        \n        df_train_labels = pd.read_csv(CFG.TRAIN_LABELS_PATH, index_col=0)\n        self.df_train_labels = df_train_labels\n        getDataframe.__instance = self","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train_labels_new = getDataframe().getInstance().df_train_labels\ndf_train_labels_new","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Visualize:\n    \n    def __init__(self):\n        pass\n    \n    def visualize_images():\n        pass","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Visualize_batch_images(Visualize):\n    \n    def __init__(self, path, image_ids, labels):\n        \n        super(Visualize_batch_images, self).__init__()\n        self.path = path\n        self.image_ids = image_ids\n        self.labels = labels\n        \n    def convert_image_id_2_path(self, image_id: str) -> str:\n        return \"../input/bms-molecular-translation/train/{}/{}/{}/{}.png\".format(\n            image_id[0], image_id[1], image_id[2], image_id \n        )\n        \n    def visualize_images(self):\n        \n        plt.figure(figsize=(16, 12))\n    \n        for ind, (image_id, label) in enumerate(zip(self.image_ids, self.labels)):\n            plt.subplot(3, 3, ind + 1)\n            image = cv2.imread(self.convert_image_id_2_path(image_id))\n            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n            plt.imshow(image)\n            plt.title(f\"{label[:30]}...\", fontsize=10)\n            plt.axis(\"off\")\n\n        plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Visualize_single_image(Visualize):\n    \n    def __init__(self, path, image_id, label):\n        \n        super(Visualize_single_image, self).__init__()\n        self.path = path\n        self.image_id = image_id\n        self.label = label\n        \n    def convert_image_id_2_path(self, image_id: str) -> str:\n        return \"../input/bms-molecular-translation/train/{}/{}/{}/{}.png\".format(\n            self.image_id[0], self.image_id[1], self.image_id[2], self.image_id \n        )\n        \n    def visualize_images(self):\n        \n        plt.figure(figsize=(10, 8))\n            \n        image = cv2.imread(self.convert_image_id_2_path(self.image_id))\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        plt.imshow(image)\n        plt.title(f\"{self.label}\", fontsize=14)\n        plt.axis(\"off\")\n\n        plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Illustrating the working of Visualizing class\nsample_row = getDataframe().getInstance().df_train_labels.sample(5)\n\nfor i in range(5):\n    \n    viz_object = Visualize_single_image('', sample_row.index[i], sample_row[\"InChI\"][i])\n    viz_object.visualize_images()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Illustrating the working of Batch Visualizing class\nsample_row = getDataframe().getInstance().df_train_labels.sample(5)\n\nimg_ids = [i for i in sample_row.index]\nlabels = [i for i in sample_row[\"InChI\"]]\n\nfor i in range(5):\n    \n    viz_object = Visualize_batch_images('', img_ids, labels)\n    viz_object.visualize_images()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Helper:\n    \n    def load_image(image_path):\n    \n        img = tf.io.read_file(image_path)\n        img = tf.image.decode_jpeg(img, channels=3)\n        img = tf.image.resize(img, (224, 224))\n        img = tf.keras.applications.mobilenet_v2.preprocess_input(img)\n        return img, image_path\n\n    def map_func(img_name, cap):\n        img_tensor = np.load('features/' + img_name.decode('utf-8')[len(CFG.Data_path):-4]+'.npy')\n        return img_tensor, cap\n    \n    def convert_image_id_2_path(image_id: str) -> str:\n        return \"../input/bms-molecular-translation/train/{}/{}/{}/{}.png\".format(\n            image_id[0], image_id[1], image_id[2], image_id \n        )\n    \n    def Train_image_paths(populateImageCaption_Path):\n        \n        image_paths = list(populateImageCaption_Path.keys())\n        #random.shuffle(image_paths)\n        # Let us take just first 6000 images for training now \n        train_image_paths = image_paths[:5500]\n        \n        return train_image_paths\n    \n    def Image_name_vector(train_image_paths):\n        \n        train_captions = []\n        img_name_vector = []\n        \n        image_path_to_caption = populateImageCaptionPath(getDataframe().getInstance().df_train_labels).populate()\n        \n        for image_path in train_image_paths:\n            caption_list = image_path_to_caption[image_path]\n            train_captions.extend(caption_list)\n            img_name_vector.extend([image_path] * len(caption_list))\n            \n        return train_captions, img_name_vector\n    \n    def calc_max_length(tensor):\n        return max(len(t) for t in tensor)\n    \n    def create_cap_val(df_train_labels):\n        \n        img_to_cap_vector = collections.defaultdict(list)\n        \n        image_path_to_caption = populateImageCaptionPath(df_train_labels).populate()\n        train_image_paths = Helper.Train_image_paths(image_path_to_caption)\n        img_name_vector = Helper.Image_name_vector(train_image_paths)[1]\n        \n        cap_vector = populateCapVector(df_train_labels).populate()\n        \n        for img, cap in zip(img_name_vector, cap_vector):\n            img_to_cap_vector[img].append(cap)\n\n        # Create training and validation sets\n        img_keys = list(img_to_cap_vector.keys())\n        random.shuffle(img_keys)\n\n        slice_index = int(len(img_keys)*0.8)\n        img_name_train_keys, img_name_val_keys = img_keys[:slice_index], img_keys[slice_index:]\n\n        img_name_val = []\n        cap_val = []\n        for imgv in img_name_val_keys:\n            capv_len = len(img_to_cap_vector[imgv])\n            img_name_val.extend([imgv] * capv_len)\n            cap_val.extend(img_to_cap_vector[imgv])\n\n            \n        return cap_val, img_name_val\n    \n    def create_cap_train(df_train_labels):\n        \n        img_to_cap_vector = collections.defaultdict(list)\n        \n        image_path_to_caption = populateImageCaptionPath(df_train_labels).populate()\n        train_image_paths = Helper.Train_image_paths(image_path_to_caption)\n        train_captions, img_name_vector = Helper.Image_name_vector(train_image_paths)\n        \n        cap_vector = populateCapVector(df_train_labels, image_path_to_caption, train_captions, train_image_paths).populate()\n        \n        for img, cap in zip(img_name_vector, cap_vector):\n            img_to_cap_vector[img].append(cap)\n\n        # print(\"img_to_cap_vector: \", len(img_to_cap_vector))\n        # print(\"img_name_vector: \", len(img_name_vector))\n        # print(\"cap_vector: \", len(cap_vector))\n        # Create training and validation sets\n        img_keys = list(img_to_cap_vector.keys())\n        #random.shuffle(img_keys)\n\n        slice_index = int(len(img_keys)*0.8)\n        img_name_train_keys, img_name_val_keys = img_keys[:slice_index], img_keys[slice_index:]\n\n        img_name_train = []\n        cap_train = []\n        for imgt in img_name_train_keys:\n            capt_len = len(img_to_cap_vector[imgt])\n            img_name_train.extend([imgt] * capt_len)\n            cap_train.extend(img_to_cap_vector[imgt])\n            \n        return cap_train, img_name_train\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Load:\n    \n    def __init__(self):\n        pass\n        \n    def load(self):\n        pass","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class LoadImage(Load):\n    \n    def __init__(self, path):\n        super(LoadImage, self).__init__()\n        self.path = path\n        \n    def load(self, path):\n        \n        img = tf.io.read_file(path)\n        img = tf.image.decode_jpeg(img, channels=3)\n        img = tf.image.resize(img, (224, 224))\n        img = tf.keras.applications.mobilenet_v2.preprocess_input(img)\n        \n        return img, path","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class LoadModel(Load):\n        \n    def load(self):\n        \n        image_model = tf.keras.applications.MobileNetV2(include_top=False,\n                                                weights='imagenet')\n        \n        return image_model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class populateData:\n    \n    def __init__(self):\n        pass\n    \n    def populate():\n        pass","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class populateImageCaptionPath(populateData):\n    \n    def __init__(self, df_train_labels):\n        \n        self.df_train_labels = df_train_labels\n        \n    def populate(self):\n        \n        image_path_to_caption = collections.defaultdict(list)\n        \n        for idx,path in enumerate(self.df_train_labels.index):\n            caption = self.df_train_labels['InChI'].iloc[idx]\n            image_path = Helper.convert_image_id_2_path(path)\n            image_path_to_caption[image_path].append(caption)\n        \n        return image_path_to_caption","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class populateCapVector(populateData):\n    \n    def __init__(self, df_train_labels, image_path_to_caption, train_captions, train_image_paths):\n        \n        self.tokenizer = CFG.tokenizer\n        self.df_train_labels = df_train_labels\n        self.image_path_to_caption = image_path_to_caption\n        self.train_image_paths = train_image_paths\n        self.train_captions = train_captions\n        \n\n    def populate(self):\n        \n        self.tokenizer.fit_on_texts(self.train_captions)\n        self.tokenizer.word_index['<pad>'] = 0\n        self.tokenizer.index_word[0] = '<pad>'\n        \n        # Create the tokenized vectors\n        train_seqs = self.tokenizer.texts_to_sequences(self.train_captions)\n        \n        # Pad each vector to the max_length of InChI\n        cap_vector = tf.keras.preprocessing.sequence.pad_sequences(train_seqs, padding='post')\n        \n        max_length = Helper.calc_max_length(train_seqs)\n        \n        return cap_vector","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!mkdir features/","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class extractFeatures:\n    \n    def __init__(self, df_train_labels):\n        pass\n        \n    def extract(self):\n        pass","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class extractFeatures_images(extractFeatures):\n    \n    def __init__(self, df_train_labels):\n        self.df_train_labels = df_train_labels\n    \n    def extract(self):\n        \n        image_model_object = LoadModel()\n        image_model = image_model_object.load()\n        new_input = image_model.input\n        hidden_layer = image_model.layers[-1].output\n\n        image_features_extract_model = tf.keras.Model(new_input, hidden_layer)\n        \n        # Extract features\n        image_path_to_caption = populateImageCaptionPath(self.df_train_labels).populate()\n        train_image_paths = Helper.Train_image_paths(image_path_to_caption)\n        encode_train = sorted(set(Helper.Image_name_vector(train_image_paths)[1]))\n        \n        image_dataset = tf.data.Dataset.from_tensor_slices(encode_train)\n        image_dataset = image_dataset.map(Helper.load_image, num_parallel_calls=tf.data.AUTOTUNE).batch(16)\n        \n        for img, path in image_dataset:\n            \n            batch_features = image_features_extract_model(img)\n            batch_features = tf.reshape(batch_features,\n                                      (batch_features.shape[0], -1, batch_features.shape[3]))\n\n            for bf, p in zip(batch_features, path):\n                \n                path_of_feature ='features/'+ p.numpy().decode(\"utf-8\")[len(CFG.Data_path):-4]\n                #print(path_of_feature)\n                np.save(path_of_feature, bf.numpy())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"extract_obj = extractFeatures_images(getDataframe().getInstance().df_train_labels)\nextract_obj.extract()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class populateDataset(populateData):\n    \n    def __init__(self, df_train_labels):\n        self.df_train_labels = df_train_labels\n        self.cap_train, self.img_name_train = Helper.create_cap_train(self.df_train_labels)\n    \n    \n    def map_funct(self, img_name, cap):\n        img_tensor = np.load('features/' + img_name.decode('utf-8')[len(CFG.Data_path):-4]+'.npy')\n        return img_tensor, cap\n    \n    def populate(self):\n        \n        dataset = tf.data.Dataset.from_tensor_slices((self.img_name_train, self.cap_train))\n        dataset = dataset.map(lambda item1, item2: tf.numpy_function(\n                  self.map_funct, [item1, item2], [tf.float32, tf.int32]),\n                  num_parallel_calls=tf.data.AUTOTUNE)\n        dataset = dataset.shuffle(CFG.BUFFER_SIZE).batch(CFG.BATCH_SIZE)\n        dataset = dataset.prefetch(buffer_size=tf.data.AUTOTUNE)\n        \n        return dataset","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls features/ | wc -l","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class checkpoint:\n    \n    def __init__(self):\n        \n        self.encoder = CNN_Encoder(CFG.embedding_dim)\n        self.decoder = RNN_Decoder(CFG.embedding_dim, CFG.units, CFG.vocab_size)\n        self.optimizer = tf.keras.optimizers.Adam()\n    \n    def return_ckpt_manager(self):\n        checkpoint_path = CFG.checkpoint_path\n        ckpt = tf.train.Checkpoint(encoder = self.encoder,\n                               decoder = self.decoder,\n                               optimizer = self.optimizer)\n\n        ckpt_manager = tf.train.CheckpointManager(ckpt, checkpoint_path, max_to_keep=5)\n        \n        return ckpt_manager","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Training:\n    \n    def __init__(self, df_train_labels):\n        \n        self.encoder = CNN_Encoder(CFG.embedding_dim)\n        self.decoder = RNN_Decoder(CFG.embedding_dim, CFG.units, CFG.vocab_size)\n        self.optimizer = tf.keras.optimizers.Adam()\n        self.start_epoch = int()\n        self.ckpt_manager = checkpoint().return_ckpt_manager()\n        self.df_train_labels = df_train_labels\n        \n    def loss_function(self, real, pred):\n        \n        loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n                      from_logits=True, reduction='none'\n        )\n        \n        mask = tf.math.logical_not(tf.math.equal(real, 0))\n        loss_ = loss_object(real, pred)\n\n        mask = tf.cast(mask, dtype=loss_.dtype)\n        loss_ *= mask\n\n        return tf.reduce_mean(loss_)\n    \n    def checkpoint(self):\n        \n        checkpoint_path = CFG.checkpoint_path\n        ckpt = tf.train.Checkpoint(encoder = self.encoder,\n                           decoder = self.decoder,\n                           optimizer = self.optimizer)\n        \n        ckpt_manager = tf.train.CheckpointManager(ckpt, checkpoint_path, max_to_keep=5)\n        \n        loss_plot = []\n        start_epoch = 0\n        if ckpt_manager.latest_checkpoint:\n            self.start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])\n            ckpt.restore(ckpt_manager.latest_checkpoint)\n        \n    @tf.function\n    def train_step(self, img_tensor, target):\n        \n        #self.checkpoint()\n        loss = 0\n        hidden = self.decoder.reset_state(batch_size=target.shape[0])\n        dec_input = tf.expand_dims([CFG.tokenizer.word_index['<unk>']] * target.shape[0], 1)\n        \n        with tf.GradientTape() as tape:\n            features = self.encoder(img_tensor)\n            for i in range(1, target.shape[1]):\n                predictions, hidden, _ = self.decoder(dec_input, features, hidden)\n                loss += self.loss_function(target[:, i], predictions)\n                dec_input = tf.expand_dims(target[:, i], 1)\n        \n        total_loss = (loss / int(target.shape[1]))\n        trainable_variables = self.encoder.trainable_variables + self.decoder.trainable_variables\n        gradients = tape.gradient(loss, trainable_variables)\n        self.optimizer.apply_gradients(zip(gradients, trainable_variables))\n        \n        return loss, total_loss\n\n    \n    def train(self):\n        \n        loss_plot = []\n        for epoch in range(self.start_epoch, CFG.EPOCHS):\n            start = time.time()\n            total_loss = 0\n            dataset = populateDataset(getDataframe().getInstance().df_train_labels).populate()\n\n            for (batch, (img_tensor, target)) in enumerate(dataset):\n\n                batch_loss, t_loss = self.train_step(img_tensor, target)\n                total_loss += t_loss\n                if batch % 100 == 0:\n                    print ('Epoch {} Batch {} Loss {:.4f}'.format(\n                      epoch + 1, batch, batch_loss.numpy() / int(target.shape[1])))\n\n            # Let us save the loss info to visualize\n            loss_plot.append(total_loss / CFG.num_steps)\n            if epoch % 5 == 0:\n                self.ckpt_manager.save()\n            print ('Epoch {} Loss {:.6f}'.format(epoch + 1, total_loss/CFG.num_steps))\n            print ('Time taken for 1 epoch {} sec\\n'.format(time.time() - start))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_obj = Training(getDataframe().getInstance().df_train_labels)\ntrain_obj.train()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**This is still A Work in Progress and I will keep making changes to it.** \n\n**So please do let me know in the comments if there's any way I can improve the code.**","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}