{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"tpu1vmV38","dataSources":[{"sourceId":19018,"databundleVersionId":2703900,"sourceType":"competition"},{"sourceId":11650,"sourceType":"datasetVersion","datasetId":8327}],"dockerImageVersionId":30299,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np\n\nimport pandas as pd\n\nfrom tqdm import tqdm  # tqdm is a library for adding progress bars to loops\n\nfrom sklearn.model_selection import train_test_split  \n\nimport tensorflow as tf \n\nfrom keras.models import Sequential  # Sequential is a Keras class to build models layer by layer\n\nfrom keras.layers.recurrent import LSTM, GRU, SimpleRNN  # LSTM, GRU, and SimpleRNN are recurrent neural network layers in Keras\n\nfrom keras.layers.core import Dense, Activation, Dropout  # Dense, Activation, and Dropout are core layers in Keras for defining neural network architectures\n\nfrom keras.layers.embeddings import Embedding  # Embedding is a layer in Keras for word embedding\n\nfrom keras.layers import BatchNormalization  # BatchNormalization is a layer in Keras for normalizing activations in neural networks\n\nfrom keras.utils import np_utils  # np_utils is a utility function in Keras for converting class vectors to binary class matrices\n\nfrom sklearn import preprocessing, decomposition, model_selection, metrics, pipeline  # scikit-learn (sklearn) is a library for machine learning in Python\n\nfrom keras.layers import GlobalMaxPooling1D, Conv1D, MaxPooling1D, Flatten, Bidirectional, SpatialDropout1D  # Additional layers in Keras for deep learning models\n\nfrom keras.preprocessing import sequence, text  # Tools in Keras for preprocessing text data\n\nfrom keras.callbacks import EarlyStopping  # EarlyStopping is a callback in Keras for stopping training when a monitored metric has stopped improving\n\nimport matplotlib.pyplot as plt  # Matplotlib is a plotting library for Python\n\nimport seaborn as sns \n\n%matplotlib inline  \n\nfrom plotly import graph_objs as go\n\nimport plotly.express as px  # Plotly Express is a high-level interface for creating interactive plots\n\nimport plotly.figure_factory as ff  # Plotly Figure Factory is a module for creating complex visualizations\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Utilizing distributed training with TensorFlow's tf.distribute.Strategy API.\n\nuse strategy.scope():\n\n+ Distributed Training:\nWhen training deep learning models on large datasets or with complex architectures, training can become time-consuming, especially on single devices. Distributed training allows you to distribute the computation across multiple devices (e.g., multiple GPUs or TPUs) to speed up the training process.\ntf.distribute.Strategy:\n+ TensorFlow's tf.distribute.Strategy API provides a high-level abstraction for distributed training. It allows you to scale your training across multiple devices without significant code changes.\n+ The strategy.scope() context manager is used to define the scope within which your model and optimizer are created. Inside this scope, TensorFlow operations will be distributed across the devices specified by the strategy.\n","metadata":{}},{"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"execution":{"iopub.status.busy":"2024-05-10T05:10:23.343274Z","iopub.execute_input":"2024-05-10T05:10:23.343678Z","iopub.status.idle":"2024-05-10T05:10:23.351513Z","shell.execute_reply.started":"2024-05-10T05:10:23.343642Z","shell.execute_reply":"2024-05-10T05:10:23.350415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## About the dataset\nThis contains the text of a comment which has been classified as toxic or non-toxic (0…1 in the toxic column). The train set’s comments are entirely in english and come either from Civil Comments or Wikipedia talk page edits. The test data's comment_text columns are composed of multiple non-English languages.\n\nPredicting the probability that a comment is toxic. A toxic comment would receive a 1.0. A benign, non-toxic comment would receive a 0.0.\n\n**jigsaw-toxic-comment-train.csv - The dataset is made up of English comments from Wikipedia’s talk page edits.**\n\n**test.csv - comments from Wikipedia talk pages in different non-English languages.**\n\n**validation.csv - comments from Wikipedia talk pages in different non-English languages.**","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv')\nvalidation = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation.csv')\ntest = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv')","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2024-05-10T05:29:11.067654Z","iopub.execute_input":"2024-05-10T05:29:11.068530Z","iopub.status.idle":"2024-05-10T05:29:12.882886Z","shell.execute_reply.started":"2024-05-10T05:29:11.068495Z","shell.execute_reply":"2024-05-10T05:29:12.881847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will drop the other columns and approach this problem as a Binary Classification Problem","metadata":{}},{"cell_type":"code","source":"train.drop(['severe_toxic','obscene','threat','insult','identity_hate'],axis=1,inplace=True)\ntrain = train.loc[:12000,:]","metadata":{"execution":{"iopub.status.busy":"2024-05-10T05:29:12.884755Z","iopub.execute_input":"2024-05-10T05:29:12.885104Z","iopub.status.idle":"2024-05-10T05:29:12.902717Z","shell.execute_reply.started":"2024-05-10T05:29:12.885070Z","shell.execute_reply":"2024-05-10T05:29:12.901377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-10T05:30:53.643686Z","iopub.execute_input":"2024-05-10T05:30:53.644114Z","iopub.status.idle":"2024-05-10T05:30:53.664251Z","shell.execute_reply.started":"2024-05-10T05:30:53.644074Z","shell.execute_reply":"2024-05-10T05:30:53.663255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def roc_auc(predictions,target):\n    '''\n    This methods returns the AUC Score when given the Predictions\n    and Labels\n    '''\n    \n    fpr, tpr, thresholds = metrics.roc_curve(target, predictions)\n    roc_auc = metrics.auc(fpr, tpr)\n    return roc_auc","metadata":{"execution":{"iopub.status.busy":"2024-05-10T05:29:52.457915Z","iopub.execute_input":"2024-05-10T05:29:52.458299Z","iopub.status.idle":"2024-05-10T05:29:52.464341Z","shell.execute_reply.started":"2024-05-10T05:29:52.458265Z","shell.execute_reply":"2024-05-10T05:29:52.463302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"xtrain, xvalid, ytrain, yvalid = train_test_split(train.comment_text.values, train.toxic.values, \n                                                  stratify=train.toxic.values, \n                                                  random_state=42, \n                                                  test_size=0.2, shuffle=True)","metadata":{"execution":{"iopub.status.busy":"2024-05-10T05:30:04.424080Z","iopub.execute_input":"2024-05-10T05:30:04.424451Z","iopub.status.idle":"2024-05-10T05:30:04.443346Z","shell.execute_reply.started":"2024-05-10T05:30:04.424419Z","shell.execute_reply":"2024-05-10T05:30:04.442038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Tokenizer takes all the unique words in the corpus,forms a dictionary with words as keys and their number of occurences as values,it then sorts the dictionary in descending order of counts. It then assigns the first value 1 , second value 2 and so on.\n\nWhen using neural networks, we usually feed an input into the network then take the output, compute the loss then back propagate, then reiterate with the next input. In practice, it is way more efficient to process data in batches, not one by one i.e. feed 64 inputs, and get 64 outputs. We’ll do it by using matrices [batch_size x sequence_length]. If we have variable length sequence, sequence_length correspond to the longest sequence. Also, in this case, we fill sequences with a pad token (usually 0) to fit the matrix size.","metadata":{}},{"cell_type":"code","source":"# using keras tokenizer here\ntoken = text.Tokenizer(num_words=None)\nmax_len = 1500\n\ntoken.fit_on_texts(list(xtrain) + list(xvalid))\nxtrain_seq = token.texts_to_sequences(xtrain)\nxvalid_seq = token.texts_to_sequences(xvalid)\n\n#zero pad the sequences\nxtrain_pad = sequence.pad_sequences(xtrain_seq, maxlen=max_len)\nxvalid_pad = sequence.pad_sequences(xvalid_seq, maxlen=max_len)\n\nword_index = token.word_index","metadata":{"execution":{"iopub.status.busy":"2024-05-10T05:51:18.848985Z","iopub.execute_input":"2024-05-10T05:51:18.849868Z","iopub.status.idle":"2024-05-10T05:51:20.769996Z","shell.execute_reply.started":"2024-05-10T05:51:18.849826Z","shell.execute_reply":"2024-05-10T05:51:20.768883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    model = Sequential()\n    model.add(Embedding(len(word_index) + 1,\n                     300,\n                     input_length=max_len))\n    model.add(SimpleRNN(100))\n    model.add(Dense(1, activation='sigmoid'))\n    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n    \nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2024-05-10T05:51:36.364841Z","iopub.execute_input":"2024-05-10T05:51:36.365761Z","iopub.status.idle":"2024-05-10T05:51:40.784679Z","shell.execute_reply.started":"2024-05-10T05:51:36.365722Z","shell.execute_reply":"2024-05-10T05:51:40.783718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.fit(xtrain_pad, ytrain, epochs=5, batch_size=64*strategy.num_replicas_in_sync)","metadata":{"execution":{"iopub.status.busy":"2024-05-10T05:51:55.975221Z","iopub.execute_input":"2024-05-10T05:51:55.976156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores = model.predict(xvalid_pad)\nprint(\"Auc: %.2f%%\" % (roc_auc(scores,yvalid)))","metadata":{"execution":{"iopub.status.busy":"2024-05-10T05:51:44.075789Z","iopub.execute_input":"2024-05-10T05:51:44.076223Z","iopub.status.idle":"2024-05-10T05:51:53.895177Z","shell.execute_reply.started":"2024-05-10T05:51:44.076188Z","shell.execute_reply":"2024-05-10T05:51:53.894002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores_model = []\nscores_model.append({'Model': 'SimpleRNN','AUC_Score': roc_auc(scores,yvalid)})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Code Explanantion\n* Tokenization<br><br>\n So if you have watched the videos and referred to the links, you would know that in an RNN we input a sentence word by word. We represent every word as one hot vectors of dimensions : Numbers of words in Vocab +1. <br>\n  What keras Tokenizer does is , it takes all the unique words in the corpus,forms a dictionary with words as keys and their number of occurences as values,it then sorts the dictionary in descending order of counts. It then assigns the first value 1 , second value 2 and so on. So let's suppose word 'the' occured the most in the corpus then it will assigned index 1 and vector representing 'the' would be a one-hot vector with value 1 at position 1 and rest zereos.<br>\n  Try printing first 2 elements of xtrain_seq you will see every word is represented as a digit now","metadata":{}},{"cell_type":"code","source":"xtrain_seq[:1]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<b>Now you might be wondering What is padding? Why its done</b><br><br>\n\nHere is the answer :\n* https://www.quora.com/Which-effect-does-sequence-padding-have-on-the-training-of-a-neural-network\n* https://machinelearningmastery.com/data-preparation-variable-length-input-sequences-sequence-prediction/\n* https://www.coursera.org/lecture/natural-language-processing-tensorflow/padding-2Cyzs\n\nAlso sometimes people might use special tokens while tokenizing like EOS(end of string) and BOS(Begining of string). Here is the reason why it's done\n* https://stackoverflow.com/questions/44579161/why-do-we-do-padding-in-nlp-tasks\n\n\nThe code token.word_index simply gives the dictionary of vocab that keras created for us","metadata":{}},{"cell_type":"markdown","source":"* Building the Neural Network\n\nTo understand the Dimensions of input and output given to RNN in keras her is a beautiful article : https://medium.com/@shivajbd/understanding-input-and-output-shape-in-lstm-keras-c501ee95c65e\n\nThe first line model.Sequential() tells keras that we will be building our network sequentially . Then we first add the Embedding layer.\nEmbedding layer is also a layer of neurons which takes in as input the nth dimensional one hot vector of every word and converts it into 300 dimensional vector , it gives us word embeddings similar to word2vec. We could have used word2vec but the embeddings layer learns during training to enhance the embeddings.\nNext we add an 100 LSTM units without any dropout or regularization\nAt last we add a single neuron with sigmoid function which takes output from 100 LSTM cells (Please note we have 100 LSTM cells not layers) to predict the results and then we compile the model using adam optimizer \n\n* Comments on the model<br><br>\nWe can see our model achieves an accuracy of 1 which is just insane , we are clearly overfitting I know , but this was the simplest model of all ,we can tune a lot of hyperparameters like RNN units, we can do batch normalization , dropouts etc to get better result. The point is we got an AUC score of 0.82 without much efforts and we know have learnt about RNN's .Deep learning is really revolutionary","metadata":{}},{"cell_type":"markdown","source":"# Word Embeddings\n\nWhile building our simple RNN models we talked about using word-embeddings , So what is word-embeddings and how do we get word-embeddings?\nHere is the answer :\n* https://www.coursera.org/learn/nlp-sequence-models/lecture/6Oq70/word-representation\n* https://machinelearningmastery.com/what-are-word-embeddings/\n<br> <br>\nThe latest approach to getting word Embeddings is using pretained GLoVe or using Fasttext. Without going into too much details, I would explain how to create sentence vectors and how can we use them to create a machine learning model on top of it and since I am a fan of GloVe vectors, word2vec and fasttext. In this Notebook, I'll be using the GloVe vectors. You can download the GloVe vectors from here http://www-nlp.stanford.edu/data/glove.840B.300d.zip or you can search for GloVe in datasets on Kaggle and add the file","metadata":{}},{"cell_type":"code","source":"# load the GloVe vectors in a dictionary:\n\nembeddings_index = {}\nf = open('/kaggle/input/glove840b300dtxt/glove.840B.300d.txt','r',encoding='utf-8')\nfor line in tqdm(f):\n    values = line.split(' ')\n    word = values[0]\n    coefs = np.asarray([float(val) for val in values[1:]])\n    embeddings_index[word] = coefs\nf.close()\n\nprint('Found %s word vectors.' % len(embeddings_index))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# LSTM's\n\n## Basic Overview\n\nSimple RNN's were certainly better than classical ML algorithms and gave state of the art results, but it failed to capture long term dependencies that is present in sentences . So in 1998-99 LSTM's were introduced to counter to these drawbacks.\n\n## In Depth Understanding\n\nWhy LSTM's?\n* https://www.coursera.org/learn/nlp-sequence-models/lecture/PKMRR/vanishing-gradients-with-rnns\n* https://www.analyticsvidhya.com/blog/2017/12/fundamentals-of-deep-learning-introduction-to-lstm/\n\nWhat are LSTM's?\n* https://www.coursera.org/learn/nlp-sequence-models/lecture/KXoay/long-short-term-memory-lstm\n* https://distill.pub/2019/memorization-in-rnns/\n* https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21\n\n# Code Implementation\n\nWe have already tokenized and paded our text for input to LSTM's","metadata":{}},{"cell_type":"code","source":"# create an embedding matrix for the words we have in the dataset\nembedding_matrix = np.zeros((len(word_index) + 1, 300))\nfor word, i in tqdm(word_index.items()):\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None:\n        embedding_matrix[i] = embedding_vector","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nwith strategy.scope():\n    \n    # A simple LSTM with glove embeddings and one dense layer\n    model = Sequential()\n    model.add(Embedding(len(word_index) + 1,\n                     300,\n                     weights=[embedding_matrix],\n                     input_length=max_len,\n                     trainable=False))\n\n    model.add(LSTM(100, dropout=0.3, recurrent_dropout=0.3))\n    model.add(Dense(1, activation='sigmoid'))\n    model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])\n    \nmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.fit(xtrain_pad, ytrain, nb_epoch=5, batch_size=64*strategy.num_replicas_in_sync)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores = model.predict(xvalid_pad)\nprint(\"Auc: %.2f%%\" % (roc_auc(scores,yvalid)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores_model.append({'Model': 'LSTM','AUC_Score': roc_auc(scores,yvalid)})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Code Explanation\n\nAs a first step we calculate embedding matrix for our vocabulary from the pretrained GLoVe vectors . Then while building the embedding layer we pass Embedding Matrix as weights to the layer instead of training it over Vocabulary and thus we pass trainable = False.\nRest of the model is same as before except we have replaced the SimpleRNN By LSTM Units\n\n* Comments on the Model\n\nWe now see that the model is not overfitting and achieves an auc score of 0.96 which is quite commendable , also we close in on the gap between accuracy and auc .\nWe see that in this case we used dropout and prevented overfitting the data","metadata":{}},{"cell_type":"markdown","source":"# GRU's\n\n## Basic  Overview\n\nIntroduced by Cho, et al. in 2014, GRU (Gated Recurrent Unit) aims to solve the vanishing gradient problem which comes with a standard recurrent neural network. GRU's are a variation on the LSTM because both are designed similarly and, in some cases, produce equally excellent results . GRU's were designed to be simpler and faster than LSTM's and in most cases produce equally good results and thus there is no clear winner.\n\n## In Depth Explanation\n\n* https://towardsdatascience.com/understanding-gru-networks-2ef37df6c9be\n* https://www.coursera.org/learn/nlp-sequence-models/lecture/agZiL/gated-recurrent-unit-gru\n* https://www.geeksforgeeks.org/gated-recurrent-unit-networks/\n\n## Code Implementation","metadata":{}},{"cell_type":"code","source":"%%time\nwith strategy.scope():\n    # GRU with glove embeddings and two dense layers\n     model = Sequential()\n     model.add(Embedding(len(word_index) + 1,\n                     300,\n                     weights=[embedding_matrix],\n                     input_length=max_len,\n                     trainable=False))\n     model.add(SpatialDropout1D(0.3))\n     model.add(GRU(300))\n     model.add(Dense(1, activation='sigmoid'))\n\n     model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])   \n    \nmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.fit(xtrain_pad, ytrain, nb_epoch=5, batch_size=64*strategy.num_replicas_in_sync)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores = model.predict(xvalid_pad)\nprint(\"Auc: %.2f%%\" % (roc_auc(scores,yvalid)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores_model.append({'Model': 'GRU','AUC_Score': roc_auc(scores,yvalid)})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores_model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Bi-Directional RNN's\n\n## In Depth Explanation\n\n* https://www.coursera.org/learn/nlp-sequence-models/lecture/fyXnn/bidirectional-rnn\n* https://towardsdatascience.com/understanding-bidirectional-rnn-in-pytorch-5bd25a5dd66\n* https://d2l.ai/chapter_recurrent-modern/bi-rnn.html\n\n## Code Implementation","metadata":{}},{"cell_type":"code","source":"%%time\nwith strategy.scope():\n    # A simple bidirectional LSTM with glove embeddings and one dense layer\n    model = Sequential()\n    model.add(Embedding(len(word_index) + 1,\n                     300,\n                     weights=[embedding_matrix],\n                     input_length=max_len,\n                     trainable=False))\n    model.add(Bidirectional(LSTM(300, dropout=0.3, recurrent_dropout=0.3)))\n\n    model.add(Dense(1,activation='sigmoid'))\n    model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])\n    \n    \nmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.fit(xtrain_pad, ytrain, nb_epoch=5, batch_size=64*strategy.num_replicas_in_sync)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores = model.predict(xvalid_pad)\nprint(\"Auc: %.2f%%\" % (roc_auc(scores,yvalid)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores_model.append({'Model': 'Bi-directional LSTM','AUC_Score': roc_auc(scores,yvalid)})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Code Explanation\n\nCode is same as before,only we have added bidirectional nature to the LSTM cells we used before and is self explanatory. We have achieve similar accuracy and auc score as before and now we have learned all the types of typical RNN architectures","metadata":{}},{"cell_type":"markdown","source":"**We are now at the end of part 1 of this notebook and things are about to go wild now as we Enter more complex and State of the art models .If you have followed along from the starting and read all the articles and understood everything , these complex models would be fairly easy to understand.I recommend Finishing Part 1 before continuing as the upcoming techniques can be quite overwhelming**","metadata":{}},{"cell_type":"markdown","source":"# Seq2Seq Model Architecture\n\n## Overview\n\nRNN's are of many types  and different architectures are used for different purposes. Here is a nice video explanining different types of model architectures : https://www.coursera.org/learn/nlp-sequence-models/lecture/BO8PS/different-types-of-rnns.\nSeq2Seq is a many to many RNN architecture where the input is a sequence and the output is also a sequence (where input and output sequences can be or cannot be of different lengths). This architecture is used in a lot of applications like Machine Translation, text summarization, question answering etc\n\n## In Depth Understanding\n\nI will not write the code implementation for this,but rather I will provide the resources where code has already been implemented and explained in a much better way than I could have ever explained.\n\n* https://www.coursera.org/learn/nlp-sequence-models/lecture/HyEui/basic-models ---> A basic idea of different Seq2Seq Models\n\n* https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html , https://machinelearningmastery.com/define-encoder-decoder-sequence-sequence-model-neural-machine-translation-keras/ ---> Basic Encoder-Decoder Model and its explanation respectively\n\n* https://towardsdatascience.com/how-to-implement-seq2seq-lstm-model-in-keras-shortcutnlp-6f355f3e5639 ---> A More advanced Seq2seq Model and its explanation\n\n* https://d2l.ai/chapter_recurrent-modern/machine-translation-and-dataset.html , https://d2l.ai/chapter_recurrent-modern/encoder-decoder.html ---> Implementation of Encoder-Decoder Model from scratch\n\n* https://www.youtube.com/watch?v=IfsjMg4fLWQ&list=PLtmWHNX-gukKocXQOkQjuVxglSDYWsSh9&index=8&t=0s ---> Introduction to Seq2seq By fast.ai","metadata":{}},{"cell_type":"code","source":"# Visualization of Results obtained from various Deep learning models\nresults = pd.DataFrame(scores_model).sort_values(by='AUC_Score',ascending=False)\nresults.style.background_gradient(cmap='Blues')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = go.Figure(go.Funnelarea(\n    text =results.Model,\n    values = results.AUC_Score,\n    title = {\"position\": \"top center\", \"text\": \"Funnel-Chart of Sentiment Distribution\"}\n    ))\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Attention Models\n\nThis is the toughest and most tricky part. If you are able to understand the intiuition and working of attention block , understanding transformers and transformer based architectures like BERT will be a piece of cake. This is the part where I spent the most time on and I suggest you do the same . Please read and view the following resources in the order I am providing to ignore getting confused, also at the end of this try to write and draw an attention block in your own way :-\n\n* https://www.coursera.org/learn/nlp-sequence-models/lecture/RDXpX/attention-model-intuition --> Only watch this video and not the next one\n* https://towardsdatascience.com/sequence-2-sequence-model-with-attention-mechanism-9e9ca2a613a\n* https://towardsdatascience.com/attention-and-its-different-forms-7fc3674d14dc\n* https://distill.pub/2016/augmented-rnns/ \n\n## Code Implementation\n\n* https://www.analyticsvidhya.com/blog/2019/11/comprehensive-guide-attention-mechanism-deep-learning/ --> Basic Level\n* https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html ---> Implementation from Scratch in Pytorch","metadata":{}},{"cell_type":"markdown","source":"# Transformers : Attention is all you need\n\nSo finally we have reached the end of the learning curve and are about to start learning the technology that changed NLP completely and are the reasons for the state of the art NLP techniques .Transformers were introduced in the paper Attention is all you need by Google. If you have understood the Attention models,this will be very easy , Here is transformers fully explained:\n\n* http://jalammar.github.io/illustrated-transformer/\n\n## Code Implementation\n\n* http://nlp.seas.harvard.edu/2018/04/03/attention.html ---> This presents the code implementation of the architecture presented in the paper by Google","metadata":{}},{"cell_type":"markdown","source":"# BERT and Its Implementation on this Competition\n\nAs Promised I am back with Resiurces , to understand about BERT architecture , please follow the contents in the given order :-\n\n* http://jalammar.github.io/illustrated-bert/ ---> In Depth Understanding of BERT\n\nAfter going through the post Above , I guess you must have understood how transformer architecture have been utilized by the current SOTA models . Now these architectures can be used in two ways :<br><br>\n1) We can use the model for prediction on our problems using the pretrained weights without fine-tuning or training the model for our sepcific tasks\n* EG: http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/ ---> Using Pre-trained BERT without Tuning\n\n2) We can fine-tune or train these transformer models for our task by tweaking the already pre-trained weights and training on a much smaller dataset\n* EG:* https://www.youtube.com/watch?v=hinZO--TEk4&t=2933s ---> Tuning BERT For your TASK\n\nWe will be using the first example as a base for our implementation of BERT model using Hugging Face and KERAS , but contrary to first example we will also Fine-Tune our model for our task\n\nAcknowledgements : https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras\n\n\nSteps Involved :\n* Data Preparation : Tokenization and encoding of data\n* Configuring TPU's \n* Building a Function for Model Training and adding an output layer for classification\n* Train the model and get the results","metadata":{}},{"cell_type":"code","source":"# Loading Dependencies\nimport os\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Dense, Input\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import ModelCheckpoint\nfrom kaggle_datasets import KaggleDatasets\nimport transformers\n\nfrom tokenizers import BertWordPieceTokenizer","metadata":{"execution":{"iopub.status.busy":"2024-05-09T19:32:56.587401Z","iopub.execute_input":"2024-05-09T19:32:56.587835Z","iopub.status.idle":"2024-05-09T19:32:57.178687Z","shell.execute_reply.started":"2024-05-09T19:32:56.587797Z","shell.execute_reply":"2024-05-09T19:32:57.177615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# LOADING THE DATA\n\ntrain1 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")\nvalid = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation.csv')\ntest = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv')\nsub = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2024-05-09T19:32:58.679611Z","iopub.execute_input":"2024-05-09T19:32:58.680040Z","iopub.status.idle":"2024-05-09T19:33:00.814124Z","shell.execute_reply.started":"2024-05-09T19:32:58.680000Z","shell.execute_reply":"2024-05-09T19:33:00.813021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Encoder FOr DATA for understanding waht encode batch does read documentation of hugging face tokenizer :\nhttps://huggingface.co/transformers/main_classes/tokenizer.html here","metadata":{}},{"cell_type":"code","source":"def fast_encode(texts, tokenizer, chunk_size=256, maxlen=512):\n    \"\"\"\n    Encoder for encoding the text into sequence of integers for BERT Input\n    \"\"\"\n    tokenizer.enable_truncation(max_length=maxlen)\n    tokenizer.enable_padding(length=maxlen)\n    all_ids = []\n    \n    for i in tqdm(range(0, len(texts), chunk_size)):\n        text_chunk = texts[i:i+chunk_size].tolist()\n        encs = tokenizer.encode_batch(text_chunk)\n        all_ids.extend([enc.ids for enc in encs])\n    \n    return np.array(all_ids)","metadata":{"execution":{"iopub.status.busy":"2024-05-09T19:33:22.783086Z","iopub.execute_input":"2024-05-09T19:33:22.783472Z","iopub.status.idle":"2024-05-09T19:33:22.791163Z","shell.execute_reply.started":"2024-05-09T19:33:22.783441Z","shell.execute_reply":"2024-05-09T19:33:22.789859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#IMP DATA FOR CONFIG\n\nAUTO = tf.data.experimental.AUTOTUNE\n\n\n# Configuration\nEPOCHS = 3\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nMAX_LEN = 192","metadata":{"execution":{"iopub.status.busy":"2024-05-09T19:33:23.246951Z","iopub.execute_input":"2024-05-09T19:33:23.247361Z","iopub.status.idle":"2024-05-09T19:33:23.252802Z","shell.execute_reply.started":"2024-05-09T19:33:23.247328Z","shell.execute_reply":"2024-05-09T19:33:23.251549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Tokenization\n\nFor understanding please refer to hugging face documentation again","metadata":{}},{"cell_type":"code","source":"# First load the real tokenizer\ntokenizer = transformers.DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')\n# Save the loaded tokenizer locally\ntokenizer.save_pretrained('.')\n# Reload it with the huggingface tokenizers library\nfast_tokenizer = BertWordPieceTokenizer('vocab.txt', lowercase=False)\nfast_tokenizer","metadata":{"execution":{"iopub.status.busy":"2024-05-09T19:33:24.781480Z","iopub.execute_input":"2024-05-09T19:33:24.781859Z","iopub.status.idle":"2024-05-09T19:33:25.607387Z","shell.execute_reply.started":"2024-05-09T19:33:24.781825Z","shell.execute_reply":"2024-05-09T19:33:25.606326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train = fast_encode(train1.comment_text.astype(str), fast_tokenizer, maxlen=MAX_LEN)\nx_valid = fast_encode(valid.comment_text.astype(str), fast_tokenizer, maxlen=MAX_LEN)\nx_test = fast_encode(test.content.astype(str), fast_tokenizer, maxlen=MAX_LEN)\n\ny_train = train1.toxic.values\ny_valid = valid.toxic.values","metadata":{"execution":{"iopub.status.busy":"2024-05-09T19:33:28.688287Z","iopub.execute_input":"2024-05-09T19:33:28.688694Z","iopub.status.idle":"2024-05-09T19:34:24.590927Z","shell.execute_reply.started":"2024-05-09T19:33:28.688659Z","shell.execute_reply":"2024-05-09T19:34:24.589728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train, y_train))\n    .repeat()\n    .shuffle(2048)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO)\n)\n\nvalid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_valid, y_valid))\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO)\n)\n\ntest_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices(x_test)\n    .batch(BATCH_SIZE)\n)","metadata":{"execution":{"iopub.status.busy":"2024-05-09T19:34:26.474629Z","iopub.execute_input":"2024-05-09T19:34:26.475433Z","iopub.status.idle":"2024-05-09T19:34:30.943097Z","shell.execute_reply.started":"2024-05-09T19:34:26.475391Z","shell.execute_reply":"2024-05-09T19:34:30.942126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_model(transformer, max_len=512):\n    \"\"\"\n    function for training the BERT model\n    \"\"\"\n    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n    sequence_output = transformer(input_word_ids)[0]\n    cls_token = sequence_output[:, 0, :]\n    out = Dense(1, activation='sigmoid')(cls_token)\n    \n    model = Model(inputs=input_word_ids, outputs=out)\n    model.compile(Adam(lr=1e-5), loss='binary_crossentropy', metrics=['accuracy'])\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2024-05-09T19:34:30.944830Z","iopub.execute_input":"2024-05-09T19:34:30.945855Z","iopub.status.idle":"2024-05-09T19:34:30.952820Z","shell.execute_reply.started":"2024-05-09T19:34:30.945812Z","shell.execute_reply":"2024-05-09T19:34:30.951872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Starting Training\n\nIf you want to use any another model just replace the model name in transformers._____ and use accordingly","metadata":{}},{"cell_type":"code","source":"%%time\nwith strategy.scope():\n    transformer_layer = (\n        transformers.TFDistilBertModel\n        .from_pretrained('distilbert-base-multilingual-cased')\n    )\n    model = build_model(transformer_layer, max_len=MAX_LEN)\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2024-05-09T19:34:30.959604Z","iopub.execute_input":"2024-05-09T19:34:30.959904Z","iopub.status.idle":"2024-05-09T19:35:09.230893Z","shell.execute_reply.started":"2024-05-09T19:34:30.959875Z","shell.execute_reply":"2024-05-09T19:35:09.229844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_steps = x_train.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n    train_dataset,\n    steps_per_epoch=n_steps,\n    validation_data=valid_dataset,\n    epochs=EPOCHS\n)","metadata":{"execution":{"iopub.status.busy":"2024-05-09T19:35:09.448742Z","iopub.execute_input":"2024-05-09T19:35:09.449519Z","iopub.status.idle":"2024-05-09T19:37:05.476745Z","shell.execute_reply.started":"2024-05-09T19:35:09.449480Z","shell.execute_reply":"2024-05-09T19:37:05.474900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_steps = x_valid.shape[0] // BATCH_SIZE\ntrain_history_2 = model.fit(\n    valid_dataset.repeat(),\n    steps_per_epoch=n_steps,\n    epochs=EPOCHS*2\n)","metadata":{"execution":{"iopub.status.busy":"2024-05-09T19:37:05.477776Z","iopub.status.idle":"2024-05-09T19:37:05.478401Z","shell.execute_reply.started":"2024-05-09T19:37:05.478111Z","shell.execute_reply":"2024-05-09T19:37:05.478139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub['toxic'] = model.predict(test_dataset, verbose=1)\nsub.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# End Notes\n\nThis was my effort to share my learnings so that everyone can benifit from it.As this community has been very kind to me and helped me in learning all of this , I want to take this forward. I have shared all the resources I used to learn all the stuff .Join me and make these NLP competitions your first ,without being overwhelmed by the shear number of techniques used . It took me 10 days to learn all of this , you can learn it at your pace and dont give in , at the end of all this you will be a different person and it will all be worth it.\n\n\n### I am attaching more resources if you want NLP end to end:\n\n1) Books\n\n* https://d2l.ai/\n* Jason Brownlee's Books\n\n2) Courses\n\n* https://www.coursera.org/learn/nlp-sequence-models/home/welcome\n* Fast.ai NLP Course\n\n3) Blogs and websites\n\n* Machine Learning Mastery\n* https://distill.pub/\n* http://jalammar.github.io/\n\n**<span style=\"color:Red\">This is subtle effort of contributing towards the community, if it helped you in any way please show a token of love by upvoting**","metadata":{}}]}