{"cells":[{"metadata":{},"cell_type":"markdown","source":"# NOTE:\n**This Notebook is not written for competetive purpose but just to inform the beginners about the NLP aspect using Deep Learning in PyTorch so please READ EVERY LINE AND COMMENT CAREFULLY**\n\nThis is the first of Multi Part series on NLP. I'll be discussing about the aspects of Natural Language Processing in details in this part and perform very basic task using Deep Learning library PyTorch. If you are not familiar with Deep Learning or PyTorch, please [check this tutorial on PyTorch](https://www.kaggle.com/deshwalmahesh/pytorch-detailed-tutorial-for-beginners-using-cnn). It'll teach you about aspects of NLP, classification of texts using Deep Learning Models, Embeddings and so on. We'll be using the Toxic Comment Dataset for this task. Our objective is to find the sentiments given in the comments whether they are toxic or not so it is a binary classification problem. We'll be using PyTorch for our building a very simple model.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# NLP?\nNLP or NAtural Language Processing is a part of AI which deals with the textual aspect of the data. We can perform different task using NLP some of the simplest being:\n\n1. Classify the kind of text data. (Science,Sports etc)\n2. Extract the summary from the text (useful in the key points for legal documents)\n3. Make a Fill in the Blanks Model\n4. ChatBots\n5. Image Captioning\n\nNLP has reached a point of complexity where the models are generating scripts for a play or generate a random story based on a random word.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Problem?\nWe have to classify the given comments in 2 parts whether these are toxis or not by using the comments. 1 shows Toxic, 0 shows Not Toxic","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Solution?\n## RNN (Recurrent Neural Networks):\nI can not draw the figure for weights,biases and all the other things that are very essential to have a deep understanding of RNN so Please check out [this insightful video](https://www.youtube.com/watch?v=2E65LDnM2cA&list=PL1F3ABbhcqa3BBWo170U4Ev2wfsF7FN8l) on RNN by Andrew NG and a very [intutive video by MIT](https://www.youtube.com/watch?v=SEnXr6v2ifU&t=1398s) before you start.\n\n**RNN helps wherever we need context from the previous input such as:**\n\n1. Language to Language Translation (Many to Many Articture: Many inputs Many Outputs)\n2. Sentimeny Analysis (Many to One Articture: Many inputs One Output)\n3. Music, Story Generation (One to Many Articture: One input Many Outputs)\n\n**We're talking about the `Many to Many Architecture Below`**\n\nRNN are specialised version of Neural Networks which deals with the time aspect as well. For instance, given a sentence like `Because Eminem is the only Rapper to win an Oscar, best selling rapper of all time, most top Billboards and has got 15 Grammy award wins so it makes him the ____` can have a complex structure because filling the right word as `GOD/GOAT/Best` is linked to the first word itself and has a relation with the entire sequence of the words. Simple Dense networds can not address the problem so to get the results in the sentence, we have to get the relation from the past where the output is based on `only,rapper,win,Oscar,most,selling,all,time,most,top,billboards,15,Grammy`. So in order to teach the model, RNN are used where an output is based only not only on the inputs but the previous outputs too.\n\nRNN have **BPTT**: **Back Propagation Through Time** which is a specialised version of Back Propagation Algorithm to train RNN. \n\nConceptually, BPTT works by unrolling all input timesteps. Each timestep has one input timestep, one copy of the network, and one output. Errors are then calculated and accumulated for each timestep. The network is rolled back up and the weights are updated.\n\nGradients will have to be called for the past cells too so if layers are in thousands so BPTT can be computationally expensive as the number of timesteps increases and it also leads to Vanishing gradients (no change in Gradients) or exploding (gradients changing too fast). To learn more about the Gradients and Back Propagation check out [this video](https://www.youtube.com/watch?v=IHZwWFHWa-w&t=1060s). \n\n## Working and feeding data into RNN for NLP?\nUnlike the data that we feed in simple DNN, we feed the data at time intervals. Let us see by an example.\nSuppose we have the Sentence `Eminem is the GOAT` and the vector representation of it suppose `[1,2,3,4]` and we want to train a super slow but intelligent model model of just 1 cell or layer in RNN `(t-1 only)`. So insted of feeding the whole instance at 12:00 PM, it'll feed starting hidden state as `h_start` (which is generally random or 0) and `1` which is notation for `Eminem` at 12:00 PM and generate an output say `99` and a hidden state `h0` at 12:01. At 12:02, We'll feed the model with past state `h0` and the current input `2` to get a result `42` and a modified hidden state from `h0 -> h1` at 12:03 and it goes on until the end of the vector. By the time our model finish training, it would have produced a vector of results and a final state `h_end`(which is of no use now except we are using Encoder-Decoder) which interprets to `He is indeed GOD. Period`. \n\n**NOTE: RNN uses SAME function (mostly`tanh`) as activation function and SAME weight matrix `W` to calculate the variables at EACH TIME STEP. Depending on the requirements, there can be multiple Weight Matrices for example one weight matrix (`Wh`) that gets multiplied by the previous hidden state (`h_old`), another one as (`Wx`) that gets multiplied by input (`x_in`).`h_now = tanh(Wh*h_old + Wx*x_in)`. To calculate the current output as (`y_now`), we use another weight matrix (`Wy`) so that `y_now = softmax(Wy * h_now)`**\n\n**Note: Hidden State is not a scaler but has dimension `d`.Hidden state represent how many different features you want to remember for either short term or long term (in LSTM's context). Hidden States tend to do something like this:**\n\n1. Remember the context\n2. Remeber the Grammar\n3. Preserve the style words have been used\n4. etc... etc...\n\n**Data should be of same length for all of the Training instances for most of the basic Deep Learning models (except sequence-to-seqence, attention etc)**. You can not pass in Vectors of different length to the model such as `Eminem is the best rapper` and `Eminem is the best rapper born EOD`. You need to either truncate the second one or to pad the first one. People mostly use the padding where the sentence with the maximum length is considered as base and a unique vector is appended to the each sentence in the end to mae the vectors of same size. By doing that, our second sentence will become `Eminem is the best rapper -PAD- PAD-` and it'll become `[1,2,3,4,777,777]`.\n\nThere might be times when your model will encounter some words that it is unaware of. For example we have the sentence `But Kanye has more Grammys. Now What?`. Our model will literally throw an error like `Kanye?? WHO???? I don't know that name. Never Heard of this word`. So to overcome the problem, another unidentified token is attached as -UN- to make the sentence as `But -UN- has more Grammys` and the result for that would be `For Producing Music mostly not Rapping, got it?`.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Imports\nImport the required libraries. With Kaggle, almost everything is given but if you need some extra, you can use `! pip install library_name` and then import it. In order to download in-built data for libraries, **Turn on the Internet**","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pandas as pd # data handling\nimport numpy as np # base of all\nimport matplotlib.pyplot as plt # plotting\nimport seaborn as sns # advance plotting\nfrom wordcloud import WordCloud # to see the words as image\nimport torch # PyTorch for building Networks\nfrom torchtext.data import Field,LabelField,BucketIterator,TabularDataset # TorchText has Text processing Function\nfrom torchtext import vocab\nfrom sklearn.model_selection import train_test_split # split the data into training and testing\nfrom sklearn.metrics import accuracy_score # accuracy metric\nfrom nltk import word_tokenize # very popular Text processing Library\nimport random # to perform randomisation of tasks\nfrom tqdm.notebook import tqdm # for a continuous progress bar style\nimport time # time module \nimport os # import operating system","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Set Defaults & Globals\nSet default conditions for some libraries and Global Variables","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"SEED = 13 # reproducible results: Same results in every run\nIN_PATH = '/kaggle/input/'\nDATA_PATH = IN_PATH+'jigsaw-multilingual-toxic-comment-classification/' # input directorypath\nOUT_PATH = '/kaggle/working/' # path for output directory \nGLOVE_TEXT_PATH = 'glove6b100dtxt/glove.6B.100d.txt' # glove directory\nEPOCH = 3 # number of epochs to run for model\n\nnp.random.seed(SEED) \ntorch.manual_seed(SEED)\ntorch.cuda.manual_seed(SEED)\ntorch.backends.cudnn.deterministic = True # cuda algorithms\nos.environ['PYTHONHASHSEED'] = str(SEED)\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # use 'cuda' if available else 'cpu'\n\nplt.style.use('seaborn') # use seaborn style plotting\n\n# nltk.download('punkt') # if using first time. Kaggle has all the things already downloaded","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Input & EDA\nCheck out the data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv(DATA_PATH+'jigsaw-toxic-comment-train.csv')\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['toxic'].value_counts().plot(kind='pie',autopct='%.2f%%',labels=['Not Toxic','Toxic'],cmap='Set2') \n# distribution of Toxic or Non Toxic. 1 detemines Toxic \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"text = \" \".join(comment for comment in df.comment_text) # join all the comments to make a new one big comment\nprint(f'There are {len(text)} unique words in the whole dataset')\n\n# show the words as an image. It can take lots of time due to large number of words so I am commenting it out\nwordcloud = '''WordCloud(max_words=150,background_color=\"white\").generate(text)\n\nfig = plt.figure(figsize=(15,7))\nplt.imshow(wordcloud, interpolation='bilinear')\nplt.axis(\"off\")\nplt.show()'''","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Pre- Processing\nRemove Null Values, Duplicates etc . Split the data, and change the structure according to the requiements of PyTorch\n\n## Remove un-necessary values\nRemove unwanted columns, Null rows and duplicated rows","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df.loc[:,['comment_text','toxic']] # we need just the two columns\ndf.drop_duplicates(subset=['comment_text'],inplace=True) # drop duplicates and it'll include empty comments too\ndf.dropna(subset=['comment_text','toxic'],inplace=True) # we do not need any of the columns with empty values\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Split Data\nSplit Data into 3-Fold Training, Validation and tsting dataframes and save to memory for later use.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df,test_df = train_test_split(df,test_size=0.25,random_state=SEED,stratify=df['toxic'])\n# stratify tries to split in a manner that distribution of 'toxic' is same in both train and test\n\ntrain_df,val_df = train_test_split(df,test_size=0.20,random_state=SEED,stratify=df['toxic'])\n\ntrain_df.reset_index(drop=True),val_df.reset_index(drop=True), test_df.reset_index(drop=True)\n# split the data while preserving the type of the data. It preserves the original Index so you need to reset\nprint(f'train_df is of type {type(train_df,).__name__} and is having a shape {train_df.shape}')\n\n# save the dataframes so that we can directly use those from disk by using PyTorch's modules\ntrain_df.to_csv(OUT_PATH+'train.csv',index=False)\nval_df.to_csv(OUT_PATH+'val.csv',index=False)\ntest_df.to_csv(OUT_PATH+'test.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Changing Structure\nChange the structure of data according to PyTorch's requirements","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"text_field = Field(tokenize=word_tokenize)\n# tokenize text using word_tokenize and convert to numerical form using default parameters\n\nlabel_field = LabelField(dtype=torch.float) \n# useful for label string to LabelEncoding. Not useful here but doesn't hurt either\n\nfields = [('comment_text',text_field),('toxic',label_field)] \n# (column name,field object to use on that column) pair for the dictonary\n\ntrain, val, test = TabularDataset.splits(path=OUT_PATH, train='train.csv',validation='val.csv',test='test.csv', \n format='csv',skip_header=True,fields=fields)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let us explore that what exactly is in the `train`, `test`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Type of \"train:\" {type(train)}\\n Length of \"train\": {len(train)}\\n' )\ni = random.randint(0,len(train)) # generate a random index within the lenth of train\nprint(f'Keys at index {i} of \"train\": {train[i].__dict__.keys()}\\n')\nprint(\"Contents at random index:\\n\",vars(train.examples[i])) \n# vars is used to see the whole dictonary when the classes or modules have __dict__() used","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build Vocab \nout of 85000000 words, we choose to have just 100000 `UNIQUE` words to build a vocab. Words that are not a part of our vocab will be represented by specialised token, let's assume ``. There will be 1 extra token apart from `` and that'll be padding token `` .","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"text_field.build_vocab(train,max_size=100000) \nlabel_field.build_vocab(train) \n\n# words are stored as integers withn the vocab for internal data structure handling. \n# let us look at the {'word':respective_integer} of first 15 \n\nprint({k: text_field.vocab.stoi[k] for k in list(text_field.vocab.stoi)[:15]}) \n# this is just pure python code to get first N elements from a a dictonary, as a dictonary","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Most common 15 words in the vocab are: {text_field.vocab.freqs.most_common(15)}\")\n# integers i int the second part of each tuples are the frequencies of words in the vocab. They show that how many\n# number of times this specific word has apprered in the whole training data set","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## BucketIterator\n`BucketIterator` is very useful for text processing because it returns the batch of texts which have almost same length. As we know that while calculating gradients during the BPTT, the gradients are calculated to the whole length of text and because we always calculate the gradients on the batches but not on an individual example to speed up the process, we'll have to pad the short length texts. So the `BatchIterator` minimizes the hassle by making a batch which has all of the texts of same length more or less.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_iter, val_iter, test_iter = BucketIterator.splits((train,val,test), batch_sizes=(32,128,128),\n sort_key=lambda x: len(x.comment_text),\n sort_within_batch=False,\n device=device) # use the cuda device if available","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Build Model\nWe'll build a class based model. Bes thing about PyTorch is that first of al it Dynamic in nature and scondly it is very close to the Neural Networks as you have to pass in the in,out and all the parameters. In case you are looking at the `PyTorch` the very first time, I'll highly recommend you to go and [check this notebook](https://www.kaggle.com/deshwalmahesh/pytorch-detailed-tutorial-for-beginners-using-cnn) which is dedicated towards the working of `PyTorch` and provides in depth knowledge.\n\nI hope you have been through the tutorials and know about the all the layers and structure. One thing that the notebook does not cover in that tutorial is `Embedding Layer`.\n\n**From the `PyTorch's` documentation:** \n\nA simple lookup table that stores embeddings of a fixed dictionary and size.\n \nThis module is often used to store word embeddings and retrieve them using indices. The input to the module is a list of indices, and the output is the corresponding word embeddings.\n\nBefore we want to give you a definition of that, let us look at a scenario:\n\nWhat if I ask you that what is common between `Eminem` and `Kanye`? Those who have something to do with Music will describe the features as the most accurately as `Rappers`. Those who have no knowledge of those but know there are English names, they'll describe that `Male English Names`. Those with just basic names will describe `English Words`. So like this, we have 3 features in our hand for 2 different attributes. What if I ask that what is difference between a `plastic bag` and `6ix9ine`??? You have no idea what that can be. You can't tell that both tend to have `garbage mostly` until yov've seen the workings of both. This is exactly what we call `Embeddings` or `Latent Features`. Our Model can generate `K` new features from the existing `N` features in the data so that the model can learn from the hidden features insted of those original features. So if we train on the `lyrics of 6ix9ne`, `essays about garbage`, `Eminem biography` etc etc, it'll tend to put the first 2 in the category of `trash` and the last one as `legend` even if the computer has no knowledge about what just happed. I Guess my model is more intelligent and has better music taste than many people out there ;)\n\n**Word embeddings are a type of word representation that allows words with similar meaning to have a similar representation.**\n\n**NOTE: While looking at the structure, please ignore the `LSTM` for now. Structure is almost same for both `RNN` and `LSTM` in `PyTorch` but working is different. To save the hassle of re-designing the network, I have used `if-else` statement. Later, I'll be talking about the LSTM too.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Network(torch.nn.Module):\n '''\n It inherits the functionality of Module class from torch.nn whic includes al the layers, weights, grads setup\n and methods to calculate the same. We just need to put in the required layers and describe the flows as\n which layers comes after which one\n '''\n \n def __init__(self,in_neuron,embedding_dim=128,hidden_size=256,out_neuron=1,m_type='rnn',drop=0.53,**kwargs):\n '''\n Constructor of the class which will instantiate the layers while initialisation.\n \n NOTE: Order of the layer defined here has nothing to do wit hthe working. Just like we can define Drouout()\n layer anywhere in ithe init() but actual working depends on the forward() method as well as the input\n and output shapes. You should be aware of the in,out shapes as the mismatch can produce error.\n \n args:\n in_neuron: input dimensions of the first layer {int}\n embedding_dim: number of latent features you want to calculate from the input data {int} default=128\n hidden_size: neurons you want to have in your hidden RNN layer {int} default=256\n out_neuron: number of outputs you want to have at the end.{int} default=1\n model: whether to use 'rnn' or 'lstm' {string} \n drop: proportion of values to dropout from the previous values randomly {float 0-1} default=0.53\n **kwargs: any torch.nn.RNN or torch.nn.LSTM args given m_type='rnn' or'lstm' {dict}\n out: \n return a tensor of shape {batch,out_neuron} as output \n '''\n super(Network,self).__init__() # call the constructor of Base Class. To know more, please visit the link\n # https://www.programiz.com/python-programming/methods/built-in/super\n self.m_type = m_type\n \n self.embedding = torch.nn.Embedding(in_neuron,embedding_dim) # embedding layer is always the first layer\n \n if self.m_type == 'lstm':\n # whether to use the LSTM type model or the RNN type model. It'll use only 1 in forward()\n self.lstm = torch.nn.LSTM(embedding_dim,hidden_size,**kwargs)\n else:\n self.rnn = torch.nn.RNN(embedding_dim,hidden_size,**kwargs) \n \n self.dropout = torch.nn.Dropout(drop) # drop the values by random which comes from previous layer\n \n self.dense = torch.nn.Linear(hidden_size,out_neuron) # last fully connected layer\n \n \n def forward(self,t):\n '''\n Activate the forward propagation of a batch at a time to transform the input bath of tensors through\n the different layers to get an out which then will be compared to original label for computing loss.\n args:\n t: tensors in the form of a batch {torch.tensor}\n '''\n # Step:1 pass the incoming tensor to the first layer to get embeddings\n embedding_t = self.embedding(t) # usually we replace the same tensor as t = self.layer(t)\n # input is a \"list\" sentences where each sentence is a vector of \"encoded words\" \n # out.shape = [sentence_length,batch_size,embedding_dimension], in.shape = [sentence_length,batch_size]\n \n # Step 2: Apply dropout\n drop_emb = self.dropout(embedding_t)\n \n # Step 3: Get hidden state and output. It'll use either LSTM or RNN\n if self.m_type == 'lstm':\n out, (hidden_state,_) = self.lstm(drop_emb)\n else:\n out, hidden_state = self.rnn(drop_emb)\n # shape of rnn_out = (seq_len, batch, num_directions * hidden_size)\n \n # Step 4: Remove the extra axis from Hidden State \n hidden_squeezed = hidden_state.squeeze(0) \n # shape of hidden_state = (num_layers * num_directions, batch, hidden_size) = (1*1,b,h) so extra 1 layer\n \n # Step 5: Assert to check. if failed, AssertionError error will be thrown\n assert torch.equal(out[-1,:,:],hidden_squeezed)\n # out_rnn is concatenation of hidden states so squeezed hidden and last value of out_rnn should be equal\n \n # Step 6: Pass the \"last\" hidden state only because we only want 1 output based on the last hidden state\n return self.dense(hidden_squeezed) # these are not the probabilities. We still need to use an activation ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_network(network,train_iter,optimizer,loss_fn,epoch_num):\n '''\n train the network using given parameters\n args:\n network: any Neural Network object \n train_batch: iterator of training data\n optimizer: optimizer for gradients calculation and updation\n loss_fn: appropriate loss function\n epoch_num = Epoch number so that it can show which epoch number in tqdm Bar\n out:\n a tuple of (average_loss,average_accuracy) of floating values for a single epoch\n '''\n epoch_loss = 0 # loss per epoch\n epoch_acc = 0 # accuracy per epoch\n \n network.train() # set the model in training mode as it requires gradients calculation and updtion\n # turn off while testing using model.eval() and torch.no_grad() block\n \n for batch in tqdm(train_iter,f\"Epoch: {epoch_num}\"): \n # data will be shown to model in batches per epoch to calculate gradients per batch\n \n optimizer.zero_grad() # clear all the calculated grdients from previous step\n \n predictions = network(batch.comment_text).squeeze(1) # squeeze out the extra dimension [batch_size,1]\n \n loss = loss_fn(predictions,batch.toxic) # calculate loss on the whole batch\n \n pred_classes = torch.round(torch.sigmoid(predictions))\n # sigmoid will convert each output value (which is a single float value for each sentence in batch) to\n # to probability between {0,1}. round is nothing but setting the threshold at 0.5 that if probability \n # is greater than 0.5, it belongs to one class and if it is less than 0.5, it belongs to other\n \n correct_preds = (pred_classes == batch.toxic).float()\n # get a floating tensors of predicted classes which match original true class \n \n accuracy = correct_preds.sum()/len(correct_preds)# it'll be a tensor of shape [1,]\n \n # below two are must and should be used only after calculation of Loss by optimizer\n loss.backward() # Start Back Propagation so that model can calculate gradients based on loss\n optimizer.step() # update the weights based on gradient corresponding to each neuron\n \n epoch_loss += loss.item() # add the loss for this batch to calculate the loss for whole epoch\n epoch_acc += accuracy.item() # .item() tend to give the exact number from the tensor of shape [1,]\n \n \n time.sleep(0.001) # for tqdm progess bar\n \n return epoch_loss/len(train_iter), epoch_acc/len(train_iter)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def evaluate_network(network,val_test_iter,optimizer,loss_fn):\n '''\n evaluate the network using given parameters\n args:\n network: any Neural Network object \n val_test_iter: iterator of validation/test data\n optimizer: optimizer for gradients calculation and updation\n loss_fn: appropriate loss function\n out:\n a tuple of (average_loss,average_accuracy) of floating values for the incoming dataset\n '''\n total_loss = 0 # total loss for the whole incoming data\n total_acc = 0 # total accuracy for the whole data\n \n network.eval() # set the model in evaluation mode to not compute gradients and reduce overhead\n \n with torch.no_grad(): # turn of gradients calculation \n \n for batch in val_test_iter:\n\n predictions = network(batch.comment_text).squeeze(1)\n\n loss = loss_fn(predictions,batch.toxic)\n\n pred_classes = torch.round(torch.sigmoid(predictions))\n\n correct_preds = (pred_classes == batch.toxic).float()\n\n accuracy = correct_preds.sum()/len(correct_preds)\n\n total_loss += loss.item() \n total_acc += accuracy.item()\n\n return total_loss/len(val_test_iter), total_acc/len(val_test_iter)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## RNN\nWe'll train our RNN First","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"in_neuron = len(text_field.vocab)\nlr = 3e-4 # learning rate = 0.0003\n\nnetwork = Network(in_neuron) # instantiate the RNN object. other parameters remain default\nif torch.cuda.is_available():\n network.cuda() # activate GPU spport\n\noptimizer = torch.optim.Adam(network.parameters(),lr=lr) # use Adam Optimizer\nloss_fn = torch.nn.BCEWithLogitsLoss() # Sigmoid activation with Binary Cross Entropy loss. This is more \n# numerically stable than using a plain Sigmoid followed by a BCELoss as, by combining the operations into one \n# layer,we take advantage of the log-sum-exp trick for numerical stability\n\nfor epoch in range(EPOCH):\n train_loss, train_acc = train_network(network,train_iter,optimizer,loss_fn,epoch+1)\n val_loss,val_acc = evaluate_network(network,val_iter,optimizer,loss_fn)\n tqdm.write(f'''End of Epoch: {epoch+1} | Train Loss: {train_loss:.3f} | Val Loss: {val_loss:.3f} | Train Acc: {train_acc*100:.2f}% | Val Acc: {val_acc*100:.2f}%''')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LSTM\nLSTM is a specialised version of RNN which has 4 Gates. Instead of updating the states by using the the previous values in RNN, these 4 gates together do the following tasks:\n1. Which Information is valuable\n2. Which information we have to keep \n3. Which information we have to forget\n4. Which information will be used to update the state\n5. Which information has become obsolete and we have to forget it\n\nI can neither draw nor explain the whole workings here so I **HIGHLY** recommend you to watch these 2 very awesome and user friendly videos.\n1. [Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM)](https://www.youtube.com/watch?v=WCUNPb-5EYI&t=660s) by Brendon Rohrer\n2. [Illustrated Guide to LSTM's and GRU's: A step by step explanation](https://www.youtube.com/watch?v=8HyCNIVRbSU) by Michael Phi\n\nIf you have seen those, you can train out network by changing just one parameter.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"network = Network(in_neuron,m_type='lstm') \n\nif torch.cuda.is_available():\n network.cuda() # activate GPU spport\n \n# optimizer and losses remains the same\n\nfor epoch in range(EPOCH):\n train_loss, train_acc = train_network(network,train_iter,optimizer,loss_fn,epoch+1)\n val_loss,val_acc = evaluate_network(network,val_iter,optimizer,loss_fn)\n tqdm.write(f'''End of Epoch: {epoch+1} | Train Loss: {train_loss:.3f} | Val Loss: {val_loss:.3f} | Train Acc: {train_acc*100:.2f}% | Val Acc: {val_acc*100:.2f}%''')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Bidirectional RNN/ LSTM\nThese are specialised version of RNN which not only look at the past but the future too. Means instead of looking at 1 word at time, they look at two words but in opposite direction. Don't worry, let us try to see with an example. Suppose we have sentence `Shady Records not shady plans`. In the birdirectional structure, we'll ave 2 `pipelines` (RNN Cells) working in different direction. First will be like a simple RNN reading words `H0` + `Shady` -> `H1` -> `H1`+ Records -> `H2` and so on .... In the same time, there will be another RNN cells working backwards in the same way.. They'll be Doing `H_0` + `plans` -> `H_1` -> `H_1` + `shady` -> `H_2` and so on..... So in this way, the first output at time step `T0` will be a combination of both **First Hidden State from One Pipeline + Last Hidden State from Second Line**. And for the Last Time stem `Tn`, it'll be **Last Hidden State of one Pipeline + First Hidden State of Second Pipeline**. In this way, it'll calculate First to last and last to first states simultaneously. So the two hidden states will updted by not just past but Future too.\n\n**This is helpful in predictions where the Past is dependent on the future**. Such as in cases where we have to predict \"it ___ that's why I had an umbrella with me\"","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Multi Layer (Deep) RNN\nIn this type of RNN, we can have `RNN/LSTM/GRU` cells not only giving Hidden States to cells to their right(Uni directional) but to cell that is above to them. We can see that like two boxes stacked over two boxes. So any cell at time T, not only uses the Hidden state from the cells before it but from the cells below it too. SO any output would then be a function of `2 hidden states and the inputs` **For a two layered** architecture.\n\nYOu can learn more about Deep RNNs by visiting [this Youtube link](https://www.youtube.com/watch?v=U7wN1x8zsG8&t=0s)\n\n**NOTE: We can use Bidirectional with Stacked to have a very powerfulNetwork. In this setup, each cell will be having 3 different `Hidden States` each from left,right and below cell**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Build Deep Bi-Directional RNN","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class DeepNetwork(torch.nn.Module):\n '''\n Deep RNN Network which can have either one both of stacked and bi-directional properties\n '''\n \n def __init__(self,in_neuron,embedding_dim=100,hidden_size=256,out_neuron=1,m_type='rnn',drop=0.53,**kwargs):\n '''\n Constructor of the class which will instantiate the layers while initialisation.\n \n args:\n in_neuron: input dimensions of the first layer {int}\n embedding_dim: number of latent features you want to calculate from the input data {int} default=100\n hidden_size: neurons you want to have in your hidden RNN layer {int} default=256\n out_neuron: number of outputs you want to have at the end.{int} default=1\n model: whether to use 'rnn','lstm' or 'gru' {string} \n drop: proportion of values to dropout from the previous values randomly {float 0-1} default=0.53\n **kwargs: any valid torch.nn.RNN, torch.nn.LSTM or torch.nn.GRU args with either 'bidirectional'=True \n or 'num_layers'>1\n out: \n return a tensor of shape {batch,out_neuron} as output \n '''\n super(DeepNetwork,self).__init__()\n \n self.m_type = m_type\n \n self.embedding = torch.nn.Embedding(in_neuron,embedding_dim)\n \n if self.m_type == 'lstm':\n self.lstm = torch.nn.LSTM(embedding_dim,hidden_size,**kwargs)\n elif self.m_type == 'gru':\n self.gru = torch.nn.GRU(embedding_dim,hidden_size,**kwargs)\n else:\n self.rnn = torch.nn.RNN(embedding_dim,hidden_size,**kwargs) \n \n self.dropout = torch.nn.Dropout(drop) \n \n self.dense = torch.nn.Linear(hidden_size*2,out_neuron)\n # Last output Linear Layer will have the two Hidden States from both the directions to have the result\n \n \n def forward(self,t):\n '''\n Activate the forward propagation\n args:\n t: tensors in the form of a batch {torch.tensor}\n '''\n t = self.dropout(self.embedding(t)) # get embeddings and dropout\n \n if self.m_type == 'lstm':\n out, (hidden,_) = self.lstm(t)\n elif self.m_type == 'gru':\n out, hidden = self.gru(t)\n else:\n out, hidden = self.rnn(t)\n # shape of rnn = (seq_len, batch, num_directions * hidden_size)\n \n # Concatenate the last and second last hidden. One is from backward and one is from forward\n t = self.dropout(torch.cat((hidden[-2,:,:],hidden[-1,:,:]),dim=1))\n \n return self.dense(t)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train and Evaluate","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"rnn_kwargs = {'num_layers':2,'bidirectional':True}\nin_neuron = len(text_field.vocab)\n\nnetwork = DeepNetwork(in_neuron,m_type='rnn',**rnn_kwargs) \n\nif torch.cuda.is_available():\n network.cuda() # activate GPU spport\n \n# optimizer and losses remains the same\n\nfor epoch in range(EPOCH):\n train_loss, train_acc = train_network(network,train_iter,optimizer,loss_fn,epoch+1)\n val_loss,val_acc = evaluate_network(network,val_iter,optimizer,loss_fn)\n tqdm.write(f'''End of Epoch: {epoch+1} | Train Loss: {train_loss:.3f} | Val Loss: {val_loss:.3f} | Train Acc: {train_acc*100:.2f}% | Val Acc: {val_acc*100:.2f}%''')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can try to work on different combinations of Deep, Bidirectional and type of RNN (LSTM, GRU, or plain RNN) to select a model. Some of the models will perform good while other will perform very poorly with the default 3 Epoch. You can change the Parameters to have a better model.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Embeddings\n## Text as Numbers\nBefore we start on Embeddings let us talk about how you can represent text? Text is, youn know \"text\" and machines only know numbers so in order to tell the machine about the text, we use different approaches most widely as `Boolean Matrix or One Hot Encoding`, `Frquency Based or TfIdf` and `Sementic Based or Embeddings`.\n\nFor example in the sentence `Messi, Eminem and Lebron, they all have achieved greatness to a certain level that it seems like nobody in the near future is going to be even closeer to the half way mark.`\n\nThere are few choices and the simplese being is giving each unique word a number and getting a vector of length equal to the total number of words in the sentence. So let us suppose Messi becomes 1, Eminem becomes 2 and so on to represent all the sentences. But the problem with this method is that the model will think Eminem `2` has more influence than Messi `1` in the sentence but in fact there is no ordering.\n\n### Boolean Vectors:\n\nWe can have a work around it. We can take every word in all the sentences and and make a vector of length equal to all the number of all the unique words in all the sentences and replace. For example in the 3 sentences, `My name is Marshall`, `Shady is my alter ego`,` Eminem is my stage name`, We'll have 9 unique words ans every sentence will be represented by a vector of length 9 where every word will be represented by a 1 if present at a certain location. For example if `name` is represented by the 2nd index, then in the **First** and **Third** sentence, the second index will be a 1 instead of zero where in the **Second** sentence, it'll be a zero because in First and thrird, `name` is present and is defined by the index position 2. This is all cool until it's not. There are lots of problems with the scenario. For example, if a sentence comes, `Hi!!`, then it'll be adding one more dto the length of vector and even though it's length is just one, it'll be represented by a vector of length 10 where 9 of the indices are filled by 0. Another problem is that when a sentence like `Shady is not shady by heart` comes, even though the word `shady` has come twice, it'll be a single 1 at the inde defined so model will have no way of knowing whether the word came 1 or 1000 times.\n\n### Frequency Based:\n\nWe can modify the above problem by giving the position a number equal to the frequeny of word ( how many times the word has repeated in the sentence) so at a defined index of `Shady`, it'll be a 2. Now let us think about pronouns, articles and connectors. They repeat a lot and lot (a,an,the,at,from,on ...bla bla bla..) So if a frequency is given, again it'll make the model think that `at` is more **TfIdf: Tern Frequency Inverse Term Frequency**. What it does is that it gives the words more weightage which appear more in a document. For example in an article Title about `Eminem is the Greatest: Why?` there will be lots of times when `Eminem` and `greatest` will be there that can easily define the Title of the Article but lots of time there will be `a,an,the,at` will bethere in article too which can hamper. So `TfIdf` surely gives more weightage to the words repeated in lines but it'll penalize the words which comes too often in the whole article which will surely be the non essential words like `a,an,the` etc. This methos eeems perfect but what about when we want to find the relation of `Eminem` to the `Greatest`?? There is no way to find that using any of the above techniques. This is where the Next Part Comes into play.\n\n## Embeddings\n\nSolution to all the problems lies in the Embeddings. Word embedding methods learn a real-valued vector representation for a predefined fixed sized vocabulary from a corpus of text. The learning process is either joint with the neural network model on some task, such as document classification, or is an unsupervised process, using document statistics.\n\nConfused?? Don't be. Word Embeddings are the Words converted to a Vector of Length `N` which consists floating values usually in range 0-1. These vectors are not at random but they are there for a reason. If the distance between 2 vectors are small in that embedding means that those two vectors are somehow (and I mean SOMEHOW) related. In this context, we'll say that the disctance of the vector formed by `Moon` will closer to`Earth` tha `Apple`. Simlarly, the embeddings can JUSt KNOW that Paris, New Delhi and London are related and will have very short distances. These Embeddings represent N dimensional HIDDEN features between words. For Example there are 3 hidden features explored by Embeddings by words as `[Scary`, `Soft` ,`Intelligent]` in the range 0-1. If we have 3 words as ``Cat``,`Gun` and `Human`, then the vector for `Human = [0.3, 0.3, 0.9], Gun = [0.8, 0.1, 0.01], Cat = [0.4, 0.9, 0,4]`. Which means that Human is somewhat scary, somewhat sft but highly intelligent and a cat is scary but super soft. The only thing to note here is that the **Machine Learning Model does not even know what it has found. To the model, these are some random dimensions and according to these dimensions, these words (one hot encoded vectors) relate.**\n\nWhatever I have just described can be seen viually and mathematically by watching [this Andrew NG video](https://www.coursera.org/lecture/nlp-sequence-models/word-representation-6Oq70)\n\n## How to?\nEmbeddings are generally generated by using the statistical means on a **HUGE CORPUS OF CLEANED DATA**. There are predefined embeddings such as Google's Word2Vec and Stanford's GloVe but you can use your own Embeddings too by using an `EmbeddingLayer` as the first layer in any Neural Network. The dimension of this layer will be the number of hidden features you want to get and the updated with each epochs will be in a manner to get the context from your data.\n\nA very detailed and intutive explanation about Embeddings and it's types is given in [this post](https://machinelearningmastery.com/what-are-word-embeddings/).\n\n## GloVe: Global Vectors for Word Representation\n\nWe'll be using **GLoVE** which is a hybrid approach to mix both the global statistics of matrix factorization techniques like LSA with the local context-based learning in Word2Vec. Rather than using a window to define local context, GloVe constructs an explicit word-context or word co-occurrence matrix using statistics across the whole text corpus. The result is a learning model that may result in generally better word embeddings.\n\nGloVe has used `word-co-occurnce` and `Nearst Neighbour` to get all these embeddings using a Neural Network. It mans that given a word, it tries to find the similar words used in the same **CONTEXT** and given a context, it tries to find words that can be used in the same context. For example let us assume that as `Lionel` and `Messi` are used most of the times together so given `Lionel`, it'll try to predict `Messi` and vice versa (word-Co-occurance). In the same way `Messi` and `Football` are used together in the same context so when you try to look up for similar words as `Messi`, it'll give you `Football` (context).\n\n## Working of GloVe\nWorking of GloVe is quite interesting. It has learned that `man+feminine` = `woman` so `king+femenine` = `queen`. In other words, it can find the relation that if you subtract the two vectors and add the third vector, it can find a 4th vector which is related to 3rd vector in the same 2nd is related to 1st. For example if you do, `woman-man+king`, it'll return `queen` as `woman-man` will give you the `femenine` dimension and adding femenine to king will be equal to queen.# Thank You\nIn the next part, We'll be covering Embeddings (such as GloVe, Word2Vec), Sequence-to-Sequence Modelling, Attention Modelling with Google's BERT to perform tasks which are not possible with fixed length inputs and outputs.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"glove = vocab.Vectors(IN_PATH+GLOVE_TEXT_PATH, OUT_PATH)\n\nprint(f'Shape of GloVe vectors is {glove.vectors.shape}')\n\n# glove = vocab.GloVe('6B',100) # You can also use this one to get vocab from internet \n# by default it'll load (300*840Billion) dimensional matrix but we'll use 100D version","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This means tht there are 400000 words and each one represented by a vector length 100. Every word in this vocab has a unique integer number **for internal storage purpose only**. Words are converted to vectors and and you can use `glove.stoi['some_small_cased_word']` to see the identifier of that word.\n\nTo access the vector representation of that word, simply use `glove['word']`. When a word is not found in the glove, it is changed by a `` vector of all zeros of same size.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"eminem is represented by the index location at: {glove.stoi['eminem']} and has the following vector values: \\n {glove['eminem']}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# GloVe Helpers","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_vector(glove,word):\n '''\n Get the vector corresponding to a word from Glove\n args:\n glove: glove embeddings\n word: any word\n out: a vector of dimensions according to the embedding size. If a word is not present, it returns zero vector\n '''\n return glove[word.lower()]\n\n\ndef find_closest(glove,input_value,n=6,vector=False):\n '''\n Find the closest words to a given word from the embedding\n args:\n glove: glove embeddings\n input_value: {string,vector} any english word or vector representation from embedding\n n: number of closest words to return\n vector: whether input type is a word or a vector\n out:\n tensor of tuple of words and distances\n '''\n if not vector:\n vector = get_vector(glove,input_value) # get vector of the current word\n else:\n vector = input_value\n \n distances = []\n for neighbour in glove.itos: # get all the words one by one\n dist = torch.dist(vector,get_vector(glove,neighbour)) # calculate distance of all the words to given\n distances.append((neighbour,dist.item()))\n \n sorted_distances = sorted(distances,key=lambda x: x[1]) # sort the value based on tuple's index=1 value\n return sorted_distances[:n] # return top n\n\n\ndef print_neatly(list_of_tuples):\n '''\n Print a tuple cleanly\n args:\n list_of_tuples: List of tuple of 2 values\n '''\n print('Distances \\t Words\\n')\n for tup in list_of_tuples:\n print('%.3f \\t\\t %s'%(tup[1],tup[0]))\n return None\n \n \ndef find_analogy(glove,w1,w11,w2,n=7):\n '''\n Find analogy of the third word given by analogy of two words\n args:\n w1: first word\n w11: analogy of the first word\n w2: second word\n n: number of analogies to find\n out:\n words that can relate to w2 in the same way w11 is related to w1\n '''\n print(f\"{w1} : {w11} :: {w2} : ?\")\n v1 = get_vector(glove,w1)\n v11 = get_vector(glove,w11)\n v2 = get_vector(glove,w2)\n v21 = (v11-v1)+v2\n \n closest_n = find_closest(glove,v21,n=n+3,vector=True) # find extra 3 so that we can remove the given 3\n \n closest_n = [i for i in closest_n if i[0] not in [w1,w11,w2]][:n]\n return closest_n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Try out the cool features","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print_neatly(find_closest(glove,'eminem'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print_neatly(find_analogy(glove,'eminem','rapper','messi'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Processing For GloVe","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"text_field = Field(tokenize=word_tokenize)\n# tokenize text using word_tokenize and convert to numerical form using default parameters\n\nlabel_field = LabelField(dtype=torch.float) \n# useful for label string to LabelEncoding. Not useful here but doesn't hurt either\n\nfields = [('comment_text',text_field),('toxic',label_field)] \n# (column name,field object to use on that column) pair for the dictonary\n\n\ntrain, val, test = TabularDataset.splits(path=OUT_PATH, train='train.csv',validation='val.csv',test='test.csv', \n format='csv',skip_header=True,fields=fields)\n\n\ntext_field.build_vocab(train,max_size=100000,vectors=glove,unk_init=torch.Tensor.zero_) \n\n# unk_init = torch.tensor.normal_ set the initial vectors of vocab as the glove vectors and \n# initialize unknown words as normal distribution instead of zeros\n\nlabel_field.build_vocab(train) \n\n\ntrain_iter, val_iter, test_iter = BucketIterator.splits((train,val,test), batch_sizes=(32,128,128),\n sort_key=lambda x: len(x.comment_text),\n sort_within_batch=False,\n device=device) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Initialize Model & Copy Weights to Embedding Layer","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"in_neuron = len(text_field.vocab)\nembedding_dim = 100 # dimensions of GloVe which we'll use as the dimension for our embedding layer too\ndrop = 0.0 # how much to drop\n\nloss_fn = torch.nn.BCEWithLogitsLoss() \nlr = 0.0003 #learning rate for optimizer\noptimizer = torch.optim.Adam(network.parameters(),lr=lr) \n\nnetwork = Network(in_neuron,embedding_dim,drop=) \n\npretrained_embeddings = text_field.vocab.vectors # get all the 100000+2 vectors\nnetwork.embedding.weight.data.copy_(pretrained_embeddings) #copy embeddings as the weights to the layer\n\n\n# now we have 2 extra embeddings so we'll have to get their index and change the values at index to zeros\n\nunknown_index = text_field.vocab.stoi[text_field.unk_token] # get index of unknown token\npadding_index = text_field.vocab.stoi[text_field.pad_token] # get index of padding token\n\nnetwork.embedding.weight.data[unknown_index] = torch.zeros(embedding_dim) #change values to zeros\nnetwork.embedding.weight.data[padding_index] = torch.zeros(embedding_dim)\n\nif torch.cuda.is_available():\n network.cuda()\n # network = network.to(device)\n\n\n# if you do not want to train your Embedding weights, you'll have to make 1 extra change\n# model.embedding.weight.requires_grad = False\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note: If you have initialised an optimizer, before freezing the weights, then it is okay but **If you Freeze the weights BEFORE passing the parameters into optimizer, then youl'll have to add `optimizer = torch.optim.Adam([ param for param in model.parameters() if param.requires_grad == True],lr=lr)`**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Train and Evaluate Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for epoch in range(EPOCH):\n train_loss, train_acc = train_network(network,train_iter,optimizer,loss_fn,epoch+1)\n val_loss,val_acc = evaluate_network(network,val_iter,optimizer,loss_fn)\n tqdm.write(f'''End of Epoch: {epoch+1} | Train Loss: {train_loss:.3f} | Val Loss: {val_loss:.3f} | Train Acc: {train_acc*100:.2f}% | Val Acc: {val_acc*100:.2f}%''')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}