{"cells":[{"metadata":{},"cell_type":"markdown","source":"NLP is a very hot topic at the moment and as believed by many experts '2020 is going to be NLP's Year'. With it's ever changing dynamics,  NLP is experiencing a boom , same as computer vision once did. Due to its popularity Kaggle launched two NLP competitions recently and me being a lover of NLP and wanted to enhance my skill with it, I decided to join this competition. \n\nIn this Notebook I will cover the following Deep Learning models :\n\n- Simple RNN's\n- LSTM's\n- GRU's\n- BI-Directional RNN's\n\nI will also talk about *seq2seq*, *Attention models* and *BERT*. \n\nNote that the aim of this notebook is not to obtained the highest score, I haven't the right computer for this nor the time. Instead, I want to fully undertand  Deep Learning techniques used for NLP.\n\nSo here's the \"Jigsaw Multilingual Toxic Comment Classification\" competition! I already participated on \"Jigsaw Toxic Comment Classification\"  on Kaggle, my notebook is on GitHub and Linkedin if you want to take a look.\nLike is predecessor, in this competition, contestants are challenged to build machine learning models that can identify toxicity in online conversations, where toxicity is defined as anything rude, disrespectful or otherwise likely to make someone leave a discussion. This problem matters because one toxic comment is enough to sour an online discussion. By identifying and filtering toxic comments online, we can have a safer, more collaborative internet, leading to greater productivity and happiness. And that's very important.\n\nIn this kernel, I will explore the data. Then, I will explain and demonstrate how various deep learning models can be used to identify toxic comments with tensorflow.keras.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"1. [Import packages](#1)\n1. [Load data](#2)\n1. [Data Exploration](#3)\n    - Workcloud for all the comments in train\n    - Number of words present in the comments.\n    - WorkCloud for toxic comments\n    - WorkCloud for clean comments\n    - WorkCloud for Obscene/Severe Toxic/Threat/Insult comments\n1. [Modeling](#4)\n1. [Text preprocessing](#4)\n1. [Logistic Regression](#13)\n    - Basic Preprocessing\n    - Model\n    - Performances\n    - Model inteprretation with LIME\n1. [SIMPLE RNN](#6)\n    - Basic Overview : What is a RNN?\n    - tokenizer and padding\n    - Designing RNN Architecture\n    - Model's performances\n    - Saving our model\n1. [LSTM's](#7)\n    - Basic Overview\n    - Word Embedding\n    - Designing LSTM Architecture\n    - Model's performances\n1. [GRU's](#9)\n    - Basic Overview\n    - Designing GRU Architecture\n    - Model's performances\n1. [Bi-Directional RNN's](#10)\n    - Designing GRU Architecture\n    - Model's performances\n1. [Function to wrap up](#14)\n1. [Conclusion](#11)\n1. [One step further with seq2seq/Attention/BERT](#12)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n## Import packages","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pickle\nimport time","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom tqdm import tqdm\nfrom sklearn.model_selection import train_test_split\nimport tensorflow as tf\nfrom keras.models import Sequential\nfrom keras.layers.recurrent import LSTM, GRU,SimpleRNN\nfrom keras.layers.core import Dense, Activation, Dropout\nfrom keras.layers.embeddings import Embedding\nfrom keras.layers.normalization import BatchNormalization\nfrom keras.utils import np_utils\nfrom sklearn import preprocessing, decomposition, model_selection, metrics, pipeline\nfrom keras.layers import GlobalMaxPooling1D, Conv1D, MaxPooling1D, Flatten, Bidirectional, SpatialDropout1D\nfrom keras.preprocessing import sequence, text\nfrom keras.callbacks import EarlyStopping\n\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n\nfrom wordcloud import WordCloud, STOPWORDS\n\nstopword=set(STOPWORDS)\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2\"></a> <br>\n## Load data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"val = pd.read_csv(\"../input/jigsaw-multilingual-toxic-comment-classification/validation.csv\")\ntest = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/test.csv')\ntrain = pd.read_csv(\"../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info(memory_usage='deep')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for dtype in ['int64','object']:\n    selected_dtype = train.select_dtypes(include=[dtype])\n    mean_usage_b = selected_dtype.memory_usage(deep=True).mean()\n    mean_usage_mb = mean_usage_b / 1024 ** 2\n    print(\"Average memory usage for {} columns: {:03.2f} MB\".format(dtype,mean_usage_mb))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n## Data exploration","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Workcloud for all the comments in train**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"wc = WordCloud(stopwords=stopword)\nplt.figure(figsize=(18,10))\nwc.generate(str(train['comment_text']))\nplt.imshow(wc)\nplt.title('Common words in comments');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the wordcloud above, we can see the most common words in the comments. These words include \"edit\", \"Explanation\", and \"background\" among other words. More offensive words like \"Bastard\" and \"Nazi\" seem to occur less often, indicating that toxic, insulting comments are seen less frequently than non-toxic comments.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Comment words **\n\nNow, I will look at the number of words present in the comments.\n\nDistribution of comment words","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def new_len(x):\n    if type(x) is str:\n        return len(x.split())\n    else:\n        return 0\n\ntrain[\"comment_words\"] = train[\"comment_text\"].apply(new_len)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the plot above, we can see that the distribution of comment words has a strong rightward (positive) skew with maximum probability denisty occuring at around 13 words. As the number of words increases beyond 13, the frequency reduces sharply.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Targets \nNow, I will visualize the targets in the dataset.\n\n**Wordclouds for different categories**\n\nNon-toxic vs. Toxic","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"Toxic = train[train['toxic'] == 1]\nNoToxic = train[train['toxic'] == 0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**WorkCloud for toxic comments**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"wc = WordCloud(stopwords= stopword)\nplt.figure(figsize = (18,12))\nwc.generate(str(Toxic['comment_text']))\nplt.imshow(wc)\nplt.title('Words frequented in Toxic Comments');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**WorkCloud for clean comments**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"wc = WordCloud(stopwords= stopword)\nplt.figure(figsize = (18,12))\nwc.generate(str(NoToxic['comment_text']))\nplt.imshow(wc)\nplt.title('Words frequented in Clean Comments')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see from the above wordclouds, that toxic comments use more insluting or hateful words such as \"f**k\" or \"idiot\", while the non-toxic comments do not usually use such words.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**WorkCloud for Obscene/Severe Toxic/Threat/Insult comments**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"obscene = train[train['obscene'] == 1]\nsevere = train[train['severe_toxic'] == 1]\nthreat = train[train.threat == 1]\ninsult = train[train.insult == 1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"wc = WordCloud(stopwords= stopword)\nplt.figure(figsize = (18,12))\nwc.generate(str(obscene['comment_text']))\nplt.imshow(wc)\nplt.title('Words frequented in obscene Comments');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"wc = WordCloud(stopwords= stopword)\nplt.figure(figsize = (18,12))\nwc.generate(str(severe['comment_text']))\nplt.imshow(wc)\nplt.title('Words frequented in severe Comments');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"wc = WordCloud(stopwords= stopword)\nplt.figure(figsize = (18,12))\nwc.generate(str(threat['comment_text']))\nplt.imshow(wc)\nplt.title('Words frequented in threat Comments');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"wc = WordCloud(stopwords= stopword)\nplt.figure(figsize = (18,12))\nwc.generate(str(insult['comment_text']))\nplt.imshow(wc)\nplt.title('Words frequented in insult Comments');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the above wordclouds, we can see that most of these categories use insulting/hateful language. But, the threat category seems to be slightly different from the remaining categories, as it uses words like \"kill\" and \"die\", indicating that most threats involve threats to kill someone.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n## Modeling\n\nNow, I will show how different deep learning models can be used to classify toxic comments.\n\n\nWe will drop the other columns and approach this problem as a Binary Classification Problem and also we will have our exercise done on a smaller subsection of the dataset(only 30000 data points) to make it easier to train the models","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.drop(['severe_toxic','obscene','threat','insult','identity_hate'],axis=1,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.loc[:30000,:]\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"5\"></a> <br>\n## Text preprocessing","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#We will check the maximum number of words that can be present in a comment , this will help us in padding later\n\ntrain['comment_text'].apply(lambda x:len(str(x).split())).max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Writing a function for getting auc score for validation\n\ndef roc_auc(predictions,target):\n    '''\n    This methods returns the AUC Score when given the Predictions\n    and Labels\n    '''\n    \n    fpr, tpr, thresholds = metrics.roc_curve(target, predictions)\n    roc_auc = metrics.auc(fpr, tpr)\n    return roc_auc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from nltk.corpus import stopwords\nimport nltk\nimport re\nimport string, collections\nfrom nltk.stem.snowball import EnglishStemmer\nfrom nltk.stem import WordNetLemmatizer\n\nstop_words = set(stopwords.words('english'))\ntokenizer = nltk.tokenize.RegexpTokenizer(r'\\w+')\nstemmer = EnglishStemmer()\nlemmatizer = WordNetLemmatizer()\n\ndef clean_text(text):\n    text = text.lower() #make text lowercase and fill na\n    text = re.sub('\\[.*?\\]', '', text) \n    text = re.sub('\\\\n', '',str(text))\n    text = re.sub(\"\\[\\[User.*\",'',str(text))\n    text = re.sub(\"\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\",'',str(text))\n    text = re.sub('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', '', text) #remove hyperlinks\n    text = re.sub(r'\\:(.*?)\\:', '', text) #remove emoticones\n    text = re.sub(r'[\\w\\.-]+@[\\w\\.-]+', '', str(text)) #remove email\n    text = re.sub(r'(?<=@)\\w+', '', text) #remove @\n    text = re.sub(r'[0-9]+', '', text) #remove numbers\n    text = re.sub(\"[^A-Za-z0-9 ]\", '', text) #remove non alphanumeric like ['@', '#', '.', '(', ')']\n    text = re.sub(r'[!\"\\$%&\\'()*+,\\-.\\/:;=#@?\\[\\\\\\]^_`{|}~]*', '', text) #remove punctuations from sentences\n    text = re.sub('<.*?>+', '', str(text))\n    text = re.sub('[%s]' % re.escape(string.punctuation), '', str(text))\n    text = re.sub('\\w*\\d\\w*', '', str(text))\n    text = tokenizer.tokenize(text)\n    text = [word for word in text if not word in stop_words]\n    #text = [lemmatizer.lemmatize(word) for word in text]\n    text = [stemmer.stem(word) for word in text]\n    final_text = ' '.join( [w for w in text if len(w)>1] ) #remove word with one letter\n    return final_text\n\n\n\n\n\n#val[\"comment_text\"] = clean_text(str(val[\"comment_text\"]))\n#test_data[\"content\"] = clean_text(str(test_data[\"content\"]))\n#train[\"comment_text\"] = clean_text(str(train[\"comment_text\"]))\n\n                  \nval['comment_text'] = val['comment_text'].apply(lambda x : clean_text(x))\n\ntest['content'] = test['content'].apply(lambda x : clean_text(x))\n\ntrain['comment_text'] = train['comment_text'].apply(lambda x : clean_text(x))  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"xtrain, xval, ytrain, yval = train_test_split(train['comment_text'].values, train['toxic'].values, \n                                                  stratify=train.toxic.values, \n                                                  random_state=42, \n                                                  test_size=0.2, shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"13\"></a> <br>\n## Logistic regression","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Basic preprocessing**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.linear_model import LogisticRegressionCV\nfrom sklearn.pipeline import make_pipeline\nfrom imblearn.over_sampling import  SMOTE\n\n\nsmt = SMOTE(random_state=777, k_neighbors=1)\n\nvec = TfidfVectorizer(min_df=3,max_features=10000,strip_accents='unicode',\n                     analyzer='word',ngram_range=(1,2),token_pattern=r'\\w{1,}',use_idf=1,smooth_idf=1,sublinear_tf=1,\n                     stop_words='english')\n\nvec_fit=vec.fit_transform(xtrain)\n\nclf = LogisticRegressionCV()\n\n\n# Over Sampling\nX_SMOTE, y_SMOTE = smt.fit_sample(vec_fit, ytrain)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from collections import Counter\n#we over sampled it \nprint(Counter(y_SMOTE))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#dealed with imbalanced","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Modeling**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import cross_val_score\nfrom sklearn.linear_model import LogisticRegression\nclf = LogisticRegression(C=0.1, solver='sag')\nscores = cross_val_score(clf, X_SMOTE,y_SMOTE, cv=5,scoring='f1_weighted')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scores.mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clf.fit(X_SMOTE,y_SMOTE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Model performances**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Writing a function for getting auc score for validation\n\ndef roc_auc(predictions,target):\n    '''\n    This methods returns the AUC Score when given the Predictions\n    and Labels\n    '''\n    \n    fpr, tpr, thresholds = metrics.roc_curve(target, predictions)\n    roc_auc = metrics.auc(fpr, tpr)\n    return roc_auc\n\n\nfrom sklearn import metrics\n\ndef print_report1(data, y):\n    y_test =  y\n    test_features=vec.transform(data)\n    y_pred = clf.predict(test_features)\n    report = metrics.classification_report(y_test, y_pred, target_names=['Toxic', 'Clean'])\n    print(report)\n    print(\"accuracy: {:0.3f}\".format(metrics.accuracy_score(y_test, y_pred)))\n    print()\n    print(\"Auc: %.2f%%\" % (roc_auc(y_pred,y_test)))\n\nprint_report1(xval, yval)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Model interpretability with LIME**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Recent times have seen a renewed focus on model interpretability. Machine Learning Experts are able to understand the importance of a model interpretability in it’s subsequent adaption by business. The problem with model explainability is that it’s very hard to define a model’s decision boundary in human understandable manner, especially with boosting model. LIME and eli5 are  python libraries which tries to solve for model interpretability by producing locally faithful explanations. Below is an example of one such explanation for my NLP problem.\n\nLIME use a representation that is understood by the humans irrespective of the actual features used by the model. This is coined as interpretable representation. An interpretable representation would vary with the type of data that we are working with for example :\n\n - For text : It represents presence/absence of words.\n- For image : It represents presence/absence of super pixels ( contiguous patch of similar pixels ).\n- For tabular data : It is a weighted combination of columns\n\nSo let's see this.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import eli5\n# import warnings filter\nfrom warnings import simplefilter\n# ignore all future warnings\nsimplefilter(action='ignore', category=FutureWarning)\neli5.show_weights(clf, vec=vec, top=15,\n                  target_names=['clean','toxic'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above figure tells that green word contributed most to Toxic comments and Red words contributed to opposite class that is Clean comments class","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(xval[2])\nprint('\\n')\nprint(yval[2])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import eli5\neli5.show_prediction(clf, xval[2], vec=vec,\n                     target_names=['clean','toxic'],top=15)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It shows probability of each of  the 2 classes and then shows which features contributed the most and which contributed the least in each class top argument shows the  top n features that contibuted to the prediction of each class\n\n\nNB : If we got the BIAS term, it occurs because we are using Linear model for classification and the Intercept added to the equation is termed BIAS here.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"6\"></a> <br>\n## SIMPLE RNN","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Here we will use 12000 rows instead of 30000 to increase calcul with Deep Learning.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"val = pd.read_csv(\"../input/jigsaw-multilingual-toxic-comment-classification/validation.csv\")\ntest = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/test.csv')\ntrain = pd.read_csv(\"../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.loc[:12000,:]\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val['comment_text'] = val['comment_text'].apply(lambda x : clean_text(x))\n\ntest['content'] = test['content'].apply(lambda x : clean_text(x))\n\ntrain['comment_text'] = train['comment_text'].apply(lambda x : clean_text(x)) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"xtrain, xval, ytrain, yval = train_test_split(train['comment_text'].values, train['toxic'].values, \n                                                  stratify=train.toxic.values, \n                                                  random_state=42, \n                                                  test_size=0.2, shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Basic Overview : What is a RNN?**\n\nRecurrent Neural Network (RNN) are a type of Neural Network where the output from previous step are fed as input to the current step. In traditional neural networks, all the inputs and outputs are independent of each other, but in cases like when it is required to predict the next word of a sentence, the previous words are required and hence there is a need to remember the previous words. Thus RNN came into existence, which solved this issue with the help of a Hidden Layer.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**tokenizer and padding**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# using keras tokenizer here\ntoken = text.Tokenizer(num_words=None)\nmax_len = 1500\n\ntoken.fit_on_texts(list(xtrain) + list(xval))\nxtrain_seq = token.texts_to_sequences(xtrain)\nxval_seq = token.texts_to_sequences(xval)\n\n#zero pad the sequences\nxtrain_pad = sequence.pad_sequences(xtrain_seq, maxlen=max_len)\nxval_pad = sequence.pad_sequences(xval_seq, maxlen=max_len)\n\nword_index = token.word_index","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What keras tokenizer does is, it produce a one hot vector (value 1 t the position of the word and rest 0) to transform every word into digits. To feed the Deep Learning algorithms with data it could manipulate. Very important step !\n\nThe pad_sequences() (padding) function in the Keras deep learning library can be used to pad variable length sequences.\nWe fill sequences with a pad token (usually 0) to fit the matrix size. This special tokens is then masked not to be accounted in loss calculation. Like that we have all of our sequences with the same length and it's exploitable for our Deep Learning models.  Deep learning libraries assume a vectorized representation of your data.\nIn the case of variable length sequence prediction problems, this requires that your data be transformed such that each sequence has the same length.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Designing RNN Architecture**\n\nLet's built a simple RNN without any pretrained embeddings and one dense layer :","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\nmodel.add(Embedding(len(word_index) + 1,\n                     300,\n                     input_length=max_len))\nmodel.add(SimpleRNN(100))\nmodel.add(Dense(1, activation='sigmoid'))\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's compile our network ! We will always use 5 epochs and a batch at the size 64 for all our models in this notebook.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Model's performances**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(xtrain_pad, ytrain, epochs = 5, batch_size = 64, verbose = 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred = model.predict(xval_pad)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Auc: %.2f%%\" % (roc_auc(pred,yval)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see our model achieves an accuracy of 0,87 very quickly which is just insane, let's not forget than maybe our model is overfitting. But even with all of this in mind,  this was the simplest model of all ,we can tune a lot of hyperparameters like RNN units, we can do batch normalization , dropouts etc to get better result. The point is we got an AUC score of 0.74 without much efforts and we know have learnt about RNN's .Deep learning is really revolutionary, especially for Natural Language Processing !","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Saving our model**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's saving our model with pickle in order to run it again without taking so much time","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"file_name = 'simpleRNN.sav'\npickle.dump(model , open(file_name, 'wb'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see how much faster it goes :","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"%%time\nmodel = pickle.load(open('simpleRNN.sav', 'rb'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"pred = model.predict(xval_pad)\nprint(\"Auc: %.2f%%\" % (roc_auc(pred,yval)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's way faster, it's very convenient when building model like Deep Learning models is quite computationnaly expensive and also quite lenghty to run.\n\nNow that our model building is done, it might be a good idea to clean up some memory before we go to the next step.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"del model\n\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"7\"></a> <br>\n## LSTM's\n\n**Basic Overview**\n\nSimple RNN's were certainly better than classical ML algorithms, in this case, and gave state of the art results, but it failed to capture long term dependencies that is present in sentences. So in 1998-99 LSTM's were introduced to counter to these drawbacks. That's why they called Long-Short-Term-Memory (LSTM).\n\n**Word Embedding**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Here we will use Word Embedding. Why ? Because it's much more efficient than just basic word representation with One Hot Encoding. \nWord embedding has nowadays become the dominant approach to vectorization. Embedding is a type of word representation that allows words with similar meaning to have a similar representation by mapping them to vectors of real numbers. Unlike older methods like One Hot enconding, word embeddings are able to represent implicit relationships between words that are useful when training on data that can benefit from contextual information\n\nThey are a distributed representation for text that is perhaps one of the key breakthroughs for the impressive performance of deep learning methods on challenging natural language processing problems.\n\nThe latest approach to getting word Embeddings is using pretained GLoVe or using Fasttext. Here I'll be using GloVe.\n\nWord Embedding will surely improve your models, it will take maybe more time but it's almost always worth your time !\n\nIf you want to acces the GloVe vectors, you can download it from here http://www-nlp.stanford.edu/data/glove.840B.300d.zip or you can search for GloVe in datasets on Kaggle and add the file.\n\nI will calculate embedding matrix for our vocabulary from the pretrained GLoVe vectors. It's a quite easy methods I found on Kaggle. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# load the GloVe vectors in a dictionary:\n\nembeddings_index = {}\nf = open('../input/glove-global-vectors-for-word-representation/glove.6B.200d.txt','r',encoding='utf-8')\nfor line in tqdm(f):\n    values = line.split(' ')\n    word = values[0]\n    coefs = np.asarray([float(val) for val in values[1:]])\n    embeddings_index[word] = coefs\nf.close()\n\nprint('Found %s word vectors.' % len(embeddings_index))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create an embedding matrix for the words we have in the dataset\nembedding_matrix = np.zeros((len(word_index) + 1, 200))\nfor word, i in tqdm(word_index.items()):\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None:\n        embedding_matrix[i] = embedding_vector","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have already tokenized and paded our text for input to our previous simple RNN so let's move on into our LSTM model right away.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Designing LSTM Architecture**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's built a simple LSTM glove embeddings and one dense layer :","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\nmodel.add(Embedding(len(word_index) + 1,\n                     200,\n                     weights=[embedding_matrix],\n                     input_length=max_len,\n                     trainable=False))\n\nmodel.add(LSTM(100, dropout=0.3, recurrent_dropout=0.3))\nmodel.add(Dense(1, activation='sigmoid'))\nmodel.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Model's performances**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(xtrain_pad, ytrain, epochs=5, batch_size = 64)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred = model.predict(xval_pad)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Auc: %.2f%%\" % (roc_auc(pred,yval)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We now see that the model achieves an auc score of 0.94 which is an huge improvement compared to simple RNN , also we close in on the gap between accuracy and auc so we could assume that our model is not overfitting . We see that in this case we used dropout and prevented overfitting the data. \n\nLSTM is much better than simple RNN, a little long to compute all the calcul but it surely worth the time.\n\nLet's clean up some memory and save our new model before we move on.","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"file_name = 'LSTM.sav'\npickle.dump(model , open(file_name, 'wb'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del model\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's move to GRU models.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a id=\"9\"></a> <br>\n## GRU's\n\n**Basic Overview**\n\nGRU (Gated Recurrent Unit), like LSTM, aims to solve the vanishing gradient problem which comes with a standard recurrent neural network. GRU's are a variation on the LSTM because both are designed similarly and, in some cases, produce equally excellent results . GRU's were designed to be simpler and faster than LSTM's and in most cases produce equally good results and thus there is no clear winner.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Designing GRU architecture**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"All of our preprocessing is already done so let's built a GRU with glove embeddings and two dense layers right away.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\nmodel.add(Embedding(len(word_index) + 1,\n                     200,\n                     weights=[embedding_matrix],\n                     input_length=max_len,\n                     trainable=False))\nmodel.add(SpatialDropout1D(0.3))\nmodel.add(GRU(300))\nmodel.add(Dense(1, activation='sigmoid'))\n\nmodel.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Model's performances**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(xtrain_pad, ytrain, epochs = 5, batch_size = 64)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred = model.predict(xval_pad)\nprint(\"Auc: %.2f%%\" % (roc_auc(pred,yval)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In theory, GRU is designed to be much faster than LSTM, less complex and produce equally good results. But here, GRU took much more time than LSTM and give us approximately the same result so in this case, that wasn't an improvement, but it has to be tested. I wanted to see how it goes and how it work. Before this notebook I didn't know GRU and I learnt a lot in the process so that's cool !","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"file_name = 'GRU.sav'\npickle.dump(model , open(file_name, 'wb'))\n\ndel model\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"10\"></a> <br>\n## Bi-Directional RNN's","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's build a simple bidirectional LSTM with glove embeddings and one dense layer","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\nmodel.add(Embedding(len(word_index) + 1,\n                     200,\n                     weights=[embedding_matrix],\n                     input_length=max_len,\n                     trainable=False))\nmodel.add(Bidirectional(LSTM(300, dropout=0.3, recurrent_dropout=0.3)))\n\nmodel.add(Dense(1,activation='sigmoid'))\nmodel.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(xtrain_pad, ytrain, epochs=5, batch_size=64)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scores = model.predict(xval_pad)\nprint(\"Auc: %.2f%%\" % (roc_auc(scores,yval)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It take forever to calcul and the result aren't better. So Il wouldn't recommend this kind of structure unless you're are fully warn about it. Maybe in other cases, it's THE solution but here it's just took too much time. \n\nMaybe with a much more powerful computer ? ","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"file_name = 'Bidirectional.sav'\npickle.dump(model , open(file_name, 'wb'))\n\ndel model\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"14\"></a> <br>\n## Function to wrap-up","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"def roc_auc(predictions,target):\n    '''\n    This methods returns the AUC Score when given the Predictions\n    and Labels\n    '''\n    \n    fpr, tpr, thresholds = metrics.roc_curve(target, predictions)\n    roc_auc = metrics.auc(fpr, tpr)\n    return roc_auc\n\nfrom nltk.corpus import stopwords\nimport nltk\nimport re\nimport string, collections\n\nstop_words = set(stopwords.words('english'))\ntokenizer = nltk.tokenize.RegexpTokenizer(r'\\w+')\n\ndef clean_text(text):\n    text = text.lower() #make text lowercase and fill na\n    text = re.sub('\\[.*?\\]', '', text) \n    text = re.sub('\\\\n', '',str(text))\n    text = re.sub(\"\\[\\[User.*\",'',str(text))\n    text = re.sub(\"\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\",'',str(text))\n    text = re.sub('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', '', text) #remove hyperlinks\n    text = re.sub(r'\\:(.*?)\\:', '', text) #remove emoticones\n    text = re.sub(r'[\\w\\.-]+@[\\w\\.-]+', '', str(text)) #remove email\n    text = re.sub(r'(?<=@)\\w+', '', text) #remove @\n    text = re.sub(r'[0-9]+', '', text) #remove numbers\n    text = re.sub(\"[^A-Za-z0-9 ]\", '', text) #remove non alphanumeric like ['@', '#', '.', '(', ')']\n    text = re.sub(r'[!\"\\$%&\\'()*+,\\-.\\/:;=#@?\\[\\\\\\]^_`{|}~]*', '', text) #remove punctuations from sentences\n    text = re.sub('<.*?>+', '', str(text))\n    text = re.sub('[%s]' % re.escape(string.punctuation), '', str(text))\n    text = re.sub('\\w*\\d\\w*', '', str(text))\n    text = tokenizer.tokenize(text)\n    text = [word for word in text if not word in stop_words]\n    final_text = ' '.join( [w for w in text if len(w)>1] ) #remove word with one letter\n    return final_text\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# load the GloVe vectors in a dictionary:\n\nembeddings_index = {}\nf = open('glove.840B.300d.txt','r',encoding='utf-8')\nfor line in tqdm(f):\n    values = line.split(' ')\n    word = values[0]\n    coefs = np.asarray([float(val) for val in values[1:]])\n    embeddings_index[word] = coefs\nf.close()\n\nprint('Found %s word vectors.' % len(embeddings_index))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# create an embedding matrix for the words we have in the dataset\nembedding_matrix = np.zeros((len(word_index) + 1, 300))\nfor word, i in tqdm(word_index.items()):\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None:\n        embedding_matrix[i] = embedding_vector","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"import math\nimport sklearn.metrics as sklm\n\n\ndef function(model, model_name):\n    \n    val = pd.read_csv(\"validation.csv\")\n    test = pd.read_csv('test.csv')\n    train = pd.read_csv(\"jigsaw-toxic-comment-train.csv\")\n\n    train.drop(['severe_toxic','obscene','threat','insult','identity_hate'],axis=1,inplace=True)\n\n    train = train.loc[:12000,:]\n    train.shape\n\n\n\n\n    val['comment_text'] = val['comment_text'].apply(lambda x : clean_text(x))\n\n    test['content'] = test['content'].apply(lambda x : clean_text(x))\n\n    train['comment_text'] = train['comment_text'].apply(lambda x : clean_text(x))  \n    \n    xtrain, xval, ytrain, yval = train_test_split(train['comment_text'].values, train['toxic'].values, \n                                                  stratify=train.toxic.values, \n                                                  random_state=42, \n                                                  test_size=0.2, shuffle=True)\n    \n    # using keras tokenizer here\n    token = text.Tokenizer(num_words=None)\n    max_len = 1500\n\n    token.fit_on_texts(list(xtrain) + list(xval))\n    xtrain_seq = token.texts_to_sequences(xtrain)\n    xval_seq = token.texts_to_sequences(xval)\n\n    #zero pad the sequences\n    xtrain_pad = sequence.pad_sequences(xtrain_seq, maxlen=max_len)\n    xval_pad = sequence.pad_sequences(xval_seq, maxlen=max_len)\n\n    word_index = token.word_index\n\n\n    #modeling\n    if model == SimpleRNN :\n        \n\n        model = Sequential()\n        model.add(Embedding(len(word_index) + 1,\n                             300,\n                             input_length=max_len))\n        model.add(SimpleRNN(100))\n        model.add(Dense(1, activation='sigmoid'))\n        model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n        print(model.summary())\n        \n        model.fit(xtrain_pad, ytrain, epochs = 5, batch_size = 64, verbose = 0)\n        \n        pred = model.predict(xval_pad)\n        \n        print(\"The AUC with {} is: {}\".format(model_name,(roc_auc(pred,yval))))\n\n        #save our model \n        \n        file_name = 'simpleRNN.sav'\n        pickle.dump(model , open(file_name, 'wb'))\n        print('Model saved !')\n    \n    elif model == LSTM:\n        \n        model = Sequential()\n        model.add(Embedding(len(word_index) + 1,\n                             300,\n                             weights=[embedding_matrix],\n                             input_length=max_len,\n                             trainable=False))\n\n        model.add(LSTM(100, dropout=0.3, recurrent_dropout=0.3))\n        model.add(Dense(1, activation='sigmoid'))\n        model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])\n        print(model.summary())\n        \n        model.fit(xtrain_pad, ytrain, epochs = 5, batch_size = 64, verbose = 0)\n        \n        pred = model.predict(xval_pad)\n        \n        print(\"The AUC with {} is: {}\".format(model_name,(roc_auc(pred,yval))))\n\n        #save our model \n        \n        file_name = 'LSTM.sav'\n        pickle.dump(model , open(file_name, 'wb'))\n        print('Model saved !')\n        \n    elif model == GRU:\n        \n        model = Sequential()\n        model.add(Embedding(len(word_index) + 1,\n                             300,\n                             weights=[embedding_matrix],\n                             input_length=max_len,\n                             trainable=False))\n        model.add(SpatialDropout1D(0.3))\n        model.add(GRU(300))\n        model.add(Dense(1, activation='sigmoid'))\n\n        model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])\n        print(model.summary())\n        \n        model.fit(xtrain_pad, ytrain, epochs = 5, batch_size = 64, verbose = 0)\n        \n        pred = model.predict(xval_pad)\n        \n        print(\"The AUC with {} is: {}\".format(model_name,(roc_auc(pred,yval))))\n\n        #save our model \n        \n        file_name = 'GRU.sav'\n        pickle.dump(model , open(file_name, 'wb'))\n        print('Model saved !')\n    \n    elif model == BiRNN:\n\n        model = Sequential()\n        model.add(Embedding(len(word_index) + 1,\n                             300,\n                             weights=[embedding_matrix],\n                             input_length=max_len,\n                             trainable=False))\n        model.add(Bidirectional(LSTM(300, dropout=0.3, recurrent_dropout=0.3)))\n\n        model.add(Dense(1,activation='sigmoid'))\n        model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])\n        print(model.summary())\n\n        \n        model.fit(xtrain_pad, ytrain, epochs = 5, batch_size = 64, verbose = 0)\n        \n        pred = model.predict(xval_pad)\n        \n        print(\"The AUC with {} is: {}\".format(model_name,(roc_auc(pred,yval))))\n\n        #save our model \n        \n        file_name = 'BiRNN.sav'\n        pickle.dump(model , open(file_name, 'wb'))\n        print('Model saved !')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"11\"></a> <br>\n## Conclusion","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"All of the methods I tried in this notebook gave good results, especially LSTM, GRU and Bi-directionnal, simple RNN gave also some nice result but was more likely overfitting. For this case, I would recommend LSTM, because it was much faster than GRU and bi-directionnal for me. But if you have the time and/or the right computer, go for it, there are performant as well ! Like as always said, it's a matter of trade-off. You want a super powerful model and you have the time and the computers ? Let's do some hyperparameter tuning on LSTM/GRU and bi-directionnal ! Let's go further in depth and found a model who achieve 99,99% accuracy and 99% AUC. In the other and, if you want to move on quickly on another project, just take the more efficient after a quick training, like here with LSTM who give really good result for the time taken.\n\nSo for now, I will not try some methods like Attention models, Seq2Seq models, BERT and so on. Not because there are useless or less interesting but because I haven't the time for that right now. Those kind of model are computationnaly expensive and my computer is not that powerful. It will take forever, it already have with GRU and Bi-directionnal RNN. Maybe later, when I have a more powerful computer. So I will not write the code implementation for this neither lauched it, but rather I will provide the resources where code has already been implemented.\n\n<a id=\"12\"></a> <br>\n## One step further with seq2seq/Attention/BERT","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"1) *seq2seq models*\n\n- https://www.coursera.org/learn/nlp-sequence-models/lecture/HyEui/basic-models : A basic idea of different Seq2Seq Models\n\n- https://towardsdatascience.com/how-to-implement-seq2seq-lstm-model-in-keras-shortcutnlp-6f355f3e5639 : A More advanced Seq2seq Model and its explanation.\n\n2) *Attention models*\n\n- https://www.coursera.org/learn/nlp-sequence-models/lecture/RDXpX/attention-model-intuition \n\n- https://towardsdatascience.com/sequence-2-sequence-model-with-attention-mechanism-9e9ca2a613a\n\n- https://towardsdatascience.com/attention-and-its-different-forms-7fc3674d14dc\n\n3) *BERT models*\n\n- http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}