{"cells":[{"metadata":{"_uuid":"92de646688a1f688ffe15319d9c798248c3d85f4"},"cell_type":"markdown","source":"# Text Classification from data preparation perspective - based on Quora competition"},{"metadata":{"_uuid":"97b2a947deac07cf3132aad8a183b2e46d66b69e"},"cell_type":"markdown","source":"## Preface"},{"metadata":{"_uuid":"8908f123f85fa95de3669704ed36203c17736e95"},"cell_type":"markdown","source":"Objective of the notebook is to show how to build text classification models using three text representations: \n* text represented by features created manually (manual feature engineering), \n* words frequency based text representation (matrix of token counts/ TF-IDF features), \n* sequence of words (word embedding) representation,\n \nOn top of that I would like to point on some solutions that I find very useful: \n* [sklearn.pipeline](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.pipeline) module that implements utilities to build a composite estimator, as a chain of transforms and estimators, \n* [sklearn.compose.ColumnTransformer](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html) module that applies transformers to columns of an array or pandas DataFrame - recently added to scikit-learn, \n* [keras.wrappers.scikit_learn.KerasClassifier](https://keras.io/scikit-learn-api/) wrappers that allow to use Sequential Keras models as part of Scikit-Learn pipeline, \n\nThe list of topics that aren't covered here:\n* advanced method of text cleaning and spell checking, \n* character-based language models, \n* ngram tokenizers, \n* word embeddings details,\n* PyTorch\n\n#### References:\n\n1) [The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) \n2) [Understanding LSTM Networks by Christopher Olah](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) \n3) [An Introduction to Bag-of-Words in NLP](https://medium.com/greyatom/an-introduction-to-bag-of-words-in-nlp-ac967d43b428) \n4) [Understanding word embeddings](https://hackernoon.com/understanding-word-embeddings-a9ff830403ce) \n5) [Useful properties of ROC curves, AUC scoring, and Gini Coefficients](https://luckytoilet.wordpress.com/2018/04/04/useful-properties-of-roc-curves-auc-scoring-and-gini-coefficients/) \n6) [Building a custom Python scikit-learn transformer for machine learning.](https://opendevincode.wordpress.com/2015/08/01/building-a-custom-python-scikit-learn-transformer-for-machine-learning/) \n7) [Getting started with the Keras Sequential model](https://keras.io/getting-started/sequential-model-guide/) \n8) [Wrappers for the Scikit-Learn API](https://keras.io/scikit-learn-api/) \n9) [GloVe: Global Vectors for Word Representation](https://nlp.stanford.edu/projects/glove/) \n10) [Empirical Evaluation of RNN Architectures on Sentence Classification Task](https://arxiv.org/pdf/1609.09171.pdf) \n11) [Getting started with the Keras functional API](https://keras.io/getting-started/functional-api-guide/) \n12) [Understanding Bidirectional RNN in PyTorch](https://towardsdatascience.com/understanding-bidirectional-rnn-in-pytorch-5bd25a5dd66) \n13) [Attention and Memory in Deep Learning and NLP](http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/)"},{"metadata":{"_uuid":"7172432b611555b71e6c0f86dbba6d44e848ff09"},"cell_type":"markdown","source":"## Problem overview - dataset used in this notebook"},{"metadata":{"_uuid":"985bb580122088ebaed5f60301bb72ddb7b2d274"},"cell_type":"markdown","source":"I decided to use dataset from [Quora Insincere Questions Classification](https://www.kaggle.com/c/quora-insincere-questions-classification) because: \n* in my opinion the qality is quite good, \n* it consist only from two columns: text and target column so I don't need to focus on other stuff that aren't so important on purpose of this notebook,\n* I know this dataset quite well because I was attending this competition :) \n\n**In this competition we were predicting whether a question asked on Quora was sincere or not.**\n\n[**Quora**](https://www.quora.com/) is a place to gain and share knowledge. It's a platform to ask questions and connect with people who contribute unique insights and quality answers.\n\nAn insincere question is defined as a question intended to make a statement rather than look for helpful answers. \nSome characteristics that can signify that a question is insincere:\n\n* Has a non-neutral tone,\n* Has an exaggerated/overemphasized tone to underscore a point about a group of people,\n* Is rhetorical and meant to imply a statement about a group of people,\n* Suggests a discriminatory idea against a protected class of people, or seeks confirmation of a stereotype,\n* Makes disparaging attacks/insults against a specific person or group of people,\n* Isn't grounded in reality,\n* Based on false information, or contains absurd assumptions,\n* Uses sexual content (incest, bestiality, pedophilia) for shock value, and not to seek genuine answers,\n\nThe training data includes the question that was asked, and target value that means whether text was identified as insincere (target = 1) or not (target = 0). \nThe ground-truth labels contain some amount of noise: they are not guaranteed to be perfect. \n \n### Examples:\n\n**insincere (target: 1)** \n* Why don't nannies want to work caring for black or half black children?\n* What is it like being black/Muslim/homosexual/immigrant etc and support Trump?\n* Do Muslims make animals watch other animals get slaughtered to trigger hormone release into their Halal meat?\n* Why do Black women hate straight men? \n* Do you dislike me for being white?\n\n**sincere (target: 0)** \n* How can I create a fake UK student ID?\n* What are some good negotiating practices? \n* How do I do good work in a new start up?\n* Is it possible for a black hole to become a nova/supernova/hypernova?\n* What will happen in each of the areas if the fortests are removed?"},{"metadata":{"_uuid":"d01495cf1abfd6d7265a98c7cff0d715a2cc403f"},"cell_type":"markdown","source":"## Text representation summary"},{"metadata":{"_uuid":"38d9f6ca7c6ae38ee294395398f7be9f0bff29af"},"cell_type":"markdown","source":"** In ML the process of converting NLP text into numbers is called vectorization **"},{"metadata":{"_uuid":"ec263ec4ebec85ff31b21b42c9b8c0cd3139d68e"},"cell_type":"markdown","source":"Let's have a look a graph that show three different ways of representing a text:"},{"metadata":{"_uuid":"aeb78b359ca8b00c0b3547fde6ffd0f9748e8b41"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"c36cc810bf1c5b88125d0e4242f0dac7f2c9ffa2"},"cell_type":"markdown","source":"### 1. manual feature engineering \na document/observation is described by a set of features created by hand ex.: \n* how long is the document/observation, \n* how many words is in document/observation, \n* what is the ratio of numbers of words to number of characters,\n* how many www links contains text, \n* how many email addresses contains text, \n* does it contain swear word, \n* etc.\n\n### 2. frequency based representation\na document/observation is represented as the bag of its words, disregarding grammar and even word order but keeping multiplicity, \njust like in a manual feature engineering approache one record/vector represent one observation.\n* **matrix of token counts - CountVectorizer** \nCountVectorizer works on Terms Frequency, i.e. counting the occurrences of tokens and building a sparse matrix of documents x tokens.\n\n ```\n matrix = [\n [0 1 1 1 0 0 1 0 1]\n [0 2 0 1 0 1 1 0 1]\n [1 0 0 1 1 0 1 1 1]\n [0 1 1 1 0 0 1 0 1]\n ]\n ```\n* **a matrix of TF-IDF features - TF-IDF Vectorizer** \nTF-IDF stands for term frequency-inverse document frequency. TF-IDF weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus.\n\n\\begin{align}\n\\text{TF-IDF score} \\:&=\\: TF * IDF\\\\\nTF \\:&=\\: \\frac{\\text{term frequency in document}}{\\text{total words in document}}\\\\\nIDF \\:&=\\: \\log\\left(\\frac{\\text{total number of documents}}{\\text{documents with term}}\\right)\\\\\n \\end{align}\n \n \n TF (Term frequency) : Reward words having high occurrence in a document [Frequent] \n \n IDF (Inverse Document Frequency) : Penalize words appearing many times in a document collection. Too general words should not have have high weight eg. \"or\" \"not\" \"is\" \"the\" [Reward Rarity]. \n \n\n\n\n```\n matrix = [\n [0.,0.46979139, 0.58028582, 0.38408524, 0., 0., 0.38408524, 0., 0.38408524],\n [0., 0.6876236, 0., 0.28108867, 0., 0.53864762, 0.28108867, 0., 0.28108867],\n [0.51184851, 0., 0., 0.26710379, 0.51184851,0., 0., 0.51184851, 0.26710379],\n [0., 0.46979139, 0.58028582, 0.38408524, 0., 0., 0.38408524, 0., 0.38408524]\n ]\n```\n\n### 3. sequence based representation \n \na document/observation is represented as a sequence of its words, each word could be represented as: \n* word embedding (word2vec, glove, etc.) - a learned representation for text where words that have the same meaning have a similar representation [- more in article](https://hackernoon.com/understanding-word-embeddings-a9ff830403ce)\n* one-hot vector (not recommended), \n\nMore on this topic later..."},{"metadata":{"_uuid":"154540e98c99087d74d31ae3f138aa8f0779553a"},"cell_type":"markdown","source":" ## Load data"},{"metadata":{"_uuid":"6399684dae6cdb2b34cb27e44ecfefb1bac82e7b"},"cell_type":"markdown","source":"Load data and divide into: \n**Train set** - 90%, data to train model, [1 mln] \n**Test set** - 5%, data to overview model performance during training and for hyperparameters tuning, [130 K] \n**Validation set** - 5%, data to estimate future performance of the model, when it will be working with new unseen spo far data, [130 K]"},{"metadata":{"trusted":true,"_uuid":"35489590e86bbb2befb9ccb8be18c9cb4c906142"},"cell_type":"code","source":"#set seed\nseed = 1029\n\n#import data\nimport pandas as pd\ntrain = pd.read_csv('../input/train.csv')\n\n#divide data info train (90%), test (5%) and valid (5%) set\nfrom sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(train[['question_text']], train[['target']], \n test_size=0.2, random_state=seed,\n stratify=train['target'].tolist(),\n shuffle = True)\nX_test, X_valid, y_test, y_valid = train_test_split(X_test, y_test, test_size=0.5, random_state=seed,\n stratify=y_test,shuffle = True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"965f702d75f1ec62a207b6e0f16e9ae632e69dee"},"cell_type":"markdown","source":"Quick peek on data:"},{"metadata":{"trusted":true,"_uuid":"605cb121b891e4479f15807cf2a922f73b6ae119"},"cell_type":"code","source":"train.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b5f86293bcb448fb0c232d636d0128d4cdd3d69d"},"cell_type":"code","source":"#clean up\nimport gc \nimport time \n\ndel(train)\ngc.collect()\ntime.sleep(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0a555792e49d6b0c6072c87eeef02132573527c3"},"cell_type":"code","source":"import matplotlib.pyplot as plt #for vizualization of data\nfrom pylab import rcParams\nimport numpy as np\nfrom collections import Counter\n%matplotlib inline\n\ncmap = plt.get_cmap(\"tab20c\")\ncolors = cmap((np.arange(10)))\n\nrcParams['figure.figsize'] = 20, 5\nfig, ax = plt.subplots(1, 4, sharex='col', sharey='row')\n\nax[0].pie([X_train.shape[0],X_test.shape[0],X_valid.shape[0]], explode=(0, 0.1,0.1), \n labels= [\"train \\n{}mln\".format(round(X_train.shape[0]/1000000,2)), \"test\\n{}K\".format(round(X_test.shape[0]/1000,1)),\"valid\\n{}K\".format(round(X_valid.shape[0]/1000,1))], autopct='%1.0f%%',\n shadow=True, startangle=75, colors=colors)\nax[0].axis('equal') # Equal aspect ratio ensures that pie is drawn as a circle.\nax[0].set_title('Data splits \\n', fontsize=20)\n\nax[1].pie(Counter(y_train.target).values(), explode=(0,0.1), labels= [\"sincere\",\"insincere\"], autopct='%1.0f%%',\n shadow=True, startangle=45, colors=colors)\nax[1].axis('equal') # Equal aspect ratio ensures that pie is drawn as a circle.\nax[1].set_title('Train set \\n', fontsize=20)\n\nax[2].pie(list(Counter(y_test.target).values())[::-1], explode=(0,0.1), labels= [\"sincere\",\"insincere\"], autopct='%1.0f%%',\n shadow=True, startangle=45, colors=colors)\nax[2].axis('equal') # Equal aspect ratio ensures that pie is drawn as a circle.\nax[2].set_title('Test set \\n', fontsize=20)\n\nax[3].pie(Counter(y_valid.target).values(), explode=(0,0.1), labels= [\"sincere\",\"insincere\"], autopct='%1.0f%%',\n shadow=True, startangle=45, colors=colors)\nax[3].axis('equal') # Equal aspect ratio ensures that pie is drawn as a circle.\nax[3].set_title('Valid set \\n', fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b34a83232d627d75e328c9f1b04a27633ddb50e2"},"cell_type":"markdown","source":"## Let's start building classifiers"},{"metadata":{"_uuid":"d5c72ff2c24c1d13b263eb57b834f09dd4ac9321"},"cell_type":"markdown","source":"### 1. Random guessing "},{"metadata":{"_uuid":"7ff0a2fc6fac76cc01f66ba4980bb8f33bb4bd58"},"cell_type":"markdown","source":"Just assign a random value **0** or **1** to the output and check the score. \nWe do this to get a reference how our evaluation metrics work. \n\n\n \n\n"},{"metadata":{"trusted":true,"_uuid":"dfa05ce94fa3c6db47816521e508780274d8f19f"},"cell_type":"code","source":"import time\nstart = time.time()\nimport numpy as np\n#y_pred = list(np.repeat(0.99,len(y_valid)))\nnp.random.seed(1029)\ny_pred = list(np.random.uniform(size=len(y_valid)))\nstop = time.time()\ntime_elapsed = stop-start","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"10f01f3784ba2a0a9a364b40f219545a9f7a45a9"},"cell_type":"code","source":"from sklearn.metrics import f1_score, accuracy_score\n\ndef bestThressholdF1(y_train_,train_preds_):\n tmp = [0,0,0] # idx, cur, max\n delta = 0\n for tmp[0] in np.arange(0.1, 0.501, 0.01):\n tmp[1] = f1_score(y_train_, np.array(train_preds_)>tmp[0])\n if tmp[1] > tmp[2]:\n delta = tmp[0]\n tmp[2] = tmp[1]\n return tmp[2]\n\ndef bestThressholdACC(y_train_,train_preds_):\n tmp = [0,0,0] # idx, cur, max\n delta = 0\n for tmp[0] in np.arange(0.1, 0.501, 0.01):\n tmp[1] = accuracy_score(y_train_, np.array(train_preds_)>tmp[0])\n if tmp[1] > tmp[2]:\n delta = tmp[0]\n tmp[2] = tmp[1]\n return tmp[2]\n\nfrom sklearn import metrics\n\ndef get_scores(y_train__,train_preds__):\n dict_ = {}\n dict_['F1']=bestThressholdF1(y_train__,train_preds__)\n dict_['accuracy'] = bestThressholdACC(y_train__,train_preds__)\n fpr, tpr, thresholds = metrics.roc_curve(y_train__,train_preds__, pos_label=1)\n dict_['auc']=metrics.auc(fpr, tpr)\n dict_['gini'] = (metrics.auc(fpr, tpr)-0.5)*2\n return dict_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"53d95cead8178f1b9a5cef0cb7cf6b47efdaf411"},"cell_type":"code","source":"scores = []\nscores.append(('Random guessing',get_scores(y_valid.target,y_pred), y_pred, \n time.strftime(\"%Mmin %Ssec\", time.gmtime(time_elapsed)),\n 0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c5db5d4af64ed2328c6b0766ed943cf8815d8ee7"},"cell_type":"code","source":"scores[-1][1]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba6f036a4a72ae53f2d3b02991422951e5edd82e"},"cell_type":"markdown","source":"* **F1 score** - this metric was used to assess resaults in competition, it is a harmonic mean of Precision and Recall, F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0\n \n\n \n\n* **accuracy** - what is the percentage of the correct predictions, perfect score is 100%, \n* **AUC** (Area Under Curve) is the area enclosed by the ROC curve. A perfect classifier has AUC = 1 and a completely random classifier has AUC = 0.5, \n* **GINI** - is 2*AUC – 1, and its purpose is to normalize the AUC so that a random classifier scores 0, and a perfect classifier scores 1. The range of possible Gini coefficient scores is [-1, 1].\n"},{"metadata":{"trusted":true,"_uuid":"4b4c544d5aa8638becfb030317540b8a13dd5236"},"cell_type":"code","source":"import gc \nimport time \n\ndel(y_pred)\ngc.collect()\ntime.sleep(2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"602c20961a2eafd926bb3873ce38a4e855f12140"},"cell_type":"markdown","source":"## 2. Manual feature engineering - Ridge Logistic Regression"},{"metadata":{"_uuid":"0f36931d43f9160d8c3c20ae62abd144e0ff6ad9"},"cell_type":"markdown","source":" \n"},{"metadata":{"_uuid":"11bb61e4df342ecb1d5903a14a450da9c3263de0"},"cell_type":"markdown","source":"Create some features based on text by hand:\n* text length,\n* number of words,\n* density,\n* number of title words,\n* number of capital words,\n* number of capital words / number of words,\n* number of unique words,\n* number of unique words / number of words,"},{"metadata":{"_uuid":"103e8a4cf5f5a045982ead441ea18387c160d803"},"cell_type":"markdown","source":"We will be using some great [scikit-learn](https://scikit-learn.org/stable/index.html) features : \n\n* [Pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) - it allows to make a data flow pipeline,\n* [custom transformer](https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer.html#sphx-glr-auto-examples-compose-plot-column-transformer-py) - need this to create custom transformer that apply custom functions on data,\n* [ColumnTransformer](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer) - need this to operate on pandas dataframe columns, \n\n\nModel: Ridge Logistic Regression"},{"metadata":{"_uuid":"daf92050de03e187203ed48e816e62bfc0ce01e2"},"cell_type":"markdown","source":"Define functions that calculates features listed above:"},{"metadata":{"trusted":true,"_uuid":"ef5460f7fb27dc01449ac7c9262d5cc56e03241a"},"cell_type":"code","source":"def text_length(_list):\n return list(map(lambda x: len(x),_list))\n\ndef text_words(_list):\n return list(map(lambda x: len(x.split()),_list))\n\ndef text_density(_list):\n return np.array(text_words(_list))/np.array(text_length(_list))\n\ndef text_title_words(_list):\n return list(map(lambda x: sum([w.istitle() for w in x.split()]),_list))\n\ndef text_capital_words(_list):\n return list(map(lambda x: sum(1 for c in x if c.isupper()),_list))\n\ndef text_caps_vs_length(_list):\n return list(map(lambda x: sum(1 for c in x if c.isupper())/len(x),_list))\n\ndef text_unique(_list):\n return list(map(lambda x: len(set(w for w in x.split())),_list))\n\ndef text_words_vs_unique(_list):\n return list(map(lambda x: len(set(w for w in x.split()))/len(x.split()),_list))\n\nfrom nltk.corpus import stopwords\nstopWords = set(stopwords.words('english'))\n\ndef text_stopwords(_list):\n return list(map(lambda x: sum([1 if i in stopWords else 0 for i in x.split()]),_list))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"50a48f5a23557dfc9448f7405a4075adfc14dd77"},"cell_type":"markdown","source":"Building a custom stateless scikit-learn transformer that take a function and apply it on data."},{"metadata":{"trusted":true,"_uuid":"9978c19849c854b1d3923c8f2b33ac2fac040964"},"cell_type":"code","source":"from sklearn.base import BaseEstimator, TransformerMixin\n\nclass ApplyFunctionCT(BaseEstimator, TransformerMixin):\n \n def __init__(self, function, **kwargs):\n self.function = function\n self.kwargs = kwargs\n \n def fit(self, x, y = None):\n return self\n \n def get_feature_names(self):\n if hasattr(self, \"columnNames\"):\n return self.columnNames\n else:\n return None \n \n def transform(self, x):\n if len(self.kwargs) == 0:\n wyn = x.apply(self.function)\n else:\n wyn = x.apply(self.function, param = self.kwargs)\n self.columnNames = wyn.columns\n return wyn","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f68766bef2eed43a4be2f124b16331ace009f06c"},"cell_type":"markdown","source":"Define dataflow pipeline:"},{"metadata":{"trusted":true,"_uuid":"6ccda88565f57621f2661e7a2d05468b26c28f88"},"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\nfrom sklearn.compose import ColumnTransformer, make_column_transformer\nfrom sklearn.preprocessing import StandardScaler, MinMaxScaler, Normalizer,RobustScaler, MaxAbsScaler\nfrom sklearn.pipeline import make_pipeline, make_union\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.feature_selection import SelectFromModel\nfrom sklearn.preprocessing import KBinsDiscretizer\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.feature_selection import SelectFromModel\nimport category_encoders as ce\n\npipe = make_pipeline(\n make_pipeline(\n ColumnTransformer([\n ('text_length', ApplyFunctionCT(function=text_length),['question_text']),\n ('text_words', ApplyFunctionCT(function=text_words),['question_text']),\n ('text_density', ApplyFunctionCT(function=text_density),['question_text']),\n ('text_title_words', ApplyFunctionCT(function=text_title_words),['question_text']),\n ('text_capital_words', ApplyFunctionCT(function=text_capital_words),['question_text']),\n ('text_caps_vs_length', ApplyFunctionCT(function=text_caps_vs_length),['question_text']),\n ('text_unique', ApplyFunctionCT(function=text_unique),['question_text']),\n ('text_words_vs_unique', ApplyFunctionCT(function=text_words_vs_unique),['question_text']),\n ]),\n KBinsDiscretizer(n_bins=10, encode='ordinal', strategy='quantile'), \n ),\n LogisticRegression(penalty = 'l2', C = 0.2, random_state=seed, solver = 'lbfgs',max_iter=400, \n verbose=1, n_jobs=-1) \n)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"90633e81889035a910a12ae9158b02e8d955aaf0"},"cell_type":"markdown","source":"Train model:"},{"metadata":{"trusted":true,"_uuid":"6dd93a8a3b69213a49919b20495147eb775685cf"},"cell_type":"code","source":"import time\nstart = time.time()\n\npipe.fit(X_train, y_train.target)\n\nstop = time.time()\ntime_elapsed = stop-start","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"20589756b3e595b5c46e4e0e6258502927dc9b2a"},"cell_type":"markdown","source":"Evaluate model:"},{"metadata":{"trusted":true,"_uuid":"62463dce62ef287ac6ace4ca9ac7ac2ef6f6e2e9"},"cell_type":"code","source":"import warnings\nwarnings.simplefilter(\"ignore\")\n\ny_predict = pipe.predict_proba(X_valid)[:,1]\nscores.append(('Manual feature engineering',get_scores(y_valid,y_predict),y_predict, \n time.strftime(\"%M:%S\", time.gmtime(time_elapsed)),\n (pipe.named_steps['logisticregression'].coef_).size))\n\nprint(scores[-1][1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9dba96c7df106e17c1b0fbcce68eb7c8bd00ae76"},"cell_type":"markdown","source":"Define function that print progess"},{"metadata":{"trusted":true,"_uuid":"4a1cff9f791a9cc436aa574d11709778a30c4a46"},"cell_type":"code","source":"import matplotlib.pyplot as plt #for vizualization of data\n%matplotlib inline\n\nfrom pylab import rcParams\nimport numpy as np\nfrom collections import Counter\nimport seaborn as sns\n\ndef plot_progress(list_):\n fig, ax = plt.subplots(figsize=(14, 6))\n #sns.set_style(\"whitegrid\")\n\n plt.plot(list(range(len(list_))), [i[1]['gini'] for i in list_], '-ok')\n for j,i in enumerate(zip([i[1]['gini'] for i in list_],\n list(range(len(list_))),\n [i[0] for i in list_],\n [i[3] for i in list_],\n [i[4] for i in list_])):\n ax.text(i[1], i[0]-0.08, \n i[2]+',\\nG: '+str(round(i[0],3))+', t: '+i[3],rotation=-25, size=10, color = ['black','blue'][j%2])\n\n ax.patch.set_facecolor('white')\n ax.grid(color='grey', linestyle='-', linewidth=0.25, alpha=0.5)\n\n ax.spines['bottom'].set_color('0.5')\n ax.spines['top'].set_color(None)\n ax.spines['left'].set_color('0.5')\n ax.spines['right'].set_color(None)\n\n plt.title('Progress Tracker', fontsize=12)\n plt.xlabel('Models', fontsize=11)\n plt.ylabel('Gini', fontsize=11)\n plt.xticks(fontsize=9)\n plt.yticks(fontsize=9)\n pass\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"82f6d55dcf854e9d72717d0f9d503c5917e3b062"},"cell_type":"code","source":"plot_progress(scores)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5795820cb39e249d8ba37f7fee888791c6c57a72"},"cell_type":"code","source":"import gc \nimport time \n\ndel(pipe, y_predict)\ngc.collect()\ntime.sleep(2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"26ee2d3e907ce6269860c6204568eee1372add6e"},"cell_type":"markdown","source":"## 3. Frequency based embedding, TF-IDF Vectorizer - Ridge Logistic Regression"},{"metadata":{"_uuid":"9819286e4668a32600c860733ed003551732b397"},"cell_type":"markdown","source":" "},{"metadata":{"_uuid":"c2633e943ee1f81153b81f00a236eb9219772e4b"},"cell_type":"markdown","source":"We are using here [TfidfVectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html) - a scikit-learn module that clean, tokenize and produce a ready to use sparse matrix with TF-IDF score. \n\nModel: Ridge Logistic Regression"},{"metadata":{"trusted":true,"_uuid":"d9fdad384f8a7b02c62b197bb1dd6d285513e319","scrolled":true},"cell_type":"code","source":"max_features_Vectorizer = None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4daa7d89e8b7ec2179c7af790b31c3ecf852d974"},"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\nfrom sklearn.compose import ColumnTransformer, make_column_transformer\nfrom sklearn.preprocessing import StandardScaler, MinMaxScaler, Normalizer,RobustScaler, MaxAbsScaler\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.linear_model import LogisticRegression\n\npipe = make_pipeline(\n ColumnTransformer([\n ('CV', TfidfVectorizer(lowercase=True, \n ngram_range=(1, 1), \n max_features=max_features_Vectorizer, \n dtype=np.float32,\n use_idf=True),'question_text')]),\n LogisticRegression(penalty = 'l2', C = 2, random_state=seed, solver = 'lbfgs',max_iter=400, \n verbose=1, n_jobs=-1)\n )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0c32336315ef67de524f899351372227112d371e","scrolled":false},"cell_type":"code","source":"import time\nstart = time.time()\n\npipe.fit(X_train, y_train.target)\n\nstop = time.time()\ntime_elapsed = stop-start","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d17307d4ec0606769a7cbafac70936366aeeeaf8"},"cell_type":"code","source":"#x = pipe.named_steps['columntransformer'].transform(X_train)\n#print(\"Sparsity equals {}- the number of zero-valued elements divided by the total number of elements\".format(\n# ((x.shape[0]*x.shape[1]) -x.getnnz())/(x.shape[0]*x.shape[1])))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2dec47da514cd9cb3e4f07a6e866cce8e0d8cad5"},"cell_type":"markdown","source":"Sparsity equals 0.9999332042305805 - the number of zero-valued elements divided by the total number of elements\n"},{"metadata":{"trusted":true,"_uuid":"bc3ebbd1131f9f798d5f119e53c79638a1aa6c8b"},"cell_type":"code","source":"y_predict = pipe.predict_proba(X_valid)[:,1]\nscores.append(('TF-IDF Vectorizer, Ridge Regression',get_scores(y_valid,y_predict),y_predict,\n time.strftime(\"%M:%S\", time.gmtime(time_elapsed)),\n (pipe.named_steps['logisticregression'].coef_).size))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7e4e5996b458690b32ad91ca5d7c8b0d70a006a1"},"cell_type":"code","source":"scores[-1][1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a23794ea8ce956d259bd7a4acbd575ba6a61cb56"},"cell_type":"code","source":"plot_progress(scores)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5bf25bda57de58a2756e400a36bc0153159297e4"},"cell_type":"code","source":"import gc \nimport time \n\ndel(pipe, y_predict)\ngc.collect()\ntime.sleep(2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b1a1a1e7144833fca0a8e1de6ef3584eb65e5bdb"},"cell_type":"markdown","source":"## 4. Frequency based embedding, MLP in Keras (The Sequential model API)"},{"metadata":{"_uuid":"8dca022a33428b1ad0deaae73f37cd396b3d22bd"},"cell_type":"markdown","source":" "},{"metadata":{"_uuid":"ef65edcd40560ad276513d4147324e0a844720ee"},"cell_type":"markdown","source":"Now instead of Ridge Logistic Regression we use Multilayer Perceptron Classifier, build on Keras Sequential model Scikit-Learn wrapper. \nFrom here we start using GPU..."},{"metadata":{"trusted":true,"_uuid":"218d2d59f570ec0dd53252d72244a27d4eab9f44"},"cell_type":"code","source":"max_features_ = 10000","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6d53ae56ae1bf7aa1946a3aa96bc3291944d70f9"},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Dense, Activation, InputLayer, BatchNormalization, Dropout\nfrom keras.wrappers.scikit_learn import KerasClassifier\nfrom keras.initializers import glorot_normal\n\n# For custom metrics\nimport keras.backend as K\n\ndef create_model():\n model = Sequential([\n Dense(units=192,input_dim=max_features_,kernel_initializer=glorot_normal(seed=seed)),\n BatchNormalization(),\n Activation('relu'),\n Dropout(0.2,seed=seed),\n Dense(units=64,input_dim=max_features_,kernel_initializer=glorot_normal(seed=seed)),\n BatchNormalization(),\n Activation('relu'),\n Dropout(0.2,seed=seed),\n Dense(units=32,input_dim=max_features_,kernel_initializer=glorot_normal(seed=seed)),\n BatchNormalization(),\n Activation('relu'),\n Dropout(0.2,seed=seed),\n Dense(1),\n Activation('sigmoid'),\n ])\n model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"782434eeabfb94eae48fcf1db21ece1383969039"},"cell_type":"markdown","source":"Model summary:"},{"metadata":{"trusted":true,"_uuid":"8a87ca938be27bcb36daf71c3479db892af82b6c"},"cell_type":"code","source":"create_model().summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"13ac890745eec404927f90484a0eafc90699564c"},"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\nfrom sklearn.compose import ColumnTransformer, make_column_transformer\nfrom sklearn.preprocessing import StandardScaler, MinMaxScaler, Normalizer,RobustScaler, MaxAbsScaler\nfrom sklearn.pipeline import make_pipeline\n\npipe = make_pipeline(\n ColumnTransformer([\n ('CV', TfidfVectorizer(lowercase=True, ngram_range=(1, 1), max_features=max_features_, dtype=np.float32,\n use_idf=True),'question_text')]),\n KerasClassifier(build_fn=create_model, epochs=3, batch_size=512, verbose=1)\n )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"189b43d7be500f71d915f921407cf3d6ef709892"},"cell_type":"code","source":"import time\nstart = time.time()\n\npipe.fit(X_train, y_train.target)\n\nstop = time.time()\ntime_elapsed = stop-start","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b2802db574669ec97f372b76da101afbb8dbc466"},"cell_type":"code","source":"y_predict = pipe.predict_proba(X_valid)[:,1]\nscores.append(('TF-IDF Vectorizer, Keras MLP',get_scores(y_valid,y_predict),y_predict,\n time.strftime(\"%M:%S\", time.gmtime(time_elapsed)),\n pipe.named_steps['kerasclassifier'].model.count_params()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ff185538fbee4e910d325cddf06649eeb8b24af1"},"cell_type":"code","source":"scores[-1][1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dcec88a01cea03fb4487b7b02e41880a5e08a317","scrolled":false},"cell_type":"code","source":"plot_progress(scores)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1335c8a539f54f0afc85ffd67f4440367784f7d9"},"cell_type":"markdown","source":"## 5. Sequence Models"},{"metadata":{"_uuid":"7b80c317df3a588fb2c28909c211048a81a501d4"},"cell_type":"markdown","source":"### Create sequences from text"},{"metadata":{"_uuid":"5d4f328310f378af92eef60f82b960aedc2c6c1e"},"cell_type":"markdown","source":"Each type of recurrent neural network (RNN, LSTM, GRU) require text dataset transformed into 3D tensor. \n\n \n\nPlease notice that each word is a vector of numbers (from pretarained embedding), we don't know what exactly specific number means but to building an intuition we could imagine that it's ex: gender, age, red, speed, scent. \nSo, car and motorbike will have similar \"speed\" feature, but for turtle this feature will be completely different, tomato and orange will have much closer value of \"red\" feature then garlic, and so on. ","attachments":{}},{"metadata":{"_uuid":"15edbe3135806d76ff5314d16a2de1a8ee531949"},"cell_type":"markdown","source":"### To prepare text in a form of 3D tensor we need apply following steps:"},{"metadata":{"_uuid":"3896ca7aba7df0d2b6011d93dca1098939268091"},"cell_type":"markdown","source":"Each type of recurrent neural network (RNN) require fixed-length input (length of text). \nWhen we are starting to prepare data for RNN we need to analize data and chose max text length. \nIt's good to limit number of unique words to - this affects computation time."},{"metadata":{"trusted":true,"_uuid":"61048905b1c5cde4616b2270ccff777eab343284"},"cell_type":"code","source":"#settings \nmaxlen = 70 # max number of words in a question to use\nmax_features = 120000 # how many unique words to use (i.e num rows in embedding vector)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e9ba8f1a6e34462247c7e28e734192a41451f8d2"},"cell_type":"markdown","source":"Next we need to do apply text cleaning and word tokenization. \nI recommend to use Keras Tokenizer, which makes it in fast and simple way - I've try many others approaches but Keras Tokenizer is the best so far."},{"metadata":{"trusted":true,"_uuid":"868ae78cabe432be3dcb7976d81688954a9a62e5"},"cell_type":"code","source":"from keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\n## Tokenize the sentences\ntokenizer = Tokenizer(num_words=max_features,\n filters='!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n', \n lower=True, \n split=' ', \n char_level=False, \n oov_token=None, \n document_count=0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"63877b0c91c21ab2edfc972b09540777879bc0ca"},"cell_type":"markdown","source":"Here's some main parameters of ```keras.preprocessing.text.Tokenizer```\n* **num_words**: the maximum number of words to keep, based on word frequency. Only the most common num_words words will be kept. \n* **filters**: a string where each element is a character that will be filtered from the texts. The default is all punctuation, plus tabs and line breaks, minus the ' character. '!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n' \n* **lower**: boolean. Whether to convert the texts to lowercase. \n* **split**: str. Separator for word splitting. \n* **char_level**: if True, every character will be treated as a token. \n* **oov_token**: if given, it will be added to word_index and used to replace out-of-vocabulary words during text_to_sequence calls"},{"metadata":{"trusted":true,"_uuid":"640d72fc1bebb7ba51c5fdf41d5a2bea3f55c799"},"cell_type":"code","source":"tokenizer.fit_on_texts(X_train.question_text.tolist())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7387306dfee38a0f651a999d172b80ddaa17a591"},"cell_type":"markdown","source":"To represent text in sequential way and use it to train a model, we need 3 elements: \n* **text represented as a list of numbers**, where each number represents specific word, \n* **word_index dictionary**- a dictionary that translates numbers to words, \n* **word embedding** - a numeric word representation: either dense word embedding (ex. GloVe) or one-hot encoded,\n"},{"metadata":{"_uuid":"83f8b45ecfbc6ec3036fbc8e4158570e7f9d14a3"},"cell_type":"markdown","source":"Keras Tokenizer can prepapare word index for you."},{"metadata":{"trusted":true,"_uuid":"d29695afbff4be0622c4ac3a41374b49167ad403"},"cell_type":"code","source":"word_index = tokenizer.word_index","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"75501c8d807a81dc472683af99e9effbc1f26fb2"},"cell_type":"code","source":"im = 10\na = {}\nfor j,i in enumerate(tokenizer.word_index):\n a[i]=tokenizer.word_index[i]\n if j>=im:\n break\na","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f3e2e57c3a863d9670056883b07e320fd27f2a8"},"cell_type":"markdown","source":"Keras Tokenizer can translate a text to list of numbers - mapped to word_index,"},{"metadata":{"trusted":true,"_uuid":"712cac74bdeef9e02f309e03df0c41650630b4d6"},"cell_type":"code","source":"X_train_seq = tokenizer.texts_to_sequences(X_train.question_text.tolist())\nX_test_seq = tokenizer.texts_to_sequences(X_test.question_text.tolist())\nX_valid_seq = tokenizer.texts_to_sequences(X_valid.question_text.tolist())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6ef4d8db387bf1dde04d50f93cd358b509215c7b"},"cell_type":"code","source":"print(X_train.question_text.tolist()[0])\nprint(X_train_seq[0])\nprint('------------------------------------------------------------------------')\nprint(X_train.question_text.tolist()[100])\nprint(X_train_seq[100])\nprint('------------------------------------------------------------------------')\nprint(X_train.question_text.tolist()[202])\nprint(X_train_seq[202])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"23af0b9e1c763d5131e2f20965e5bed0759a88cb"},"cell_type":"markdown","source":"To assure that each text has a fixed size we need to apply sequence padding. We can use Keras pad_sequences method."},{"metadata":{"trusted":true,"_uuid":"7b4d9166e16b070808c9d604a1c38749943bfb16"},"cell_type":"code","source":"X_train_seq = pad_sequences(X_train_seq, maxlen=maxlen)\nX_test_seq = pad_sequences(X_test_seq, maxlen=maxlen)\nX_valid_seq = pad_sequences(X_valid_seq, maxlen=maxlen)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4ee1eac45259529f2c5d5c55344844b70632af9e"},"cell_type":"code","source":"print(X_train.question_text.tolist()[0])\nprint(X_train_seq[0])\nprint('------------------------------------------------------------------------')\nprint(X_train.question_text.tolist()[100])\nprint(X_train_seq[100])\nprint('------------------------------------------------------------------------')\nprint(X_train.question_text.tolist()[202])\nprint(X_train_seq[202])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2adf9712cbf338c73aa6679ab900bdef37d2d01e"},"cell_type":"code","source":"import gc \nimport time \n\ndel(tokenizer)\ngc.collect()\ntime.sleep(2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5a12c11ea847f2f0eeee4f89b6cce06b0408cf27"},"cell_type":"markdown","source":"### Load GloVe: pretrained word embedding /pretrained word vectors."},{"metadata":{"_uuid":"fa94cfd3dddb87f9def0dd3c663057dd161be5c2"},"cell_type":"markdown","source":"Step by step:\n1. Load file with pretrained word vectors, \n2. Select only words that are in word_index dictionary, \n3. If there is not pretrained word vector for our word from word_index - we replace it with embedding average value,\n4. As a result we will get an 2D array,"},{"metadata":{"trusted":true,"_uuid":"8f1ae74832991f2af6c65e540ddec401b9b51133"},"cell_type":"code","source":"def load_glove(word_index, max_features__ = max_features):\n EMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\n def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')[:300]\n embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))\n \n all_embs = np.stack(embeddings_index.values())\n emb_mean,emb_std = -0.005838499,0.48782197\n embed_size = all_embs.shape[1]\n\n nb_words = min(max_features__, len(word_index))\n np.random.seed(1029)\n embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n for word, i in word_index.items():\n if i >= max_features__: continue\n embedding_vector = embeddings_index.get(word)\n if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n \n return embedding_matrix ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d6849a7218a49dd1efdecd290d0be640857aeb4b"},"cell_type":"code","source":"glove_embeddings = load_glove(word_index)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5e56a46dbbdb0f155d392fa5fee412fca9535d86"},"cell_type":"markdown","source":"Combining all things together: text represented as a list of numbers, word_index, word embedding and one additional layer (Keras Embedding) we produced 3D tensor. \n \n Let's have a look at embedding for sentence `the quick fast brown fox jumps over the lazy slow dog` (30 of 300 vecotors), \n\nCould you find a word features that have similar values for words `quick` and `fast`?"},{"metadata":{"trusted":true,"_uuid":"57a44105277fcb27511db432a1ee71b0e7696761"},"cell_type":"code","source":"import seaborn as sns; sns.set()\nsz = 30\nplt.figure(figsize=(9,9))\nplt.title('Embedding for sentence `the quick fast brown fox jumps over the lazy slow dog` ({} of 300 vecotors) \\n'.format(str(sz)))\nax = sns.heatmap(pd.DataFrame(np.hstack([glove_embeddings[word_index['the'],:].reshape(-1,1),\n glove_embeddings[word_index['quick'],:].reshape(-1,1),\n glove_embeddings[word_index['fast'],:].reshape(-1,1),\n glove_embeddings[word_index['brown'],:].reshape(-1,1),\n glove_embeddings[word_index['fox'],:].reshape(-1,1),\n glove_embeddings[word_index['jumps'],:].reshape(-1,1),\n glove_embeddings[word_index['over'],:].reshape(-1,1),\n glove_embeddings[word_index['the'],:].reshape(-1,1),\n glove_embeddings[word_index['lazy'],:].reshape(-1,1),\n glove_embeddings[word_index['slow'],:].reshape(-1,1),\n glove_embeddings[word_index['dog'],:].reshape(-1,1) \n ])[:sz,:],columns = ['the','quick','fast','brown','fox','jumps','over','the','lazy','slow','dog']\n ,index = ['word feature {}'.format(str(i)) for i in range(sz)]\n ), \n cbar=False,annot=True,annot_kws={\"size\": 10})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2196d833e60a3ccfb17ef685e6d59f5d572219ec"},"cell_type":"markdown","source":"## 5.1 Simple RNN, Many to One (Keras Functional API)"},{"metadata":{"_uuid":"819b0be0a32ca5be485964c476493b2ddad14981"},"cell_type":"markdown","source":"\n \n"},{"metadata":{"_uuid":"8687220d1439e2f514222801923d2f2d7da7810b"},"cell_type":"markdown","source":"There is a lot to say about Recurrent Neural Networks, but to make a long story short: \nThink of it a s many single neural networks (NN) - as many as the fixed text length (in this case 70). \nEach of these single NN takes one vector that represent a word, makes some predictions on it and pass a prediction to the next NN, which concatenate the output of the previous NN + input word and makes some predictions on it. Then it passes it to the next NN and so on, and so on. \nThe one thing that you need to know is that all of this NN share the same weights - you will find more about it at the end of this notebook."},{"metadata":{"trusted":true,"_uuid":"3cea62e42f8d6f361aa88e1f9280dfaa079489c8"},"cell_type":"code","source":"#nb_words = len(word_index)+1\nnb_words = glove_embeddings.shape[0]\nWV_DIM = glove_embeddings.shape[1]\nmaxlen = maxlen\nSEED = 1029","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"26de323c4a25ef7b007f353c7252aea468b5dea7"},"cell_type":"code","source":"from keras import Input\nfrom keras.layers import Embedding, SpatialDropout1D, CuDNNLSTM, CuDNNGRU, Dropout, Dense, SimpleRNN\nfrom keras.layers import concatenate, Bidirectional, GlobalAveragePooling1D, GlobalMaxPooling1D\nfrom keras.initializers import glorot_normal, orthogonal\n# First layer\n# create input\nmain_input = Input(shape=(maxlen,), dtype='int32',name='main_input')\n# creating the embedding\nembedded_sequences = Embedding(input_dim = nb_words,\n output_dim = WV_DIM,\n mask_zero=False,\n weights=[glove_embeddings],\n input_length=maxlen,\n trainable=False)(main_input)\n#Second layer\nembedded_sequences = SpatialDropout1D(0.2, seed=seed)(embedded_sequences)\nx = SimpleRNN(64, return_sequences=False,\n kernel_initializer=glorot_normal(seed=seed),\n recurrent_initializer=orthogonal(seed=seed))(embedded_sequences)\n\n#output (batch, 64)\n#The input format should be three-dimensional: the three components represent sample size, number of time steps and output dimension\npreds = Dense(16, activation=\"relu\", kernel_initializer=glorot_normal(seed=seed))(x)\npreds = Dropout(0.1,seed=seed)(preds)\npreds = Dense(1, activation=\"sigmoid\", kernel_initializer=glorot_normal(seed=seed))(preds)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8e298737abd166b57e35057fe843932399cb5d93"},"cell_type":"code","source":"from keras.models import Model\nfrom keras.optimizers import Adam\nmodel = Model(inputs=main_input, outputs=preds)\nmodel.compile(loss='binary_crossentropy', optimizer=Adam(clipvalue=1), metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"96fb04ebf75a660e062246f550867ec0cf2f7ec9"},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"9955e6280b4a7d3b9eb2ef732b10adef535d6e20"},"cell_type":"code","source":"import time\nstart = time.time()\n\nhist = model.fit(X_train_seq, y_train, batch_size=1024, epochs=5, \n validation_data=(X_test_seq, y_test))\n\nstop = time.time()\ntime_elapsed = stop-start","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5d100c8d2558ba24c4dc7d75c25b146ecef9eae7"},"cell_type":"code","source":"pred_val = model.predict(X_valid_seq, batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ee0687b88385b3967768d13c8399b666c06a1444"},"cell_type":"code","source":"scores.append(('RNN, Many to One',get_scores(y_valid.target.tolist(),list(pred_val[:,0])),list(pred_val[:,0]),\n time.strftime(\"%M:%S\", time.gmtime(time_elapsed)),\n model.count_params()\n ))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fc16faf71ee0c8880e9507d7222c2ef4bce339c7"},"cell_type":"code","source":"scores[-1][1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d3cc7bf397f56dc76ec1707f4d2d46a6a2c06f07"},"cell_type":"code","source":"plot_progress(scores)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1221028855d0129bbeb2aa4e879dc57db8ec0366"},"cell_type":"code","source":"import gc \nimport time \n\ndel(model, hist, pred_val)\ngc.collect()\ntime.sleep(2)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8018c14e1bab3cab321739a59b41fc2cf42c286a"},"cell_type":"markdown","source":"## 5.2 LSTM, Many to One"},{"metadata":{"_uuid":"d406ae7dffcc4bc320686bbae371681a91ddd6b9"},"cell_type":"markdown","source":"\n"},{"metadata":{"_uuid":"9c440fbb98ccea28d822a8c7b5cd8332c71806cb"},"cell_type":"markdown","source":"Now we try to use LSTM instead of RNN. LSTM prevent the vanishing gradient problem by applying solution to better \"remember\" things from the past time steps. \nMore about the differences at the end of the notebook."},{"metadata":{"trusted":true,"_uuid":"3ca458d67e8be8261f641894e72858338509f9d5"},"cell_type":"code","source":"from keras import Input\nfrom keras.layers import Embedding, SpatialDropout1D, CuDNNLSTM, CuDNNGRU, Dropout, Dense, SimpleRNN\nfrom keras.layers import concatenate, Bidirectional, GlobalAveragePooling1D, GlobalMaxPooling1D\nfrom keras.initializers import glorot_normal, orthogonal\n# First layer\n# create input\nmain_input = Input(shape=(maxlen,), dtype='int32',name='main_input')\n# creating the embedding\nembedded_sequences = Embedding(input_dim = nb_words,\n output_dim = WV_DIM,\n mask_zero=False,\n weights=[glove_embeddings],\n input_length=maxlen,\n trainable=False)(main_input)\n#Second layer\nembedded_sequences = SpatialDropout1D(0.2, seed=SEED)(embedded_sequences)\nx = CuDNNLSTM(64, return_sequences=False,\n kernel_initializer=glorot_normal(seed=seed),\n recurrent_initializer=orthogonal(seed=seed))(embedded_sequences)\n\n#output (batch, 64)\n#The input format should be three-dimensional: the three components represent sample size, number of time steps and output dimension\npreds = Dense(16, activation=\"relu\", kernel_initializer=glorot_normal(seed=seed))(x)\npreds = Dropout(0.1,seed=seed)(preds)\npreds = Dense(1, activation=\"sigmoid\", kernel_initializer=glorot_normal(seed=seed))(preds)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"82421347e70326712e3818368b3f48deb9282f21"},"cell_type":"markdown","source":"we use clipvalue here"},{"metadata":{"trusted":true,"_uuid":"4a2c1ed74bd08ee47527bb5dda01f5de2fc37f8d"},"cell_type":"code","source":"from keras.models import Model\nfrom keras.optimizers import Adam\nmodel = Model(inputs=main_input, outputs=preds)\nmodel.compile(loss='binary_crossentropy', optimizer=Adam(clipvalue=1), metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"19134e404e03596eee07129b8be3f7ba98ec92b7","scrolled":true},"cell_type":"code","source":"import time\nstart = time.time()\n\nhist = model.fit(X_train_seq, y_train, batch_size=1024, epochs=5, \n validation_data=(X_test_seq, y_test))\n\nstop = time.time()\ntime_elapsed = stop-start","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"54d944fb610145de0b37069832ac0c79b7d72ea9"},"cell_type":"code","source":"pred_val = model.predict(X_valid_seq, batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b6ca699f03ca003c45a89c38f281a5f514d63e78"},"cell_type":"code","source":"scores.append(('LSTM, Many to One',get_scores(y_valid.target.tolist(),list(pred_val[:,0])),list(pred_val[:,0]),\n time.strftime(\"%M:%S\", time.gmtime(time_elapsed)),\n model.count_params()\n ))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"15518ed91d5dae2ad0deeb1ef79670199ccbeb47"},"cell_type":"code","source":"scores[-1][1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a37caf324b156fc2f5a816d780c47fa94b9ba001"},"cell_type":"code","source":"plot_progress(scores)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8821e917c12a6e5f2f3de25a5d8a067322443288"},"cell_type":"code","source":"import gc \nimport time \n\ndel(model, hist, pred_val)\ngc.collect()\ntime.sleep(2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"063ad76e17b73ed51e4f315ce3c0219f0826c858"},"cell_type":"markdown","source":"## 5.3 Bidirectional LSTM, Many to One"},{"metadata":{"_uuid":"5a27d0c75ac097ebe85b7e7d1c45e12d0bf3e85b"},"cell_type":"markdown","source":"\n\n\n\n"},{"metadata":{"_uuid":"ce6fa24a18fdbffaf9541d03d4a91bf2fc11a5b2"},"cell_type":"markdown","source":"We have nowe similar model to the last one, but instead of single LSTM we use Bidirectional LSTM. \n\n\"Bidirectional recurrent neural networks(RNN) are really just putting two independent RNNs together. The input sequence is fed in normal time order for one network, and in reverse time order for another. The outputs of the two networks are usually concatenated at each time step, though there are other options, e.g. summation.\""},{"metadata":{"trusted":true,"_uuid":"968d5c1d4fa963af6ed59d6efeebb483e3c48793"},"cell_type":"code","source":"from keras import Input\nfrom keras.layers import Embedding, SpatialDropout1D, CuDNNLSTM, CuDNNGRU, Dropout, Dense, SimpleRNN\nfrom keras.layers import concatenate, Bidirectional, GlobalAveragePooling1D, GlobalMaxPooling1D\nfrom keras.initializers import glorot_normal, orthogonal\n# First layer\n# create input\nmain_input = Input(shape=(maxlen,), dtype='int32',name='main_input')\n# creating the embedding\nembedded_sequences = Embedding(input_dim = nb_words,\n output_dim = WV_DIM,\n mask_zero=False,\n weights=[glove_embeddings],\n input_length=maxlen,\n trainable=False)(main_input)\n#Second layer\nembedded_sequences = SpatialDropout1D(0.2, seed=SEED)(embedded_sequences)\nx = Bidirectional(CuDNNLSTM(64, return_sequences=False,\n return_state = False,\n kernel_initializer=glorot_normal(seed=seed),\n recurrent_initializer=orthogonal(seed=seed)))(embedded_sequences)\n\n#The input format should be three-dimensional: the three components represent sample size, number of time steps and output dimension\npreds = Dense(16, activation=\"relu\", kernel_initializer=glorot_normal(seed=seed))(x)\npreds = Dropout(0.1,seed=seed)(preds)\npreds = Dense(1, activation=\"sigmoid\", kernel_initializer=glorot_normal(seed=seed))(preds)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"67e989f9fef0db7506b9baed6b19f1fd7e1b5c8d"},"cell_type":"code","source":"from keras.models import Model\nfrom keras.optimizers import Adam\nmodel = Model(inputs=main_input, outputs=preds)\nmodel.compile(loss='binary_crossentropy', optimizer=Adam(clipvalue=1), metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3f9f930cee5f4c909630c9ff6651e0e266dd4a69"},"cell_type":"code","source":"import time\nstart = time.time()\n\nhist = model.fit(X_train_seq, y_train, batch_size=1024, epochs=5, \n validation_data=(X_test_seq, y_test))\n\nstop = time.time()\ntime_elapsed = stop-start","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c297a54cfe15bee2dfc6858fa5e2c29f367ff7ec"},"cell_type":"code","source":"pred_val = model.predict(X_valid_seq, batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"036fc8c643c580451e8dd1ae8edf73e667c12ca5"},"cell_type":"code","source":"scores.append(('BiLSTM, Many to One',get_scores(y_valid.target.tolist(),list(pred_val[:,0])),\n list(pred_val[:,0]),\n time.strftime(\"%M:%S\", time.gmtime(time_elapsed)),\n model.count_params()\n ))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"70a52af067a11f4bf4b4c3bcac8e3d225af43987"},"cell_type":"code","source":"scores[-1][1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e9d2cca2420b30323d322e990c55cb9bacb83a00"},"cell_type":"code","source":"plot_progress(scores)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1e1c3c08093d7dd0eb2b6096e51d450b54e0c0de"},"cell_type":"code","source":"import gc \nimport time \n\ndel(model, hist, pred_val)\ngc.collect()\ntime.sleep(2)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cbb6197382812692018abfb36a7bbf9249e19d4f"},"cell_type":"markdown","source":"## 5.4 Bidirectional LSTM, Many to Many + Pooling"},{"metadata":{"_uuid":"f2adfe19fe84b871b4b80350d387494c74763698"},"cell_type":"markdown","source":"\n\n\n"},{"metadata":{"_uuid":"579a1f313dc9dae8667fc069ef90b6381ebd6ae2"},"cell_type":"markdown","source":"Instead of using only the last LSTM prediction we could use predictions from all the time steps to predict the score. \nThe widely used RNN structure for this is “Pooling Model”. \nThe pooling BiLSTM model can be regarded as each LSTM nodes voting for the feature layer. The “max pooling” is a form for voting, that's choose always the highest value. “Mean pooling” calculates the value of the i’th position in vector h by averaging the corresponding value of each hidden state vector from BLSTM nodes."},{"metadata":{"trusted":true,"_uuid":"af2e864732f4ad24d32cba320d924921f1f09f9b"},"cell_type":"code","source":"from keras import Input\nfrom keras.layers import Embedding, SpatialDropout1D, CuDNNLSTM, CuDNNGRU, Dropout, Dense, SimpleRNN\nfrom keras.layers import concatenate, Bidirectional, GlobalAveragePooling1D, GlobalMaxPooling1D\nfrom keras.initializers import glorot_normal, orthogonal\n# First layer\n# create input\nmain_input = Input(shape=(maxlen,), dtype='int32',name='main_input')\n# creating the embedding\nembedded_sequences = Embedding(input_dim = nb_words,\n output_dim = WV_DIM,\n mask_zero=False,\n weights=[glove_embeddings],\n input_length=maxlen,\n trainable=False)(main_input)\n#Second layer\nembedded_sequences = SpatialDropout1D(0.2, seed=SEED)(embedded_sequences)\nx, forward_h, forward_c, backward_h, backward_c = Bidirectional(CuDNNLSTM(64, return_sequences=True,return_state = True,\n kernel_initializer=glorot_normal(seed=seed),\n recurrent_initializer=orthogonal(seed=seed)))(embedded_sequences)\nstate_h = concatenate([forward_h, backward_h])\nstate_c = concatenate([forward_c, backward_c])\n\n#x (?, 70, 128)\n#forward_h (?, 64)\n#forward_c (?, 64)\n#backward_h (?, 64)\n#backward_c (?, 64)\n\navg_pool = GlobalAveragePooling1D()(x)\n#avg_poll = (?, 128)\nmax_pool = GlobalMaxPooling1D()(x)\n#max_pool = (?, 128)\n\nconc = concatenate([state_h, avg_pool, max_pool])\n#conc (?, 384)\n\n#The input format should be three-dimensional: the three components represent sample size, number of time steps and output dimension\npreds = Dense(16, activation=\"relu\", kernel_initializer=glorot_normal(seed=seed))(conc)\npreds = Dropout(0.1,seed=seed)(preds)\npreds = Dense(1, activation=\"sigmoid\", kernel_initializer=glorot_normal(seed=seed))(preds)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f5db5a29e73f364d3d306642c71bf03d9d52c046"},"cell_type":"code","source":"from keras.models import Model\nfrom keras.optimizers import Adam\nmodel = Model(inputs=main_input, outputs=preds)\nmodel.compile(loss='binary_crossentropy', optimizer=Adam(clipvalue=1), metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3a63e99ff53cb099a457057f0fe71a71d8de77fe"},"cell_type":"code","source":"import time\nstart = time.time()\n\nhist = model.fit(X_train_seq, y_train, batch_size=1024, epochs=5, \n validation_data=(X_test_seq, y_test))\n\nstop = time.time()\ntime_elapsed = stop-start","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4d7c7dc485e032f46340b828d55e9ea1d5bea49b"},"cell_type":"code","source":"pred_val = model.predict(X_valid_seq, batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"342422b891755712cc122ddc4058ad6eb12ea0bc"},"cell_type":"code","source":"scores.append(('BiLSTM, Many to Many, Max Pooling',get_scores(y_valid.target.tolist(),list(pred_val[:,0])),\n list(pred_val[:,0]),\n time.strftime(\"%M:%S\", time.gmtime(time_elapsed)),\n model.count_params()\n ))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"84fa62b311390f88fd8a43c5feb59607fff14941"},"cell_type":"code","source":"scores[-1][1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f725edd82fc04133b12a85b52620830c1e5d8468"},"cell_type":"code","source":"plot_progress(scores)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2488cd0c8f8f38e44ca7d2ed175230ebde942790"},"cell_type":"code","source":"import gc \nimport time \n\ndel(model, hist, pred_val)\ngc.collect()\ntime.sleep(2)\n\n#get_scores(y_valid,np.average(np.vstack([i[2] for i in scores[2:]]),axis=0).reshape(-1))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"515e6193397729d6e2d7235116465c049f00d71f"},"cell_type":"markdown","source":"## 5.4 Bidirectional LSTM + Bidirectional GRU, Many to Many + Pooling"},{"metadata":{"_uuid":"c9f46810bda3728c2a1c78d62739d26699b70350"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"a44251280e0d01110bb56540b933b96bb35de44b"},"cell_type":"markdown","source":"RNN could be stack just like as deep Neutral Networks, in this case we stack 2 layers of RNN (LSTM ad GRU). Its rather rare to stack more then 4 layer of RNN. "},{"metadata":{"trusted":true,"_uuid":"c7761a013bd3e97c43f7d76c4dc81fd0326e510b"},"cell_type":"code","source":"from keras import Input\nfrom keras.layers import Embedding, SpatialDropout1D, CuDNNLSTM, CuDNNGRU, Dropout, Dense, SimpleRNN\nfrom keras.layers import concatenate, Bidirectional, GlobalAveragePooling1D, GlobalMaxPooling1D\nfrom keras.initializers import glorot_normal, orthogonal\n# First layer\n# create input\nmain_input = Input(shape=(maxlen,), dtype='int32',name='main_input')\n# creating the embedding\nembedded_sequences = Embedding(input_dim = nb_words,\n output_dim = WV_DIM,\n mask_zero=False,\n weights=[glove_embeddings],\n input_length=maxlen,\n trainable=False)(main_input)\n#Second layer\nembedded_sequences = SpatialDropout1D(0.2, seed=seed)(embedded_sequences)\n\n#Third layer \nx = Bidirectional(CuDNNLSTM(64, return_sequences=True,\n kernel_initializer=glorot_normal(seed=seed),\n recurrent_initializer=orthogonal(seed=seed)))(embedded_sequences)\n\n#Fourth layer \nx, x_h, x_c = Bidirectional(CuDNNGRU(64, return_sequences=True,return_state = True,\n kernel_initializer=glorot_normal(seed=seed),\n recurrent_initializer=orthogonal(seed=seed)))(x)\n\n#concatenate\n\navg_pool = GlobalAveragePooling1D()(x)\nmax_pool = GlobalMaxPooling1D()(x)\nconc = concatenate([x_h, avg_pool, max_pool])\n#The input format should be three-dimensional: the three components represent sample size, number of time steps and output dimension\npreds = Dense(16, activation=\"relu\", kernel_initializer=glorot_normal(seed=seed))(conc)\npreds = Dropout(0.1,seed=seed)(preds)\npreds = Dense(1, activation=\"sigmoid\", kernel_initializer=glorot_normal(seed=seed))(preds)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"81e838c158a5262a9c02f498d362380da72ebb19"},"cell_type":"code","source":"from keras.models import Model\nfrom keras.optimizers import Adam\nmodel = Model(inputs=main_input, outputs=preds)\nmodel.compile(loss='binary_crossentropy', optimizer=Adam(clipvalue=1), metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"923a6c6ecbf804e695daa2c518309d0f0dd7809b"},"cell_type":"code","source":"import time\nstart = time.time()\n\nhist = model.fit(X_train_seq, y_train, batch_size=1024, epochs=5, \n validation_data=(X_test_seq, y_test))\n\nstop = time.time()\ntime_elapsed = stop-start","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b7cd603b92605b7b0fa65797b7f3a62c5c76e923"},"cell_type":"code","source":"pred_val = model.predict(X_valid_seq, batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"06ae180ec31ad8df6c1778db6e4cf3ed6a7606e5"},"cell_type":"code","source":"scores.append(('BiLSTM+BiGRU, M2M, Max Pooling',get_scores(y_valid.target.tolist(),list(pred_val[:,0])),\n list(pred_val[:,0]),\n time.strftime(\"%M:%S\", time.gmtime(time_elapsed)),\n model.count_params()\n ))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a53ef946e0c0e4104729c267b5a9992845b892d1"},"cell_type":"code","source":"scores[-1][1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b6733e47ae4dd08ac5940a43b28ee1f329becee9"},"cell_type":"code","source":"plot_progress(scores)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"85e058fc380427c72814649e3ad0c8ad7ff7c17f"},"cell_type":"code","source":"import gc \nimport time \n\ndel(model, hist, pred_val)\ngc.collect()\ntime.sleep(2)\n\n#get_scores(y_valid,np.average(np.vstack([i[2] for i in scores[2:]]),axis=0).reshape(-1))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"65fd94d4ca7c7e284ffe3c590c3137dc2a7fc745"},"cell_type":"markdown","source":"## 5.5 Bidirectional LSTM + Bidirectional GRU, Many to Many + Pooling + Attention"},{"metadata":{"_uuid":"36069daf07eda30366ba8122ac0fd631be23951a"},"cell_type":"markdown","source":"An example of use an attention mechanism in RNN."},{"metadata":{"trusted":true,"_uuid":"f82e65c441588b49f6d9bbc1d96600628bd9825a"},"cell_type":"code","source":"from keras import backend as K\nfrom keras.engine.topology import Layer\n#from keras import initializations\nfrom keras import initializers, regularizers, constraints\n\n#https://www.kaggle.com/qqgeogor/keras-lstm-attention-glove840b-lb-0-043\nclass Attention(Layer):\n def __init__(self, step_dim,\n W_regularizer=None, b_regularizer=None,\n W_constraint=None, b_constraint=None,\n bias=True, **kwargs):\n \"\"\"\n Keras Layer that implements an Attention mechanism for temporal data.\n Supports Masking.\n Follows the work of Raffel et al. [https://arxiv.org/abs/1512.08756]\n # Input shape\n 3D tensor with shape: `(samples, steps, features)`.\n # Output shape\n 2D tensor with shape: `(samples, features)`.\n :param kwargs:\n Just put it on top of an RNN Layer (GRU/LSTM/SimpleRNN) with return_sequences=True.\n The dimensions are inferred based on the output shape of the RNN.\n Example:\n model.add(LSTM(64, return_sequences=True))\n model.add(Attention())\n \"\"\"\n self.supports_masking = True\n #self.init = initializations.get('glorot_uniform')\n self.init = initializers.get('glorot_uniform')\n\n self.W_regularizer = regularizers.get(W_regularizer)\n self.b_regularizer = regularizers.get(b_regularizer)\n\n self.W_constraint = constraints.get(W_constraint)\n self.b_constraint = constraints.get(b_constraint)\n\n self.bias = bias\n self.step_dim = step_dim\n self.features_dim = 0\n super(Attention, self).__init__(**kwargs)\n\n def build(self, input_shape):\n assert len(input_shape) == 3\n\n self.W = self.add_weight((input_shape[-1],),\n initializer=self.init,\n name='{}_W'.format(self.name),\n regularizer=self.W_regularizer,\n constraint=self.W_constraint)\n self.features_dim = input_shape[-1]\n\n if self.bias:\n self.b = self.add_weight((input_shape[1],),\n initializer='zero',\n name='{}_b'.format(self.name),\n regularizer=self.b_regularizer,\n constraint=self.b_constraint)\n else:\n self.b = None\n\n self.built = True\n\n def compute_mask(self, input, input_mask=None):\n # do not pass the mask to the next layers\n return None\n\n def call(self, x, mask=None):\n # eij = K.dot(x, self.W) TF backend doesn't support it\n\n # features_dim = self.W.shape[0]\n # step_dim = x._keras_shape[1]\n\n features_dim = self.features_dim\n step_dim = self.step_dim\n\n eij = K.reshape(K.dot(K.reshape(x, (-1, features_dim)), K.reshape(self.W, (features_dim, 1))), (-1, step_dim))\n\n if self.bias:\n eij += self.b\n\n eij = K.tanh(eij)\n\n a = K.exp(eij)\n\n # apply mask after the exp. will be re-normalized next\n if mask is not None:\n # Cast the mask to floatX to avoid float64 upcasting in theano\n a *= K.cast(mask, K.floatx())\n\n # in some cases especially in the early stages of training the sum may be almost zero\n a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n\n a = K.expand_dims(a)\n weighted_input = x * a\n #print weigthted_input.shape\n return K.sum(weighted_input, axis=1)\n\n def compute_output_shape(self, input_shape):\n #return input_shape[0], input_shape[-1]\n return input_shape[0], self.features_dim","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6f6e2a79f709fb624f0d827f1e1d3f6b349fd1aa"},"cell_type":"code","source":"from keras import Input\nfrom keras.layers import Embedding, SpatialDropout1D, CuDNNLSTM, CuDNNGRU, Dropout, Dense, SimpleRNN\nfrom keras.layers import concatenate, Bidirectional, GlobalAveragePooling1D, GlobalMaxPooling1D\nfrom keras.initializers import glorot_normal, orthogonal\n# First layer\n# create input\nmain_input = Input(shape=(maxlen,), dtype='int32',name='main_input')\n# creating the embedding\nembedded_sequences = Embedding(input_dim = nb_words,\n output_dim = WV_DIM,\n mask_zero=False,\n weights=[glove_embeddings],\n input_length=maxlen,\n trainable=False)(main_input)\n#Second layer\nembedded_sequences = SpatialDropout1D(0.2, seed=seed)(embedded_sequences)\n\n#Third layer \nx = Bidirectional(CuDNNLSTM(64, return_sequences=True,\n kernel_initializer=glorot_normal(seed=seed),\n recurrent_initializer=orthogonal(seed=seed)))(embedded_sequences)\n\n#Fourth layer \nx, x_h, x_c = Bidirectional(CuDNNGRU(64, return_sequences=True,return_state = True,\n kernel_initializer=glorot_normal(seed=seed),\n recurrent_initializer=orthogonal(seed=seed)))(x)\n\n#concatenate\n\natt = Attention(maxlen)(x)\navg_pool = GlobalAveragePooling1D()(x)\nmax_pool = GlobalMaxPooling1D()(x)\nconc = concatenate([x_h, avg_pool, max_pool, att])\n\n#The input format should be three-dimensional: the three components represent sample size, number of time steps and output dimension\npreds = Dense(16, activation=\"relu\", kernel_initializer=glorot_normal(seed=seed))(conc)\npreds = Dropout(0.1,seed=seed)(preds)\npreds = Dense(1, activation=\"sigmoid\", kernel_initializer=glorot_normal(seed=seed))(preds)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"25269bfd4ee8f6999585697f289b86dceed08db1"},"cell_type":"code","source":"from keras.models import Model\nfrom keras.optimizers import Adam\nmodel = Model(inputs=main_input, outputs=preds)\nmodel.compile(loss='binary_crossentropy', optimizer=Adam(clipvalue=1), metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a80ce0951c437821836c11543e88d5d51d28f599"},"cell_type":"code","source":"import time\nstart = time.time()\n\nhist = model.fit(X_train_seq, y_train, batch_size=1024, epochs=5, \n validation_data=(X_test_seq, y_test))\n\nstop = time.time()\ntime_elapsed = stop-start","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"946deef5cb238b01a0b805b1dac6c49c5b639699"},"cell_type":"code","source":"pred_val = model.predict(X_valid_seq, batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a4379111f390529bc0ad50b1b1e31c3e2257b5c7"},"cell_type":"code","source":"scores.append(('BiLSTM+BiGRU, M2M, Pool + Att',get_scores(y_valid.target.tolist(),list(pred_val[:,0])),\n list(pred_val[:,0]),\n time.strftime(\"%M:%S\", time.gmtime(time_elapsed)),\n model.count_params()\n ))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5ec9e7e59f0129f2cebad1e8f03c7c28204c0eff"},"cell_type":"code","source":"scores[-1][1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cb17898c698bd28db5b2440acc38e804f2ff049d"},"cell_type":"code","source":"plot_progress(scores)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7ea250cf4e427587f96cd8b4edfb125608d5c9bc"},"cell_type":"markdown","source":"## 6. Blend results"},{"metadata":{"_uuid":"6e5488c8ae126c56337a845d202a054d8fc68b69"},"cell_type":"markdown","source":"To tweak a results even further just average the results from best models.\n\n\n"},{"metadata":{"trusted":true,"_uuid":"ebb27923c05ed64b40c689abb2db9a01e444b6a0"},"cell_type":"code","source":"import time\nstart = time.time()\npred_val = np.average(np.vstack([i[2] for i in scores[-4:]]),axis=0).reshape(-1)\n\nstop = time.time()\ntime_elapsed = stop-start","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"068541fc8e4e92272e2c64cd39b22767b528ea5e"},"cell_type":"code","source":"scores.append(('Blend last 4 models',get_scores(y_valid,pred_val),list(pred_val),\n time.strftime(\"%M:%S\", time.gmtime(time_elapsed)),\n 'n/a'\n ))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9978ee75e2e91c392b7714eb604c9a87f7ea7d27"},"cell_type":"code","source":"scores[-1][1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5b921bf39d5e8817c4061c19a483a1d811b208cc"},"cell_type":"code","source":"plot_progress(scores)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"415f02e18d70c82515ef221a7dc51348eb0c2c71"},"cell_type":"markdown","source":"# Conclusions "},{"metadata":{"_uuid":"a673e19186a068b4b33b8611bfabbf6c399ce8ca"},"cell_type":"markdown","source":"* NLP is not easy but it's a lot of fun. \n* There is many possibilities left to try to build the best model in this problem but I think that these mentioned here are a good starter to do something on your own. \n* To find the best model you have to try and try and try. \n* You need a GPU unit to train your models, fortunately Kaggle have it now. \n* Keras is great - easy to learn and to use, but has some huge flaw when you use it on GPU: the results are not fully deterministic. \n* Many practitioners recommend to move to PyTorch when you are on higher level of experience with LSTM. Check my other kernel if you want to see a model made in PyTorch: https://www.kaggle.com/nicke1/fine-text-preproc-concat-embedding-lstm-gru-att"},{"metadata":{"_uuid":"c8387c8586e977b667755b684cfd0d440447d781"},"cell_type":"markdown","source":"# Questions that I've asked\n\n1) [Why are the weights of RNN/LSTM networks shared across time?](https://stats.stackexchange.com/questions/221513/why-are-the-weights-of-rnn-lstm-networks-shared-across-time) \n\"The 'shared weights' perspective comes from thinking about RNNs as feedforward networks unrolled across time. If the weights were different at each moment in time, this would just be a feedforward network. But, I suppose another way to think about it would be as an RNN whose weights are a time-varying function (and that could let you keep the ability to process variable length sequences).\n\nIf you did this, the number of parameters would grow linearly with the number of time steps.\"\n\n2) [Difference between logistic regression and neural networks](https://stats.stackexchange.com/questions/43538/difference-between-logistic-regression-and-neural-networks) \n\"Logistic regression: The simplest form of Neural Network, that results in decision boundaries that are a straight line\"\n\n3) [Different Between LSTM and LSTMCell Function in PyTorch](https://discuss.pytorch.org/t/different-between-lstm-and-lstmcell-function/5657/3) \n\"LSTMCell is more flexible and you need less code with LSTM .\n\nSo with LSTMCell,\n```\ndef forward(self, x):\n h = self.get_hidden() \n for input in x: \n h = self.rnn(input, h) # self.rnn = self.LSTMCell(input_size, hidden_size)\n```\nwhile with LSTM it is\n```\ndef forward(self, x):\n h_0 = self.get_hidden()\n output, h = self.rnn(x, h_0) # self.rnn = self.LSTM(input_size, hidden_size)\n```\n\"\n4) [Why is the F-Measure a harmonic mean and not an arithmetic mean of the Precision and Recall measures?](https://stackoverflow.com/questions/26355942/why-is-the-f-measure-a-harmonic-mean-and-not-an-arithmetic-mean-of-the-precision) \nConsider a trivial method (e.g. always returning class A). There are infinite data elements of class B, and a single element of class A:\n\n```\nPrecision: 0.0\nRecall: 1.0\n```\nWhen taking the arithmetic mean, it would have 50% correct. Despite being the worst possible outcome! With the harmonic mean, the F1-measure is 0.\n```\nArithmetic mean: 0.5\nHarmonic mean: 0.0\n```\nIn other words, to have a high F1, you need to both have a high precision and recall.\n\n5) [Difference between feedback RNN and LSTM/ GRU](https://stats.stackexchange.com/questions/222584/difference-between-feedback-rnn-and-lstm-gru): \n\"All RNNs have feedback loops in the recurrent layer. This lets them maintain information in 'memory' over time. But, it can be difficult to train standard RNNs to solve problems that require learning long-term temporal dependencies. This is because the gradient of the loss function decays exponentially with time (called the vanishing gradient problem). LSTM networks are a type of RNN that uses special units in addition to standard units. LSTM units include a 'memory cell' that can maintain information in memory for long periods of time. A set of gates is used to control when information enters the memory, when it's output, and when it's forgotten. This architecture lets them learn longer-term dependencies. GRUs are similar to LSTMs, but use a simplified structure. They also use a set of gates to control the flow of information, but they don't use separate memory cells, and they use fewer gates.:\n\n6) [What is gradient clipping and why is it necessary?](https://www.quora.com/What-is-gradient-clipping-and-why-is-it-necessary). \nGradient clipping limits the magnitude of the gradient and can make stochastic gradient descent (SGD) behave better in the vicinity of steep cliffs. \nThe steep cliffs commonly occur in recurrent networks in the area where the recurrent network behaves approximately linearly. SGD without gradient clipping overshoots the landscape minimum, while SGD with gradient clipping descends into the minimum. \n\n7) [how could i get both the final hidden state and sequence in a LSTM layer when using a bidirectional wrapper](https://stackoverflow.com/questions/49313650/how-could-i-get-both-the-final-hidden-state-and-sequence-in-a-lstm-layer-when-us) \n\nThe call Bidirectional(LSTM(128, return_sequences=True, return_state=True))(input) returns 5 tensors: \n* The entire sequence of hidden states, by default it'll be the concatenation of forward and backward states.\n* The last hidden state h for the forward LSTM\n* The last cell state c for the forward LSTM\n* The last hidden state h for the backward LSTM\n* The last cell state c for the backward LSTM\n\n```\nlstm, forward_h, forward_c, backward_h, backward_c = Bidirectional(LSTM(128, return_sequences=True, return_state=True))(input)\nstate_h = Concatenate()([forward_h, backward_h])\nstate_c = Concatenate()([forward_c, backward_c])\n``` \n\n"},{"metadata":{"_uuid":"c80a3394534e3ad38481845aa6900a9bdb2fbb98"},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}