{"cells":[{"metadata":{"_uuid":"bbd3b6a46c3fadd9c72eb4ee55cea5997969ed6a"},"cell_type":"markdown","source":"# NLP Capstone Project - Movie Review - Sentiment Analysis -  Classify the sentiment of sentences from the Rotten Tomatoes dataset\n## Developing Deep Learning Models using GLoVE pretrained word embeddings as feature extraction / representation\n\n![](https://cdn.steemitimages.com/DQmQZCf7ME7Haj3X3MzXtG8R8JtGmTpuh5NXDSd3wKueva7/rottentomatoes.png)\n\n![Sentiment](https://www.kdnuggets.com/images/sentiment-fig-1-689.jpg)\n\n![DL](https://www.doulos.com/images/logos/DeepLearning.jpg)\n\n![W2v](https://newvitruvian.com/images/term-vector-word2vec-4.png)\n\nMoving one from EDA and Machine Leaning models to Deep Learning with the movie review sentiment analysis dataset with Keras. In particular the DL models will be developed in conjunction with the pre-trained Word Embeddings.\n\nThe pretrained word embeddings will be downloaded from Stanford NLP GloVe: Global Vectors for Word Representation [source](https://nlp.stanford.edu/projects/glove/).\n\nAt first, There must be mention that after EDA an odd conclusion was made. The dataset of this competition turned to have some unique features. we have only phrases as data. And a phrase can contain a single word. And one punctuation mark can cause phrase to receive a different sentiment. Also assigned sentiments can be strange. This means several things:\n\n- using stopwords can be a bad idea, especially when phrases contain one single stopword;\n- puntuation could be important, so it should be used;\n\n** This thought will be enhanced later with my anomaly detection insights **"},{"metadata":{"_uuid":"7d7425aa225c7dcf03e704b8c8774b7013b11c35"},"cell_type":"markdown","source":"## Loading Main Libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\n\nimport matplotlib\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport seaborn as sns\n\nimport nltk","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"df = pd.read_csv(\"../input/train.tsv\", sep=\"\\t\")\n\ndf_test = pd.read_csv(\"../input/test.tsv\", sep=\"\\t\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"377968fad9e3f1a90c312fd1663a2b67da2af60e"},"cell_type":"code","source":"df.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"281b040ce6f593dfbe04c3b4dfde5f04534ffc62"},"cell_type":"code","source":"df_test.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ce53897d7dccf3f6a3461743278bc36c20535b81"},"cell_type":"markdown","source":"### Thoughts before training DL models\nHere are couple of instances where punctuations appeared to be predictive. So if we \"cleanedup\" the data in the name of data preparation some predictiveness will be lost."},{"metadata":{"trusted":true,"_uuid":"bbf262264d45fb2d0b1c33ffaad332cd9c5a0bde"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 0) & (df['PhraseId'] <= 2)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"21d3788394760163cd09f9bf17c9e76cc3c9d785"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 517) & (df['PhraseId'] <= 518)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"764cdb3cde4a32ffc0afe645acb54c150f2206c3"},"cell_type":"markdown","source":"Below another example that the appearance punctuation symbol \",\" is important"},{"metadata":{"trusted":true,"_uuid":"1ca61bab78eeec037b886fd7252dbb07244d3abb"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 68) & (df['PhraseId'] <= 69)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"02dde7d31efdf2a33d0bdbaa7f8f477c924d14e7"},"cell_type":"markdown","source":"Below another example that the appearance punctuation symbol \"!\" is important"},{"metadata":{"trusted":true,"_uuid":"d927e4b460e24554b3e92f2919a1ab498f1a8c1b"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 10737) & (df['PhraseId'] <= 10738)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d49d07430867a4e5d5c424fcf02d4daeb9c28d0f"},"cell_type":"markdown","source":"Another strange thing that I discovered is that there are phrases with a single word only and if they disappear at the following phrases the sentiment changes."},{"metadata":{"trusted":true,"_uuid":"10d4b2c8a8785461f991fdcd823224a74a109ceb"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 22) & (df['PhraseId'] <= 24)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])\n\nprint()\n\nprint(example[\"Phrase\"].values[2], \" - Sentiment:\", example[\"Sentiment\"].values[2])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"255c71ce650d436836345b428077ed00dd24bbff"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 46) & (df['PhraseId'] <= 47)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2923746a2e058ffc8e4b67183aaa3458b1228571"},"cell_type":"markdown","source":"As you can see sentence id denotes a single review with the phrase column having the entire review text as an input instance followed by random suffixes of the same sentence to form multiple phrases with subsequent phrase ids. This repeats for every single new sentence id (or new review per se). The sentiment is coded with 5 values 0= Very negative to 4=Very positive and everything else in between.\n\nA quick glance will show you that the data is a little weird for a sentiment corpus:\n\n- Phrases of sentences are** chopped up compeltely randomly**. So logic like sentence tokenization based on periods or punctuations or something of that sort doesn't apply\n- Certain phrases are **with one single word!**.\n- For some phrases inclusion of a punctuation like a comma or a full stop changes the sentiment from say 2 to 3 i.e neutral to positive.\n- Some phrases **starts** with a punctuation like a **backquote**.\n- Some phrases **end** with a **punctuation**\n- There are some ** weird ** words such as ** -RRB-,  -LRB- **\n\nAll these weird aspects of this dataset, can be helpful and may be predictive. Afterall, we are looking for patterns in data. Therefore, it would be easier for us to engineer features, I mean apart from the text features that can be extracted from the corpus.\n\nSo, after all this train of thought, let us move on to Deep Learning and Predictive Models."},{"metadata":{"_uuid":"68578726f6be9f4cfdebf288bb359a908dc09eb7"},"cell_type":"markdown","source":"### Helper Functions to tokenize and create a vocabulary\nThey both will be needed later in Deep Learning functions"},{"metadata":{"trusted":true,"_uuid":"a2446d50a5870d84b7fe2e937475a1b7fc0b721e"},"cell_type":"code","source":"def tokenize_the_text(phrases):\n    \n    from nltk.tokenize import word_tokenize\n    from nltk.text import Text\n    \n    tokens = [word for word in phrases]\n    tokens = [word.lower() for word in tokens]\n    tokens = [word_tokenize(word) for word in tokens]\n    \n    return tokens\n\ncrude_tokens = tokenize_the_text(df.Phrase)\nprint(crude_tokens[0:10])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aa14c209c7b9108cef0fd7a786704012d309c640"},"cell_type":"code","source":"def create_a_vocab(tokens):\n    \n    vocab = set()\n\n    for sentence in tokens:\n        for word in sentence:\n            vocab.add(word)\n\n    vocab = list(vocab)\n\n    return vocab\n    \nvocab = create_a_vocab(crude_tokens)\n\nprint(len(vocab))\n#print(type(vocab))\n#print(sorted(vocab))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f04bba93c2eeddcc435558e643a81178c8141ed"},"cell_type":"markdown","source":"## Deep Learning Techniques"},{"metadata":{"_uuid":"26c294218674c7ebc4a76ae7eddb92fc50a9dc3c"},"cell_type":"markdown","source":"### Load Keras functions"},{"metadata":{"trusted":true,"_uuid":"fb9bced6a514c0d5e5bfb41d1a118ea6fc406d38"},"cell_type":"code","source":"# Keras Libraries\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Flatten, LSTM, GRU, Conv1D, MaxPooling1D, Dropout, SpatialDropout1D, Bidirectional, Activation,GlobalAveragePooling1D, GlobalMaxPooling1D\nfrom keras.layers.embeddings import Embedding\nfrom sklearn.model_selection import train_test_split\nfrom keras.models import load_model\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import precision_score\nfrom sklearn.metrics import recall_score\nfrom sklearn.metrics import f1_score\nfrom sklearn.metrics import classification_report\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0cfd4a7a9c4adee890bad3d7cc7ecbdee8cf5dc"},"cell_type":"markdown","source":"## Word to feature Extraction with pre-trained Word Embeddings\n![w2v2](https://cdn-images-1.medium.com/max/1600/1*jpnKO5X0Ii8PVdQYFO2z1Q.png)\n\nWord embedding is one of the most popular representation of document vocabulary. It is capable of capturing context of a word in a document, semantic and syntactic similarity, relation with other words, etc.\n\nWord Embeddings originates from the idea of generating distributed representations. Intuitively, there is some dependence of one word on the other words. The words in context of this word would get a greater share of this dependence. In one hot encoding representations, all the words are independent of each other, as mentioned earlier.\n\nWord Embeddings are vector representations of a particular word. Word Embeddings is a method to construct such an embedding. It can be obtained using two methods (both involving Neural Networks): Skip Gram and Common Bag Of Words. [source](https://towardsdatascience.com/introduction-to-word-embedding-and-word2vec-652d0c2060fa).\n\nTo get Word Embeddings they will be downloaded from Stanford NLP GloVe: **Global Vectors for Word Representation**. GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space [source](https://nlp.stanford.edu/projects/glove/).\n\nSo we continue with the download of the GLoVe embeddings, The embedding size of 300 will be used for increased expressibility."},{"metadata":{"trusted":true,"_uuid":"3c9fd3e091a3ad3dce2508677bec6ce246e25d2f"},"cell_type":"code","source":"import os,requests\ndef download(url):\n    get_response = requests.get(url,stream=True)\n    file_name  = url.split(\"/\")[-1]\n    with open(file_name, 'wb') as f:\n        for chunk in get_response.iter_content(chunk_size=1024):\n            if chunk: # filter out keep-alive new chunks\n                f.write(chunk)\n        \n\ndownload(\"http://nlp.stanford.edu/data/glove.6B.zip\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b748c8837fa30ec1ee7eb968bc5926ecb9baf2a"},"cell_type":"markdown","source":"### Export the GLoVe zip file."},{"metadata":{"trusted":true,"_uuid":"240e10078a434db7b74bf5feaee47644ba39b5e2"},"cell_type":"code","source":"import zipfile\nwith zipfile.ZipFile('glove.6B.zip', 'r') as zip_ref:\n    zip_ref.extractall('.')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a8c24525327b4e3723b961c0c10c3878f073a119"},"cell_type":"code","source":"import os\nos.listdir()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5a1a0cad76156aef0df8ddb17b639f25f2d9212"},"cell_type":"markdown","source":"### Reading the GLoVe embedding txt file"},{"metadata":{"trusted":true,"_uuid":"459f61db581e75d063b35f9846197c5f21ccf4e2"},"cell_type":"code","source":"import numpy as np\n\nembedding_dim = 300\nfilename = 'glove.6B.'+ str(embedding_dim) +'d.txt'\n\nglove_w2v_embeddings_index = dict()\nf = open(filename, \"r\", encoding='utf-8')\nfor line in f:\n    values = line.split()\n    word = values[0]\n    coefs = np.asarray(values[1:], dtype='float32')\n    glove_w2v_embeddings_index[word] = coefs\nf.close()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e0190ecb7fe3d28ea42e58ae2e0ab9739d6c1524"},"cell_type":"markdown","source":"### Converting the train and test set into one-hot encoding and preparing for DL model deployment\nThe following python code snippet converts the train and test Sets into one-hot encoding using the Keras tokenizer. Due to its phrase has different number of words and different one hot encoding vectors will be created, all vectors were padded to meet an equal number of length."},{"metadata":{"trusted":true,"_uuid":"cf73cb273a0822c12709bcab0f8377d38b16140a"},"cell_type":"code","source":"df_test = pd.read_csv(\"../input/test.tsv\", sep=\"\\t\")\n\n###\ntokens_uncleaned = tokenize_the_text(df.Phrase.values)\ntokens_uncleaned_test = tokenize_the_text(df_test.Phrase.values)\nsentences = [' '.join(sent) for sent in tokens_uncleaned]\nsentences_test = [' '.join(sent) for sent in tokens_uncleaned_test]\nall_corpus = sentences + sentences_test\n###\n\nvocab_all_corpus = create_a_vocab(tokens_uncleaned + tokens_uncleaned_test)\nmax_len = max([len(elem.split()) for elem in all_corpus])\n#print(max_len)\n\n\ntokenizer = Tokenizer(lower=True, filters='')\ntokenizer.fit_on_texts(all_corpus)\n\nvocabulary_size = len(tokenizer.word_index) + 1\n#print(vocabulary_size)\n\nX = tokenizer.texts_to_sequences(sentences)\ny = df.Sentiment.values\nX = pad_sequences(X, maxlen=max_len)\n\n\nxtrain, xvalid, ytrain, yvalid = train_test_split(X, y, stratify=y, random_state=42, test_size=0.2, shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1fe076c8a6a88e9e8a5f644ce920e231d49ac3cb"},"cell_type":"markdown","source":"### Prepare the embedding matrix\nCreating the embedding matrix for each word in the dataset's corpus using the embedding vectors from the GLoVe zip file. Moreover for each word its polarity and subjectivity from Textblob NLP library will be added. If a word does not appear in the GLoVe embeddings then a vector with zeros and with size equal to embedding_dim."},{"metadata":{"trusted":true,"_uuid":"cc8d1ca3ba8f837a5538d1de405f833bb4946614"},"cell_type":"code","source":"import textblob\n\nembedding_matrix = np.zeros((vocabulary_size, embedding_dim + 2))\n\nfor word, i in tokenizer.word_index.items():\n    if i > vocabulary_size:\n        continue\n    embedding_vector = glove_w2v_embeddings_index.get(word)\n    word_sentiment = textblob.TextBlob(word).sentiment\n    if embedding_vector is not None:\n        embedding_matrix[i] = np.append(embedding_vector, [word_sentiment.polarity, word_sentiment.subjectivity])\n    else:\n        embedding_matrix[i, -2:] = [word_sentiment.polarity, word_sentiment.subjectivity]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fb4f45de1855978dbf1cb72c5499ea80eb9308b5"},"cell_type":"markdown","source":"### Function to plot the DL fitting history results\nTHe function below will be utilized to plot the DL models' fitting history to visualise their performance"},{"metadata":{"trusted":true,"_uuid":"f6919b5b37c5def0d2b8a0072266cda216ba3e6c"},"cell_type":"code","source":"def plot_history(history):\n    acc = history.history['categorical_accuracy']\n    val_acc = history.history['val_categorical_accuracy']\n    loss = history.history['loss']\n    val_loss = history.history['val_loss']\n    x = range(1, len(acc) + 1)\n\n    plt.figure(figsize=(12, 5))\n    plt.subplot(1, 2, 1)\n    plt.plot(x, acc, 'b', label='Training acc')\n    plt.plot(x, val_acc, 'r', label='Validation acc')\n    plt.title('Training and validation accuracy')\n    plt.legend()\n    plt.subplot(1, 2, 2)\n    plt.plot(x, loss, 'b', label='Training loss')\n    plt.plot(x, val_loss, 'r', label='Validation loss')\n    plt.title('Training and validation loss')\n    plt.legend()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b721700901bf0f3c738e2111406b32cab706133"},"cell_type":"markdown","source":"## Deep Learning Techiniques for Multiclass Sentiment Analysis\n\nThe Deep Learning models that will be deployed is the following:\n- Long Short Term Memory networks (LSTM)\n- Bidirectional Long Short Term Memory networks (BiLSTM)\n- Convolutional Neural Networks (CNN)\n- Long Short Term Memory networks followed by Convolutional Neural Networks (LSTM + CNN)\n- Bidirectional Long Short Term Memory networks followed by Convolutional Neural Networks (LSTM + BiLSTM)\n\nThe idea of using these 5 networks came from the scientific literature and from browsing the web [source_1](https://www.isca-speech.org/archive/interspeech_2012/i12_0194.html), [source_2](https://arxiv.org/abs/1609.02745), [source_3](http://www.aclweb.org/anthology/P16-2037),  [source_4](https://arxiv.org/ftp/arxiv/papers/1801/1801.07883.pdf).\n\nSince the competion evaluates the models based on accuracy then the models will be evaluated based on accuracy and because the dataset is unbalanced (based on its EDA) us a secondary statistical evaluation metric I will use the F1 score.\n \n The train set will be split in train and validation sets with ratio **80:20** .\n \n For all the DL models the random state will be set to 42 in order to the models be reproducable and create the same results in every run.\n \n Finally as a benchmark model, due to the fact that LSTM is the one of the simpliest DL techniques from the other 4. The performance results between LSTM model and the rest of the DL models will be compared and try to outperform."},{"metadata":{"trusted":true,"_uuid":"26aa795e610929e581904551a830f18d4fb64608"},"cell_type":"code","source":"dl_performance_metrics_df = pd.DataFrame(columns=['accuracy','F1-score','training-time'], index=['LSTM', 'BiLSTM', 'CNN', 'LSTM_CNN', 'BiLSTM_CNN'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c69c02ef349e9c4b3878f569c5520b3b544b1d74"},"cell_type":"markdown","source":"### Deep Learning Model - Long Short Term Memory networks (LSTM)\n\nLong Short Term Memory networks LSTMs are a special kind of RNN, capable of learning long-term dependencies. They were introduced by Hochreiter & Schmidhuber (1997), and were refined and popularized by many people in following work. They work tremendously well on a large variety of problems, and are now widely used.\n\nLSTMs are explicitly designed to avoid the long-term dependency problem. Remembering information for long periods of time is practically their default behavior, not something they struggle to learn!\n\nAll recurrent neural networks have the form of a chain of repeating modules of neural network. In standard RNNs, this repeating module will have a very simple structure, such as a single tanh layer.\n\nLSTMs also have this chain like structure, but the repeating module has a different structure. Instead of having a single neural network layer, there are four, interacting in a very special way [source](http://colah.github.io/posts/2015-08-Understanding-LSTMs/).\n\nThe following function builds a DL sequential network. The layers are the following:\n- The Embedding Layers where word embeddings are trained which reduces the one-hot-encoding phrases vectors to a dense with size of 300\n- A dropout Layers where with regularize the connection between the embedding layer and the following LSTM layer\n- Then the LSTM layer with number of cells equal to 150\n- A dropout Layers where with regularize the connection between the the LSTM layer and the following 5 dense neurons\n- Finally the last layer is 5 neurons equal to the number of sentiment classes\n\nAs optimizer we use the *nadam* and the fitting goal is to maximize the model's accuracy, hence we use a checkpointer Keras function for that to save the best model. Furthemore, the earlyStopping function will be utlized to avoid overfitting.\n\nAs the loss / coss function since we have categorical variables to be predicted we used the categorical_crossentropy loss function.\n\nIn every epoch the DL model will be evaluated in the validation set. If the accuracy improves in the new epoch the model will be saved. But after 3 iterations of no improvement the model the fitting will be stopped.\n\nFor all the DL models the random state will be set to 42 in order to the models be reproducable and create the same results in every run."},{"metadata":{"trusted":true,"_uuid":"158ed7c11814d988bb43d53ae3fce820d8736ddd"},"cell_type":"code","source":"def build_dl_lstm_model(xtrain, ytrain, xvalid, yvalid, num_of_epochs, filename):\n\n    ## Network architecture\n    from keras.utils import to_categorical\n    from keras.callbacks import ModelCheckpoint\n    from keras.callbacks import EarlyStopping\n    from keras.layers import Masking\n    from keras.initializers import Constant\n    import time\n    \n    import warnings\n    warnings.simplefilter(action='ignore', category=FutureWarning)\n    \n    start_time = time.time()\n    \n    \n    from numpy.random import seed\n    seed(42)\n    from tensorflow import set_random_seed\n    set_random_seed(42)\n\n    embedding_size= embedding_dim + 2\n    batch_size = 128\n    dropouts = 0.2\n    epochs = num_of_epochs\n    \n    model=Sequential()\n    model.add(Embedding(input_dim = vocabulary_size, output_dim = embedding_size, input_length = max_len, \n                        weights=[embedding_matrix], trainable = False, mask_zero=True))\n    \n    model.add(SpatialDropout1D(dropouts))\n    \n    model.add(LSTM(int(embedding_size/2), recurrent_dropout=dropouts, dropout=dropouts, return_sequences=False))\n    \n    model.add(Dense(5, activation='softmax'))\n\n    print(model.summary())\n\n    model.compile(loss='categorical_crossentropy', optimizer = 'nadam', metrics=['categorical_accuracy']) # RMSprop\n\n    '''\n    saves the model weights after each epoch if the val_acc loss decreased\n    '''\n    checkpointer = ModelCheckpoint(monitor='val_categorical_accuracy', mode='max', filepath=''+filename+'.hdf5', verbose=2, save_best_only=True)\n    earlyStopping = EarlyStopping(monitor='val_categorical_accuracy', min_delta=0, patience=3, verbose=0, mode='max')\n\n    history = model.fit(x = xtrain, y = to_categorical(ytrain), validation_data=(xvalid, to_categorical(yvalid)), epochs=epochs, batch_size=batch_size, verbose=2, callbacks=[checkpointer, earlyStopping])\n\n    model = load_model(''+filename+'.hdf5')\n    \n    elapsed_time = time.time() - start_time\n    \n    return model, history, elapsed_time","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"00d465d505fca5883c5b4636539a928cc14781ba","scrolled":false},"cell_type":"code","source":"\ndl_lstm_model, history_lstm, elapsed_time = build_dl_lstm_model(xtrain, ytrain, xvalid, yvalid, num_of_epochs=30, filename=\"lstm\")\nprint(\"Elapsed time in seconds:\", elapsed_time)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5447e6963f9a42a555eb303d7d90ffbd1c303cc5"},"cell_type":"markdown","source":"### Model Summary"},{"metadata":{"trusted":true,"_uuid":"628d4454ef3339c802153b6dbc016e8970b7c890"},"cell_type":"code","source":"from keras.utils.vis_utils import plot_model\n\nplot_model(dl_lstm_model, to_file='dl_lstm_model.png', show_shapes=True, show_layer_names=True)\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom PIL import Image\n\nimg = Image.open('dl_lstm_model.png')\nplt.rcParams[\"figure.figsize\"] = (10,10)\nplt.axis('off')\nplt.grid(False)\nimgplot = plt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bbc055f6856bccc46cab2c8ea68b726cf798237c"},"cell_type":"markdown","source":"### Model Evaluation"},{"metadata":{"trusted":true,"_uuid":"7f261651899208eff5bdeb273fb8142aa0d44896"},"cell_type":"code","source":"plot_history(history_lstm)\n\ny_pred_lstm = dl_lstm_model.predict_classes(xvalid, verbose=1)\nprint()\n\nprint(classification_report(yvalid, y_pred_lstm))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, y_pred_lstm))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, y_pred_lstm, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, y_pred_lstm, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, y_pred_lstm, average='weighted'))\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, y_pred_lstm)\n\ndl_performance_metrics_df.loc['LSTM']['training-time'] = elapsed_time\ndl_performance_metrics_df.loc['LSTM']['accuracy'] = accuracy_score(yvalid, y_pred_lstm)\ndl_performance_metrics_df.loc['LSTM']['F1-score'] = f1_score(yvalid, y_pred_lstm, average='weighted')\n\nprint(\"elapsed time:\", round(elapsed_time), \"seconds\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"99984f42124a3db094631323dc130a3673949dea"},"cell_type":"markdown","source":"The LSTM architucture shows promising results; The accuracy is 0.67 and the F1-score is 0.67. However there are cases that are missclassified based on the high recall score per sentiment class."},{"metadata":{"_uuid":"53e93c3211f16488915b2b5fd640483e4781799f"},"cell_type":"markdown","source":"### Deep Learning Model - Bidirectional Long Short Term Memory networks (BiLSTM)\n\nA major issue with all of the Recurrent networks is that they learn representations from previous time steps. Sometimes, you might have to learn representations from future time steps to better understand the context and eliminate ambiguity. Take the following examples, “He said, Teddy bears are on sale” and “He said, Teddy Roosevelt was a great President”. In the above two sentences, when we are looking at the word “Teddy” and the previous two words “He said”, we might not be able to understand if the sentence refers to the President or Teddy bears. Therefore, to resolve this ambiguity, we need to look ahead. This is what Bidirectional RNNs accomplish. The repeating module in a Bidirectional RNN could be a conventional RNN, LSTM or GRU. [source](https://towardsdatascience.com/introduction-to-sequence-models-rnn-bidirectional-rnn-lstm-gru-73927ec9df15).\n\nThe following function builds a Deep Learning sequential network with the following stracture:\n\n- The Embedding Layers where word embeddings are trained which reduces the one-hot-encoding phrases vectors to a dense with size of 300\n- A dropout Layers where with regularize the connection between the embedding layer and the following Bidirectional LSTM layer\n- Then the Bidirectional LSTM layer with number of cells equal to 150\n- A dropout Layers where with regularize the connection between the the LSTM layer and the following 5 dense neurons\n- Finally the last layer is 5 neurons equal to the number of sentiment classes\n\nAs optimizer we use the *nadam* and the fitting goal is to maximize the model's accuracy, hence we use a checkpointer Keras function for that to save the best model. Furthemore, the earlyStopping function will be utlized to avoid overfitting\n\nAs the loss / coss function since we have categorical variables to be predicted we used the categorical_crossentropy loss function.\n\nIn every epoch the DL model will be evaluated in the validation set. If the accuracy improves in the new epoch the model will be saved. But after 3 iterations of no improvement the model the fitting will be stopped.\n\nFor all the DL models the random state will be set to 42 in order to the models be reproducable and create the same results in every run."},{"metadata":{"trusted":true,"_uuid":"a472ca7ce69d68ba46f8181c3696f6044da18398"},"cell_type":"code","source":"def build_dl_bidirectional_lstm_model(xtrain, ytrain, xvalid, yvalid, num_of_epochs, filename):\n\n    ## Network architecture\n    from keras.utils import to_categorical\n    from keras.callbacks import ModelCheckpoint\n    from keras.callbacks import EarlyStopping\n    from keras.optimizers import RMSprop, SGD, adam\n    from keras.layers import Masking\n    from keras.initializers import Constant\n    import time\n    \n    import warnings\n    warnings.simplefilter(action='ignore', category=FutureWarning)\n    \n    start_time = time.time()\n    \n    \n    from numpy.random import seed\n    seed(42)\n    from tensorflow import set_random_seed\n    set_random_seed(42)\n\n\n    embedding_size= embedding_dim + 2\n    batch_size = 256\n    dropouts = 0.2\n    epochs = num_of_epochs\n    \n    model=Sequential()\n    model.add(Embedding(input_dim = vocabulary_size, output_dim = embedding_size, input_length=max_len, \n                        weights=[embedding_matrix], trainable = False, mask_zero=True))\n    \n    model.add(SpatialDropout1D(dropouts))\n    \n    model.add(Bidirectional(LSTM(int(embedding_size/2), recurrent_dropout=dropouts, dropout=dropouts, return_sequences=False)))\n    \n    model.add(Dense(5, activation='softmax'))\n\n    print(model.summary())\n\n    model.compile(loss='categorical_crossentropy', optimizer = 'nadam', metrics=['categorical_accuracy']) # RMSprop\n\n    '''\n    saves the model weights after each epoch if the val_acc loss decreased\n    '''\n    checkpointer = ModelCheckpoint(monitor='val_categorical_accuracy', mode='max', filepath=''+filename+'.hdf5', verbose=2, save_best_only=True)\n    earlyStopping = EarlyStopping(monitor='val_categorical_accuracy', min_delta=0, patience=3, verbose=0, mode='max')\n\n    history = model.fit(x = xtrain, y = to_categorical(ytrain), validation_data=(xvalid, to_categorical(yvalid)), epochs=epochs, batch_size=batch_size, verbose=2, callbacks=[checkpointer, earlyStopping])\n\n    model = load_model(''+filename+'.hdf5')\n    \n    elapsed_time = time.time() - start_time\n    \n    return model, history, elapsed_time\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2d2de8d4797e8ec2f60cc02256ec9a955f63ba19"},"cell_type":"code","source":"\ndl_bidirectional_lstm_model, history_bidirectional_lstm, elapsed_time = build_dl_bidirectional_lstm_model(xtrain, ytrain, xvalid, yvalid, num_of_epochs=50, filename=\"bidirectional_lstm\")\nprint(\"Elapsed time in seconds:\", elapsed_time)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6279f44761f2384f37db931b81c333fbec35a5f9"},"cell_type":"markdown","source":"### Model Summary"},{"metadata":{"trusted":true,"_uuid":"335e5567ff43f76492c2917bff18ab8c065c1429"},"cell_type":"code","source":"from keras.utils.vis_utils import plot_model\n\nplot_model(dl_bidirectional_lstm_model, to_file='dl_bidirectional_lstm_model.png', show_shapes=True, show_layer_names=True)\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom PIL import Image\n\nimg = Image.open('dl_bidirectional_lstm_model.png')\nplt.rcParams[\"figure.figsize\"] = (10,10)\nplt.axis('off')\nplt.grid(False)\nimgplot = plt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8d4e498fe0ef55602581352316962c940b4130d8"},"cell_type":"markdown","source":"### Model Evaluation"},{"metadata":{"trusted":true,"_uuid":"73d386f2f3f55432fa82cffca327be830c7042c3"},"cell_type":"code","source":"plot_history(history_bidirectional_lstm)\n\ny_pred_bidirectional_lstm = dl_bidirectional_lstm_model.predict_classes(xvalid, verbose=1)\nprint()\n\nprint(classification_report(yvalid, y_pred_bidirectional_lstm))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, y_pred_bidirectional_lstm))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, y_pred_bidirectional_lstm, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, y_pred_bidirectional_lstm, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, y_pred_bidirectional_lstm, average='weighted'))\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, y_pred_bidirectional_lstm)\n\n\ndl_performance_metrics_df.loc['BiLSTM']['training-time'] = elapsed_time\ndl_performance_metrics_df.loc['BiLSTM']['accuracy'] = accuracy_score(yvalid, y_pred_bidirectional_lstm)\ndl_performance_metrics_df.loc['BiLSTM']['F1-score'] = f1_score(yvalid, y_pred_bidirectional_lstm, average='weighted')\n\nprint(\"elapsed time:\", round(elapsed_time), \"seconds\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dc5dd7143a39c2f84110c2eb0fa417a0e0df1995"},"cell_type":"markdown","source":"The Bidirectional LSTM (BiLSTM) architucture presents some good results; The accuracy is 0.67 and the F1-score is 0.67. Unfortunately, there are many cases that are missclassified based on the high recall score per sentiment class."},{"metadata":{"_uuid":"cee3ef0b4822e2a9f1f7f1a3838d031029cdcca2"},"cell_type":"markdown","source":"### Deep Learning Models - Convolutional Neural Networks\n\nConvolutional Neural Networks are very famous for applications in image classification. The whole idea about ConvNets stems from the notion that by adding more and more layers to the network the DL model can understand more and more features from an image and categotize it easier and more efficiently [source](https://adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/). Moreover, the same architecture presents great results with Text classification problems [source](http://www.joshuakim.io/understanding-how-convolutional-neural-network-cnn-perform-text-classification-with-word-embeddings/).\n\nThe following function builds a Deep Learning sequential network with the following stracture:\n\n- The Embedding Layers where word embeddings are trained which reduces the one-hot-encoding phrases vectors to a dense with size of 300\n- A dropout Layers where with regularize the connection between the embedding layer and the following ConvNets layer\n- 3 fully connected Convolutional Layers, each time the number of units are increasing and the same happens to the kernel_size / filter\n- A GlobalMaxPooling to pool the best features from the ConvNets\n- They the best features are fed to a fully connected Dense neural units equal to 150 neurons\n- A dropout Layers where with regularize the connection between the the LSTM layer and the following 5 dense neurons\n- Finally the last layer is 5 neurons equal to the number of sentiment classes\n\nAs optimizer we use the *nadam* and the fitting goal is to maximize the model's accuracy, hence we use a checkpointer Keras function for that to save the best model. Furthemore, the earlyStopping function will be utlized to avoid overfitting\n\nAs the loss / coss function since we have categorical variables to be predicted we used the categorical_crossentropy loss function.\n\nIn every epoch the DL model will be evaluated in the validation set. If the accuracy improves in the new epoch the model will be saved. But after 3 iterations of no improvement the model the fitting will be stopped.\n\nFor all the DL models the random state will be set to 42 in order to the models be reproducable and create the same results in every run."},{"metadata":{"trusted":true,"_uuid":"c21e5eb31c0d7d3cd5e02e2f11334c516380bebc"},"cell_type":"code","source":"def build_dl_cnn_model(xtrain, ytrain, xvalid, yvalid, num_of_epochs, filename):\n\n    ## Network architecture\n    from keras.utils import to_categorical\n    from keras.callbacks import ModelCheckpoint\n    from keras.callbacks import EarlyStopping\n    from keras.optimizers import RMSprop, SGD, adam\n    from keras.layers import Masking\n    from keras.initializers import Constant\n    import time\n    \n    import warnings\n    warnings.simplefilter(action='ignore', category=FutureWarning)\n    \n    start_time = time.time()\n    \n    \n    from numpy.random import seed\n    seed(42)\n    from tensorflow import set_random_seed\n    set_random_seed(42)\n\n    embedding_size= embedding_dim + 2\n    batch_size = 128\n    dropouts = 0.2\n    epochs = num_of_epochs\n    \n    model=Sequential()\n    model.add(Embedding(input_dim = vocabulary_size, output_dim = embedding_size, input_length=max_len, \n                        weights=[embedding_matrix], trainable = False))\n    \n    model.add(SpatialDropout1D(dropouts))\n        \n    model.add(Conv1D(128, kernel_size = 1, strides = 1,  padding='valid', activation='relu'))\n    model.add(Conv1D(256, kernel_size = 3, strides = 1,  padding='valid', activation='relu'))\n    model.add(Conv1D(512, kernel_size = 5, strides = 1,  padding='valid', activation='relu'))\n    \n    model.add(GlobalMaxPooling1D())\n    model.add(Dropout(dropouts))\n\n    model.add(Dense(int(embedding_size/2), activation=\"relu\"))\n    model.add(Dropout(dropouts))\n    \n    model.add(Dense(5, activation='softmax'))\n\n    print(model.summary())\n\n    model.compile(loss='categorical_crossentropy', optimizer = 'nadam', metrics=['categorical_accuracy'])\n\n    '''\n    saves the model weights after each epoch if the val_acc loss decreased\n    '''\n    checkpointer = ModelCheckpoint(monitor='val_categorical_accuracy', mode='max', filepath=''+filename+'.hdf5', verbose=2, save_best_only=True)\n    earlyStopping = EarlyStopping(monitor='val_categorical_accuracy', min_delta=0, patience=3, verbose=0, mode='max')\n\n    history = model.fit(x = xtrain, y = to_categorical(ytrain), validation_data=(xvalid, to_categorical(yvalid)), epochs=epochs, batch_size=batch_size, verbose=2, callbacks=[checkpointer, earlyStopping])\n\n    model = load_model(''+filename+'.hdf5')\n    \n    elapsed_time = time.time() - start_time\n    \n    return model, history, elapsed_time\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"018937c09103fd7f6b7ffe098cf98b7543c63d0c"},"cell_type":"code","source":"\ndl_cnn_model, history_cnn, elapsed_time = build_dl_cnn_model(xtrain, ytrain, xvalid, yvalid, num_of_epochs=30, filename=\"cnn\")\nprint(\"Elapsed time in seconds:\", elapsed_time)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"24aab373c24ec1c8fa7fd162c9b52558812d8ae5"},"cell_type":"markdown","source":"### Model Summary"},{"metadata":{"trusted":true,"_uuid":"d9539d531d35144f2b70deea56eb90fe5cd27d93"},"cell_type":"code","source":"from keras.utils.vis_utils import plot_model\n\nplot_model(dl_cnn_model, to_file='dl_cnn_model.png', show_shapes=True, show_layer_names=True)\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom PIL import Image\n\nimg = Image.open('dl_cnn_model.png')\nplt.rcParams[\"figure.figsize\"] = (21,22)\nplt.axis('off')\nplt.grid(False)\nimgplot = plt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c250d35a7524d8cd17aa39c9ebde6f8c3493656c"},"cell_type":"markdown","source":"### Model Evaluation"},{"metadata":{"trusted":true,"_uuid":"23be55b23dc858185fdc9f1d3edffd3a571a790a"},"cell_type":"code","source":"plot_history(history_cnn)\n\ny_pred_cnn = dl_cnn_model.predict_classes(xvalid, verbose=1)\nprint()\n\nprint(classification_report(yvalid, y_pred_cnn))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, y_pred_cnn))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, y_pred_cnn, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, y_pred_cnn, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, y_pred_cnn, average='weighted'))\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, y_pred_cnn)\n\n\ndl_performance_metrics_df.loc['CNN']['training-time'] = elapsed_time\ndl_performance_metrics_df.loc['CNN']['accuracy'] = accuracy_score(yvalid, y_pred_cnn)\ndl_performance_metrics_df.loc['CNN']['F1-score'] = f1_score(yvalid, y_pred_cnn, average='weighted')\n\nprint(\"elapsed time:\", round(elapsed_time), \"seconds\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5a37ea24664f927cb849f7e3978d927db02dfd0"},"cell_type":"markdown","source":"The deep CNN network architucture has some promising results; The accuracy is 0.67 and the F1-score is 0.67. Nevertheless, there are many cases that are missclassified based on the high recall score per sentiment class."},{"metadata":{"_uuid":"fdb43ef2a3620036672abe412e9be50984438774"},"cell_type":"markdown","source":"### Deep Learning Model: Combining LSTM and CNN together (LSTM_CNN)\nThere are papers in the scientific literature that combine both LSTM and CNN to improve the DL models performance by deepening the network [source](https://arxiv.org/abs/1511.08630). Thus the same will be done and here!\n\nThe following function builds a Deep Learning sequential network with the following stracture:\n\n- The Embedding Layers where word embeddings are trained which reduces the one-hot-encoding phrases vectors to a dense with size of 300\n- A dropout Layers where with regularize the connection between the embedding layer and the following LSTM layer\n- The LSTM layer with 150 units\n- 3 fully connected Convolutional Layers, each time the number of units are increasing and the same happens to the kernel_size / filter\n- A GlobalMaxPooling to pool the best features from the ConvNets\n- They the best features are fed to a fully connected Dense neural units equal to 150 neurons\n- A dropout Layers where with regularize the connection between the the LSTM layer and the following 5 dense neurons\n- Finally the last layer is 5 neurons equal to the number of sentiment classes\n\nAs optimizer we use the *nadam* and the fitting goal is to maximize the model's accuracy, hence we use a checkpointer Keras function for that to save the best model. Furthemore, the earlyStopping function will be utlized to avoid overfitting\n\nAs the loss / coss function since we have categorical variables to be predicted we used the categorical_crossentropy loss function.\n\nIn every epoch the DL model will be evaluated in the validation set. If the accuracy improves in the new epoch the model will be saved. But after 3 iterations of no improvement the model the fitting will be stopped.\n\nFor all the DL models the random state will be set to 42 in order to the models be reproducable and create the same results in every run.\n\n"},{"metadata":{"trusted":true,"_uuid":"32a1e8350a8157b8b4f50ca1e2ff75736d88c484"},"cell_type":"code","source":"def build_dl_lstm_cnn_model(xtrain, ytrain, xvalid, yvalid, num_of_epochs, filename):\n\n    ## Network architecture\n    from keras.utils import to_categorical\n    from keras.callbacks import ModelCheckpoint\n    from keras.callbacks import EarlyStopping\n    from keras.optimizers import RMSprop, SGD, adam\n    from keras.layers import Masking\n    from keras.initializers import Constant\n    import time\n    \n    import warnings\n    warnings.simplefilter(action='ignore', category=FutureWarning)\n    \n    start_time = time.time()\n    \n    \n    from numpy.random import seed\n    seed(42)\n    from tensorflow import set_random_seed\n    set_random_seed(42)\n\n\n    embedding_size= embedding_dim + 2 #128\n    batch_size = 128\n    dropouts = 0.2\n    epochs = num_of_epochs\n    \n    model=Sequential()\n    model.add(Embedding(input_dim = vocabulary_size, output_dim = embedding_size, input_length=max_len, \n                        weights=[embedding_matrix], trainable = False))\n    \n    model.add(SpatialDropout1D(dropouts))\n    \n    model.add(LSTM(int(embedding_size/2), recurrent_dropout=dropouts, dropout=dropouts, \n                   return_sequences=True))\n    \n    model.add(Conv1D(128, kernel_size = 1, strides = 1,  padding='valid', activation='relu'))\n    model.add(Conv1D(256, kernel_size = 3, strides = 1,  padding='valid', activation='relu'))\n    model.add(Conv1D(512, kernel_size = 5, strides = 1,  padding='valid', activation='relu'))\n    \n    model.add(GlobalMaxPooling1D())\n    model.add(Dropout(dropouts))\n\n    model.add(Dense(int(embedding_size/2), activation=\"relu\"))\n    model.add(Dropout(dropouts))\n    \n    model.add(Dense(5, activation='softmax'))\n\n    print(model.summary())\n\n    model.compile(loss='categorical_crossentropy', optimizer = 'nadam', metrics=['categorical_accuracy']) # RMSprop\n\n    '''\n    saves the model weights after each epoch if the val_acc loss decreased\n    '''\n    checkpointer = ModelCheckpoint(monitor='val_categorical_accuracy', mode='max', filepath=''+filename+'.hdf5', verbose=2, save_best_only=True)\n    earlyStopping = EarlyStopping(monitor='val_categorical_accuracy', min_delta=0, patience=3, verbose=0, mode='max')\n\n    history = model.fit(x = xtrain, y = to_categorical(ytrain), validation_data=(xvalid, to_categorical(yvalid)), epochs=epochs, batch_size=batch_size, verbose=2, callbacks=[checkpointer, earlyStopping])\n\n    model = load_model(''+filename+'.hdf5')\n    \n    elapsed_time = time.time() - start_time\n    \n    return model, history, elapsed_time\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"a40c4ef26cadef0de182e5812aec7f6c286e3420"},"cell_type":"code","source":"\ndl_lstm_cnn_model, history_lstm_cnn, elapsed_time = build_dl_lstm_cnn_model(xtrain, ytrain, xvalid, yvalid, num_of_epochs=30, filename=\"lstm_cnn\")\nprint(\"Elapsed time in seconds:\", elapsed_time)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"85c53246ee2c4a6c1d35a0e6706e4033fad57706"},"cell_type":"markdown","source":"### Model Summary"},{"metadata":{"trusted":true,"_uuid":"ec155c8f3ca3cd18eb1c02d0e9413ca10dbd2675"},"cell_type":"code","source":"from keras.utils.vis_utils import plot_model\n\nplot_model(dl_lstm_cnn_model, to_file='dl_lstm_cnn_model.png', show_shapes=True, show_layer_names=True)\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom PIL import Image\n\nimg = Image.open('dl_lstm_cnn_model.png')\nplt.rcParams[\"figure.figsize\"] = (20,25)\nplt.axis('off')\nplt.grid(False)\nimgplot = plt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6cc8419dfc9660604f066d23a87ca9196c40abd3"},"cell_type":"markdown","source":"### Model Evaluation"},{"metadata":{"trusted":true,"_uuid":"7c8994d493bf2d4fe51118baecb219972640e4ff"},"cell_type":"code","source":"plot_history(history_lstm_cnn)\n\ny_pred_lstm_cnn = dl_lstm_cnn_model.predict_classes(xvalid, verbose=1)\nprint()\n\nprint(classification_report(yvalid, y_pred_lstm_cnn))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, y_pred_lstm_cnn))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, y_pred_lstm_cnn, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, y_pred_lstm_cnn, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, y_pred_lstm_cnn, average='weighted'))\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, y_pred_lstm_cnn)\n\n\ndl_performance_metrics_df.loc['LSTM_CNN']['training-time'] = elapsed_time\ndl_performance_metrics_df.loc['LSTM_CNN']['accuracy'] = accuracy_score(yvalid, y_pred_lstm_cnn)\ndl_performance_metrics_df.loc['LSTM_CNN']['F1-score'] = f1_score(yvalid, y_pred_lstm_cnn, average='weighted')\n\nprint(\"elapsed time:\", round(elapsed_time), \"seconds\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"057f896c062fec29990ccfc3320db08b9ee8647e"},"cell_type":"markdown","source":"The DL combination of LSTM and CNN (LSTM_CNN) network architucture has some promising results; The accuracy is 0.67 and the F1-score is 0.67. Nevertheless, there are many cases that are missclassified based on the high recall score per sentiment class."},{"metadata":{"_uuid":"88fbbd1bbd9d0399633dc5539418fd6cd1eb1a69"},"cell_type":"markdown","source":"### Deep Learning Model: Combining Bidirectional LSTM and CNN together (BiLSTM_CNN)\nFollowing the precious model lets took the liberty to combine and a bidirectional LSTM and a CNN together!\n\nThe following function builds a Deep Learning sequential network with the following stracture:\n\n- The Embedding Layers where word embeddings are trained which reduces the one-hot-encoding phrases vectors to a dense with size of 300\n- A dropout Layers where with regularize the connection between the embedding layer and the following Bidirectional LSTM layer\n- The Bidirectional LSTM layer with 150 units\n- 3 fully connected Convolutional Layers, each time the number of units are increasing and the same happens to the kernel_size / filter\n- A GlobalMaxPooling to pool the best features from the ConvNets\n- They the best features are fed to a fully connected Dense neural units equal to 150 neurons\n- A dropout Layers where with regularize the connection between the the LSTM layer and the following 5 dense neurons\n- Finally the last layer is 5 neurons equal to the number of sentiment classes\n\nAs optimizer we use the *nadam* and the fitting goal is to maximize the model's accuracy, hence we use a checkpointer Keras function for that to save the best model. Furthemore, the earlyStopping function will be utlized to avoid overfitting\n\nAs the loss / coss function since we have categorical variables to be predicted we used the categorical_crossentropy loss function.\n\nIn every epoch the DL model will be evaluated in the validation set. If the accuracy improves in the new epoch the model will be saved. But after 3 iterations of no improvement the model the fitting will be stopped.\n\nFor all the DL models the random state will be set to 42 in order to the models be reproducable and create the same results in every run."},{"metadata":{"trusted":true,"_uuid":"7a3b23fcd629c943670466c11e5ec2ad4a050b88"},"cell_type":"code","source":"def build_dl_bidirectional_lstm_cnn_model(xtrain, ytrain, xvalid, yvalid, num_of_epochs, filename):\n\n    ## Network architecture\n    from keras.utils import to_categorical\n    from keras.callbacks import ModelCheckpoint\n    from keras.callbacks import EarlyStopping\n    from keras.optimizers import RMSprop, SGD, adam\n    from keras.layers import Masking\n    from keras.initializers import Constant\n    import time\n    \n    import warnings\n    warnings.simplefilter(action='ignore', category=FutureWarning)\n    \n    start_time = time.time()\n    \n    \n    from numpy.random import seed\n    seed(42)\n    from tensorflow import set_random_seed\n    set_random_seed(42)\n    \n    \n    embedding_size = embedding_dim + 2\n    batch_size = 256\n    dropouts = 0.2\n    epochs = num_of_epochs\n    \n    model=Sequential()\n    model.add(Embedding(vocabulary_size, embedding_size, input_length=max_len, \n                        weights=[embedding_matrix], trainable = False))\n    \n    model.add(SpatialDropout1D(dropouts))\n    \n    model.add(Bidirectional(LSTM(int(embedding_size/2), recurrent_dropout=dropouts, dropout=dropouts, \n                   return_sequences=True)))\n    \n    model.add(Conv1D(128, kernel_size=2, strides = 1,  padding='valid', activation='relu'))\n    model.add(Conv1D(256, kernel_size=3, strides = 1,  padding='valid', activation='relu'))\n    model.add(Conv1D(512, kernel_size=5, strides = 1,  padding='valid', activation='relu'))\n    \n    model.add(GlobalMaxPooling1D())\n    model.add(Dropout(dropouts))\n    \n    model.add(Dense(int(embedding_size/2), activation=\"relu\"))\n    model.add(Dropout(dropouts))\n    \n    model.add(Dense(5, activation='softmax'))\n\n    print(model.summary())\n\n    model.compile(loss='categorical_crossentropy', optimizer = 'nadam', metrics=['categorical_accuracy']) # RMSprop\n\n    '''\n    saves the model weights after each epoch if the val_acc loss decreased\n    '''\n    checkpointer = ModelCheckpoint(monitor='val_categorical_accuracy', mode='max', filepath=''+filename+'.hdf5', verbose=2, save_best_only=True)\n    earlyStopping = EarlyStopping(monitor='val_categorical_accuracy', min_delta=0, patience=3, verbose=0, mode='max')\n\n    history = model.fit(x = xtrain, y = to_categorical(ytrain), validation_data=(xvalid, to_categorical(yvalid)), epochs=epochs, batch_size=batch_size, verbose=2, callbacks=[checkpointer, earlyStopping])\n\n    model = load_model(''+filename+'.hdf5')\n    \n    elapsed_time = time.time() - start_time\n    \n    return model, history, elapsed_time\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"72124b5bef79e02d1a9ef9e22f9c735d729a2820","scrolled":false},"cell_type":"code","source":"\ndl_bidirectional_lstm_cnn_model, history_bidirectional_lstm_cnn, elapsed_time = build_dl_bidirectional_lstm_cnn_model(xtrain, ytrain, xvalid, yvalid, num_of_epochs=50, filename=\"bidirectional_lstm_cnn\")\nprint(\"Elapsed time in seconds:\", elapsed_time)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9358fb4dc1945b78a42c8177dd69ade897f09141"},"cell_type":"markdown","source":"### Model Summary"},{"metadata":{"trusted":true,"_uuid":"2d58180c0fad97f920dc6e9c988dac93a5e00fa8"},"cell_type":"code","source":"from keras.utils.vis_utils import plot_model\n\nplot_model(dl_bidirectional_lstm_cnn_model, to_file='dl_bidirectional_lstm_cnn_model.png', show_shapes=True, show_layer_names=True)\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom PIL import Image\n\nimg = Image.open('dl_lstm_cnn_model.png')\nplt.rcParams[\"figure.figsize\"] = (20,25)\nplt.axis('off')\nplt.grid(False)\nimgplot = plt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c623b6789a3ede49f4b084f832bc4da457245acd"},"cell_type":"markdown","source":"### Model Evaluation"},{"metadata":{"trusted":true,"_uuid":"aea43e360c9842325ca6c8689e08f60dab457d5b"},"cell_type":"code","source":"plot_history(history_bidirectional_lstm_cnn)\n\ny_pred_bidirectional_lstm_cnn = dl_bidirectional_lstm_cnn_model.predict_classes(xvalid, verbose=1)\nprint()\n\nprint(classification_report(yvalid, y_pred_bidirectional_lstm_cnn))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, y_pred_bidirectional_lstm_cnn))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, y_pred_bidirectional_lstm_cnn, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, y_pred_bidirectional_lstm_cnn, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, y_pred_bidirectional_lstm_cnn, average='weighted'))\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, y_pred_bidirectional_lstm_cnn)\n\n\ndl_performance_metrics_df.loc['BiLSTM_CNN']['training-time'] = elapsed_time\ndl_performance_metrics_df.loc['BiLSTM_CNN']['accuracy'] = accuracy_score(yvalid, y_pred_bidirectional_lstm_cnn)\ndl_performance_metrics_df.loc['BiLSTM_CNN']['F1-score'] = f1_score(yvalid, y_pred_bidirectional_lstm_cnn, average='weighted')\n\nprint(\"elapsed time:\", round(elapsed_time), \"seconds\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2c975ea222afb05d719b65f57fd94a44fcd53961"},"cell_type":"markdown","source":"The DL combination of Bidirectional LSTM and CNN (Didirectional_LSTM_CNN) network architucture has the best results so far; The accuracy is 0.68 and the F1-score is 0.68. However, there are many cases that are missclassified based on the high recall score per sentiment class."},{"metadata":{"_uuid":"ba16dc919faec4066b9a6fcbededeb2a04f0355a"},"cell_type":"markdown","source":"### Summarizing DL Models based on their accuracy with Trainable Word Embeddings as Feature Extraction / Representation"},{"metadata":{"trusted":true,"_uuid":"86e677d0348833f4e98b98218aaaf6ac29c47e1d"},"cell_type":"code","source":"dl_performance_metrics_df.sort_values(by=\"accuracy\", ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9de47a94fd5969fa8d015be1bcf54940abda593b"},"cell_type":"code","source":"sns.set(rc={'figure.figsize':(15.27,6.27)})\ndl_performance_metrics_df.sort_values(by=\"accuracy\", ascending=False).accuracy.plot(kind=\"bar\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"14dbc536fd27a98610ddd26788a11de18c441e13"},"cell_type":"markdown","source":"### Summarizing DL Models based on their F1-score with Trainable Word Embeddings as Feature Extraction / Representation"},{"metadata":{"trusted":true,"_uuid":"92cd198418eaea1837b4922d3c135a6423af5e87"},"cell_type":"code","source":"dl_performance_metrics_df.sort_values(by=\"F1-score\", ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c6ab48faf235b2bf9451f3155b14e4561defb286"},"cell_type":"code","source":"sns.set(rc={'figure.figsize':(15.27,6.27)})\ndl_performance_metrics_df.sort_values(by=\"F1-score\", ascending=False)[\"F1-score\"].plot(kind=\"bar\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a7368799ea5a107afaca1bf4c0658a3e7ea505fd"},"cell_type":"markdown","source":"### Summarizing DL Models based on their training time with Trainable Word Embeddings as Feature Extraction / Representation"},{"metadata":{"trusted":true,"_uuid":"61210930a8e25f44373bca92372c406a9e5d5538"},"cell_type":"code","source":"dl_performance_metrics_df.sort_values(by=\"training-time\", ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b5838dd29e6d48c5e30a184f7fcd60449f124a7f"},"cell_type":"code","source":"sns.set(rc={'figure.figsize':(15.27,6.27)})\ndl_performance_metrics_df.sort_values(by=\"training-time\", ascending=False)[\"training-time\"].plot(kind=\"bar\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5b9bf897ba65971fbda214edefd50b01653e07ca"},"cell_type":"markdown","source":"### Ensemble Models\nEnsemble DL Models using the statistical mode for the predicted classed over the validation set."},{"metadata":{"trusted":true,"_uuid":"8cda2adb7d761f8c0d7ae7bba783db477123e8ba"},"cell_type":"code","source":"y_pred_lstm = dl_lstm_model.predict_classes(xvalid, verbose=1)\n\ny_pred_cnn = dl_cnn_model.predict_classes(xvalid, verbose=1)\n\ny_pred_bidirectional_lstm = dl_bidirectional_lstm_model.predict_classes(xvalid, verbose=1)\n\ny_pred_lstm_cnn = dl_lstm_cnn_model.predict_classes(xvalid, verbose=1)\n\ny_pred_bidirectional_lstm_cnn =  dl_bidirectional_lstm_cnn_model.predict_classes(xvalid, verbose=1)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0afe4654b7826439d21dcb5d13d2face4102cd70"},"cell_type":"markdown","source":"### Model Evaluation"},{"metadata":{"trusted":true,"_uuid":"37d85a02393f41e72feb08888a78600d118f057f"},"cell_type":"code","source":"ensemble_all_dl_pred_df = pd.DataFrame({'model_lstm':y_pred_lstm,\n                                                'model_bidirectional_lstm':y_pred_bidirectional_lstm,\n                                                'model_cnn':y_pred_cnn,\n                                                'model_bidirectional_lstm_cnn':y_pred_bidirectional_lstm_cnn,\n                                                'model_lstm_cnn':y_pred_lstm_cnn,\n                                                })\n\n\npred_mode = ensemble_all_dl_pred_df.agg('mode',axis=1)[0].values\n\nprint()\nprint(classification_report(yvalid, pred_mode))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, pred_mode))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, pred_mode, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, pred_mode, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, pred_mode, average='weighted'))\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, pred_mode)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dc0a64098e4d270008acb4a038cc8a5a4b785763"},"cell_type":"markdown","source":"Ensembling the DL models here over the validation set using the statistical mode on the predicted classes to conclude the ensemble new class based on the majority of the predictions. Ensemble technique improved even more the performance;Accuracy and F1-score are 0.69 and 0.69 respectively. Still high recall per sentiment means that there are many missclassified cases."},{"metadata":{"_uuid":"7678acedcde2a0b54b52719a25492f2b51b27695"},"cell_type":"markdown","source":"### Making Predictions and preparing the submissions' csv files"},{"metadata":{"trusted":true,"_uuid":"7d483faab80f4fe45225f3a14ec9bd8812da9211"},"cell_type":"code","source":"xtest = tokenizer.texts_to_sequences(df_test.Phrase.values)\nxtest = pad_sequences(xtest, maxlen=max_len)\n\ny_pred_test_lstm = dl_lstm_model.predict_classes(xtest, verbose=1)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = y_pred_test_lstm\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_lstm.csv',index=False)\n\n\ny_pred_test_bidirectional_lstm = dl_bidirectional_lstm_model.predict_classes(xtest, verbose=1)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = y_pred_test_bidirectional_lstm\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_bidirectional_lstm.csv',index=False)\n\n\ny_pred_test_cnn = dl_cnn_model.predict_classes(xtest, verbose=1)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = y_pred_test_cnn\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_cnn.csv',index=False)\n\n\ny_pred_test_lstm_cnn = dl_lstm_cnn_model.predict_classes(xtest, verbose=1)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = y_pred_test_lstm_cnn\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_lstm_cnn.csv',index=False)\n\n\ny_pred_test_bidirectional_lstm_cnn = dl_bidirectional_lstm_cnn_model.predict_classes(xtest, verbose=1)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = y_pred_test_bidirectional_lstm_cnn\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_bidirectional_lstm_cnn.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c2f8a55c2a2e397c0efc410f126be6f0bd470bae"},"cell_type":"markdown","source":"### Ensebling Predictions over the test set"},{"metadata":{"trusted":true,"_uuid":"3caf5ec939eebd841209c67930bd387f1fedf8d0"},"cell_type":"code","source":"ensemble_all_dl_pred_test_df = pd.DataFrame({'model_lstm':y_pred_test_lstm,\n                                                'model_bidirectional_lstm':y_pred_test_bidirectional_lstm,\n                                                'model_cnn':y_pred_test_cnn,\n                                                'model_lstm_cnn':y_pred_test_lstm_cnn,\n                                                'model_bidirectional_lstm_cnn':y_pred_test_bidirectional_lstm_cnn})\n\n\npred_test_mode = ensemble_all_dl_pred_test_df.agg('mode',axis=1)[0].values\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = pred_test_mode\nsubmission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_ensemble.csv',index=False)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9c177e1d0f18509ce87a74ddebc0120a694baa15"},"cell_type":"markdown","source":"## Summary\nThe Deep Learning models that were developed here combined with pre-trained Word Embeddings from Stanford NLP GLoVE vectors size of 300 as feature extraction / representation fitted in most of the cases and modeled the dataset with accuracy up to almost 0.69 in the validation set. Maybe more exhaustive experiments must be done with the current installement to improve accuracy even do we should not forget that the datasrt is very strange and the sentiments alter even with the absence of a single word or punctuation.\n\n__________________________"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}