{"cells":[{"metadata":{"_uuid":"01f361ddc47e0b386595316fe3d7f4dabbd260db"},"cell_type":"markdown","source":"**Notebook Objective:**\n\nObjective of the notebook is to look at the different pretrained embeddings provided in the dataset and to see how they are useful in the model building process. \n\nFirst let us import the necessary modules and read the input data."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport time\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom tqdm import tqdm\nimport math\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics\n\nimport keras\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Dense, Input, LSTM, Embedding, Dropout, Activation, CuDNNGRU, Conv1D, CuDNNLSTM\nfrom keras.layers import Bidirectional, GlobalMaxPool1D\nfrom keras.models import Model\nfrom keras import initializers, regularizers, constraints, optimizers, layers\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\")\ntest_df = pd.read_csv(\"../input/test.csv\")\nprint(\"Train shape : \",train_df.shape)\nprint(\"Test shape : \",test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ad1e9fdf97f3291a7d1797b25f0c7a0c5d1f1edd"},"cell_type":"markdown","source":"Next steps are as follows:\n * Split the training dataset into train and val sample. Cross validation is a time consuming process and so let us do simple train val split.\n * Fill up the missing values in the text column with '_na_'\n * Tokenize the text column and convert them to vector sequences\n * Pad the sequence as needed - if the number of words in the text is greater than 'max_len' trunacate them to 'max_len' or if the number of words in the text is lesser than 'max_len' add zeros for remaining values."},{"metadata":{"trusted":true,"_uuid":"ba5a1b8109dee2c9fbc628d5da4a7c3447d42fb8"},"cell_type":"code","source":"## split to train and val\ntrain_df, val_df = train_test_split(train_df, test_size=0.1, random_state=2018)\n\n## some config values \nembed_size = 300 # how big is each word vector\nmax_features = 50000 # how many unique words to use (i.e num rows in embedding vector)\nmaxlen = 100 # max number of words in a question to use\n\n## fill up the missing values\ntrain_X = train_df[\"question_text\"].fillna(\"_na_\").values\nval_X = val_df[\"question_text\"].fillna(\"_na_\").values\ntest_X = test_df[\"question_text\"].fillna(\"_na_\").values\n\n## Tokenize the sentences\ntokenizer = Tokenizer(num_words=max_features)\ntokenizer.fit_on_texts(list(train_X))\ntrain_X = tokenizer.texts_to_sequences(train_X)\nval_X = tokenizer.texts_to_sequences(val_X)\ntest_X = tokenizer.texts_to_sequences(test_X)\n\n## Pad the sentences \ntrain_X = pad_sequences(train_X, maxlen=maxlen)\nval_X = pad_sequences(val_X, maxlen=maxlen)\ntest_X = pad_sequences(test_X, maxlen=maxlen)\n\n## Get the target values\ntrain_y = train_df['target'].values\nval_y = val_df['target'].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2a5f324273d8e4726a6f0f9206170845d5ead890"},"cell_type":"markdown","source":"**1. Without Pretrained Embeddings - Bidirectional GRU model:**\n\nNow that we are done with all the necessary preprocessing steps, we can first train a Bidirectional GRU model. We will not use any pre-trained word embeddings for this model and the embeddings will be learnt from scratch. Please check out the model summary for the details of the layers used. "},{"metadata":{"trusted":true,"_uuid":"3cfab26c6cced33ef7ab84f0d36997113131d530"},"cell_type":"code","source":"inp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size)(inp)\nx = Bidirectional(CuDNNGRU(64, return_sequences=True))(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(16, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b71267b19ec4b8a4c977cb3ad87b636390e5cdb9"},"cell_type":"markdown","source":"Train the model using train sample and monitor the metric on the valid sample. This is just a sample model running for 2 epochs. Changing the epochs, batch_size and model parameters might give us a better model."},{"metadata":{"trusted":true,"_uuid":"ef1e1015e7c3ab5bc5d9774e49820c4b286d7847"},"cell_type":"code","source":"## Train the model \nhistory = model.fit(train_X, train_y, batch_size=512, epochs=2, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a72ba82481de9f96c62c334cc40bd1d134d38e2d"},"cell_type":"markdown","source":"Now let us get the validation sample predictions and also get the best threshold for F1 score. "},{"metadata":{"trusted":true,"_uuid":"47b63dca0247a08a808db7ae6eea33065c554948"},"cell_type":"code","source":"pred_noemb_val_y_gru = model.predict([val_X], batch_size=1024, verbose=1)\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"At threshold {0}, F1 score is {1:.4f}, Precision is {2:.4f} and Recall is {3:.4f}\".format(thresh, metrics.f1_score(val_y, (pred_noemb_val_y_gru>thresh).astype(int)), metrics.precision_score(val_y, (pred_noemb_val_y_gru>thresh).astype(int)),metrics.recall_score(val_y, (pred_noemb_val_y_gru>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot of F1 score, Precision and Recall for different thresholds \nf1s = []\nprecisions = []\nrecalls = []\n    \nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    f1 = metrics.f1_score(val_y, (pred_noemb_val_y_gru>thresh).astype(int))\n    precision = metrics.precision_score(val_y, (pred_noemb_val_y_gru>thresh).astype(int))\n    recall = metrics.recall_score(val_y, (pred_noemb_val_y_gru>thresh).astype(int))\n\n    f1s.append(f1)\n    precisions.append(precision)\n    recalls.append(recall)\n\n    data = pd.DataFrame(data = {\n        'F1': f1s,\n        'Precision': precisions,\n        'Recall': recalls})\nsns.lineplot(data=data, palette='muted', linewidth=2.5, dashes=False)\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Train/Test loss v/s Epoch\ndata = pd.DataFrame(data={'Train': history.history['loss'], 'Test': history.history['val_loss']})\nax = sns.lineplot(data=data, palette=\"pastel\", linewidth=2.5, dashes=False)\nax.set(xlabel='Epoch', ylabel='Loss', title='Loss')\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"383e51177bf33da0b8fee42dd6a093908b808f64"},"cell_type":"markdown","source":"Now let us get the test set predictions as well and save them"},{"metadata":{"trusted":true,"_uuid":"a88df747f43259bab84447b50e45aa9e978f2cee"},"cell_type":"code","source":"pred_noemb_test_y_gru = model.predict([test_X], batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that our model building is done, it might be a good idea to clean up some memory before we go to the next step."},{"metadata":{"trusted":true},"cell_type":"code","source":"del model, inp, x, history, data, ax\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**2. Without Pretrained Embeddings - Bidirectional LSTM model:**\n\nNow second we are training a Bidirectional LSTM model. We will not use any pre-trained word embeddings for this model and the embeddings will be learnt from scratch. Please check out the model summary for the details of the layers used. "},{"metadata":{"trusted":true},"cell_type":"code","source":"inp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size)(inp)\nx = Bidirectional(CuDNNLSTM(64, return_sequences=True))(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(16, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Train the model using train sample and monitor the metric on the valid sample. This is just a sample model running for 2 epochs. Changing the epochs, batch_size and model parameters might give us a better model."},{"metadata":{"trusted":true},"cell_type":"code","source":"## Train the model \nhistory = model.fit(train_X, train_y, batch_size=512, epochs=2, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let us get the validation sample predictions and also get the best threshold for F1 score. "},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_noemb_val_y_lstm = model.predict([val_X], batch_size=1024, verbose=1)\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"At threshold {0}, F1 score is {1}, Precision is {2} and Recall is {3}\".format(thresh, metrics.f1_score(val_y, (pred_noemb_val_y_lstm>thresh).astype(int)), metrics.precision_score(val_y, (pred_noemb_val_y_lstm>thresh).astype(int)),metrics.recall_score(val_y, (pred_noemb_val_y_lstm>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot of F1 score, Precision and Recall for different thresholds \nf1s = []\nprecisions = []\nrecalls = []\n    \nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    f1 = metrics.f1_score(val_y, (pred_noemb_val_y_lstm>thresh).astype(int))\n    precision = metrics.precision_score(val_y, (pred_noemb_val_y_lstm>thresh).astype(int))\n    recall = metrics.recall_score(val_y, (pred_noemb_val_y_lstm>thresh).astype(int))\n\n    f1s.append(f1)\n    precisions.append(precision)\n    recalls.append(recall)\n\n    data = pd.DataFrame(data = {\n        'F1': f1s,\n        'Precision': precisions,\n        'Recall': recalls})\nsns.lineplot(data=data, palette='muted', linewidth=2.5, dashes=False)\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Train/Test loss v/s Epoch\ndata = pd.DataFrame(data={'Train': history.history['loss'], 'Test': history.history['val_loss']})\nax = sns.lineplot(data=data, palette=\"pastel\", linewidth=2.5, dashes=False)\nax.set(xlabel='Epoch', ylabel='Loss', title='Loss')\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let us get the test set predictions as well and save them"},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_noemb_test_y_lstm = model.predict([test_X], batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f3831ee610fee119c9851b5ca29b2d80b102ae6a"},"cell_type":"markdown","source":"Now that our model building is done, it might be a good idea to clean up some memory before we go to the next step."},{"metadata":{"trusted":true,"_uuid":"a36a071fb50f6c120e099b5fe27ad6ac977f1125"},"cell_type":"code","source":"del model, inp, x, history, data, ax\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"717f7dcd5ccf71e83d0f062e221c46db39845e4d"},"cell_type":"markdown","source":"So we got some baseline GRU and LSTM model without pre-trained embeddings. Now let us use the provided embeddings and rebuild the model again to see the performance. \n\n"},{"metadata":{"trusted":true,"_uuid":"b9d263852f653e466e24f9827548d7d1a7ee7262"},"cell_type":"code","source":"!ls ../input/embeddings/","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7c0010e518288bc7f588776c58610949140a139a"},"cell_type":"markdown","source":"We have four different types of embeddings.\n * GoogleNews-vectors-negative300 - https://code.google.com/archive/p/word2vec/\n * glove.840B.300d - https://nlp.stanford.edu/projects/glove/\n * paragram_300_sl999 - https://cogcomp.org/page/resource_view/106\n * wiki-news-300d-1M - https://fasttext.cc/docs/en/english-vectors.html\n \n A very good explanation for different types of embeddings are given in this [kernel](https://www.kaggle.com/sbongo/do-pretrained-embeddings-give-you-the-extra-edge). Please refer the same for more details..\n\n**   **\n\n\n**3. Glove Embeddings - Bidirectional GRU model:**\n\nIn this section, let us use the Glove embeddings and rebuild the GRU model."},{"metadata":{"trusted":true,"_uuid":"23f130e80159bb1701e449e2e91199dbfff1f1d4"},"cell_type":"code","source":"EMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n        \ninp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)\nx = Bidirectional(CuDNNGRU(64, return_sequences=True))(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(16, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a560ab0dbab9cf6fdbdae6721ec030e300f19d78"},"cell_type":"code","source":"history = model.fit(train_X, train_y, batch_size=512, epochs=2, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ff43855164472de035a5a1d80b3db4838684701a"},"cell_type":"code","source":"pred_glove_val_y_gru = model.predict([val_X], batch_size=1024, verbose=1)\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"At threshold {0}, F1 score is {1}, Precision is {2} and Recall is {3}\".format(thresh, metrics.f1_score(val_y, (pred_glove_val_y_gru>thresh).astype(int)), metrics.precision_score(val_y, (pred_glove_val_y_gru>thresh).astype(int)),metrics.recall_score(val_y, (pred_glove_val_y_gru>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot of F1 score, Precision and Recall for different thresholds \nf1s = []\nprecisions = []\nrecalls = []\n    \nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    f1 = metrics.f1_score(val_y, (pred_glove_val_y_gru>thresh).astype(int))\n    precision = metrics.precision_score(val_y, (pred_glove_val_y_gru>thresh).astype(int))\n    recall = metrics.recall_score(val_y, (pred_glove_val_y_gru>thresh).astype(int))\n\n    f1s.append(f1)\n    precisions.append(precision)\n    recalls.append(recall)\n\n    data = pd.DataFrame(data = {\n        'F1': f1s,\n        'Precision': precisions,\n        'Recall': recalls})\nsns.lineplot(data=data, palette='muted', linewidth=2.5, dashes=False)\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Train/Test loss v/s Epoch\ndata = pd.DataFrame(data={'Train': history.history['loss'], 'Test': history.history['val_loss']})\nax = sns.lineplot(data=data, palette=\"pastel\", linewidth=2.5, dashes=False)\nax.set(xlabel='Epoch', ylabel='Loss', title='Loss')\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d2a33c252f31fddcc65896053184226128562776"},"cell_type":"markdown","source":"Results seem to be better than the model without pretrained embeddings."},{"metadata":{"trusted":true,"_uuid":"d51ff8ed6a87b488fec3ac84ca50df661d7c8193"},"cell_type":"code","source":"pred_glove_test_y_gru = model.predict([test_X], batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"39d4fedab4ac170863a0ee1ca3aa9be1ee58fe02"},"cell_type":"code","source":"del word_index, embeddings_index, all_embs, embedding_matrix, model, inp, x, history, data, ax\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**4. Glove Embeddings - Bidirectional LSTM model:**\n\n* In this section, let us use the Glove embeddings and rebuild the LSTM model."},{"metadata":{"trusted":true},"cell_type":"code","source":"EMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n        \ninp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)\nx = Bidirectional(CuDNNLSTM(64, return_sequences=True))(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(16, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(train_X, train_y, batch_size=512, epochs=2, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_glove_val_y_lstm = model.predict([val_X], batch_size=1024, verbose=1)\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"At threshold {0}, F1 score is {1}, Precision is {2} and Recall is {3}\".format(thresh, metrics.f1_score(val_y, (pred_glove_val_y_lstm>thresh).astype(int)), metrics.precision_score(val_y, (pred_glove_val_y_lstm>thresh).astype(int)),metrics.recall_score(val_y, (pred_glove_val_y_lstm>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot of F1 score, Precision and Recall for different thresholds \nf1s = []\nprecisions = []\nrecalls = []\n    \nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    f1 = metrics.f1_score(val_y, (pred_glove_val_y_lstm>thresh).astype(int))\n    precision = metrics.precision_score(val_y, (pred_glove_val_y_lstm>thresh).astype(int))\n    recall = metrics.recall_score(val_y, (pred_glove_val_y_lstm>thresh).astype(int))\n\n    f1s.append(f1)\n    precisions.append(precision)\n    recalls.append(recall)\n\n    data = pd.DataFrame(data = {\n        'F1': f1s,\n        'Precision': precisions,\n        'Recall': recalls})\nsns.lineplot(data=data, palette='muted', linewidth=2.5, dashes=False)\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Train/Test loss v/s Epoch\ndata = pd.DataFrame(data={'Train': history.history['loss'], 'Test': history.history['val_loss']})\nax = sns.lineplot(data=data, palette=\"pastel\", linewidth=2.5, dashes=False)\nax.set(xlabel='Epoch', ylabel='Loss', title='Loss')\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Results seem to be better than the model without pretrained embeddings."},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_glove_test_y_lstm = model.predict([test_X], batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del word_index, embeddings_index, all_embs, embedding_matrix, model, inp, x, history, data, ax\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc6bab22dd12a09378f4b8b159cb7a5d88a3e7c0"},"cell_type":"markdown","source":"**5. Wiki News FastText Embeddings - Bidirectional GRU model:**\n\nNow let us use the FastText embeddings trained on Wiki News corpus in place of Glove embeddings and rebuild the GRU model."},{"metadata":{"trusted":true,"_uuid":"6f3d0fd28dd2b04eaccb732b96b872e5a223d962"},"cell_type":"code","source":"EMBEDDING_FILE = '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE) if len(o)>100)\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n        \ninp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)\nx = Bidirectional(CuDNNGRU(64, return_sequences=True))(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(16, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"47238831a4701c8a67dc7ecb130ac1402baf7bb2"},"cell_type":"code","source":"history = model.fit(train_X, train_y, batch_size=512, epochs=2, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b7ab4100f723ad535528865b1edc7896bce80223"},"cell_type":"code","source":"pred_fasttext_val_y_gru = model.predict([val_X], batch_size=1024, verbose=1)\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"At threshold {0}, F1 score is {1}, Precision is {2} and Recall is {3}\".format(thresh, metrics.f1_score(val_y, (pred_fasttext_val_y_gru>thresh).astype(int)), metrics.precision_score(val_y, (pred_fasttext_val_y_gru>thresh).astype(int)),metrics.recall_score(val_y, (pred_fasttext_val_y_gru>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot of F1 score, Precision and Recall for different thresholds \nf1s = []\nprecisions = []\nrecalls = []\n    \nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    f1 = metrics.f1_score(val_y, (pred_fasttext_val_y_gru>thresh).astype(int))\n    precision = metrics.precision_score(val_y, (pred_fasttext_val_y_gru>thresh).astype(int))\n    recall = metrics.recall_score(val_y, (pred_fasttext_val_y_gru>thresh).astype(int))\n\n    f1s.append(f1)\n    precisions.append(precision)\n    recalls.append(recall)\n\n    data = pd.DataFrame(data = {\n        'F1': f1s,\n        'Precision': precisions,\n        'Recall': recalls})\nsns.lineplot(data=data, palette='muted', linewidth=2.5, dashes=False)\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Train/Test loss v/s Epoch\ndata = pd.DataFrame(data={'Train': history.history['loss'], 'Test': history.history['val_loss']})\nax = sns.lineplot(data=data, palette=\"pastel\", linewidth=2.5, dashes=False)\nax.set(xlabel='Epoch', ylabel='Loss', title='Loss')\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3216362afb0f49579d287a06f13adf8cd7d8b0cf"},"cell_type":"code","source":"pred_fasttext_test_y_gru = model.predict([test_X], batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f24f9753ff1d933fa4f75a0ba34df305632d6e93"},"cell_type":"code","source":"del word_index, embeddings_index, all_embs, embedding_matrix, model, inp, x, history, data, ax\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**6. Wiki News FastText Embeddings - Bidirectional LSTM model:**\n\nNow let us use the FastText embeddings trained on Wiki News corpus in place of Glove embeddings and rebuild the LSTM model."},{"metadata":{"trusted":true},"cell_type":"code","source":"EMBEDDING_FILE = '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE) if len(o)>100)\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n        \ninp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)\nx = Bidirectional(CuDNNLSTM(64, return_sequences=True))(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(16, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(train_X, train_y, batch_size=512, epochs=2, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_fasttext_val_y_lstm = model.predict([val_X], batch_size=1024, verbose=1)\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"At threshold {0}, F1 score is {1}, Precision is {2} and Recall is {3}\".format(thresh, metrics.f1_score(val_y, (pred_fasttext_val_y_lstm>thresh).astype(int)), metrics.precision_score(val_y, (pred_fasttext_val_y_lstm>thresh).astype(int)),metrics.recall_score(val_y, (pred_fasttext_val_y_lstm>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot of F1 score, Precision and Recall for different thresholds \nf1s = []\nprecisions = []\nrecalls = []\n    \nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    f1 = metrics.f1_score(val_y, (pred_fasttext_val_y_lstm>thresh).astype(int))\n    precision = metrics.precision_score(val_y, (pred_fasttext_val_y_lstm>thresh).astype(int))\n    recall = metrics.recall_score(val_y, (pred_fasttext_val_y_lstm>thresh).astype(int))\n\n    f1s.append(f1)\n    precisions.append(precision)\n    recalls.append(recall)\n\n    data = pd.DataFrame(data = {\n        'F1': f1s,\n        'Precision': precisions,\n        'Recall': recalls})\nsns.lineplot(data=data, palette='muted', linewidth=2.5, dashes=False)\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Train/Test loss v/s Epoch\ndata = pd.DataFrame(data={'Train': history.history['loss'], 'Test': history.history['val_loss']})\nax = sns.lineplot(data=data, palette=\"pastel\", linewidth=2.5, dashes=False)\nax.set(xlabel='Epoch', ylabel='Loss', title='Loss')\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_fasttext_test_y_lstm = model.predict([test_X], batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del word_index, embeddings_index, all_embs, embedding_matrix, model, inp, x, history, data, ax\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4ca44ac68bf404b9c26e07fbcc9c8ac793e04510"},"cell_type":"markdown","source":"**7. Paragram Embeddings - Bidirectional GRU model:**\n\nIn this section, we can use the paragram embeddings and build the model and make predictions."},{"metadata":{"trusted":true,"_uuid":"25ec1aac4aedbf431a2d30de64030ce8e3203c18"},"cell_type":"code","source":"EMBEDDING_FILE = '../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE, encoding=\"utf8\", errors='ignore') if len(o)>100)\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n        \ninp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)\nx = Bidirectional(CuDNNGRU(64, return_sequences=True))(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(16, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cc188f2787ea7b98d3a40953a95a5fc09ff2764d"},"cell_type":"code","source":"history = model.fit(train_X, train_y, batch_size=512, epochs=2, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9abdfd1cf15257f2c0c2181a13327796e8d4584e"},"cell_type":"code","source":"pred_paragram_val_y_gru = model.predict([val_X], batch_size=1024, verbose=1)\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"At threshold {0}, F1 score is {1}, Precision is {2} and Recall is {3}\".format(thresh, metrics.f1_score(val_y, (pred_paragram_val_y_gru>thresh).astype(int)), metrics.precision_score(val_y, (pred_paragram_val_y_gru>thresh).astype(int)),metrics.recall_score(val_y, (pred_paragram_val_y_gru>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot of F1 score, Precision and Recall for different thresholds \nf1s = []\nprecisions = []\nrecalls = []\n    \nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    f1 = metrics.f1_score(val_y, (pred_paragram_val_y_gru>thresh).astype(int))\n    precision = metrics.precision_score(val_y, (pred_paragram_val_y_gru>thresh).astype(int))\n    recall = metrics.recall_score(val_y, (pred_paragram_val_y_gru>thresh).astype(int))\n\n    f1s.append(f1)\n    precisions.append(precision)\n    recalls.append(recall)\n\n    data = pd.DataFrame(data = {\n        'F1': f1s,\n        'Precision': precisions,\n        'Recall': recalls})\nsns.lineplot(data=data, palette='muted', linewidth=2.5, dashes=False)\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Train/Test loss v/s Epoch\ndata = pd.DataFrame(data={'Train': history.history['loss'], 'Test': history.history['val_loss']})\nax = sns.lineplot(data=data, palette=\"pastel\", linewidth=2.5, dashes=False)\nax.set(xlabel='Epoch', ylabel='Loss', title='Loss')\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"99cb9f6145da909bd7436e46d47547efc097499d"},"cell_type":"code","source":"pred_paragram_test_y_gru = model.predict([test_X], batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"af087d21bdb4358701e31aded6b522accd5a8a64"},"cell_type":"code","source":"del word_index, embeddings_index, all_embs, embedding_matrix, model, inp, x, history, data, ax\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**8. Paragram Embeddings - Bidirectional LSTM model:**\n\nIn this section, we can use the paragram embeddings and build the model and make predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"EMBEDDING_FILE = '../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE, encoding=\"utf8\", errors='ignore') if len(o)>100)\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n        \ninp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)\nx = Bidirectional(CuDNNLSTM(64, return_sequences=True))(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(16, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(train_X, train_y, batch_size=512, epochs=2, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_paragram_val_y_lstm = model.predict([val_X], batch_size=1024, verbose=1)\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"At threshold {0}, F1 score is {1}, Precision is {2} and Recall is {3}\".format(thresh, metrics.f1_score(val_y, (pred_paragram_val_y_lstm>thresh).astype(int)), metrics.precision_score(val_y, (pred_paragram_val_y_lstm>thresh).astype(int)),metrics.recall_score(val_y, (pred_paragram_val_y_lstm>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot of F1 score, Precision and Recall for different thresholds \nf1s = []\nprecisions = []\nrecalls = []\n    \nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    f1 = metrics.f1_score(val_y, (pred_paragram_val_y_lstm>thresh).astype(int))\n    precision = metrics.precision_score(val_y, (pred_paragram_val_y_lstm>thresh).astype(int))\n    recall = metrics.recall_score(val_y, (pred_paragram_val_y_lstm>thresh).astype(int))\n\n    f1s.append(f1)\n    precisions.append(precision)\n    recalls.append(recall)\n\n    data = pd.DataFrame(data = {\n        'F1': f1s,\n        'Precision': precisions,\n        'Recall': recalls})\nsns.lineplot(data=data, palette='muted', linewidth=2.5, dashes=False)\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Train/Test loss v/s Epoch\ndata = pd.DataFrame(data={'Train': history.history['loss'], 'Test': history.history['val_loss']})\nax = sns.lineplot(data=data, palette=\"pastel\", linewidth=2.5, dashes=False)\nax.set(xlabel='Epoch', ylabel='Loss', title='Loss')\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_paragram_test_y_lstm = model.predict([test_X], batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del word_index, embeddings_index, all_embs, embedding_matrix, model, inp, x, history, data, ax\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e1312b7a4c3b67ca4ebd26fb083dbac3b6635dc2"},"cell_type":"markdown","source":"**Observations:**\n * Overall pretrained embeddings seem to give better results comapred to non-pretrained model. \n * The performance of the different pretrained embeddings are almost similar.\n \n**Final Blend:**\n\nThough the results of the models with different pre-trained embeddings are similar, there is a good chance that they might capture different type of information from the data. So let us do a blend of these three models by averaging their predictions."},{"metadata":{},"cell_type":"markdown","source":"**Averaging all 3 embeddings - Bidirectional GRU Model**"},{"metadata":{"trusted":true,"_uuid":"449bc59fdc9a719aa0759ac51a4481df113604ca"},"cell_type":"code","source":"pred_val_y_gru = 0.34*pred_glove_val_y_gru + 0.33*pred_fasttext_val_y_gru + 0.33*pred_paragram_val_y_gru \nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"At threshold {0}, F1 score is {1}, Precision is {2} and Recall is {3}\".format(thresh, metrics.f1_score(val_y, (pred_val_y_gru>thresh).astype(int)), metrics.precision_score(val_y, (pred_val_y_gru>thresh).astype(int)),metrics.recall_score(val_y, (pred_val_y_gru>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot of F1 score, Precision and Recall for different thresholds \nf1s = []\nprecisions = []\nrecalls = []\n    \nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    f1 = metrics.f1_score(val_y, (pred_val_y_gru>thresh).astype(int))\n    precision = metrics.precision_score(val_y, (pred_val_y_gru>thresh).astype(int))\n    recall = metrics.recall_score(val_y, (pred_val_y_gru>thresh).astype(int))\n\n    f1s.append(f1)\n    precisions.append(precision)\n    recalls.append(recall)\n\n    data = pd.DataFrame(data = {\n        'F1': f1s,\n        'Precision': precisions,\n        'Recall': recalls})\nsns.lineplot(data=data, palette='muted', linewidth=2.5, dashes=False)\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Averaging all 3 embeddings - Bidirectional LSTM Model**"},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_val_y_lstm = 0.34*pred_glove_val_y_lstm + 0.33*pred_fasttext_val_y_lstm + 0.33*pred_paragram_val_y_lstm \nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"At threshold {0}, F1 score is {1}, Precision is {2} and Recall is {3}\".format(thresh, metrics.f1_score(val_y, (pred_val_y_lstm>thresh).astype(int)), metrics.precision_score(val_y, (pred_val_y_lstm>thresh).astype(int)),metrics.recall_score(val_y, (pred_val_y_lstm>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plot of F1 score, Precision and Recall for different thresholds \nf1s = []\nprecisions = []\nrecalls = []\n    \nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    f1 = metrics.f1_score(val_y, (pred_val_y_lstm>thresh).astype(int))\n    precision = metrics.precision_score(val_y, (pred_val_y_lstm>thresh).astype(int))\n    recall = metrics.recall_score(val_y, (pred_val_y_lstm>thresh).astype(int))\n\n    f1s.append(f1)\n    precisions.append(precision)\n    recalls.append(recall)\n\n    data = pd.DataFrame(data = {\n        'F1': f1s,\n        'Precision': precisions,\n        'Recall': recalls})\nsns.lineplot(data=data, palette='muted', linewidth=2.5, dashes=False)\nsns.despine()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4fdbeffc0f84643d2832eec49234bd9d6c6e216b"},"cell_type":"markdown","source":"The result seems to better than individual pre-trained models and so we let us create a submission file using this model blend."},{"metadata":{"trusted":true,"_uuid":"c90fb4a4ef1b3b2ea06563a6901deac1b38822f3"},"cell_type":"code","source":"pred_test_y_gru = 0.34*pred_glove_test_y_gru + 0.33*pred_fasttext_test_y_gru + 0.33*pred_paragram_test_y_gru\npred_test_y_gru = (pred_test_y_gru>0.35).astype(int)\nout_df = pd.DataFrame({\"qid\":test_df[\"qid\"].values})\nout_df['prediction'] = pred_test_y_gru\nout_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f6797ab73bdd5bdb8c8f6d80ec361c50a2b0f56f"},"cell_type":"markdown","source":"\n**References:**\n\nThanks to the below kernels which helped me with this one. \n1. https://www.kaggle.com/jhoward/improved-lstm-baseline-glove-dropout\n2. https://www.kaggle.com/sbongo/do-pretrained-embeddings-give-you-the-extra-edge"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}