{"cells":[{"metadata":{"_uuid":"f0c0400943ea1ec794a3bbd7776ddeb79971074c"},"cell_type":"markdown","source":"# Analysis on pre trained embedding which will give best baseline results \n# Full Depth analysis "},{"metadata":{"_uuid":"cf7235f966fb3b78196a37d94653e3116d1f25e9"},"cell_type":"markdown","source":"![](https://www.pyimagesearch.com/wp-content/uploads/2017/12/not_santa_detector_dl_logos.jpg)\n**This notebook attempts to tackle this classification problem by using Keras LSTM. While there are many notebook out there that are already tackling using this approach, I feel that there isn't enough explanation to what is going on each step. As someone who has been using vanilla Tensorflow, and recently embraced the wonderful world of Keras, I hope to share with fellow beginners the intuition that I gained from my research and study. **\n\n**Join me as we walk through it. **"},{"metadata":{"_uuid":"01f361ddc47e0b386595316fe3d7f4dabbd260db"},"cell_type":"markdown","source":"**Notebook Objective:**\n\nObjective of the notebook is to look at the different pretrained embeddings provided in the dataset and to see how they are useful in the model building process. \n\nFirst let us import the necessary modules and read the input data."},{"metadata":{"_uuid":"5bba6c67af6cf6190a855c481d56a3f6afee5c0e"},"cell_type":"markdown","source":"In this kernel, we shall see if pretrained embeddings like Word2Vec, GLOVE and Fasttext, which are pretrained using billions of words could improve our accuracy score as compared to training our own embedding. We will compare the performance of models using these pretrained embeddings against the baseline model that doesn't use any pretrained embeddings in my previous kernel [here](https://www.kaggle.com/sbongo/for-beginners-tackling-toxic-using-keras).\n\n![](https://qph.fs.quoracdn.net/main-qimg-3e812fd164a08f5e4f195000fecf988f)\n\nPerhaps it's a good idea to briefly step in the world of word embeddings and see what's the difference between Word2Vec, GLOVE and Fasttext.\n\nEmbeddings generally represent geometrical encodings of words based on how frequently appear together in a text corpus. Various implementations of word embeddings described below differs in the way as how they are constructed.\n\n**Word2Vec**\n\nThe main idea behind it is that you train a model on the context on each word, so similar words will have similar numerical representations.\n\nJust like a normal feed-forward densely connected neural network(NN) where you have a set of independent variables and a target dependent variable that you are trying to predict, you first break your sentence into words(tokenize) and create a number of pairs of words, depending on the window size. So one of the combination could be a pair of words such as ('cat','purr'), where cat is the independent variable(X) and 'purr' is the target dependent variable(Y) we are aiming to predict.\n\nWe feed the 'cat' into the NN through an embedding layer initialized with random weights, and pass it through the softmax layer with ultimate aim of predicting 'purr'. The optimization method such as SGD minimize the loss function \"(target word | context words)\" which seeks to minimize the loss of predicting the target words given the context words. If we do this with enough epochs, the weights in the embedding layer would eventually represent the vocabulary of word vectors, which is the \"coordinates\" of the words in this geometric vector space.\n\n![](https://i.imgur.com/R8VLFs2.png)\n\nThe above example assumes the skip-gram model. For the Continuous bag of words(CBOW), we would basically be predicting a word given the context. \n\n**GLOVE**\n\nGLOVE works similarly as Word2Vec. While you can see above that Word2Vec is a \"predictive\" model that predicts context given word, GLOVE learns by constructing a co-occurrence matrix (words X context) that basically count how frequently a word appears in a context. Since it's going to be a gigantic matrix, we factorize this matrix to achieve a lower-dimension representation. There's a lot of details that goes in GLOVE but that's the rough idea.\n\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport time\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom tqdm import tqdm\nimport math\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Dense, Input, LSTM, Embedding, Dropout, Activation, CuDNNGRU, Conv1D\nfrom keras.layers import Bidirectional, GlobalMaxPool1D\nfrom keras.models import Model\nfrom keras import initializers, regularizers, constraints, optimizers, layers","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\")\ntest_df = pd.read_csv(\"../input/test.csv\")\nprint(\"Train shape : \",train_df.shape)\nprint(\"Test shape : \",test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ad1e9fdf97f3291a7d1797b25f0c7a0c5d1f1edd"},"cell_type":"markdown","source":"Next steps are as follows:\n * Split the training dataset into train and val sample. Cross validation is a time consuming process and so let us do simple train val split.\n * Fill up the missing values in the text column with '_na_'\n * Tokenize the text column and convert them to vector sequences\n * Pad the sequence as needed - if the number of words in the text is greater than 'max_len' trunacate them to 'max_len' or if the number of words in the text is lesser than 'max_len' add zeros for remaining values."},{"metadata":{"trusted":true,"_uuid":"ba5a1b8109dee2c9fbc628d5da4a7c3447d42fb8"},"cell_type":"code","source":"## split to train and val\ntrain_df, val_df = train_test_split(train_df, test_size=0.1, random_state=104)\n\n## some config values \nembed_size = 300 # how big is each word vector\nmax_features = 40000 # how many unique words to use (i.e num rows in embedding vector)\nmaxlen = 100 # max number of words in a question to use\n\n## fill up the missing values\ntrain_X = train_df[\"question_text\"].fillna(\"_na_\").values\nval_X = val_df[\"question_text\"].fillna(\"_na_\").values\ntest_X = test_df[\"question_text\"].fillna(\"_na_\").values\n\n## Tokenize the sentences\ntokenizer = Tokenizer(num_words=max_features)\ntokenizer.fit_on_texts(list(train_X))\ntrain_X = tokenizer.texts_to_sequences(train_X)\nval_X = tokenizer.texts_to_sequences(val_X)\ntest_X = tokenizer.texts_to_sequences(test_X)\n\n## Pad the sentences \ntrain_X = pad_sequences(train_X, maxlen=maxlen)\nval_X = pad_sequences(val_X, maxlen=maxlen)\ntest_X = pad_sequences(test_X, maxlen=maxlen)\n\n## Get the target values\ntrain_y = train_df['target'].values\nval_y = val_df['target'].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"63a7d93e0622141b71d11327de57f173a43d1412"},"cell_type":"markdown","source":"# plotting the length of question we have"},{"metadata":{"trusted":true,"_uuid":"5d0d30131f912125f7b45a864695d0e924de4103"},"cell_type":"code","source":"totalNumWords = [len(one_comment) for one_comment in train_X]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"31d66cf622034b33461e30f11067b0e8aaadc8d1"},"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.hist(totalNumWords,bins = np.arange(0,410,10))#[0,50,100,150,200,250,300,350,400])#,450,500,550,600,650,700,750,800,850,900])\nplt.xlabel(\"Distribution of comment\")\nplt.ylabel(\"no of comments\")\nplt.title(\"no of comments vs no of words distribution \")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b9d263852f653e466e24f9827548d7d1a7ee7262"},"cell_type":"markdown","source":"# Embedding Analysis Started down"},{"metadata":{"_uuid":"7c0010e518288bc7f588776c58610949140a139a"},"cell_type":"markdown","source":"We have four different types of embeddings.\n * GoogleNews-vectors-negative300 - https://code.google.com/archive/p/word2vec/\n * glove.840B.300d - https://nlp.stanford.edu/projects/glove/\n * paragram_300_sl999 - https://cogcomp.org/page/resource_view/106\n * wiki-news-300d-1M - https://fasttext.cc/docs/en/english-vectors.html\n \n A very good explanation for different types of embeddings are given in this [kernel](https://www.kaggle.com/sbongo/do-pretrained-embeddings-give-you-the-extra-edge). Please refer the same for more details..\n\n**Glove Embeddings:**\n\nIn this section, let us use the Glove embeddings and rebuild the GRU model."},{"metadata":{"_uuid":"6a892bc4969619e26f3fd0aafb1d024632eeedc6"},"cell_type":"markdown","source":"Since we are going to evaluate a few word embeddings, let's define a function so that we can run our experiment properly. I'm going to put some comments in this function below for better intuitions.\n\nNote that there are quite a few GLOVE embeddings in Kaggle datasets, and I feel that it would be more applicable to use the one that was trained based on Twitter text. Since the comments in our dataset consists of casual, user-generated short message, the semantics used might be very similar. Hence, we might be able to capture the essence and use it to produce a good accurate score.\n\nSimilarly, I have used the Word2Vec embeddings which has been trained using Google Negative News text corpus, hoping that it's negative words can work better in our \"toxic\" context."},{"metadata":{"_uuid":"09bd2d9fe2d7193b11006f358478be7232eedcb5"},"cell_type":"markdown","source":"The function would return a new embedding matrix that has the loaded weights from the pretrained embeddings for the common words we have, and randomly initialized numbers that has the same mean and standard deviation for the rest of the weights in this matrix."},{"metadata":{"_uuid":"e2f4077f9f7fd093bb7a33ce83268c619f07cccf"},"cell_type":"markdown","source":"With the embedding weights, we can proceed to build a LSTM layer. The whole architecture is pretty much the same as the previous one I have done in the earlier kernel here, except that I have turned the LSTM into a bidirectional one, and added a dropout factor to it. \n\nWe start off with defining our input layer. By indicating an empty space after comma, we are telling Keras to infer the number automatically."},{"metadata":{"_uuid":"09b49f86c814771b7493b8bd168dfddf1041e1e5"},"cell_type":"markdown","source":"Next, we pass it to a LSTM unit. But this time round, we will be using a Bidirectional LSTM instead because there are several kernels which shows a decent gain in accuracy by using Bidirectional LSTM.\n\nHow does Bidirectional LSTM work? \n\n![](https://i.imgur.com/jaKiP0S.png)\n\nImagine that the LSTM is split between 2 hidden states for each time step. As the sequence of words is being feed into the LSTM in a forward fashion, there's another reverse sequence that is feeding to the different hidden state at the same time. You might noticed later at the model summary that the output dimension of LSTM layer has doubled to 120 because 60 dimensions are used for forward, and another 60 are used for reverse.\n\nThe greatest advantage in using Bidirectional LSTM is that when it runs backwards you preserve information from the future and using the two hidden states combined, you are able in any point in time to preserve information from both past and future.\n"},{"metadata":{"_uuid":"db93324e73a32af3eb5f8a593f1bc02aaff4b363"},"cell_type":"markdown","source":"We are also introducing 2 more new mechanisms in this notebook: **LSTM Drop out and recurrent drop out.**\n\nWhy are we using dropout? You might have noticed that it's easy for LSTM to overfit, and in my previous notebook, overfitting problem starts to surface in just 2 epochs! Drop out is not something new to most of us, and these mechanisms applies the same dropout principles in a LSTM context.\n\n![](https://i.imgur.com/ksSyArD.png)\nLSTM Dropout is a probabilistic drop out layer on the inputs in each time step, as depict on the left diagram(arrows pointing upwards). On the other hand, recurrent drop out is something like a dropout mask that applies drop out between the hidden states throughout the recursion of the whole LSTM network, which is depicted on the right diagram(arrows pointing to the right). \n\nThese mechanisms could be set via the \"dropout\" and \"recurrent_dropout\" parameters respectively. Please ignore the colors in the picture."},{"metadata":{"_uuid":"28d3e0dc31e680a4d65a4406f751d7cec11c78a1"},"cell_type":"markdown","source":"# Architecture is same only dimensions are changed "},{"metadata":{"_uuid":"83edfd14e1aefe88b04150c0f91f35ec554e9d10"},"cell_type":"markdown","source":"This is the architecture of the model we are trying to build. It's always to good idea to list out the dimensions of each layer in the model to think visually and help you to debug later on.\n![](https://i.imgur.com/txJomEa.png)"},{"metadata":{"trusted":true,"_uuid":"23f130e80159bb1701e449e2e91199dbfff1f1d4"},"cell_type":"code","source":"EMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n        \ninp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)\nx = Bidirectional(LSTM(60, return_sequences=True,name='lstm_layer',dropout=0.1,recurrent_dropout=0.1))(x)\nx = GlobalMaxPool1D()(x)\nx = Dropout(0.3)(x)\nx = Dense(50, activation=\"relu\")(x)\nx = Dropout(0.3)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy',\n                  optimizer='adam',\n                  metrics=['accuracy'])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a560ab0dbab9cf6fdbdae6721ec030e300f19d78"},"cell_type":"code","source":"model.fit(train_X, train_y, batch_size=1024, epochs=2, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ff43855164472de035a5a1d80b3db4838684701a"},"cell_type":"code","source":"pred_glove_val_y = model.predict([val_X], batch_size=1024, verbose=1)\nmax_glove=[]\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    max_glove.append(metrics.f1_score(val_y, (pred_glove_val_y>thresh).astype(int)))\n    print(\"F1 score at threshold {0} is {1}\".format(thresh, metrics.f1_score(val_y, (pred_glove_val_y>thresh).astype(int))))\nmax_glove=max(max_glove)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d2a33c252f31fddcc65896053184226128562776"},"cell_type":"markdown","source":"Results seem to be better than the model without pretrained embeddings."},{"metadata":{"trusted":true,"_uuid":"d51ff8ed6a87b488fec3ac84ca50df661d7c8193"},"cell_type":"code","source":"pred_glove_test_y = model.predict([test_X], batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"39d4fedab4ac170863a0ee1ca3aa9be1ee58fe02"},"cell_type":"code","source":"del word_index, embeddings_index, all_embs, embedding_matrix, model, inp, x\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc6bab22dd12a09378f4b8b159cb7a5d88a3e7c0"},"cell_type":"markdown","source":"**Wiki News FastText Embeddings:**\n\nNow let us use the FastText embeddings trained on Wiki News corpus in place of Glove embeddings and rebuild the model."},{"metadata":{"trusted":true,"_uuid":"6f3d0fd28dd2b04eaccb732b96b872e5a223d962"},"cell_type":"code","source":"EMBEDDING_FILE = '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE) if len(o)>100)\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n        \ninp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)\nx = Bidirectional(LSTM(60, return_sequences=True,name='lstm_layer',dropout=0.1,recurrent_dropout=0.1))(x)\nx = GlobalMaxPool1D()(x)\nx = Dropout(0.3)(x)\nx = Dense(50, activation=\"relu\")(x)\nx = Dropout(0.3)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy',\n                  optimizer='adam',\n                  metrics=['accuracy'])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"47238831a4701c8a67dc7ecb130ac1402baf7bb2"},"cell_type":"code","source":"model.fit(train_X, train_y, batch_size=1024, epochs=2, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b7ab4100f723ad535528865b1edc7896bce80223"},"cell_type":"code","source":"pred_fasttext_val_y = model.predict([val_X], batch_size=1024, verbose=1)\nmax_fast_text=[]\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    max_fast_text.append(metrics.f1_score(val_y, (pred_fasttext_val_y>thresh).astype(int)))\n    print(\"F1 score at threshold {0} is {1}\".format(thresh, metrics.f1_score(val_y, (pred_fasttext_val_y>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3216362afb0f49579d287a06f13adf8cd7d8b0cf"},"cell_type":"code","source":"pred_fasttext_test_y = model.predict([test_X], batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f24f9753ff1d933fa4f75a0ba34df305632d6e93"},"cell_type":"code","source":"del word_index, embeddings_index, all_embs, embedding_matrix, model, inp, x\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4ca44ac68bf404b9c26e07fbcc9c8ac793e04510"},"cell_type":"markdown","source":"**Paragram Embeddings:**\n\nIn this section, we can use the paragram embeddings and build the model and make predictions."},{"metadata":{"trusted":true,"_uuid":"25ec1aac4aedbf431a2d30de64030ce8e3203c18"},"cell_type":"code","source":"EMBEDDING_FILE = '../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE, encoding=\"utf8\", errors='ignore') if len(o)>100)\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n        \ninp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)\nx = Bidirectional(LSTM(60, return_sequences=True,name='lstm_layer',dropout=0.1,recurrent_dropout=0.1))(x)\nx = GlobalMaxPool1D()(x)\nx = Dropout(0.3)(x)\nx = Dense(50, activation=\"relu\")(x)\nx = Dropout(0.3)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy',\n                  optimizer='adam',\n                  metrics=['accuracy'])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cc188f2787ea7b98d3a40953a95a5fc09ff2764d"},"cell_type":"code","source":"model.fit(train_X, train_y, batch_size=512, epochs=2, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9abdfd1cf15257f2c0c2181a13327796e8d4584e"},"cell_type":"code","source":"pred_paragram_val_y = model.predict([val_X], batch_size=1024, verbose=1)\nmax_paragram=[]\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    max_paragram.append(metrics.f1_score(val_y, (pred_paragram_val_y>thresh).astype(int)))\n    print(\"F1 score at threshold {0} is {1}\".format(thresh, metrics.f1_score(val_y, (pred_paragram_val_y>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"99cb9f6145da909bd7436e46d47547efc097499d"},"cell_type":"code","source":"pred_paragram_test_y = model.predict([test_X], batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"af087d21bdb4358701e31aded6b522accd5a8a64"},"cell_type":"code","source":"del word_index, embeddings_index, all_embs, embedding_matrix, model, inp, x\nimport gc; gc.collect()\ntime.sleep(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e1312b7a4c3b67ca4ebd26fb083dbac3b6635dc2"},"cell_type":"markdown","source":"**Observations:**\n * Overall pretrained embeddings seem to give better results comapred to non-pretrained model. \n * The performance of the different pretrained embeddings are almost similar.\n \n**Final Blend:**\n\nThough the results of the models with different pre-trained embeddings are similar, there is a good chance that they might capture different type of information from the data. So let us do a blend of these three models by averaging their predictions."},{"metadata":{"trusted":true,"_uuid":"449bc59fdc9a719aa0759ac51a4481df113604ca"},"cell_type":"code","source":"pred_val_y = 0.33*pred_glove_val_y + 0.33*pred_fasttext_val_y + 0.34*pred_paragram_val_y\nmax_blend=[]\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    max_blend.append(metrics.f1_score(val_y, (pred_val_y>thresh).astype(int)))\n    print(\"F1 score at threshold {0} is {1}\".format(thresh, metrics.f1_score(val_y, (pred_val_y>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4fdbeffc0f84643d2832eec49234bd9d6c6e216b"},"cell_type":"markdown","source":"The result seems to better than individual pre-trained models and so we let us create a submission file using this model blend."},{"metadata":{"trusted":true,"_uuid":"a0dd799c5b92bb4b5d11e5ca1d1be2e73073b418"},"cell_type":"code","source":"max_paragram","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"160601dc20e3c43717012ea48353175c81e0c0d0"},"cell_type":"code","source":"max_paragram=max(max_paragram)\nmax_fast_text = max(max_fast_text)\nmax_blend = max(max_blend)\nimport seaborn as sns\nsns.lineplot([\"Glove\",\"Fasttext\",\"Paragram\",\"Blend\"],y=[max_glove,max_fast_text,max_paragram,max_blend])\nplt.xlabel(\"models\")\nplt.ylabel(\"F1-Score\")\nplt.title(\"Score Board\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c90fb4a4ef1b3b2ea06563a6901deac1b38822f3"},"cell_type":"code","source":"pred_test_y = 0.33*pred_glove_test_y + 0.33*pred_fasttext_test_y + 0.34*pred_paragram_test_y\npred_test_y = (pred_test_y>0.35).astype(int)\nout_df = pd.DataFrame({\"qid\":test_df[\"qid\"].values})\nout_df['prediction'] = pred_test_y\nout_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}