{"cells":[{"metadata":{"_uuid":"e0333873d816065e12c5e4eef74258bfb72f8747"},"cell_type":"markdown","source":"# Quora notebook (Glove embedding LSTM)\n### I have converted to markdown the code that I wish to keep in the the Jupyter Notebook, but to not run when comitted (because it increases runtime). I will write \"(converted to markdown)\" above these sections of code. I did not remove them because they explain the train of thought.\nThis is a kernel with an untrained embedding baseline model and pretrained Glove embedding structured model.\n\nGeneral guidance and help from Jannes Klaas' ML workshop series and the provided materials. <br>\nF1 scoring and further inspiration thanks to SRK: https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","scrolled":true,"trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\npath = \"../input\" # Change to \"../input\" on Kaggle, or \"all\" on local\nprint(os.listdir(path))\n\n# Any results you write to the current directory are saved as output.\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom sklearn.model_selection import train_test_split\nfrom keras.models import Sequential, Model\nfrom keras.layers import Embedding, Flatten, Dense, LSTM, Bidirectional, Dropout, Activation, GlobalMaxPool1D, Input\nfrom tqdm import tqdm\nfrom sklearn import metrics","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0cb116e24332c10a143b9f83902cd3f486d70e5d"},"cell_type":"markdown","source":"# Setting up the input data and looking at the structure of it"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","scrolled":true,"trusted":true},"cell_type":"code","source":"train = pd.read_csv(path + \"/train.csv\")\ntest = pd.read_csv(path + \"/test.csv\")\nsample = pd.read_csv(path + \"/sample_submission.csv\")\n\nprint(\"training data: \" + str(train.shape))\n# print(train.head()) I have commented this code out to save space\n\nprint(\"test data: \" + str(test.shape))\n# print(test.head()) I have commented this code out to save space\n\nprint(\"sample data: \" + str(sample.shape))\n# print(sample.head()) I have commented this code out to save space","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f0bba7604e168559278a721c3a4341c28d111a5"},"cell_type":"markdown","source":"### Initially I printed a few of the insincere questions to get an idea of what we're looking for (converted to markdown)"},{"metadata":{"trusted":true,"_uuid":"e37e35e250e210a551ff752905109ea505ea4874"},"cell_type":"markdown","source":"print(train.loc[train['target'] == 1]['question_text'])"},{"metadata":{"_uuid":"e11f515a5d3d48ac8e8158486556b9f618b094f4"},"cell_type":"markdown","source":"# Preprocess with wordvectors and run it through a logistic regressor to see how well untrained embeddings work"},{"metadata":{"_uuid":"1659a6abc10eedbd025d757e16e6d7d93dbee564","scrolled":false,"trusted":true},"cell_type":"code","source":"max_words = 60000 # This has been tested in the range 50,000-100,000 and this gave a good score\n\ntokenizer_class = Tokenizer(num_words=max_words, lower=True) # Setting up the tokenizer class\ntokenizer_class.fit_on_texts(train['question_text'])\ntrain_data = tokenizer_class.texts_to_sequences(train['question_text']) # List of number sequences, each corresponding to a question\noutput_test_data = tokenizer_class.texts_to_sequences(test['question_text']) # Test sequences for later\n\ntrain_targets = train['target']\n\nword_dict = tokenizer_class.word_index # The dictionary for this tokenizer class\nnumber_of_words = min(max_words, len(word_dict)) # We work out how many words we're actually using (capped by max_words value)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"956193bf441a983435cf0a48c6eb46493a8423a6"},"cell_type":"markdown","source":"### Examine the lengths of the sequences before padding"},{"metadata":{"_uuid":"df3bbc44e930554ff56155b1542c3e18c4cbed54","scrolled":false,"trusted":true},"cell_type":"code","source":"sequence_lengths = []\nfor sequence in train_data:\n    sequence_lengths.append(len(sequence))\n\nprint(np.amax(sequence_lengths))\nprint(np.mean(sequence_lengths))\nprint(np.std(sequence_lengths))\nprint(np.sum(np.array(sequence_lengths) > 70)) # Number of sequences of length greater than 70","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6621fb7ea8bc09b97fd30ec0c226dcc3be33a172"},"cell_type":"markdown","source":"### From the above and the fact there are only 3 sequences longer than 70, we assign this value to maxlen"},{"metadata":{"trusted":true,"_uuid":"f69a64f6eb3dcb8b7b7623e000e1a7da3b263d6b"},"cell_type":"code","source":"maxlen = 70 # Maximum number of words to use in each question\ntrain_data = pad_sequences(train_data, maxlen=maxlen) # Padding the lengths of sequences\noutput_test_data = pad_sequences(output_test_data, maxlen=maxlen) # Padding the lengths of the test_sequences for later","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a0b316fc13f95a28c364db77529dbc869846d22a"},"cell_type":"markdown","source":"### Splitting the train_data into train/test data for model training"},{"metadata":{"_uuid":"e6d44260ffb458b744c9a1d756c55ed42d3e5663","trusted":true},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(train_data, train_targets,\n                                                    test_size = 0.1,\n                                                    shuffle=False,\n                                                    random_state = 42)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1a7b57dbfcb0e7bb027b739f1723fe5b01ae0cd3"},"cell_type":"markdown","source":"### Building the first model (converted to markdown)"},{"metadata":{"trusted":true,"_uuid":"0e01237ac8517d14ad85c421cab0ba044e933459"},"cell_type":"markdown","source":"untrained_embed_size = 120 # We choose embedding dimension of 120 (this was arbitrarily chosen)\n\ninput_class = Input(shape=(maxlen,))\nx = Embedding(max_words, untrained_embed_size)(input_class)\nx = Flatten()(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=input_class, outputs=x)\n\nprint(model.summary())"},{"metadata":{"_uuid":"92076345b1309f33ad1383e222e12f457a7e5c54"},"cell_type":"markdown","source":"### Training the first model (converted to markdown)"},{"metadata":{"_uuid":"de6072ac04abdacbc91a3cbdb99ba8359f5ecd39","trusted":true},"cell_type":"markdown","source":"model.compile(optimizer='adam',\n              loss='binary_crossentropy',\n              metrics=['acc'])\n\nhistory = model.fit(X_train, y_train,\n                    epochs=3,\n                    batch_size=256,\n                    validation_data=(X_test, y_test))"},{"metadata":{"_uuid":"7eb1643ee58e4d3e4355cacf34ace14dc4ca2a86"},"cell_type":"markdown","source":"### Calculating F1 score threshold (code due to SRK's Kernel)(converted to markdown)"},{"metadata":{"trusted":true,"_uuid":"4791f814cb87c970b52a952a83f61a13a8ea0388"},"cell_type":"markdown","source":"first_model_pred = model.predict([X_test], batch_size=1024, verbose=1)\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"F1 score at threshold {0} is {1}\".format(thresh, metrics.f1_score(y_test, (first_model_pred>thresh).astype(int))))"},{"metadata":{"_uuid":"bd6d39c19aa2df56066f7f9334eadb34992925d7"},"cell_type":"markdown","source":"### The val_loss is going up and the val_accuracy is going down which is a classic symptom of overfitting, so we must delete the model and try again (converted to markdown)"},{"metadata":{"_uuid":"95cbc99b5cbcce2cf59388f290d922222677b140"},"cell_type":"markdown","source":"del model"},{"metadata":{"_uuid":"2d389775ef2a29dea9125dba32dcb62e89619704"},"cell_type":"markdown","source":"# Instead try a pre-trained word embedding with more structured layers"},{"metadata":{"_uuid":"6b1ec03d3eb538d364118cdbf2c5081df17088d2"},"cell_type":"markdown","source":"### Compiling the Glove dictionary from the txt file"},{"metadata":{"_uuid":"b47204c4b47a5623bbb41b983e7569d33cea12c0","trusted":true,"scrolled":true},"cell_type":"code","source":"f = open(os.path.join('../input/embeddings/glove.840B.300d', 'glove.840B.300d.txt'))\n\nembeddings_index = {}\n\nfor line in tqdm(f): # Wrapped this iterable with a tqdm to see the progress\n    values = line.split(' ')\n    word = values[0]\n    embedding = np.asarray(values[1:], dtype='float32')\n    embeddings_index[word] = embedding\nf.close()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3d3e18673d8a669608eaef4652649e4caf5b0baf"},"cell_type":"markdown","source":"### Calculating the mean, standard deviation and dimension of these embeddings"},{"metadata":{"_uuid":"ed6cffe75ffb29644b993d2a475421c1433f8616","trusted":true},"cell_type":"code","source":"all_embs = np.stack(embeddings_index.values())\nemb_mean = all_embs.mean()\nemb_std = all_embs.std()\n\nprint(all_embs.shape) # Examine the dimensions of the vectors\nembedding_dim = all_embs.shape[1] # Asign the vector dimension to a variable","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1b6a0387c4b683dffc17e8bc6fe5a31aea2c97b0"},"cell_type":"markdown","source":"### Generate a random matrix with the same mean and std as the embeddings"},{"metadata":{"_uuid":"0fd46896a2bc7bd6c2855d71d7b004a0414e8495","trusted":true},"cell_type":"code","source":"embedding_matrix = np.random.normal(emb_mean,\n                                    emb_std,\n                                    (number_of_words, embedding_dim))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1f1237434d2e432b8de0f2ba3861a0c2b4608a46"},"cell_type":"markdown","source":"### Replace the words for which Glove provides trained embeddings"},{"metadata":{"trusted":true,"_uuid":"efe4e8e8f998f010f950405ad730e4980732f805"},"cell_type":"code","source":"for word, i in word_dict.items():\n    if i >= max_words:\n        continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None:\n        embedding_matrix[i] = embedding_vector","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6170d5f5e265b33645f498d275c56225b494a52a"},"cell_type":"markdown","source":"### Building the second model"},{"metadata":{"trusted":true,"_uuid":"c22dabaff08f68eaecfe6174f856440207ea572c"},"cell_type":"code","source":"input_class = Input(shape=(maxlen,))\nx = Embedding(max_words, embedding_dim, weights=[embedding_matrix], trainable=False)(input_class)\nx = Bidirectional(LSTM(64, return_sequences=True))(x)\nx = Bidirectional(LSTM(32, return_sequences=True))(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(16, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=input_class, outputs=x)\n\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3e0d4c11cc3b0d4bc77115350e9276b4c95d83a5"},"cell_type":"markdown","source":"### Training the second model"},{"metadata":{"trusted":true,"_uuid":"3cd1890663c39bc3d7f593899701b4c7c3134b73"},"cell_type":"code","source":"model.compile(optimizer='adam',\n              loss='binary_crossentropy',\n              metrics=['acc'])\n\nhistory = model.fit(X_train, y_train,\n                    epochs=3,\n                    batch_size=256,\n                    validation_data=(X_test, y_test))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3a4947461ee08dfd1da8418410b733a4751c3af4"},"cell_type":"markdown","source":"### No longer is the val_loss going up and the val_accuracy going down so we have eliminated overfitting. This model with pretrained embeddings has a slightly better loss and accuracy."},{"metadata":{"_uuid":"846a59cee9984252091c23981b5e2c4be7c72ecc"},"cell_type":"markdown","source":"### Calculating F1 score threshold (code due to SRK's Kernel)"},{"metadata":{"trusted":true,"_uuid":"5d5fe629f5e633484c2f8c83c74c7fe26b9802e1"},"cell_type":"code","source":"pretrained_model_pred = model.predict([X_test], batch_size=1024, verbose=1)\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"F1 score at threshold {0} is {1}\".format(thresh, metrics.f1_score(y_test, (pretrained_model_pred>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4f3c922b0dc9c3eddee93cc9040d453aa6ce3d42"},"cell_type":"markdown","source":"# Then we finally output the test predictions using the second model"},{"metadata":{"trusted":true,"_uuid":"a987bcca5b04a6c51fab336ea3400e5e1f87991b"},"cell_type":"code","source":"test_predictions = model.predict([output_test_data], batch_size=256, verbose=1)\ntest_predictions = (test_predictions>0.44).astype(int) # Using the best threshold from above\noutput = pd.DataFrame({\"qid\":test[\"qid\"].values})\noutput['prediction'] = test_predictions\noutput.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}