{"cells":[{"metadata":{"_uuid":"dc89df0d458f53eee3783e31aa2a623e72957869"},"cell_type":"markdown","source":"# Quora preprocessing and BLSTM + Attention model\n\n1. Preprocessing  \n1.1. Build vocabulary   \n1.2. Load embeddings  \n1.3. Check coverage  \n1.4. Lowerization  \n1.5. Contractions  \n1.6. Punctuation removal  \n1.7. Split train val sets  \n1.8. Fill N/A values  \n1.9. Tokenize sentences  \n1.10. Pad sentences  \n1.11. Get target value  \n1.12. Shuffle data  \n2. Model  \n2.1. Attention layer  \n2.2. Define F1-score  \n2.3. Define model: bLSTM + Attention  \n2.4. Define training function  \n2.5. Blend embeddings  \n2.6. Create model and train   \n2.7. Find best threshold for F1-score  \n2.8. Predict test set and save submission  \n3. References  \n"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import os\nimport re\nimport string\n\nimport numpy as np \nimport pandas as pd\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\nimport nltk\nfrom nltk.corpus import stopwords","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-output":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")\n\nprint(\"Train shape : \",train.shape)\nprint(\"Test shape : \",test.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"49b43168bacd6b9b939b0a89301b59720ab3c536"},"cell_type":"markdown","source":"# 1. Preprocessing\n\n### How to approach the preprocessing stage? [Dieter's approach](https://www.kaggle.com/christofhenkel):\n\n> 1. **Don't use standard preprocessing steps like stemming or stopword removal when you have pre-trained embeddings**\n> \n> Some of you might used standard preprocessing steps when doing word count based feature extraction (e.g. TFIDF) such as removing stopwords, stemming etc. The reason is simple: You lose valuable information, which would help your NN to figure things out.\n> \n> 2. **Get your vocabulary as close to the embeddings as possible**"},{"metadata":{"trusted":true,"_uuid":"350cfcb5ad270677117f8829cb9c8f5fb2f6bce5"},"cell_type":"code","source":"EMBED_SIZE = 300 # size of word vector\nMAX_FEATURES = 100000 # how many unique words to use (i.e num rows in embedding vector)\nMAXLEN = 40 # max length of question","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"59adafa6a5d7a5ed93ec591e2f9e57656dd08a25"},"cell_type":"markdown","source":"## 1.1. Build vocabulary"},{"metadata":{"_uuid":"09061d8d79c37d43eeb159383be53b9568738809","trusted":true},"cell_type":"code","source":"# tracks the training vocabulary, which goes through all our text and counts the occurrence of the contained words.\ndef build_vocab(texts):\n    \"\"\"\n    input: list of list of words\n    output: dictionary of words and their count\n    \"\"\"\n    sentences = texts.apply(lambda x: x.split()).values\n    vocab = {}\n    for sentence in sentences:\n        for word in sentence:\n            try:\n                vocab[word] += 1\n            except KeyError:\n                vocab[word] = 1\n    return vocab","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bacf2c61a1f03e236c5a73dfb4e214886d2b3c0b","trusted":true},"cell_type":"code","source":"# populate the vocabulary\ndf = pd.concat([train ,test], sort=False)\nvocab = build_vocab(df['question_text'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"693bb6a700f80925ea9c9fd55f79a2394dd628d3"},"cell_type":"code","source":"# display the first 5 elements and their count\ni = 0\nfor k in list(vocab):\n    print(k, vocab[k])\n    i += 1\n    if i > 5:\n        break","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e80b6eb40860fcd98448ce867baac043960103da"},"cell_type":"markdown","source":"   ## 1.2. Load embedding index and embedding matrix\n   \n   * Embedding index: a dictionary where the keys are embeddings and the values are arrays of their embedding representation\n   * Embedding matrix: a matrix representation of the embeddings\n   \n Due to the big size of the embeddings (5.6GB), the necessary RAM is around 10.6GB."},{"metadata":{"_uuid":"3a2812853014e6355bf64210a391c343fffaaccf","trusted":true},"cell_type":"code","source":"def load_embed(file):\n    def get_coefs(word,*arr): \n        return word, np.asarray(arr, dtype='float32')\n    \n    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(file, encoding='latin'))\n    \n    return embeddings_index","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"db8c569ec68c20f47aa42965aa1ac16f558853a8","trusted":true},"cell_type":"code","source":"glove = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\n\nprint(\"Extracting GloVe embedding\")\nembed_glove = load_embed(glove)\nprint(\"Embedding extraction done\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5c5063caee19f6bdc0245ae4740e6a27d7e86e43"},"cell_type":"code","source":"len(embed_glove)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c80dfcd2819b5e9797c7214dfd7f3a1d26f9d7ce"},"cell_type":"code","source":"def load_glove_matrix(word_index, embeddings_index):\n\n    all_embs = np.stack(embeddings_index.values())\n    emb_mean, emb_std = all_embs.mean(), all_embs.std()\n    EMBED_SIZE = all_embs.shape[1]\n    \n    nb_words = min(MAX_FEATURES, len(word_index))\n    embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, EMBED_SIZE))\n\n    for word, i in word_index.items():\n        if i >= MAX_FEATURES:\n            continue\n        embedding_vector = embeddings_index.get(word)\n        if embedding_vector is not None:\n            embedding_matrix[i] = embedding_vector\n\n    return embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aaf846fa0e5f1a47e8388b27f2c47bf58703b652"},"cell_type":"markdown","source":"## 1.3. Check coverage\n\n* This function checks the intersection between our vocabulary and the embeddings\n* The output is a list of out of vocabulary (oov) words"},{"metadata":{"_uuid":"16c92507ec3ed05998b7e3e2218f6dc113d13bd9","trusted":true},"cell_type":"code","source":"import operator\n\ndef check_coverage(vocab, embeddings_index):\n    known_words = {}\n    unknown_words = {}\n    nb_known_words = 0\n    nb_unknown_words = 0\n    for word in vocab.keys():\n        try:\n            known_words[word] = embeddings_index[word]\n            nb_known_words += vocab[word]\n        except:\n            unknown_words[word] = vocab[word]\n            nb_unknown_words += vocab[word]\n            pass\n\n    print('Found embeddings for {:.3%} of vocabulary'.format(len(known_words)/len(vocab)))\n    print('Found embeddings for {:.3%} of all text'.format(nb_known_words/(nb_known_words + nb_unknown_words)))\n    unknown_words = sorted(unknown_words.items(), key=operator.itemgetter(1))[::-1]\n\n    return unknown_words","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"32fb6aa49c305f405b1f6461a7a467541671ce67","trusted":true},"cell_type":"code","source":"print(\"Glove: \")\noov_glove = check_coverage(vocab, embed_glove)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"757d05bae4e3b997dbc358059a721ef60e4db2f8"},"cell_type":"markdown","source":"## Remarks\n\n* 12% of our text is more or less useless, since it won't be detected by the embedding\n* To improve this, we can check which words are out of vocabulary (oov)"},{"metadata":{"_uuid":"2f536fb0710beaf6e83396c2890a225cc3d6965f","trusted":true},"cell_type":"code","source":"oov_glove[:20]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa9015a35cf7316b85e94aa5b3eb3cac69e94ee6"},"cell_type":"markdown","source":"## 1.4. Lowerization\n\n* The top oov words include punctuations(question mark) and contractions\n* First we focus on questions with capital letters"},{"metadata":{"_uuid":"739c41f0577dd9c7f668e1884c6556bbf431d15e","trusted":true},"cell_type":"code","source":"def add_lower(embedding, vocab):\n    count = 0\n    for word in vocab:\n        if word in embedding and word.lower() not in embedding:  \n            embedding[word.lower()] = embedding[word]\n            count += 1\n    print(f\"Added {count} words to embedding\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c7d21885e1de594d4b3d1932a0bca67b5188e8d2","trusted":true},"cell_type":"code","source":"train['question_text'] = train['question_text'].apply(lambda x: x.lower())\ntest['question_text'] = test['question_text'].apply(lambda x: x.lower())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0a6fca573a785c9f6c97019acf11a4f574bca4c6","trusted":true},"cell_type":"code","source":"print(\"Glove: \")\noov_glove = check_coverage(vocab, embed_glove)\nadd_lower(embed_glove, vocab) # operates on the same vocabulary\noov_glove = check_coverage(vocab, embed_glove)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7be49bd5ae936c9673b85b04dbf27d8d91205a9e"},"cell_type":"markdown","source":"## Remarks\n\n* There's barely any change, but the first word of every question has been treated/lowerized."},{"metadata":{"_uuid":"6186e7961ac307e2843ab249141831f90f24afc1","trusted":true},"cell_type":"code","source":"oov_glove[:10]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"532df671a7572c14e77891ad779b6ee365aeb131"},"cell_type":"markdown","source":"## 1.5. Contractions\n\n* One option would be to remove/modify punctuations, such as: \"John's\" --> \" Johns\" or \"John ' s\"\n* But if the word is a contraction, such as \"haven't\", and if we treat the punctuation first, the word now is \"haven ' t\" or \"havent\", and that is no longer a contraction\n* Most oov words either are:  \n    a) Contractions  \n    b) Words with adjacent punctuations\n\nReference: https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/77758"},{"metadata":{"_uuid":"2a288f89230620c46b061f30ee5b0f02a9e759aa","trusted":true},"cell_type":"code","source":"contraction_mapping = {\"ain't\": \"is not\", \"aren't\": \"are not\",\"can't\": \"cannot\", \"'cause\": \"because\", \"could've\": \"could have\", \"couldn't\": \"could not\", \"didn't\": \"did not\",  \"doesn't\": \"does not\", \"don't\": \"do not\", \"hadn't\": \"had not\", \"hasn't\": \"has not\", \"haven't\": \"have not\", \"he'd\": \"he would\",\"he'll\": \"he will\", \"he's\": \"he is\", \"how'd\": \"how did\", \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \"how's\": \"how is\",  \"I'd\": \"I would\", \"I'd've\": \"I would have\", \"I'll\": \"I will\", \"I'll've\": \"I will have\",\"I'm\": \"I am\", \"I've\": \"I have\", \"i'd\": \"i would\", \"i'd've\": \"i would have\", \"i'll\": \"i will\",  \"i'll've\": \"i will have\",\"i'm\": \"i am\", \"i've\": \"i have\", \"isn't\": \"is not\", \"it'd\": \"it would\", \"it'd've\": \"it would have\", \"it'll\": \"it will\", \"it'll've\": \"it will have\",\"it's\": \"it is\", \"let's\": \"let us\", \"ma'am\": \"madam\", \"mayn't\": \"may not\", \"might've\": \"might have\",\"mightn't\": \"might not\",\"mightn't've\": \"might not have\", \"must've\": \"must have\", \"mustn't\": \"must not\", \"mustn't've\": \"must not have\", \"needn't\": \"need not\", \"needn't've\": \"need not have\",\"o'clock\": \"of the clock\", \"oughtn't\": \"ought not\", \"oughtn't've\": \"ought not have\", \"shan't\": \"shall not\", \"sha'n't\": \"shall not\", \"shan't've\": \"shall not have\", \"she'd\": \"she would\", \"she'd've\": \"she would have\", \"she'll\": \"she will\", \"she'll've\": \"she will have\", \"she's\": \"she is\", \"should've\": \"should have\", \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\", \"so've\": \"so have\",\"so's\": \"so as\", \"this's\": \"this is\",\"that'd\": \"that would\", \"that'd've\": \"that would have\", \"that's\": \"that is\", \"there'd\": \"there would\", \"there'd've\": \"there would have\", \"there's\": \"there is\", \"here's\": \"here is\",\"they'd\": \"they would\", \"they'd've\": \"they would have\", \"they'll\": \"they will\", \"they'll've\": \"they will have\", \"they're\": \"they are\", \"they've\": \"they have\", \"to've\": \"to have\", \"wasn't\": \"was not\", \"we'd\": \"we would\", \"we'd've\": \"we would have\", \"we'll\": \"we will\", \"we'll've\": \"we will have\", \"we're\": \"we are\", \"we've\": \"we have\", \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\", \"what're\": \"what are\",  \"what's\": \"what is\", \"what've\": \"what have\", \"when's\": \"when is\", \"when've\": \"when have\", \"where'd\": \"where did\", \"where's\": \"where is\", \"where've\": \"where have\", \"who'll\": \"who will\", \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\", \"why's\": \"why is\", \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\", \"won't've\": \"will not have\", \"would've\": \"would have\", \"wouldn't\": \"would not\", \"wouldn't've\": \"would not have\", \"y'all\": \"you all\", \"y'all'd\": \"you all would\",\"y'all'd've\": \"you all would have\",\"y'all're\": \"you all are\",\"y'all've\": \"you all have\",\"you'd\": \"you would\", \"you'd've\": \"you would have\", \"you'll\": \"you will\", \"you'll've\": \"you will have\", \"you're\": \"you are\", \"you've\": \"you have\", 'colour': 'color', 'centre': 'center', 'favourite': 'favorite', 'travelling': 'traveling', 'counselling': 'counseling', 'theatre': 'theater', 'cancelled': 'canceled', 'labour': 'labor', 'organisation': 'organization', 'wwii': 'world war 2', 'citicise': 'criticize', 'youtu ': 'youtube ', 'Qoura': 'Quora', 'sallary': 'salary', 'Whta': 'What', 'narcisist': 'narcissist', 'howdo': 'how do', 'whatare': 'what are', 'howcan': 'how can', 'howmuch': 'how much', 'howmany': 'how many', 'whydo': 'why do', 'doI': 'do I', 'theBest': 'the best', 'howdoes': 'how does', 'mastrubation': 'masturbation', 'mastrubate': 'masturbate', \"mastrubating\": 'masturbating', 'pennis': 'penis', 'Etherium': 'Ethereum', 'narcissit': 'narcissist', 'bigdata': 'big data', '2k17': '2017', '2k18': '2018', 'qouta': 'quota', 'exboyfriend': 'ex boyfriend', 'airhostess': 'air hostess', \"whst\": 'what', 'watsapp': 'whatsapp', 'demonitisation': 'demonetization', 'demonitization': 'demonetization', 'demonetisation': 'demonetization'}\nlen(contraction_mapping)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"76bc0e5c4205556396549b99a62152aebd8d1821","trusted":true},"cell_type":"code","source":"def known_contractions(embed):\n    known = []\n    for contract in contraction_mapping:\n        if contract in embed:\n            known.append(contract)\n    return known","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"caa9e7c83ab17fcd1b9509ac03eb0a00fe368f00","trusted":true},"cell_type":"code","source":"print(\"Known contractions Glove:\")\nprint(known_contractions(embed_glove))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d1817ace4c6a3bba4c14273d6c76c6abcfeb315c"},"cell_type":"markdown","source":"## Remark\n\nGloVe recognizes these 19 contractions, the other contractions aren't recognized and we lose information"},{"metadata":{"_uuid":"3cea9566f77e7d5b977b85d5bdeb8fffee15d555","trusted":true},"cell_type":"code","source":"def clean_contractions(text, mapping):\n    specials = [\"’\", \"‘\", \"´\", \"`\"]\n    for s in specials:\n        text = text.replace(s, \"'\")\n    text = ' '.join([mapping[t] if t in mapping else t for t in text.split(\" \")])\n    return text","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5bd114d40be132769c5341d903234ab7b0ac7ff0","trusted":true},"cell_type":"code","source":"train['question_text'] = train['question_text'].apply(lambda x: clean_contractions(x, contraction_mapping))\ntest['question_text'] = test['question_text'].apply(lambda x: clean_contractions(x, contraction_mapping))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"28782b734a20f377c2b280bda51da3ed5bfa79d8","trusted":true},"cell_type":"code","source":"df = pd.concat([train ,test], sort=False)\nvocab = build_vocab(df['question_text']) # create vocabulary with new treated questions\nprint(\"Glove: \")\noov_glove = check_coverage(vocab, embed_glove)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"80f21cfb0893960ebb100066d77ac11d54b19217","trusted":true},"cell_type":"code","source":"oov_glove[:10]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"238d4ed445f46dc4bb0ac4498026e70480bfeb7f"},"cell_type":"markdown","source":"## Remarks\n\nVery small improvement, it seems like the biggest \"problem\" are punctuations"},{"metadata":{"_uuid":"9861ad959ab03ed578ab7f4601e3d556a7b49407"},"cell_type":"markdown","source":"## 1.6. Punctuation removal"},{"metadata":{"_uuid":"7d885566c01a062ca5d5bc6b48db1e85629cdc00","trusted":true},"cell_type":"code","source":"punct_mapping = \"/-'?!.,#$%\\'()*+-/:;<=>@[\\\\]^_`{|}~\" + '\"\"“”’' + '∞θ÷α•à−β∅³π‘₹´°£€\\×™√²—–&'\npunct_mapping += '©^®` <→°€™› ♥←ק″′Â█½à…“★”–●â►−¢²¬░¶↑±¿▾═¦║―¥▓—‹─▒：¼⊕▼▪†■’▀¨▄♫☆é¯♦¤▲踾Ã⋅‘∞∙）↓、│（»，♪╩╚³・╦╣╔╗▬❤ïع≤‡√'\n\ndef unknown_punct(embed, punct):\n    unknown = ''\n    for p in punct:\n        if p not in embed:\n            unknown += p\n            unknown += ' '\n    return unknown","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"753116744911a618a26d6d40c8414d2f80e35ff5","trusted":true},"cell_type":"code","source":"print(\"Glove:\")\nprint(unknown_punct(embed_glove, punct_mapping))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6164f42ae447ffa143fcb5f1d4a274975a8c49ab"},"cell_type":"markdown","source":"GloVe only recognizes these punctuations. Any word containing other punctuations is considered as unknown."},{"metadata":{"_uuid":"a4e56c2af38825e6fb06314ce22c63f8f7302b24","trusted":true},"cell_type":"code","source":"puncts = {\"‘\": \"'\", \"´\": \"'\", \"°\": \"\", \"€\": \"e\", \"—\": \"-\", \"–\": \"-\", \"’\": \"'\", \"_\": \"-\", \"`\": \"'\", '“': '\"', '”': '\"', '“': '\"', \"£\": \"e\", '∞': 'infinity', 'θ': 'theta', '÷': '/', 'α': 'alpha', '•': '.', 'à': 'a', '−': '-', 'β': 'beta', '∅': '', '³': '3', 'π': 'pi', '…': ' '}\n\ndef clean_special_chars(text, punct, mapping):\n    for p in mapping:\n        text = text.replace(p, mapping[p])\n    \n    for p in punct:\n        text = text.replace(p, f' {p} ')\n    \n    return text","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8cbf6bd19a1233519fd30c97f027d5a27297a078","trusted":true},"cell_type":"code","source":"train['question_text'] = train['question_text'].apply(lambda x: clean_special_chars(x, punct_mapping, puncts))\ntest['question_text'] = test['question_text'].apply(lambda x: clean_special_chars(x, punct_mapping, puncts))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a357f8ed7a72122aa35afa4884e605b1e74b7628","trusted":true},"cell_type":"code","source":"df = pd.concat([train ,test], sort=False)\nvocab = build_vocab(df['question_text'])\nprint(\"Glove: \")\noov_glove = check_coverage(vocab, embed_glove)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b8202066be702c7e2fd0e3961e86193969a24abc"},"cell_type":"markdown","source":"## Remarks\n\nBig improvement now. 68% of vocabulary is covered, but that 68% of vocabulary amounts to 99.5% of all text\n\nA **possible improvement** to this could to treat misspelled words or strange/mathematical/latex symbols."},{"metadata":{"_uuid":"7a489d2ae316d6f5f8a38565e52dcde8c9423057","trusted":true},"cell_type":"code","source":"oov_glove[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3d5692856019d7eab55db32ed1649b1533e3b160"},"cell_type":"code","source":"train.head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a26de07f50c2839906ecf46b6aed4f58ac5e7d99"},"cell_type":"markdown","source":"## 1.7 Split train val\n\nSplit trainset into trainset and valset in order to check the training performance before submitting."},{"metadata":{"trusted":true,"_uuid":"a6f002669b6f234ec65e9c038b293cf7137f12be"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ntrain, val = train_test_split(train, test_size=0.2, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f86d74aa0872668974fa4ba5ecf048ba6bc904ef"},"cell_type":"markdown","source":"## 1.8. Fill N/A values\nThe questions are already clean, but just in case"},{"metadata":{"trusted":true,"_uuid":"d439ed8af549b9f372b7ce0a6f562aeb5a31be21"},"cell_type":"code","source":"# Fill up the missing values\nxtrain = train['question_text'].fillna('_na_').values\nxval = val['question_text'].fillna('_na_').values\nxtest = test['question_text'].fillna('_na_').values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d6582458d2350501dd6119f5278337086305675f"},"cell_type":"markdown","source":"## 1.9. Tokenize sentences"},{"metadata":{"trusted":true,"_uuid":"94f98f30c5c71818b8b52be70264ab40bb01f8a2"},"cell_type":"code","source":"from keras.preprocessing.text import Tokenizer\n\ntokenizer = Tokenizer(num_words=MAX_FEATURES)\ntokenizer.fit_on_texts(list(xtrain))\n\nxtrain = tokenizer.texts_to_sequences(xtrain)\nxval = tokenizer.texts_to_sequences(xval)\nxtest = tokenizer.texts_to_sequences(xtest)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4ce715676da6951db989338cc88994f5f2de90c3"},"cell_type":"markdown","source":"## 1.10. Pad sentences\nMost questions are shorter than MAXLEN (=70), but in case any is longer, not much information will be lost when padding"},{"metadata":{"trusted":true,"_uuid":"158f8f1570458833a4f026761874a6708c231f94"},"cell_type":"code","source":"from keras.preprocessing.sequence import pad_sequences\n\nxtrain = pad_sequences(xtrain, maxlen=MAXLEN)\nxval = pad_sequences(xval, maxlen=MAXLEN)\nxtest = pad_sequences(xtest, maxlen=MAXLEN)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba803aac53fab1e905f916bb40c18d4a24f446ee"},"cell_type":"markdown","source":"## 1.11. Get target value"},{"metadata":{"trusted":true,"_uuid":"2516d60c4f124a11b81439da5ae179e23e882890"},"cell_type":"code","source":"ytrain = train['target'].values\nyval = val['target'].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b608645872ec55eab17ca3d5959ec66a00b8574"},"cell_type":"markdown","source":"## 1.12. Shuffle data"},{"metadata":{"trusted":true,"_uuid":"a48a6d987dd13e319d6215fe2f1a53a0a498e4cf"},"cell_type":"code","source":"np.random.seed(42)\n\ntrn_idx = np.random.permutation(len(xtrain))\nval_idx = np.random.permutation(len(xval))\n\nxtrain = xtrain[trn_idx]\nytrain = ytrain[trn_idx]\nxval = xval[val_idx]\nyval = yval[val_idx]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"47fcfb32fe4a5d18ed73f128b506a70fe00bc270"},"cell_type":"code","source":"print(\"Extracting GloVe embedding matrix\")\nembedding_matrix_glove = load_glove_matrix(tokenizer.word_index, embed_glove)\nprint(\"Embedding matrix extraction done\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e29fa9c308fe156679e0bb154c3f199c3179d2bf"},"cell_type":"markdown","source":"# 2. Model"},{"metadata":{"trusted":true,"_uuid":"1e69e0f10c1e9745b9972620ddacce174129fe67"},"cell_type":"code","source":"from sklearn import metrics\nfrom sklearn.linear_model import LinearRegression\n\nfrom keras.layers import Dense, Input, CuDNNLSTM, Embedding, Dropout, Activation, CuDNNGRU, Conv1D\nfrom keras.layers import Bidirectional, GlobalMaxPool1D, GlobalMaxPooling1D, GlobalAveragePooling1D\nfrom keras.layers import Input, Embedding, Dense, Conv2D, MaxPool2D, concatenate, Lambda\nfrom keras.layers import Reshape, Flatten, Concatenate, Dropout, SpatialDropout1D\nfrom keras.optimizers import Adam\nfrom keras.models import Model\nfrom keras import backend as K\nfrom keras.engine.topology import Layer\nfrom keras import initializers, regularizers, constraints, optimizers, layers\nfrom keras.layers import concatenate","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1f7d7aea215e6dec67b23b04cac696d5845f8572","trusted":true},"cell_type":"code","source":"train.head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2f10c4e2207ee25768b2a79e888601438c40818f"},"cell_type":"markdown","source":"## 2.1. Attention layer\n\n* Without attention, from sentence length 20 to higher, performance is lower. \n* With attention, the performance in longer sentences is equal or even better. attention uses all hidden states. values, how much attention we want to pay to this state, how much focus. with all these values we create a context vector/matrix.\n* attention in lstm: uses current input and previous context encoding\n* self-attention in lstm: uses all other inputs. query, keys and values all come from the same place"},{"metadata":{"trusted":true,"_uuid":"d80a66fd70b61dd7c8394bcc05a2172d736fc353"},"cell_type":"code","source":"class Attention(Layer):\n    def __init__(self, step_dim,\n                 W_regularizer=None, b_regularizer=None,\n                 W_constraint=None, b_constraint=None,\n                 bias=True, **kwargs):\n        self.supports_masking = True\n        self.init = initializers.get('glorot_uniform')\n\n        self.W_regularizer = regularizers.get(W_regularizer)\n        self.b_regularizer = regularizers.get(b_regularizer)\n\n        self.W_constraint = constraints.get(W_constraint)\n        self.b_constraint = constraints.get(b_constraint)\n\n        self.bias = bias\n        self.step_dim = step_dim\n        self.features_dim = 0\n        super(Attention, self).__init__(**kwargs)\n\n    def build(self, input_shape):\n        assert len(input_shape) == 3\n\n        self.W = self.add_weight((input_shape[-1],),\n                                 initializer=self.init,\n                                 name='{}_W'.format(self.name),\n                                 regularizer=self.W_regularizer,\n                                 constraint=self.W_constraint)\n        self.features_dim = input_shape[-1]\n\n        if self.bias:\n            self.b = self.add_weight((input_shape[1],),\n                                     initializer='zero',\n                                     name='{}_b'.format(self.name),\n                                     regularizer=self.b_regularizer,\n                                     constraint=self.b_constraint)\n        else:\n            self.b = None\n\n        self.built = True\n\n    def compute_mask(self, input, input_mask=None):\n        return None\n\n    def call(self, x, mask=None):\n        features_dim = self.features_dim\n        step_dim = self.step_dim\n\n        eij = K.reshape(K.dot(K.reshape(x, (-1, features_dim)),\n                        K.reshape(self.W, (features_dim, 1))), (-1, step_dim))\n\n        if self.bias:\n            eij += self.b\n\n        eij = K.tanh(eij)\n        a = K.exp(eij)\n\n        if mask is not None:\n            a *= K.cast(mask, K.floatx())\n\n        a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n\n        a = K.expand_dims(a)\n        weighted_input = x * a\n        return K.sum(weighted_input, axis=1)\n\n    def compute_output_shape(self, input_shape):\n        return input_shape[0], self.features_dim\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68d40a78be75322b65397cb7c95822adaa577f93"},"cell_type":"markdown","source":"## 2.2 Define F1-score"},{"metadata":{"trusted":true,"_uuid":"2b2eee063ee56e2f15c1ec4c757bf8c0d9d88c09"},"cell_type":"code","source":"def f1(y_true, y_pred):\n\n    def recall(y_true, y_pred):\n        \n        true_positives = K.sum(K.round(K.clip(y_true*y_pred, 0, 1)))\n        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n        recall = true_positives/(possible_positives + K.epsilon())\n        return recall\n\n    def precision(y_true, y_pred):\n        \n        true_positives = K.sum(K.round(K.clip(y_true*y_pred, 0, 1)))\n        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n        precision = true_positives/(predicted_positives + K.epsilon())\n        return precision\n\n    precision = precision(y_true, y_pred)\n    recall = recall(y_true, y_pred)\n\n    return 2*((precision*recall)/(precision+recall+K.epsilon()))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f56cc03228f17f6f54f23ceac6baa589d3fe0b9"},"cell_type":"markdown","source":"## 2.3. Define model: LSTM with Attention"},{"metadata":{"trusted":true,"_uuid":"001b89c69648eb291bb72461793956acfa92b497"},"cell_type":"code","source":"def model_lstm_att(embedding_matrix):\n    \n    inp = Input(shape=(MAXLEN,))\n    x = Embedding(MAX_FEATURES, EMBED_SIZE, weights=[embedding_matrix], trainable=False)(inp)\n    x = Bidirectional(CuDNNLSTM(64, return_sequences=True))(x)\n    x = Bidirectional(CuDNNLSTM(32, return_sequences=True))(x)\n    \n    att = Attention(MAXLEN)(x)\n    \n    y = Dense(32, activation='relu')(att)\n    y = Dropout(0.1)(y)\n    outp = Dense(1, activation='sigmoid')(y)    \n\n    model = Model(inputs=inp, outputs=outp)\n    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[f1])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"70d9ab0e9ee88e788156bd59ad6dd348b7e479b2"},"cell_type":"markdown","source":"## 2.4. Define training function"},{"metadata":{"trusted":true,"_uuid":"92f9f73024e6f9b9f880b9c3161905a9eac895b2"},"cell_type":"code","source":"def train_pred(model, epochs=2):\n    \n    for e in range(epochs):\n        model.fit(xtrain, ytrain, batch_size=512, epochs=3, validation_data=(xval, yval))\n        pred_val_y = model.predict([xval], batch_size=1024, verbose=0)\n\n        best_thresh = 0.5\n        best_score = 0.0\n        for thresh in np.arange(0.1, 0.501, 0.01):\n            thresh = np.round(thresh, 2)\n            score = metrics.f1_score(yval, (pred_val_y > thresh).astype(int))\n            if score > best_score:\n                best_thresh = thresh\n                best_score = score\n\n        print(\"Val F1 Score: {:.4f}\".format(best_score))\n\n    pred_test_y = model.predict([xtest], batch_size=1024, verbose=0)\n\n    return pred_val_y, pred_test_y, best_score","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f9e087d9698f20c9e7d397d20d0719a1b16060b"},"cell_type":"markdown","source":"## 2.5. Blend embeddings"},{"metadata":{"trusted":true,"_uuid":"d1e68227b4fd965c0e59c9cb658ac187dbc1aae1"},"cell_type":"code","source":"paragram = '../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt'\nembedding_matrix_para = load_glove_matrix(tokenizer.word_index, load_embed(paragram))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9ac1a996cd777324cdab95b2581e74360d4e679d"},"cell_type":"code","source":"embedding_matrix = np.mean([embedding_matrix_glove, embedding_matrix_para], axis=0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f2f3bae53bd62614b77d0369bb5b48b02afdbb8c"},"cell_type":"markdown","source":"## 2.6. Create model and train"},{"metadata":{"trusted":true,"_uuid":"b60a4142e76d15b50fcfe2c10840dfb7d41bc170"},"cell_type":"code","source":"model_lstm = model_lstm_att(embedding_matrix)\nmodel_lstm.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7a80c449238791ee2d81ddf14b431837c989f5b5"},"cell_type":"code","source":"outputs = []\npred_val_y, pred_test_y, best_score = train_pred(model_lstm, epochs=3)\noutputs.append([pred_val_y, pred_test_y, best_score, 'model_lstm_att only Glove'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fe2a8239dfa7f2797b4d0294f24b73fb929fff0d"},"cell_type":"markdown","source":"## 2.7. Find best threshold"},{"metadata":{"trusted":true,"_uuid":"f4de5c8ccd8d1af6e7d704864445e1c83b11cb62"},"cell_type":"code","source":"outputs.sort(key=lambda x: x[2]) \nweights = [i for i in range(1, len(outputs) + 1)]\nweights = [float(i) / sum(weights) for i in weights] \n\npred_val_y = np.mean([outputs[i][0] for i in range(len(outputs))], axis = 0)\n\nthresholds = []\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    res = metrics.f1_score(yval, (pred_val_y > thresh).astype(int))\n    thresholds.append([thresh, res])\n    print(\"F1 score at threshold {0} is {1}\".format(thresh, res))\n    \nthresholds.sort(key=lambda x: x[1], reverse=True)\nbest_thresh = thresholds[0][0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"73e36a78767d3d672ab38040c035051f79313190"},"cell_type":"code","source":"print(\"Best threshold:\", best_thresh, \"and F1 score\", thresholds[0][1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b19d8c4efcef21044366f8feeb963575e6e1abfa"},"cell_type":"markdown","source":"## 2.8. Predict test set and save sumbission"},{"metadata":{"trusted":true,"_uuid":"1976c49983e15087e4f737ec697b26fb2723b3c0"},"cell_type":"code","source":"pred_test_y = np.mean([outputs[i][1] for i in range(len(outputs))], axis = 0)\npred_test_y = (pred_test_y > best_thresh).astype(int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"459e73bfc43f916cb688271f5b3a8fdc9c9494c2"},"cell_type":"code","source":"sub = pd.read_csv('../input/sample_submission.csv')\nout_df = pd.DataFrame({\"qid\":sub[\"qid\"].values})\nout_df['prediction'] = pred_test_y\nout_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba7e84f79ed40d1f0f58f83f2165d7cf06d69065"},"cell_type":"markdown","source":"## References\n\nReferences for preprocessing:  \n* [Preprocess when using embeddings](https://www.kaggle.com/christofhenkel/how-to-preprocessing-when-using-embeddings)\n* [Preprocessing](https://www.kaggle.com/theoviel/improve-your-score-with-some-text-preprocessing)\n* [Preprocessing v2](https://www.kaggle.com/theoviel/improve-your-score-with-text-preprocessing-v2)\n\nReferences for the model:  \n* [Model LSTM + Attention](https://www.kaggle.com/kiraplenkin/model-lstm-attention/notebook)\n* [blend LSTM + CNN (concatenates all 4 embeddings)](https://www.kaggle.com/shujian/blend-of-lstm-and-cnn-with-4-embeddings-1200d)\n* [keras LSTM + Attention + embedding blending](https://www.kaggle.com/seynog/fork-attention-with-changed-weights)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}