{"cells":[{"metadata":{"_uuid":"d929dee7aa5b533cc1480f1a11e5e0722da25b32"},"cell_type":"markdown","source":"Our solution utilizes some tricks that were public and some that we kept secret. Please don't use these in the future. You guys are making it too hard. "},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"seed_nb=16\nimport numpy as np \nnp.random.seed(seed_nb)\nimport tensorflow as tf\ntf.set_random_seed(seed_nb)\n\n%matplotlib inline\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\n#os.environ['CUDA_VISIBLE_DEVICES']='0'\nimport string\nimport re\n#from sklearn.model_selection import train_test_split\n\n#from sklearn.utils.class_weight import compute_class_weight\nfrom gensim.models import KeyedVectors\nimport matplotlib.pyplot as plt\n#from sklearn.metrics import f1_score as off1\n#from sklearn.metrics import log_loss\nimport gc\nfrom tqdm import tqdm\nfrom tqdm import tqdm_notebook\nimport time\nimport h5py\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\ntqdm.pandas()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"INPUT_PATH = '../input/'\nprint(os.listdir(INPUT_PATH))\ndebug = False\ndo_submit = True\nsimulate_test = False\nopt = \"val_loss\"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"620fbcb89aaab6fc0574da0871ae4e8268f9ee41"},"cell_type":"markdown","source":"One thing we did along the way was test out how our model would perform with a much larger testing set. We did this simply by loading in the test set multiple times and concatenating."},{"metadata":{"_uuid":"3eced255e6b4a85b1cef5e67f7a00d4dd40899ec","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(INPUT_PATH + 'train.csv')\n# if submit use original test set else split off from train\nif simulate_test:\n    train_df, test_df = train_test_split(train_df, test_size = 0.1, random_state = 28)\nelse:\n    test_df = pd.read_csv(INPUT_PATH + 'test.csv')\n#     for i in range(6):\n#         test_df = pd.concat([pd.read_csv(INPUT_PATH + 'test.csv'), test_df], axis = 0)\nprint(train_df.shape)\nprint(test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"18a7f488d3c82a45f8d38adb2a69e70ed69a9fad"},"cell_type":"markdown","source":"As most others have already shown in their strong placing kernels, preprocessing was very important. I think we probably could have done better here. It looks like many of the lemmatizing and tokenizing tricks we wrote off were actually useful. You can see how we went through a progression of text preprocessing based off of what is commented out. For a long time we were using the preprocessing that was publicly available in @dieters kernel https://www.kaggle.com/christofhenkel/how-to-preprocessing-when-using-embeddings. Eventually we settled on the below preprocessing by systematically testing each option  and combination of options and seeing how they performed. "},{"metadata":{"_uuid":"47b691a4526c573ef130f40eec8f89b378f83395","trusted":true},"cell_type":"code","source":"import string\nfrom nltk.tokenize.treebank import TreebankWordTokenizer\ntokenizer = TreebankWordTokenizer()\n# constants\n\n# from https://stackoverflow.com/questions/2013451/test-if-string-contains-only-letters-a-z-é-ü-ö-ê-å-ø-etc\nlatin_similar = \"’'‘ÆÐƎƏƐƔIJŊŒẞÞǷȜæðǝəɛɣijŋœĸſßþƿȝĄƁÇĐƊĘĦĮƘŁØƠŞȘŢȚŦŲƯY̨Ƴąɓçđɗęħįƙłøơşșţțŧųưy̨ƴÁÀÂÄǍĂĀÃÅǺĄÆǼǢƁĆĊĈČÇĎḌĐƊÐÉÈĖÊËĚĔĒĘẸƎƏƐĠĜǦĞĢƔáàâäǎăāãåǻąæǽǣɓćċĉčçďḍđɗðéèėêëěĕēęẹǝəɛġĝǧğģɣĤḤĦIÍÌİÎÏǏĬĪĨĮỊIJĴĶƘĹĻŁĽĿʼNŃN̈ŇÑŅŊÓÒÔÖǑŎŌÕŐỌØǾƠŒĥḥħıíìiîïǐĭīĩįịijĵķƙĸĺļłľŀʼnńn̈ňñņŋóòôöǒŏōõőọøǿơœŔŘŖŚŜŠŞȘṢẞŤŢṬŦÞÚÙÛÜǓŬŪŨŰŮŲỤƯẂẀŴẄǷÝỲŶŸȲỸƳŹŻŽẒŕřŗſśŝšşșṣßťţṭŧþúùûüǔŭūũűůųụưẃẁŵẅƿýỳŷÿȳỹƴźżžẓ\"\nwhite_list = string.ascii_letters + string.digits + latin_similar + ' '\n# from checking symbols_iv = {char:char_vocab[char] for char in symbols if char in embeddings_index}\nsymbols_iv = \"\"\"?,./-()\"$=…*&+′[ɾ̃]%:^\\xa0\\\\{}–“”;!<`®ạ°#²|~√_α→>—£，。´×@π÷？ʿ€の↑∞ʻ℅в•−а年！∈∩⊆§℃θ±≤͡⁴™си≠∂³ி½△¿¼∆≥⇒¬∨∫▾Ω＾γµº♭ー̂ɔ∑εντσ日Γ∪φβ¹∘¨″⅓ɑː✅✓（）∠«»்ுλ∧∀،＝ɨʋδɒ¸☹μΔʃɸηΣ₅₆◦·ВΦ☺❤♨✌≡ʌʊா≈⁰‛：fi„¾ρ⟨⟩˂⅔≅－＞¢⁸ʒは⬇♀؟¡⋅ɪ₁₂ɤ◌ʱ、▒ْ；☉＄∴✏ωɹ̅।ـ☝♏̉̄♡₄∼́̀⁶⁵¦¶ƒˆ‰©¥∅・ﾟ⊥ª†ℕ│ɡ∝♣／☁✔❓∗➡ℝ位⎛⎝¯⎞⎠↓ɐ∇⋯˚⁻ˈ₃⊂˜̸̵̶̷̴̡̲̳̱̪̗̣̖̎̿͂̓̑̐̌̾̊̕\\x92\"\"\"\n\ndef get_chars(train_X):\n    char_vocab = {}\n    for sent in tqdm_notebook(train_X):\n        for char in sent:\n            try:\n                char_vocab[char] += 1\n            except KeyError:\n                char_vocab[char] = 1\n    return char_vocab\n\ndef split_off_symbols_iv(x):\n    for punct in symbols_iv:\n        x = x.replace(punct, f' {punct} ')\n    return x\n\ndef preprocess(text):\n    \n    text = split_off_symbols_iv(text) #increase score\n    #text = delete_oov_symbols(text,translate_map)\n    #text = ' '.join(text.split())\n    #text = clean_apostrophes(text)\n    #text = clean_contractions(text, contraction_mapping)\n    #text = split_off_s(text)\n    #text = replace_acronyms(text,acronyms)\n    ##text = fix_mispells(text,mispells) decrease score\n    #text = fix_apo(text)\n    return text","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b790c0ebee70bd43431fe1e3fe950b7903ebb2d0","scrolled":true,"trusted":true},"cell_type":"code","source":"char_vocab = get_chars(train_df[\"question_text\"].values)\nsymbols = {char:char_vocab[char] for char in char_vocab if not char in white_list}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8ba91019507ff2e836c1af64bd9f3db5be7eb528","trusted":true},"cell_type":"code","source":"train_df[\"question_text\"] = train_df[\"question_text\"].progress_apply(lambda x: preprocess(x))\ntrain_df[\"question_text\"] = train_df[\"question_text\"].progress_apply(lambda x: tokenizer.tokenize(x, return_str=True))\ntest_df[\"question_text\"] = test_df[\"question_text\"].progress_apply(lambda x: preprocess(x))\ntest_df[\"question_text\"] = test_df[\"question_text\"].progress_apply(lambda x: tokenizer.tokenize(x, return_str=True))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2195fc5b36f080299d1d5497b1553fb7e33589eb"},"cell_type":"markdown","source":"This is all roughly standard stuff. Only thing I think we could have improved here was a longer maxlen and larger vocabulary size, but we were having issues early on with running out of memory and killing kernels. We never really revisited this, but it seems everyone else was able to fit in many more words and multiple embeddings at a time. "},{"metadata":{"_uuid":"2c361bb3c71044528378aa551b0b7de76c45175d","trusted":true},"cell_type":"code","source":"# some config values \nembed_size = 300 # how big is each word vector\nmax_features = 100000 # how many unique words to use (i.e num rows in embedding vector)\nmaxlen = 50 # max number of words in a question to use\n\ntrain_X = train_df[\"question_text\"].fillna(\"_na_\").values\ntest_X = test_df[\"question_text\"].fillna(\"_na_\").values\n## Tokenize the sentences\ntokenizer = Tokenizer(num_words=max_features, filters = '', lower = False)\ntokenizer.fit_on_texts(list(train_X) + list(test_X) )\ntrain_X = tokenizer.texts_to_sequences(train_X)\ntest_X = tokenizer.texts_to_sequences(test_X)\n## Pad the sentences \ntrain_X = pad_sequences(train_X, maxlen=maxlen)\ntest_X = pad_sequences(test_X, maxlen=maxlen)\nword_index = tokenizer.word_index","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6683124ae22ba417ffee16da8717ba9af5f8ae24","trusted":true},"cell_type":"code","source":"del tokenizer\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c901a544ae551841c07f5cd77ef8b6a6a8b0a70f"},"cell_type":"markdown","source":"Using the gensim function to load in the embeddings ended up being much more time efficient than most of the things available in the public kernels. I am sure a few other people used this same function but I did not see it widespread. "},{"metadata":{"_uuid":"daceae2e3c8387d7ec5185fc90fad23ecf04ebbf","trusted":true},"cell_type":"code","source":"from gensim import utils\n\ndef load_word2vec(fname, encoding='utf8', unicode_errors='strict',datatype=np.float32, max_vocab=3000000, word_index=None):\n    #emb_mean,emb_std = -0.0051106834, 0.18445626\n    #embedding_matrix = np.random.normal(emb_mean, emb_std, (max_features, 300))\n    embedding_index = {}\n    with utils.smart_open(fname) as fin:\n        header = utils.to_unicode(fin.readline(), encoding=encoding)\n        vocab_size, vector_size = (int(x) for x in header.split())\n        binary_len = np.dtype(datatype).itemsize * vector_size\n        \n        for _ in tqdm(range(min(vocab_size,max_vocab))):\n            # mixed text and binary: read text first, then binary\n            word = []\n            while True:\n                ch = fin.read(1)\n                if ch == b' ':\n                    break\n                if ch == b'':\n                    raise EOFError(\"unexpected end of input\")\n                if ch != b'\\n':\n                    word.append(ch)\n            word = utils.to_unicode(b''.join(word), encoding=encoding, errors=unicode_errors)\n            weights = np.fromstring(fin.read(binary_len), dtype=datatype).astype(np.float16)\n            embedding_index[word] = weights\n    return embedding_index","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5939ead65b7fde889408741b8ee6f00dbb8f227b"},"cell_type":"markdown","source":"Our functions for loading our embeddings"},{"metadata":{"_uuid":"c38e4b4d8b51f3d10eeaaccb8f39696730223627","trusted":true},"cell_type":"code","source":"def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n\ndef load_news(embed_dir = '../input/GoogleNews-vectors-negative300.bin'):\n    embeddings_index = KeyedVectors.load_word2vec_format(embed_dir, binary=True)\n    emb_ind = {}\n    for i, vec in tqdm(enumerate(embeddings_index.wv.vectors)):\n        emb_ind[embeddings_index.wv.index2word[i]] = vec\n    del embeddings_index\n    gc.collect()\n    return emb_ind\n        \ndef load_glove(word_index,embed_dir = \"../input/embeddings/glove.840B.300d/glove.840B.300d.txt\"):\n    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in tqdm_notebook(open(embed_dir, encoding = \"utf8\")) if o.split(\" \")[0] in word_index) \n    return embeddings_index\n\ndef load_para(embed_dir ='../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt'):\n    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in tqdm_notebook(open(embed_dir, encoding=\"utf8\", errors='ignore')) if len(o)>100)\n    return embeddings_index","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9724829776ed56541e61da2357f0ed2344925f5c"},"cell_type":"markdown","source":"Our standard embedding matrix builder. We will revisit this in a second because we heavily altered things after the fact."},{"metadata":{"_uuid":"0a3b4d6db77fc946dddb12f6b03b1715576670ff","trusted":true},"cell_type":"code","source":"def build_embedding_matrix(word_index,embeddings_index, max_features, maxlen, lower = False):\n    #all_embs = np.stack(embeddings_index.values())\n    #emb_mean,emb_std = all_embs.mean(), all_embs.std()\n\n    #embedding_matrix = np.random.normal(emb_mean, emb_std, (max_features, 300), dty)\n    embedding_matrix = np.zeros((max_features, 300))\n    for word, i in word_index.items():\n        if lower:\n            word = word.lower()\n        if i >= max_features: continue\n        embedding_vector = embeddings_index.get(word)\n        if embedding_vector is not None: \n            embedding_matrix[i] = embedding_vector\n    return embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"73cdd0816d6149c716b30ffa6976d9a4e522aa19","trusted":true},"cell_type":"markdown","source":"We only ended up using two of the below functions, add_lower_and_upper and add_symbols. What these do are add things to our embeddings which were missing. So if a word didnt show up in our embedding matrix, but we found that the lower case or uppercase was available in the pretrained embeddings then we would add that in where the blanks were. "},{"metadata":{"_uuid":"e758f8d5748115119e7e37d69711cc8daff9e732","trusted":true},"cell_type":"code","source":"def add_lower_and_upper(embedding, vocab):\n    count_l = 0\n    count_u = 0\n    for word in tqdm(vocab):\n        if word not in embedding and word.lower() in embedding:  \n            embedding[word] = embedding[word.lower()]\n            count_l += 1\n        if word not in embedding and word.title() in embedding:\n            embedding[word] = embedding[word.title()]\n            count_u += 1\n    print(f\"Added {count_l} lower word embeddings\")\n    print(f\"Added {count_u} upper word embeddings\")\n\ndef clean_numbers(x):\n\n    x = re.sub('[0-9]{5,}', '#####', x)\n    x = re.sub('[0-9]{4}', '####', x)\n    x = re.sub('[0-9]{3}', '###', x)\n    x = re.sub('[0-9]{2}', '##', x)\n    return x\n\ndef add_numbers(embedding, word_index):\n    count_d = 0\n    count_s = 0\n    for word in tqdm(word_index):\n        if word not in embedding:\n            cln = clean_numbers(word)\n            if cln in embedding:  \n                embedding[word] = embedding[cln]\n                count_d += 1\n    print(f\"Added {count_d} number embeddings\")\n    \ndef add_symbols(embedding, word_index):\n    \n    quotes=[\"``\",\"''\"]\n    symbs_iv = [char for char in symbols if char in embedding]\n    mean_symbol = np.zeros((300,))\n    for s in symbs_iv:\n        mean_symbol += embedding[s]\n    \n    mean_symbol /= len(symbs_iv)\n\n    count_s = 0\n    for word in tqdm(word_index):\n        if word not in embedding:\n            if len(word) == 1:\n                embedding[word] = mean_symbol\n                count_s += 1\n            if word in quotes:\n                embedding[word] = mean_symbol\n                count_s += 1\n    print(f\"Added {count_s} symbol embeddings\")\n    \ndef build_glove_matrix():\n    embeddings_index = load_glove(word_index=word_index,embed_dir = INPUT_PATH + \"embeddings/glove.840B.300d/glove.840B.300d.txt\")\n    add_lower_and_upper(embeddings_index, word_index)\n    add_symbols(embeddings_index, word_index)\n    embedding_matrix1 = build_embedding_matrix(word_index,embeddings_index, max_features, maxlen, lower = False)\n    return embedding_matrix1","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"76ddfc8cb974535ec857e8109a3ad3fb0987b59c","trusted":true},"cell_type":"code","source":"embedding_matrix1 = build_glove_matrix()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f0cf7382b448be61f9fccbb78f51b8aaa7d3c89d","trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1fc48f1f94d1fcd5a32e50dfb971148de2b9f593","trusted":true},"cell_type":"code","source":"train_y = train_df[\"target\"].values\nif simulate_test: \n    test_y = test_df[\"target\"].values\nelse:\n    test_y = None","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cecf1b52afaf9f8e75e7eefd8a6db4557df8290b","trusted":true},"cell_type":"code","source":"from keras.models import Model, load_model\nfrom keras.layers import Input, Dense, Embedding, concatenate, CuDNNGRU, CuDNNLSTM, Bidirectional, GlobalAveragePooling1D, GlobalMaxPooling1D, SpatialDropout1D, Flatten, Lambda, Permute, Reshape, merge, Dropout, Conv2D, MaxPool2D, Concatenate, Conv1D, MaxPool1D, add, MaxPooling1D\nfrom keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau\nfrom keras import backend as K\nfrom keras.losses import binary_crossentropy\n\nfrom keras.engine.topology import Layer\nfrom keras import initializers, regularizers, constraints, optimizers, layers\nfrom keras.initializers import glorot_uniform, he_uniform, he_normal\nfrom keras.optimizers import Adam\nfrom keras.layers import LeakyReLU, multiply\nfrom keras.layers import Reshape, Permute, multiply","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"89334e7e31e6fce680686de49e8e9fcf41bbc1ed"},"cell_type":"markdown","source":"Here is where our first major trick occurred. Full credit for Dieter for noticing this. What we did here was actually really clever to save time. After looking at the histograms of our predictions Dieter noticed that almost 70 percent of our predictions were under .01 meaning that the model was very confident that they were sincere questions. We also measured the accuracy of thes predictions and found that it was nearly perfect. Our models were able to find these easy ones within a few epochs. We exploited this by training one model in order to filter out the bottom 70 percent of the data and then retraining all of our models on the remaining 30%. This allowed us to train many more models in a quick span and more completely. After we train this model we simply throw it away so as not to leak any results into our validation set. All models after this one are trained from scratch only on the subset of data "},{"metadata":{"_uuid":"47bf82b82bc4383477c1f1251e3f4331cf0ddf39","trusted":true},"cell_type":"code","source":"def build_gru_model(embedding_matrix, name = \"gru\"):\n    \n    inp = Input(shape=(maxlen, ))\n    x = Embedding(max_features,300,weights=[embedding_matrix],input_length=maxlen,trainable=False)(inp)\n    x = SpatialDropout1D(.4,seed=seed_nb)(x)\n    x = CuDNNGRU(128, return_sequences=True,kernel_initializer=glorot_uniform(seed=seed_nb))(x)\n    x = CuDNNGRU(64, return_sequences=True,kernel_initializer=glorot_uniform(seed=seed_nb))(x)\n    last = Flatten()(Lambda(lambda x: x[:,-1:,:]) (x))\n    avg_pool = GlobalAveragePooling1D()(x)\n    max_pool = GlobalMaxPooling1D()(x)\n    conc = concatenate([avg_pool, max_pool, last])\n    #x = Dropout(.3,seed=seed_nb)(x)\n    x = Dense(32, activation = 'relu',kernel_initializer=he_uniform(seed=seed_nb))(conc)\n    outp = Dense(1, activation=\"sigmoid\",kernel_initializer=he_uniform(seed=seed_nb))(x)\n    \n    model = Model(inputs=[inp], outputs=outp, name = name)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1db96986e11c89c87d17366d39012f18672a843f","scrolled":true,"trusted":true},"cell_type":"code","source":"K.clear_session()\nbatch_size = 2048\nmodel = build_gru_model(embedding_matrix1, name = 'gru_stage0')\nmodel.compile(loss=binary_crossentropy,optimizer=Adam())\nmodel.fit(train_X, train_y, batch_size=batch_size, epochs=3,verbose=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4819135e6603b13ce55d18e3141d959a0497c844","trusted":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, accuracy_score, log_loss, f1_score\n\n\nfilter_pred = model.predict(train_X, batch_size = batch_size, verbose =True)[:,0]\nthresh = np.percentile(filter_pred,70.)\nprint(thresh)\nkeep_index = (filter_pred > thresh)\n#trash_index = np.logical_not(keep_index)\n#trashed_pred = filter_pred[trash_index]\n#trashed_train = train_X[trash_index]\n#trashed_y = train_y[trash_index]\n\n#print('trashing data with these metrics:')\n#print('confusion matrix:')\n#print(confusion_matrix(trashed_pred>0.5,trashed_y))\n#print('acc:')\n#print(accuracy_score(trashed_pred>0.5,trashed_y))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f0c6821be2086718db47af401d0bdecd5291b896","trusted":true},"cell_type":"code","source":"#k = np.where(keep_index)\n#t = np.where(np.logical_not(keep_index))\n#train_X_old = train_X.copy()\n#train_y_old = train_y.copy()\ntrain_X = train_X[np.where(keep_index)]\ntrain_y = train_y[np.where(keep_index)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ccd610b41dffdecbcab7cb9bb01451d204195a9b"},"cell_type":"code","source":"train_X.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fe2036b86fa5d8715154b9646e3879866fec4fab"},"cell_type":"markdown","source":"Now you can see that our training set is much smaller and we only need to focus on the difficult problems. It is a waste of time in order to continue training on training samples the model is already confident and accurate on. These examples are trivial for one reason or another. We found that dropping these samples does not negatively affect our accuracy at all and our models trained on the subset can still easily classify those that were dropped and never trained on. "},{"metadata":{"_uuid":"19451c0d9497e2318bc337897b3148a2a8354189","trusted":true},"cell_type":"code","source":"del filter_pred, model\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"43a451c0a6900f4dade3d56a6915b770bc495279"},"cell_type":"markdown","source":"After filtering out these samples we used 5 models. These were found by training an array of models with different embeddings and configurations and then seeing how they ensembled together. Our offline stacker eventually found that the best ensemble of models was:\n* DPCNN with reversing and glove embeddings\n* A bidirectional gru into an lstm with the glove embeddings. (this was very similar to what we used for the toxic comment challenge and was our strongest individual model here as well)\n* a parrallel lstm and gru model w/glove embeddings\n* parts of speech bidirectional lstm and gru model w/paragram embeddings\n* parts of speech parallel lstm and gru model w/news embeddings\n\nI will talk about the POS models more in a second\n\nOne thing I wish we had explored a bit more was stacked embeddings like many used. We simply were running into too many issues early on with killing kernels so I wrote it off as a viable option. In the toxic comment challenge we heavily utilized this with even stacking 3 embeddings at once and also encoding our own information in the embeddings like if the word was all_caps and other various features. We simply didnt have the throughput in order to use all of these techniques at once. "},{"metadata":{"_uuid":"e7a3ed60b660f8bd699cac03a1dede642cbcf5f7","trusted":true},"cell_type":"code","source":"class Attention(Layer):\n    def __init__(self, step_dim,\n                 W_regularizer=None, b_regularizer=None,\n                 W_constraint=None, b_constraint=None,\n                 bias=True, **kwargs):\n        self.supports_masking = True\n        self.init = initializers.get('glorot_uniform')\n\n        self.W_regularizer = regularizers.get(W_regularizer)\n        self.b_regularizer = regularizers.get(b_regularizer)\n\n        self.W_constraint = constraints.get(W_constraint)\n        self.b_constraint = constraints.get(b_constraint)\n\n        self.bias = bias\n        self.step_dim = step_dim\n        self.features_dim = 0\n        super(Attention, self).__init__(**kwargs)\n\n    def build(self, input_shape):\n        assert len(input_shape) == 3\n\n        self.W = self.add_weight((input_shape[-1],),\n                                 initializer=self.init,\n                                 name='{}_W'.format(self.name),\n                                 regularizer=self.W_regularizer,\n                                 constraint=self.W_constraint)\n        self.features_dim = input_shape[-1]\n\n        if self.bias:\n            self.b = self.add_weight((input_shape[1],),\n                                     initializer='zero',\n                                     name='{}_b'.format(self.name),\n                                     regularizer=self.b_regularizer,\n                                     constraint=self.b_constraint)\n        else:\n            self.b = None\n\n        self.built = True\n\n    def compute_mask(self, input, input_mask=None):\n        return None\n\n    def call(self, x, mask=None):\n        features_dim = self.features_dim\n        step_dim = self.step_dim\n\n        eij = K.reshape(K.dot(K.reshape(x, (-1, features_dim)),\n                        K.reshape(self.W, (features_dim, 1))), (-1, step_dim))\n\n        if self.bias:\n            eij += self.b\n\n        eij = K.tanh(eij)\n\n        a = K.exp(eij)\n\n        if mask is not None:\n            a *= K.cast(mask, K.floatx())\n\n        a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n\n        a = K.expand_dims(a)\n        weighted_input = x * a\n        return K.sum(weighted_input, axis=1)\n\n    def compute_output_shape(self, input_shape):\n        return input_shape[0],  self.features_dim\n\ndef squash(x, axis=-1):\n    # s_squared_norm is really small\n    # s_squared_norm = K.sum(K.square(x), axis, keepdims=True) + K.epsilon()\n    # scale = K.sqrt(s_squared_norm)/ (0.5 + s_squared_norm)\n    # return scale * x\n    s_squared_norm = K.sum(K.square(x), axis, keepdims=True)\n    scale = K.sqrt(s_squared_norm + K.epsilon())\n    return x / scale\n\n# A Capsule Implement with Pure Keras\nclass Capsule(Layer):\n    def __init__(self, num_capsule, dim_capsule, routings=3, kernel_size=(9, 1), share_weights=True,\n                 activation='default', **kwargs):\n        super(Capsule, self).__init__(**kwargs)\n        self.num_capsule = num_capsule\n        self.dim_capsule = dim_capsule\n        self.routings = routings\n        self.kernel_size = kernel_size\n        self.share_weights = share_weights\n        if activation == 'default':\n            self.activation = squash\n        else:\n            self.activation = Activation(activation)\n\n    def build(self, input_shape):\n        super(Capsule, self).build(input_shape)\n        input_dim_capsule = input_shape[-1]\n        if self.share_weights:\n            self.W = self.add_weight(name='capsule_kernel',\n                                     shape=(1, input_dim_capsule,\n                                            self.num_capsule * self.dim_capsule),\n                                     # shape=self.kernel_size,\n                                     initializer='glorot_uniform',\n                                     trainable=True)\n        else:\n            input_num_capsule = input_shape[-2]\n            self.W = self.add_weight(name='capsule_kernel',\n                                     shape=(input_num_capsule,\n                                            input_dim_capsule,\n                                            self.num_capsule * self.dim_capsule),\n                                     initializer='glorot_uniform',\n                                     trainable=True)\n\n    def call(self, u_vecs):\n        if self.share_weights:\n            u_hat_vecs = K.conv1d(u_vecs, self.W)\n        else:\n            u_hat_vecs = K.local_conv1d(u_vecs, self.W, [1], [1])\n\n        batch_size = K.shape(u_vecs)[0]\n        input_num_capsule = K.shape(u_vecs)[1]\n        u_hat_vecs = K.reshape(u_hat_vecs, (batch_size, input_num_capsule,\n                                            self.num_capsule, self.dim_capsule))\n        u_hat_vecs = K.permute_dimensions(u_hat_vecs, (0, 2, 1, 3))\n        # final u_hat_vecs.shape = [None, num_capsule, input_num_capsule, dim_capsule]\n\n        b = K.zeros_like(u_hat_vecs[:, :, :, 0])  # shape = [None, num_capsule, input_num_capsule]\n        for i in range(self.routings):\n            b = K.permute_dimensions(b, (0, 2, 1))  # shape = [None, input_num_capsule, num_capsule]\n            c = K.softmax(b)\n            c = K.permute_dimensions(c, (0, 2, 1))\n            b = K.permute_dimensions(b, (0, 2, 1))\n            outputs = self.activation(tf.keras.backend.batch_dot(c, u_hat_vecs, [2, 2]))\n            if i < self.routings - 1:\n                b = tf.keras.backend.batch_dot(outputs, u_hat_vecs, [2, 3])\n\n        return outputs\n\n    def compute_output_shape(self, input_shape):\n        return (None, self.num_capsule, self.dim_capsule)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2051f6715169fd62b1a8abd258b591f1bbd58a8d","trusted":true},"cell_type":"code","source":"def f1(y_true, y_pred):\n    def recall(y_true, y_pred):\n        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n        recall = true_positives / (possible_positives + K.epsilon())\n        return recall\n\n    def precision(y_true, y_pred):\n        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n        precision = true_positives / (predicted_positives + K.epsilon())\n        return precision\n    precision = precision(y_true, y_pred)\n    recall = recall(y_true, y_pred)\n    return 2*((precision*recall)/(precision+recall+K.epsilon()))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"627a9574c8db3aed8c31dbb9152354fbad082e2f"},"cell_type":"markdown","source":"One major improvement we were able to add to all of our models because of this time savings was much higher spatial dropout and also a secondary training phase where we trained only the embedding layer. \n\nSpatial Dropout we found to be exceptionally effective to prevent overfitting in the toxic comment challenge. What this does is mask out dimensions of the embeddings. This is well described a couple places in the forums https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/79911. The key is that your model is only seeing subsets of the dimensions of the embeddings at a time so it does not fit to all of them at once. Some people were doing this incorrectly in the pytorch variant where Dropout2D was dropping out entire words rather than just dimensions of the embeddings. We did some experiments of our own with PyTorch and confirmed this was not correct so @MihaS tried to correct this by manually creating a new dropout layer for us. We did not end up spending much more time with PyTorch though because the majority of us were proficient with Keras. \n\nThe second portion, finetuning the embeddings is very crucial. Many people I saw used either unfrozen embeddings the entire time or left them frozen the entire training process. In my opinion this is not the correct way to approach it. If we look at how we do transfer learning with imagenet models and the like what people typically do is freeze the network except for the last layer and train that and then tune the rest of the network after that last layer is at least somewhat trained. The reason behind doing this is if you train the entire network and your last layer is randomly initialized then it will essentially just blow out whatever was there because the gradient will be so noisy. If you do that you are using them as initialization points rather than to actually transfer knowledge to a new problem. The same is true with word embeddings. We train the rest of the network around the embeddings and then once that is adequately trained we can do some tweaking of the embeddings. In this way we are able to maintain all of the juicy information captured in these embeddings and also impart some of the information from this new dataset"},{"metadata":{"_uuid":"6eca7e28ba773553f53a44b6c2d3388ded4d7a54","trusted":true},"cell_type":"code","source":"def train_model(model,train_x, y_tra,epochs,batch_size):\n    \n    \n    #complete_hist = {'model_name':model.name}\n    #complete_hist['epochs'] = [0,0,0]\n    #tic = time.time()\n    model.compile(loss='binary_crossentropy',optimizer=Adam(),metrics=['acc', f1])\n\n    hist = model.fit(train_x, y_tra, batch_size=batch_size, epochs=epochs[1],verbose=True)\n\n    for layer in model.layers:\n        layer.trainable = False\n    model.layers[1].trainable = True\n    \n    model.compile(loss='binary_crossentropy',optimizer=Adam(),metrics=['acc', f1])\n    hist = model.fit(train_x, y_tra, batch_size=batch_size, epochs=epochs[2],verbose=True)\n    \n    return hist","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9d74f9700d282d37f11d26d5714f8615ca50a6bb","trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ebef366dc1ce73699fb3dda7502ecffd480b3981","trusted":true},"cell_type":"code","source":"def build_dpcnn_model(embedding_matrix, num_block = 5, k = 3, units = 64,name = 'dpcnn'):\n    pad = 'same'\n    inp = Input(shape = (None, ))\n    embedding_layer1 = Embedding(max_features, embed_size, weights=[embedding_matrix], embeddings_initializer = he_uniform(seed=seed_nb), trainable = False)(inp)\n    embedding_layer1 = SpatialDropout1D(0.3)(embedding_layer1)\n    emb_short_cut = Conv1D(units, kernel_size = 1, padding = pad, activation = 'relu')(embedding_layer1)\n    emb_short_cut = Conv1D(units, kernel_size = 1, padding = pad, activation = 'relu')(emb_short_cut)\n    # Main block\n    for b in range(1, num_block + 1):\n        if b == 1:\n            block = embedding_layer1\n            short_cut = emb_short_cut\n        else:\n            block = block\n            short_cut = block\n        block = Conv1D(units, kernel_size = k, padding = pad, activation = 'relu')(block)\n        block = Conv1D(units, kernel_size = k, padding = pad, activation = 'relu')(block)\n        block = add([short_cut, block])\n        block = MaxPooling1D(pool_size = 3, strides = 2, padding = pad)(block)\n    # Final block\n    short_cut = block\n    block = Conv1D(units, kernel_size = k, padding = pad, activation = 'relu')(block)\n    block = Conv1D(units, kernel_size = k, padding = pad, activation = 'relu')(block)\n    block = add([short_cut, block])\n    max_pool = GlobalMaxPooling1D()(block)\n    avg_pool = GlobalAveragePooling1D()(block)\n    last = (Lambda(lambda x: x[:,-1,:]) (block))\n    block = concatenate([max_pool, avg_pool, last])\n    #reverse\n    rev_embedding_layer = Lambda(lambda x: K.reverse(x,axes=-1))(embedding_layer1)\n    rev_emb_short_cut = Conv1D(units, kernel_size = 1, padding = pad, activation = 'relu')(rev_embedding_layer)\n    rev_emb_short_cut = Conv1D(units, kernel_size = 1, padding = pad, activation = 'relu')(rev_emb_short_cut)\n    # Main block\n    for b in range(1, num_block + 1):\n        if b == 1:\n            rev_block = rev_embedding_layer\n            rev_short_cut = rev_emb_short_cut\n        else:\n            rev_block = rev_block\n            rev_short_cut = rev_block\n        rev_block = Conv1D(units, kernel_size = k, padding = pad, activation = 'relu')(rev_block)\n        rev_block = Conv1D(units, kernel_size = k, padding = pad, activation = 'relu')(rev_block)\n        rev_block = add([rev_short_cut, rev_block])\n        rev_block = MaxPooling1D(pool_size = 3, strides = 2, padding = pad)(rev_block)\n    # Final block\n    rev_short_cut = rev_block\n    rev_block = Conv1D(units, kernel_size = k, padding = pad, activation = 'relu')(rev_block)\n    rev_block = Conv1D(units, kernel_size = k, padding = pad, activation = 'relu')(rev_block)\n    rev_block = add([rev_short_cut, rev_block])\n    rev_max_pool = GlobalMaxPooling1D()(rev_block)\n    rev_avg_pool = GlobalAveragePooling1D()(rev_block)\n    rev_last = (Lambda(lambda x: x[:,-1,:]) (rev_block))\n    rev_block = concatenate([rev_max_pool, rev_avg_pool, rev_last])\n    block = concatenate([rev_block, block])\n    # output block\n    out_put = Dense(64, activation = 'relu')(block)\n    outp = Dense(1, activation=\"sigmoid\")(out_put)\n    model = Model(inputs=inp, outputs=outp, name=name)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"521bcee948279e41dcb473a8499ac31b76aabdc5","trusted":true},"cell_type":"code","source":"n_models = 5\npreds_test_all = np.zeros((test_X.shape[0],n_models))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68f8de54adcd1e005938f248709f8b1fe5559367","trusted":true},"cell_type":"code","source":"K.clear_session()\nmodel_name = 'dpcnn'\nbatch_size = 1000\nepochs = [0,12,3] # checked locally\n\nmodel = build_dpcnn_model(embedding_matrix1)\ndpcnn_hist = train_model(model,train_X, train_y, epochs,batch_size)\npreds_test_all[:,0] = model.predict(test_X, batch_size=batch_size,verbose = True)[:,0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa7d5336a15af5bd18eb59a7c32e626c9f493a79","trusted":true},"cell_type":"code","source":"\n#ens_pred1 = model.predict(X_ens, batch_size=batch_size,verbose = True)\n#os.remove(\"dpcnn.h5\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"82513b59008ff5605f24b8aa0b9907e17884a27d","trusted":true},"cell_type":"code","source":"del model, dpcnn_hist\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"79e4ee4e5ceabcb2f6c7762f433e0af9abffe238"},"cell_type":"code","source":"def build_gru_lstm_model(embedding_matrix, name = \"\"):\n    \n    inp = Input(shape=(maxlen, ))\n    x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)\n    x = SpatialDropout1D(.4, seed = seed_nb)(x)\n    x = Bidirectional(CuDNNGRU(128, return_sequences=True,kernel_initializer=glorot_uniform(seed=seed_nb)))(x)\n    x = Dropout(.2, seed = seed_nb)(x)\n    x = Bidirectional(CuDNNLSTM(64, return_sequences=True,kernel_initializer=glorot_uniform(seed=seed_nb)))(x)\n\n    last = Flatten()(Lambda(lambda x: x[:,-1:,:]) (x))\n    avg_pool = GlobalAveragePooling1D()(x)\n    max_pool = GlobalMaxPooling1D()(x)\n    conc = concatenate([avg_pool,max_pool,last])\n    x = Dropout(.3, seed = seed_nb)(x)\n    outp = Dense(1, activation=\"sigmoid\",kernel_initializer=he_uniform(seed=seed_nb))(conc)\n    \n    model = Model(inputs=[inp], outputs=outp, name = name)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5da9751999b71ebfbbdbd32f85c8981f3e9f1f88"},"cell_type":"code","source":"K.clear_session()\nmodel_name = 'gru_lstm'\nbatch_size = 2048\nepochs = [0,12,3] # checked locally\n\nmodel = build_gru_lstm_model(embedding_matrix1)\nlstm_gru_hist = train_model(model,train_X, train_y, epochs,batch_size)\npreds_test_all[:,1] = model.predict(test_X, batch_size=batch_size,verbose = True)[:,0]\ndel model, lstm_gru_hist\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"437dbd18eb8a3873f044865c6f901c87b15621df"},"cell_type":"code","source":"def build_parallel_lstm_gru(embedding_matrix, name = \"\"):\n    inp = Input(shape=(maxlen,))\n    x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)\n    x = SpatialDropout1D(.4, seed = seed_nb)(x)\n    x1 = CuDNNGRU(128, return_sequences=True,kernel_initializer=glorot_uniform(seed=seed_nb))(x)\n    x2 = CuDNNLSTM(128, return_sequences=True,kernel_initializer=glorot_uniform(seed=seed_nb))(x)\n    x = Concatenate()([x1,x2])\n    x = Dropout(.2, seed = seed_nb)(x)\n    x1 = CuDNNGRU(64, return_sequences=True,kernel_initializer=glorot_uniform(seed=seed_nb))(x)\n    x2 = CuDNNLSTM(64, return_sequences=True,kernel_initializer=glorot_uniform(seed=seed_nb))(x)\n    x = Concatenate()([x1,x2])\n    last = Flatten()(Lambda(lambda x: x[:,-1:,:]) (x))\n    avg_pool = GlobalAveragePooling1D()(x)\n    max_pool = GlobalMaxPooling1D()(x)\n    conc = concatenate([avg_pool, max_pool,  last])\n    #x = Dropout(.3, seed = seed_nb)(x)\n    outp = Dense(1, activation=\"sigmoid\",kernel_initializer=he_uniform(seed=seed_nb))(conc)\n    \n    model = Model(inputs=[inp], outputs=outp, name = name)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2858093c20852a6692edfc1ec15ffc4846f7d2fe"},"cell_type":"code","source":"K.clear_session()\nmodel_name = 'parallel_lstm_gru2'\nbatch_size = 2048\nepochs = [0,11,4] # checked locally\n\nmodel = build_parallel_lstm_gru(embedding_matrix1)\nparallel_lstm_gru2_hist = train_model(model,train_X, train_y, epochs,batch_size)\npreds_test_all[:,2] = model.predict(test_X, batch_size=batch_size,verbose = True)[:,0]\ndel model, parallel_lstm_gru2_hist\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5374ae7bb08148fe00f1d4580eea21df3f1f7311"},"cell_type":"markdown","source":"The below section is where we did POS tagging in order to disambiguate the words. I have a more detailed write up of this in my other public kernel https://www.kaggle.com/ryches/parts-of-speech-disambiguation-error-analysis. This technique on its own performs worse (I have some hypotheses of why this might be the case. It has to do with the way I am initializing them in the same spot. If I were to do something to break their symmetry I think it would likely perform even better) but when ensembled, since the model is learning based off of slightly more expressive embeddings, it actually performs significantly better. The offline stacker gave these much higher weights than initially expected. I used this same technique in toxic comment, but I was not able to extensively test how it worked and my method is much more elegant now. Last competition I simply sent the results to alex and he gave me the thumbs up saying it seemed to get weight in the stacker because we found out so late in the competition. This time I got to actually play with it and see it for myself. This took our 5 model cv from .702 to upper .705. when stacked. We were only able to do this by training on the subset. With a full training set we would have run out of time. Especially with the expanded test set. \n\nOur other submission was able to fit 8 models in one kernel, but it turns out it was worth dropping those models in place of the POS tagging. "},{"metadata":{"_uuid":"aa95ae17536308295381c13bc60729b5e764c385"},"cell_type":"markdown","source":"There are various difficulties with the POS tagging, specifically making the text actually line up with the parts of speech that are output. Keras tokenizer will typically break these so I had to take care with this."},{"metadata":{"trusted":true,"_uuid":"30555ee5044cf2e48969a476c0c4d73a44e067f4"},"cell_type":"code","source":"train_X = train_df[\"question_text\"].fillna(\"_na_\").values\ntest_X = test_df[\"question_text\"].fillna(\"_na_\").values\ntrain_X = train_X[np.where(keep_index)]\nfrom spacy.tokens import Doc\nfrom spacy.vocab import Vocab\nimport spacy\nclass WhitespaceTokenizer(object):\n    def __init__(self, vocab):\n        self.vocab = vocab\n    def __call__(self, text):\n        words = text.strip().split(' ')\n        words = [word for word in words if word is not '']\n        spaces = [True] * len(words)\n        return Doc(self.vocab, words=words, spaces=spaces)\nnlp = spacy.load('en', disable = [\"parser\", \"ner\", \"textcat\"])\nnlp.tokenizer = WhitespaceTokenizer(nlp.vocab)\ntokens = []\nfor doc in tqdm(nlp.pipe(train_X, n_threads = 8)):\n    tokens.append(\" \".join([n.text + \"_\"  + n.pos_ for n in doc]))\ntrain_X = tokens\ntokens = []\nfor doc in tqdm(nlp.pipe(test_X, n_threads = 8)):\n    tokens.append(\" \".join([n.text + \"_\"  + n.pos_ for n in doc]))\ntest_X = tokens\n## Tokenize the sentences\ntokenizer = Tokenizer(num_words=max_features, filters = '', lower = False)\ntokenizer.fit_on_texts(list(train_X) + list(test_X) )\ntrain_X = tokenizer.texts_to_sequences(train_X)\ntest_X = tokenizer.texts_to_sequences(test_X)\n## Pad the sentences \ntrain_X = pad_sequences(train_X, maxlen=maxlen)\ntest_X = pad_sequences(test_X, maxlen=maxlen)\nword_index = tokenizer.word_index","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"46e5011e79e9e52b1c8e7d2e406338160d7841db"},"cell_type":"code","source":"def build_embedding_matrix(word_index,embeddings_index, max_features, maxlen, lower = False):\n    all_embs = np.stack(embeddings_index.values())\n    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n\n#     embedding_matrix = np.random.normal(emb_mean, emb_std, (max_features, 300))\n    embedding_matrix = np.zeros((max_features, 300))\n    for word, i in word_index.items():\n        if i >= max_features: continue\n        word_part = word.split(\"_\")[0]\n        embedding_vector = embeddings_index.get(word_part)\n        if embedding_vector is not None: \n            embedding_matrix[i] = embedding_vector\n    return embedding_matrix\ndef add_lower_and_upper(embedding, vocab):\n    count_l = 0\n    count_u = 0\n    for word in tqdm(vocab):\n        word = word.split(\"_\")[0]\n        if word not in embedding and word.lower() in embedding:  \n            embedding[word] = embedding[word.lower()]\n            count_l += 1\n        if word not in embedding and word.title() in embedding:\n            embedding[word] = embedding[word.title()]\n            count_u += 1\n    print(f\"Added {count_l} lower word embeddings\")\n    print(f\"Added {count_u} upper word embeddings\")\nembeddings_index2 = load_para(embed_dir = INPUT_PATH + 'embeddings/paragram_300_sl999/paragram_300_sl999.txt')\nadd_lower_and_upper(embeddings_index2, word_index)\nadd_symbols(embeddings_index2, word_index)\nembedding_matrix1 = build_embedding_matrix(word_index,embeddings_index2, max_features, maxlen, lower = True)\ndel embeddings_index2\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"64e942f57e191a305719895e858f9b5b7eece003","trusted":true},"cell_type":"code","source":"def build_lstm_gru_model(embedding_matrix, name = \"\"):\n    \n    inp = Input(shape=(maxlen, ))\n    x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)\n    x = SpatialDropout1D(.4, seed = seed_nb)(x)\n    x = Bidirectional(CuDNNLSTM(128, return_sequences=True,kernel_initializer=glorot_uniform(seed=seed_nb)))(x)\n    x = Dropout(.2, seed = seed_nb)(x)\n    x = Bidirectional(CuDNNGRU(64, return_sequences=True,kernel_initializer=glorot_uniform(seed=seed_nb)))(x)\n\n    last = Flatten()(Lambda(lambda x: x[:,-1:,:]) (x))\n    avg_pool = GlobalAveragePooling1D()(x)\n    max_pool = GlobalMaxPooling1D()(x)\n    conc = concatenate([avg_pool,max_pool,last])\n    x = Dropout(.3, seed = seed_nb)(x)\n    outp = Dense(1, activation=\"sigmoid\",kernel_initializer=he_uniform(seed=seed_nb))(conc)\n    \n    model = Model(inputs=[inp], outputs=outp, name = name)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"19bd8c92cfa5bdc5441c6d31e02c6eba5b03e4c8","trusted":true},"cell_type":"code","source":"K.clear_session()\nmodel_name = 'lstm_gru'\nbatch_size = 2048\nepochs = [0,13,3] # checked locally\n\nmodel = build_lstm_gru_model(embedding_matrix1)\nlstm_gru_hist = train_model(model,train_X, train_y, epochs,batch_size)\npreds_test_all[:,3] = model.predict(test_X, batch_size=batch_size,verbose = True)[:,0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"98a758069947526feb16202410e82b0228a9bc78","trusted":true},"cell_type":"code","source":"del model, lstm_gru_hist\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c2d8fd9d3b23ac359f537108be3130d390de14b","trusted":true},"cell_type":"code","source":"del embedding_matrix1\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d8bd4c560eb044df347003617314c00936f4a76b","scrolled":true,"trusted":true},"cell_type":"code","source":"embeddings_index3 = load_word2vec(INPUT_PATH + 'embeddings/GoogleNews-vectors-negative300/GoogleNews-vectors-negative300.bin')\nadd_lower_and_upper(embeddings_index3, word_index)\nadd_numbers(embeddings_index3, word_index)\nadd_symbols(embeddings_index3, word_index)\nembedding_matrix1 = build_embedding_matrix(word_index,embeddings_index3, max_features, maxlen, lower = False)\ndel embeddings_index3\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba3441f9d8f9c7dfc1db092d4465e0b7884e11b0","scrolled":true,"trusted":true},"cell_type":"code","source":"K.clear_session()\nmodel_name = 'parallel_lstm_gru3'\nbatch_size = 2048\nepochs = [0,13,3] # checked locally\n\nmodel = build_parallel_lstm_gru(embedding_matrix1)\nparallel_lstm_gru3_hist = train_model(model,train_X, train_y, epochs,batch_size)\npreds_test_all[:,4] = model.predict(test_X, batch_size=batch_size,verbose = True)[:,0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c16bd80147e6a43c1490e0331a8995d447404e88","trusted":true},"cell_type":"code","source":"del model, parallel_lstm_gru3_hist\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ae0d6983937b4dcaf89cb10ec90ffd1f44c9c39e","trusted":true},"cell_type":"code","source":"del embedding_matrix1\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"88ead96e6c3a76d1718958c55a2bf26ec7bd6be0"},"cell_type":"markdown","source":"These are the weights that were found offline using hillclimbing. These are not really all that precise. At the end we ran out of time so I simply ran things once and this is what was popped out. "},{"metadata":{"_uuid":"152837f3c34ce1ab4811e5badf3ad72c71ef606c","trusted":true},"cell_type":"code","source":"w = np.array([41, 95, 66, 68, 87])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"481e3a963a572523860bab152199643114c18847","trusted":true},"cell_type":"code","source":"preds = np.zeros((test_X.shape[0]))\nfor i in range(5):\n    preds += w[i]*preds_test_all[:,i]\npreds/= np.sum(w)\npreds.mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"16fba592031ac6be62f8323d7175732dbf276835"},"cell_type":"markdown","source":"Our threshold was set to .34 because this seemed to work reasonably well in our CV. One thing to note is we totally dropped any sort of validation come rollout time in order to maximize the training data our model was exposed to. This ended up having a significant effect on our performance. \n\nThe threshold likely could have been more precisely selected. \n\nIt's unfortunate we weren't able to further exploit the tricks we came across, but glad the decision to give a shot to the time consuming POS tagging step paid off. Once again it shows how important preprocessing is. "},{"metadata":{"_uuid":"a47507e809ff5df6917191e1522d30d083448a63","trusted":true},"cell_type":"code","source":"y_te = np.array(preds>0.34, dtype=np.int8)\nsubmit_df = pd.DataFrame({\"qid\": test_df[\"qid\"], \"prediction\": y_te})\nsubmit_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c326301fcb3c89e1a0ef7f419c23faba197a2ebc","trusted":true},"cell_type":"code","source":"# submit_df = pd.DataFrame({\"qid\": test_df[\"qid\"], \"prediction\": pred})\n# submit_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6bbdd782c2625373c91cb40678a835fad0da932a"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}