{"cells":[{"metadata":{"_uuid":"be476743688e9e7ab2dea38d017f8cd981fffad2"},"cell_type":"markdown","source":"# Embeddings-keras-v02\nFrom v1 we draw the conclusion:\nA good idea would be to wrap everything into a function that we can feed with train/val/test-data as well as an embedding matrix and all hyper parameters. This function could then just train and validate the model, get the threshold that gave the best f1 score and then return the predictions based on this threshold. This function can then be executed for multiple embeddings thus getting predictions based on different embeddings.\n\nThat's what I will do in this notebook"},{"metadata":{"_uuid":"f42c2fdca66aeb3f7aaf8232f2380a5f89241328"},"cell_type":"markdown","source":"As usual, start with the imports..."},{"metadata":{"trusted":false,"_uuid":"d3ce82c68e61888caa708bc54edd5f5f59781e2c"},"cell_type":"code","source":"import pandas as pd\nfrom collections import Counter\nimport numpy as np\nimport time\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics\n\n# We'll have to import all keras stuff here later\nfrom keras.layers import Bidirectional, Dense, Dropout, Embedding, CuDNNLSTM, CuDNNGRU, Input, GlobalMaxPool1D\nfrom keras.callbacks import ModelCheckpoint\nfrom keras.models import Model, load_model\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras import optimizers\n\n# Can keras find a gpu?\nfrom keras import backend as K\nprint(K.tensorflow_backend._get_available_gpus())","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"0769aa3a73ea2cf91fa70c6925c97e98a557adb4"},"cell_type":"code","source":"# Helper function that will be used in all cells!\ndef timeSince(t0):\n    ''' This function will be used to print the time since t0. \n        Will be called in every cell to give me some measurement. '''\n    print('Cell complete in {:.0f}m {:.0f}s'.format((time.time()-t0) // 60, (time.time()-t0) % 60))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f817700c4ffe7375951a67fced25f05e80fb1cb"},"cell_type":"markdown","source":"## Define constants"},{"metadata":{"trusted":false,"_uuid":"0a0cf0e75b8e3dcc5e672d1740fcad092a2d66de"},"cell_type":"code","source":"# Dataset path\n_traindataset = '../input/train.csv'\n_testdataset = '../input/test.csv'\n\n# Embeddings path\n_glove = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\n_paragram =  '../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt'\n_wiki_news = '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec'\n_google_news = '../input/embeddings/GoogleNews-vectors-negative300/GoogleNews-vectors-negative300.bin'\n\nembeddings = [{'name': 'glove', 'path': _glove},\n              {'name': 'paragram', 'path': _paragram},\n              {'name': 'fasttext', 'path': _wiki_news}]\n\n# Other constants here?","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fd8e9f728ea567a727808a5223e0968c750528d1"},"cell_type":"markdown","source":"Load dataset"},{"metadata":{"trusted":false,"_uuid":"6230df4828405a5fb6c52e6696455dc537851d10"},"cell_type":"code","source":"t0 = time.time()\ndf_train = pd.read_csv(_traindataset)\ndf_test = pd.read_csv(_testdataset)\ntimeSince(t0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b0b2a534feb5ffc5c13b2fc7bc894213e3fd740"},"cell_type":"markdown","source":"## Define hyperparameters"},{"metadata":{"trusted":false,"_uuid":"a0c1882ff95234cc623315b42b71d96b8a4f8774"},"cell_type":"code","source":"t0 = time.time()\n\nhparam = {}\nhparam['VOCAB_SIZE'] = 50000\nhparam['PAD_LENGTH'] = 100\nhparam['MINIBATCH_SIZE'] = 512\nhparam['LEARNING_RATE'] = 1e-3\nhparam['EPOCHS'] = 5\nhparam['LSTM_HIDDEN_SIZE'] = 128\nhparam['WORD_EMB_DIM'] = 300 # This will be set per embedding but all should be 300\n    \ntimeSince(t0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a42d958d0d0364959c0b07779fa632b92db539c2"},"cell_type":"markdown","source":"Create train/val split"},{"metadata":{"trusted":false,"_uuid":"e11ebb8acd742f87112c505ac38687de4cbc596f"},"cell_type":"code","source":"t0 = time.time()\ndf_train, df_val = train_test_split(df_train, test_size=0.1, random_state=2019)\ntimeSince(t0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b3b55d537d8b9c5feae25aa50d893a74d36d5a77"},"cell_type":"markdown","source":"## Preprocessing\nRight now there isn't much preprocessing going on in the cells below. This will be in v03!\nRight now we extract the data we're interested in, convert every question to lowercase, tokenize and pad to a fix size..."},{"metadata":{"_uuid":"9aed51a0dcfb73e83297145ebecbe35e2ec705e5"},"cell_type":"markdown","source":"Extract data we're interested in"},{"metadata":{"trusted":false,"_uuid":"6c29b8cfa139a466dfce0015dfd2626db1fa7922"},"cell_type":"code","source":"t0 = time.time()\n        \nX_train = df_train['question_text'].fillna(\"_nan_\").str.lower().values\nX_val = df_val['question_text'].fillna(\"_nan_\").str.lower().values  \nX_test = df_test['question_text'].fillna(\"_nan_\").str.lower().values  \ny_train = df_train['target'].values\ny_val = df_val['target'].values\nqid_test = df_test['qid'].values\n\nprint(\"Lenth X_train, y_train = {}, {}\".format(len(X_train), len(y_train)))\nprint(\"Lenth X_val, y_val = {}, {}\".format(len(X_val), len(y_val)))\nprint(\"Lenth X_test, qid_test = {}, {}\".format(len(X_test), len(qid_test)))\n\ntimeSince(t0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"523c251be4d8588b4f42650bea55e9217e0e36c6"},"cell_type":"markdown","source":"Tokenize and pad"},{"metadata":{"trusted":false,"_uuid":"9fc6b415a8b929e372da35690711ba2b9842b8b1"},"cell_type":"code","source":"t0 = time.time()\n\nvocab_size = hparam['VOCAB_SIZE']\npad_length = hparam['PAD_LENGTH']\n\n## Tokenize the sentences\ntokenizer = Tokenizer(num_words=vocab_size)\ntokenizer.fit_on_texts(list(X_train) + list(X_test))\nX_train_token = tokenizer.texts_to_sequences(X_train)\nX_val_token = tokenizer.texts_to_sequences(X_val)\nX_test_token = tokenizer.texts_to_sequences(X_test)\n\n## Pad the sentences \nX_train_pad = pad_sequences(X_train_token, maxlen=pad_length)\nX_val_pad = pad_sequences(X_val_token, maxlen=pad_length)\nX_test_pad = pad_sequences(X_test_token, maxlen=pad_length)\n\n# pack into placeholder\ndataset = {'X_train': X_train_pad, 'y_train': y_train,\n          'X_val': X_val_pad, 'y_val': y_val,\n          'X_test': X_test_pad}\n\nprint(\"X_train_pad.shape: {}\".format(X_train_pad.shape))\nprint(\"y_train.shape: {}\".format(y_train.shape))\nprint(\"X_val_pad.shape: {}\".format(X_val_pad.shape))\nprint(\"y_val.shape: {}\".format(y_val.shape))\nprint(\"X_test_pad.shape: {}\".format(X_test_pad.shape))\n\nprint(X_train[3])\nprint(X_train_pad[3])\nprint(len(tokenizer.word_counts))\nprint(len(tokenizer.word_index))\n\ntimeSince(t0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f46bc0f491d585a4f4ffcfd633d63f8797db6973"},"cell_type":"markdown","source":"## Start procedure"},{"metadata":{"_uuid":"26799562a417146b69dcc66434163926f6be1526"},"cell_type":"markdown","source":"Create helper functions for loading embeddings & creating embedding matrices"},{"metadata":{"trusted":false,"_uuid":"08284c1ec250440153e91c43e1fde3f680b4d22f"},"cell_type":"code","source":"t0 = time.time()\n\ndef load_embed(file):\n    ''' Load the embedding from file '''\n    def get_coefs(word, *arr):\n        return word, np.asarray(arr, dtype='float32')\n    \n    if file.split('/')[-1] == 'wiki-news-300d-1M.vec':\n        embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(file) if len(o) > 100)\n    else:\n        embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(file, encoding='latin'))\n        \n    return embeddings_index\n\ndef create_emb_matrix(emb_mean, emb_std, nb_words, embed_size):\n    ''' Creates a initial random embedding matrix \n        All words that are not found in the embedding will thus be a random vector\n        Can maybe be imrpoved by initializing all words not found to the same random vector '''\n    return np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n\ndef fill_emb_matrix(word_idx, emb_matrix, emb_index, vocab_size):\n    ''' Created a word2vec format matrix that we can use to embed our words '''\n    for word, i in word_idx:\n        if i >= vocab_size:\n            return emb_matrix\n        emb_vector = emb_index.get(word)\n        if emb_vector is not None:\n            emb_matrix[i] = emb_vector\n    return emb_matrix\n\ntimeSince(t0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1186d66e830bf8898b53610ec177cdaef6f338de"},"cell_type":"markdown","source":"Define the function that I've been talking about!"},{"metadata":{"trusted":false,"_uuid":"165dc5902456792a05508669cf8ee39b3fbd3f35"},"cell_type":"code","source":"t0 = time.time()\n\ndef train_val_pred(dataset, hparam, embedding_matrix):\n    ''' This function will train a model using some embedding matrix.\n        The prediction threshold will then be calculated based on the threshold that gives\n        the best f1 score on the validation data. The predictions on the test set will then\n        be returned along with the best calculated f1 score '''\n    \n    # Get data from dataset\n    X_train = dataset['X_train']\n    y_train = dataset['y_train']\n    X_val = dataset['X_val']\n    y_val = dataset['y_val']\n    X_test = dataset['X_test']\n\n    # Get hyperparameters\n    VOCAB_SIZE = hparam['VOCAB_SIZE']\n    PAD_LENGTH = hparam['PAD_LENGTH']\n    MINIBATCH_SIZE = hparam['MINIBATCH_SIZE']\n    LEARNING_RATE = hparam['LEARNING_RATE']\n    EPOCHS = hparam['EPOCHS']\n    LSTM_HIDDEN_SIZE = hparam['LSTM_HIDDEN_SIZE']\n    WORD_EMB_DIM = hparam['WORD_EMB_DIM']\n    \n    # Create the model\n    inp = Input(shape=(PAD_LENGTH,))\n    x = Embedding(VOCAB_SIZE, WORD_EMB_DIM, weights=[embedding_matrix], trainable=True)(inp)\n    x = Bidirectional(CuDNNLSTM(LSTM_HIDDEN_SIZE, return_sequences=True))(x)\n    x = GlobalMaxPool1D()(x)\n    x = Dense(16, activation=\"relu\")(x)\n    x = Dropout(0.1)(x)\n    x = Dense(1, activation=\"sigmoid\")(x)\n    model = Model(inputs=inp, outputs=x)\n    adam = optimizers.Adam(lr=LEARNING_RATE)\n    model.compile(loss='binary_crossentropy', optimizer=adam, metrics=['accuracy'])\n    model.summary()\n    \n    # Train model\n    model.fit(X_train, y_train, epochs=EPOCHS, batch_size=MINIBATCH_SIZE, \n          validation_data = (X_val, y_val))\n    \n    # Get threshold that gives best f1 score on validation set\n    val_preds = model.predict(X_val, batch_size=MINIBATCH_SIZE, verbose=1)\n    best_f1 = -1\n    best_thresh = -1\n    for thresh in np.arange(0.1, 0.501, 0.01):\n        thresh = np.round(thresh, 2)\n        f1 = metrics.f1_score(y_val, (val_preds > thresh).astype(int))\n        if f1 > best_f1:\n            best_f1 = f1\n            best_thresh = thresh\n    print(\"Best f1 score = {} at tresh {}\".format(best_f1, best_thresh))\n    \n    # Get predictions on test set\n    test_preds = model.predict(X_test)\n    \n    # Some memory management!\n    del embedding_matrix, model, inp, x, adam\n    import gc; gc.collect()\n    \n    # Return predictions and the thresh that gave best f1 score on validation data\n    return test_preds, val_preds, best_thresh, best_f1\n    \ntimeSince(t0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"67a230353f5f865b603dd40b03f28b24aef0ed75"},"cell_type":"markdown","source":"Now we iterate over the embeddings and run the above function!"},{"metadata":{"trusted":false,"_uuid":"836bacaa097dc4a13720856b8f273a2da991f18e"},"cell_type":"code","source":"t0 = time.time()\n\n# Non-iterable vars\nword_index = tokenizer.word_index\nnb_words = min(hparam['VOCAB_SIZE'], len(word_index))\n\ntimeSince(t0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"0318945fb4792ef2bf95f45914cc0dbb6d6fe407"},"cell_type":"code","source":"t0 = time.time()\n\n# Define some list where we save the results!\nresults = []\n    \nfor embedding in embeddings:\n    emb_name = embedding['name']\n    emb_path = embedding['path']\n    print(\"Running procedure on {}\".format(emb_name))\n    \n    # Load embedding\n    emb_index = load_embed(emb_path)\n    all_emb = np.stack(list(emb_index.values()))\n    emb_mean, emb_std = all_emb.mean(), all_emb.std()\n    emb_size = all_emb.shape[1]\n    hparam['WORD_EMB_DIM'] = emb_size # Set this! Not really needed tho...\n    print(\"{} mean: {}, std: {}, size: {}\".format(emb_name, emb_mean, emb_std, emb_size))\n    \n    # Convert emb to word2vec format\n    emb_matrix = create_emb_matrix(emb_mean, emb_std, nb_words, emb_size)\n    emb_matrix = fill_emb_matrix(word_index.items(), emb_matrix, emb_index, hparam['VOCAB_SIZE'])\n    \n    # Run entire procedure and get predictions, best threshold and best f1 score\n    test_preds, val_preds, thresh, f1 = train_val_pred(dataset, hparam, emb_matrix)\n    \n    print(\"len(test_preds) = {}, len(val_preds) = {}, thresh = {} at f1 = {}\".format(len(test_preds), \n                                                                                     len(val_preds), \n                                                                                     thresh, \n                                                                                     f1))\n    # Save into results\n    new_result = {'name': emb_name, \n                  'test_preds': test_preds, \n                  'val_preds': val_preds, \n                  'thresh': thresh, \n                  'f1': f1}\n    results.append(new_result)\n    \n    # Memory management!\n    del emb_index, all_emb, emb_mean, emb_std, emb_size, emb_matrix\n    import gc; gc.collect()\n    time.sleep(10)\n\ntimeSince(t0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dbf1ea919474064b4480c97bdffd708f343f6abe"},"cell_type":"markdown","source":"Get som statistics about our results"},{"metadata":{"trusted":false,"_uuid":"502ef8e83d2e26fdc4ecda1296b9dc35852a1e75"},"cell_type":"code","source":"t0 = time.time()\n\nprint(\"Got {} number of results!\".format(len(results)))\navg_thresh = 0\nfor result in results:\n    print(\"{} gave f1 score {} with thresh {}\".format(result['name'], result['f1'], result['thresh']))\n    avg_thresh += result['thresh']\n\navg_thresh = avg_thresh / len(results)\nprint(\"Got an average threshold at {}\".format(avg_thresh))\n\ntimeSince(t0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"64076f5a44b5f5f9a0c000ce03d5b8bc987bc701"},"cell_type":"markdown","source":"Some testing"},{"metadata":{"trusted":false,"_uuid":"ae0a2e25c9a34e057c4ea77fed6df1aacc6ea5e1"},"cell_type":"code","source":"t0 = time.time()\n\npred_glove_val = results[0]['val_preds']\npreds_paragram_val = results[1]['val_preds']\npreds_fasttext_val = results[2]['val_preds']\n\npred_val_y = 0.33*pred_glove_val + 0.33*preds_paragram_val + 0.34*preds_fasttext_val \n\nbest_f1_combined = -1\nbest_thresh_combined = -1\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    f1 = metrics.f1_score(dataset['y_val'], (pred_val_y > thresh).astype(int))\n    if f1 > best_f1_combined:\n        best_f1_combined = f1\n        best_thresh_combined = thresh\n    #print(\"F1 score at threshold {0} is {1}\".format(thresh, f1))\n    \nprint(\"Best f1 score = {} at tresh {}\".format(best_f1_combined, best_thresh_combined))\n\ntimeSince(t0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8cd95d802861cf512bf91466b693f7ba21e1b185"},"cell_type":"code","source":"t0 = time.time()\n\nprint(\"Using treshold {}\".format(best_thresh_combined))\n\npred_glove_test = results[0]['test_preds']\npreds_paragram_test = results[1]['test_preds']\npreds_fasttext_test = results[2]['test_preds']\n\npred_test_y = 0.33*pred_glove_test + 0.33*preds_paragram_test + 0.34*preds_fasttext_test\npred_test_y_res = (pred_test_y > best_thresh_combined).astype(int)\n\nresults_dict = {'qid':qid_test, 'prediction':[]}\n\nfor prediction in pred_test_y_res:\n    results_dict['prediction'].append(prediction[0])\n\n    \nprint(results_dict['qid'][:15])\nprint(results_dict['prediction'][:15])\n\n    \n# Save results\ndf = pd.DataFrame(data=results_dict)\ndf.to_csv('submission.csv', index=False)\nprint(\"Saved csv to disk!\")\n\ntimeSince(t0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"20c789b1e133058e2bd2f353305dc839705325e7"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}