{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"Have come across a lot of interesting blended models in the public kernels. Most of these models have used a single train-val split. I guess because of the 2-hour time limit, people have been focusing on blending multiple models together and not on K-fold cross validation which is expensive. I wanted to explore how much additional juice can be extracted by using a K-fold vcross-validation on a single simple model. K = 5 in this example.\n\nAlso I want to acknowledge[ Shujian Liu](https://www.kaggle.com/shujian),  [SRK](https://www.kaggle.com/sudalairajkumar/), [Dieter](https://www.kaggle.com/christofhenkel) and [Khoi Nguyen ](https://www.kaggle.com/suicaokhoailang) for their excellent public kernels. They have been inspiring and motivating, and extremely useful.\n\nThe model is based on SRK's original public kernel, with my own minor modifications  : \nhttps://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings\n\n\n    input = Input(shape=(max_len,))\n        embed = Embedding(max_words, embed_size, weights=[embedding_matrix])(input)\n    \n    x = Bidirectional(CuDNNLSTM(64, return_sequences=True))(embed)\n    x = Bidirectional(CuDNNGRU(64, return_sequences=True))(x)\n    x = GlobalMaxPool1D()(x)\n    x = Dense(16, activation=\"relu\")(x)\n    x = Dropout(0.1)(x)\n    \n    y = Bidirectional(CuDNNGRU(64, return_sequences=True))(embed)\n    y = Bidirectional(CuDNNLSTM(64, return_sequences=True))(y)\n    y = GlobalMaxPool1D()(y)\n    y = Dense(16, activation=\"relu\")(y)\n    y = Dropout(0.1)(y)\n    \n    z= Concatenate()([x,y])\n    \n    output = Dense(1, activation=\"sigmoid\")(z)"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import os\nimport time\nimport numpy as np \nimport pandas as pd\nimport gc\nimport tensorflow as tf\nimport pickle\nimport glob\n\nnp.random.seed(7418880)\ntf.set_random_seed(7418880)\nfrom collections import defaultdict\n\nfrom sklearn import metrics\nfrom sklearn.metrics import f1_score\nfrom sklearn.model_selection import  StratifiedKFold\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.models import Model\nfrom keras.layers import Bidirectional, Embedding, GlobalMaxPool1D, Input\nfrom keras.layers import CuDNNLSTM, CuDNNGRU, Concatenate, Dense,  Dropout\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1dd6cd665c9bd8a9968493f1e4499b7cf7ede4e2"},"cell_type":"markdown","source":"Let us load the train and test sets. Set max_words to be 50000 and max_len to be 70 "},{"metadata":{"trusted":true,"_uuid":"8b9a0c9a08f4e0d9c7bb5526480ad9356fbef643"},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\")\ntest_df = pd.read_csv(\"../input/test.csv\")\nprint(\"Train shape : \", train_df.shape)\nprint(\"Test_shape : \", test_df.shape)\n\ntrain_X = train_df[\"question_text\"].fillna(\"_na_\").values\ntest_X = test_df[\"question_text\"].fillna(\"_na_\").values\ntrain_y = train_df['target'].values\n\nembed_size = 300\nmax_words = 50000 # number of unique words\nmax_len = 70\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6ab2cfa1c59acdc117881bb3e17d9e48c3883d94"},"cell_type":"markdown","source":"Tokenize the sentences"},{"metadata":{"trusted":true,"_uuid":"a9773cd344bc1945b1b0fa6db98bbeb4d4c48df1"},"cell_type":"code","source":"start_time = time.time()\n\ntokenizer = Tokenizer(num_words=max_words)\ntokenizer.fit_on_texts(list(train_X))\ntrain_X=tokenizer.texts_to_sequences(train_X)\ntest_X=tokenizer.texts_to_sequences(test_X)\n\ntrain_X = pad_sequences(train_X, maxlen=max_len)\ntest_X = pad_sequences(test_X, maxlen=max_len)\n\nprint (train_X.shape, test_X.shape)\nprint(\"Total time for tokenizing = {:.0f} s\".format(time.time()-start_time))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"67d1223852ef53a808318d6fdce6a68460b827c2"},"cell_type":"markdown","source":"Tokenizing takes about 65 s on average in my experience. Interesting to know these numbers since we have a 7200 s time limit for these competitiom"},{"metadata":{"trusted":true,"_uuid":"60c4978eed01a4f59c742176bb68cdefe3b56a8c"},"cell_type":"code","source":"start_time = time.time()\n\nEMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\ndef get_coefs(word, *arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.rstrip().rsplit(' ')) for o in open(EMBEDDING_FILE))\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_words, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= max_words: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n\nprint(\"Total time for embedding = {:.0f} s\".format(time.time()-start_time))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"006b59ed9e4208a61d47399baec3eec7707e3b76"},"cell_type":"markdown","source":"Embedding the glove vectors takes > 3 mins on average. \nLet us define the model now. Nothing fancy. A GRU/LSTM bilayer concated with a LSTM/GRU layer (don't ask me why) with a few pooling, dense and dropout layers thrown in to the mix. Based on SRK's kernel."},{"metadata":{"trusted":true,"_uuid":"1e6e577224d52fce3361261c5786ff49c7c181dc"},"cell_type":"code","source":"def generate_model ():\n    input = Input(shape=(max_len,))\n    embed = Embedding(max_words, embed_size, weights=[embedding_matrix])(input)\n    \n    x = Bidirectional(CuDNNLSTM(64, return_sequences=True))(embed)\n    x = Bidirectional(CuDNNGRU(64, return_sequences=True))(x)\n    x = GlobalMaxPool1D()(x)\n    x = Dense(16, activation=\"relu\")(x)\n    x = Dropout(0.1)(x)\n    \n    y = Bidirectional(CuDNNGRU(64, return_sequences=True))(embed)\n    y = Bidirectional(CuDNNLSTM(64, return_sequences=True))(y)\n    y = GlobalMaxPool1D()(y)\n    y = Dense(16, activation=\"relu\")(y)\n    y = Dropout(0.1)(y)\n    \n    z= Concatenate()([x,y])\n    \n    output = Dense(1, activation=\"sigmoid\")(z)\n    \n    model = Model (inputs=input, outputs=output)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"da5891a89af13bd57c4749409e8b386fed535913"},"cell_type":"markdown","source":"Let us train this model using 5-fold stratified validation. Through trial and error,  running 2 epochs  seemed like a reasonable choice for this model, to prevent overfitting."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"0e55a607f399f193fbd4e578aba398f2d22a0899"},"cell_type":"code","source":"start_time = time.time()\n\nuid = 1\nversion =1 \nn_splits = 5\nn_epochs =2\nbatch_size =1024\n\nskf = StratifiedKFold(n_splits=n_splits, random_state = 7418880, shuffle=False)\nval_preds = defaultdict(list)\ntest_preds = {}\nhistoryD={}\nfor ii, (train_index, val_index)  in enumerate(skf.split(train_X, train_y)):\n    \n    X_train, X_val = train_X[train_index], train_X[val_index]\n    y_train, y_val = train_y[train_index], train_y[val_index]\n    \n    model = generate_model()\n    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n    hist = model.fit(X_train, y_train, batch_size=batch_size, epochs=n_epochs, \n              validation_data=(X_val, y_val), verbose= True)\n    \n    historyD[\"fold{}\".format(ii+1)] = hist.history\n    \n    pred_val_y = model.predict([X_val], batch_size=batch_size, verbose=1)\n    val_preds['fold{}'.format(ii+1)] = [pred_val_y.ravel(),y_val]\n    \n    pred_test_y = model.predict([test_X], batch_size=batch_size, verbose=1)\n    test_preds['fold{}'.format(ii+1)] =  pred_test_y.ravel()\n    \n    del model\n    gc.collect()\n    \nwith open('train_history_uid{}_v{}.pkl'.format(uid, version),'wb') as pklfile:\n    pickle.dump(historyD,pklfile)\nwith open('val_preds_uid{}_v{}.pkl'.format(uid, version),'wb') as pklfile:\n    pickle.dump(val_preds,pklfile)\nwith open('test_preds_uid{}_v{}.pkl'.format(uid, version),'wb') as pklfile:\n    pickle.dump(test_preds,pklfile)\nprint(\"Total time for training = {:.0f} s\".format(time.time()-start_time))    \n    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cbe818a12907f5b7d53b6188a9f9d9530bb0c9fe"},"cell_type":"markdown","source":"So total training time is ~ 65 minutes.\n\nFor each fold, a validation prediction and a test prediction are made. The threshold value to use for each of these test predictions is  obtained by a threshold scan  that yields the maximum  F1 score on the validation set. Basically what others have been using. Ad-hoc, but works.\n\nIn separate kernels, I submitted the test prediction from each of the 5 folds. The public LB scores I got were \n0.662, 0.663, 0.664, 0.665, 0.665. An average of ~0.664. Now, how does the average prediction from these five folds perform?"},{"metadata":{"trusted":true,"_uuid":"7c1822c885d1bcb5ba54c361951a85bcf66a0229"},"cell_type":"code","source":"test01_df = pd.DataFrame()\n#print(val_preds['fold6'])\nfor ii in range(len(val_preds)):\n    threshL = []\n    y_preds, y_actual = (val_preds['fold{}'.format(ii+1)])\n    for idx, thresh in enumerate(np.arange(0.1, 0.51, 0.01)):\n        thresh = np.round(thresh,2)\n        y_01 = [ 0 if x <thresh else 1 for x in y_preds]\n        threshL.append((metrics.f1_score(y_actual,y_01), thresh))\n    threshL=sorted(threshL)\n    best_F1, opt_thresh = threshL[-1]\n    print (\"For fold {0}, best validation F1 of {1:.5f} at threshold {2:.2f}\".format(ii+1, best_F1,opt_thresh))\n    test01_df['fold{}'.format(ii)] = [ 0 if x<opt_thresh else 1 for x in test_preds['fold{}'.format(ii+1)]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3d0e85b1a605f0fc34d4f3fec9577a4831d985db"},"cell_type":"code","source":"print(test01_df.sum(axis=1).value_counts())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d697500f4d7c646b3f25b2c6630cc6fb1a6ffb82"},"cell_type":"markdown","source":"So,  the models disagree on (594+470+354+354) = 1772 samples . \n\nA pearson corelation coefficient between the predictions is also an useful number to know to see how correlated the differnt folds are. "},{"metadata":{"trusted":true,"_uuid":"080ba13fa75940186813d9d059efa5ca46920bd6"},"cell_type":"code","source":" test01_df.corr(method='pearson')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aebfead9db4ac9137cbf64fbad6b18ead61331a8"},"cell_type":"markdown","source":"We get a correlation coefficient of ~ 0.88 between prediction from the various folds. Heavily correlated, but still some variance.  So averaging these slightly different predictions should give us a better score than each of the  individual predictions.\n\n Let us create the submission file:"},{"metadata":{"trusted":true,"_uuid":"72c1266605e03cf22a31bf38d9f78005dd13c483"},"cell_type":"code","source":"out_df = pd.DataFrame({\"qid\":test_df[\"qid\"].values})\nout_df['prediction'] = [ 0 if x<2.5 else 1 for x in test01_df.sum(axis=1)]\nout_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d9ab103b20e5f6832820f37248eade14bd562b2c"},"cell_type":"markdown","source":"On submission, this should yields an LB score of 0.671or thereabouts, which is a reasonable improvement of 0.007 or so. One should expect a similar improvement on other \"single models\" while using K-fold validation. Hope this kernel was useful and gives an idea of how to balance K-fold validation of a single model  vs. adding additional models to the soup."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}