{"cells":[{"metadata":{"_uuid":"c570a4c3a99ef5ac8c7e33f3ffebc598eeb073ab"},"cell_type":"markdown","source":"In this kernel I tried to stack some of the high-performing single models and validate on a holdout set. I have been having trouble to  reach an agreement between local cv score and lb score. \n\nSome conclusions I believe its true:\n1. Do not trust the cv score obtained through searching an optimal threshold, because this process of validating includes data leakage.\n2. The random seed seems to have at least some significance.\n\nSome remaining problems:\n1. Validaing with a holdout set is not working in this kernel. I thought this was because the distribution of labels in test data and train data was significantly different, but that did not seem true either. So what is the exact problem?\n"},{"metadata":{"_uuid":"693859c710488a003fb927654efe653d3a8f8a06"},"cell_type":"markdown","source":"some reference:\n\nhttps://www.kaggle.com/shujian/single-rnn-with-4-folds-clr \n\nhttps://www.kaggle.com/bminixhofer/a-validation-framework-impact-of-the-random-seed/notebook\n\nhttps://www.kaggle.com/gmhost/gru-capsule\n\n\nTell me if missed any"},{"metadata":{"trusted":true,"_uuid":"b378958a9606ac48fe0dc54e24bed4cd503e0ac7","_kg_hide-output":false,"_kg_hide-input":false},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom tqdm import tqdm\ntqdm.pandas()\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\nfrom sklearn import metrics\nfrom sklearn.metrics import f1_score \n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import *\nfrom keras.models import *\nfrom keras import initializers, regularizers, constraints, optimizers, layers\nfrom keras.initializers import *\nfrom keras.optimizers import *\nimport keras.backend as K\nfrom keras.callbacks import *\nimport tensorflow as tf\nimport os\nimport time\nimport gc\nimport re\nfrom unidecode import unidecode","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")\nprint(\"Train shape : \", train.shape)\nprint(\"Test shape : \", test.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"565dc7106c788db0b971c8c01c2cf09c2740e985"},"cell_type":"markdown","source":"**Text Preprocessing and Embedding**"},{"metadata":{"trusted":true,"_uuid":"6d336bfef799c16f12f2d02ffa4f3c2eaaf6ef34"},"cell_type":"code","source":"train[\"question_text\"] = train[\"question_text\"].str.lower()\ntest[\"question_text\"] = test[\"question_text\"].str.lower()\n\npuncts = [',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', ';', \"'\", '$', '&', '/', '[', ']', '>', '%', '=', '#', '*', '+', '\\\\', '•',  '~', '@', '£', \n '·', '_', '{', '}', '©', '^', '®', '`',  '<', '→', '°', '€', '™', '›',  '♥', '←', '×', '§', '″', '′', 'Â', '█', '½', 'à', '…', \n '“', '★', '”', '–', '●', 'â', '►', '−', '¢', '²', '¬', '░', '¶', '↑', '±', '¿', '▾', '═', '¦', '║', '―', '¥', '▓', '—', '‹', '─', \n '▒', '：', '¼', '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', '▄', '♫', '☆', 'é', '¯', '♦', '¤', '▲', 'è', '¸', '¾', 'Ã', '⋅', '‘', '∞', \n '∙', '）', '↓', '、', '│', '（', '»', '，', '♪', '╩', '╚', '³', '・', '╦', '╣', '╔', '╗', '▬', '❤', 'ï', 'Ø', '¹', '≤', '‡', '√', ]\ndef clean_text(x):\n\n    x = str(x)\n    for punct in puncts:\n        x = x.replace(punct, f' {punct} ')\n    return x\n\n\ntrain[\"question_text\"] = train[\"question_text\"].apply(lambda x: clean_text(x))\ntest[\"question_text\"] = test[\"question_text\"].apply(lambda x: clean_text(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"87b0cf1880df72c47d8a882e6441aaa07dacfd9c"},"cell_type":"code","source":"## some config values \nembed_size = 300 # how big is each word vector\nmax_features = None # how many unique words to use (i.e num rows in embedding vector)\nmaxlen = 72 # max number of words in a question to use #99.99%\n\n## fill up the missing values\nX = train[\"question_text\"].fillna(\"_na_\").values\nX_test = test[\"question_text\"].fillna(\"_na_\").values\n\n## Tokenize the sentences\ntokenizer = Tokenizer(num_words=max_features, filters='')\ntokenizer.fit_on_texts(list(X))\n\nX = tokenizer.texts_to_sequences(X)\nX_test = tokenizer.texts_to_sequences(X_test)\n\n## Pad the sentences \nX = pad_sequences(X, maxlen=maxlen)\nX_test = pad_sequences(X_test, maxlen=maxlen)\n\n## Get the target values\nY = train['target'].values\n\nsub = test[['qid']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"259d359c2fd45efd7ccd1a18db69fbb7fe7ad8d2"},"cell_type":"code","source":"del train, test\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"26e110fe2c74167bf02fda50e7cd9cbb897dca57"},"cell_type":"code","source":"word_index = tokenizer.word_index\nmax_features = len(word_index)+1\ndef load_glove(word_index):\n    EMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\n    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE) if o.split(\" \")[0] in word_index)\n\n    all_embs = np.stack(embeddings_index.values())\n    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n    embed_size = all_embs.shape[1]\n\n    embedding_matrix = np.random.normal(emb_mean, emb_std, (max_features, embed_size))\n    for word, i in word_index.items():\n        if i >= max_features: continue\n        embedding_vector = embeddings_index.get(word)\n        if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n            \n    return embedding_matrix \n    \ndef load_fasttext(word_index):    \n    EMBEDDING_FILE = '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec'\n    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE) if len(o)>100 and o.split(\" \")[0] in word_index )\n\n    all_embs = np.stack(embeddings_index.values())\n    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n    embed_size = all_embs.shape[1]\n\n    embedding_matrix = np.random.normal(emb_mean, emb_std, (max_features, embed_size))\n    for word, i in word_index.items():\n        if i >= max_features: continue\n        embedding_vector = embeddings_index.get(word)\n        if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n\n    return embedding_matrix\n\ndef load_para(word_index):\n    EMBEDDING_FILE = '../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt'\n    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE, encoding=\"utf8\", errors='ignore') if len(o)>100 and o.split(\" \")[0] in word_index)\n\n    all_embs = np.stack(embeddings_index.values())\n    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n    embed_size = all_embs.shape[1]\n    \n    embedding_matrix = np.random.normal(emb_mean, emb_std, (max_features, embed_size))\n    for word, i in word_index.items():\n        if i >= max_features: continue\n        embedding_vector = embeddings_index.get(word)\n        if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n    \n    return embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2505794d8ebc30884ccc6a9a9a17a3a1af12bd29"},"cell_type":"code","source":"embedding_matrix_1 = load_glove(word_index)\n#embedding_matrix_2 = load_fasttext(word_index)\nembedding_matrix_3 = load_para(word_index)\nembedding_matrix = np.mean((embedding_matrix_1, embedding_matrix_3), axis=0)  \n#del embedding_matrix_1, embedding_matrix_3\n#gc.collect()\nnp.shape(embedding_matrix)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6c667e241e4e028db3190f41f31df8c3cbfafd65"},"cell_type":"code","source":"def f1(y_true, y_pred):\n    def recall(y_true, y_pred):\n        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n        recall = true_positives / (possible_positives + K.epsilon())\n        return recall\n\n    def precision(y_true, y_pred):\n        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n        precision = true_positives / (predicted_positives + K.epsilon())\n        return precision\n    precision = precision(y_true, y_pred)\n    recall = recall(y_true, y_pred)\n    return 2*((precision*recall)/(precision+recall+K.epsilon()))\n\ndef f1_smart(y_true, y_pred):\n    args = np.argsort(y_pred)\n    tp = y_true.sum()\n    fs = (tp - np.cumsum(y_true[args[:-1]])) / np.arange(y_true.shape[0] + tp - 1, tp, -1)\n    res_idx = np.argmax(fs)\n    return 2 * fs[res_idx], (y_pred[args[res_idx]] + y_pred[args[res_idx + 1]]) / 2","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e14fe407ed4cb95e3b5484df1caafd59f8f51345"},"cell_type":"markdown","source":"# Defining the models"},{"metadata":{"_uuid":"07b177cc8ed96bded4cfe434ab0243ebdb0c9644"},"cell_type":"markdown","source":"The 4 stacking models I used are:  GRU+Capsule with the mean of embeddings, RNN with attention layer, Conv1D with 400 features and window size 4, GRU+Capsule with only Glove"},{"metadata":{"_uuid":"3f8eb6a922db4c6936750ed356668f2f07cc7db4"},"cell_type":"markdown","source":"**GRU + Capsule Model**"},{"metadata":{"_uuid":"b75ab7890f8d51c008b47d994011f88a35de36a7","trusted":true,"_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"def squash(x, axis=-1):\n    # s_squared_norm is really small\n    # s_squared_norm = K.sum(K.square(x), axis, keepdims=True) + K.epsilon()\n    # scale = K.sqrt(s_squared_norm)/ (0.5 + s_squared_norm)\n    # return scale * x\n    s_squared_norm = K.sum(K.square(x), axis, keepdims=True)\n    scale = K.sqrt(s_squared_norm + K.epsilon())\n    return x / scale\n\n# A Capsule Implement with Pure Keras\nclass Capsule(Layer):\n    def __init__(self, num_capsule, dim_capsule, routings=3, kernel_size=(9, 1), share_weights=True,\n                 activation='default', **kwargs):\n        super(Capsule, self).__init__(**kwargs)\n        self.num_capsule = num_capsule\n        self.dim_capsule = dim_capsule\n        self.routings = routings\n        self.kernel_size = kernel_size\n        self.share_weights = share_weights\n        if activation == 'default':\n            self.activation = squash\n        else:\n            self.activation = Activation(activation)\n\n    def build(self, input_shape):\n        super(Capsule, self).build(input_shape)\n        input_dim_capsule = input_shape[-1]\n        if self.share_weights:\n            self.W = self.add_weight(name='capsule_kernel',\n                                     shape=(1, input_dim_capsule,\n                                            self.num_capsule * self.dim_capsule),\n                                     # shape=self.kernel_size,\n                                     initializer='glorot_uniform',\n                                     trainable=True)\n        else:\n            input_num_capsule = input_shape[-2]\n            self.W = self.add_weight(name='capsule_kernel',\n                                     shape=(input_num_capsule,\n                                            input_dim_capsule,\n                                            self.num_capsule * self.dim_capsule),\n                                     initializer='glorot_uniform',\n                                     trainable=True)\n\n    def call(self, u_vecs):\n        if self.share_weights:\n            u_hat_vecs = K.conv1d(u_vecs, self.W)\n        else:\n            u_hat_vecs = K.local_conv1d(u_vecs, self.W, [1], [1])\n\n        batch_size = K.shape(u_vecs)[0]\n        input_num_capsule = K.shape(u_vecs)[1]\n        u_hat_vecs = K.reshape(u_hat_vecs, (batch_size, input_num_capsule,\n                                            self.num_capsule, self.dim_capsule))\n        u_hat_vecs = K.permute_dimensions(u_hat_vecs, (0, 2, 1, 3))\n        # final u_hat_vecs.shape = [None, num_capsule, input_num_capsule, dim_capsule]\n\n        b = K.zeros_like(u_hat_vecs[:, :, :, 0])  # shape = [None, num_capsule, input_num_capsule]\n        for i in range(self.routings):\n            b = K.permute_dimensions(b, (0, 2, 1))  # shape = [None, input_num_capsule, num_capsule]\n            c = K.softmax(b)\n            c = K.permute_dimensions(c, (0, 2, 1))\n            b = K.permute_dimensions(b, (0, 2, 1))\n            outputs = self.activation(tf.keras.backend.batch_dot(c, u_hat_vecs, [2, 2]))\n            if i < self.routings - 1:\n                b = tf.keras.backend.batch_dot(outputs, u_hat_vecs, [2, 3])\n\n        return outputs\n\n    def compute_output_shape(self, input_shape):\n        return (None, self.num_capsule, self.dim_capsule)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6d9ba67784a576692b6f38c931e3c012eff539d7"},"cell_type":"code","source":"def create_capsule(embedding_matrix):\n    K.clear_session()       \n    inp = Input(shape=(maxlen,))\n    x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)\n    x = SpatialDropout1D(rate=0.2)(x)\n    x = Bidirectional(CuDNNGRU(100, return_sequences=True, \n                                kernel_initializer=glorot_normal(seed=12300), recurrent_initializer=orthogonal(gain=1.0, seed=10000)))(x)\n\n    x = Capsule(num_capsule=10, dim_capsule=10, routings=4, share_weights=True)(x)\n    x = Flatten()(x)\n\n    x = Dense(100, activation=\"relu\", kernel_initializer=glorot_normal(seed=12300))(x)\n    x = Dropout(0.5)(x)\n    x = BatchNormalization()(x)\n\n    x = Dense(1, activation=\"sigmoid\")(x)\n    model = Model(inputs=inp, outputs=x)\n    model.compile(loss='binary_crossentropy', optimizer=Adam(),)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"67bcaa08f3f8344f0dd8a105cfe40538c6132b4e"},"cell_type":"markdown","source":"**RNN + attention layer + clr**"},{"metadata":{"trusted":true,"_uuid":"f98c6bea60e8a656ae7e96c1cfd48b8e75dede21"},"cell_type":"code","source":"class CyclicLR(Callback):\n    def __init__(self, base_lr=0.001, max_lr=0.006, step_size=2000., mode='triangular',\n                 gamma=1., scale_fn=None, scale_mode='cycle'):\n        super(CyclicLR, self).__init__()\n\n        self.base_lr = base_lr\n        self.max_lr = max_lr\n        self.step_size = step_size\n        self.mode = mode\n        self.gamma = gamma\n        if scale_fn == None:\n            if self.mode == 'triangular':\n                self.scale_fn = lambda x: 1.\n                self.scale_mode = 'cycle'\n            elif self.mode == 'triangular2':\n                self.scale_fn = lambda x: 1/(2.**(x-1))\n                self.scale_mode = 'cycle'\n            elif self.mode == 'exp_range':\n                self.scale_fn = lambda x: gamma**(x)\n                self.scale_mode = 'iterations'\n        else:\n            self.scale_fn = scale_fn\n            self.scale_mode = scale_mode\n        self.clr_iterations = 0.\n        self.trn_iterations = 0.\n        self.history = {}\n\n        self._reset()\n\n    def _reset(self, new_base_lr=None, new_max_lr=None,\n               new_step_size=None):\n        \"\"\"Resets cycle iterations.\n        Optional boundary/step size adjustment.\n        \"\"\"\n        if new_base_lr != None:\n            self.base_lr = new_base_lr\n        if new_max_lr != None:\n            self.max_lr = new_max_lr\n        if new_step_size != None:\n            self.step_size = new_step_size\n        self.clr_iterations = 0.\n        \n    def clr(self):\n        cycle = np.floor(1+self.clr_iterations/(2*self.step_size))\n        x = np.abs(self.clr_iterations/self.step_size - 2*cycle + 1)\n        if self.scale_mode == 'cycle':\n            return self.base_lr + (self.max_lr-self.base_lr)*np.maximum(0, (1-x))*self.scale_fn(cycle)\n        else:\n            return self.base_lr + (self.max_lr-self.base_lr)*np.maximum(0, (1-x))*self.scale_fn(self.clr_iterations)\n        \n    def on_train_begin(self, logs={}):\n        logs = logs or {}\n\n        if self.clr_iterations == 0:\n            K.set_value(self.model.optimizer.lr, self.base_lr)\n        else:\n            K.set_value(self.model.optimizer.lr, self.clr())        \n            \n    def on_batch_end(self, epoch, logs=None):\n        \n        logs = logs or {}\n        self.trn_iterations += 1\n        self.clr_iterations += 1\n\n        self.history.setdefault('lr', []).append(K.get_value(self.model.optimizer.lr))\n        self.history.setdefault('iterations', []).append(self.trn_iterations)\n\n        for k, v in logs.items():\n            self.history.setdefault(k, []).append(v)\n        \n        K.set_value(self.model.optimizer.lr, self.clr())\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"336ee310bec7a801615471d6e2adbb2161494cb8"},"cell_type":"code","source":"class Attention(Layer):\n    def __init__(self, step_dim,\n                 W_regularizer=None, b_regularizer=None,\n                 W_constraint=None, b_constraint=None,\n                 bias=True, **kwargs):\n        self.supports_masking = True\n        self.init = initializers.get('glorot_uniform')\n\n        self.W_regularizer = regularizers.get(W_regularizer)\n        self.b_regularizer = regularizers.get(b_regularizer)\n\n        self.W_constraint = constraints.get(W_constraint)\n        self.b_constraint = constraints.get(b_constraint)\n\n        self.bias = bias\n        self.step_dim = step_dim\n        self.features_dim = 0\n        super(Attention, self).__init__(**kwargs)\n\n    def build(self, input_shape):\n        assert len(input_shape) == 3\n\n        self.W = self.add_weight((input_shape[-1],),\n                                 initializer=self.init,\n                                 name='{}_W'.format(self.name),\n                                 regularizer=self.W_regularizer,\n                                 constraint=self.W_constraint)\n        self.features_dim = input_shape[-1]\n\n        if self.bias:\n            self.b = self.add_weight((input_shape[1],),\n                                     initializer='zero',\n                                     name='{}_b'.format(self.name),\n                                     regularizer=self.b_regularizer,\n                                     constraint=self.b_constraint)\n        else:\n            self.b = None\n\n        self.built = True\n\n    def compute_mask(self, input, input_mask=None):\n        return None\n\n    def call(self, x, mask=None):\n        features_dim = self.features_dim\n        step_dim = self.step_dim\n\n        eij = K.reshape(K.dot(K.reshape(x, (-1, features_dim)),\n                        K.reshape(self.W, (features_dim, 1))), (-1, step_dim))\n\n        if self.bias:\n            eij += self.b\n\n        eij = K.tanh(eij)\n\n        a = K.exp(eij)\n\n        if mask is not None:\n            a *= K.cast(mask, K.floatx())\n\n        a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n\n        a = K.expand_dims(a)\n        weighted_input = x * a\n        return K.sum(weighted_input, axis=1)\n\n    def compute_output_shape(self, input_shape):\n        return input_shape[0],  self.features_dim","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"702d25aa78bdd3ed2db4567205295d77a0b9d696"},"cell_type":"code","source":"def create_lstm_atten(embedding_matrix):\n    K.clear_session()\n    inp = Input(shape=(maxlen,))\n    x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)\n    x = SpatialDropout1D(0.1)(x)\n    x = Bidirectional(CuDNNLSTM(40, return_sequences=True))(x)\n    y = Bidirectional(CuDNNGRU(40, return_sequences=True))(x)\n    \n    atten_1 = Attention(maxlen)(x) # skip connect\n    atten_2 = Attention(maxlen)(y)\n    avg_pool = GlobalAveragePooling1D()(y)\n    max_pool = GlobalMaxPooling1D()(y)\n    \n    conc = concatenate([atten_1, atten_2, avg_pool, max_pool])\n    conc = Dense(16, activation=\"relu\")(conc)\n    conc = Dropout(0.1)(conc)\n    outp = Dense(1, activation=\"sigmoid\")(conc)    \n\n    model = Model(inputs=inp, outputs=outp)\n    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[f1])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8293c95849968b291eeec15d734902c5017a6d68"},"cell_type":"markdown","source":"**CNN**"},{"metadata":{"trusted":true,"_uuid":"26957cbaed959c0eeb4b3402276502dd9b1d1c50"},"cell_type":"code","source":"def create_cnn(embedding_matrix):\n    K.clear_session()       \n    inp = Input(shape=(maxlen,))\n    x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)\n    x = SpatialDropout1D(rate=0.2)(x)\n    \n    conv1 = Conv1D(400, 4, activation = 'relu')(x)\n    conv1 = GlobalMaxPool1D()(conv1)\n    \n    out = Dropout(0.5)(conv1)\n    out = Dense(400, activation=\"relu\", kernel_initializer=glorot_normal(seed=12300))(out)\n    out = Dense(1, activation=\"sigmoid\")(out)\n    model = Model(inputs=inp, outputs=out)\n    model.compile(loss='binary_crossentropy', optimizer=Adam(),)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5b63c8814b43699f3edc832c1d9f5f380f3a9abd"},"cell_type":"markdown","source":"# Splitting the data"},{"metadata":{"_uuid":"efeccf5f45d469e8097f185a1d4d838c6137b0cd"},"cell_type":"markdown","source":"X_val is never seen in the training process of the base models, so it should simulate the situation of prediciting unseen data."},{"metadata":{"trusted":true,"_uuid":"43a81dcef6b0a0868a936769bad3042e426a22c5"},"cell_type":"code","source":"X_train, X_val, Y_train, Y_val = train_test_split(X, Y, random_state = 2018, test_size = 0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"46c47412aaf1bffd46e0421f1a290e86466440cb"},"cell_type":"code","source":"val_pred = {}\ntest_pred = {}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a42155a6b9338b9f22bf8cc1f09bfc4a011d5214"},"cell_type":"markdown","source":"# Training"},{"metadata":{"_uuid":"144bc69dd84828af65034a5e949b95d6ed95d28e"},"cell_type":"markdown","source":"The training approach I took was to use the epoch with the least validation loss.\nThe optimal f1 score obtained in the process is probably overestimated, since we are overfitting the validation data to find an optimal threshold. Instead, I believe when tuning single models, threshold should be a hyperparamater worthy of considering as well. But since I am stacking these models later, I do not need to pick a threshold."},{"metadata":{"_uuid":"db56bca7dcab8c0603e68c5d1311b6c41616797c"},"cell_type":"markdown","source":"**Train with GRU + Capsule**"},{"metadata":{"trusted":true,"_uuid":"02e8c0bb4071f1758d260c490bdc396080587a00"},"cell_type":"code","source":"filepath1=\"weights1_best.h5\"\ncheckpoint = ModelCheckpoint(filepath1, monitor='val_loss', verbose=2, save_best_only=True, mode='min')\nreduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.6, patience=1, min_lr=0.0001, verbose=2)\nearlystopping = EarlyStopping(monitor='val_loss', min_delta=0.0001, patience=2, verbose=2, mode='auto')\ncallbacks = [checkpoint, reduce_lr]\nmodel = create_capsule(embedding_matrix)\nprint(model.summary()) \nmodel.fit(X_train, Y_train, batch_size=512, epochs=6, validation_data=(X_val, Y_val), verbose=2, callbacks=callbacks, )\nmodel.load_weights(filepath1)\ny_pred = model.predict([X_val], batch_size=1024, verbose=2)\nval_pred['model1'] = y_pred\ny_test = model.predict([X_test], batch_size=1024, verbose=2)\ntest_pred['model1'] = y_test\nf1_score, threshold = f1_smart(np.squeeze(Y_val), np.squeeze(y_pred))\nprint('Optimal F1: {:.4f} at threshold: {:.4f}'.format(f1_score, threshold))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"024629d050c81607c37e0a5d11014177c5ef45d9"},"cell_type":"markdown","source":"**Train with attention LSTM**"},{"metadata":{"trusted":true,"_uuid":"7bebc6b5403959b4e423cdac3d13ae8ef2c982cf"},"cell_type":"code","source":"clr = CyclicLR(base_lr=0.001, max_lr=0.002,\n               step_size=300., mode='exp_range',\n               gamma=0.99994)\nfilepath2=\"weights2_best.h5\"\ncheckpoint = ModelCheckpoint(filepath2, monitor='val_loss', verbose=2, save_best_only=True, mode='min')\ncallbacks = [checkpoint, clr]\nmodel = create_lstm_atten(embedding_matrix)\nprint(model.summary()) \nmodel.fit(X_train, Y_train, batch_size=1024, epochs=8, validation_data=(X_val, Y_val), verbose=2, callbacks=callbacks,)\nmodel.load_weights(filepath2)\ny_pred = model.predict([X_val], batch_size=1024, verbose=2)\nval_pred['model2'] = y_pred\ny_test = model.predict([X_test], batch_size=1024, verbose=2)\ntest_pred['model2'] = y_test\nf1_score, threshold = f1_smart(np.squeeze(Y_val), np.squeeze(y_pred))\nprint('Optimal F1: {:.4f} at threshold: {:.4f}'.format(f1_score, threshold))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b753d12772307904a2e5a148b7c68ba24aacb3f3"},"cell_type":"markdown","source":"**Train with Conv1D**"},{"metadata":{"trusted":true,"_uuid":"b82db110188df05bd3a650463ee9d010c7d33206"},"cell_type":"code","source":"filepath3=\"weights3_best.h5\"\ncheckpoint = ModelCheckpoint(filepath3, monitor='val_loss', verbose=2, save_best_only=True, mode='min')\nreduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.6, patience=1, min_lr=0.0001, verbose=2)\nearlystopping = EarlyStopping(monitor='val_loss', min_delta=0.0001, patience=2, verbose=2, mode='auto')\ncallbacks = [checkpoint, reduce_lr]\nmodel = create_cnn(embedding_matrix)\nprint(model.summary()) \nmodel.fit(X_train, Y_train, batch_size=512, epochs=10, validation_data=(X_val, Y_val), verbose=2, callbacks=callbacks, )\nmodel.load_weights(filepath3)\ny_pred = model.predict([X_val], batch_size=1024, verbose=2)\nval_pred['model3'] = y_pred\ny_test = model.predict([X_test], batch_size=1024, verbose=2)\ntest_pred['model3'] = y_test\nf1_score, threshold = f1_smart(np.squeeze(Y_val), np.squeeze(y_pred))\nprint('Optimal F1: {:.4f} at threshold: {:.4f}'.format(f1_score, threshold))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5084a2b156ee5a04c633737b867fea8cf9e009c9"},"cell_type":"markdown","source":"**GRU + Capsul only Glove**"},{"metadata":{"trusted":true,"_uuid":"34850f184b099232e1af11ed9ee6e726fcbd277b"},"cell_type":"code","source":"filepath4=\"weights4_best.h5\"\ncheckpoint = ModelCheckpoint(filepath4, monitor='val_loss', verbose=2, save_best_only=True, mode='min')\nreduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.6, patience=1, min_lr=0.0001, verbose=2)\nearlystopping = EarlyStopping(monitor='val_loss', min_delta=0.0001, patience=2, verbose=2, mode='auto')\ncallbacks = [checkpoint, reduce_lr]\nmodel = create_capsule(embedding_matrix_1)\nprint(model.summary()) \nmodel.fit(X_train, Y_train, batch_size=512, epochs=6, validation_data=(X_val, Y_val), verbose=2, callbacks=callbacks, )\nmodel.load_weights(filepath4)\ny_pred = model.predict([X_val], batch_size=1024, verbose=2)\nval_pred['model4'] = y_pred\ny_test = model.predict([X_test], batch_size=1024, verbose=2)\ntest_pred['model4'] = y_test\nf1_score, threshold = f1_smart(np.squeeze(Y_val), np.squeeze(y_pred))\nprint('Optimal F1: {:.4f} at threshold: {:.4f}'.format(f1_score, threshold))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2f61349e3773726b5043c076d906f08832724f59"},"cell_type":"markdown","source":"# Ensembling"},{"metadata":{"_uuid":"3e9b00e1a1b5363508e4ba9be6ced6f48fbe8d32"},"cell_type":"markdown","source":"The model I used to stack the base models are lightgbm. I tried linear regression as well, which also boosted the score on holdout set but did not help a lot on the lb score."},{"metadata":{"trusted":true,"_uuid":"569b08579a2a8b4b6fec6292193c47e9d0bad135"},"cell_type":"code","source":"from sklearn.metrics import f1_score as f1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e2988e1dd04af8d1842dc5faf58f1243e0f7fcb1","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"len(val_pred)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1c32438870e7257cdebbdc718a320cda4cd963f2","_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"len(test_pred)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fe55d03b30f1cae21e35370f1863a10942982ef2"},"cell_type":"code","source":"stack_x = np.concatenate([i for i in val_pred.values()], axis = 1)\nstack_test = np.concatenate([i for i in test_pred.values()], axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0885e77afc93079ea0075028171aa73abd699b46"},"cell_type":"code","source":"print('shape of training data: ', stack_x.shape)\nprint('shape of test data: ', stack_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"154343c1d59a4d28cc9c4adf783b560947016eec"},"cell_type":"code","source":"y_test = np.zeros((56370,))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6e58230a10cb01bf1770bf420b0513cfc9697643"},"cell_type":"markdown","source":"First, split the training data into train set and validation set. The training set is not seen by the base models, but seen by the stacking model. The validation set is not seen by any model, and therefore should be able to simulate the result of predicting on unseen data. Also the size of validation data is chosen to simulate the actual test data deciding lb score."},{"metadata":{"trusted":true,"_uuid":"cf15e49562a8311e87d6614c779accbf1dab15cf"},"cell_type":"code","source":"stack_train, oof_train, stack_train_target, oof_target = train_test_split(stack_x, Y_val, random_state = 2018, test_size = 0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9d1ace85334feb2df5bcd31dc8848275cb4cdcbc"},"cell_type":"code","source":"oof_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"87bb954fbbd67581576737bebb0a07cb8e4e2e29"},"cell_type":"code","source":"oof_test = np.zeros(oof_train.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"68b56ebefca0a35bda915e145d014f158c3da0dd"},"cell_type":"code","source":"import lightgbm as lgb\nparams = {}\nparams['learning_rate'] = 0.05\nparams['boosting_type'] = 'gbdt'\nparams['objective'] = 'binary'\nparams['metric'] = 'binary_logloss'\nparams['sub_feature'] = 0.5\nparams['num_leaves'] = 10\nparams['min_data'] = 50\nparams['max_depth'] = 6","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4cc11e0bd0f4c9cc1e86c1d9e66029921bd15ecd"},"cell_type":"markdown","source":"We train the model with 10-fold cross validation, and take the mean of each fold to predict the unseen data. As said above, it would be better to pick a universe threshold instead of finding the best threshold, in order to avoid data leakage. I chose the threshold as 0.36, becasue it turned out to give the best result."},{"metadata":{"trusted":true,"_uuid":"9e69fa9f03d4fa98195feeacb6986d492c4985c7"},"cell_type":"code","source":"kfold = StratifiedKFold(n_splits=10, random_state=2018, shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"156c24a3adee18e5f806653aabc43a41308a42ff"},"cell_type":"code","source":"for i, (train_index, valid_index) in enumerate(kfold.split(stack_train, stack_train_target)):\n    stack_x_train, stack_x_val, stack_y_train, stack_y_val = stack_train[train_index], stack_train[valid_index], stack_train_target[train_index], stack_train_target[valid_index]\n    d_train = lgb.Dataset(stack_x_train, label=stack_y_train)\n    clf = lgb.train(params, d_train, 100)\n    stack_y_pred = (clf.predict(stack_x_val) > 0.36).astype(int)\n    f1_score = f1(stack_y_val, stack_y_pred)\n    print('f1 score for fold ', i, ': ', f1_score)\n    y_test += (clf.predict(stack_test))/10\n    oof_test += (clf.predict(oof_train))/10","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9bab86d15451ca42cabe9f1556dcf4ef89beaa7"},"cell_type":"markdown","source":"The f1 score for a single fold might not be stable, but it would be much more robust if we combine the mean of results from 10 fold. The result on hold out set (which hasn't been seen by any model) would reach 0.705+ with threshold 0.36. There might be better thresholds, but once again we should not always pick the optimal threshold to avoid overfitting validation data."},{"metadata":{"trusted":true,"_uuid":"c4806102665c6fcb628631da00ff51fd8cca7213"},"cell_type":"code","source":"for i in [0.30, 0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38]:\n    print('threshold: ', i, \"f1: \", f1(oof_target, (oof_test > i).astype(int)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"af8a0806d63b0b6ef210e89ee94959cb9c90879a"},"cell_type":"code","source":"y_test = y_test.reshape((-1, 1))\npred_test_y = (y_test>0.36).astype(int)\nsub['prediction'] = pred_test_y\nsub.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"00ae94b15b693513e4cfb677022aa2097374bf42"},"cell_type":"markdown","source":"# The Problem with Local CV Score"},{"metadata":{"_uuid":"2d620676f3eb93d0d87d1fc621fef88de0503861"},"cell_type":"markdown","source":"However, even the model performs well in predicting the unseen holdout set, its score on public lb is usually much lower (around 0.687) compared to the score obtained through local validation. What might cause the problem?"},{"metadata":{"_uuid":"657c15c3bde932c2e6504c025ac7d90d51c06128"},"cell_type":"markdown","source":"**Randomness**  \nFirst, it might be possible that the model just happens to fit well on the holdout data I chose. Therefore I tried to simulate different holdout data with different random seeds. According to my results the random seeds do have some effect on the result (just like the threshold! but we should not use the best one, but rather a universal one to avoid data leakage)  But the result is consistently 0.70+, with minimum of 0.695."},{"metadata":{"trusted":true,"_uuid":"3ad5b9688f8664d14bd8760e6746fd2c8085ead6"},"cell_type":"code","source":"def tune_seed(seed):\n    stack_train, oof_train, stack_train_target, oof_target = train_test_split(stack_x, Y_val, random_state = seed, test_size = 0.2)\n    oof_test = np.zeros(oof_target.size)\n    for i, (train_index, valid_index) in enumerate(kfold.split(stack_train, stack_train_target)):\n        stack_x_train, stack_x_val, stack_y_train, stack_y_val = stack_train[train_index], stack_train[valid_index], stack_train_target[train_index], stack_train_target[valid_index]\n        d_train = lgb.Dataset(stack_x_train, label=stack_y_train)\n        clf = lgb.train(params, d_train, 100)\n        oof_test += (clf.predict(oof_train))/10\n    score = f1(oof_target, (oof_test > 0.34).astype(int))\n    return score","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"786bdb7e22c3391cc467a6f79737cf76be21285b"},"cell_type":"code","source":"holdout_scores = []\nfor s in range(1999, 2020):\n    holdout_score = tune_seed(s)\n    print('f1 score at seed ', s, ' is ', holdout_score)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"011b90b048cb3ce22cb58ee5ba198035774db3a9"},"cell_type":"markdown","source":"Seems that the difference is not due to randomness? I then thought that the distribution of labels in the lb test data is significantly different from the data provided. So I calculated the distribution."},{"metadata":{"trusted":true,"_uuid":"85e9dbb5ab14be18885a512b68de529550c8357e"},"cell_type":"code","source":"#proportion of of 1s in the entire labeled data\nnp.sum(Y) / len(Y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f49b88072fd260803262e51900a5b5c00a173adc"},"cell_type":"code","source":"#number of 1s in the holdout set\nnp.sum(oof_target)/(len(oof_target))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2334567fc055b38c8cf842b4e1a1c5bcb69353eb"},"cell_type":"code","source":"#proportion of 1s in predicted test set\nnp.sum(pred_test_y)/len(pred_test_y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d92217e0d5afe01f52aa0fb33cb69f4da9915b46"},"cell_type":"markdown","source":"There seems to be a difference of about 0.007, but that might be due to my model mislabeling. So the distribution of lb test data is around the same as the labeld data provided."},{"metadata":{"_uuid":"7f65926dae8e07940c047a443d7e44e294e4de12"},"cell_type":"markdown","source":"The difference between lb score (0.687) and cv score (0.70+) should not be due to randomness or difference in label distribution."},{"metadata":{"_uuid":"37e6017c44098bb03417e08ae408461e36fa2ba7"},"cell_type":"markdown","source":"**So, what's wrong with the local validation method? Or what's wrong with the lb test data?**"},{"metadata":{"_uuid":"ec8998d905273d6cb43b0ec97bc425c62f975451"},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}